ArcMap

Struggling to design final project for my students. Backwards and forwards between ArcMap 10.6, ArcGIS PRO and QGIS. A fun little observation from ArcMap Not sure how it manages that. But checking the stats under "Source" it seems to have the mean and the stdev swapped. All good fun. I really wish the SCP toolbox in QGIS was easier to use, I might let my students use then. As it is now, no chance.

Just checking in

Nothing new under the sun but Google sent me a gentle nudge. Well, not nothing new. I now have my PhD and am teaching GIS and RS in a town called Gävle.

awk saves the day (again)

A quick post on using awk to thin data.
I have a file containing thousands of coordinates recorded using dGPS every 5 seconds. The spatial density in the data is often far too high for practical purposes and made kriging a nightmare.
So I wrote the following awk command to thin the data (note that Eastings are in column 3, Northings in column 2 and Elevation in column 4)

awk -v OFS="," -F"," 'NR == 1{xo = $3; yo = $2; zo = $4; print xo,yo,zo; next} 
    {x = $3; y = $2; z =$4; xd = xo -x; yd = yo - y; xyzd = sqrt((xd^2 + yd^2 + zd^2)); if (xyzd < 10) next; else print x,y,z; xo = x; yo = y; zo = z}' Points_e1.csv > Points_e2.csv

Breaking it down:

1. firstly, set the output delimiter set to “,” using -v OFS=","
   • then the input delimiter to “,” using -F","
   • at line one set the initial values for x,y,z and print them
2. set new values for x,y,z to those from current line
   • calculate differences to the old values
   • calculate 3D distance
   • if the distance is less than 10 then skip to the next line without doing anything more
   • if the distance is greater than 10 then print the current values of x,y,z and then put them into the variables for the “old” values and move on

Running this in the command line was so quick that at first I thought it must have failed.

​

Python shapely

I have to admit that I haven't scripted anything using "shapely" yet so I thought I should give it a try. Well there's a long way round to installing it and a short way round.
The long way:

Read the instructions:

GEOS >=3.3 (Shapely 1.2.x requires only GEOS 3.1 but YMMV)

Try to install GEOS and find that it requires SWIG for the Python bindings.

Try to install SWIG on Mavericks only to find it requires PCRE to install.

Install PCRE, install SWIG, then try GEOS

    make: *** [check] Error 2

There's a ticket on this from 5 years ago. Great!

The short way:

I then recall that GEOS is included in the KyngChaos installer for GDAL and sure enough there is GEOS and shapely, but shapely is only installed for the system Python. I am using Anaconda. Put a .pth file into the site-packages folder under Anaconda and we are up and running.

Sometimes it is very easy to make life difficult.

Yosemite will work if you throw enough money at it

Quick update on Yosemite on my MacBook Pro from 2010. For about a little over a month now I have been struggling to run more than one programme at a time due to Yosemite’s memory hogging. Having to sudo purge all the time was a pain and didn’t really help too much. In the end I bought 8 Gig of RAM and put that in to my machine. It has helped. I now have a functioning machine again.
Thanks for nothing, Apple.

​

Yosemite, no cat-like reflexes found here

Well, I have waited with this post for a while, I wanted to give Yosemite a chance. I had been running Lion on my MacBook Pro (2010) for some time but a few weeks ago I followed the advice and listened to AppStore's incessant nagging. I upgraded to Yosemite. This was a big mistake, as clunky as Lion was, it zoomed and zipped along in comparison to Yosemite. Apparently, Yosemite really requires a great deal of RAM, my machine has only 4GB but 16GB would be optimal, oh and an SSD. I understand that (sort of, although I'm not sure I understand the perceived need for that sort of "bells and whistles" performance) but what I don't understand is Apple's recommendation to upgrade. They knew that my machine couldn't handle it so this can only be explained as a shortsighted, shareholder driven manipulation to get me to buy new hardware. It has failed in my case; I have been considering switching to Linux for some time now, as OS X has progressively become bulkier, slower (for my workflow) and buggier and so when I finally upgrade it will be away from Apple. Can I really stand the ugliness of Ubuntu? I guess my choices are limited.

aRrrrrrrrgghhh!

A recent excursion into "R" has left me raging, embarrassed and tired.

Many people I respect and like use R in their work and like the way it works. It is backed up by "R Studio" as well as help files and a large, user community. After an enormous amount of frustration caused by Pandas for Python I thought I might try R instead. Let's start with the good points. R Studio is a great way to use R. It is for R what "Spyder" would like (but fails) to be for Python. It is considerably more stable and smooth than Spyder and looks good too. R Studio provides some pretty useful tricks for coding, such as running only the marked text in a script file or running step by step, line by line.
R itself comes with many packages and tools, especially for statistics. This is the main focus of R: statistics for ecologists and biologists. This explains and is explained by the strong connexion with these research communities. Plotting (at least with the standard "plot" function and not the hideous "ggplot") is simple and elegant, much easier than "matplotlib" in Python.

The bad points. If you are new to scripting then R is probably fine, you will learn its "psychology" and work with it. If you have learnt any other language first then R will feel unreasonably and pointlessly quirky and counter-intuitive. Indexing is confusing, lists have names for their elements, which might set them apart from vectors, except that so do vectors. Many hours have been spent by many people developing the packages for R, which provides all that functionality, but why R? It doesn't provide any data structure that is superior to other scripting languages. It's not as if "factors" provide a quick and easy mechanism for sub-setting and analysis, certainly no easier than in other languages. The help files are many and mostly useless for the newbie. They read and look like poorly written, first draft man pages.

I suppose my biggest complaint about R is "what's the point?" All that time and effort could have been put into another language and the statistics would work just as well. As I have stated, R provides no intrinsic advantage for statistical calculations, it is simply that engaged enthusiasts have written the libraries in R. Learning another language is fine but in this case, the language sits at some uncomfortable angle to most others and provides no clear advantage, making learning frustrating but without any real incentive. I suppose I shall just have to learn to accept Pandas and its idiotic time stamp behaviour and Python's general love of bloating with evermore object types for no good reason whatsoever.

Or just stop bitching about this sort of thing.