awk saves the day (again)

A quick post on using awk to thin data.
I have a file containing thousands of coordinates recorded using dGPS every 5 seconds. The spatial density in the data is often far too high for practical purposes and made kriging a nightmare.
So I wrote the following awk command to thin the data (note that Eastings are in column 3, Northings in column 2 and Elevation in column 4)

awk -v OFS="," -F"," 'NR == 1{xo = $3; yo = $2; zo = $4; print xo,yo,zo; next} 
    {x = $3; y = $2; z =$4; xd = xo -x; yd = yo - y; xyzd = sqrt((xd^2 + yd^2 + zd^2)); if (xyzd < 10) next; else print x,y,z; xo = x; yo = y; zo = z}' Points_e1.csv > Points_e2.csv

Breaking it down:

1. firstly, set the output delimiter set to “,” using -v OFS=","
   • then the input delimiter to “,” using -F","
   • at line one set the initial values for x,y,z and print them
2. set new values for x,y,z to those from current line
   • calculate differences to the old values
   • calculate 3D distance
   • if the distance is less than 10 then skip to the next line without doing anything more
   • if the distance is greater than 10 then print the current values of x,y,z and then put them into the variables for the “old” values and move on

Running this in the command line was so quick that at first I thought it must have failed.

​

Python shapely

I have to admit that I haven't scripted anything using "shapely" yet so I thought I should give it a try. Well there's a long way round to installing it and a short way round.
The long way:

Read the instructions:

GEOS >=3.3 (Shapely 1.2.x requires only GEOS 3.1 but YMMV)

Try to install GEOS and find that it requires SWIG for the Python bindings.

Try to install SWIG on Mavericks only to find it requires PCRE to install.

Install PCRE, install SWIG, then try GEOS

    make: *** [check] Error 2

There's a ticket on this from 5 years ago. Great!

The short way:

I then recall that GEOS is included in the KyngChaos installer for GDAL and sure enough there is GEOS and shapely, but shapely is only installed for the system Python. I am using Anaconda. Put a .pth file into the site-packages folder under Anaconda and we are up and running.

Sometimes it is very easy to make life difficult.

Yosemite will work if you throw enough money at it

Quick update on Yosemite on my MacBook Pro from 2010. For about a little over a month now I have been struggling to run more than one programme at a time due to Yosemite’s memory hogging. Having to sudo purge all the time was a pain and didn’t really help too much. In the end I bought 8 Gig of RAM and put that in to my machine. It has helped. I now have a functioning machine again.
Thanks for nothing, Apple.

​