A few bash one liners I have found useful

Once you start working via the terminal, using gdal, python etc. you start wondering if bash might not be capable of a lot more than just triggering those nice programmes. Of course it is, but is it easy? No. Well, sort of. Sometimes the syntax (especially regular expressions) can seem arcane and impenetrable. I'm not here to rescue you, I am a duffer at this but I have learnt how to do a few things that I think are useful and above all, fast.

Let's say you want to reorder the columns of a csv file, you want to completely drop the first column and switch the 4th and 5th columns around. The following does just that and exports the rearranged columns to a new file. The $ signs indicate column numbers and their order in the list determines their order in the output.

awk -v OFS="," -F"," '{print $2,$3,$5,$4}' file1.csv > file1_e.csv

What if we want to do some more editing of the csv file? Lets say you want to remove the rows where the last entry is empty.

sed -i ".bak" '/,$/d' file1_e.csv

Or remove rows containing "nan".

sed -i ".bak" '/nan/d' file1_e.csv

Or remove both (drop the -E on Linux using gsed).

sed -i ".bak" -E '/,$|nan/d' file1_e.csv

How about renaming a folder full of files, removing some part of the name and adding something else. The following code removes the string "bs04" from all tiff file names, no matter where in the name and prefixes all with "trial".

for i in *.tif; do mv "$i" trial"${i/bs04/}"; done

Alternatively remove "bs04" and append "trial" to all png files.

for i in *.png; do mv "$i" "${i/bs04/trial}"; done

Now that the files have been renamed, lets make a folder, called "results" and move them to it.

mkdir results; mv *trial* results

There are many other blogs and pages dedicated to bash code and one liners (e.g. bashoneliners.com) but you will have to wade through a great deal of chat and geek nonsense. I hope these lines were to the point and generic enough to be useful. I shall try and update this list with useful (for me) code as I use it but lots of stuff is so specific and/or hard for me to understand, let alone explain, that it won't make it on to here.