One liners

Today, I was running several simulations with torchcvnn, each of them saving their results a metrics.csv file within a specific directory. Each metrics.csv file was like :

16,33.1491584777832,33.149155968594066,0.8940797582559719

with an index followed by the metrics.

The question was : how to easily compute the mean and standard deviation of the metrics by accumulating the measures from all the simulations.

First, we can locate all the metrics file and build a list of these filenames

find . -name "metrics.csv" | tr "\n"  " "

The tr command allows to remove the “new line”, in order to have a sequence of filepaths on a single line.

And then, serialize a second command, this time using awk based. The computation of the standard deviation is the difference between the mean of the squared valued and the squared mean value.

for f in $(find . -name "metrics.csv" | tr "\n"  " "); do cat -- "$f"; printf "\n"; done | awk -F ',' '{s+=$3; ss+=$3^2} END{print m=s/NR, sqrt(ss/NR-m^2)}'