Using jq with Bro logs

We switched to JSON output from Bro when we started feeding logs into an ELK cluster. While I can still grep the raw logs, it’s a bit ugly. The ever-brilliant Justin Azoff recommended jq to me. I played around a bit with this today. The current release version of jq is 1.4, which doesn’t seem to have gmtime(). I checked out the git version and that worked, or you can get 1.5rc(whatever) from the releases page. By the way, those of you who know me, know that I generally despise the “cat file | something” convention, but I’m using it here. Sorry, not sorry.

If you just want to get the timestamps and convert them to human-readable (assuming you don’t think in epoch):

$ cat test | jq -c '.ts | gmtime'
[2015,7,2,15,3,56.80596899986267,0,213]
[2015,7,2,15,3,58.65949010848999,0,213]
[2015,7,2,15,5,30.574909925460815,0,213]
$

I used the -c flag so it wouldn’t actually break each timestamp up one line per element. Still, boring – who just wants slightly-more-readable-than-epoch timestamps? Timestamp, source and destination IPs, and the requested URI:

$ cat test | jq -c '[(.ts | gmtime), ."id.orig_h", ."id.resp_h", .uri]'
[[2015,7,2,15,3,56.80596899986267,0,213],"10.0.0.2","50.56.3.60","/sites/default/files/css/css_7KJjOARp2EdJe7HGme_KJe6Y7Rq6npDiv9Uq6onbQY0.css"]
[[2015,7,2,15,3,58.65949010848999,0,213],"10.0.0.2","50.56.3.60","/sites/all/themes/custom/tbs_v03/css/i/navbar/icon-facebook.png"]
[[2015,7,2,15,5,30.574909925460815,0,213],"10.0.0.2","160.153.16.10","/wordpress/wp-content/plugins/cookie-law-info/css/cli-style.css?ver=1.5.2"]
$

(I was checking the Beer Store hours for this holiday weekend.) The square brackets around the fields collects everything into a single array, the quotation marks around the field names tell jq to interpret those keys as literals – otherwise it gets angry about the dots in them. Parentheses allow piping the timestamp to the gmtime function.

gmtime output is pretty ugly though.

$ cat test | jq -c '[(.ts | todateiso8601), ."id.orig_h", ."id.resp_h", .uri]'
["2015-08-02T15:03:56Z","10.0.0.2","50.56.3.60","/sites/default/files/css/css_7KJjOARp2EdJe7HGme_KJe6Y7Rq6npDiv9Uq6onbQY0.css"]
["2015-08-02T15:03:58Z","10.0.0.2","50.56.3.60","/sites/all/themes/custom/tbs_v03/css/i/navbar/icon-facebook.png"]
["2015-08-02T15:05:30Z","10.0.0.2","160.153.16.10","/wordpress/wp-content/plugins/cookie-law-info/css/cli-style.css?ver=1.5.2"]
$

Note the timestamps actually are in UTC. Depending on your use case, you may want to omit the -c flag – compact output including any of the longer fields isn’t a lot better to read than the raw JSON logs.