Leigh Honeywell wrote an excellent blog post for the ACLU about staying safe online. It’s mostly a primer for people who need a quick jumpstart into how to keep themselves safe, but be warned, some of it takes some time, and won’t be 100% effective. For example, scrubbing your online presence is a lot easier if you’ve never maintained much of one – search engines cache and don’t always clear. If you’ve forgotten about accounts with re-used passwords, those can still come back to bite you.
Since this bites me every time, documenting here, worked with Python 2.7.11 and OpenSSL from MacPorts. Extract Python source, edit Modules/Setup, uncomment the lines starting with SSL= (and lines following) and set that to /opt/local. Run configure with appropriate prefix and make install. An LD_LIBRARY_PATH might help too.
For pip to install with, use
And while you’re at it,
pip install gnureadline pyopenssl
We switched to JSON output from Bro when we started feeding logs into an ELK cluster. While I can still grep the raw logs, it’s a bit ugly. The ever-brilliant Justin Azoff recommended jq to me. I played around a bit with this today. The current release version of jq is 1.4, which doesn’t seem to have gmtime(). I checked out the git version and that worked, or you can get 1.5rc(whatever) from the releases page. By the way, those of you who know me, know that I generally despise the “cat file | something” convention, but I’m using it here. Sorry, not sorry.
If you just want to get the timestamps and convert them to human-readable (assuming you don’t think in epoch):
$ cat test | jq -c '.ts | gmtime' [2015,7,2,15,3,56.80596899986267,0,213] [2015,7,2,15,3,58.65949010848999,0,213] [2015,7,2,15,5,30.574909925460815,0,213] $
I used the -c flag so it wouldn’t actually break each timestamp up one line per element. Still, boring – who just wants slightly-more-readable-than-epoch timestamps? Timestamp, source and destination IPs, and the requested URI:
$ cat test | jq -c '[(.ts | gmtime), ."id.orig_h", ."id.resp_h", .uri]' [[2015,7,2,15,3,56.80596899986267,0,213],"10.0.0.2","220.127.116.11","/sites/default/files/css/css_7KJjOARp2EdJe7HGme_KJe6Y7Rq6npDiv9Uq6onbQY0.css"] [[2015,7,2,15,3,58.65949010848999,0,213],"10.0.0.2","18.104.22.168","/sites/all/themes/custom/tbs_v03/css/i/navbar/icon-facebook.png"] [[2015,7,2,15,5,30.574909925460815,0,213],"10.0.0.2","22.214.171.124","/wordpress/wp-content/plugins/cookie-law-info/css/cli-style.css?ver=1.5.2"] $
(I was checking the Beer Store hours for this holiday weekend.) The square brackets around the fields collects everything into a single array, the quotation marks around the field names tell jq to interpret those keys as literals – otherwise it gets angry about the dots in them. Parentheses allow piping the timestamp to the gmtime function.
gmtime output is pretty ugly though.
$ cat test | jq -c '[(.ts | todateiso8601), ."id.orig_h", ."id.resp_h", .uri]' ["2015-08-02T15:03:56Z","10.0.0.2","126.96.36.199","/sites/default/files/css/css_7KJjOARp2EdJe7HGme_KJe6Y7Rq6npDiv9Uq6onbQY0.css"] ["2015-08-02T15:03:58Z","10.0.0.2","188.8.131.52","/sites/all/themes/custom/tbs_v03/css/i/navbar/icon-facebook.png"] ["2015-08-02T15:05:30Z","10.0.0.2","184.108.40.206","/wordpress/wp-content/plugins/cookie-law-info/css/cli-style.css?ver=1.5.2"] $
Note the timestamps actually are in UTC. Depending on your use case, you may want to omit the -c flag – compact output including any of the longer fields isn’t a lot better to read than the raw JSON logs.
Since I just finally figured it out, after years of saying “I really should figure this out”…
- Set LANG to something that is utf-8. I used en_GB.UTF-8.
- Start screen with -U flag set.
- In irssi, set term_charset to utf-8.
And that’s that.