Analyzing Log Files With Dave Taylor

Remember my previous post about Carl? It was about the set of automated scripts we use to monitor our Lighttpd logs.

In the recent issue of Linux Journal, Dave Taylor, the 26-year veteran of UNIX, shows how to analyze your logs old school:
http://www.linuxjournal.com/article/9168

For me it is always amazing to see how you can get two or three simple UNIX utilities and do a complex task in one command. Pure magic.

I loved the scripts so much that I went to our 1Passwd web server and modified them to query the information I need. Here is a few examples:

This command will print the list of the status codes generated by your server:

awk '{print $9}' lighttpd_access.log | sort | uniq -c | sort -rn

In my case, it displayed and it looks like the server is working OK — there were just three 404 (Page Not Found) errors. I am still not sure how to interpret the 206 code though:

Another interesting command is this:

awk '{print $11}' lighttpd_access.log | grep -v 'http://1passwd.com' | sort | uniq -c | sort -rn | head -10

It prints the list of top 10 referrers, excluding the http://1passwd.com itself:

The best part about using these commands is how easy it is to change them to do what you need.

Comments

[Dave Taylor]

Thanks for the positive words about my column! I'm not entirely sure what the error 206 means, but it's definition is referenced in this somewhat cryptic note on the Apache.org site: "apache replies 206 partial content instead of 401"

:-)