tail: can tail multiple files simultaneously, who knew? and other tail tricks.

After using tail for a long time, I've only recently had a need to became familiar with tail's ability to watch multiple files.

you can easily watch a single file, thats the tail we all know and love.

tail -f /var/log/syslog

But I've got another rsyslog directory that concentrates logs from a bunch of different servers with specific naming conventions that I can match by filespec.

For each cluster there are multiple app, database, loadbalancers and memcache servers. trying to debug a problem, I needed to tail all of the app servers at the same time.

it's dead simple, particularly if you are in the directory all the files you want to tail reside.

tail -f *production-app*

Where it matches any filenames that contain "production-app".

If I need to watch the mysql servers of my testing cluster

tail -f *testing-mysql*

Incidentally you can also tail multiple files without using a filespec

tail -f /var/log/apache2/access.log /var/log/apache2/error.log

it tails the files that match the filespec and interleaves the output with markers so that you know which log file you are looking at. Beautiful.

Other tail tricks:

combine with grep to watch for your needle before it gets buried in the haystack:

tail -f /var/log/syslog | grep "my needle"

combine with grep to exclude a bunch of annoying messages that you don't need.

tail -f /var/log/syslog | grep -v "annoying message I don't want to see"

Here's one that I commonly use to cut the cruft out of watching logs on my EC2 instances, it eliminates any lines with Connection OR Kernel in them.

tail -f /var/log/syslog | grep -v 'Connection\|kernel'