Using swatch to Monitor Logfiles

A very useful tool for monitoring logfiles is Swatch (aka Simple WATCHer of Logfiles). This is a perl program that essentially watches a logfile. You can run as many instances of it as you want, but you can only watch one logfile per instance.
Depending on the log file you want to watch, you may have to run swatch with some additional privilege. For instance, /var/log/messages is likely to be the primary target for anyone interested in using swatch. But this file should be readable by root on most, if not all, systems. So Joe User can't just monitor the logfile.
A good solution here is to use sudo to give root priv to a given swatch command. This reduces the risk of a) opening a root shell and using it to execute swatch and b) giving root access to swatch, thus allowing one to view any file they want. For instance, you may use the following in your /etc/sudoers file:

joe     ALL = /usr/local/swatch/swatchmsgs

Then create the /usr/local/swatch/swatchmsgs script which may look like this:

#!/bin/sh
/usr/bin/swatch --config-file=/usr/local/swatch/swatchrc \
  --script-dir=/usr/local/swatch \
  --tail-file=/var/log/messages

And make sure that the swatchmsgs script is mode 700 so that no user other than root can modify it (to change the --tail-file, for example). It is probably a good idea to make the /usr/local/swatch directory accessible only by root as well, and make /usr/local/swatch/swatchrc (the configuration file) mode 600.
So now, user joe can execute:

$ sudo /usr/local/swatch/swatchmsgs

and watch the log messages according to the system configuration in /usr/local/swatch/swatchrc. I suggest taking this sort of precaution because logfiles are sources for a wealth of information and you don't want the wrong person able to browse them.
Configuring swatch is pretty easy. The configuration file has an extremely straightforward syntax. A typical stanza for something to monitor begins with the keywords watchfor or ignore. Each keyword takes a perl regular expression to match the message you wish to act upon. Following that, you can take a number of actions upon the matched line. You can use these actions:

• echo [modes]; this echos the matched line to stdout and the text mode can be normal, bold, inverse, a color (red, green, magenta, etc.) or a highlighted color (red_h, green_h, magenta_h, etc.).
• bell [N]; echo the matched line and send a bell N times (default is once).
• exec command; executes command, and command may contain variables with are substituted with fields from the matched line. Ie. $N will be replaced with the Nth field of the line ($3 being the third field), while $0 or $* will be replaced with the entire line. (Note that this is why you want your configuration file to be writeable only by root if you do need root privilege to view a logfile).
• mail [addresses=address:address:...][,subject=your_text_here]; send an email to the address(es) containing the matched lines as they appear in the body of the message, with the specified subject.
• pipe command[,keep_open]; pipe matched lines into command. If you specify keep_open then the pipe will be forced to remain open until a different pipe action is run or until swatch exits.
• write [user:user:...]; use write to send matched lines to user(s) (the lines will show up on any terminals they have open).
• regex]; use this action to limit the number of times that the matched pattern has actions performed on it. If use=regex is specified, it will cause throttling to be based on the regular expression defined than on the returned message (using the message is the default).
• continue; this action causes swatch to continue to try to match other stanzas when it is finished with the current one.
• quit; causes swatch to clean up and quit immediately.

There is one other special action that can be used with any of of the above:

• when=day_of_week:hour_of_day; this specifies a window of times and days that this action can be performed.

For example, to send an email to a pager Monday through Saturday, from 8:00am to 5:00pm, you would use:

mail=pager@somehost.com,when=1-6:8-17

All very straightforward. The big thing is to know some perl regular expressions so you can fine-tune your rules. Let's look at a few examples.

# Bad login attempts
watchfor   /INVALID|REPEATED|INCOMPLETE|[Ff]ail /
        echo magenta_h
        bell 3
        mail addresses=sysadmin\@myhost.com,subject=Bad_login_attempt
        throttle 01:00

This stanza will send an email to the specified email address (note you have to escape the @ sign, like you would in a perl script), with the subject "Bad login attempt". It will echo the message to the screen in highlighted magenta and will throttle the message to display once per minute, and ring a bell three times.

# Logins, password changes and such
watchfor   /workstation.*(LOGIN|passwd|ruserok|session opened|session closed)/
        echo yellow
        bell 3
watchfor   /firewall.*(LOGIN|passwd|ruserok|session opened|session closed)/
        echo yellow_h
        bell 3

This is two different stanzas grouped together. They are almost identical except one uses a highlight yellow, and the regular expression to watchfor is slightly different. In this example, messages coming from the machine named "workstation" are echoed in yellow, while those coming from the machine named "firewall" are yellow highlighted. This illustrates how you can configure swatch on a central loghost to differentiate between messages from different machines. If you wanted, you could have emails sent for these attempts on the firewall machine, and nothing on the other machine. Specifying the machine name (on a machine that receives logs for more than just itself) is useful when trying to write some fine-grained rules.
Rules can be small and simple as well. Remember, swatch only displays what you tell it to, so if you want to see something without making it fancy, you still have to write a rule for it.

watchfor   /(su|sudo):/
        echo bold

watchfor  /firewall kernel.*(DROP|REJECT)/
        echo blue_h

watchfor   /rsyncd.*from/
        echo

This is three very simple rules. The first just bolds anything having to do with su or sudo. The second shows any iptables DROP or REJECT logs from the firewall machine, and the last simple echos to the screen (with absolutely no highlighting or coloring), messages from rsyncd that contain the word "from" (these lines contain the IP address or reverse-DNS name of connecting systems), so we exclude statistical stuff from rsyncd like how much was written or received.
As you can see, swatch is fairly flexible, particulary due to the use of perl regular expressions. And colorizing the output is nice if you have a virtual desktop you can spare to maximize an xterm (or transparent borderless Eterm which looks very slick) and watch swatch's output. It doesn't prevent you from having to read log files; no program can really do that, but it can reduce the "noise" so that you're only reading what is important (to you).