The crontab system is an embodiment of several facets of the “Unix Way”. One of the famous rules of this philosophy is the “Rule of Silence”, meaning that unless something alarming or unexpected has happened, it should remain silent. By design, the scripts you are running in your crontab should produce no output, they are simply meant to run at the assigned time and do a job.
Sadly, our software is not perfect and our scripts are fallible. Servers go offline, software updates change syntax, our old assumptions may not always be valid. So we need to find some sort of happy medium between running invisibly and constantly being in our face.
Case Study – A Backup System That Could Use Some Improvement
We are starting with a very basic backup routine. Data on our file server is stored in /storage and we want to clone it nightly to a removable USB drive mounted at /backups. This is accomplished by the following script.
We run this script nightly from crontab as follows.
So this works pretty well, our backup job executes and since the v switch is being used we exploit a side effect of a crontab – any output is emailed to the cron job owner. We end up with a nice output of the backup job, if everything runs well we see a verbose list of files synced, if something goes wrong we see that output as well.
This couldn’t be simpler.
But there is a problem. After a few days of smooth sailing it will be very tempting to skim the daily reports and just assume that everything is going well. Then as more time passes we will simple tune them out and delete these reports. The challenge we are facing is that we still want some feed back from our backup system, but not too much that we ignore it. Essentially, we want to know:
- Is the system even running?
- Did something bad happen?
Luckily, with just a little instrumentation added to our backup script, we can use this to create an RSS feed that we can follow in our favorite feed reader to see the health of our system at a glance.
Modifying Our Backup Script
For our added instrumentation we will be exploiting another side effect of the “Unix Way”. Specifically, the return value of the command. A “well behaved” program will return an exit code of 0 meaning “everything ran as expected”. Any other code will mean “something went wrong”.
We, aren’t going to dive into the different return codes and what they mean went wrong, we just want to know if it went as expected or not.
Let’s review the changes we have made:
- #!/usr/bin/env bash – Switch from sh to bash since sh might be linked or another shell that is not compatible with the variables and if statements we are using
- RSYNC_RESULT=$? – This tells the script to capture the return code of the last command executed. As we’ve discussed, 0 is good, anything else means “something went wrong”
- if [ $RSYNC_RESULT ] – If the result is 0 execute the following command, otherwise execute the later
- echo "Rsync to backup drive success, `date`" >> /storage/BACKUPS/jobstatus.txt – The results was good so we will append this new status to jobstatus.txt
- echo "Rsync to backup drive failed, `date`" >> /storage/BACKUPS/jobstatus.txt – The results was no so good, append this new status to jobstatus.txt
Using more complicated decision structures and Boolean logic tests we could create very complicated, multi-step tests, but for now let’s stick with something simple.
Now that we have our backup script added updates the jobstatus.txt the final step is to take this primitive database and use to to create an RSS feed.
Generating the RSS Feed
We aren’t going to get too deep into the explanation of this script (If you are new to Python, please check out the excellent guide, “Learn Python The Hard Way”).
Our “database” is set in the DATALOC, the script opens this file and iterates through each line. It reads in the line and assigns the first half (before the comma) to the description and assigns the date to the second half. If you want to, feel free to create a third field (separated by another comma) for the title of the RSS item.
After the script has parsed each RSS item it generates a feed our of all of these items and writes it to FEEDLOC. This location should be in accessible by a web-server running on this server. From there it’s just a matter of your feed reader picking up this update.
Below is a sample of our new crontab, first running the backup script, then running our RSS generator:
Our Daily Status Report
And with a few minutes of work we can now monitor our daily scripts and jobs from a simple glance at our RSS feed. This is definitely not suited for a real-time status, but it does well for daily or weekly tasks.
There are many improvements that could be made, better checking of status codes to report the exact nature of the error if the status is not 0, more bullet-proof scripts to begin with, but this is a good start to looking at and thinking about your repeating jobs in a different way. Keeping the status brief and to the point it can easily be glanced over and is less likely to be forgotten.