Next Generation Cron With systemd Creating A Timer

Do you need to schedule some task in the future on your computer? This may look simple - after all, your dishwasher is able to wait before launching with the help of a button - but sometimes computers is making such simple tasks so hard.But if you have some background, you will have probably heard of cron, this piece of software fully dedicated to launch the right task at the right time. But this tool has really been designed with simplicity in mind and you may in the end have bad surprises.If you ever succeeded to schedule a task on Windows, you have used the Windows Task Planner. It does have a GUI by default but it doesn't make it so simple to use as well: these two systems just launch a process at a fixed time and date.

In order to understand how systemd can be helpful to you there, I'll take an example.

What pitfalls systemd timers will avoid you?

If you ever own a machine with data you care about in it, you'll want to have a copy of your data in another, probably safer place. If you manage a server, it's mandatory: after all, how will you recover if your hard disk fails and prevent you to recover any data?

So as a responsible person you set up backup every week or every day. You may set it up using cron, you schedule it at 4:24 AM, but here starts the problem: what if your server is shutdown from 4:10 AM to 4:30 AM for any reason?

Well it's likely cron will just skip that backup. This might be critical if that happens often and silently or if your code relies on the fact it runs and it may fail otherwise. Generally this happens when you set up a cleanup task via cron and it doesn't launch. Suddenly your code may have insufficient space to continue and will break - it's sad, so sad situation, right Mr Elton John.

However, if a missed launch can be a problem, imagine one second - wow, John Lennon now? - that your task is too much slow. If your task is set to run every 10 minutes but takes 15 minutes to complete, cron or Windows will happily launch another task even if the current task isn't gone yet - and so, you'll have 2 instances of your task running concurrently, which is the perfect recipe for disaster. When a program is running concurrently while it's not designed to do so, it will really likely corrupt files, other softwares, databases - and your server becomes suddenly a sinking ship like Titanic.

OK, maybe I am going too far with Titanic but you get the idea. While systemd couldn't have done much to save this ship, it can help you with all these shortfalls and ensure  you a longer Christmas vacation thanks to the bugs it will avoid you. It's time now to get to know how to set up systemd timers.

How to schedule automated server backup?

First of all, systemd timers triggers a systemd service, so before scheduling your task, you'll need to make it first a service. Fortunately, I've written a guide to create systemd service, this way it will introduce you with systemd's way of working. You should read it before going on. Unless if you exactly know what you're doing, your systemd service file should not contain any WantedBy= setting. If you want to start your service at a specific time, you probably don't want to start it on boot.

Thanks to the systemd service system, it's impossible to have multiple instances of your task running by mistake: if a task is already running, it will just skip that launch and leave the currently running task finish its job.

Once you have a systemd service to schedule, create a file with the same filename as your service except that it should end with .timer instead of .service. In our automated  backup example, the service would be automated-backup.service and the timer would be automated-backup.timer. Both files should be in the same directory. As I told you in the systemd service article, I recommend you to write these files in a normal place such as your home directory and then, copy them to a systemd folder, once you've finished your edits.

So, let me show you what our timer file is looking like:

[Unit]
Description=Schedule backups during off peak hours
[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=7200
Persistent=true
[Install]
WantedBy=timers.target

Much like in systemd services, there's 3 sections. [Unit] or [Install] work exactly the same as explained in my systemd services article. Please note that WantedBy= is important here because timers can be started or stopped, so if you don't tell systemd to start your timer during boot, it will never trigger. timers.target is a special systemd target for timers.

Now, the [Timer] section. Inside it, you'll find all settings related to when the timer should trigger. For our automated backup, I've told systemd to run it between 3 AM and 5 AM at server's timezone. The exact time is random on each day.

OnCalendar= sets the timer related to your server's time (wallclock), such as every Sundays at 1 PM. If you have used cron previously, you should be really familiar with this syntax. However it has some added benefits.

For example, if you want something to happen hourly, you can do like this:

OnCalendar=hourly

and daily:

OnCalendar=daily

In fact, it supports all of the following values:

1. minutely
2. hourly
3. daily
4. monthly
5. weekly
6. yearly
7. quarterly
8. semiannually

There's however a problem with these keywords: for example, daily triggers always a midnight, which is often a peak hour in computing systems. That's why it's recommended to use RandomizedDelaySec= (its usage is specified below). Anyway for backup it's not a good option: midnight isn't off peak hours, it's rather the reverse. So we need to set more accurately when we want to see that task launched.

If you want more control, you can write a date like 2018-12-06 12:49:37. Well if you're that specific you'll just trigger the timer once. To make it recurrent, you'll replace any of these elements by * asterisk.

OnCalendar=*-*-* 03:00:00

As you can see above, in our backup example, all the date part is *-*-*, meaning it should occur every day of every month of every year. Now if you do:

OnCalendar=*-12-25 03:00:00

Then it runs every 25th December at 3 AM. Perfect systemd timer for Santa Claus - even if I doubt he will ever need one! So asterisk adds recurrence where you put it. If you put it in year field, it means “every year”, etc.

Finally, you can add UTC at the end of the line to use UTC time instead of local timezone. For example, some services reset their API quotas at midnight but to avoid any timezone bias it uses UTC. So for such tasks, you would do:

OnCalendar=daily UTC

Now, let's solve another problem: rush hours. systemd has also a setting to fight against that.

RandomizedDelaySec= allows to delay the task of a random amount of time. The value is the max number of seconds the timer will delay. It's specifically intended for such cases. You remember that in systemd, daily always triggers at midnight? Well, weekly always triggers at Monday midnight, and yearly triggers at January 1 midnight, one of the worst peaks in the year with network outages everywhere. You certainly don't want that to happen.

By adding a delay, you remove that problem: it will automatically delay at a unknown time your task. Randomness here is important because it's far more likely to be even when it's random and an even load allows to better optimize your tasks.

Say you need to run your tasks around 7 AM for the morning but you want to allow a small delay of max 15 minutes, you would do like this:

RandomizedDelaySec=900

That should be enough for delays. Sometimes even milliseconds delays are enough to prevent unintended spikes.

Persistent= takes care of missed timer triggers. What if your server is shutdown during night? Well, the backup would never trigger at all. Setting it to true allows systemd run it on the next boot in such cases. This way you know in one way or another, the timer's task will be run. Its usage is simple, you just do this:

Persistent=true

This has however one drawback that's really hard to avoid anyway: when multiple tasks from different timers are missed, they will all run at boot and slow down that boot. In my opinion that's much better than if it never runs and after all that's normal, the most appropriate moment to run the timer is when it is scheduled, afterwards it will be probably inappropriate anyway.

OnBootSec= is the last option I'll show you (but not the least). It's if you want to trigger a timer some time after boot instead of based on calendar. For example, if you need to check on startup if your server is started properly and working as intended, you could write a check service and use that timer setting to trigger it after the system had enough time to boot.

Let's say the system needs 3 minutes to boot, you could do:

OnBootSec=180

And despite its name, you can also do:

OnBootSec=3 minutes

If you precise both OnBootSec= and OnCalendar=, it will start the service whenever any of these 2 events happen.

Okay, now it's time to save your file, copy it to the system folder if you followed my advice above, and test if your timer is working properly.

Enable your new timer and monitoring

In order to test your new timer, you have to tell systemd you added a new timer, so you need to type this command:

$ sudo systemctl daemon-reload

Now, systemd will take in account your new timer and look closely on when to run your task. As systemd is always running, it's after all one of the best candidates to manage and run your scheduled tasks.

One thing you might find counterintuitive though: a timer is by default disabled. In order to enable it, you need to do this command:

$ sudo systemctl enable --now automated-backup.timer

You will then probably want to see if your timer acts as expected. Good news: systemd is even kind enough to have a command telling you when it was last launched and when the next launch is scheduled (except if timer is set to run only at boot, as systemd doesn't know when the system will boot again, obviously). Here's that command:

$ systemctl status automated-backup.timer

Finally, when you no longer need the timer, you can disable it as well:

$ sudo systemctl disable --now automated-backup.timer

Conclusion

Using systemd timers, your management of scheduled tasks are to a next level: honestly, I personally feel like scheduled tasks should have been this way since years.

Oh, one little surprise for you: all systemd timers are logged in a well structured system with filtering, log rotation and all the like. So I invite you to see how you can see logs about your scheduled tasks!