After=time-sync.target. Then, if you’re using
systemd-timesyncd.service, enable systemd-time-wait-sync.service(8). Otherwise, tweak your NTP service to create /run/systemd/timesync/synchronized, or write a service that blocks until the time is synchronized.
I am using Orange Pi PC 2 and Zabbix to collect statistics about my local network. PC 2 is an excellent device for this: small, silent, and power-efficient. It also boots up immediately when the power is applied, so it automatically restores itself after power outages. I’ve been using this board for over a year now, and it never required any maintenance.
Imagine my surprise, then, as I opened the stats the other day and found that Zabbix is not running! (Who’s gonna watch the watchers, right?) Luckily, the logs had a complete picture of the situation, but it requires a bit of context to explain.
Orange Pi PC 2 lacks a hardware clock, meaning it can’t count the time while it’s turned off. To compensate for that, Armbian (the distro I’m using) employs two techniques. First of all, it occasionally saves the current time to disk. Upon boot, this time is restored. That’s obviously not very accurate, so Armbian then waits for the network to come up, and uses NTP to synchronize the time to within milliseconds.
While reading a file from a disk is near-instant, waiting for network and NTP is not. Furthermore, the wait doesn’t block the boot process, so Zabbix was launched right away. Once Chrony (the NTP client) got online, it jerked the time full 17 minutes ahead — a move that impressed Zabbix so much that it promptly died. (
journald noted that “the service has successfully entered the ‘dead’ state”. How cheerful.)
Now on to the solution. Armbian 20.08.17 uses systemd 241, and that provides a special unit called
time-sync.target. According to the systemd.special(8), “all services where correct time is essential should be ordered after this unit”. Perfect! Except it doesn’t work: on Armbian, the target is reached right after Chrony start-up, without waiting for it to synchronize.
Clearly, I needed to delay the target until the synchronization is complete. A bit of search led me to this NixOS pull request, which inserts a new service in between Chrony and
time-sync.target. That service runs
chronyc waitsync to, um, wait for the sync. This worked for me, but the solution felt a bit too obscure, so I decided to blog about it.
It’s while conducting the due diligence that I stumbled upon systemd-time-wait-sync.service(8). Added in systemd 239, it does what the NixOS PR did, but more generically: it waits for either /run/systemd/timesync/synchronized to appear, or for adjtimex(2) call to happen. The latter approach is unreliable, so the file always takes precedence.
systemd-time-wait-sync.service and doing a few test reboots, I got to experience the unreliability first-hand. Chrony made such a small adjustment that it didn’t register with the waiting service, and a dozen units just hung, waiting for the sync to happen. To get around that, I ditched Chrony and switched to
systemd-timesyncd.service, which touches the file mentioned above and thus reliably unblocks
Long story short, here are the steps to make a systemd service wait until the system clock is synchronized:
make sure you’re using
sudo apt remove chrony sudo systemctl enable --now systemd-timesyncd.service
Alternatively, you could amend your existing time-synching tool so that it creates /run/systemd/timesync/synchronized once it finishes synchronizing.
As yet another alternative, you can write a helper service that blocks until the time is synchronized, and order it in between
time-sync.targetand your time-synching tool. You then make the service pull both the
time-sync.targetand the helper. See this pull request for an example of this approach.
time-sync.targetwait until the clock is synchronized:
sudo systemctl enable --now systemd-time-wait-sync.service
make the service wait until
sudo systemctl edit <name>.service
An editor will open; type this in:
[Unit] After=time-sync.target Wants=time-sync.target
If you’re the author of the service that you’re delaying, you can just edit its .service file directly;
systemctl editis only necessary for the services provided by your OS.
to test the result, reboot and run the following:
sudo journalctl -b \ -u systemd-timesyncd.service \ -u systemd-time-wait-sync.service \ -u time-sync.target \ -u <name>.service
This will output a part of the boot log, from which it should be clear that your service doesn’t start until the clock is synchronized.
Drop me a line! (wonder where’s the comments form?)