Time Difference Warning on Prometheus Server UI

Last Updated:

Prometheus relies heavily on time series data. A time drift can lead to inaccurate data collection, affecting the reliability of metrics and graphs.

When visiting your Prometheus server, you may be presented with the following warning:

Warning! Detected 33.52 seconds time difference between your browser and the server. Prometheus relies on accurate time and time drift might cause unexpected query results.

Prometheus collects data that includes timestamps, marking exactly when each piece of data was gathered. This approach is known as time series data collection.

If there’s a mismatch between the time recorded by Prometheus and the time on your browser, the data you see might be incorrect or misleading.

How to Fix this Warning

First, check if the time on your computer is accurate and synchronized. If your computer’s time is correct, and the issue persists, then follow these steps.

For Prometheus running on a Linux server, begin by verifying the server’s time. Ensure that it is set to the correct timezone and that the time is accurate.

To see the time on a Linux machine, run date command:

date

The date command will return the date and time of the server.

It’s possible that the server time is behind or ahead only a few seconds, so it needs to be accurate.

If the date and time of the server is different, you need to adjust it.

We have a few options here.

The recommended approach is to have a time service running like NTP or Chrony. These services will make sure the time is always synchronized against the time servers you define on their configurations.

If you server is already using these services, you can check their configuration.

To see if NTP (Network Time Protocol) is running, use systemctl:

sudo systemctl status ntpd

If you server is using NTP, you should get the status of the service as “Running”.

You can do the same to check the status of chrony:

sudo systemctl status chronyd

If any of those are installed, a service restart is recommended to see if that fixes the issue:

sudo systemctl restart ntpd

or

sudo systemctl restart chronyd

If that does not fix the issue, check the configuration of these files and look for the time servers they’re configured against.

After restarting the NTP service, check the system time again with the date command, and make sure it’s now synced with your local machine (browser).

Install and Configure NTP or Chrony

In case your server time is set statically and it does not talk to a time server, you can install either NTP or Chrony on the server and configure the service.

For the purpose of this guide, I’ll install chrony

On Redhat/CenOS systems:

sudo yum install chrony

On Debian-based systems:

sudo apt install chrony

Then edit its configuration file which is located at /etc/chrony.conf and add the following:

cat /etc/chrony.conf
server time.google.com iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
keyfile /etc/chrony.keys
leapsectz right/UTC
logdir /var/log/chrony

On that configuration, the time (NTP) server is time.google.com, which is a public time server provided by Google. You can use a different time server based on your location, just Google “Public Time Servers” to get list of time servers.

After adding that to /etc/chrony,conf, you can now restart the chronyd service using systemctl:

sudo systemctl restart chronyd

And check it’s status to make sure it’s running:

sudo systemctl status chronyd

It should be ‘active’ and ‘running’

Now check back the Prometheus server and see if the warning has disappeared.

How to set the Linux System Time manually

If you don’t want to use a NTP server for whatever reason, these instructions can help you to set your system time manually

Set the time:

sudo timedatectl set-time 'YYYY-MM-DD hh:mm:ss'

For example:

sudo timedatectl set-time '2023-11-28 19:57:00'

Replace the date and time in that snippet with your current date/time.

To set the timezone:

sudo timedatectl set-timezone America/New_York

To see a full list of timezones, run:

sudo timedatectl list-timezones

This will get you a list of all available timezones.

Conclusion

As Prometheus relies on time-series data, it’s crucial to make sure the Prometheus server time is synchronized otherwise inaccurate data may be presented or even cause delays on your incident alerting systems. In this guide, we walked through the process of synchronizing the time on Prometheus server to resolve the time difference warning displayed on Prometheus UI.

RECENT POSTS

Get Ops Pro Tips in Your Inbox!