Setting the Timezone and Locale on Linux

2018-10-01 · linux, timezone, locale

Timestamps that disagree with each other are a quiet source of wasted hours. The MySQL slow log says 09:14, nginx says 00:14, the cron job that "ran at midnight" shows up at 15:00 in the mail, and it takes a while to realise the server is on UTC, the database was started with a different TZ, and the developer's laptop is on KST. Locale problems are subtler: sort orders differently, Perl warns on every login, and a script that worked on one box prints ? instead of Korean characters on another.

I've chased that kind of mismatch enough that timezone and locale are now the first thing I fix on a new image. Here's how on CentOS 7 and Ubuntu 16.04/18.04, plus the two services that don't inherit the system setting.

What do you have now?

timedatectl
localectl status
locale

timedatectl shows the current time in local and UTC, the configured zone, and whether NTP is active. localectl status shows the system locale; locale shows what your shell session actually has, which can differ if your SSH client forwards LANG (more on that below).

Timezone

I keep servers on UTC unless there's a strong reason not to. No daylight saving, and logs from machines in different countries line up. If a server must be on local time, the command is the same:

timedatectl list-timezones | grep -i seoul
sudo timedatectl set-timezone Asia/Seoul

For UTC: sudo timedatectl set-timezone UTC. Under the hood this points the /etc/localtime symlink at the right file under /usr/share/zoneinfo/ and, on Ubuntu, updates /etc/timezone. Existing processes keep the old zone until restarted, so after changing it:

sudo systemctl restart rsyslog
sudo systemctl restart crond    # CentOS 7
sudo systemctl restart cron     # Ubuntu

Cron in particular reads the timezone at startup and will keep firing jobs on the old schedule until you restart it.

Is the clock actually synchronised?

A correct zone on a drifting clock is still wrong. CentOS 7 uses chrony by default, Ubuntu 16.04 uses systemd-timesyncd (and 18.04 as well unless you installed ntp or chrony):

# CentOS 7
sudo systemctl enable --now chronyd
chronyc tracking
chronyc sources

# Ubuntu
sudo timedatectl set-ntp true
timedatectl | grep -E 'NTP|synchronized'

If chronyc sources shows no reachable server, edit /etc/chrony.conf and put an internal NTP server or a regional pool in front of the defaults, for example server 0.kr.pool.ntp.org iburst. On Ubuntu the equivalent is NTP= in /etc/systemd/timesyncd.conf followed by systemctl restart systemd-timesyncd.

Locale

On CentOS 7, the available locales are all pre-generated. Pick one:

localectl list-locales | grep -E 'en_US|ko_KR'
sudo localectl set-locale LANG=en_US.UTF-8

This writes /etc/locale.conf. On Ubuntu, locales have to be generated first. Enable the ones you need in /etc/locale.gen and run the generator:

sudo sed -i 's/^# *en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen
sudo sed -i 's/^# *ko_KR.UTF-8 UTF-8/ko_KR.UTF-8 UTF-8/' /etc/locale.gen
sudo locale-gen
sudo update-locale LANG=en_US.UTF-8

update-locale writes /etc/default/locale. Either way, the new value takes effect at the next login. I use en_US.UTF-8 on servers even in Korea: error messages stay in English (which makes them searchable), and UTF-8 handles any characters that show up in file names or data.

Services that don't follow the system

Two common ones on database hosts:

MySQL/MariaDB keep their own time_zone setting, which defaults to SYSTEM, but NOW() and TIMESTAMP columns will silently follow whatever the OS had when mysqld started. After changing the OS timezone, restart the server or set it explicitly in /etc/my.cnf:

[mysqld]
default-time-zone = '+00:00'

PHP ignores the OS zone entirely and warns if date.timezone is not set in php.ini:

date.timezone = Asia/Seoul

For a single cron job that should think in a different zone, prefix the command: TZ=Asia/Seoul /usr/local/bin/report.sh, or set CRONTAB_TZ/TZ= at the top of the crontab (supported by cronie on CentOS 7).

Checking everything

Log out and back in, then:

date
date -u
locale
perl -e 'print "ok\n"'
mysql -e "SELECT @@global.time_zone, @@system_time_zone, NOW(), UTC_TIMESTAMP();"

date and date -u should differ by exactly your offset, locale should show the new LANG on every line without errors, Perl should not complain about locale settings, and the MySQL row should show the zone you expect with NOW() and UTC_TIMESTAMP() differing accordingly.

Things that bit me

perl: warning: Setting locale failed. ... LC_ALL = (unset), LC_CTYPE = "ko_KR.UTF-8", LANG = "en_US.UTF-8": your SSH client sent its own LC_* variables (macOS Terminal does this) and the server does not have that locale generated. Either generate it on the server or stop forwarding it: comment out SendEnv LANG LC_* in /etc/ssh/ssh_config on the client, or AcceptEnv in sshd_config on the server.
Failed to set locale: No such file or directory from localectl: on Ubuntu, the locale is not in /etc/locale.gen yet. Generate it first.
timedatectl set-ntp true returns Failed to set ntp: NTP not supported: neither systemd-timesyncd nor chrony is installed and enabled. Install one; timedatectl only toggles whichever exists.
MySQL ERROR 1298 (HY000): Unknown or incorrect time zone: 'Asia/Seoul': the named zone tables are empty. Load them with mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql, or use a numeric offset as shown above.

Do it on the base image and every clone starts out consistent. Five minutes now, versus an afternoon of working out why three logs disagree.