DB-유

AppArmor Basics on Ubuntu

· linux, ubuntu, apparmor, security

If you come from CentOS, you know SELinux as the thing that produces mysterious Permission denied errors when you move a data directory. Ubuntu's equivalent is AppArmor, and it causes exactly the same kind of confusion when MySQL refuses to start after you point datadir at a new disk. The difference is that AppArmor profiles are per-program path-based rules in plain text under /etc/apparmor.d/, which makes them much easier to read and adjust than SELinux policy.

I got bitten by exactly that, and now it's the first thing I check. This covers Ubuntu 18.04 (16.04 is nearly identical): what is confined, reading the MySQL profile, adjusting it for a non-standard data directory, and generating rules from the log. AppArmor is on by default, but the userspace tools need sudo apt install apparmor-utils.

What is confined?

sudo aa-status

The output lists loaded profiles, split into enforce mode (violations are blocked and logged) and complain mode (violations are only logged). On a server with MySQL 5.7 installed you will see /usr/sbin/mysqld in enforce mode. Each running process that is confined is listed with its PID.

The profile files are named after the binary path with slashes replaced by dots, so the MySQL profile is /etc/apparmor.d/usr.sbin.mysqld. Read it:

less /etc/apparmor.d/usr.sbin.mysqld

Rules look like /var/lib/mysql/ r, and /var/lib/mysql/** rwk, (read, write, lock). Anything not listed is denied.

Finding the denials

When a confined program is blocked, the kernel logs an audit message. On Ubuntu without auditd running, it goes to /var/log/kern.log and /var/log/syslog, and dmesg:

sudo grep -i 'apparmor="DENIED"' /var/log/syslog | tail

A typical line after moving the MySQL data directory:

apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/data/mysql/ibdata1" pid=1423 comm="mysqld" requested_mask="rw" denied_mask="rw" fsuid=112 ouid=112

profile tells you which profile to fix, name is the path, and denied_mask is what was attempted. That is everything needed to write the rule.

Adding rules through the local override

Ubuntu's packaged profiles include a line near the end: #include <local/usr.sbin.mysqld>. The file /etc/apparmor.d/local/usr.sbin.mysqld exists, is empty, and is not touched by package upgrades. That is where your additions go:

sudo nano /etc/apparmor.d/local/usr.sbin.mysqld

# MySQL data directory moved to /data/mysql
/data/mysql/ r,
/data/mysql/** rwk,

Then reload just that profile:

sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.mysqld
sudo systemctl restart mysql

apparmor_parser -r replaces the loaded profile; systemctl reload apparmor reloads all of them. Neither restarts the confined program, so MySQL itself still needs restarting to pick up the new datadir anyway.

Complain mode and aa-logprof

For a program with many unknown paths, put the profile in complain mode, exercise the program fully, then let aa-logprof turn the logged events into rules:

sudo aa-complain /usr/sbin/mysqld
# run the workload: start MySQL, run backups, load data, etc.
sudo aa-logprof

aa-logprof walks through each logged access and asks whether to allow it, with options to generalise the path (for example turn /data/mysql/ib_logfile0 into /data/mysql/**). When done, return to enforce mode:

sudo aa-enforce /usr/sbin/mysqld

Note that aa-logprof writes into the main profile file, not the local/ include, so for packaged profiles I prefer to read its suggestions and copy them into the local file by hand.

Turning a profile off

Sometimes the right answer is to unconfine one program rather than maintain a profile for it. Disable it without deleting the file:

sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld

The symlink in disable/ stops it loading at boot; -R unloads it now. Do not disable AppArmor globally with the apparmor=0 kernel parameter just to fix one service.

Checking the result

sudo aa-status | grep mysqld
sudo systemctl status mysql
sudo grep 'apparmor="DENIED"' /var/log/syslog | grep mysqld | tail -3

MySQL should be running, aa-status should list the profile in enforce mode, and no new DENIED lines should appear after the restart. Run a backup and a table import to exercise the paths you added; check the log once more.

Gotchas

  • MySQL fails with Can't create/write to file '/data/mysql/ibdata1' (Errcode: 13 - Permission denied) even though ls -l shows the mysql user owns everything: this is AppArmor, not filesystem permissions. Check syslog for the DENIED line.
  • AppArmor parser error for /etc/apparmor.d/usr.sbin.mysqld ... syntax error, unexpected TOK_ID: a rule in the local file is missing the trailing comma, or the path has a typo. Every rule ends with ,.
  • aa-complain: command not found: apparmor-utils is not installed.
  • A snap or a program run from a different path is unaffected by your changes: profiles attach by executable path. /usr/local/sbin/mysqld is not /usr/sbin/mysqld. Check aa-status for the exact path listed.
  • Nothing is logged although access is clearly failing: auditd is installed and is receiving the messages instead. Look in /var/log/audit/audit.log for type=AVC.

My rule now: mysterious permission error after a path change on Ubuntu, grep syslog for DENIED before touching chmod. Usually it's a two-line addition under /etc/apparmor.d/local/.