DB-유

Monitoring Linux with top, htop, and vmstat

· linux, monitoring, performance

Before reaching for a monitoring stack with dashboards and agents, it pays to be fluent in the three tools that are already installed on nearly every Linux box or one yum install away. When a database server is "slow", the first two minutes on the console with top, htop and vmstat usually tell me whether the problem is CPU, memory, disk or something else entirely. That decides where to look next.

Most of the skill is knowing which columns matter and which are misleading. top and vmstat come with procps-ng, which is installed by default on CentOS 7 and Ubuntu 16.04/18.04. htop needs yum install epel-release && yum install htop on CentOS 7 or apt install htop on Ubuntu, and iostat (from sysstat) is a handy fourth tool.

Reading the top header

I ignore the process list at first.

Run top and look at the summary lines before the process list:

top

The line that matters most is the load average, and the most common misreading is comparing it to 1.0. Load is the average number of runnable and uninterruptible (usually waiting on disk) processes; the threshold for "busy" is the number of CPU cores, which nproc tells you. A load of 6 on a 16-core box is quiet; on a 2-core VPS it means work is queuing.

On the %Cpu(s) line, the columns to watch:

  • us user time, sy system time: normal work.
  • wa I/O wait: CPUs idle because processes are blocked on disk. Sustained wa above 10-20% points at storage.
  • st steal: time the hypervisor gave to other guests. Anything consistently above a few percent on a VPS means you are fighting neighbours.

The memory lines are misleading on their own because Linux uses free RAM for the page cache. Look at avail Mem (procps 3.3.10 and later, so on CentOS 7 and Ubuntu 16.04 onward), which estimates what is really available without swapping. Then check Swap: ... used. Some swap in use is not a problem; swap that is growing is.

Useful interactive keys: 1 to show each CPU separately, M to sort by memory, P by CPU, c to show full command lines, k to kill a PID, W to save the current layout as the default.

htop for the process view

I reach for htop over top whenever it's installed. Same data, per-core bars, a tree view, and much nicer interaction:

htop

Press F5 for the tree view, which shows that the forty php-fpm processes belong to one master, or that a MySQL server has a single process with hundreds of threads (press H to toggle showing threads). F6 chooses the sort column, F4 filters by name, F9 sends a signal. t toggles the tree.

Two columns to add via F2 (Setup) that are not shown by default: IO_RATE (needs root, shows per-process read/write bandwidth) and NLWP (thread count). Settings save to ~/.config/htop/htoprc.

In the memory bar, green is used by processes, blue is buffers, and yellow/orange is cache. If the bar is mostly yellow, the machine is not short of memory even if free says otherwise.

top and htop are snapshots. vmstat prints a line every interval, which shows change, and is what I leave running in a second terminal while reproducing a problem:

vmstat 2 30

That prints 30 samples at 2-second intervals. The first line is an average since boot; ignore it. Columns, grouped:

  • r: processes waiting for CPU. Consistently higher than core count means CPU saturation.
  • b: processes blocked on I/O. Non-zero for many samples means disk trouble.
  • si / so: swap in and out, in KB/s. Any sustained so is bad news for a database; it means active memory is being written to disk.
  • bi / bo: blocks read/written from block devices.
  • us / sy / id / wa / st: same meaning as in top.

Add -S M to display memory columns in megabytes instead of kilobytes, and -t for a timestamp on each line so you can match it to the application log afterwards:

vmstat -S M -t 5

iostat, when the disk looks guilty

When wa and b point to storage, iostat says which device and how badly:

iostat -xz 2

-x gives extended statistics and -z hides idle devices. The columns to read are %util (how busy the device is; near 100% is saturated), await (average ms per request including queue time; on an SSD this should be single digits, on spinning disks under ~20 ms), and avgqu-sz (queue depth). High await with low %util on a virtual disk means the latency is coming from the storage layer underneath, not from your workload.

Take a baseline

Run these on a known-good day. On a database server I record something like:

uptime
vmstat -S M 5 6
iostat -xz 5 3

and keep the output in the server's notes. When the box is slow later, the same three commands and a diff against the baseline usually make the bottleneck obvious.

Notes

  • Load average includes processes in uninterruptible sleep (state D in top). A stuck NFS mount can push load to 50 with the CPU completely idle. Check wa and b before assuming CPU.
  • RES in top is resident memory per process, but shared libraries are counted in every process using them, so summing RES overstates real usage. smem (in EPEL) reports proportional set size if you need an honest total.
  • top in batch mode is handy for logging: top -b -n 1 | head -20 >> /var/log/top-snapshot.log in a cron job gives you a record of what was running when the alert fired.
  • On a VM, if st in vmstat is high, no amount of tuning inside the guest will help; the fix is on the hypervisor or with the provider.

It's no substitute for a monitoring system with history and alerts. But it answers "what is this machine doing right now" on a box with nothing else installed.