Operating System CentOS

Use iotop to Find High Disk I/O on AlmaLinux 9 and Rocky Linux 9

Install and use iotop on AlmaLinux 9, Rocky Linux 9 and CentOS Stream 9 to see which processes read and write the most, including batch logging and the delay-accounting setting.

When a server feels slow but the CPU is mostly idle, the disk is the usual suspect. iotop shows disk activity per process or thread in a top-like view, so you can see which program is responsible instead of only seeing that the disk is busy. This guide installs iotop on AlmaLinux 9, Rocky Linux 9 and CentOS Stream 9, explains its display and options, and shows how to log I/O for later analysis.

CentOS 6 reached end of life in November 2020 and no longer receives updates; AlmaLinux 9 and Rocky Linux 9 are the current replacements. The commands are the same on all three RHEL 9 family systems.

Requirements

  • AlmaLinux 9, Rocky Linux 9 or CentOS Stream 9.
  • A user with sudo rights. iotop reads kernel accounting data that is only available to root.

Install iotop

The package is in the BaseOS repository, so no extra repository is needed:

sudo dnf install iotop

Confirm the installed version:

sudo iotop --version

Enable delay accounting for the IO and SWAPIN columns

Since Linux 5.14, per-task delay accounting is switched off by default to save a little overhead, and the RHEL 9 kernel is based on 5.14. With it off, iotop still reports read and write rates, but the IO> and SWAPIN percentages are unavailable and you may see a warning about CONFIG_TASK_DELAY_ACCT. Check the current state:

sysctl kernel.task_delayacct

Turn it on while you investigate:

sudo sysctl kernel.task_delayacct=1

Only processes started after the switch are measured, so long-running daemons show percentages only after they are restarted. Turn it off again when you are finished:

sudo sysctl kernel.task_delayacct=0

If you want it enabled permanently, for example on a database server you monitor regularly, write it to a sysctl file:

echo 'kernel.task_delayacct = 1' | sudo tee /etc/sysctl.d/90-delayacct.conf
sudo sysctl --system

Run iotop and read the display

Start the interactive view:

sudo iotop

The two lines at the top summarise the whole system. Total DISK READ / Total DISK WRITE is what processes asked the kernel for; Actual DISK READ / Actual DISK WRITE is what really went to the block devices in the same interval. They differ because of the page cache: a process can write a lot that is flushed to disk later.

Below that is one row per thread:

ColumnMeaning
TIDThread ID (shown as PID when you group by process)
PRIOI/O scheduling class and level, for example be/4 (best effort, level 4)
USEROwner of the thread
DISK READRead rate during the last sampling interval
DISK WRITEWrite rate during the last sampling interval
SWAPINShare of time the thread spent waiting for pages to be swapped in
IO>Share of time the thread spent waiting for I/O; the > marks the sort column
COMMANDCommand line of the process

A process with a modest write rate but a high IO> value is being held up by the disk; that is usually the more interesting signal than raw throughput.

Interactive keys

  • Left and right arrow keys: change the sort column.
  • r: reverse the sort order.
  • o: show only threads that are doing I/O right now.
  • p: switch between threads and whole processes.
  • a: switch between current rates and totals accumulated since iotop started.
  • i: change the I/O priority of a thread.
  • q: quit.

Useful command-line options

The combination most administrators use shows only active entries, groups threads into processes and accumulates totals, which makes the heavy hitters rise to the top over a minute or two:

sudo iotop -oPa
OptionEffect
-o, --onlyOnly show processes or threads actually doing I/O
-P, --processesShow processes instead of individual threads
-a, --accumulatedShow totals since start instead of rates
-d SEC, --delay=SECSeconds between updates (default 1)
-p PID, --pid=PIDMonitor only the given process or thread
-u USER, --user=USERMonitor only the given user's processes
-b, --batchNon-interactive output, suitable for logs and scripts
-n NUM, --iter=NUMStop after NUM updates
-t, --timeAdd a timestamp to every line (implies --batch)
-k, --kilobytesAlways use kilobytes, which is easier to parse
-q, --quietSuppress header lines in batch mode; repeat it for less output

Watch everything owned by the database user:

sudo iotop -oP -u mysql

Watch a single process, here PID 1234, with a five-second interval:

sudo iotop -p 1234 -d 5

Log disk I/O to a file

Load spikes rarely happen while you are watching. Batch mode lets you record activity and read it afterwards. This command samples every 5 seconds for 10 minutes (120 iterations) and writes timestamped lines for active processes only:

sudo iotop -botqqqk -d 5 -n 120 | sudo tee /var/log/iotop-sample.log > /dev/null

To capture a nightly problem window, schedule the same command from root's crontab (sudo crontab -e). This entry starts a one-hour recording at 02:00 every day:

0 2 * * * /usr/sbin/iotop -botqqqk -d 10 -n 360 >> /var/log/iotop-night.log 2>&1

Remove the entry and the log file when you have found the cause, because the file grows without limit.

What to do with the result

Once you know the process, the fix depends on what it is: a backup or rsync job that can be moved to a quieter hour, a database missing an index, a log file written at debug level, or memory pressure that shows up as swap activity. Two tools from the same toolbox help to narrow it down.

Lower the I/O priority of a running job so interactive work is served first. Class 3 is idle; class 2 with level 7 is the lowest best-effort priority. Whether this has an effect depends on the I/O scheduler in use for the device.

sudo ionice -c 2 -n 7 -p 1234

Check the device side with iostat from the sysstat package. High %util together with a growing aqu-sz means the device itself is saturated:

sudo dnf install sysstat
iostat -xz 5

The same package provides pidstat -d 5, a per-process I/O view that does not need root for your own processes and is handy in scripts.

Frequently asked questions

Why does iotop show kernel threads such as jbd2 or kworker at the top?

Buffered writes are flushed to disk by kernel threads on behalf of the process that wrote the data. The journal thread (jbd2) or a flush worker is then charged with the actual device write. Use the accumulated view (-a) and look at which user process has the matching DISK WRITE total to find the originator.

Is it safe to leave delay accounting enabled?

It is safe, but it adds a small amount of bookkeeping to every task, which is why the kernel leaves it off by default. On most servers the cost is not noticeable; on very busy systems enable it only while you are diagnosing a problem.

What is iotop-c?

iotop-c is a rewrite of iotop in C with extra features such as a history graph and a key to toggle delay accounting. The package in BaseOS is the original Python implementation; iotop-c is available for the RHEL 9 family from the EPEL repository if you prefer it. The basic options covered above work in both.

Does iotop work inside a container or on OpenVZ-style VPS plans?

It needs access to the kernel's taskstats interface, which unprivileged containers and some container-based VPS products do not expose. On a KVM-based VPS or a dedicated server it works normally.

Share story:

Continue reading other guides on How VPS.