Use iotop to Find High Disk I/O on AlmaLinux 9 and Rocky Linux 9
Install and use iotop on AlmaLinux 9, Rocky Linux 9 and CentOS Stream 9 to see which processes read and write the most, including batch logging and the delay-accounting setting.
When a server feels slow but the CPU is mostly idle, the disk is the usual suspect. iotop shows disk activity per process or thread in a top-like view, so you can see which program is responsible instead of only seeing that the disk is busy. This guide installs iotop on AlmaLinux 9, Rocky Linux 9 and CentOS Stream 9, explains its display and options, and shows how to log I/O for later analysis.
CentOS 6 reached end of life in November 2020 and no longer receives updates; AlmaLinux 9 and Rocky Linux 9 are the current replacements. The commands are the same on all three RHEL 9 family systems.
Requirements
- AlmaLinux 9, Rocky Linux 9 or CentOS Stream 9.
- A user with
sudorights.iotopreads kernel accounting data that is only available to root.
Install iotop
The package is in the BaseOS repository, so no extra repository is needed:
sudo dnf install iotop
Confirm the installed version:
sudo iotop --version
Enable delay accounting for the IO and SWAPIN columns
Since Linux 5.14, per-task delay accounting is switched off by default to save a little overhead, and the RHEL 9 kernel is based on 5.14. With it off, iotop still reports read and write rates, but the IO> and SWAPIN percentages are unavailable and you may see a warning about CONFIG_TASK_DELAY_ACCT. Check the current state:
sysctl kernel.task_delayacct
Turn it on while you investigate:
sudo sysctl kernel.task_delayacct=1
Only processes started after the switch are measured, so long-running daemons show percentages only after they are restarted. Turn it off again when you are finished:
sudo sysctl kernel.task_delayacct=0
If you want it enabled permanently, for example on a database server you monitor regularly, write it to a sysctl file:
echo 'kernel.task_delayacct = 1' | sudo tee /etc/sysctl.d/90-delayacct.conf
sudo sysctl --system
Run iotop and read the display
Start the interactive view:
sudo iotop
The two lines at the top summarise the whole system. Total DISK READ / Total DISK WRITE is what processes asked the kernel for; Actual DISK READ / Actual DISK WRITE is what really went to the block devices in the same interval. They differ because of the page cache: a process can write a lot that is flushed to disk later.
Below that is one row per thread:
| Column | Meaning |
|---|---|
| TID | Thread ID (shown as PID when you group by process) |
| PRIO | I/O scheduling class and level, for example be/4 (best effort, level 4) |
| USER | Owner of the thread |
| DISK READ | Read rate during the last sampling interval |
| DISK WRITE | Write rate during the last sampling interval |
| SWAPIN | Share of time the thread spent waiting for pages to be swapped in |
| IO> | Share of time the thread spent waiting for I/O; the > marks the sort column |
| COMMAND | Command line of the process |
A process with a modest write rate but a high IO> value is being held up by the disk; that is usually the more interesting signal than raw throughput.
Interactive keys
- Left and right arrow keys: change the sort column.
r: reverse the sort order.o: show only threads that are doing I/O right now.p: switch between threads and whole processes.a: switch between current rates and totals accumulated sinceiotopstarted.i: change the I/O priority of a thread.q: quit.
Useful command-line options
The combination most administrators use shows only active entries, groups threads into processes and accumulates totals, which makes the heavy hitters rise to the top over a minute or two:
sudo iotop -oPa
| Option | Effect |
|---|---|
-o, --only | Only show processes or threads actually doing I/O |
-P, --processes | Show processes instead of individual threads |
-a, --accumulated | Show totals since start instead of rates |
-d SEC, --delay=SEC | Seconds between updates (default 1) |
-p PID, --pid=PID | Monitor only the given process or thread |
-u USER, --user=USER | Monitor only the given user's processes |
-b, --batch | Non-interactive output, suitable for logs and scripts |
-n NUM, --iter=NUM | Stop after NUM updates |
-t, --time | Add a timestamp to every line (implies --batch) |
-k, --kilobytes | Always use kilobytes, which is easier to parse |
-q, --quiet | Suppress header lines in batch mode; repeat it for less output |
Watch everything owned by the database user:
sudo iotop -oP -u mysql
Watch a single process, here PID 1234, with a five-second interval:
sudo iotop -p 1234 -d 5
Log disk I/O to a file
Load spikes rarely happen while you are watching. Batch mode lets you record activity and read it afterwards. This command samples every 5 seconds for 10 minutes (120 iterations) and writes timestamped lines for active processes only:
sudo iotop -botqqqk -d 5 -n 120 | sudo tee /var/log/iotop-sample.log > /dev/null
To capture a nightly problem window, schedule the same command from root's crontab (sudo crontab -e). This entry starts a one-hour recording at 02:00 every day:
0 2 * * * /usr/sbin/iotop -botqqqk -d 10 -n 360 >> /var/log/iotop-night.log 2>&1
Remove the entry and the log file when you have found the cause, because the file grows without limit.
What to do with the result
Once you know the process, the fix depends on what it is: a backup or rsync job that can be moved to a quieter hour, a database missing an index, a log file written at debug level, or memory pressure that shows up as swap activity. Two tools from the same toolbox help to narrow it down.
Lower the I/O priority of a running job so interactive work is served first. Class 3 is idle; class 2 with level 7 is the lowest best-effort priority. Whether this has an effect depends on the I/O scheduler in use for the device.
sudo ionice -c 2 -n 7 -p 1234
Check the device side with iostat from the sysstat package. High %util together with a growing aqu-sz means the device itself is saturated:
sudo dnf install sysstat
iostat -xz 5
The same package provides pidstat -d 5, a per-process I/O view that does not need root for your own processes and is handy in scripts.
Frequently asked questions
Why does iotop show kernel threads such as jbd2 or kworker at the top?
Buffered writes are flushed to disk by kernel threads on behalf of the process that wrote the data. The journal thread (jbd2) or a flush worker is then charged with the actual device write. Use the accumulated view (-a) and look at which user process has the matching DISK WRITE total to find the originator.
Is it safe to leave delay accounting enabled?
It is safe, but it adds a small amount of bookkeeping to every task, which is why the kernel leaves it off by default. On most servers the cost is not noticeable; on very busy systems enable it only while you are diagnosing a problem.
What is iotop-c?
iotop-c is a rewrite of iotop in C with extra features such as a history graph and a key to toggle delay accounting. The package in BaseOS is the original Python implementation; iotop-c is available for the RHEL 9 family from the EPEL repository if you prefer it. The basic options covered above work in both.
Does iotop work inside a container or on OpenVZ-style VPS plans?
It needs access to the kernel's taskstats interface, which unprivileged containers and some container-based VPS products do not expose. On a KVM-based VPS or a dedicated server it works normally.