Using dstat to analyze important system parameters

Gobbling up memory

Memory capacity in older systems and SBCs like the Pi is very limited. If a systems does not react as expected and if the results of the CPU metrics are inconclusive, then a memory hungry process is the culprit. Investigate further by calling dstat with the parameters -g -m -s --top-mem and take a look at the pertinent values (Figure 3).

Figure 3: Dstat helps you locate memory hogs quickly.

The relevant fields Paging (-g), Memory usage (-m), Swap (-s) and Most expensive (--top-mem) have to do with the virtual system memory as well as the real main memory that is available to applications for completing tasks.

Normally idle processes do not permanently require memory because the kernel will page out some or all of the memory for an idle process to the hard disk. See the second column, out, in Figure 3 under the paging sections for more information. Once this is done, additional main memory will become available for other processes. The reverse happens when the system reads the memory back into RAM, as soon as the memory is required by the application. See the first column, in, in Figure 3 under the paging section.

The memory usage area indicates the metrics for the real memory. This area tells you how much RAM is currently being used (used), how much data is in the system queue waiting to be written to the hard disk (buff), how much data is in the queue waiting to be read from storage media into RAM (cach) and how much RAM is still available (free).

In general, the greater the value in the used column and the smaller the measurement under free, the greater the system load caused by the programs will be. If the unused RAM capacity is tending toward zero, then the kernel will begin swapping. This means that it will write memory reserved by the processes to the hard disk. Since this works considerably slower than using the main memory, the speed of the entire system will slow down. The third area in the output shown in Figure 3 also indicates how much data the kernel has swapped out (used) and how much space is left for this (free).

The --top-mem option shows the largest consumer of resources and provides you with quick information. The benchmarking tool Mbw [6] was used for Figure 3 to determine the amount of memory taken up by the application Compiz. Dstat shows how the kernel increasingly uses paging, moving more and more data into the swap area. This explains why the system reacted slowly when dstat was called.

When looking at memory load, you will find that the more actively the swapping and paging is taking place, the slower the system will work. Swapping has a lot to do with decreasing the available I/O bandwidth on the hard disk.

Data Throughput

Just as with CPU and RAM workloads, data storage devices containing the operating system and applications will have an impact on the performance of a system. Dstat has options that permit measurements of disk access and data throughput per second. These are usually labeled Disk I/O (Figure 4).

Figure 4: Dstat swings toward the color green when writing occurs, as an indication that there has been a change.

The command dstat -d --disk-util --disk-tps helps create an overview of all data pertinent to these actions. The Dsk/total (-d) field shows the throughput for reading and writing for all data storage devices. It also shows the current workload in percentage points for each known storage device. In Figure 4 this is the column sda produced by the option --disk-util. The third area dsk/total indicates the read and write operations per intervals of time.

Once the load reaches 100%, the system grinds to a halt. At this point, the number of processes queuing for disk I/O (CPU statistics, column wait) will increase dramatically.