Testing and comparing small-board computers

Benchmarks

The operating system of choice was Ubuntu Mate 16.04 for the RPi2, RPi3, and Odroid-C2, because it is suitable for all three SBCs, and the official Xubuntu 16.04 for the Pine A64+. All of the benchmarks are triggered with SSH, so the desktop environment was not a factor. As a result, Ubuntu 16.04 generally proved to be a sound basis for drawing comparisons.

Before testing, all images were updated on a SanDisk class 10 microSD card with 32GB of capacity. The C2 was tested with a 16GB eMMC module, as well. This is the only technology that Odroid supports for purposes of this round of comparisons.

The benchmark software was Sysbench, iPerf, and P7zip. Sysbench [11] determines the CPU integer performance as well as the I/O speed to the SD card. iPerf [12] determines the Ethernet and WiFi interface speeds. The P7zip [13] compression tool offers a less well documented option for benchmarking how the Lempel-Ziv-Markov chain algorithm (LZMA) compresses and decompresses the software packages. The results of this benchmark are comparatively more meaningful than the Sysbench integer calculations, but also more difficult to interpret. Therefore, you should refer to the website for more about the meaning of individual values.

Classification

The results for the benchmark are summarized in Figures 5 through 7. The Sysbench integer performance results (Figure 5) should be taken with a grain of salt. In fact, the Odroid-C2 CPU does not actually work 10 to 15 times faster than the RPi3. Instead, the Sysbench test results only point to tendencies.

Figure 5: On paper, it looks like the Sysbench results produce a clear statement for the user. In reality, these results are not entirely reliable for a number of reasons.

The reasons are many. First, measuring different architectures affects the way the software runs. If the integer performance test is performed on 200,000 instead of 20,000 prime numbers, then the RPi2 comes out ahead because it has fewer thermal issues. The CPUs on the RPi3 and Odroid-C2 would probably throttle back because of overheating. However this does not change the results in terms of overall tendencies. The same thing goes for the I/O portion of the Sysbench test.

As expected, the third generation Rasp Pi clearly beats its predecessors and proves to be approximately one-third faster in practice. The two sister computers come out fairly even in the I/O test. The limiting factor proves to be the SD card, as revealed by the test using the Odroid-C3 with an eMMC module.

The Odroid-C2 clearly distinguishes itself from the two Rasp Pis in the integer test. It has a significantly faster CPU and two times as much main memory; even so, Sysbench produces results that are perhaps too optimistic. For some reason, the Odroid does less well than the two Rasp Pis when an SD card is used for the I/O test. The Odroid achieved eight times more performance in the I/O test with the eMMC module plugged to the back side of the board than it did with the SD card.

The iPerf test (Figure 6) indicated an efficient 10/100Mbps Ethernet interface on the two Rasp Pi models. The RPi3 WiFi achieved about 25Mbps (data not shown); however 10/100 proved to be the limiting factor in the comparison with the Odroid-C2 and the Pine A64+. The C2 utilizes its Gigabit Ethernet interface almost completely, at 937Mbps, but the A64+ is disappointing in this regard because its 10/100/1000 network interface only delivers about half of the possible throughput rate. This problem is well known and attributable to the kernel. A future update is supposed to resolve this issue.

Figure 6: The Rasp Pis and their Fast Ethernet interface have a disadvantage over their competitors, which are equipped with GigE.

The results for the P7zip benchmark (Figure 7) are more instructive than those produced by Sysbench. However, the compression results are more difficult to interpret. The MIPS (million instructions per second) values were compared with the I/O throughput (KBps). Each of these was examined in compressed and decompressed form. The Pine A64+ with 2GB of RAM and Gigabit Ethernet is just biting at the heels of the latest Raspberry Pi with significantly fewer resources.

Figure 7: I/O performance of the Pine A64+ and RPi3 are similar, although the RPi3 is less well endowed.