os-test

os-test is a set of test suites for POSIX operating systems designed to make it easy to compare differences between operating systems and to find operating system bugs. It consists of test suites that focus on different operating system areas.

This page visualizes the results for the POSIX operating systems that are relevant today.

os-test has been generously funded by the Next Generation Internet Zero Commons fund managed by the NLnet Foundation.

Source Code

The os-test source code is available on gitlab. You also can view the source code for each test by clicking the test title in the leftmost entry of each row. os-test is licensed under the ISC license, see the LICENSE file in the source code for details.

Please feel free to contribute more tests and suites and I'll be happy to run them on every operating system and publish the data.

Setup

I run os-test on my host Linux system; AIX, macOS, and Solaris are graciously provided by cfarm; Managarm and Redox results are pulled from their continuous integration; and the rest of the systems run in qemu virtual machines. The testing harness connects to each machine over ssh and runs tests there. The systems are:

• AIX
• DragonFly
• FreeBSD
• Haiku
• Hurd
• Linux
• macOS
• Managarm
• Minix
• NetBSD
• OmniOS
• OpenBSD
• Redox
• Solaris
• Sortix

I'm still looking for access to Integrity, LynxOS, HP-UX, OpenServer, QNX, UnixWare, VxWorks, and z/OS systems, or any other relevant maintained POSIX systems with a sshd port. Please get in touch if you can help.

Legend

This page is an overview of the test results from the contained suites.

Each cell contains the percentage scores of the test results for a operating system on a suite, along with the number of good test results and the total number of non-neutral test results. Test outcomes that are unknown, neutral, or otherwise non-good and non-bad are not counted.

Click on the the percentages to explore the detailed suite test results.

If the cell scored any partial test outcomes, then a smaller alternate percentage is shown beneath, showing what the score would have been if the following minor problems had not occurred:

• extension - The POSIX feature macro values did not work, but implementation-specific feature macros did work. This outcome typically indicates a recently standardized feature.
• previous_posix - The latest POSIX feature macro values did not work, but the previous values worked. This outcome indicates the headers are not forward compatible with new standards.

The table is sliced into the mandatory POSIX section, and a section for each applicable optional part of POSIX. Each table section is headed by an overall score for the option if multiple suites cover that option.

Suites

os-test contains the following suites:

• basic - This suite tests basic invocations of all functions.
• include - This suite tests whether headers include the correct declarations.
• io - This suite tests input/output system calls.
• limits - This suite tests whether the limits.h constants have the correct values.
• malloc - This suite tests memory allocation.
• namespace - This suite tests for namespace pollution in the standard headers.
• paths - This suite tests whether common paths exist in the filesystem.
• process - This suite tests process system calls.
• pty - This suite tests pseudoterminals.
• signal - This suite tests signal system calls.
• stdio - This suite tests functionality from the stdio.h header.
• udp - This suite tests the UDP stack.

Results