os-test

os-test is a set of test suites for POSIX operating systems designed to make it easy to compare differences between operating systems and to find operating system bugs. It consists of test suites that focus on different operating system areas.

This page visualizes the results for the free software POSIX operating systems that are relevant today.

Source Code

The os-test source code is available on gitlab. You also can view the source code for each test by clicking the test title in the leftmost entry of each row. os-test is licensed under the ISC license, see the LICENSE file in the source code for details.

Setup

I run os-test on my host Linux system and the rest of the systems run in VirtualBox virtual machines of every other system. The testing harness connects to each virtual machine over ssh and runs tests there. The systems are:

• DragonFly (Twitter thread on installation)
• FreeBSD (Twitter thread on installation)
• Haiku (Twitter thread on installation)
• Hurd (Twitter thread on installation)
• Linux (My development environment)
• macOS (Contributed data - how do I get such a virtual machine?)
• Minix (Twitter thread on installation)
• NetBSD (Twitter thread on installation)
• OpenBSD
• OpenIndiana (Twitter thread on installation) and OmniOS Community Edition (Twitter thread on installation) (the results are identical, so their columns are combined to save space)
• Sortix (Virtual Machine of my own OS)

Legend

Good. A cell is good if its output belongs to the set of expected valid outputs for that test, as determined by the applicable standards, specifications, expected behavior, or the interpretation of the os-test authors. If all the good results in a row have the same output, all the good result cells are colored in the unanimous color. Otherwise, each different kind of good outcome is colored in an unique color.

Unknown. A cell is unknown if the output is known, but it has not yet been determined if the output is a good result. If all the unknown results in a row have the same output, all the unknown result cells are colored in the unanimous color. Otherwise, each different kind of unknown outcome is colored in an unique color.

Bad. A cell is bad if its output is neither good nor unknown. If all the bad results in a row have the same output, all the bad result cells are colored in the unanimous color. Otherwise, each different kind of bad outcome is colored in an unique color.

§. The § link on the left of each row links to that row.

Suites

os-test currently contains these suites:

• io - This suite tests input/output system calls.
• udp - This suite tests the UDP stack.

io

This suite tests input/output system calls.

udp

This suite tests the UDP stack.

Open questions:

• What should happen if you bind to the broadcast address 255.255.255.255? It is an explicitly forbidden source address. Should it fail with EADDRNOTAVAIL, or some other error, or succeed (and in which case, what does it mean?). (bind-broadcast-0-getpeername, bind-broadcast-0-getsockname)
• What should happen if you connect to the broadcast address 255.255.255.255? Must SO_BROADCAST be set first? (connect-broadcast-getpeername-so-broadcast, connect-broadcast-getpeername, connect-broadcast-getsockname-so-broadcast, connect-broadcast-getsockname)
• What should happen if you bind to the first address in the lan subnet? (bind-lan-subnet-first)
• What should happen if you connect to the any address 0.0.0.0? Is it the same as connecting to the loopback address? (connect-any-getpeername and connect-any-getsockname)
• What should happen if you connect to port 0? Does connecting to port 0 unconnect the socket on DragonFly and Linux? (connect-any-0-getpeername, connect-any-0-getsockname, connect-loopback-0-getpeername, connect-loopback-0-getsockname)
• What should happen if you send to the any address 0.0.0.0? (sendto-any-so-error)
• What should happen if you send to port 0? (sendto-loopback-0-so-error)
• Should it be possible to shutdown a socket without having connected it?
• Should reading on a socket shutdown for read return 0 or should it fail with EWOULDBLOCK?
• Should writing to socket shutdown for write send SIGPIPE in addition to failing with ESPIPE?
• After shutdown for read, should data already received be avalilable for read? Some systems deliver already received asyncronous errors after shutdown, yet don't deliver already received data after shutdown.
• After shutdown for read, should data be received? Should POLLIN be set if data is received yet recv of it would return 0.
• Should POLLOUT be set after shutdown for write? Should POLLHUP be set?
• Should POLLIN be set after shutdown for write if there's already received data?
• If an asynchronous socket error is pending, which of POLLIN, POLLOUT, POLLERR should be set?
• Should asynchronous socket errors be delivered on connect?
• What should happen if you connect to an address on the loopback interface and then reconnect to the public internet? What happens to the locally bound address? Is the socket rebound? Or does the connect fail because there's no longer a valid route from the loopback interface to the public internet.
• If a socket is connected, can you supply a destination address to sendto? What if the destination address is the current remote address?
• What is the local socket name after connect to loopback and then unconnect?
• How does unconnecting work? Can you pass a sa_family_t set to AF_UNSPEC to connect, or do you need to wrap it in a struct sockaddr, or a struct sockaddr_in?
• What happens if you bind after unconnect?
• Does unconnect on a freshly made socket bind the socket? (unconnect-getsockname)
• Can you unconnect a socket that hasn't been connected yet? (unconnect-getsockname)
• If a socket is shutdown for read, should recv return any data received after the shutdown.
• If you receive on freshly made socket, what address should getsockname return?
• Can you bind to the any address in the loopback network? (bind-loopback-other)
• Can you bind to the broadcast address in the loopback network? (bind-loopback-broadcast)
• Does SO_REUSEADDR need to set on both sockets or just the second? (bind-conflict-any-loopback-so-reuseaddr, bind-conflict-any-loopback-so-reuseaddr-both)
• Does SO_REUSEADDR allow the same address and port to be bound twice? (bind-conflict-loopback-loopback-so-reuseaddr, bind-conflict-loopback-loopback-so-reuseaddr-both)

Running this suite has the following side effects:

• Datagrams containing just the byte 'x' are sent to 8.8.8.8:53 (Google DNS) because I needed a legitimate address that would not send back any ICMP connection refused address. Please configure this address in udp/udp.h using BLACKHOLE_HOST and BLACKHOLE_PORT to a site of your choice.
• Datagrams containing just the byte 'x' are sent to port 65534 and port 65535 on the loopback address 127.0.0.1.
• This suite requires port 65535 on the loopback address 127.0.0.1 to be unused.
• UDP sockets will be bound to the loopback interface and on the interface leading to the public internet.