osv

OSv was originally designed and implemented by Cloudius Systems (now ScyllaDB) however currently, it is being maintained and enhanced by a small community of volunteers. If you are into systems programming or want to learn and help us improve OSv, then please contact us on OSv Google Group forum or feel free to pick up any good issues for newcomers. For details on how to format and send patches, please read this wiki (we do accept pull requests as well).

OSv

OSv is an open-source versatile modular unikernel designed to run single unmodified Linux application securely as microVM on top of a hypervisor, when compared to traditional operating systems which were designed for a vast range of physical machines. Built from the ground up for effortless deployment and management of microservices and serverless apps, with superior performance.

OSv has been designed to run unmodified x86-64 and aarch64 Linux binaries as is, which effectively makes it a Linux binary compatible unikernel (for more details about Linux ABI compatibility please read this doc). In particular, OSv can run many managed language runtimes including JVM, Python, Node.JS, Ruby, Erlang, and applications built on top of those runtimes. It can also run applications written in languages compiling directly to native machine code like C, C++, Golang and Rust as well as native images produced by GraalVM and WebAssembly/Wasmer.

OSv can boot as fast as ~5 ms on Firecracker using as low as 11 MB of memory. OSv can run on many hypervisors including QEMU/KVM, Firecracker, Cloud Hypervisor, Xen, VMWare, VirtualBox and Hyperkit as well as open clouds like AWS EC2, GCE and OpenStack.

For more information about OSv, see the main wiki page and http://osv.io/.

Building and Running Apps on OSv

To run an application on OSv, one needs to build an image by fusing the OSv kernel, and the application files together. This, at a high level, can be achieved in two ways, either:

• by using the shell script located at ./scripts/build that builds the kernel from sources and fuses it with application files, or
• by using the capstan tool that uses pre-built kernel and combines it with application files to produce a final image.

If you intend to try to run your app on OSv with the least effort possible, you should pursue the capstan route. For introduction please read this crash course. For more details about capstan please read this more detailed documentation. Pre-built OSv kernel files (osv-loader.qemu) can be automatically downloaded by capstan from the OSv regular releases page or manually from the nightly releases repo.

If you are comfortable with make and GCC toolchain and want to try the latest OSv code, then you should read this part of the readme to guide you how to set up your development environment and build OSv kernel and application images.

Releases

We aim to release OSv 2-3 times a year. You can find the latest one on github along with several published artifacts including kernel and some modules.

In addition, we have set up Travis-based CI/CD pipeline where each commit to the master and ipv6 branches triggers full build of the latest kernel and publishes some artifacts to the nightly releases repo. Each commit also triggers the publishing of new Docker "build toolchain" images to the Docker hub.

Design

A good bit of the design of OSv is pretty well explained in the Components of OSv wiki page. You can find even more information in the original USENIX paper and its presentation.

In addition, you can find a lot of good information about the design of specific OSv components on the main wiki page and http://osv.io/ and http://blog.osv.io/. Unfortunately, some of that information may be outdated (especially on http://osv.io/), so it is always best to ask on the mailing list if in doubt.

Component Diagram

In the diagram below, you can see the major components of OSv across the logical layers. Starting with libc at the top, which is greatly based on musl, the core layer in the middle comprises ELF dynamic linker, VFS, networking stack, thread scheduler, page cache, RCU, and memory management components. Then finally down, the layer is composed of the clock, block, and networking device drivers that allow OSv to interact with hypervisors like VMware and VirtualBox or the ones based on KVM and XEN.

Metrics and Performance

There are no official up-to date performance metrics comparing OSv to other unikernels or Linux. In general, OSv lags behind Linux in disk-I/O-intensive workloads partially due to coarse-grained locking in VFS around read/write operations as described in this issue. In network-I/O-intensive workloads, OSv should fare better (or at least used to as Linux has advanced a lot since) as shown with performance tests of Redis and Memcached. You can find some old "numbers" on the main wiki, http://osv.io/benchmarks, and some papers listed at the bottom of this readme.

So OSv is probably not best suited to run MySQL or ElasticSearch, but should deliver pretty solid performance for general stateless applications like microservices or serverless (at least as some papers show).

Kernel Size

At this moment (as of December 2022) the size of the universal OSv kernel (loader.elf artifact) built with all symbols hidden is around 3.6 MB. The size of the kernel linked with the full libstdc++.so.6 library and ZFS filesystem library included is 6.8 MB. Please read the Modularization wiki to better understand how kernel can be built and further reduced in size and customized to run on a specific hypervisor or a specific app.

The size of OSv kernel may be considered quite large compared to other unikernels. However, bear in mind that OSv kernel (being unikernel) provides subset of the functionality of the following Linux libraries (see their approximate size on Linux host):

• libresolv.so.2 (100 K)
• libc.so.6 (2 MB)
• libm.so.6 (1.4 MB)
• ld-linux-x86-64.so.2 (184 K)
• libpthread.so.0 (156 K)
• libdl.so.2 (20 K)
• librt.so.1 (40 K)
• libstdc++.so.6 (2 MB)
• libaio.so.1 (16 K)
• libxenstore.so.3.0 (32 K)
• libcrypt.so.1 (44 K)

Boot Time

OSv, with Read-Only FS and networking off, can boot as fast as ~5 ms on Firecracker and even faster around ~3 ms on QEMU with the microvm machine. However, in general, the boot time will depend on many factors like hypervisor including settings of individual para-virtual devices, filesystem (ZFS, ROFS, RAMFS, or Virtio-FS), and some boot parameters. Please note that by default OSv images get built with ZFS filesystem.

For example, the boot time of ZFS image on Firecracker is ~40 ms, and regular QEMU ~200 ms these days. Also, newer versions of QEMU (>=4.0) are typically faster to boot. Booting on QEMU in PVH/HVM mode (aka direct kernel boot, enabled by -k option of run.py) should always be faster as OSv is directly invoked in 64-bit long mode. Please see this Wiki for a brief review of the boot methods OSv supports.

Finally, some boot parameters passed to the kernel may affect the boot time:

• --console serial - this disables VGA console that is slow to initialize and can shave off 60-70 ms on QEMU
• --nopci - this disables enumeration of PCI devices especially if we know none are present (QEMU with microvm or Firecracker) and can shave off 10-20 ms
• --redirect=/tmp/out - writing to the console can impact the performance quite severely (30-40%) if application logs a lot, so redirecting standard output and error to a file might speed up performance quite a lot

You can always see boot time breakdown by adding --bootchart parameter:

…

Memory Utilization

OSv needs at least 11 M of memory to run a hello world app. Even though it is a third of what it was 4 years ago, it is still quite a lot compared to other unikernels. The applications spawning many threads may take advantage of building the kernel with the option conf_lazy_stack=1 to further reduce memory utilization (please see the comments of this patch to understand this feature better).

We are planning to further lower this number by adding self-tuning logic to L1/L2 memory pools.

Testing

OSv comes with around 140 unit tests that get executed upon every commit and run on ScyllaDB servers. There are also a number of extra tests located under tests/ sub-tree that are not automated at this point.

You can run unit tests in a number of ways:

./scripts/build check                  # Create ZFS test image and run all tests on QEMU

./scripts/build check fs=rofs          # Create ROFS test image and run all tests on QEMU

./scripts/build image=tests && \       # Create ZFS test image and run all tests on Firecracker
./scripts/test.py -p firecracker

./scripts/build image=tests && \       # Create ZFS test image and run all tests on QEMU
./scripts/test.py -p qemu_microvm      # with microvm machine

In addition, there is an Automated Testing Framework that can be used to run around 30 real apps, some of them under stress using ab or wrk tools. The intention is to catch any regressions that may be missed by unit tests.

Finally, one can use Docker files to t