Reworking the filesystem setup (pivot_root)

Context

I'm working on making some changes to firejail, adding an option which allows forming a directory with arbitrary content, something like:

# Allow the app to run some system tools
private-dir /usr/libexec /usr/libexec/convert-*
# Put some more executables in the same folder
private-dir /usr/libexec /opt/other-app/bin/*

Also, the changes I proposed in https://github.com/netblue30/firejail/pull/7067 also touch this topic.

Problem

firejail builds the application's filesystem in place, that is it mounts a series of tmpfs'es on top of the current top level directories. This makes it so that if a profile specifies a lot of rules like the ones suggested above, it could run the risk of stepping into its own shoes, that is whenever a rule references the file "/usr/share/convert-image" it could accidentally pick not the original file, but another file that has been bind-mounted on that path by a previous rule. firejail code is currently avoiding these issues by performing the rules in a precise order, which blacklist rules being applied last. My proposal in https://github.com/netblue30/firejail/pull/7067 also had to deal with this issues, by saving the file descriptor of the source file during the first iteration of the whitelist rules, and in the later phase performing the mount using the saved descriptor. An implementation of the private-dir command mentioned above would have to follow the same logic.

All this introduces a couple of problems:

• We can run out of file descriptors
• Creating multiple tmpfs probably wastes more RAM than creating a single one

This seems to be somehow related to these other issues I found through a quick search:

• https://github.com/netblue30/firejail/issues/2985
• https://github.com/netblue30/firejail/issues/5269#issuecomment-1322921552
• https://github.com/netblue30/firejail/issues/354

Describe the solution you'd like

Build the filesystem on a separate tmpfs:

• Execute whitelist rules immediately as they are found
• Execute all blacklist rules
• Once the filesystem is ready, pivot_root to it