Cannot create dynamic inputs from the dynamic outputs of another target efficiently
The problem I have is this: A project using Fortran has a number of libraries. Those libraries generate module files, and those mod files need to be consumed across target boundaries. For example:
src/
util/
util1.f90
util2.f90
util3.f90
lib1/
lib1.f90
lib2/
lib2.f90
exe1/
exe1.f90For simplicity, let's assume each of these fortran files has a single module of the same name, so util1.f90 -> uilt1.mod, and lets assume that each directory of sources is a single target
Ideally, any source that consumes one of these modules would be able to be scheduled for compilation as soon as the .mod exists. So if lib1.f90 only needs util1.mod, it can be scheduled as soon as util1.f90 has been compiled and util1.mod has been generated, and if lib2.f90 only needs util2.mod, etc.
Only, There isn't a way to do that, because Ninja can schedule lib1.f90 before util1.mod exists, since the dyndep file for lib1.f90 adds the dependency, but ninja doesn't ensure that it has an node that provides util1.mod.
There are two ways to work around this, both with their caveats. The solution proposed here of having lib1.a || util.a works, because it ensures that ninja has loaded the dyndep for util1.f90 before lib1.f90 is scheduled. It looks like to me this is what CMake does. The problem with this is that during the initial compilation nothing consuming util1.mod will be scheduled until util.a is linked, if we have several levels of small libraries consuming each other like this, we bottle neck the build and prevent maximum parallelism. The other option, what Meson does, is to use the object file as a proxy for the mod. This makes the first build fast because parallelism is maximized, but means for rebuilds that some objects are recompiled even though they don't need to be.
I see a few ways that this could be resolved:
- The original proposal of the linked issue issue: Ninja should treat dynamic inputs special, and not allow a job to be scheduled if it does not yet know how to build a dynamic input. This is user transparent, and would be my preferred solution.
- Add order-only dependencies to dyndeps. This would allow you to write something like
build lib1.f90.o : dyndep | util1.mod || util1.f90.o. That would give ninja something that's statically defined in the build.ninja to act as a proxy, since we can know that the the dyndeps declaring util1.mod has been loaded by the time that util1.f90.o is compiled, but we also don't cause spurious recompiles like depending on the object directly does - Allow dyndeps to declare build rules for other targets to fulfill the implicit ins of the main target:
build util1.f90.o | util1.mod
build lib1.f90.o : dyndep | util1.modThis feels like a worse version of 2 to me 4. provide a way to specify that a build statement requires additional dyndeps to be loaded before its own dyndeps are loaded 5. ninja will always schedule and load all dyndeps before beginning other target types. This is a big hammer, and while it would work, I think there are better options.
Source: ninja-build/ninja