We had a race condition on cold cache builds. When several nix-gl-host
instances were called, they were trying to concurrently build the
cache.
This lead to some weird errors and busted caches.
We introduce a file lock preventing any concurrent access to the
cache.
We take advantage of this bug to rethink the way we build the cache
and do it in a more robust way. Instead of building it in place, we're
now building it first in a temporary directory (while making sure to
patchelf the DSOs according their final destination). We then move
this directory to the actual cache destination iff the cache has been
built successfully.
It seems like some drivers installations are missing some NVidia
subsystems. We stumbled upon the case of somebody missing the Cuda
libraries.
It did end up making fail the patchelf call.
Preventing the copying/patch routine to run when we do not have any
DSO to copy/patch.
We manage to improve the cold cache generation from 98s to ~30s on my
desktop.
Two things have been done to improve that performance:
1. This one was stupid. I forgot a debug tracing routine that should
have been removed in the code… This tracing routine was forcing us
to cache the libraries… …twice. Massive facepalm. Addressing this
reduced the cold runtime by 50%.
2. Instead of spinning up a patchelf subprocess for each library, we
batch these operations as much as possible in a single subprocess.
This trick shaves about 30% of the remaining runtime.
After profiling a nixglhost hot run, it turns out that we were
spending more than 98% of the run time reading and sha256-hashing
files.
Let's give up on content hashing the files and assume that using their
name, size and last write time is good enough.
On a hot run, we reduce the run time from about 3s to 0.3s on a
nvme-powered ryzen 7 desktop.
I guess this 10x speedup probably worth the little cache corectness we
lose on the way.
We made the incorrect assumption that the first DSO we'd stumble upon
in the load path would be the most appropriate one for the host
system. IE. it'd be the x86_64-gnu-linux on such a system. It turned
out not being the case, meaning we can't take such a shortcut and have
to handle the case where we end up with multiple same libraries for
different archs.
This forced us to do two things:
1. It forced us to rethink the cache directory structure. We now have
to cache each library directory separately. Meaning we now have to
inject the GLX/EGL/Cuda directories of *all* these library
directories subpaths to the LD_LIBRARY_PATH. The dynamic linker
will then try to open these DSOs (and fail) until it stumble upon
the appropriate one.
2. We had to give up on the way we injected EGL libraries using
asolute paths. We don't know which DSO is adapted to the wrapped
program arch. Instead of injecting the absolute paths through the
JSON configuration files, we just stipulate the libraries names in
them. We then inject the various EGL DSOs we find through the
LD_LIBRARY_PATH, similarly to what we already do for GLX and Cuda.
We add a -d/--driver-directory flag allowing the user to circomvent
the DSO automatic lookup and instead force nix-gl-host to load its
dynamic libraries from a specific directory.
Copying & patching all the DSOs is a time consuming process (~10s on a
slow hard drive computer). We definitely don't want to go through it
for each process start, we need to introduce a cache.
For this cache, we go the concervative way. We're going to "resolve" a
DSO name (ie. find the DSO absolute path) and sha256-hash each DSO.
We're then going to compare the fingerprints to determine whether or
not we need to nuke and rebuild the DSO cache.
The cache state is persisted through a JSON file saved in the cache dir.
Félix Baylac Jacqué
Ivor Wanders