We try to insert the rows directly from the Rust software. Sadly at a massive
perf cost :(
real 2m14,645s
user 1m1,134s
sys 1m15,060s
We seem to be CPU-bound during the DB generation. I tried disabling the
insert journal *and* the sync pragma. I have several hypothesis
explaining this slowdown:
1. The pragmas are getting incorrectly set.
2. I'm missing some necessary pragmas.
To move forward, I think I should check what kind of pragma the native
CSV sqlite import uses and somehow dump all the active pragmas
somewhere.
Does not even terminate after a full night worth of run (~12h).
I probably should implement some kind of progress bar to estimate the
total run duration.
Perf:
$ time target/release/fast-fantoir FANTOIR0721
real 0m0,608s
user 0m0,526s
sys 0m0,081s
$ time target/debug/fast-fantoir FANTOIR0721
real 0m11,664s
user 0m11,549s
sys 0m0,113s
I highly suspect the release build to optimize out the useless file
read, partially explaining the *MASSIVE* perf difference between the
release build and the debug one.
This gives up a baseline speed order though. We should probably be
able to stay below 1 minute to generate the DB.
Félix Baylac-Jacqué