Félix Baylac-Jacqué
7a43fdcdd6
We try to insert the rows directly from the Rust software. Sadly at a massive perf cost :( real 2m14,645s user 1m1,134s sys 1m15,060s We seem to be CPU-bound during the DB generation. I tried disabling the insert journal *and* the sync pragma. I have several hypothesis explaining this slowdown: 1. The pragmas are getting incorrectly set. 2. I'm missing some necessary pragmas. To move forward, I think I should check what kind of pragma the native CSV sqlite import uses and somehow dump all the active pragmas somewhere. |
||
---|---|---|
scripts | ||
src | ||
.gitignore | ||
Cargo.lock | ||
Cargo.toml | ||
README.md | ||
shell.nix |
README.md
Fast Fantoir
time ./scripts/gen-fantoir.sh FANTOIR0721
Generating fantoir CSV
Finished release [optimized] target(s) in 0.00s
Running `target/release/fast-fantoir FANTOIR0721`
Generating fantoir SQLite DB
DB generated at /home/ninjatrappeur/code-root/git.alternativebit.fr/fast-fantoir/fantoir.sqlite
real 0m13,319s
user 0m9,804s
sys 0m3,458s
DB Schema:
sqlite> .tables
fantoir
sqlite> .schema fantoir
CREATE TABLE IF NOT EXISTS "fantoir"(
"full_insee" TEXT,
"rivoli_with_key" TEXT,
"libelle" TEXT
);
Data
You can download the latest FANTOIR data over there: https://www.data.gouv.fr/fr/datasets/fichier-fantoir-des-voies-et-lieux-dits/