Go to file

Félix Baylac-Jacqué 7a43fdcdd6 "native" SQLite insertion. We try to insert the rows directly from the Rust software. Sadly at a massive perf cost :( real 2m14,645s user 1m1,134s sys 1m15,060s We seem to be CPU-bound during the DB generation. I tried disabling the insert journal and the sync pragma. I have several hypothesis explaining this slowdown: 1. The pragmas are getting incorrectly set. 2. I'm missing some necessary pragmas. To move forward, I think I should check what kind of pragma the native CSV sqlite import uses and somehow dump all the active pragmas somewhere.		2022-02-11 11:07:17 +01:00
scripts	CSV intermediate file approach.	2021-11-17 21:32:02 +01:00
src	"native" SQLite insertion.	2022-02-11 11:07:17 +01:00
.gitignore	Reference read	2021-11-03 21:27:07 +01:00
Cargo.lock	"native" SQLite insertion.	2022-02-11 11:07:17 +01:00
Cargo.toml	"native" SQLite insertion.	2022-02-11 11:07:17 +01:00
README.md	"native" SQLite insertion.	2022-02-11 11:07:17 +01:00
shell.nix	Naive SQLite implementation	2021-11-17 20:45:16 +01:00

README.md

Fast Fantoir

time ./scripts/gen-fantoir.sh FANTOIR0721
Generating fantoir CSV
    Finished release [optimized] target(s) in 0.00s
     Running `target/release/fast-fantoir FANTOIR0721`
Generating fantoir SQLite DB
DB generated at /home/ninjatrappeur/code-root/git.alternativebit.fr/fast-fantoir/fantoir.sqlite

real	0m13,319s
user	0m9,804s
sys	0m3,458s

DB Schema:

sqlite> .tables
fantoir

sqlite> .schema fantoir
CREATE TABLE IF NOT EXISTS "fantoir"(
  "full_insee" TEXT,
  "rivoli_with_key" TEXT,
  "libelle" TEXT
);

Data

You can download the latest FANTOIR data over there: https://www.data.gouv.fr/fr/datasets/fichier-fantoir-des-voies-et-lieux-dits/