Reliability
Execution time and memory by engine version
The same databases, the same panels of activities, and every engine release in turn. One column per release, so what moves from one to the next is the engine.
Every load below is a parse from the source files: the engine keeps a parsed cache, and that cache is deleted immediately before the reading is taken. The cache is what every later start reads instead, which is the Load from cache row.
Loading from source also measures the operating system's own file cache, which the release measured just before it emptied by holding tens of gigabytes. Read that row as an order of magnitude rather than as a comparison between neighbouring columns; the scoring row and the two memory rows do not carry that noise.
One database is loaded at a time, in an engine started for it and stopped afterwards. Peak memory is the highest figure the kernel saw the process hold, and it never falls, so the scored reading includes the loaded one. Measuring two databases in one engine would have made the second inherit the first.
All of these were built the same way: GHC 9.12.4, rts_thr, x86_64. A difference there would move the numbers without the engine changing at all.
| Measurement | v0.9.1 | v0.9.2 | v0.9.3 | v0.9.4 | v0.9.5 | v0.10.0 | v0.11.0 | v0.12.0 |
|---|---|---|---|---|---|---|---|---|
| ecoinvent 3.12 cut-off EcoSpold 2, scored under EF v3.1 | ||||||||
| Activities scoredhow many of the published panel this release resolved | 26,533 | 26,533 | 26,533 | 26,533 | 26,533 | 26,533 | 26,533 | 26,533 |
| Load from sourceparsed with the cache deleted first | 17.8 s | 14.6 s | 40.4 s | 16.6 s | 17.1 s | 19 s | 17.8 s | 18 s |
| Load from cachewhat every later start costs | 1.8 s | 1.5 s | 1.6 s | 1.5 s | 1.5 s | 1.6 s | 1.6 s | 1.5 s |
| Score the published panelone batch call, every activity in it | 371.7 s | 363.5 s | 360.8 s | 428.2 s | 471.6 s | 473.2 s | 475.5 s | 106 s |
| Peak memory, loadedhighest the kernel saw | 9,267 MB | 9,287 MB | 8,162 MB | 8,957 MB | 8,951 MB | 7,790 MB | 7,777 MB | 7,832 MB |
| Peak memory, scoredthe same mark after scoring, so it carries the load | 26,881 MB | 27,437 MB | 28,241 MB | 29,768 MB | 29,926 MB | 31,414 MB | 31,994 MB | 24,736 MB |
| Agribalyse 3.2 SimaPro CSV, scored under EF 3.1 adapted 1.0 | ||||||||
| Activities scoredhow many of the published panel this release resolved | 2,389 | 2,389 | 2,389 | 2,389 | 2,389 | 2,389 | 2,389 | 2,389 |
| Load from sourceparsed with the cache deleted first | 29.4 s | 29.5 s | 31.2 s | 30.1 s | 29.6 s | 29.6 s | 30.5 s | 29.8 s |
| Load from cachewhat every later start costs | 5.2 s | 5.8 s | 5.7 s | 5.5 s | 5.7 s | 5.1 s | 5.6 s | 5.2 s |
| Score the published panelone batch call, every activity in it | 94.9 s | 90.6 s | 91.2 s | 89.1 s | 92.1 s | 94.4 s | 95.6 s | 29.3 s |
| Peak memory, loadedhighest the kernel saw | 11,120 MB | 11,099 MB | 11,074 MB | 12,556 MB | 12,640 MB | 12,618 MB | 12,514 MB | 12,714 MB |
| Peak memory, scoredthe same mark after scoring, so it carries the load | 11,178 MB | 11,157 MB | 11,131 MB | 12,616 MB | 12,698 MB | 12,677 MB | 12,573 MB | 12,777 MB |
| BAFU 2026 v1 EcoSpold 1, scored under EF 3.1 adapted 1.05 Not loaded by v0.9.1, v0.9.2, v0.9.3: RuntimeError: the engine refused to load bafu-2026-v1: Unknown unit conversion: "unit" → "p" in Natural gas, production GQ, at evaporation plant — add these units to [[units]] CSV Not loaded by v0.9.4, v0.9.5, v0.10.0, v0.11.0: RuntimeError: the engine refused to load bafu-2026-v1: Unknown unit conversion: "unit" → "p" in Natural gas, production US, at evaporation plant — add these units to [[units]] CSV | ||||||||
| Activities scoredhow many of the published panel this release resolved | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | 11,837 |
| Load from sourceparsed with the cache deleted first | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | 3.2 s |
| Load from cachewhat every later start costs | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | 0.7 s |
| Score the published panelone batch call, every activity in it | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | 22 s |
| Peak memory, loadedhighest the kernel saw | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | 1,464 MB |
| Peak memory, scoredthe same mark after scoring, so it carries the load | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | not loaded | 6,855 MB |
A database marked not loaded carries, under its name, the reason the releases that refused it gave. An older engine failing to read a database is an answer about that release, not a gap in the measurement.
The panel of activities comes from the publisher's own table rather than from the engine, so every release is asked for the same work. How much of it a release could take up is the first row of each block: two columns whose counts differ are not timing the same thing, and the number is there to be checked rather than assumed. The agreement those same panels produced is on the comparison by version.