Processing pipeline¶
Diagrams of the MULD2 processing steps and their dependencies,
from the overall pipeline down to the individual attribute batches.
Source: muld2_pipeline.drawio. Click an image to open it at full size.
Overview¶
One chain, transform → geofix → assemble → peat decay → rasterize. Only the
last steps write the final products, under fixed names in the run directory
($MULD2_RUN_DIR, see integration/run_layout.py); everything before is an
intermediate below work/:
| Step | pixi task | Writes |
|---|---|---|
| 1 | transform |
work/1_transform/parsed_parquet/ |
| 1b | geofix |
work/2_geofix/ |
| 2 | assemble |
work/3_assemble/mv_mullakaart_parsed.gpkg (base map) |
| 3 | peat-subset, peat-clip-all, peat-clip-assemble |
work/4_peat/ |
| 3 | peat-decay |
mv_mullakaart_parsed.gpkg (final) |
| 3 | gpkg-to-fgb |
mv_mullakaart_parsed.fgb (final) |
| 4 | rasterize-l1 |
rasters/{sand,silt,clay,rock}_{lo}_{hi}.tif (final) |
pixi run run-pipeline runs all of them in this order. In GitLab CI (karula,
manual jobs from "Run pipeline") the same tasks are the jobs transform
(1–2), peat-decay (3), rasterise-l1 (4) and publish-s3, which uploads
the final products only.
1 · transform (Dask)¶
Per-polygon corrections by unique_id first, then the Huumus, Siffer + WRB and
Loimis tracks, with the attribute batches each step adds.
1b · geofix¶
Turvasmuld polygons without prims: each one is dissolved into a turvasmuld neighbour, gets its prims, or goes to manual review.
2 · assemble¶
Streaming fid-window join of the parsed attributes onto the source geometry.
The geofix decisions and the end-of-chain rules (empty model depths without
layers, peat thickness from the prims) are applied per batch, and every touched
polygon is labelled in fix_applied.
3 · peat decay¶
The peat soils lose thickness where drainage and cultivation have decomposed
the peat. This step sits between assemble and rasterize: it reads the base map
of assemble and writes the final map, mv_mullakaart_parsed.gpkg, which the
rasters, the FlatGeobuf, the exports and PostGIS read. The base map is an
intermediate (work/3_assemble/) and is never overwritten, because the
remaining-peat raster derives from it and decaying a decayed map would decay
the peat twice. Code in integration/peat_decay/.
peat-subset: the peat polygons near the < 30 cm remaining-peat mask (below_30cm_peat_polygons_simplified.gpkg), with their neighbours, in independent chunks.peat-clip-all: the QGIS model (peat_clipping_qgis/*.model3, run headless through CWL in themuld2-qgiscontainer) cuts the peat polygons along the mask, per chunk. A chunk the model fails on is retried with a rounded mask and then split by group; groups it still fails on keep their base geometry (peat_clip = model_failed).peat-clip-assemble: the cut pieces replace their polygons, split pieces get new UUIDs (parent_unique_idkeeps the base id), and the remaining peat (median ofpeat_remaining_cm.tif) and land use (haritav / looduslik, ELME2 baaskaart) are computed per piece.peat-decay(decay_apply.py):- Tarn polygons get Loimis1
t₃(also in transform); - the peat layers lose thickness down to the remaining peat.
The modelled depths always change; the Loimis formula and the exact
Z1–Z4only where the formula has a thickness for the peat layer (t₃40/l→t₃15/l). A layer that is gone keeps its typetwith 0 cm; - the change matrix (
matrix.py) gives cultivated pieces a newsiffer_1from the remaining peat, e.g.M’’with 20 cm →G1; - siffer, WRB and all Loimis1 columns are re-derived for the changed pieces with the functions of transform, and the end-of-chain rules of assemble are applied.
- Tarn polygons get Loimis1
Every change is labelled in fix_applied (77:peat_depth:…, 79:matrix:…,
80:tarn_loimis_t3) and listed in the decay_log layer; peat_clip_log and
geofix_log are carried along. All columns the peat decay adds:
Field Reference, peat decay.
Scope: siffer_1 and Loimis1 only
The peat decay changes the first soil unit and the Loimis1 formula only.
The change matrix rewrites siffer_1; siffer_2–siffer_4 of a compound
siffer stay as mapped. The peat layers lose thickness only in Loimis1
(loimis1, Z1–Z4, z1_model–zmx_model); Loimis2 (the lm2_
columns) and huumus keep the values of the base map, and a loss that
reaches the huumus peat is only flagged (77:huumus_peat_check). The same
holds for Tarn → t₃ and for the peat thickness from the prims in the
model depths (assemble), and the rasters are made from Loimis1 too.
siffer_2–siffer_4 are read in one place only: whether the
remaining-peat raster covers a piece (peat_modelled, R, S, M or AM in any
unit). Exception: geofix puts its prims on every peat unit of a siffer
without prims (DG;M → DG;M’), with the thickness from the Loimis1 layers.
4 · rasterize¶
Vertical depth weighting of the final map (after the peat decay) and
rasterisation to COGs on the reference grid
(est_ref_grid_500m_buffer_2025.tif; despite the name 10 m pixels,
37 100 × 25 800). Sand, silt, clay and rock are averaged per depth interval
(0–10 … 100–200 cm), weighted by the layer thicknesses from z1_model–z4_model
and clipped at zmx_model.
What an interval value means:
- Peat and humus layers (
LXTYPEPEAT/HUMUS) have no mineral fine earth and count as 0 % sand/silt/clay over their thickness. The value is the mineral fine-earth share, so sand + silt + clay < 100 where part of the interval is organic. Peat and its decay are therefore visible in the upper intervals. Rock of a peat layer comes from its coarse fragments as before. - A 0 cm layer (peat that has decayed completely,
t₃0/l) carries no weight. The layer below starts at 0 cm. - GRAVELS layers have unknown fine earth and are skipped; the neighbouring layer fills their depth range, as before.
- No layer in the interval (below
zmx_model): NoData (-9999), not 0. Earlier raster sets burned these cells as 0 % (GDAL reads an empty attribute as 0).
5 · Repairs: in place and open¶
Some repairs cannot be made by the per-field parsers, because they need the
value of a single polygon, more than one attribute, or the geometry. The list
below shows where these repairs attach in the chain (status 2026-10-08, numbers
from the run of 2026-10-07). Every polygon touched by one of them is labelled
in the fix_applied column, and the geofix decisions are kept in the
geofix_log layer of the output GeoPackage.
The following repairs are in place:
- per-polygon attribute corrections by uuid: raw
SifferandLoimis1values are replaced byunique_idfromuid_fixes_2026.csvat the start of transform, before any repair rule or parser (83 polygons, from the loimis and prim reviews); - turvasmuld (AM, M, S, R) without prims and with a peat thickness in the
Loimis formula: prims are set from the thickness in
1b · geofix, with 30–50 cm giving’, 50–100 cm’’and more than 100 cm’’’(63 polygons); - turvasmuld without prims, smaller than 0.3 ha and without a peat thickness
(the R, M, S "holes"): the polygon is dissolved into a turvasmuld
neighbour in
1b · geofix(60 polygons), otherwise it gets one prim and keeps its geometry (6 596 polygons); - siffer and WRB of the polygons changed by geofix: re-derived for these polygons only, with the parser and lookups of transform;
- modelled depths (
zmx_model,z1_model–z4_model): left empty in assemble where a polygon has no texture layers, and taken from the prims (40, 75 or 100 cm) for peat soils whose formula gives no peat thickness.
The following repairs are still open:
- turvasmuld without prims of 0.3 ha or more and without a peat thickness
(436 polygons): these stay in manual review and can be exported with
pixi run export-open-cases; - geometry fixes for lakes, rivers etc. (before transform);
- geometry fixes for Pp soils (by uuid, before transform?).





