Skip to content

Processing pipeline

Diagrams of the MULD2 processing steps and their dependencies, from the overall pipeline down to the individual attribute batches. Source: muld2_pipeline.drawio. Click an image to open it at full size.

Overview

One chain, transform → geofix → assemble → peat decay → rasterize. Only the last steps write the final products, under fixed names in the run directory ($MULD2_RUN_DIR, see integration/run_layout.py); everything before is an intermediate below work/:

Step pixi task Writes
1 transform work/1_transform/parsed_parquet/
1b geofix work/2_geofix/
2 assemble work/3_assemble/mv_mullakaart_parsed.gpkg (base map)
3 peat-subset, peat-clip-all, peat-clip-assemble work/4_peat/
3 peat-decay mv_mullakaart_parsed.gpkg (final)
3 gpkg-to-fgb mv_mullakaart_parsed.fgb (final)
4 rasterize-l1 rasters/{sand,silt,clay,rock}_{lo}_{hi}.tif (final)

pixi run run-pipeline runs all of them in this order. In GitLab CI (karula, manual jobs from "Run pipeline") the same tasks are the jobs transform (1–2), peat-decay (3), rasterise-l1 (4) and publish-s3, which uploads the final products only.

Pipeline overview

1 · transform (Dask)

Per-polygon corrections by unique_id first, then the Huumus, Siffer + WRB and Loimis tracks, with the attribute batches each step adds.

transform

1b · geofix

Turvasmuld polygons without prims: each one is dissolved into a turvasmuld neighbour, gets its prims, or goes to manual review.

geofix

2 · assemble

Streaming fid-window join of the parsed attributes onto the source geometry. The geofix decisions and the end-of-chain rules (empty model depths without layers, peat thickness from the prims) are applied per batch, and every touched polygon is labelled in fix_applied.

assemble

3 · peat decay

The peat soils lose thickness where drainage and cultivation have decomposed the peat. This step sits between assemble and rasterize: it reads the base map of assemble and writes the final map, mv_mullakaart_parsed.gpkg, which the rasters, the FlatGeobuf, the exports and PostGIS read. The base map is an intermediate (work/3_assemble/) and is never overwritten, because the remaining-peat raster derives from it and decaying a decayed map would decay the peat twice. Code in integration/peat_decay/.

  1. peat-subset: the peat polygons near the < 30 cm remaining-peat mask (below_30cm_peat_polygons_simplified.gpkg), with their neighbours, in independent chunks.
  2. peat-clip-all: the QGIS model (peat_clipping_qgis/*.model3, run headless through CWL in the muld2-qgis container) cuts the peat polygons along the mask, per chunk. A chunk the model fails on is retried with a rounded mask and then split by group; groups it still fails on keep their base geometry (peat_clip = model_failed).
  3. peat-clip-assemble: the cut pieces replace their polygons, split pieces get new UUIDs (parent_unique_id keeps the base id), and the remaining peat (median of peat_remaining_cm.tif) and land use (haritav / looduslik, ELME2 baaskaart) are computed per piece.
  4. peat-decay (decay_apply.py):
    • Tarn polygons get Loimis1 t₃ (also in transform);
    • the peat layers lose thickness down to the remaining peat. The modelled depths always change; the Loimis formula and the exact Z1–Z4 only where the formula has a thickness for the peat layer (t₃40/l → t₃15/l). A layer that is gone keeps its type t with 0 cm;
    • the change matrix (matrix.py) gives cultivated pieces a new siffer_1 from the remaining peat, e.g. M’’ with 20 cm → G1;
    • siffer, WRB and all Loimis1 columns are re-derived for the changed pieces with the functions of transform, and the end-of-chain rules of assemble are applied.

Every change is labelled in fix_applied (77:peat_depth:…, 79:matrix:…, 80:tarn_loimis_t3) and listed in the decay_log layer; peat_clip_log and geofix_log are carried along. All columns the peat decay adds: Field Reference, peat decay.

peat decay

Scope: siffer_1 and Loimis1 only

The peat decay changes the first soil unit and the Loimis1 formula only. The change matrix rewrites siffer_1; siffer_2–siffer_4 of a compound siffer stay as mapped. The peat layers lose thickness only in Loimis1 (loimis1, Z1–Z4, z1_model–zmx_model); Loimis2 (the lm2_ columns) and huumus keep the values of the base map, and a loss that reaches the huumus peat is only flagged (77:huumus_peat_check). The same holds for Tarn → t₃ and for the peat thickness from the prims in the model depths (assemble), and the rasters are made from Loimis1 too. siffer_2–siffer_4 are read in one place only: whether the remaining-peat raster covers a piece (peat_modelled, R, S, M or AM in any unit). Exception: geofix puts its prims on every peat unit of a siffer without prims (DG;M → DG;M’), with the thickness from the Loimis1 layers.

4 · rasterize

Vertical depth weighting of the final map (after the peat decay) and rasterisation to COGs on the reference grid (est_ref_grid_500m_buffer_2025.tif; despite the name 10 m pixels, 37 100 × 25 800). Sand, silt, clay and rock are averaged per depth interval (0–10 … 100–200 cm), weighted by the layer thicknesses from z1_model–z4_model and clipped at zmx_model.

What an interval value means:

  • Peat and humus layers (LXTYPE PEAT/HUMUS) have no mineral fine earth and count as 0 % sand/silt/clay over their thickness. The value is the mineral fine-earth share, so sand + silt + clay < 100 where part of the interval is organic. Peat and its decay are therefore visible in the upper intervals. Rock of a peat layer comes from its coarse fragments as before.
  • A 0 cm layer (peat that has decayed completely, t₃0/l) carries no weight. The layer below starts at 0 cm.
  • GRAVELS layers have unknown fine earth and are skipped; the neighbouring layer fills their depth range, as before.
  • No layer in the interval (below zmx_model): NoData (-9999), not 0. Earlier raster sets burned these cells as 0 % (GDAL reads an empty attribute as 0).

rasterize

5 · Repairs: in place and open

Some repairs cannot be made by the per-field parsers, because they need the value of a single polygon, more than one attribute, or the geometry. The list below shows where these repairs attach in the chain (status 2026-10-08, numbers from the run of 2026-10-07). Every polygon touched by one of them is labelled in the fix_applied column, and the geofix decisions are kept in the geofix_log layer of the output GeoPackage.

The following repairs are in place:

  • per-polygon attribute corrections by uuid: raw Siffer and Loimis1 values are replaced by unique_id from uid_fixes_2026.csv at the start of transform, before any repair rule or parser (83 polygons, from the loimis and prim reviews);
  • turvasmuld (AM, M, S, R) without prims and with a peat thickness in the Loimis formula: prims are set from the thickness in 1b · geofix, with 30–50 cm giving ’, 50–100 cm ’’ and more than 100 cm ’’’ (63 polygons);
  • turvasmuld without prims, smaller than 0.3 ha and without a peat thickness (the R, M, S "holes"): the polygon is dissolved into a turvasmuld neighbour in 1b · geofix (60 polygons), otherwise it gets one prim and keeps its geometry (6 596 polygons);
  • siffer and WRB of the polygons changed by geofix: re-derived for these polygons only, with the parser and lookups of transform;
  • modelled depths (zmx_model, z1_model–z4_model): left empty in assemble where a polygon has no texture layers, and taken from the prims (40, 75 or 100 cm) for peat soils whose formula gives no peat thickness.

The following repairs are still open:

  • turvasmuld without prims of 0.3 ha or more and without a peat thickness (436 polygons): these stay in manual review and can be exported with pixi run export-open-cases;
  • geometry fixes for lakes, rivers etc. (before transform);
  • geometry fixes for Pp soils (by uuid, before transform?).