zettelkasten.tables_build

zettelkasten.tables_build

Top-level matrix build / generate / stream orchestration helpers.

Mechanically split out of tables.py: table load/peek/preview, the build_matrix and generate_matrix entry points, materialize/summarize helpers and the reconcile projection.

reconcile

reconcile(get_graph: GetGraph, *, overlay: Any = None, **build_kwargs: Any) -> dict[str, Any]

Reproject (build the grid) then :func:apply_overlay — the read path.

The lens flow: a fresh build is the projection and the stored overlay is re-applied on top, so a regenerate never loses editorial work. build_kwargs are forwarded verbatim to :func:build_matrix. The rebuilt corpus's note ids are passed to :func:apply_overlay so a member_add pinning a note that has since vanished is dropped (as removed_upstream) rather than resurrected.

Source code in zettelkasten/tables_build.py

def reconcile(
    get_graph: GetGraph,
    *,
    overlay: Any = None,
    **build_kwargs: Any,
) -> dict[str, Any]:
    """Reproject (build the grid) then :func:`apply_overlay` — the read path.

    The lens flow: a fresh build is the projection and the stored overlay is
    re-applied on top, so a regenerate never loses editorial work. ``build_kwargs``
    are forwarded verbatim to :func:`build_matrix`. The rebuilt corpus's note ids
    are passed to :func:`apply_overlay` so a ``member_add`` pinning a note that has
    since vanished is dropped (as ``removed_upstream``) rather than resurrected.
    """
    table = build_matrix(get_graph, **build_kwargs)
    try:
        valid_note_ids = corpus_note_ids(
            get_graph,
            project=build_kwargs.get("project", ""),
            graph=build_kwargs.get("graph", ""),
            graphs_dir=build_kwargs.get("graphs_dir"),
            localize=build_kwargs.get("localize"),
        )
    except Exception:  # noqa: BLE001 — never fail reconcile on the existence probe
        valid_note_ids = None
    return apply_overlay(table, overlay, valid_note_ids=valid_note_ids)

build_matrix

build_matrix(get_graph: GetGraph, *, project: str = '', graph: str = '', name: str = '', table_id: str = '', title: str = '', columns: 'list[dict[str, Any]] | None' = None, row_axis: 'dict[str, Any] | None' = None, force: bool = False, peek: bool = False, preview: bool = False, extract_fn: 'Callable[[str, str], str] | None' = None, summarize_fn: 'Callable[[str, str], str] | None' = None, graphs_dir: 'Path | None' = None, localize: 'Callable[[str], str] | None' = None, matrix_view: 'dict[str, Any] | None' = None) -> dict[str, Any]

GATHER → cache → deterministic-fill → EXTRACT → synthesize → persist.

peek is cost-free: returns the stored grid (if any) + a staleness flag, never gathering, extracting, or writing. preview is the live builder probe: it GATHERs + deterministic-fills the current (possibly unsaved) column/axis config and returns it WITHOUT calling the agent (no AI) and WITHOUT persisting — so prompt columns stay [GAP] but every deterministic column fills, giving an instant grid as the user edits. A cached grid whose signature still matches the corpus + columns is returned without re-gathering unless force. Columns backed by prompt (or deterministic cells left as [GAP]) are filled by the injectable extract_fn; when none is supplied those cells stay [GAP] (the deterministic grid is still useful).

matrix_view (§11.3 spine tree → grid flatten) is wired here for real, both axes:

• rollup decides whether a dimension cell aggregates its component-of subtree.
• cols_level PIVOTS the COLUMN AXIS: on a spine-backed table it replaces the configured columns with the spine's structure nodes at that component-of depth (:func:spine_columns_at_level), each cell reading the rolled-up membership at the cut (members below the pivot roll up into it, no silent loss). See :func:_pivot_columns_for_matrix_view.

When not passed explicitly it is resolved from the backing organization's stored matrix_view (so the persisted config is no longer dead storage), defaulting to the flat grid (cols_level=None, rollup=True). That default — and any non-spine / lens table — renders BYTE-IDENTICALLY to pre-V2b behavior, so existing orgs/routes/frontend are unaffected.

Source code in zettelkasten/tables_build.py

def build_matrix(
    get_graph: GetGraph,
    *,
    project: str = "",
    graph: str = "",
    name: str = "",
    table_id: str = "",
    title: str = "",
    columns: "list[dict[str, Any]] | None" = None,
    row_axis: "dict[str, Any] | None" = None,
    force: bool = False,
    peek: bool = False,
    preview: bool = False,
    extract_fn: "Callable[[str, str], str] | None" = None,
    summarize_fn: "Callable[[str, str], str] | None" = None,
    graphs_dir: "Path | None" = None,
    localize: "Callable[[str], str] | None" = None,
    matrix_view: "dict[str, Any] | None" = None,
) -> dict[str, Any]:
    """GATHER → cache → deterministic-fill → EXTRACT → synthesize → persist.

    ``peek`` is cost-free: returns the stored grid (if any) + a staleness flag,
    never gathering, extracting, or writing. ``preview`` is the live builder
    probe: it GATHERs + deterministic-fills the *current* (possibly unsaved)
    column/axis config and returns it WITHOUT calling the agent (no AI) and
    WITHOUT persisting — so ``prompt`` columns stay ``[GAP]`` but every
    deterministic column fills, giving an instant grid as the user edits. A
    cached grid whose signature still matches the corpus + columns is returned
    without re-gathering unless ``force``. Columns backed by ``prompt`` (or
    deterministic cells left as ``[GAP]``) are filled by the injectable
    ``extract_fn``; when none is supplied those cells stay ``[GAP]`` (the
    deterministic grid is still useful).

    ``matrix_view`` (§11.3 spine tree → grid flatten) is wired here for real, both
    axes:

    * ``rollup`` decides whether a dimension cell aggregates its ``component-of``
      subtree.
    * ``cols_level`` PIVOTS the COLUMN AXIS: on a spine-backed table it replaces
      the configured columns with the spine's structure nodes at that
      ``component-of`` depth (:func:`spine_columns_at_level`), each cell reading the
      rolled-up membership at the cut (members below the pivot roll up into it, no
      silent loss). See :func:`_pivot_columns_for_matrix_view`.

    When not passed explicitly it is resolved from the backing organization's
    stored ``matrix_view`` (so the persisted config is no longer dead storage),
    defaulting to the flat grid (``cols_level=None``, ``rollup=True``). That
    default — and any non-spine / lens table — renders BYTE-IDENTICALLY to pre-V2b
    behavior, so existing orgs/routes/frontend are unaffected.
    """
    base = Path(graphs_dir) if graphs_dir is not None else GRAPHS_DIR
    if not name:
        name = graph or project or "all"
    validate_id(name, kind="review name", for_filename=True)
    if not table_id:
        table_id = "matrix"
    validate_id(table_id, kind="table id", for_filename=True)

    cols = normalize_columns(columns or [])
    axis = normalize_row_axis(row_axis)
    # Load the backing org ONCE and derive from it both (a) the spine matrix-view
    # config and (b) the grid's spine origin. For ``mv`` an explicit arg wins;
    # otherwise read the org's stored ``matrix_view`` so the persisted pivot/rollup
    # takes effect without routes.py having to thread it. The default (no org / no
    # config) is the flat grid, so the gather is byte-identical to pre-V2b.
    backing_org = _load_backing_org(
        name, table_id, project=project, graph=graph, base=base
    )
    if matrix_view is not None:
        mv = matrix_view
    else:
        _org_mv = backing_org.get("matrix_view") if isinstance(backing_org, dict) else None
        mv = _org_mv if isinstance(_org_mv, dict) else None
    # Echoed on every exit path so the frontend stamps ``gridSpineId`` cross-session.
    spine_id = _spine_id_for_org(backing_org)

    # A ported BLANK-spine axis carries an UNRESOLVED-collision risk: a foreign
    # synthesis graph minting the same bare dim id can appear AFTER the last build
    # without moving this grid's per-scope corpus signature. So for such an axis we
    # (a) load the durable attribution decisions and (b) fold a cross-graph +
    # attribution term into the cache key (:func:`_blank_spine_sig_extra`) so the
    # signature DOES move when a colliding graph or a decision changes — making a
    # cache hit safe (the persisted grid's exclusions are still valid) and a miss
    # re-gather + re-scan against the CURRENT corpus (no stale leak).
    is_blank_spine = _is_blank_spine_axis(axis)
    attributions = (
        _load_attributions(name, table_id, project=project, graph=graph, graphs_dir=base)
        if is_blank_spine
        else None
    )
    blank_sig_extra = _blank_spine_sig_extra(base, attributions) if is_blank_spine else ""

    store = _load_tables(name, base)
    stored = store["tables"].get(table_id)

    # ── peek: read-only staleness probe ──────────────────────────────────────
    if peek:
        return _peek_table(
            stored,
            table_id=table_id,
            title=title,
            cols=cols,
            axis=axis,
            project=project,
            graph=graph,
            name=name,
            base=base,
            blank_sig_extra=blank_sig_extra,
            spine_id=spine_id,
        )

    # ── spine/schema axis: rows = member spines, cols = curated dimensions ────
    # A ``spine`` group strategy adopts a schema-matrix (deterministic, agent-free)
    # rather than the generic gather: rows are the schema's member spines and cells
    # come from each spine's own membership (see :func:`_spine_schema_table`). It
    # persists as a normal table so it reopens/regenerates like any saved grid.
    if axis.get("kind") == "group" and axis.get("strategy") == "spine":
        # Author per-cell summary PARAGRAPHS as part of generation: when an agent
        # is supplied (a real build, not a peek/preview), each scaffold dimension
        # node with members gets a grounded paragraph materialized onto it. The
        # node refs are carried only to drive that synthesis and stripped before
        # the table is persisted/returned.
        want_synth = summarize_fn is not None and not preview
        table = _spine_schema_table(
            get_graph,
            axis=axis,
            project=project,
            graph=graph,
            name=name,
            table_id=table_id,
            title=title,
            cols=cols,
            base=base,
            localize=localize,
            spine_id=spine_id,
            with_node_refs=want_synth,
        )
        refs = _extract_node_refs(table)
        if preview:
            out = dict(table)
            out["stale"] = False
            out["cached"] = False
            out["exists"] = True
            out["preview"] = True
            return out
        if not force and stored is not None and stored.get("signature") == table["signature"]:
            out = dict(stored)
            out.setdefault("row_axis", normalize_row_axis(stored.get("row_axis")))
            out.setdefault("schema_version", _TABLE_SCHEMA_VERSION)
            out["stale"] = False
            out["cached"] = True
            out["exists"] = True
            out["spine_id"] = spine_id
            return out
        if want_synth and refs:
            _materialize_schema_summaries(
                get_graph, localize, table, refs,
                summarize_fn=summarize_fn, force=force, graphs_dir=base,
            )
            table["synthesis"] = compute_synthesis(table["columns"], table["rows"])
            table["attached_note_ids"] = _attached_note_ids(table["rows"])
        _persist_table(name, table, base, project=project, graph=graph)
        out = dict(table)
        out["stale"] = False
        out["cached"] = False
        out["exists"] = True
        return out

    # ── column-axis pivot (cols_level) ───────────────────────────────────────
    # When a spine-backed table requests a ``cols_level`` cut, the rendered column
    # axis becomes the spine's structure nodes at that depth. Applied here so the
    # pivoted columns feed BOTH the signature/cache key and the gather below, and
    # are returned consistently with the rows' cells. A no-op (returns ``cols``
    # unchanged) for ``cols_level=None`` / non-spine / lens — byte-identical.
    cols = _pivot_columns_for_matrix_view(get_graph, cols, mv, localize=localize)

    # ── preview: gather + deterministic fill, no AI, no persist ───────────────
    # Powers the wizard's live grid. Reflects the *current* (unsaved) config, so
    # it always re-gathers (no cache short-circuit) and never writes; ``prompt``
    # columns stay ``[GAP]`` because the agent is intentionally skipped.
    if preview:
        return _preview_table(
            get_graph,
            cols=cols,
            axis=axis,
            project=project,
            graph=graph,
            base=base,
            localize=localize,
            mv=mv,
            attributions=attributions,
            table_id=table_id,
            title=title,
            stored=stored,
            spine_id=spine_id,
        )

    # ── cache check ──────────────────────────────────────────────────────────
    signature = _table_signature(
        cols, row_axis=axis, project=project, graph=graph, name=name, base=base,
        extra=blank_sig_extra,
    )
    # A blank spine is cacheable too: its ``blank_sig_extra`` folds the cross-graph
    # corpus + attribution state into the signature, so a cache hit means no
    # colliding graph appeared and no decision changed since the build — the
    # persisted grid (exclusions baked in) is still correct. A new collider or a
    # claim/foreign edit moves the signature → miss → full re-gather + re-scan.
    if not force and stored is not None and stored.get("signature") == signature:
        out = dict(stored)
        out.setdefault("row_axis", normalize_row_axis(stored.get("row_axis")))
        out.setdefault("schema_version", _TABLE_SCHEMA_VERSION)
        out["stale"] = False
        out["cached"] = True
        out["exists"] = True
        out["spine_id"] = spine_id
        return out

    # ── GATHER + deterministic fill ──────────────────────────────────────────
    # Only digest each row's note material when we will actually EXTRACT — it is
    # grounding input for the agent, not grid data, and is stripped before the
    # rows are persisted/returned.
    prompt_cols = [c for c in cols if c["backing"] == "prompt"]
    need_material = bool(prompt_cols) and extract_fn is not None
    # A semantic ``group`` axis needs the agent to partition the corpus into rows;
    # the same injected ``extract_fn`` doubles as the grouping seam.
    group_fn = extract_fn if axis.get("strategy") == "semantic" else None
    collision_out: dict[str, Any] = {}
    rows = gather_rows(
        get_graph,
        cols,
        project=project,
        graph=graph,
        row_axis=axis,
        graphs_dir=base,
        localize=localize,
        with_material=need_material,
        group_fn=group_fn,
        matrix_view=mv,
        attributions=attributions,
        collision_out=collision_out,
    )

    # ── incremental plan: reuse unchanged cells, target the agent at changed ──
    # Compares each freshly-gathered cell's material signature to the prior grid's
    # (using ``row["material"]`` while it is still present, before it is stripped
    # below): unchanged cells are reused VERBATIM here, and the EXTRACT/summarize
    # passes below run only over the CHANGED set. ``force`` (or no prior grid)
    # marks every cell changed → the full rebuild, unchanged.
    fresh_sigs, changed = _plan_incremental_cells(rows, cols, stored, force=force)

    # ── EXTRACT (agent) for prompt columns + any leftover gaps ───────────────
    if need_material:
        _extract_prompt_cells(
            rows, prompt_cols, extract_fn, only=None if force else changed
        )
        for r in rows:
            r.pop("material", None)
    else:
        for r in rows:
            r.pop("material", None)

    # ── per-cell summaries: agent synthesis over 2+-member cells ─────────────
    if summarize_fn is not None:
        for r in rows:
            row_only = (
                None
                if force
                else {ck for (rid, ck) in changed if rid == r["id"]}
            )
            _summarize_row_cells(r, cols, summarize_fn, only=row_only)

    # Stamp the freshly-computed signatures onto every cell so the NEXT rebuild
    # can diff against them, then recompute synthesis only for columns that
    # contain a changed cell (unchanged columns keep their prior synthesis).
    _stamp_cell_signatures(rows, cols, fresh_sigs)
    dirty_cols = _synthesis_dirty_cols(cols, rows, stored, changed, force=force)
    synthesis = _merge_synthesis(cols, rows, stored, dirty_cols)
    table = {
        "table_id": table_id,
        "title": title or table_id,
        "columns": cols,
        "row_axis": axis,
        "rows": rows,
        "synthesis": synthesis,
        "signature": signature,
        "schema_version": _TABLE_SCHEMA_VERSION,
        # Drift stamps: the signature the grid was built against and the set of
        # note ids actually routed into a cell (see :func:`count_unrouted`).
        "built_signature": signature,
        "attached_note_ids": _attached_note_ids(rows),
        "generated_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
        # The grid's spine origin, PERSISTED so the persisted-list endpoint and a
        # later cross-session ``peek`` carry it (old grids lack it → read DEFAULT).
        "spine_id": spine_id,
        # Blank-spine collision bucket: the unattributable members EXCLUDED from
        # the rows (no leak) plus whether any still await attribution. Persisted so
        # a later peek surfaces the same signal. PERSISTENT + non-self-clearing: it
        # is driven by unresolved members, not signature equality, so a forced
        # regenerate cannot clear it while unclaimed ambiguous members remain.
        "needs_attribution": list(collision_out.get("needs_attribution") or []),
        "unresolved_collision": bool(collision_out.get("unresolved_collision")),
        # Persisted so a later peek can tell a fail-safe (scan-error) ambiguity from
        # a real collision; transient by nature but recomputed on every live build.
        "collision_scan_failed": bool(collision_out.get("collision_scan_failed")),
    }
    _persist_table(name, table, base, project=project, graph=graph)

    out = dict(table)
    out["stale"] = False
    out["cached"] = False
    out["exists"] = True
    return out

generate_matrix

generate_matrix(get_graph: GetGraph, *, project: str = '', graph: str = '', name: str = '', table_id: str = '', title: str = '', columns: 'list[dict[str, Any]] | None' = None, row_axis: 'dict[str, Any] | None' = None, org_id: str = '', themes: 'list[dict[str, Any]] | None' = None, force: bool = False, extract_fn: 'Callable[[str, str], str] | None' = None, summarize_fn: 'Callable[[str, str], str] | None' = None, group_fn: 'Callable[[str, str], str] | None' = None, graphs_dir: 'Path | None' = None, localize: 'Callable[[str], str] | None' = None) -> dict[str, Any]

Generate a matrix by MATERIALIZING a spine, then rendering over it.

The write path collapse (design §5): committing a matrix reifies the builder's column config into a spine and renders the matrix over the materialized graph, rather than persisting an ephemeral lens table. The flow:

1. Bind to the open matrix's org id. Identity is org_id (defaulting to organizations.org_id_for(name, table_id) — the open matrix ↔ one org binding, NEVER a hash of the column config, per §8.3).
2. Regenerate = resync in place, or readback when in sync. When that org is ALREADY a materialized spine (state == 'spine'): if it is still in sync (organizations.spine_is_synced — no corpus drift and no new routing corrections since the last materialize), no themes were supplied, and it is not a ported BLANK spine (whose staleness also tracks cross-graph/attribution state the sync stamp does not), SKIP the resync and just render over its existing edges (materialized='render'). Otherwise call resync_organization — re-route drift, re-synthesize, and preserve manual corrections/edges via reconcile — instead of minting a near-duplicate spine.
3. New / unbound matrix mints a spine. Otherwise build + persist the lens grid (so promotion sees the reconciled membership) and promote_organization materializes it: apex + one dimension node per materializable column, route the in-scope corpus into them (spine-member edges), embed the generated schema (via schema_from_columns). Base notes are NEVER mutated — the north-star invariant is honored by promote/resync, which this never bypasses.
4. Render the matrix over the spine. A single spine renders the grounded rows = sources × columns = dimensions grid (the lit-review shape) over the promoted link-form definition. When ≥2 spines share a spine-schema (group_spines_by_spine_schema), the spine-group comparison matrix (spine_group_matrix: rows = the topic apexes, columns = the shared dimensions) is returned instead.

Returns a build_matrix-shaped dict with these ADDITIONAL contract fields:

• matrix_shape — "single-spine" (rows = sources), "spine-group" (rows = topic apexes; ≥2 same-schema spines), or "lens" (degraded: no owner scope or no materializable column, so nothing was materialized).
• org_id — the bound org id (the materialize identity).
• spine_ref — the materialized synthesis graph ("" when lens).
• materialized — "promote" (minted), "resync" (updated in place), "render" (readback only — the bound spine was already in sync with the corpus + corrections, so nothing was re-materialized), or "" (degraded / not materialized).
• schema_id + spine_members — present ONLY for spine-group: the shared spine-schema fingerprint id and the member org ids.
• orphaned_corrections — ALWAYS present ([] when none): human corrections a resync's / theme prune's defunct-hub cleanup surfaced but did NOT apply (mirrors _resync_locked's channel). A build→correct→drift→ regenerate sequence would otherwise SILENTLY lose these; the frontend shows them so the loss is visible.

themes (design §8.1/§8.2) is the OPTIONAL approved-theme channel: [{"label": str, "note_ids": ["<graph>::<id>", …]}]. Each kept theme is materialized as a ROW-HUB grouping its member sources under the single spine (additive over the default per-source rows) via :func:organizations.materialize_themes — spine-side edges only, base notes untouched, hub identity durable (reconcile by row id). An empty / omitted list is the flat source-row grid (today's behavior). Themes are materialized only on a successful mint/resync (a degraded lens has no spine to hang them on).

A degenerate config (no MEMBER-BEARING dimension — an identity-only OR a pure-prompt lens, which would mint an ungrounded empty spine) or a degenerate scope degrades to a plain persisted build_matrix grid tagged matrix_shape='lens' — Generate never fails just because there is nothing to materialize. force is threaded to the underlying builds.

Source code in zettelkasten/tables_build.py

def generate_matrix(
    get_graph: GetGraph,
    *,
    project: str = "",
    graph: str = "",
    name: str = "",
    table_id: str = "",
    title: str = "",
    columns: "list[dict[str, Any]] | None" = None,
    row_axis: "dict[str, Any] | None" = None,
    org_id: str = "",
    themes: "list[dict[str, Any]] | None" = None,
    force: bool = False,
    extract_fn: "Callable[[str, str], str] | None" = None,
    summarize_fn: "Callable[[str, str], str] | None" = None,
    group_fn: "Callable[[str, str], str] | None" = None,
    graphs_dir: "Path | None" = None,
    localize: "Callable[[str], str] | None" = None,
) -> dict[str, Any]:
    """Generate a matrix by MATERIALIZING a spine, then rendering over it.

    The write path collapse (design §5): committing a matrix reifies the builder's
    column config into a spine and renders the matrix over the materialized graph,
    rather than persisting an ephemeral lens table. The flow:

    1. **Bind to the open matrix's org id.** Identity is ``org_id`` (defaulting to
       ``organizations.org_id_for(name, table_id)`` — the open matrix ↔ one org
       binding, NEVER a hash of the column config, per §8.3).
    2. **Regenerate = resync in place, or readback when in sync.** When that org is
       ALREADY a materialized spine (``state == 'spine'``): if it is still in sync
       (``organizations.spine_is_synced`` — no corpus drift and no new routing
       corrections since the last materialize), no ``themes`` were supplied, and it is
       not a ported BLANK spine (whose staleness also tracks cross-graph/attribution
       state the sync stamp does not), SKIP the resync and just render over its
       existing edges (``materialized='render'``). Otherwise call
       ``resync_organization`` — re-route drift, re-synthesize, and preserve manual
       corrections/edges via reconcile — instead of minting a near-duplicate spine.
    3. **New / unbound matrix mints a spine.** Otherwise build + persist the lens
       grid (so promotion sees the reconciled membership) and ``promote_organization``
       materializes it: apex + one dimension node per materializable column, route
       the in-scope corpus into them (``spine-member`` edges), embed the generated
       ``schema`` (via ``schema_from_columns``). Base notes are NEVER mutated — the
       north-star invariant is honored by promote/resync, which this never bypasses.
    4. **Render the matrix over the spine.** A single spine renders the grounded
       ``rows = sources × columns = dimensions`` grid (the lit-review shape) over
       the promoted link-form definition. When ≥2 spines share a **spine-schema**
       (``group_spines_by_spine_schema``), the spine-group comparison matrix
       (``spine_group_matrix``: rows = the topic apexes, columns = the shared dimensions)
       is returned instead.

    Returns a ``build_matrix``-shaped dict with these ADDITIONAL contract fields:

    * ``matrix_shape`` — ``"single-spine"`` (rows = sources), ``"spine-group"``
      (rows = topic apexes; ≥2 same-schema spines), or ``"lens"`` (degraded: no
      owner scope or no materializable column, so nothing was materialized).
    * ``org_id`` — the bound org id (the materialize identity).
    * ``spine_ref`` — the materialized synthesis graph (``""`` when ``lens``).
    * ``materialized`` — ``"promote"`` (minted), ``"resync"`` (updated in place),
      ``"render"`` (readback only — the bound spine was already in sync with the
      corpus + corrections, so nothing was re-materialized), or ``""`` (degraded /
      not materialized).
    * ``schema_id`` + ``spine_members`` — present ONLY for ``spine-group``: the
      shared spine-schema fingerprint id and the member org ids.
    * ``orphaned_corrections`` — ALWAYS present (``[]`` when none): human
      corrections a resync's / theme prune's defunct-hub cleanup surfaced but did
      NOT apply (mirrors ``_resync_locked``'s channel). A build→correct→drift→
      regenerate sequence would otherwise SILENTLY lose these; the frontend shows
      them so the loss is visible.

    ``themes`` (design §8.1/§8.2) is the OPTIONAL approved-theme channel:
    ``[{"label": str, "note_ids": ["<graph>::<id>", …]}]``. Each kept theme is
    materialized as a ROW-HUB grouping its member sources under the single spine
    (additive over the default per-source rows) via
    :func:`organizations.materialize_themes` — spine-side edges only, base notes
    untouched, hub identity durable (reconcile by row id). An empty / omitted list
    is the flat source-row grid (today's behavior). Themes are materialized only
    on a successful mint/resync (a degraded lens has no spine to hang them on).

    A degenerate config (no MEMBER-BEARING dimension — an identity-only OR a
    pure-``prompt`` lens, which would mint an ungrounded empty spine) or a
    degenerate scope degrades to a plain persisted ``build_matrix`` grid tagged
    ``matrix_shape='lens'`` — Generate never fails just because there is nothing to
    materialize. ``force`` is threaded to the underlying builds.
    """
    from zettelkasten import organizations as _orgs

    base = Path(graphs_dir) if graphs_dir is not None else GRAPHS_DIR
    if not name:
        name = graph or project or "all"
    validate_id(name, kind="review name", for_filename=True)
    if not table_id:
        table_id = "matrix"
    validate_id(table_id, kind="table id", for_filename=True)
    loc = localize or (lambda s: s)

    cols = normalize_columns(columns or [])
    axis = normalize_row_axis(row_axis)

    def _build(cols_in: list[dict[str, Any]], axis_in: dict[str, Any], ttl: str) -> dict[str, Any]:
        return build_matrix(
            get_graph,
            project=project,
            graph=graph,
            name=name,
            table_id=table_id,
            title=ttl,
            columns=cols_in,
            row_axis=axis_in,
            force=force,
            extract_fn=extract_fn,
            summarize_fn=summarize_fn,
            graphs_dir=base,
            localize=localize,
        )

    # ── spine-group readback axis: a matrix whose ROW AXIS already points at a
    # materialized spine group (strategy 'spine') is a READBACK of existing spines,
    # not a promotable lens — delegate to build_matrix's schema-matrix branch and
    # tag the shape so the caller renders it as a comparison. ────────────────────
    if axis.get("kind") == "group" and axis.get("strategy") == "spine":
        table = _build(cols, axis, title)
        table["matrix_shape"] = "spine-group"
        table.setdefault("org_id", "")
        table.setdefault("materialized", "")
        table.setdefault("orphaned_corrections", [])
        return table

    owner = _mirror_owner(name, table_id, project, graph, base)
    # ── bind the materialize identity (P2-E: never TRUST a client org_id). The
    # deterministic default is ``org_id_for(name, table_id)`` — the open matrix ↔
    # one org binding. A SUPPLIED ``org_id`` is honored ONLY when it resolves to an
    # org actually owned by THIS ``(name, table_id)`` (same review + table_id under
    # the resolved owner); a stale/copied/renamed id that loads a foreign org — or
    # nothing — falls back to the default so it can never drive promote/resync
    # against the wrong org (minting a duplicate spine). ──────────────────────────
    default_oid = _orgs.org_id_for(name, table_id) if owner is not None else ""
    supplied_oid = (org_id or "").strip()
    if owner is None:
        oid = ""  # degenerate scope: nothing to bind, client id is meaningless.
    elif not supplied_oid or supplied_oid == default_oid:
        oid = default_oid
    else:
        candidate = _orgs.load_organization(
            owner[0], owner[1], supplied_oid, graphs_dir=base, migrate=False
        )
        belongs = (
            bool(candidate)
            and str(candidate.get("review") or "") == name
            and str(candidate.get("table_id") or "") == table_id
        )
        oid = supplied_oid if belongs else default_oid

    # ── degrade: no owner scope, or NO MEMBER-BEARING column → keep a lens grid. ─
    # A materializable column becomes a spine dimension node, but only a
    # MEMBER-BEARING one (schema_tag / note_type / ccc_slot) attaches actual note
    # members. A pure-``prompt`` (or identity-only) config has none, so a promote
    # would mint an apex + empty dimension/hub nodes with ZERO grounding — an empty
    # spine (P1-A). Gate on the SAME ``_MEMBER_BEARING_BACKINGS`` the materializer
    # uses (NOT ``derive_dimension_tags``, which counts ``prompt`` as
    # materializable) so the degrade and the attach agree, and return the
    # deterministic lens grid so Generate still yields a usable matrix.
    member_bearing = [
        c for c, _tag in derive_dimension_tags(cols)
        if c.get("backing") in _orgs._MEMBER_BEARING_BACKINGS
    ]
    if owner is None or not member_bearing:
        table = _build(cols, axis, title)
        table["matrix_shape"] = "lens"
        table["org_id"] = oid
        table["spine_ref"] = ""
        table["materialized"] = ""
        table["orphaned_corrections"] = []
        return table

    owner_type, owner_name = owner
    existing = _orgs.load_organization(owner_type, owner_name, oid, graphs_dir=base, migrate=False)
    is_spine = bool(existing) and existing.get("state") == "spine"

    lens_table: "dict[str, Any] | None" = None
    materialized = ""
    # Human corrections a resync's / theme prune's defunct-hub cleanup SURFACED but
    # did not apply. Captured here and propagated onto the result so a build→
    # correct→drift→regenerate sequence never SILENTLY loses them (P1-B).
    orphaned_corrections: list[dict[str, Any]] = []
    # A ported BLANK-spine axis's staleness also depends on cross-graph collisions +
    # the attribution overlay (``_blank_spine_sig_extra``) — state ``spine_is_synced``
    # (corpus + routing overlay) deliberately does NOT observe. Its drift banner keys
    # on that fuller signal, so allowing a blank spine to readback here could leave
    # the banner saying "resync" while Generate quietly reads back. Blank spines are
    # rare and were always resynced before the readback fast-path, so exclude them:
    # they keep resyncing, which matches the banner exactly. A normal single spine's
    # ``spine_is_synced`` is a SUPERSET of its banner staleness (both cover corpus
    # drift; the gate adds overlay corrections), so the banner never asks for a resync
    # the readback would skip.
    existing_axis = normalize_row_axis((existing or {}).get("row_axis"))
    if (
        is_spine and not themes
        and not _is_blank_spine_axis(existing_axis)
        and _orgs.spine_is_synced(existing, graphs_dir=base)
    ):
        # Readback: a matrix built on a spine that hasn't drifted (same corpus, same
        # routing corrections since the last materialize) needs NO re-materialize —
        # its ``spine-member`` edges already encode the whole grid. Skip the resync
        # (a fresh gather + agent re-route + edge writes + re-synthesis) and fall
        # straight through to the render below, which projects rows × dimensions over
        # the existing edges. "Open a matrix I already built" is thus O(read), not an
        # agent round-trip. ``themes`` asks for row-hub materialization, so its
        # presence always takes the resync path (handled by the ``and not themes``).
        materialized = "render"
    elif is_spine:
        # Regenerate with drift: resync the BOUND spine in place. Resync re-routes
        # through the agent seam via a fresh gather (it does NOT reuse a stale
        # persisted grid), so no lens build is needed first — and re-building the lens
        # grid here would clobber the promoted link-form definition.
        try:
            resync_result = _orgs.resync_organization(
                owner_type, owner_name, oid,
                get_graph=get_graph, graphs_dir=base, group_fn=group_fn,
            ) or {}
            materialized = "resync"
            # Propagate the surfaced-loss channel ``_resync_locked`` deliberately
            # returns ("surfaced, not applied — the loss must never be SILENT").
            orphaned_corrections.extend(resync_result.get("orphaned_corrections") or [])
        except ValueError:
            materialized = ""
    else:
        # Mint: persist the lens grid FIRST so promotion materializes the reconciled
        # membership the author saw (a semantic/agent lens has no deterministic grid
        # otherwise), then promote it into a spine.
        lens_table = _build(cols, axis, title)
        try:
            _orgs.promote_organization(
                owner_type, owner_name, oid,
                get_graph=get_graph, graphs_dir=base, group_fn=group_fn,
            )
            materialized = "promote"
        except ValueError:
            # Empty-spine / degenerate promote → fall back to the lens grid rather
            # than surfacing a materialization error for a still-useful grid.
            materialized = ""

    # A matrix rename updates the mirrored org title (``rename_definition``) but not
    # the already-materialized apex NODE. Re-title it here so a regenerate (readback
    # OR resync) propagates the rename to the apex without a route change. The apex
    # is keyed by the durable sentinel id, so this re-titles the SAME node in place
    # (never mints a second apex) and is a no-op when the title is unchanged. The
    # mint path is skipped — a fresh promote stamps the title from the org already.
    if materialized in ("render", "resync"):
        try:
            _orgs.retitle_spine_apex(owner_type, owner_name, oid, get_graph=get_graph, graphs_dir=base)
        except Exception:  # noqa: BLE001 — an apex re-title must never fail Generate
            logger.warning("apex re-title failed for org '%s'", oid, exc_info=True)

    if not materialized:
        # Degrade to a plain grid. When the bound org is ALREADY a materialized
        # spine (a resync that failed), render over its stored link-form definition
        # so the fallback build never clobbers it back to lens-form columns; a
        # never-materialized bind falls back to the just-built lens grid.
        if lens_table is not None:
            table = dict(lens_table)
        elif is_spine and existing is not None:
            table = _build(existing.get("columns"), existing.get("row_axis"), title or str(existing.get("title") or ""))
        else:
            table = _build(cols, axis, title)
        table["matrix_shape"] = "lens"
        table["org_id"] = oid
        table["spine_ref"] = str((existing or {}).get("spine_ref") or "") if is_spine else ""
        table["materialized"] = ""
        table["orphaned_corrections"] = orphaned_corrections
        return table

    # ── themes → row-hubs (design §8.1/§8.2, P1-C). On a successful mint/resync,
    # materialize each approved theme as a durable row-hub grouping its member
    # sources under the spine (spine-side edges only; base notes untouched). An
    # empty/omitted list is the flat source-row grid (today's behavior). A theme
    # prune can surface orphaned corrections too — fold them into the channel. ────
    if themes:
        try:
            theme_result = _orgs.materialize_themes(
                owner_type, owner_name, oid, themes,
                get_graph=get_graph, graphs_dir=base,
            ) or {}
            orphaned_corrections.extend(theme_result.get("orphaned_corrections") or [])
        except Exception:  # noqa: BLE001 — a theme materialize must never fail Generate
            logger.warning("theme materialize failed for org '%s'", oid, exc_info=True)

    org = _orgs.load_organization(owner_type, owner_name, oid, graphs_dir=base, migrate=False)
    spine_ref = str((org or {}).get("spine_ref") or "").strip()

    # ── render: spine-group when ≥2 same-schema spines exist, else single spine. ─
    orgs = _orgs.list_organizations(owner_type, owner_name, graphs_dir=base, migrate=False)
    groups = group_spines_by_spine_schema(orgs)
    my_group = next(
        (g for g in groups if oid in (g.get("members") or [])), None
    )
    if my_group is not None:
        by_id = {str(o.get("id") or ""): o for o in orgs}
        members = [by_id[m] for m in my_group.get("members", []) if m in by_id]
        grid = spine_group_matrix(
            get_graph, loc,
            members=members,
            dim_keys=list(my_group.get("dimensions") or []),
            title=str(my_group.get("label") or title or ""),
        )
        grid["matrix_shape"] = "spine-group"
        grid["schema_id"] = my_group["id"]
        grid["spine_members"] = list(my_group.get("members") or [])
        grid["org_id"] = oid
        grid["spine_ref"] = spine_ref
        grid["materialized"] = materialized
        grid["orphaned_corrections"] = orphaned_corrections
        # P2-F: the spine-group grid is ephemeral (keyed by apex, never persisted),
        # but the mint/render path DID persist a single-row grid under ``table_id``
        # (the lens build / single-spine gather). On reload a cost-free ``peek``
        # serves that persisted single grid and SILENTLY reverts the spine-group
        # view. Stamp a durable shape marker onto the persisted table so a peek
        # reports ``matrix_shape='spine-group'`` (+ ``schema_id``) and the frontend
        # re-derives the group matrix instead of showing the stale single grid.
        _stamp_stored_table_shape(
            name, table_id, base,
            matrix_shape="spine-group",
            schema_id=str(my_group["id"]),
            org_id=oid,
            spine_ref=spine_ref,
        )
        return grid

    # Single spine: render the grounded rows = sources × columns = dimensions grid
    # over the promoted (link-form) definition — the same gather machinery a
    # promoted spine's matrix already uses (it reads ``spine-member`` edges), so
    # cells are grounded per-(source, dimension). Uses the org's CURRENT link-form
    # columns/row_axis, not the pre-promotion lens columns.
    table = _build(
        (org or {}).get("columns"),
        (org or {}).get("row_axis"),
        title or str((org or {}).get("title") or ""),
    )
    # Reuse the matrix synthesis to give the spine free overview paragraphs: author
    # a corpus-wide portrait per dimension and write it onto the (scaffold)
    # dimension node, so the materialized spine's notes carry real prose instead of
    # placeholder bodies. Scaffold-guarded via ``_should_synthesize`` — it never
    # overwrites a human/agent-authored body unless ``force``, so the ``render``
    # fast-path (nodes already materialized) is a cheap no-op with no LLM calls,
    # while ``force`` re-authors them and a pre-feature spine gets backfilled. Base
    # notes are untouched. Best-effort — never fails a Generate.
    if summarize_fn is not None and spine_ref:
        try:
            _materialize_single_spine_dimensions(
                get_graph, loc, table, spine_ref,
                summarize_fn=summarize_fn, force=force, graphs_dir=base,
            )
        except Exception:  # noqa: BLE001 — dimension enrichment must never fail Generate
            logger.warning("single-spine dimension synthesis failed for org '%s'", oid, exc_info=True)
    table["matrix_shape"] = "single-spine"
    table["org_id"] = oid
    table["spine_ref"] = spine_ref
    table["materialized"] = materialized
    table["orphaned_corrections"] = orphaned_corrections
    # P2-F symmetry: the single-spine render persisted this grid under ``table_id``
    # (via ``_build``), so stamp the shape marker on it too. This ALSO heals a
    # table that was previously stamped ``spine-group`` but has since dropped back
    # to a single spine (a sibling was deleted) — without it a stale spine-group
    # marker would keep a now-single matrix rendering as a (broken) group.
    _stamp_stored_table_shape(
        name, table_id, base,
        matrix_shape="single-spine",
        schema_id="",
        org_id=oid,
        spine_ref=spine_ref,
    )
    return table