zettelkasten.tables_common

zettelkasten.tables_common

Shared foundation for the synthesis-matrix engine (see :mod:zettelkasten.tables).

Mechanically split out of tables.py: constants, the _RowScope dataclass, column/cell/member builders, normalizers and other leaf helpers used across the other tables_* submodules. Import surface is preserved by re-export from zettelkasten.tables.

normalize_row_axis

normalize_row_axis(raw: Any) -> dict[str, Any]

Validate + normalize a table's row-axis descriptor to a canonical dict.

Shape: {"kind": "source"|"note"|"tag"|"note_type"|"group", "ref": str, "include": [str]}. A missing / malformed descriptor falls back to the historical source axis, so every legacy table (which has no row_axis) keeps its one-row-per-source behavior unchanged. include curates which auto-listed values become rows (empty = all).

A group axis additionally carries the aggregation controls (strategy/relation/direction/apex/instruction). These keys are emitted ONLY for group so every other axis keeps its exact historical three-key shape (callers and stored tables compare on it).

Source code in zettelkasten/tables_common.py

def normalize_row_axis(raw: Any) -> dict[str, Any]:
    """Validate + normalize a table's row-axis descriptor to a canonical dict.

    Shape: ``{"kind": "source"|"note"|"tag"|"note_type"|"group", "ref": str,
    "include": [str]}``. A missing / malformed descriptor falls back to the
    historical ``source`` axis, so every legacy table (which has no ``row_axis``)
    keeps its one-row-per-source behavior unchanged. ``include`` curates which
    auto-listed values become rows (empty = all).

    A ``group`` axis additionally carries the aggregation controls
    (``strategy``/``relation``/``direction``/``apex``/``instruction``). These keys
    are emitted ONLY for ``group`` so every other axis keeps its exact historical
    three-key shape (callers and stored tables compare on it).
    """
    if not isinstance(raw, dict):
        raw = {}
    kind = str(raw.get("kind") or "source")
    if kind not in _ROW_AXES:
        kind = "source"
    ref = str(raw.get("ref") or "").strip()
    include = [str(v).strip() for v in (raw.get("include") or []) if str(v).strip()]
    axis: dict[str, Any] = {"kind": kind, "ref": ref, "include": include}
    if kind == "group":
        strategy = str(raw.get("strategy") or "semantic").strip()
        if strategy not in _GROUP_STRATEGIES:
            strategy = "semantic"
        direction = str(raw.get("direction") or "outgoing").strip()
        if direction not in ("outgoing", "incoming"):
            direction = "outgoing"
        axis["strategy"] = strategy
        axis["relation"] = str(raw.get("relation") or "").strip()
        axis["direction"] = direction
        axis["apex"] = str(raw.get("apex") or "").strip()
        axis["instruction"] = str(raw.get("instruction") or "").strip()
        # The synthesis graph a promoted spine's hubs live in. When set, the
        # ``link`` strategy is SPINE-SCOPED: only edges whose target graph equals
        # this name count, so two spines that generated identical hub ids never
        # bleed members into each other. Empty for a non-spine link/field axis.
        axis["graph"] = str(raw.get("graph") or "").strip()
        # Target-id row scoping (additive, backward-compatible). When set, an
        # ``incoming`` link axis counts ONLY edges whose TARGET resolves to one of
        # these GRAPH-QUALIFIED nodes, so the row apex must link to one of THIS
        # spine's materialized dimension nodes — identified by (home graph + id),
        # NOT a bare id string that could collide with an unrelated graph's
        # same-minute/same-title node. Lets a ported legacy spine whose attach
        # edges carry a BLANK ``target_graph`` scope its rows to exactly its own
        # dimension nodes instead of relaxing the graph filter to "" (which drops
        # all graph scoping and leaks every foreign ``input-to`` member).
        #
        # Stored STRUCTURALLY as ``{"graph": g, "id": nid}`` dicts (NOT a
        # ``"<graph>::<id>"`` string): a synthesis-graph name can in principle
        # carry any character, and a ``::``-joined string would silently mis-split
        # — dropping the graph qualifier and rendering an EMPTY matrix — for a
        # graph name containing the separator. The dict form has no separator to
        # break on, so target scoping is robust regardless of the graph name.
        #
        # The key is emitted ONLY when populated: an empty/absent ``target_ids`` is
        # a no-op filter, and omitting it keeps every ordinary ``group`` axis
        # byte-identical to its pre-target_ids shape — so ``_table_signature``
        # (which folds the normalized axis) does NOT see a spurious change and no
        # existing grid is force-rebuilt / flagged stale on upgrade.
        target_ids: list[dict[str, str]] = []
        for v in raw.get("target_ids") or []:
            if isinstance(v, dict):
                g = str(v.get("graph") or "").strip()
                i = str(v.get("id") or "").strip()
                if i:
                    target_ids.append({"graph": g, "id": i})
        if target_ids:
            axis["target_ids"] = target_ids
        # Source-graph row scoping (additive). When set, an ``incoming`` link axis
        # admits a row ONLY when the apex (the linking claim) lives in one of these
        # source graphs. Combined with ``target_ids``, this closes the residual
        # id-collision leak that target qualification alone cannot: two ported
        # BLANK spines whose dimension nodes mint the SAME bare id string (in
        # different synthesis graphs) attribute a blank attach edge to whichever
        # spine is rendered — but the apex claim's HOME source graph disambiguates
        # them when the spines draw from DIFFERENT source-graph sets. Emitted ONLY
        # when populated, so ordinary group axes stay byte-identical.
        source_graphs = [
            str(v).strip() for v in (raw.get("source_graphs") or []) if str(v).strip()
        ]
        if source_graphs:
            axis["source_graphs"] = source_graphs
    return axis

normalize_matrix_view

normalize_matrix_view(raw: Any) -> dict[str, Any]

Validate + normalize a spine's MATRIX view config (the tree → grid flatten).

A spine is a structure TREE (§11.3); the matrix is one 2-axis rendering of it — a small config picks which tree cut is the COLUMNS axis and whether a column's cell rolls up its subtree:

• cols_level (int | None) — the component-of depth whose structure nodes become columns (apex = 0, top-level dimensions = 1, …). None (the default) means "every materialized dimension is its own column", which reproduces the historical flat depth-1 grid exactly — so an org with no matrix_view behaves identically to before V2b.
• rollup (bool, default True) — when set, a column node's cell AGGREGATES its descendants' members (the confirmed §11.3 rollup). A flat spine has no descendants, so this is a no-op there.

The grid is a VIEW CONFIG over the tree, not the storage; the tree itself (nodes + component-of edges + spine-member edges) is the source of truth. It is applied by :func:build_matrix (the column-axis pivot via :func:_pivot_columns_for_matrix_view + the rolled-up cell membership in :func:gather_rows) and also read back directly via :func:spine_readback_matrix.

Source code in zettelkasten/tables_common.py

def normalize_matrix_view(raw: Any) -> dict[str, Any]:
    """Validate + normalize a spine's MATRIX view config (the tree → grid flatten).

    A spine is a structure TREE (§11.3); the matrix is one 2-axis *rendering* of
    it — a small config picks which tree cut is the COLUMNS axis and whether a
    column's cell rolls up its subtree:

    * ``cols_level`` (int | None) — the ``component-of`` depth whose structure
      nodes become columns (apex = 0, top-level dimensions = 1, …). ``None`` (the
      default) means "every materialized dimension is its own column", which
      reproduces the historical flat depth-1 grid exactly — so an org with no
      ``matrix_view`` behaves identically to before V2b.
    * ``rollup`` (bool, default True) — when set, a column node's cell AGGREGATES
      its descendants' members (the confirmed §11.3 rollup). A flat spine has no
      descendants, so this is a no-op there.

    The grid is a VIEW CONFIG over the tree, not the storage; the tree itself
    (nodes + ``component-of`` edges + ``spine-member`` edges) is the source of
    truth. It is applied by :func:`build_matrix` (the column-axis pivot via
    :func:`_pivot_columns_for_matrix_view` + the rolled-up cell membership in
    :func:`gather_rows`) and also read back directly via :func:`spine_readback_matrix`.
    """
    if not isinstance(raw, dict):
        raw = {}
    cols_level = raw.get("cols_level")
    if cols_level is not None:
        try:
            cols_level = int(cols_level)
        except (TypeError, ValueError):
            cols_level = None
        else:
            if cols_level < 0:
                cols_level = None
    return {"cols_level": cols_level, "rollup": bool(raw.get("rollup", True))}

normalize_columns

normalize_columns(raw: Any) -> list[dict[str, Any]]

Validate + normalize a table's column list to canonical dicts.

Each column carries key (unique), label, type (render/summary semantics), backing (how it fills), ref (the schema tag / note type / CCC slot it reads), and prompt (the extraction instruction for a prompt backing). Unknown values fall back to safe defaults so a slightly malformed client payload still produces a usable table.

RELABEL-KEY CONTRACT (§11.4): a column's key is its DURABLE identity — a promoted spine keys the dimension node's source['node_id'] on it, so the key must survive a relabel for the node (and its spine-member edges + overlay corrections) to survive. This function PRESERVES an explicitly-supplied key and only slugs one from the label as a last resort when none is given; callers relabeling a column MUST send its existing key back so a label change never silently re-mints the key (and thus the durable node).

Source code in zettelkasten/tables_common.py

def normalize_columns(raw: Any) -> list[dict[str, Any]]:
    """Validate + normalize a table's column list to canonical dicts.

    Each column carries ``key`` (unique), ``label``, ``type`` (render/summary
    semantics), ``backing`` (how it fills), ``ref`` (the schema tag / note type /
    CCC slot it reads), and ``prompt`` (the extraction instruction for a
    ``prompt`` backing). Unknown values fall back to safe defaults so a slightly
    malformed client payload still produces a usable table.

    RELABEL-KEY CONTRACT (§11.4): a column's ``key`` is its DURABLE identity — a
    promoted spine keys the dimension node's ``source['node_id']`` on it, so the
    key must survive a relabel for the node (and its ``spine-member`` edges +
    overlay corrections) to survive. This function PRESERVES an explicitly-supplied
    ``key`` and only slugs one from the ``label`` as a last resort when none is
    given; callers relabeling a column MUST send its existing ``key`` back so a
    label change never silently re-mints the key (and thus the durable node).
    """
    cols: list[dict[str, Any]] = []
    seen: set[str] = set()
    for i, c in enumerate(raw or []):
        if not isinstance(c, dict):
            continue
        label = str(c.get("label") or c.get("ref") or c.get("key") or f"Column {i + 1}").strip()
        # KEY-ECHO CONTRACT (§11.4): this function is STATELESS — it cannot see the
        # prior column list, so it cannot tell a label-only RELABEL from a brand-new
        # column. An explicit ``key`` is therefore the caller's durable handle and is
        # PRESERVED verbatim; a key is slugged from the label ONLY when none is sent.
        # DOCUMENTED RESIDUAL: a relabel that arrives WITHOUT echoing its existing
        # ``key`` re-mints the key from the NEW label (key churn) — which on a
        # promoted spine re-derives the dimension ``node_id`` and orphans its
        # ``spine-member`` edges + overlay corrections. Callers relabeling a column
        # MUST send its existing ``key`` back; matching old↔new columns is the
        # caller's job (e.g. the org/lens layer), not this normalizer's.
        # The explicit key is STRIPPED, and a WHITESPACE-ONLY key (e.g. " ") is
        # treated like an absent one and falls back to the label slug. A blank key
        # cannot be a durable identity, and an un-stripped whitespace key would slip
        # past ``derive_dimension_tags``'s empty-tag skip and silently DROP the
        # dimension from one consumer (the embedded schema) while the spine node
        # still materialized it — so normalize a non-empty key here at the root.
        key = str(c.get("key") or "").strip() or _slug(label)
        # De-duplicate keys so a cell map never collides two columns.
        base_key = key
        n = 2
        while key in seen:
            key = f"{base_key}-{n}"
            n += 1
        seen.add(key)
        backing = str(c.get("backing") or "prompt")
        if backing not in _BACKINGS:
            backing = "prompt"
        ctype = str(c.get("type") or ("identity" if backing == "identity" else "extractive"))
        if ctype not in _COLUMN_TYPES:
            ctype = "extractive"
        col: dict[str, Any] = {
            "key": key,
            "label": label,
            "type": ctype,
            "backing": backing,
            "ref": str(c.get("ref") or "").strip(),
            "prompt": str(c.get("prompt") or "").strip(),
        }
        # Spine-dimension routing controls — emitted ONLY for ``schema_tag`` so
        # every other backing keeps its exact historical six-key shape. ``relation``
        # (+ ``apex``, the dimension node id) opts a column into the structural
        # link tier: notes wired to the dimension node via the schema's
        # ``attach_relation`` route in alongside exact-tag matches. Empty for a
        # lens (no materialized edges), where routing degrades to tag+semantic.
        if backing == "schema_tag":
            col["relation"] = str(c.get("relation") or "").strip()
            col["apex"] = str(c.get("apex") or "").strip()
            # The synthesis graph the dimension node (``apex``) lives in. When set,
            # the structural link tier is SPINE-SCOPED: only edges whose target
            # graph equals this name route in, so two spines sharing a generated
            # dimension id never bleed members. Empty for a lens column.
            col["graph"] = str(c.get("graph") or "").strip()
        cols.append(col)
    return cols

compute_synthesis

compute_synthesis(columns: list[dict[str, Any]], rows: list[dict[str, Any]]) -> dict[str, Any]

Per-column mechanical summary (counts / ranges / coverage) — no LLM.

For categorical columns: value counts. For numeric: count/min/max/mean. For everything else: fill coverage (how many rows are non-[GAP]). The optional AI narrative is layered on top via :func:generate_column_synthesis.

Source code in zettelkasten/tables_common.py

def compute_synthesis(columns: list[dict[str, Any]], rows: list[dict[str, Any]]) -> dict[str, Any]:
    """Per-column mechanical summary (counts / ranges / coverage) — no LLM.

    For categorical columns: value counts. For numeric: count/min/max/mean. For
    everything else: fill coverage (how many rows are non-``[GAP]``). The optional
    AI narrative is layered on top via :func:`generate_column_synthesis`.
    """
    total = len(rows)
    out: dict[str, Any] = {}
    for col in columns:
        key = col["key"]
        filled = [r["cells"][key] for r in rows if key in r["cells"] and not r["cells"][key].get("gap")]
        # Member-aware values: numeric/categorical read the cells' member titles
        # (each member contributes a value); a member-less cell (identity / AI
        # prompt) falls back to its derived ``value``.
        values: list[str] = []
        for c in filled:
            members = c.get("members") or []
            if members:
                values.extend(str(m.get("title") or "").strip() for m in members)
            else:
                values.append(str(c.get("value") or "").strip())
        values = [v for v in values if v]
        entry: dict[str, Any] = {"filled": len(filled), "total": total, "kind": col["type"]}
        if col["type"] == "numeric":
            nums = _numbers(values)
            if nums:
                entry["stats"] = {
                    "count": len(nums),
                    "min": min(nums),
                    "max": max(nums),
                    "mean": round(sum(nums) / len(nums), 4),
                }
        elif col["type"] == "categorical":
            counts: dict[str, int] = {}
            for v in values:
                counts[v] = counts.get(v, 0) + 1
            entry["counts"] = dict(sorted(counts.items(), key=lambda kv: -kv[1]))
        out[key] = entry
    return out