zettelkasten.grounding

zettelkasten.grounding

Mechanical grounding checks for extracted notes.

The extraction protocol asks the agent to back load-bearing notes with verbatim quote notes and to never fabricate a quote (see _EVIDENCE_PROTOCOL in :mod:zettelkasten.server). That discipline used to be prompt-only. This module makes it enforceable: given a quote body and the source's own text (the cached full text plus any saved Zotero highlights), it decides whether the quote is actually present in the source and, when it can, recovers the page it sits on.

Design (deterministic, no LLM — mirrors the engine/agent split in backfill.py): - A quote is grounded when its text appears in the source. PDF text extraction is noisy (line-break hyphenation, ligatures, smart quotes, runaway whitespace), so matching is done on an aggressively normalized form and tolerates small drift via a fuzzy fallback above a high floor. - Matching against a Zotero highlight is preferred: the highlight is curated and page-labeled, so it yields the page locator the flat full-text cache cannot. - When a quote cannot be verified the result carries the closest passage actually found in the source (near_miss) so the caller can hand the agent the real text to copy instead of its invented version.

normalize_for_match

normalize_for_match(text: str) -> str

Collapse a string to a comparison form robust to PDF extraction noise.

Lowercases, removes soft hyphens / zero-width characters, folds common ligatures and smart quotes to ASCII, de-hyphenates words split across a line break (exam-\nple -> example), and collapses all whitespace to single spaces. The result is what every match in this module compares against.

Source code in zettelkasten/grounding.py

def normalize_for_match(text: str) -> str:
    """Collapse a string to a comparison form robust to PDF extraction noise.

    Lowercases, removes soft hyphens / zero-width characters, folds common
    ligatures and smart quotes to ASCII, de-hyphenates words split across a line
    break (``exam-\\nple`` -> ``example``), and collapses all whitespace to single
    spaces. The result is what every match in this module compares against.
    """
    if not text:
        return ""
    s = text
    for ch in _INVISIBLES:
        s = s.replace(ch, "")
    for src, dst in _LIGATURES.items():
        s = s.replace(src, dst)
    # De-hyphenate line-break splits: a hyphen followed by (optional spaces and)
    # a newline rejoins the word. Do this BEFORE whitespace is collapsed.
    s = re.sub(r"-\s*\n\s*", "", s)
    s = s.lower()
    s = re.sub(r"\s+", " ", s).strip()
    return s

strip_reader_note

strip_reader_note(body: str) -> str

Return the verbatim portion of a quote body, dropping any reader-note tail.

backfill._quote_body appends \n\n_Reader note: ..._ for the reader's own comment; only the quoted text before it should be checked against the source.

Source code in zettelkasten/grounding.py

def strip_reader_note(body: str) -> str:
    """Return the verbatim portion of a quote body, dropping any reader-note tail.

    ``backfill._quote_body`` appends ``\\n\\n_Reader note: ..._`` for the reader's
    own comment; only the quoted text before it should be checked against the
    source.
    """
    return body.split("\n\n_Reader note:", 1)[0]

strip_math_delimiters

strip_math_delimiters(body: str) -> str

Return the inner LaTeX of an equation body, dropping a wrapping delimiter.

Source code in zettelkasten/grounding.py

def strip_math_delimiters(body: str) -> str:
    """Return the inner LaTeX of an equation body, dropping a wrapping delimiter."""
    t = (body or "").strip()
    for open_d, close_d in _MATH_DELIMS:
        if (
            len(t) >= len(open_d) + len(close_d)
            and t.startswith(open_d)
            and t.endswith(close_d)
        ):
            return t[len(open_d): len(t) - len(close_d)].strip()
    return t

verify_equation

verify_equation(body: str, *, page: str | int | None = None, fulltext: str | None = None, context: str | None = None) -> dict[str, Any]

Decide whether an equation body is well-formed (render-valid) LaTeX.

Unlike :func:verify_quote this performs NO verbatim source match — it validates that the LaTeX is structurally sound (the render-validity guard) and records a best-effort page locator. It never rejects on a locator miss.

Parameters:

Name	Type	Description	Default
body	str	The equation note body (LaTeX, optionally delimiter-wrapped).	required
page	str | int | None	A claimed page locator (e.g. from a Zotero annotation or the reader). Echoed back and marked claimed unless a context match upgrades it.	None
fulltext	str | None	The source's cached text, used only for the advisory context match (never for validating the math itself).	None
context	str | None	Optional surrounding caption/label text (e.g. "Equation (3.4)" or the variable names in prose). When found in fulltext the locator is upgraded to context-matched — a soft provenance signal, not a gate.	None

Returns a dict with

valid (bool — render-validity), method (render-valid | empty | mojibake | unbalanced-braces | unbalanced-delimiters | unbalanced-environments), reason (str, empty when valid), page (str | None), and locator (annotation | context-matched | claimed | none).

Source code in zettelkasten/grounding.py

def verify_equation(
    body: str,
    *,
    page: str | int | None = None,
    fulltext: str | None = None,
    context: str | None = None,
) -> dict[str, Any]:
    """Decide whether an equation body is well-formed (render-valid) LaTeX.

    Unlike :func:`verify_quote` this performs NO verbatim source match — it
    validates that the LaTeX is structurally sound (the render-validity guard) and
    records a best-effort page locator. It never rejects on a locator miss.

    Args:
        body: The equation note body (LaTeX, optionally delimiter-wrapped).
        page: A claimed page locator (e.g. from a Zotero annotation or the
            reader). Echoed back and marked ``claimed`` unless a context match
            upgrades it.
        fulltext: The source's cached text, used only for the advisory context
            match (never for validating the math itself).
        context: Optional surrounding caption/label text (e.g. ``"Equation
            (3.4)"`` or the variable names in prose). When found in ``fulltext``
            the locator is upgraded to ``context-matched`` — a soft provenance
            signal, not a gate.

    Returns a dict with:
        ``valid`` (bool — render-validity), ``method``
        (``render-valid`` | ``empty`` | ``mojibake`` | ``unbalanced-braces`` |
        ``unbalanced-delimiters`` | ``unbalanced-environments``), ``reason``
        (str, empty when valid), ``page`` (str | None), and ``locator``
        (``annotation`` | ``context-matched`` | ``claimed`` | ``none``).
    """
    latex = strip_math_delimiters(body or "")
    resolved_page = str(page) if page not in (None, "") else None
    locator = "claimed" if resolved_page else "none"
    # Advisory context match — upgrades the locator, never rejects.
    if context and fulltext:
        ctx = normalize_for_match(context)
        if ctx and ctx in normalize_for_match(fulltext):
            locator = "context-matched"

    def _result(valid: bool, method: str, reason: str = "") -> dict[str, Any]:
        return {
            "valid": valid,
            "method": method,
            "reason": reason,
            "page": resolved_page,
            "locator": locator,
        }

    if not latex.strip():
        return _result(False, "empty", "Equation body is empty.")

    # Mojibake floor: a body dominated by replacement characters is mangled OCR,
    # not a transcribable equation.
    moji = sum(1 for ch in latex if ch in _MOJIBAKE_CHARS)
    if moji and moji / max(len(latex), 1) > 0.1:
        return _result(
            False,
            "mojibake",
            "Equation body is dominated by replacement/mojibake characters — "
            "transcribe the formula as LaTeX rather than pasting extracted text.",
        )

    if not _balanced_braces(latex):
        return _result(False, "unbalanced-braces", "Unbalanced { } in the LaTeX.")
    if not _balanced_left_right(latex):
        return _result(
            False, "unbalanced-delimiters", "Unbalanced \\left / \\right in the LaTeX."
        )
    if not _balanced_environments(latex):
        return _result(
            False,
            "unbalanced-environments",
            "Unbalanced or mismatched \\begin{…} / \\end{…} in the LaTeX.",
        )

    return _result(True, "render-valid")

verify_figure

verify_figure(snapshot: dict[str, Any] | None, *, page: str | int | None = None, fulltext: str | None = None, caption: str | None = None) -> dict[str, Any]

Decide whether a figure note is grounded by a resolvable snapshot.

The visual analogue of :func:verify_equation: it performs NO verbatim text match. A figure (chart/diagram/plot) is grounded when its snapshot pointer resolves to a committed image on disk — the visual ground-truth — and inferred otherwise. Like verify_equation this is ADVISORY: a missing or unresolvable snapshot lowers the grounding but NEVER rejects the note (unlike a quote, a figure has no source text to reproduce verbatim), and a locator miss is never a gate.

Parameters:

Name	Type	Description	Default
snapshot	dict[str, Any] | None	The figure's snapshot pointer ({path, content_hash, page?, rect?, ...}) or None. Grounding hinges on whether path resolves to an existing sidecar image.	required
page	str | int | None	A claimed page locator (e.g. from a Zotero annotation or the reader). Echoed back and marked claimed unless a caption match upgrades it.	None
fulltext	str | None	The source's cached text, used only for the advisory caption match (never for deciding whether the figure is grounded).	None
caption	str | None	Optional figure caption/label text (e.g. "Figure 3"). When found in fulltext the locator is upgraded to context-matched — a soft provenance signal, not a gate.	None

Returns a dict with

grounded (bool — the snapshot image resolves), method (snapshot-resolved | snapshot-missing | no-snapshot), reason (str, empty when grounded), page (str | None), and locator (context-matched | claimed | none).

Source code in zettelkasten/grounding.py

def verify_figure(
    snapshot: dict[str, Any] | None,
    *,
    page: str | int | None = None,
    fulltext: str | None = None,
    caption: str | None = None,
) -> dict[str, Any]:
    """Decide whether a ``figure`` note is grounded by a resolvable snapshot.

    The visual analogue of :func:`verify_equation`: it performs NO verbatim text
    match. A figure (chart/diagram/plot) is *grounded* when its ``snapshot``
    pointer resolves to a committed image on disk — the visual ground-truth — and
    *inferred* otherwise. Like ``verify_equation`` this is ADVISORY: a missing or
    unresolvable snapshot lowers the grounding but NEVER rejects the note (unlike
    a quote, a figure has no source text to reproduce verbatim), and a locator
    miss is never a gate.

    Args:
        snapshot: The figure's snapshot pointer (``{path, content_hash, page?,
            rect?, ...}``) or ``None``. Grounding hinges on whether ``path``
            resolves to an existing sidecar image.
        page: A claimed page locator (e.g. from a Zotero annotation or the
            reader). Echoed back and marked ``claimed`` unless a caption match
            upgrades it.
        fulltext: The source's cached text, used only for the advisory caption
            match (never for deciding whether the figure is grounded).
        caption: Optional figure caption/label text (e.g. ``"Figure 3"``). When
            found in ``fulltext`` the locator is upgraded to ``context-matched``
            — a soft provenance signal, not a gate.

    Returns a dict with:
        ``grounded`` (bool — the snapshot image resolves), ``method``
        (``snapshot-resolved`` | ``snapshot-missing`` | ``no-snapshot``),
        ``reason`` (str, empty when grounded), ``page`` (str | None), and
        ``locator`` (``context-matched`` | ``claimed`` | ``none``).
    """
    resolved_page = str(page) if page not in (None, "") else None
    locator = "claimed" if resolved_page else "none"
    # Advisory caption match — upgrades the locator, never gates grounding.
    if caption and fulltext:
        cap = normalize_for_match(caption)
        if cap and cap in normalize_for_match(fulltext):
            locator = "context-matched"

    def _result(grounded: bool, method: str, reason: str = "") -> dict[str, Any]:
        return {
            "grounded": grounded,
            "method": method,
            "reason": reason,
            "page": resolved_page,
            "locator": locator,
        }

    rel = snapshot.get("path") if isinstance(snapshot, dict) else None
    if not rel:
        return _result(
            False, "no-snapshot", "Figure has no snapshot image pointer to ground it."
        )

    # The snapshot is grounded only when its committed sidecar actually resolves
    # on disk (path-jailed to the graphs dir by ``resolve_snapshot_path``).
    from zettelkasten.equation_snapshot import resolve_snapshot_path

    if resolve_snapshot_path(str(rel)) is None:
        return _result(
            False,
            "snapshot-missing",
            "Snapshot pointer does not resolve to a committed image on disk.",
        )
    return _result(True, "snapshot-resolved")

verify_quote

verify_quote(body: str, fulltext: str | None, annotations: list[dict[str, Any]] | None = None, *, fuzzy_floor: float = DEFAULT_FUZZY_FLOOR) -> dict[str, Any]

Decide whether a quote body is grounded in the source's own text.

Parameters:

Name	Type	Description	Default
body	str	The quote note body (the verbatim passage, possibly with a reader note appended).	required
fulltext	str | None	The source's cached/extracted full text, or None when the source has no recoverable text (cold cache, no pointer).	required
annotations	list[dict[str, Any]] | None	Saved Zotero highlights (each {text, page, ...}). Preferred match because they carry a page label.	None
fuzzy_floor	float	Minimum similarity for the fuzzy fallback to count as verified.	DEFAULT_FUZZY_FLOOR

Returns a dict with

verified (bool), method (annotation | exact | fuzzy | none | unavailable), score (0..1), page (str | None, only from an annotation match), near_miss (str | None, the closest passage actually found when not verified), and confirmed_source_text (str | None). The last is the text the verifier actually PROVED is present in the source (the matched span, under this module's normalization): for an exact match, the SOURCE tokens at the matched offset (the needle's token sequence confirmed as a contiguous, word-aligned sublist of the haystack's tokens — equal to the needle for a genuine verbatim match, derived from the source for safety); for an annotation match, the OVERLAP that lies inside the real Zotero highlight (the matched highlight span when the needle's tokens are a contiguous sublist of the highlight's, else the whole highlight in the superset case) — a source-derived span, NOT the raw superset body. Containment is WORD/NUMBER-ALIGNED (token-subsequence), so a character-substring that carves across word boundaries ("2000 dollars" inside "12000 dollars") is rejected, never confirmed. Numeric tokens preserve SIGN and UNIT (-5 vs 5, 5% vs 5 are DISTINCT tokens), so a sign/magnitude/unit flip can never match and the confirmed span is itself sign/unit-preserving — a faithful quote of a -5 source confirms ...-5..., not ...5.... It is None for fuzzy (the accepted window may diverge from the supplied body) and for none / unavailable. Callers that license downstream text off a quote MUST use this source-derived span, never the caller-supplied body (which may carry a reader-note tail, a fuzzy-tolerated drift, or an annotation-superset fabrication).

Source code in zettelkasten/grounding.py

def verify_quote(
    body: str,
    fulltext: str | None,
    annotations: list[dict[str, Any]] | None = None,
    *,
    fuzzy_floor: float = DEFAULT_FUZZY_FLOOR,
) -> dict[str, Any]:
    """Decide whether a quote body is grounded in the source's own text.

    Args:
        body: The quote note body (the verbatim passage, possibly with a reader
            note appended).
        fulltext: The source's cached/extracted full text, or ``None`` when the
            source has no recoverable text (cold cache, no pointer).
        annotations: Saved Zotero highlights (each ``{text, page, ...}``).
            Preferred match because they carry a page label.
        fuzzy_floor: Minimum similarity for the fuzzy fallback to count as
            verified.

    Returns a dict with:
        ``verified`` (bool), ``method`` (``annotation`` | ``exact`` | ``fuzzy`` |
        ``none`` | ``unavailable``), ``score`` (0..1), ``page`` (str | None,
        only from an annotation match), ``near_miss`` (str | None, the closest
        passage actually found when not verified), and ``confirmed_source_text``
        (str | None). The last is the text the verifier actually PROVED is present
        in the source (the matched span, under this module's normalization): for an
        ``exact`` match, the SOURCE tokens at the matched offset (the needle's token
        sequence confirmed as a contiguous, word-aligned sublist of the haystack's
        tokens — equal to the needle for a genuine verbatim match, derived from the
        source for safety); for an ``annotation`` match, the OVERLAP that lies inside
        the real Zotero highlight (the matched highlight span when the needle's tokens
        are a contiguous sublist of the highlight's, else the whole highlight in the
        superset case) — a source-derived span, NOT the raw superset body. Containment
        is WORD/NUMBER-ALIGNED (token-subsequence), so a character-substring that carves
        across word boundaries (``"2000 dollars"`` inside ``"12000 dollars"``) is
        rejected, never confirmed. Numeric tokens preserve SIGN and UNIT (``-5`` vs
        ``5``, ``5%`` vs ``5`` are DISTINCT tokens), so a sign/magnitude/unit flip can
        never match and the confirmed span is itself sign/unit-preserving — a faithful
        quote of a ``-5`` source confirms ``...-5...``, not ``...5...``. It is ``None``
        for ``fuzzy`` (the accepted window may
        diverge from the supplied body) and for ``none`` / ``unavailable``. Callers
        that license downstream text off a quote MUST use this source-derived span,
        never the caller-supplied ``body`` (which may carry a reader-note tail,
        a fuzzy-tolerated drift, or an annotation-superset fabrication).
    """
    verbatim = strip_reader_note(body or "")
    needle = normalize_for_match(verbatim)
    if not needle:
        return {"verified": False, "method": "none", "score": 0.0, "page": None,
                "near_miss": None, "confirmed_source_text": None}

    # The source text is genuinely unavailable: the caller decides whether to
    # soft-allow. We never claim "verified" without evidence.
    if fulltext is None:
        return {"verified": False, "method": "unavailable", "score": 0.0, "page": None,
                "near_miss": None, "confirmed_source_text": None}

    # 1) Zotero highlight match — preferred, and the only path that yields a page.
    #    The supplied needle may be a SUPERSET of the highlight, so the raw body is
    #    not a proven source substring. But the real highlight IS source-derived, so
    #    the CONFIRMED span is the OVERLAP that lies inside the highlight: the matched
    #    needle span when the needle's tokens are a contiguous sublist of the
    #    highlight's, else the whole highlight (the superset case, where only the
    #    highlight is proven source). Containment is WORD-ALIGNED (token-subsequence,
    #    not raw character-substring). This is a source-derived span (never the raw
    #    caller body), so it may license a restatement of the genuinely-highlighted
    #    text while the unverified remainder cannot.
    for ann in annotations or []:
        ann_text = ann.get("text") or ""
        ann_norm = normalize_for_match(ann_text)
        if not ann_norm:
            continue
        # Word-aligned containment (NOT raw character-substring): the needle is a
        # match only when its TOKEN sequence is a contiguous sublist of the
        # highlight's tokens (needle ⊆ highlight), or the highlight's tokens are a
        # contiguous sublist of the needle's (the superset case: the caller quoted
        # MORE than the highlight). Either way the confirmed span is derived from the
        # real highlight tokens, never the caller's needle — so a char-substring carve
        # across word boundaries (``"2000 dollars"`` inside ``"12000 dollars"``) does
        # not launder to verified.
        needle_tokens = _match_tokens(needle)
        ann_tokens = _match_tokens(ann_norm)
        confirmed: str | None = None
        idx = _token_sublist_index(needle_tokens, ann_tokens)
        if idx != -1:
            # needle ⊆ highlight: the matched highlight span (source-derived).
            confirmed = " ".join(ann_tokens[idx:idx + len(needle_tokens)])
        elif _token_sublist(ann_tokens, needle_tokens):
            # highlight ⊆ needle (superset): only the real highlight is licensed.
            confirmed = " ".join(ann_tokens)
        if confirmed is not None:
            return {
                "verified": True,
                "method": "annotation",
                "score": 1.0,
                "page": (str(ann.get("page")) or None) if ann.get("page") else None,
                "near_miss": None,
                "confirmed_source_text": confirmed,
            }

    haystack = normalize_for_match(fulltext)

    # 2) Exact (normalized) WORD-ALIGNED match: the needle's tokens must appear as a
    #    contiguous sublist of the full text's tokens (a token-subsequence, NOT a raw
    #    character substring — so ``"2000 dollars"`` inside ``"12000 dollars"`` is not
    #    a match). Only here (and the annotation path) is the matched text provably
    #    present in the source, so the source-derived span is surfaced for licensing.
    needle_tokens = _match_tokens(needle)
    haystack_tokens = _match_tokens(haystack)
    exact_idx = _token_sublist_index(needle_tokens, haystack_tokens)
    if needle_tokens and exact_idx != -1:
        # Confirm the SOURCE-derived contiguous span, not the caller's raw needle.
        # For a genuine verbatim match these are equal; deriving from the source
        # tokens is the safe invariant that keeps a char-substring carve out.
        confirmed = " ".join(haystack_tokens[exact_idx:exact_idx + len(needle_tokens)])
        return {"verified": True, "method": "exact", "score": 1.0, "page": None,
                "near_miss": None, "confirmed_source_text": confirmed}

    # 3) Fuzzy fallback against the full text for extraction drift. The supplied
    #    body may diverge from the matched window (tolerated drift), so it confers
    #    no confirmed source text even though the node still verifies/grounds.
    score, window = _fuzzy_best(needle, haystack)
    if score >= fuzzy_floor:
        return {"verified": True, "method": "fuzzy", "score": round(score, 3), "page": None,
                "near_miss": None, "confirmed_source_text": None}

    near_miss = window[:240] if window else None
    return {
        "verified": False,
        "method": "none",
        "score": round(score, 3),
        "page": None,
        "near_miss": near_miss,
        "confirmed_source_text": None,
    }