zettelkasten.framing

zettelkasten.framing

Question framing: reframe the existing corpus along a research question.

A research question is treated as a lens over material you already have, not a new structure to curate. frame_question decomposes a question into facets (either the landscape concept hubs most relevant to it, or agent-supplied sub-questions), semantically projects the relevant notes per facet, and characterizes each facet as established / contested / thin. It also bundles the citations referenced by each facet's evidence.

The logic is parametrized by a get_graph callable so both the MCP server and the dashboard backend (which keep separate graph caches) can share it.

frame_search_sources

frame_search_sources(project: str, graphs_dir: 'Path | None' = None, namespace: Callable[[str], str] = lambda s: s) -> list[str]

Resolve the graphs to project a question across (sources + _cross).

graphs_dir points the source/project/cross lookups at a specific .zettelkasten/ tree (defaults to the local GRAPHS_DIR). namespace maps each resolved local graph name to the name the caller's get_graph expects — for federation it prefixes <repo_id>: so the names route to the federated repo and stay correct as source_graph on the way back.

Source code in zettelkasten/framing.py

def frame_search_sources(
    project: str,
    graphs_dir: "Path | None" = None,
    namespace: Callable[[str], str] = lambda s: s,
) -> list[str]:
    """Resolve the graphs to project a question across (sources + _cross).

    ``graphs_dir`` points the source/project/cross lookups at a specific
    ``.zettelkasten/`` tree (defaults to the local ``GRAPHS_DIR``). ``namespace``
    maps each resolved local graph name to the name the caller's ``get_graph``
    expects — for federation it prefixes ``<repo_id>:`` so the names route to the
    federated repo and stay correct as ``source_graph`` on the way back.
    """
    base = graphs_dir or GRAPHS_DIR
    if project:
        proj = load_project(project, graphs_dir=base)
        names = list(proj.get("sources", [])) if proj else []
    else:
        names = [g for g in note_graph_names(graphs_dir=base) if g != "_cross"]
    if (base / "_cross").is_dir():
        names.append("_cross")
    seen: list[str] = []
    for n in names:
        if n not in seen and (base / n).is_dir():
            seen.append(n)
    return [namespace(n) for n in seen]

project_query

project_query(query: str, search_sources: list[str], top_k: int, get_graph: GetGraph, *, preserve_membership: bool = False, project: str = '', graphs_dir: 'Path | None' = None, extra_recall: 'Callable[[str, list[str], int, set[str]], tuple[set[str], dict]] | None' = None) -> list[Hit]

Hybrid (keyword + semantic) retrieval of notes for a query across graphs.

Per graph a semantic ranking (embeddings) and a tiered keyword ranking (title → tag/alias → body substring, same tiering the MCP search_notes uses) are fused with Reciprocal Rank Fusion; the per-graph fused lists are then merged by fused rank (cosine breaks ties). This is the same hybrid the rest of the store defaults to — it stops the sparse concept hubs from being drowned out by the quote/finding majority in a pure-semantic top-k, and it rewards exact factor terms (e.g. "z-score", "momentum") a vector search alone misses.

Each hit is (note_id, cosine_similarity, source_graph, keyword_tier) where cosine_similarity is the true embedding score (0.0 when the note surfaced ONLY via keyword) and keyword_tier is 0/1/2 for a title/tag/body match or None when it surfaced only via embeddings. Downstream can therefore treat an exact-term match as relevant even when its cosine is below the floor.

The similarity-recalled candidate set is then blended-reranked by :func:zettelkasten.retrieval.rerank_hits (similarity + graph-centrality importance + access-recency, diversified with MMR) and truncated to top_k. To give the reranker room, the semantic channel OVER-FETCHES (overfetch × top_k). Reranking only REORDERS the recalled set and always returns the TRUE cosine per hit, so the relevance gate in :func:_frame_facet (which filters on the real cosine or a high-precision keyword tier) is preserved exactly — an importance/recency boost can never pull a sub-floor note into a facet. When reranking is configured off (w_importance == w_recency == 0) the legacy fused-rank order is returned unchanged.

preserve_membership (used by :func:_frame_facet) fixes facet MEMBERSHIP to the legacy fused-rank top_k and lets the reranker only REORDER within it — so an importance/recency boost can never change WHICH notes are members of a facet (it may only set their presentation order). Other surfaces leave it False so the reranker can over-fetch and promote notes INTO the top_k.

project and graphs_dir are threaded to the reranker so the (opt-in) syllabus source-work importance channel scopes its cohort percentiles to the current project (and caches per scope) rather than the global corpus.

Source code in zettelkasten/framing.py

def project_query(query: str, search_sources: list[str], top_k: int,
                  get_graph: GetGraph, *, preserve_membership: bool = False,
                  project: str = "", graphs_dir: "Path | None" = None,
                  extra_recall: "Callable[[str, list[str], int, set[str]], tuple[set[str], dict]] | None" = None) -> list[Hit]:
    """Hybrid (keyword + semantic) retrieval of notes for a query across graphs.

    Per graph a semantic ranking (embeddings) and a tiered keyword ranking
    (title → tag/alias → body substring, same tiering the MCP ``search_notes``
    uses) are fused with Reciprocal Rank Fusion; the per-graph fused lists are then
    merged by fused rank (cosine breaks ties). This is the same hybrid the rest of
    the store defaults to — it stops the sparse ``concept`` hubs from being drowned
    out by the quote/finding majority in a pure-semantic top-k, and it rewards
    exact factor terms (e.g. "z-score", "momentum") a vector search alone misses.

    Each hit is ``(note_id, cosine_similarity, source_graph, keyword_tier)`` where
    ``cosine_similarity`` is the true embedding score (0.0 when the note surfaced
    ONLY via keyword) and ``keyword_tier`` is 0/1/2 for a title/tag/body match or
    ``None`` when it surfaced only via embeddings. Downstream can therefore treat
    an exact-term match as relevant even when its cosine is below the floor.

    The similarity-recalled candidate set is then blended-reranked by
    :func:`zettelkasten.retrieval.rerank_hits` (similarity + graph-centrality
    importance + access-recency, diversified with MMR) and truncated to ``top_k``.
    To give the reranker room, the semantic channel OVER-FETCHES
    (``overfetch × top_k``). Reranking only REORDERS the recalled set and always
    returns the TRUE cosine per hit, so the relevance gate in :func:`_frame_facet`
    (which filters on the real cosine or a high-precision keyword tier) is
    preserved exactly — an importance/recency boost can never pull a sub-floor
    note into a facet. When reranking is configured off (``w_importance ==
    w_recency == 0``) the legacy fused-rank order is returned unchanged.

    ``preserve_membership`` (used by :func:`_frame_facet`) fixes facet MEMBERSHIP
    to the legacy fused-rank ``top_k`` and lets the reranker only REORDER within
    it — so an importance/recency boost can never change WHICH notes are members
    of a facet (it may only set their presentation order). Other surfaces leave
    it ``False`` so the reranker can over-fetch and promote notes INTO the
    ``top_k``.

    ``project`` and ``graphs_dir`` are threaded to the reranker so the (opt-in)
    syllabus source-work importance channel scopes its cohort percentiles to the
    current project (and caches per scope) rather than the global corpus.
    """
    from memory.ranking import rrf_order
    from zettelkasten import config as zk_config

    cfg = zk_config.rerank_config()
    disabled = zk_config.rerank_disabled(cfg)
    # Over-fetch the semantic recall so the reranker has a real candidate pool to
    # promote from; when reranking is off (or membership is pinned) there is
    # nothing to gain by over-fetching, so recall exactly top_k.
    over_fetch = not disabled and not preserve_membership
    recall_k = max(top_k, cfg.overfetch * top_k) if over_fetch else top_k

    # ── Semantic recall ───────────────────────────────────────────────────────
    # Prefer ONE pre-filtered global-ANN query for the local boxes it covers, so a
    # broad scope is a single vector search instead of an O(#sources) per-box
    # fan-out (each of which cold-builds an embedding index). Boxes the global
    # index does NOT cover — the memory tree, federated peers, or a box not yet
    # mirrored in — keep the per-box search. A disabled/cold index, or a non-
    # default ``graphs_dir`` (the global store is built over the DEFAULT
    # GRAPHS_DIR, so a custom dir — e.g. tests — must stay per-box), falls fully
    # back to the legacy per-box path.
    sem_by_box: dict[str, list[tuple[str, float]]] = {}
    covered: set[str] = set()
    gi = _global_semantic_index(graphs_dir)
    if gi is not None:
        covered = gi.covered_boxes(search_sources)
        if covered:
            try:
                for box, nid, score in gi.search(query, top_k=recall_k, boxes=list(covered)):
                    sem_by_box.setdefault(box, []).append((nid, score))
            except Exception:
                covered = set()  # ANN failed — fall back to per-box for all

    # Caller-injected supplemental recall (dependency inversion, no import of the
    # caller here). Synapse uses it to route FEDERATED peers through their own
    # global ANN index — ONE pre-filtered search per peer repo, exactly like the
    # local block above — returning the boxes each peer covers (added to
    # ``covered`` so they skip the per-box fan-out below AND the keyword/rerank
    # pass unless they carry a top recall hit) and the recalled hits. Without this
    # a federated box would fan out per-box, inflating the candidate pool ~#boxes×
    # and dominating rerank. Best-effort: any failure leaves those boxes to the
    # per-box path.
    if extra_recall is not None:
        try:
            extra_covered, extra_sem = extra_recall(query, search_sources, recall_k, covered)
            covered |= set(extra_covered)
            for box, hits in extra_sem.items():
                if hits:
                    sem_by_box.setdefault(box, []).extend(hits)
        except Exception:
            pass

    # Per-box semantic for the UNCOVERED boxes (memory tree, federated peers, or a
    # box not yet in the global store).
    for sname in search_sources:
        if sname in covered:
            continue
        try:
            zg = get_graph(sname)
        except Exception:
            continue
        if getattr(zg, "embeddings", None) is None:
            continue
        try:
            sem_by_box[sname] = zg.embeddings.search(query, top_k=recall_k)
        except Exception:
            pass

    # ── Keyword channel + per-box RRF fusion ──────────────────────────────────
    # Fuse ONLY the boxes that produced a semantic hit (ANN-recalled) or are
    # searched per-box (uncovered); NEVER iterate the covered-but-unrecalled
    # boxes — not touching them is the whole point of the pre-filtered ANN.
    # Trade-off: an exact keyword-only hit in a covered box the ANN did not recall
    # is not surfaced at universe scale (semantic recall drives, keyword refines
    # within the recalled/local set). The keyword tiering is otherwise unchanged:
    # the query is tokenized so a natural-language question matches its salient
    # terms (e.g. "z-score") instead of the whole prose string as one substring.
    tokens = _salient_tokens(query)
    fuse_boxes = set(sem_by_box) | {s for s in search_sources if s not in covered}
    merged: list[tuple[str, float, str, "int | None", int]] = []
    for sname in search_sources:
        if sname not in fuse_boxes:
            continue
        try:
            zg = get_graph(sname)
        except Exception:
            continue
        if not zg.notes:
            continue

        sem = [(nid, score) for nid, score in sem_by_box.get(sname, []) if nid in zg.notes]
        sem_by_id = {nid: score for nid, score in sem}
        semantic_rank = [nid for nid, _ in sem]

        scored_kw: list[tuple[int, int, str]] = []
        if tokens:
            for pos, note in enumerate(zg.notes.values()):
                # Precise fields (title=tier0, tag/alias=tier1) match on the
                # word-boundary TOKEN SET (same tokenizer as the query), so
                # "score" hits the word `score` but not "underscore"/"scoreboard".
                title_tokens = _salient_token_set(note.title)
                title_match = any(tok in title_tokens for tok in tokens)
                tag_alias_tokens: set[str] = set()
                for t in note.tags:
                    tag_alias_tokens |= _salient_token_set(t)
                for a in note.aliases:
                    tag_alias_tokens |= _salient_token_set(a)
                tag_match = any(tok in tag_alias_tokens for tok in tokens)
                # Body (tier2) stays free-text substring matching; tier-2 hits
                # still must clear the semantic floor in `_frame_facet`, so this
                # looseness can't inflate a facet on its own.
                body = note.body.lower()
                body_match = any(tok in body for tok in tokens)
                if title_match or tag_match or body_match:
                    tier = 0 if title_match else (1 if tag_match else 2)
                    scored_kw.append((tier, pos, note.id))
        scored_kw.sort(key=lambda t: (t[0], t[1]))
        keyword_rank = [nid for _, _, nid in scored_kw]
        kw_tier = {nid: tier for tier, _, nid in scored_kw}

        fused = rrf_order([semantic_rank, keyword_rank])
        for rank, nid in enumerate(fused):
            merged.append((nid, sem_by_id.get(nid, 0.0), sname, kw_tier.get(nid), rank))

    # Global order across graphs: best fused rank first, cosine breaks ties. This
    # is the similarity-recalled candidate POOL (over-fetched) in recall order.
    merged.sort(key=lambda h: (h[4], -h[1]))
    candidates: list[Hit] = [
        (nid, sim, sname, tier) for nid, sim, sname, tier, _ in merged
    ]
    if disabled:
        # Reproduce today's fused-rank ordering byte-for-byte (pure similarity
        # reranking would reorder by cosine and lose the keyword fusion).
        return candidates[:top_k]

    from zettelkasten import retrieval

    if preserve_membership:
        # Membership = legacy fused-rank top_k; rerank only REORDERS within it
        # (input already == top_k, so no note is displaced out of the set).
        members = candidates[:top_k]
        return retrieval.rerank_hits(members, get_graph, cfg=cfg, top_k=top_k,
                                     project=project, graphs_dir=graphs_dir)

    return retrieval.rerank_hits(candidates, get_graph, cfg=cfg, top_k=top_k,
                                 project=project, graphs_dir=graphs_dir)

frame_question

frame_question(question: str, get_graph: GetGraph, project: str = '', facets: list[str] | None = None, max_facets: int = 5, per_facet_k: int = 12, thin_min_results: int = 2, thin_min_similarity: float = 0.15, graphs_dir: 'Path | None' = None, namespace: Callable[[str], str] | None = None, search_sources: list[str] | None = None, extra_recall: 'Callable | None' = None) -> dict[str, Any]

Reframe the corpus along a research question. Returns a result dict.

Returns {"error": ...} on validation/empty-corpus problems. Callers serialize as needed (the MCP tool json.dumps it; the dashboard returns it).

graphs_dir + namespace let the dashboard frame a federated repo: sources/citations are read from that repo's .zettelkasten/ and each source name is namespaced so the injected get_graph routes to it (read-only). Both default to the local store, leaving the MCP tool's behavior unchanged.

search_sources overrides the project-derived source set. When supplied, those exact source names are framed against (and get_graph must resolve every one of them). This is how synapse frames across the memory tree + intersecting ZK projects in a single pass — it hands in the union of each project's sources plus the _memory source. project then only steers curated _cross facet scoping and should normally be left empty.

Source code in zettelkasten/framing.py

def frame_question(
    question: str,
    get_graph: GetGraph,
    project: str = "",
    facets: list[str] | None = None,
    max_facets: int = 5,
    per_facet_k: int = 12,
    thin_min_results: int = 2,
    thin_min_similarity: float = 0.15,
    graphs_dir: "Path | None" = None,
    namespace: Callable[[str], str] | None = None,
    search_sources: list[str] | None = None,
    extra_recall: "Callable | None" = None,
) -> dict[str, Any]:
    """Reframe the corpus along a research question. Returns a result dict.

    Returns ``{"error": ...}`` on validation/empty-corpus problems. Callers
    serialize as needed (the MCP tool json.dumps it; the dashboard returns it).

    ``graphs_dir`` + ``namespace`` let the dashboard frame a *federated* repo:
    sources/citations are read from that repo's ``.zettelkasten/`` and each source
    name is namespaced so the injected ``get_graph`` routes to it (read-only).
    Both default to the local store, leaving the MCP tool's behavior unchanged.

    ``search_sources`` overrides the project-derived source set. When supplied,
    those exact source names are framed against (and ``get_graph`` must resolve
    every one of them). This is how synapse frames across the memory tree +
    intersecting ZK projects in a single pass — it hands in the union of each
    project's sources plus the ``_memory`` source. ``project`` then only steers
    curated ``_cross`` facet scoping and should normally be left empty.
    """
    if not question.strip():
        return {"error": "question must not be empty.", "type": "ValidationError"}

    ns = namespace or (lambda s: s)
    if search_sources is None:
        search_sources = frame_search_sources(project, graphs_dir=graphs_dir, namespace=ns)
    if not search_sources:
        return {"error": "No sources to frame against.", "type": "NotFound"}

    citations = load_citations(graphs_dir=graphs_dir)

    facet_specs: list[dict] = []
    if facets:
        facet_origin = "agent"
        facet_specs = [{"label": f, "query": f, "hub_id": None} for f in facets if f.strip()]
    else:
        base = graphs_dir or GRAPHS_DIR
        proj = load_project(project, graphs_dir=base) if project else {}
        cross_graph = ns("_cross")
        # Derive facets from the concept hubs most relevant to the question via a
        # dedicated, type-restricted retrieval pass (so sparse hubs aren't buried).
        # Framing is a lightweight, greenfield semantic lens — it is deliberately
        # NOT tied to a project's curated spine (a spine's columns are card
        # dimensions, not research-question facets); its output can instead seed
        # NEW spines. Ordering matters: (2) curated-cross scoping runs BEFORE dedup
        # (so an unclaimed _cross hub can't win dedup against a same-titled local
        # hub and then be dropped, losing the facet), and dedup runs BEFORE the
        # final max_facets cut (so a pre-dedup truncation can't drop distinct
        # facets below max_facets).
        facet_specs = _dedup_facets(
            _scope_cross_facets(
                _concept_hub_specs(question, search_sources, get_graph, max_facets),
                proj, cross_graph,
            )
        )[:max_facets]
        facet_origin = "landscape-hubs"
        if len(facet_specs) < 2:
            # Adaptive fallback: a degenerate/sparse corpus yielded <2 hubs.
            facet_specs = _broaden_facets(
                facet_specs, search_sources, get_graph, proj, cross_graph,
                question, max_facets,
            )[:max_facets]
            facet_origin = "concept-fallback"
        # If nothing hub-anchored survived, this is effectively the whole-question lens.
        if not any(f.get("hub_id") for f in facet_specs):
            facet_origin = "whole-question"

    if not facet_specs:
        facet_specs = [{"label": question, "query": question, "hub_id": None}]
        facet_origin = "whole-question"

    framed = [
        _frame_facet(f, search_sources, get_graph, citations,
                     per_facet_k, thin_min_results, thin_min_similarity,
                     project=project, graphs_dir=graphs_dir, extra_recall=extra_recall)
        for f in facet_specs
    ]

    gaps = [f["facet"] for f in framed if f["coverage"] == "thin"]
    counts = {"established": 0, "contested": 0, "thin": 0}
    for f in framed:
        counts[f["coverage"]] += 1

    return {
        "question": question,
        "project": project or "all",
        "facet_origin": facet_origin,
        "facets": framed,
        "gaps": gaps,
        "uncovered": all(f["coverage"] == "thin" for f in framed),
        "summary": {
            "facet_count": len(framed),
            "established": counts["established"],
            "contested": counts["contested"],
            "thin": counts["thin"],
        },
        "message": (
            f"Framed '{question[:60]}' across {len(search_sources)} graph(s) into "
            f"{len(framed)} facet(s) ({facet_origin}): "
            f"{counts['established']} established, {counts['contested']} contested, {counts['thin']} thin."
        ),
    }