zettelkasten.retrieval

zettelkasten.retrieval

Wiring for the blended recall-then-rerank retrieval policy.

Glue between the pure scoring core (:mod:zettelkasten.reranking), the access-recency sidecar (:mod:zettelkasten.usage), the reranker config (:mod:zettelkasten.config), and the retrieval surfaces (framing + server.query_topic / project_search / note search). It builds the three inputs :func:zettelkasten.reranking.rerank needs from a candidate set:

• an importance_map — each candidate graph's :func:~zettelkasten.reranking.note_importance (graph centrality), keyed by the composite (source_graph, note_id) and restricted to POSITIVE scores so an isolated note carries no importance signal (absence, never a penalty). When RerankConfig.use_source_importance is on, each note additionally inherits its source graph's syllabus work-importance (blended by max);
• a recency_map — :func:zettelkasten.usage.recency_scores per graph against the querying repo's :func:~zettelkasten.usage.session_timeline, keyed by the same composite (source_graph, note_id); and
• a get_vector accessor backed by each source graph's embedding index, so the reranker's MMR diversity pass has real vectors (and tolerates the None vectors of unembedded notes, e.g. quotes).

Everything is parametrized by a get_graph callable (the MCP server and the dashboard keep separate graph caches), and every step degrades gracefully — a missing embedder, empty usage DB, or unreadable graph never breaks retrieval, it just drops that channel's signal.

Federation caveat. Recency for a candidate whose source_graph is namespaced (<repo_id>:<box> — a federated sibling repo) resolves against the NOTE's home-repo usage DB, but :func:zettelkasten.usage.recency_scores reads only the LOCAL <ANGELO_DIR>/zettel/usage.db (it exposes no repo-root arg, and federated repos gitignore their runtime caches, so a sibling's usage DB is typically absent anyway). We therefore compute recency for LOCAL candidate graphs only and skip federated ones — they simply carry no recency signal (absence, not a penalty), which keeps the non-punitive invariant intact. Importance and MMR are unaffected (they read the live federated graph the dashboard already builds).

build_get_vector

build_get_vector(get_graph: GetGraph) -> 'Callable[[str, str], list[float] | None]'

A (source_graph, note_id) -> vector | None accessor for MMR.

Backed by each source graph's embedding index. Returns None (no vector) for an unembedded note (e.g. a quote) or any lookup failure, which the reranker treats as "no diversity contribution" rather than an error.

Source code in zettelkasten/retrieval.py

def build_get_vector(get_graph: GetGraph) -> "Callable[[str, str], list[float] | None]":
    """A ``(source_graph, note_id) -> vector | None`` accessor for MMR.

    Backed by each source graph's embedding index. Returns ``None`` (no vector)
    for an unembedded note (e.g. a quote) or any lookup failure, which the
    reranker treats as "no diversity contribution" rather than an error.
    """

    def _get_vector(source_graph: str, note_id: str) -> "list[float] | None":
        try:
            zg = get_graph(source_graph)
        except Exception:
            return None
        idx = getattr(zg, "embeddings", None)
        if idx is None:
            return None
        try:
            return idx.get_vector(note_id)
        except Exception:
            return None

    return _get_vector

rerank_hits

rerank_hits(candidates: 'list[Hit]', get_graph: GetGraph, *, cfg: 'Any' = None, top_k: 'int | None' = None, repo_root: 'Any' = None, project: str = '', graphs_dir: 'Any' = None) -> 'list[Hit]'

Rerank a similarity-recalled candidate set, returning the same hit shape.

candidates are (note_id, cosine, source_graph, keyword_tier) tuples (the shape framing.project_query yields). Blends similarity + graph-centrality importance + access-recency and diversifies with MMR per cfg (defaults to :func:zettelkasten.config.rerank_config), then returns the reranked hits — truncated to top_k when given. The returned cosine is the TRUE embedding score for each hit (never the fused score), so a caller's cosine gate stays correct.

project and graphs_dir scope the (opt-in) syllabus source-work importance channel so its cohort percentiles are project-scoped and cached per scope; they are ignored unless cfg.use_source_importance is on.

When cfg reproduces pure similarity order (both boost weights zero, :func:zettelkasten.config.rerank_disabled) the caller should have skipped this entirely, but this function ALSO self-honors that contract: it short-circuits to pure similarity order with NO blend and NO MMR, so a future surface that forgets the call-site guard can't silently reintroduce MMR diversification. (The low-level :func:reranking.rerank primitive is UNCHANGED — called directly with zero weights and mmr_lambda < 1.0 it still MMR-diversifies.)

Source code in zettelkasten/retrieval.py

def rerank_hits(
    candidates: "list[Hit]",
    get_graph: GetGraph,
    *,
    cfg: "Any" = None,
    top_k: "int | None" = None,
    repo_root: "Any" = None,
    project: str = "",
    graphs_dir: "Any" = None,
) -> "list[Hit]":
    """Rerank a similarity-recalled candidate set, returning the same hit shape.

    ``candidates`` are ``(note_id, cosine, source_graph, keyword_tier)`` tuples
    (the shape framing.project_query yields). Blends similarity + graph-centrality
    importance + access-recency and diversifies with MMR per ``cfg`` (defaults to
    :func:`zettelkasten.config.rerank_config`), then returns the reranked hits —
    truncated to ``top_k`` when given. The returned ``cosine`` is the TRUE
    embedding score for each hit (never the fused score), so a caller's cosine
    gate stays correct.

    ``project`` and ``graphs_dir`` scope the (opt-in) syllabus source-work
    importance channel so its cohort percentiles are project-scoped and cached per
    scope; they are ignored unless ``cfg.use_source_importance`` is on.

    When ``cfg`` reproduces pure similarity order (both boost weights zero,
    :func:`zettelkasten.config.rerank_disabled`) the caller should have skipped
    this entirely, but this function ALSO self-honors that contract: it
    short-circuits to pure similarity order with NO blend and NO MMR, so a future
    surface that forgets the call-site guard can't silently reintroduce MMR
    diversification. (The low-level :func:`reranking.rerank` primitive is
    UNCHANGED — called directly with zero weights and ``mmr_lambda < 1.0`` it
    still MMR-diversifies.)
    """
    if not candidates:
        return []
    cfg = cfg or zk_config.rerank_config()

    # Disabled contract: with BOTH boost channels off there is nothing to blend,
    # and per the "no silent diversification" rule MMR is off too. Force
    # mmr_lambda=1.0 through the same rerank path so the output is pure similarity
    # order (composite dedup preserved) regardless of the configured mmr_lambda.
    if (
        getattr(cfg, "w_importance", 0.0) == 0.0
        and getattr(cfg, "w_recency", 0.0) == 0.0
        and getattr(cfg, "w_calendar_recency", 0.0) == 0.0
    ):
        from dataclasses import replace

        pure_cfg = replace(cfg, mmr_lambda=1.0)
        results = reranking.rerank(
            candidates, importance_map={}, recency_map={}, cfg=pure_cfg,
            get_vector=None, limit=top_k,
        )
        hits = [
            (r["id"], r["sim"], r["source_graph"], r["keyword_tier"]) for r in results
        ]
        return hits[:top_k] if top_k is not None else hits

    ids_by_graph: dict[str, list[str]] = {}
    for c in candidates:
        nid = str(c[0])
        sg = c[2]
        ids_by_graph.setdefault(sg, []).append(nid)

    importance_map = _importance_map(get_graph, list(ids_by_graph))
    if getattr(cfg, "use_source_importance", False):
        _fold_source_importance(
            importance_map, get_graph, ids_by_graph,
            project=project, graphs_dir=graphs_dir,
        )
    timeline = usage.session_timeline(repo_root)
    recency_map = _recency_map(ids_by_graph, timeline, zk_config.session_half_life())
    calendar_map = (
        _calendar_map(
            get_graph, ids_by_graph, getattr(cfg, "calendar_half_life", 90.0),
            graphs_dir=graphs_dir,
        )
        if getattr(cfg, "w_calendar_recency", 0.0) > 0.0
        else {}
    )

    # MMR is the only consumer of get_vector; prefetch all candidate vectors in one
    # batch per source graph when it's active, else skip the fetch entirely.
    if getattr(cfg, "mmr_lambda", 1.0) < 1.0:
        _vec_cache = _prefetch_vectors(get_graph, ids_by_graph)
        get_vector = lambda sg, nid: _vec_cache.get((sg, nid))  # noqa: E731
    else:
        get_vector = None

    results = reranking.rerank(
        candidates,
        importance_map=importance_map,
        recency_map=recency_map,
        calendar_map=calendar_map,
        cfg=cfg,
        get_vector=get_vector,
        limit=top_k,
    )
    hits: list[Hit] = [
        (r["id"], r["sim"], r["source_graph"], r["keyword_tier"]) for r in results
    ]
    if top_k is not None:
        hits = hits[:top_k]
    return hits

record_notes_access

record_notes_access(pairs: 'list[tuple[str, str]]', *, session_id: 'str | None' = None, weight: float = 1.0) -> None

Record explicit/synthesis access for a batch of (graph, note_id) pairs.

Thin fan-out over :func:zettelkasten.usage.record_access (which never raises) so the synthesis hooks (remine apply, spine promote, concept-hub accept) share one call site and one weight convention. Blank entries are skipped; duplicates within a batch are de-duplicated so a note surfaced twice in one action is counted once.

Source code in zettelkasten/retrieval.py

def record_notes_access(
    pairs: "list[tuple[str, str]]", *, session_id: "str | None" = None, weight: float = 1.0
) -> None:
    """Record explicit/synthesis access for a batch of ``(graph, note_id)`` pairs.

    Thin fan-out over :func:`zettelkasten.usage.record_access` (which never
    raises) so the synthesis hooks (remine apply, spine promote, concept-hub
    accept) share one call site and one weight convention. Blank entries are
    skipped; duplicates within a batch are de-duplicated so a note surfaced twice
    in one action is counted once.
    """
    seen: set[tuple[str, str]] = set()
    for graph, note_id in pairs:
        g = str(graph or "")
        nid = str(note_id or "")
        if not g or not nid or (g, nid) in seen:
            continue
        seen.add((g, nid))
        usage.record_access(g, nid, session_id=session_id, weight=weight)