Skip to content

Datasets

Most of a zettelkasten is prose — claims, quotes, findings — but some evidence is numbers: a price series, a table of holdings, a column of returns. A dataset note is how the zettelkasten carries that numeric evidence without stuffing a spreadsheet into a markdown body. It is a typed note whose numbers live outside the note, pointed at by a small data block, and it is the thing a data-grounded claim re-computes against.

Why the numbers live outside the note

A zettelkasten note is markdown-with-frontmatter, committed to git and diffed like prose. Numbers do not belong in that body: a 50,000-row price table would swamp the note, blow up every diff, and still not be queryable. So a dataset note keeps only a pointer to where the bytes live and how to read them; the bytes themselves sit in a sidecar file, a DVC-tracked file, or a fetch cache. The note stays small and reviewable; the data stays whole and machine-readable.

This mirrors the same split the rest of angelo uses for large or regenerable artifacts — the note (like a memory entry) is the committed source of truth, and the heavy bytes are content-addressed and rebuildable.

The data block

A dataset note carries a data block in its frontmatter that records where the bytes are and how to parse them. Everything but the location is optional and inferred when omitted:

data:
  backend: sidecar | dvc | api      # optional; inferred from path/fetch when omitted
  format: csv | json | parquet      # optional; inferred from the path suffix
  content_hash: "sha256:..."        # stable identity + cache key + reproducibility pin
  schema:                           # optional column declaration
    - {name: quarter, dtype: date}
    - {name: top5_concentration, dtype: float}
  path: berkshire_13f.csv           # sidecar/dvc: a file in the source folder
  fetch: {provider: url, params: {url: "https://..."}}   # api: how to fetch
Field Meaning
backend Where the bytes come from — sidecar, dvc, or api. Inferred as api when a fetch spec is present, else sidecar.
format csv (also tsv), json (also ndjson), or parquet. Inferred from the path suffix when omitted; a fetch with no format defaults to csv.
content_hash The sha256 of the bytes. Doubles as the cache key and the reproducibility pin (see Content hash + snapshot).
schema Optional column names/dtypes. For CSV it renames/validates columns; it never implies headerless data (set header: false for that).
path A file inside the source folder, resolved with safe_join so it cannot escape the box. Covers both a committed sidecar and a DVC-tracked file.
fetch A {provider, params} spec for an api dataset — how to pull the bytes live.

Three backends, one resolver

The point of the data block is that a claim's verifier does not care where the numbers came from — it just asks the resolver for a table. There are three backends, and resolution is deliberately unified:

  • sidecar — a small file committed next to the source (e.g. a derived top5_concentration.csv). Read directly from disk. This is the right home for the small, derived frame a claim actually grounds on.
  • dvc — a large file tracked by DVC, present in git only as a <path>.dvc pointer. A path covers both sidecar and DVC: if the data file itself is absent but its .dvc pointer sits beside it, the resolver runs dvc pull to materialize it, then reads. Same path, the difference is just whether the bytes are already local.
  • api — bytes pulled from a provider (url, file, or a host-registered fetcher) and cached under <ANGELO_DIR>/zettel/datasets/<hash>.<ext> (gitignored and re-derivable, exactly like the full-text cache). The cache is served first so a later read is offline and cheap.

New api providers register with register_fetcher(provider, fn) without touching the resolver — the same host-extension seam datasets share with registered derivations and custom agents.

The resolver never raises for routine problems: a missing file, an unknown provider, or an unreadable parquet resolves to an empty table with a warning so the caller (a dashboard, an auditor) can surface it instead of crashing. Parquet support is optional — it needs pandas or pyarrow, and without either the table resolves empty with an explanatory warning rather than a hard failure.

Content hash and snapshot

A data-grounded claim is only reproducible if the frame it grounds on is frozen. That is what content_hash is for: it is the sha256 of the resolved bytes, and it serves three jobs at once — the dataset's stable identity, the api cache key, and the reproducibility pin a claim cites.

The snapshot action is how you freeze a frame. It re-reads the current source bytes (forcing past the api fetch cache) and pins their hash into the note as content_hash:

  • For an api/file dataset it also (re)caches those exact bytes under that hash, so every later read reproduces this snapshot offline — even if the live source keeps changing.
  • For a sidecar/dvc file it aligns the note's declared hash with the file on disk, clearing any drift.

At read time, a declared content_hash that no longer matches the bytes is a soft warning, not a hard error — the bytes may have legitimately changed. But when a claim pins that hash, a mismatch becomes a hard verification failure (dataset_hash_mismatch): the data moved under the claim, so its grounding no longer holds. The rule of thumb is: snapshot first, then write the claim — so the citation grounds on a frame that cannot silently shift.

The dataset tool

One dispatch tool keyed on action manages dataset notes. Read actions are always available; write actions are refused when the store is write-free (ZK_DISABLE_WRITE).

Action What it does
add Create a dataset note from a title/body plus the storage pointer (backend/format/path or fetch, content_hash, schema, tags, links).
get The note plus resolved metadata — columns, row count, backend, any resolution warning — without the row data.
values Resolve the dataset and return a paginated rows payload (offset/limit, default 200).
snapshot Re-read the current bytes and pin their hash into the note; returns the new + previous hash and whether it changed.
attach Link this dataset to a target_id (optionally cross-graph) with a relation — default measures, or supports to ground a claim, derives-from for a derived series.

From a dataset to evidence: grounding a claim

A dataset on its own is just numbers. It becomes evidence when a claim cites it via a grounding.method: data block — a data-citation that names the dataset note, its pinned content_hash, a deterministic row/column selection, the derivation that computes a statistic, and the asserted value with a tolerance. To verify, the auditor re-resolves that exact frozen frame, re-runs the derivation, and checks the result lands within tolerance of the asserted value — grounding by re-computation, not by string-matching a quote.

This is the third grounding modality alongside quote and equation, and it is produced by the data trio (data-extractor → data-scribe → auditor) when a source is declared kind: data. See Ground a claim in data for the concrete recipe.

Angelo verifies and stores — it never composes

A data-grounded claim is one evidence contribution: a value, a confidence, an epistemic status, and the pin that reproduces it. Angelo re-runs the derivation, confirms the value, and stamps the provenance — but it never interprets what a derivation means or combines several claims into a belief. A derivation is an opaque, deterministic function the store can only invoke and compare. Composition (say, fitting a genome from many verified cells) is the application's job. Keeping that line sharp is what lets the generalist store carry quantitative evidence without absorbing domain logic.

A shipped starter: dataset-profile

The zettelkasten ships one schema that exercises this path end to end — dataset-profile, in extraction-schemas.yaml. It characterizes a numeric column with re-computable summary statistics, and every dimension is grounded by an expr derivation:

dataset-profile:
  description: "Characterize a numeric column of a dataset with re-computable summary statistics."
  link_relation: measures
  dimensions:
    - {tag: central-tendency, desc: "...", evidence: data, derivation: mean:value}
    - {tag: spread,           desc: "...", evidence: data, derivation: std:value}
    - {tag: upper-tail,       desc: "...", evidence: data, derivation: quantile:value:0.95}
    - {tag: coverage,         desc: "...", evidence: data, derivation: count:value}

Because those derivations are all whitelisted expr forms (mean:col, std:col, quantile:col:<q>, count:col), they need no host registration and re-verify in any repo — so this schema is runnable as-is. A project extends it by swapping value for the column it profiles, and, for a statistic no expr can express, naming a registered derivation instead. Dropping the evidence/derivation fields from a row turns it back into a normal prose dimension grounded by a quote, so prose and data dimensions coexist in one schema.

The concrete, finance-specific schema (a decision profile grounded in 13F derivations) deliberately lives in the application repo, not this generalist library — the same domain-agnostic boundary the data-grounding design draws throughout.

Design notes

The decisions behind this, drawn as a slice of the memory tree.

  • D angelo Phase 1: data-grounded evidence substrate shippedactive
    Shipped the generalist data-grounding evidence layer for the zettelkasten so a claim can be grounded by RE-COMPUTING a statistic over a stored dataset, not only by matching a verbatim quote.
  • D Moved zettelkasten storage to committed .zettelkasten/ (mirrors .memory/)active
    Zettelkasten note data now lives in a committed .zettelkasten/ tree at the repo root, mirroring how .memory/ is committed, instead of the gitignored .angelo/zettelkasten/.
  • R Angelo
    • P zettelkastenactive
      • P Phase 2: Literature Review Systemactive
      • P Phase: Data-grounded evidence substrate (verifiable data-citations + epistemic status)active
        • D angelo Phase 1: data-grounded evidence substrate shippedactive
          • E Review-panel findings + 3 fix cycles: coverage-surface consistencyactive
          • C Checkpoint: Phase 1 data-grounding complete (125 green, P1 closed)
          • N Shipped dataset-profile schema + documented evidence/derivation in schema header
          • N Add Datasets docs page under the Zettelkasten section

Semantically related entries from the memory graph.