Plan: cloud-first experiment rerun with S3 job queue and AWS runner

Angelo › Experiment Infrastructure: Memory Hardening + DVC Artifacts › Phase 2: DVC + S3 Artifact System with Experiment Manifests › Phase 2 infrastructure plan: guaranteed remote artifact state before DVC pipelines

Type: Plan · Status: active · Created: 2026-06-11

Tags: phase2,rerun,sandbox,s3,job-queue,runner,reproducibility,plan

Confirmed plan for one-click experiment reruns. Dashboard (any machine) enqueues a job spec to an S3 queue (jobs/pending|running|done|failed + jobs/logs). A cassius-runner daemon on the AWS machine claims jobs, checks out the original commit into a git-worktree sandbox under .cassius/sandboxes/, pulls DVC inputs by original content hash, runs the manifest command, copies outputs to per-run folders (.memory/artifacts/experiments/<family>/runs/<timestamp>/), pushes via DVC, and writes a new dated run manifest with rerun_of, executor, and a replication check (exact/divergent output hashes). Experiments are modeled as series of dated runs sharing a family id; dashboard groups one card per family, sorted by latest run. Also adding one-step backup_artifact (dvc add + push + verify) for the local-file-with-S3-backup pattern. Scope: memory/jobs.py, rerun_experiment() in memory/artifacts.py, cassius-runner daemon, dashboard endpoints/UI, MCP tools, tests, README docs.

---

Source: memory entry plan-c9b3afda.