Production Semantic Spine Codex Checkpoint
A checkpoint on the Codex-led production-readiness lane: sensitive fallback bridge landed, active embeddings restored to a passing authority snapshot, and the failed 5k scale attempt preserved without promoting it.
This checkpoint advanced the production semantic spine without letting a failed scale-up become active authority. The engine now has a 22-row sensitive fallback bridge, refreshed source-authority receipts, a passing AIN-510 exact-cosine gate over 7,010 active Gemini vectors, and a repeatable quarantine command that preserved but removed the failed 5,000-row semantic-review expansion from active vector authority.
Local Authority
The repo is /srv/aina/aina-data-engine-room on branch codex/aina-prod-readiness-2026-06-14. This work is VDS-local only. Fusion stays paused/read-only, donor repos stay read-only, and there was no GitHub push or public release as part of this checkpoint.
The runtime boundary remains explicit: public runtime, real-user data, external writes, production telemetry, and production runtime embedding authority are still off. Claude CLI was not used for execution because the earlier checkpoint auth check returned 401; Codex handled orchestration and verification locally.
Milestones And Slices
| Milestone | Status | What changed |
|---|---|---|
| M0 - Reconcile, Freeze, And Gate The Root | Done | Gate stack refreshed, receipt/path exposure issues fixed, artifact policy proved. |
| M1 - Sensitive Fallback Bridge And Route Hardening | Done | The bridge now uses exactly 22 sensitive eval-passing rows and excludes the 3 blocked carryover rows by identity. |
| M2 - Clean, Repair, And Embed Source Families | Partial | 500 semantic-review live embeddings passed. The 5,000 scale-up failed quality gates and was quarantined. |
| M3 - Platform-Live Boundary Preparation | Pending | Final platform integration, auth/tenant/privacy/telemetry/release receipts still need a dedicated slice. |
| M4 - Donor Retirement And Founder Release Pack | Pending | Source authority is refreshed; final donor retirement ledger and founder release package remain. |
M0 and M1 are complete for this checkpoint. M2 is deliberately partial because the scale gate did its job: the 5,000-row semantic-review expansion had zero Gemini API failures, but it lowered the known-pair cosine gap below the floor. The correct response was to stop, preserve, and quarantine rather than continue to 25,000 or batch.
Failed Scale-Up Preserved, Not Promoted
The live Gemini path used gemini-embedding-2 at 768 dimensions through Vertex ADC on project aina-495702. The first 500 semantic-review live vectors passed. The 5,000 expansion completed technically, but failed quality gates because the known-pair cosine gap fell to 0.146566, below the 0.15 floor.
A new command, production-embedding-vector-authority-quarantine, now separates failed progressive tranches from active authority. For this run it matched exactly 5,000 rows after 2026-06-15T02:34:59Z in source family semantic_review, preserved the 12,010-row pre-quarantine snapshot, wrote the quarantined rows under ignored bulk storage, and restored active Parquet plus DuckDB to 7,010 rows.
uv run aina-data-engine --root /srv/aina/aina-data-engine-room \ production-embedding-vector-authority-quarantine \ --source-family semantic_review \ --created-after 2026-06-15T02:34:59Z \ --expected-quarantine-count 5000 \ --reason failed_5k_semantic_review_progressive_quality_gate
What Passed
| Check | Result |
|---|---|
| AIN-510 retrieval promotion gate | Pass, promotion_ready, cosine gap 0.190463, stale vectors 0. |
| Production chunk/vector reconciliation | Pass, vector rows match AIN-510 and DuckDB matches vector Parquet. |
| Source authority registry v2 | Pass, 35 registry rows, 25 chunk families, 7,010 vectors. |
| Source-authority start-here | Pass, top 500 and top 1,000 complete, semantic-review active vectors 1,000. |
| AIN-506 P0 gate | Pass, Vertex ADC project aina-495702, runtime authority false. |
| Runtime readiness | Pass, headless runtime ready to harden; public/runtime/user/telemetry writes false. |
| Artifact exposure scan | Pass, active findings 0; only deterministic synthetic learner IDs exempted. |
| Focused pytest and ruff | 35 passed; All checks passed. |
| Full validate | Pass. |
The artifact policy was also proved: bulk quarantine Parquet remains ignored by artifacts/*, while the small JSON and Markdown receipts are allowed through the selective unignore rules.
Do Not Skip The Diagnosis
The next agent should not run 25,000 or batch for semantic_review. First diagnose the quarantined 5,000 rows: inspect family distribution, label contamination, generic-neighbor effects, and known-pair drift. Future vector writes should carry embedding_run_id, authority_status, quality_gate_status, and quarantine_ref so rollbacks do not depend on timestamp cutoffs.
M3 and M4 remain: finalize platform-live boundary contracts and release receipts, then complete the donor retirement ledger and founder release pack.
Start Here Next
Resume in /srv/aina/aina-data-engine-room on branch codex/aina-prod-readiness-2026-06-14. Read docs/handoff/2026-06-14-production-semantic-spine-codex-checkpoint.md first. Do not unpause Fusion. Do not mutate donor repos. Do not use public runtime, real-user data, external writes, production telemetry, or production runtime embedding authority. Start by running: uv run aina-data-engine --root /srv/aina/aina-data-engine-room source-authority-start-here uv run aina-data-engine --root /srv/aina/aina-data-engine-room ain-510-retrieval-promotion-gate uv run aina-data-engine --root /srv/aina/aina-data-engine-room validate Then continue M2 by diagnosing the quarantined 5,000 semantic_review vectors, not by running 25,000 or batch. Preserve the quarantine receipt and active 7,010-vector authority unless a new gate-proven replacement snapshot is created.
Start with the quarantine diagnosis, because the engine is safest when failed semantic scale-ups are preserved as evidence instead of promoted as authority.