The Evaluation Roadmap

What VerifiMind is not yet

Before the roadmap, the disclaimer block — written so an enterprise buyer who finds this page through search cannot misread the paradox page as proof of certification:

What VerifiMind is, today, is a structured hypothesis about how a three-agent critique pipeline (novelty / scoring / code-security) might improve on single-call frontier-model critique. This roadmap is the plan to make that hypothesis falsifiable.

Pre-registered thresholds

These are stated before any data is collected. Failure to clear them does not kill the project. It triggers a public retrospective that names what changed and why.

The labeled-set size is sized to detect a 10-point F1 improvement at α = 0.05, power = 0.8: 800 items total, ≥ 400 positives per primary slice; 400 adversarial items (≥ 100 per attack family); 200 retest items for test-retest reliability; 200 human-anchor items.

The Public Roadmap — 10 milestones

Each milestone has (a) a concrete artifact, (b) a pre-registered numerical threshold for what “passes,” and (c) a named external witness identified before the work starts. Every milestone closes with a public retrospective. Silent edits to this page are blocked by a CI workflow that fails the build if roadmap-v*.md is modified without a new git tag.

Kill-conditions

The roadmap is real only if abandonment conditions are stated up front. VerifiMind-as-tool will be retired and the work continued as methodology research if any of the following observable conditions occurs, with a public retrospective explaining the trigger:

1. No external human validation by day 180 — fewer than 3 independent non-friend users complete a structured evaluation task and consent to be cited.
2. No ML co-maintainer by day 120 — no qualified collaborator signs a scoped RFC role or completes a substantive PR / review cycle despite a public role spec.
3. Label reliability failure — after two label-protocol revisions, core label κ remains below 0.60 on a representative sample.
4. Calibration failure — Z cannot show materially better-than-baseline calibrated confidence on the labeled set, with high-confidence false positives remaining frequent and unexplained.
5. Security gating failure — CS requires code execution for its advertised claims but no sandbox + threat model are shipped by day 180.
6. Benchmark falsification without learning — external benchmark shows weak performance and no credible error taxonomy is produced within 30 days.
7. Cost-revenue inversion — frontier-model and hosting costs exceed project revenue or committed sponsor funding for 3 consecutive months with no credible unit-cost path.
8. Trust harm — a buyer, vendor, or public reviewer reasonably interprets VerifiMind as claiming validated ML rigor this roadmap admits it does not yet possess.

A kill-condition firing produces a documented decision, not an automatic pivot. The decision is published, dated, and signed.

Commitment mechanism

A roadmap that lives only on a webpage is editable. Four mechanisms operate together to convert it from aspirational to binding:

1. Git tags as commitments. This document is tagged roadmap-v1.0. Any modification of a milestone date or definition requires a new tag with a diff and a public reason. git log --tags is the audit trail.
2. Milestone-keyed retrospectives. Each milestone closes with a public retrospective post. The retrospective is the artifact; the milestone is not complete until the retrospective is published.
3. Pre-named third-party witnesses. For M3, M5, M7, M8, the external party is named in advance, publicly. A milestone with a named external reviewer cannot be unilaterally marked complete.
4. Pre-registered failure conditions. Every milestone has an explicit “this counts as a miss” definition stated now, not after the fact. Post-hoc rationalization is visible because the rationalization comes after the pre-registration.

Git tags make silent edits visible. Retrospectives make silent skips visible. Witnesses make false completions visible. Pre-registered failure conditions make rationalization visible. Together they form the cheapest available substitute for the institutional review structure VerifiMind cannot afford.

What this roadmap deliberately does not promise

• It does not promise revenue, users, or adoption metrics.
• It does not promise that κ will be high — only that κ will be measured and published.
• It does not promise the co-maintainer search succeeds — only that the search happens publicly and its outcome is reported.
• It does not promise VerifiMind becomes a credible eval framework by April 2027. It promises that by April 2027, any external reader can answer “is this project methodologically serious?” from public evidence — in either direction.

Technical RFC appendix

The full technical appendix — metric definitions (Cohen’s κ, ECE, Brier, F1, ESR with formulas), evaluation dataset spec, inter-rater reliability plan, cross-family judge triangulation, external methodology mapping (RAGAS, ARES, G-Eval, Prometheus-Eval, SWE-Bench, HumanEval), sandbox & security plan, MCP version-pinning, reproducibility checklist (NeurIPS-style), co-maintainer role and compensation terms, and acceptance criteria for the co-maintainer’s first “done” — is published as the canonical markdown source on GitHub. It is the appendix an ML-literate co-maintainer needs answered before joining.