Where Falcon PR QA sits against falcon's code surface — 882 Go files across four entry-point types (measured 2026-07-30).
| Entry point | Go files | Share | Today | How it grows |
|---|---|---|---|---|
| HTTP handlers | 624 | 70.7% | coveredfull pipeline, LOCAL + LIVE | deepeningAWS-coupled handlers still bail |
| Tasks | 141 | 16.0% | bails with stub verdict | nextseparate skill, shared harness |
| Schedules | 110 | 12.5% | bails with stub verdict | future skill |
| Consumers | 7 | 0.8% | bails with stub verdict | future skill |
| Total | 882 | 100% | 70.7%of the surface in scope | 86.7%once tasks land |
Every falcon ticket the runner has touched — 25 tickets, 41 recorded runs, full evidence folder behind every verdict.
| Ticket | Mode | Scenarios | Verdict | What it showed |
|---|---|---|---|---|
| OPT-187521 | LOCAL | — | PASS_WITH_WARN | Side-effects, not just status codes: the value the handler wrote was read back from UCD and verified. |
| OPT-226244 | LIVE | 8 | ALL_PASS | Merged PR re-verified against production, side-effect absence proven for the negative cases. |
| OPT-227886 | LIVE | 9 | PASS_WITH_WARN | Prod pass with warnings surfaced instead of swallowed. |
| OPT-227950 | — | 0 | BAIL_OUT_OF_SCOPE | A non-handler PR was refused with an explicit stub verdict — the scope guard working as designed. |
| OPT-228110 | LOCAL | 8 | PARTIAL | Boot, config inject and plan synthesis completed; stack handed to the developer before an execution verdict was recorded. |
| OPT-228967 | staging | — | — | Staging-deploy verification (pods + field state), not a full QA run. |
| OPT-229081 | staging ×2 | — | — | Staging-deploy verification, not a full QA run. |
| OPT-229286 | LOCAL + LIVE | 12 | ALL_PASS | Same 12-scenario plan green pre-merge and on production. |
| OPT-229941 | LOCAL ×2 + LIVE | 12 | PARTIAL → ALL_PASS | 12 scenarios: 10 PASS — and the two that didn't were both test-data problems, not code and not the runner. One test identity was already enrolled on the partner's UAT before the test ran (the handler correctly took the already-member branch), and the happy-path identity wasn't registered on the partner's prod side at all. The verdict said exactly that instead of faking green; a clean-data re-run went 10/10 and the LIVE run confirmed it. |
| OPT-230055 | staging | — | — | Staging-deploy verification, not a full QA run. |
| OPT-230440 | LOCAL ×2 | 17 | WARN | All 17 scenarios pass on the wire — the handler works. The WARN exists because the PR's own cURL examples wrap the payload in {"body":{"params":{…}}} while the handler actually reads a flat shape: anyone copying those examples into an integration gets it wrong. A documentation bug inside the PR — not a code bug, not a runner failure — flagged loudly instead of silently reconciled. This is the discrepancy brake doing its job. |
| OPT-230702 | LIVE | 27 | ALL_PASS | Largest single plan to date — 27 scenarios against production. |
| OPT-231123 | LOCAL | 10 | ALL_PASS | Pre-merge isolated boot, full pass. |
| OPT-231958 | LOCAL + LIVE | 8 | ALL_PASS | Pre-merge pass confirmed on production after merge. |
| OPT-232074 | LOCAL + LIVE | 7 | ALL_PASS | Pre-merge pass confirmed on production after merge. |
| OPT-232075 | LOCAL | 7 | ALL_PASS | Pre-merge isolated boot, full pass. |
| OPT-233659 | — | — | — | Schedule-job trace investigation — outside v1 handler scope, evidence archived anyway. |
| OPT-233698 | LOCAL ×2 + LIVE | 7 | ALL_PASS | Two alternative config options each got their own isolated run before the LIVE confirmation. |
| OPT-235343 | LIVE | 13 | ALL_PASS | Verdict derived independently from code + this run only — pre-existing QA in Jira deliberately not used as input. |
| OPT-235603 | LOCAL | 16 | ALL_PASS | Pre-merge isolated boot, full pass. |
| OPT-235741 | staging | — | — | Staging-deploy + test-environment verification, not a full QA run. |
| OPT-236114 | LOCAL ×2 + LIVE | 14–18 | ALL_PASS | Parity proof: production responses byte-identical to the local run across all 12 comparable scenarios. |
| OPT-236399 | LIVE | 6 | ALL_PASS | Run from a developer machine over zero-trust egress — same pipeline, different seat. |
| OPT-237515 | LOCAL + LIVE ×2 | 24 | BLOCKED — external | 24/24 green locally. The LIVE re-check then isolated the blocker with three separate proofs: the prod config exists (no config-not-found), the code is deployed and correct (10/10 validation scenarios byte-identical to the LOCAL run), and falcon's own IP whitelist passed. What remains is on the partner's side: their IP allowlist doesn't cover falcon's prod egress, so the upstream sign-in returns 403 — re-checked an hour later, still 403. Entirely external — not a PR bug, not a missing config, and not a runner failure. The action item (whitelist falcon's prod egress IPs) went to the right party, with evidence attached. |
| OPT-237769 | LOCAL + LIVE | 9 / 8 | ALL_PASS | Pre-merge pass confirmed on production after merge. |
The highlighted rows are the point. A QA runner that only ever says ALL_PASS is a rubber stamp. Note what the three non-green verdicts have in common: none of them is a failure of the runner itself. Stale test data, a documentation bug inside the PR, a partner-side allowlist gap — in every case the runner's contribution was to isolate where the failure lives and say it plainly, with evidence, instead of reporting a vague red or a dishonest green.
The comment-triggered LIVE flow, exactly as it ran on falcon PR #1709: a reviewer comments falcon-qa-live → GitHub Actions runs the pipeline against production → the bot posts the verdict. Partner identifiers are redacted in these captures.
Four classes of bug it has already caught in real PRs — each one evidence-backed, not hypothetical.
cURL examples with the wrong payload shape, typo'd field names the handler silently ignores — flagged by the live typo review even when the handler shares the typo.
The LIVE re-check runs the same plan against production. On one merged PR it proved the code was deployed and configured — and that the real blocker was the partner's own IP allowlist. External blockers get named, not absorbed as failures.
What the handler wrote to Redis and UCD is read back and compared. A handler that answers politely and writes garbage still fails.
Because both modes run the identical plan, prod responses can be diffed against the pre-merge run — 12/12 comparable scenarios byte-identical on the last check.
And every finding ships with receipts. The QA comment carries reproducible, secret-free cURLs for every executed request — a reviewer can replay the whole run verbatim.
Open work, not accepted limits. The CI and test-environment tracks are the two big ones.
Post-merge QA already runs in CI: a reviewer with write access comments falcon-qa-live on a merged PR and the pipeline runs against production from the org runner. Comment-triggered by design — never auto-fired.
Pre-merge (docker-boot) QA in CI is close: the runner's cross-account image pull is already proven. The one open piece is a safe config-secret source for the runner. Until it lands, pre-merge runs happen on the developer's machine.
On-demand falcon environments via the internal Vision MCP are built — currently blocked on a DevOps-side pod fix (a service-env collision that crashes falcon's redis boot). Meanwhile the lightweight path is proven: the existing runner does functional UCD read+write against the test environment for config-less and UCD-only handlers.
Merged ≠ deployed. Next: wire the internal release-data MCP into Phase 0 so a LIVE run fires only after the change has actually reached production — today that gap is handled by judgment.
Handlers that talk to AWS services mid-request can't be exercised in the isolated stack yet — they bail to manual testing with an explicit verdict instead of pretending.
Tasks, schedules and consumers — 29% of the surface — bail out today. Each becomes its own skill on the shared harness (Jira posting, evidence layout, masking) rather than bloating this one; prod-side side-effect verification rides the same track once a prod-VPC runner exists.