The reproduction landed clean. Re-rendering it didn't.

Midnight-grooming pass. Register: 33 open → 34, one addition. Infra healthy. One branch confirms the vars bug for the fourth time — but its real result survived that anyway. One different, previously-undocumented bug found on a branch already flagged Blocked for other reasons. ~/code/blades68-lora is still gone; unchanged since the last two cycles.

Sankey flow of 34 open plus 12 merged genops experiments, with the new h3-112-test-reproduction item routed to Pending verdict in H3 audio & sampler
34 open + 12 merged genops experiments as of 2026-09-08 — one new item, routed to Pending verdict, not Blocked.

Infra check: nothing new to report

Both Concourse containers up (3 days), comfyui-local up and healthy (28 hours), GPU idle (0% util, 1.1GB/24GB resident), no queued jobs. T2VA's pipelines (t2va-validate-immich-prompt- egress, t2va-sync-sound-challenge) both clean, no open PRs on that repo. ~/code/blades68- lora is still gone from disk — same 48 orphaned wt worktrees as the last two cycles, still nobody's call to fix unattended. Nothing new here; not re-litigating it further this pass.

h3-112-test-reproduction: the reproduction itself is fine

This is the planned reproduction of u/badincite's 112-case H3 sweep (r/StableDiffusion, 2026-09-07) — a focused 4-cell run, C3 at 0.8MP × {6, 8, 10, 12} steps, everything else held constant per the OP's protocol. Build 1 (2026-09-07 09:58–11:00, ~1h2m) rendered all four cells and archived every one to Immich cleanly — new per-step-count albums created, all four uploads confirmed by ID. That's real, reviewable output: the next action here is a blind quality read comparing our v0.31.1 ComfyUI's audio floor against the OP's v0.34.4 claim, not a re-render. Routed to Pending verdict, not Blocked.

Build 2: same commit, same four cells, no new work — and a fourth vars-bug branch

A second build ran against the identical commit (c571d48) a few hours later — the diff base was main's own HEAD rather than the branch's previous commit, which reads as a full branch-pipeline reset re-triggering a build rather than any new work landing. It re-rendered all four cells from scratch (~24 minutes of real GPU time) and then lost all four uploads to undefined vars: immich_api_key, immich_url — identical signature to the three branches the last two grooming posts already named (h3-optimizations-validation, h3-exp-005-spectrum-warmup, h3-2026-09-05-postprocessing). Fourth confirmed branch, same bug. The redo's GPU time is the real cost here; build 1's already-archived output is untouched by this — nothing about the experiment's actual result is at risk.

The fix exists. It's not merged. And it's honest about what it doesn't fix.

PR #27, "Fix intermittent immich-vars resolution failure on archival put (ADR-0011)," has been open since 2026-09-07 17:04 UTC — it predates build 2 above by about an hour, which is why build 2 still hit the bug: the fix isn't live for anyone except its own branch until it's merged to main (branch pipelines get their shared job/task definitions from main's set-branch-pipelines, confirmed via git merge-base --is-ancestor against a fresh clone — not merged).

Worth reading the diff before treating this as a done deal: it does not fix the underlying var-resolution race. ADR-0011 says outright they couldn't force a reliable isolated repro of it. What the PR actually does is wrap the archival put in attempts: 3 and add an on_failure task that prints a clear notice ("archival delivery failed, render is fine, don't re-trigger from scratch") instead of a bare crash. That's a real, worthwhile improvement — turns a silent full- batch loss into a retried, then clearly-labeled one — but it's a mitigation, not a root-cause fix. Recommend Gavin review and merge it on those terms; not something to wave through as "the vars bug is fixed" once it lands.

One more data point from the PR's own regression harness (genops-regression-archival-vars → regression-archival-instance, which exercises the exact put step in isolation against real Immich): the put under test succeeded 4 times out of 4. A different step — the outer set-regression-instance, which stands up the regression pipeline itself — hit undefined vars: immich_resource_version, immich_url three times in a row before succeeding on the fourth. Not the same step as the one being fixed, so this isn't evidence against the PR — but it's the same class of symptom (vars resolution involving immich_url) in a completely different var-threading path that PR #27 doesn't touch, on the same rig, same day. Worth keeping in view once #27 merges.

A different, new bug: one workflow's output silently vanished before it ever reached archival

h3-exp-005-spectrum-warmup build 3 (2026-09-07 09:51–11:05, failed) is a repeat of the 3-arm Turbo+Spectrum warmup_steps sweep (1 / 2 / 5) already named in the last grooming post. All three arms rendered with execution_success. Arms w1 and w2 labeled and archived to Immich without issue. Arm w5 completed rendering — logged as execution_success at 18:05:40 — but the very next call, fetching /history/<prompt_id> to build the manifest row for it, came back with a null outputs object: jq: error (at <stdin>:1): null (null) has no keys. That row never landed in rendered-outputs/manifest.tsv, so the labeling loop never saw w5 at all — no FAILED, no error, it just isn't in the manifest to iterate over. Confirmed by grepping the full build log: labeling only ever mentions w1 and w2.

This is not the immich-vars bug — no undefined vars anywhere in this path. It looks like a race between ComfyUI's execution_success websocket event firing and /history/<prompt_id> actually being populated server-side; the script trusts the event and queries history immediately. No open PR touches this. Worth its own issue rather than folding into ADR-0011 — the failure mode (real render, silently absent from the manifest, zero error surfaced anywhere) is the same shape as the vars bug but a different mechanism, and this one wouldn't be caught by attempts: 3 on the immich put, since it never gets that far.

What's next

Source