Concourse CI
Open-source, container-native CI/CD engine, deployed to a local Docker host tonight as a git-backed orchestrator for GPU-bound generation jobs — replacing ad hoc, uncoordinated agent dispatch with a single declarative queue.
Dogfooded 2026-08-18/19 during a long blades68-lora session: after three independent style/eval jobs collided over one shared RTX 3090 in a single night (see the GPU-queue write-up in memory), Gavin asked to try putting the actual orchestration — job definitions, sequencing, exclusivity — into git-tracked pipeline config instead of hand-rolled GPU_QUEUE.md conventions and dispatched-agent discipline.
Goals
Get a real, working, git-versioned pipeline running locally that (a) can trigger the existing ai-toolkit (Krea-2 LoRA/generation) and comfyui-local render jobs, and (b) enforces mutual exclusion on the single GPU so two jobs can never collide again, without hand-maintaining a queue file that multiple independent sessions all have to remember to update correctly.
Effectiveness
In progress, but with two real, proven wins along the way. Stood up cleanly from the official release-build docker-compose (after first grabbing the wrong, source-build compose file and having to swap it) — Postgres-backed, one worker registered, running at 127.0.0.1:8080 in about the time it takes to docker compose up. The max_in_flight: 1 step modifier proved out as a genuinely cleaner exclusivity primitive than a hand-written lock file: built a real two-job pipeline and captured an actual fly log showing one job visibly wait for the other, not just a claim that it works. The harder problem — getting the host's single GPU into Concourse's task-level containers — split into two answers depending on which tool needed it: comfyui-local already runs as a persistent host service, so a task just needs to hit its existing HTTP API on localhost:8188, no passthrough needed at all; ai-toolkit is a bare CLI script with no server, so something still has to execute a process on the host on the pipeline's behalf. First pass at that used SSH with a forced-command, allowlist-only authorized_keys entry (tested against path traversal and shell injection first, then run for real against a live low-stakes job) — safety-reviewed clean, and it worked. But Gavin found the SSH-transport layer itself hard to reason about ("byzantine and not very discoverable") and asked for an ADR — which surfaced, once written down plainly, that stock Concourse's own oci-hooks GPU-passthrough flags could let ai-toolkit just run as a normal task step with real GPU access, no bespoke transport layer at all. That's confirmed working (nvidia-smi sees the 3090 from inside a plain task container) but landed mid-session as a genuine architecture pivot, not a finished migration — and, separately, the LoRA itself got dropped from the picture entirely partway through (a real, unconfounded ablation test proved it was doing visible work, but Gavin decided a retrain wasn't worth doing right now and asked to just run base Krea-2 via the API instead).
What made it effective
The fly-log-as-proof discipline held throughout — every claim ("the lock works," "SSH-only allows one job," "GPU passthrough works") got a real trigger and a real log pasted back, not just a description of expected behavior. That habit caught the SSH allowlist actually rejecting the attacks it was supposed to reject before ever declaring it safe. Concourse's own primitives (across/max_in_flight, the /prompt→WebSocket→/history→/free job lifecycle it expects) mapped almost exactly onto what the ad hoc GPU-queue dispatch had been hand-building all night anyway — recognizing that meant the pivot away from custom locking wasn't a detour, it was converging on the same shape with less bespoke code.
Friction, pain points, surprises
The SSH-transport spike burned real GPU-adjacent effort (design, safety review, a working end-to-end test) before Gavin's "why is this so byzantine" question and the ADR it produced revealed a simpler native option had been available the whole time — worth a five-minute "does Concourse already do this?" pass before building custom plumbing next time. Nested-container GPU access is genuinely two different problems depending on whether the target already has an API (no passthrough needed) or is a bare CLI (passthrough or a transport layer required), and it's easy to conflate them mid-investigation. Two later attempts to actually carry out the oci-hooks containerization as a dispatched coding-agent task both failed with zero output for unrelated infra reasons (a CLI timeout, then a run that silently lost its execution context after an hour) — a reminder that "the technical answer is proven" and "the implementation task will actually complete" are separate risks, especially for long-running dispatched work late in a session.
Update — 2026-08-20: mutex proven under real load, GPU_QUEUE.md retired
The pipeline went from "proven with a toy job" to actually load-bearing overnight: a dispatched coding-agent push of the workflow repo reported "done" but turned out to be just a flat folder of Python scripts — the real ComfyUI workflow graphs it existed to version (krea2_turbo.json etc.) had been left out entirely, caught only by Gavin opening the repo himself. Fixed with a proper scripts//configs/workflows/ split and a rename to gavmor/comfyui-workflows once it was clear the repo was never just about one LoRA project. With that fixed, serial_groups: [gpu] got a genuine stress test: two full 44-image Krea2 batches (darkroom-realism, then Disney-cartoon) queued as separate Concourse jobs against the one shared 3090, and the second cleanly waited for the first — not a smoke test, a real production-scale confirmation. On the strength of that, Gavin gave the order to fully retire GPU_QUEUE.md as the coordination mechanism, on the spot ("Let's migrate off GPU_QUEUE.md. Right now. Top priority.").
Two sharp new gotchas surfaced doing this: fly set-pipeline silently overwrites an already-resolved secret (e.g. a deploy key) with the literal ((var)) placeholder if you omit -l <vars.yml> on a re-apply — no error, just a pipeline that breaks on its next real run. And serial_groups turned out to be pipeline-scoped, not global — two different pipelines can't actually mutex each other through it, even though jobs in the same pipeline do; real cross-pipeline exclusivity for one shared GPU still has to come from the resource itself, not Concourse's scheduling.
Status update: training/LoRA work and quick ad hoc one-off experiments — most of GPU_QUEUE.md's actual historical traffic — still have no Concourse job and no mutex protection. Flagged openly in the retirement notice rather than pretending coverage is complete.
Update — 2026-08-21: four parallel experiments, real branch-instance-group use, and a trust gap
First night the pipeline carried genuinely parallel, independent work instead of a sequential stress test: four separate H3 attention/audio experiments (Sage+SolAttn, sparse-attention, prompt-builder consistency, audio-latent-mask) each got their own git branch, and Concourse's branch-instance-group pattern turned each branch into its own pipeline instance automatically — 8 live pipelines by the end of the night (2 standalone, 6 in the comfyui-workflows instance group) with zero manual pipeline wiring per experiment. serial_groups: [gpu] held all four to the single 3090 without a collision, same guarantee the 2026-08-20 update proved, now exercised on four genuinely different workloads instead of two copies of the same job. Merge to main doubled as the actual quality gate — no test suite exists or is needed for generative-render output, so "does the human like what came out" is the CI check, and all four branches merged clean (--no-ff, no conflicts) once Gavin reviewed the renders.
Separately, the gpu-lock "release" bug flagged as unexplained on 2026-08-20 got root-caused: concourse-locks' unclaimed/ directory doesn't survive being empty in git (no .gitkeep), so a fully-released lock silently vanished from version control rather than reflecting reality — fixed by committing placeholder files, confirmed via commit 905eaef.
The sharper lesson wasn't about Concourse's mechanics but about trusting a report of what it's running versus checking git state directly. A pasted "forensic analysis" doc made specific claims about this exact setup (the .gitkeep root cause above, plus an uncommitted Immich captioning job) and got dismissed as fabricated on first read — reasonably, since neither showed up in session memory. Both turned out to be true; the concourse-locks repo is private and had simply never been cloned locally, so its existence was invisible to anything short of git log against the real remote. Once Concourse is the thing actually running your infrastructure, "I don't remember this" stops being evidence of anything — the pipeline and its git history are the source of truth, not the agent's summarized recollection of past sessions.
What made it effective (update)
The branch-instance-group pattern removed the last bit of manual toil from running parallel experiments: name a branch, push it, get a pipeline. Combined with fly-log-as-proof discipline holding from day one, every one of the four experiment claims ("SLA sparse-attention saves 7%," "audio-latent-mask reproduces word-for-word") got verified against a real render and a real transcript, not a description of expected behavior — which is exactly what caught one experiment ("sage+sol-attn") as a wash rather than a win once the actual numbers were compared side by side.
Friction, pain points, surprises (update)
The trust-gap incident above is the more interesting failure than any Concourse mechanic: a system that's genuinely running unattended, 24/7, across private repos will eventually generate true claims about itself that don't match anyone's summarized memory of it. The fix isn't "trust reports more" or "trust reports less" — it's "check the git history for anything checkable before calling it fabricated," which took under three minutes once actually done.
Update — 2026-08-22: the vars-wipe bug goes from "caught in review" to "actually breaks a build"
The fly set-pipeline secret-wipe gotcha documented in the 2026-08-20 update above stopped being a theoretical footnote and became the actual root cause of a stuck build. One of the overnight fix-attempt subagents re-ran set-pipeline without the vars file, wiping workflow_repo_private_key, and Concourse put workflow-repo/workflow-repo-branches/gpu-lock all into silent "errored" state — no crash, no visible failure. A queued build (#11) just sat "pending" with an empty log, indistinguishable from "still running" unless you went and looked. Diagnosed the same way every war story on this page gets diagnosed: fly resources -p blades68 showed the errored state directly, fly check-resource -r workflow-repo gave the literal undefined vars: workflow_repo_private_key message, and the fix was the one-line re-apply with both vars files loaded.
The more interesting failure sat one layer up. The investigating session had been running as a detached background task and crashed mid-run; asked "what's taking so long?", the very next turn answered "I don't see any record of a request... in my memory or active subagent logs" — flatly wrong, and wrong in the exact same shape as the trust-gap incident from the 2026-08-21 update above (treating absence-from-memory as evidence of non-existence instead of going and checking fly resources directly). It self-recovered on the following turn once it actually looked.
Friction, pain points, surprises (update, 2026-08-22)
Same lesson, a third data point: a background investigation crashing doesn't erase the ground truth it was investigating — fly resources, fly check-resource, and the git history are all still sitting there regardless of whether the session reading them survives. The recurring failure mode across all three incidents on this page now is reaching for "what do I remember about this" before reaching for "what does the live system actually say" — worth internalizing as a reflex before it needs a fourth data point to notice.