Concourse CI

Open-source, container-native CI/CD engine, deployed to a local Docker host tonight as a git-backed orchestrator for GPU-bound generation jobs — replacing ad hoc, uncoordinated agent dispatch with a single declarative queue.

Dogfooded 2026-08-18/19 during a long blades68-lora session: after three independent style/eval jobs collided over one shared RTX 3090 in a single night (see the GPU-queue write-up in memory), Gavin asked to try putting the actual orchestration — job definitions, sequencing, exclusivity — into git-tracked pipeline config instead of hand-rolled GPU_QUEUE.md conventions and dispatched-agent discipline.

Goals

Get a real, working, git-versioned pipeline running locally that (a) can trigger the existing ai-toolkit (Krea-2 LoRA/generation) and comfyui-local render jobs, and (b) enforces mutual exclusion on the single GPU so two jobs can never collide again, without hand-maintaining a queue file that multiple independent sessions all have to remember to update correctly.

Effectiveness

In progress, but with two real, proven wins along the way. Stood up cleanly from the official release-build docker-compose (after first grabbing the wrong, source-build compose file and having to swap it) — Postgres-backed, one worker registered, running at 127.0.0.1:8080 in about the time it takes to docker compose up. The max_in_flight: 1 step modifier proved out as a genuinely cleaner exclusivity primitive than a hand-written lock file: built a real two-job pipeline and captured an actual fly log showing one job visibly wait for the other, not just a claim that it works. The harder problem — getting the host's single GPU into Concourse's task-level containers — split into two answers depending on which tool needed it: comfyui-local already runs as a persistent host service, so a task just needs to hit its existing HTTP API on localhost:8188, no passthrough needed at all; ai-toolkit is a bare CLI script with no server, so something still has to execute a process on the host on the pipeline's behalf. First pass at that used SSH with a forced-command, allowlist-only authorized_keys entry (tested against path traversal and shell injection first, then run for real against a live low-stakes job) — safety-reviewed clean, and it worked. But Gavin found the SSH-transport layer itself hard to reason about ("byzantine and not very discoverable") and asked for an ADR — which surfaced, once written down plainly, that stock Concourse's own oci-hooks GPU-passthrough flags could let ai-toolkit just run as a normal task step with real GPU access, no bespoke transport layer at all. That's confirmed working (nvidia-smi sees the 3090 from inside a plain task container) but landed mid-session as a genuine architecture pivot, not a finished migration — and, separately, the LoRA itself got dropped from the picture entirely partway through (a real, unconfounded ablation test proved it was doing visible work, but Gavin decided a retrain wasn't worth doing right now and asked to just run base Krea-2 via the API instead).

What made it effective

The fly-log-as-proof discipline held throughout — every claim ("the lock works," "SSH-only allows one job," "GPU passthrough works") got a real trigger and a real log pasted back, not just a description of expected behavior. That habit caught the SSH allowlist actually rejecting the attacks it was supposed to reject before ever declaring it safe. Concourse's own primitives (across/max_in_flight, the /prompt→WebSocket→/history→/free job lifecycle it expects) mapped almost exactly onto what the ad hoc GPU-queue dispatch had been hand-building all night anyway — recognizing that meant the pivot away from custom locking wasn't a detour, it was converging on the same shape with less bespoke code.

Friction, pain points, surprises

The SSH-transport spike burned real GPU-adjacent effort (design, safety review, a working end-to-end test) before Gavin's "why is this so byzantine" question and the ADR it produced revealed a simpler native option had been available the whole time — worth a five-minute "does Concourse already do this?" pass before building custom plumbing next time. Nested-container GPU access is genuinely two different problems depending on whether the target already has an API (no passthrough needed) or is a bare CLI (passthrough or a transport layer required), and it's easy to conflate them mid-investigation. Two later attempts to actually carry out the oci-hooks containerization as a dispatched coding-agent task both failed with zero output for unrelated infra reasons (a CLI timeout, then a run that silently lost its execution context after an hour) — a reminder that "the technical answer is proven" and "the implementation task will actually complete" are separate risks, especially for long-running dispatched work late in a session.