Dagger
Evaluated 2026-10-02/03 as the in-task execution engine UNDER Concourse for the genops GPU pipelines. Concourse keeps scheduling, the gpu-lock, and put: egress; Dagger was only ever a candidate for what happens inside one task: step.
Verdict: don't adopt. The GPU passthrough problem is solved and the fix is real. The reason for wrapping anything in Dagger — content-addressed caching of an expensive CUDA compile — was then measured on this host and lost by a factor of four, with a corrupt content store at the end of it.
Goals
One genops pipeline stage genuinely computes inside the Concourse task instead of curling a long-lived host service: the gsplat reconstruct. Its build-env stage clones gsplat v1.5.3 and AOT-compiles the CUDA extensions at sm_86 — roughly 14 minutes of cc1plus on every single run, for inputs (gsplat_ref, torch_image) that change approximately never. Concourse has no way to skip that; a task step is a task step. Dagger's pitch is that its content-addressed DAG would skip it automatically.
So the question was narrow: does dagger call inside a Concourse task cache that compile across runs, without giving up the scheduler, the single-3090 mutex, or versioned egress?
The comfyui-driven pipelines were deliberately excluded. They curl a server on the host and compute nothing in-task, so wrapping them in Dagger would cache an HTTP request, not a render.
Effectiveness
Two answers, and the second one decides it.
GPU passthrough: works. Build dagger-gsplat-reconstruct/4 succeeded in 1h20m17s, and the acceptance line is nvidia-smi -L stdout from a container Dagger itself spawned, not the Concourse task container:
11 : Container.experimentalWithAllGPUs DONE [0.0s]
12 : [16m42s] | GPU 0: NVIDIA GeForce RTX 3090 (UUID: GPU-425e29c4-f9cd-38f9-32a4-31f5e6ab4b2c)
9 : Gsplat.gpuCheck DONE [16m49s]
gpu-check exit=0
dagger call reconstruct --steps 1000 then drove build_env → dataset → GPU training end to end, exit 0.
Caching: loses, badly. Two runs on lab, same pipeline, same job name, same step name (Concourse scopes task caches to pipeline/job/step/path, so renaming either silently gives you two cold runs and a fake answer):
| COLD (build 6) | WARM (build 7) | |
|---|---|---|
| Result | succeeded | failed |
| Wall clock | 17m38s | 1h9m46s |
/var/lib/dagger at start |
4 KB | 30 GB |
/var/lib/dagger at end |
30 GB | 96 GB |
| Changed input | — | steps 1000 → 1200 |
Stage breakdown, quoted from the build logs:
COLD STAGE build-env exit=0 seconds=844 cached_markers=0
STAGE dataset exit=0 seconds=71 cached_markers=2
STAGE reconstruct exit=0 seconds=34 cached_markers=15
WARM STAGE build-env exit=0 seconds=2718 cached_markers=0
STAGE dataset exit=0 seconds=212 cached_markers=0
STAGE reconstruct exit=1 seconds=1122 cached_markers=5
Nothing was skipped. Not one stage. build-env — the 14-minute CUDA compile, the single layer this whole exercise exists to reuse, with both its inputs unchanged — re-ran and took 3.2x longer than the cold compile. dataset, also unchanged, re-ran 3x slower with fewer cache hits than the cold run had. The warm run paid for the cache twice: once to restore a 30 GB volume into the task, and again to redo all the work that volume was supposed to contain.
Then it broke. The warm reconstruct did not fail on anything to do with gsplat; it failed pulling its base image out of the cache Dagger had written 40 minutes earlier:
import image "docker.io/pytorch/pytorch:2.9.1-cuda12.8-cudnn9-devel@sha256:39236c0a...":
failed to get reader from content store:
content digest sha256:d78851750f0bfb7eaa5a50636afcde177ad1090c845a4eb2d1c7fe7e0f5bf4d8: not found
An 18m31s timeout against a content store missing a digest it is itself indexing. The identical image pulled cleanly in the cold run. Carrying /var/lib/dagger across runs as a Concourse task cache does not produce a warm cache; it produces a corrupt one.
N=1 per arm — a smoke test, not a matched comparison, and one pair of runs can't fully separate "Dagger's cache is wrong here" from "this host was IO-starved that hour" (load 37.66, 28% iowait, 45 processes blocked, swap in use, on a disk already 71% full). But the decision only needed one question answered — is there a win worth the complexity? — and the answer is no by a wide margin with a hard failure on top. A tighter comparison is only worth running if something first explains why a 30 GB cache restore produced zero cache hits.
What made it effective
The one thing that genuinely worked, and the thing to steal if we ever revisit this: experimentalWithAllGPUs is a real, working primitive once the engine image has the toolkit userspace. Nested GPU containers from inside an already-containerized CI task is not a thing most runners can do at all.
dataset() returning dagger.Directory is the right shape for passing artifacts between stages — typed, content-addressed, no bind-mount archaeology. The problem is that nothing forces you to use it on the output side (see below).
Pinning the grafted toolkit to 1.20.1 to match the host's nvidia-ctk is load-bearing. The hook in the container and the driver on the host are two halves of one mechanism; let them drift and the failure is silent.
Bonus utility
The exercise was worth more as a harness audit than as a Dagger evaluation.
Builds 3 and 4 reported succeeded with reconstruct exit=0 in 10 and 9 seconds — for what is nominally 1000 GPU optimisation steps. Neither ran gsplat at all. Two defects stacked: cv2 couldn't import (libGL.so.1: cannot open shared object file; the pytorch base image ships no libGL and gsplat's colmap.py imports cv2 unconditionally), and the trainer was piped into tail — python simple_trainer.py ... 2>&1 | tail -40. A pipeline's exit status is the last command's, so that reports tail's 0 no matter how the trainer died. set -e cannot see it. An unconditional echo 'RECONSTRUCT OK' then printed a success line three lines below ls: cannot access '/out'.
Had those been believed, the cold baseline would have been 10 seconds and the entire cache comparison would have been meaningless in the flattering direction. The cold number is only trustworthy because build 6 is backed by artifacts rather than an exit code: ckpt_999_rank0.pt at 46,725,481 bytes, train_step0999_rank0.json, and using /data/tandt/truck/images (251 images) in the log.
The fix generalizes past this pipeline: redirect to a file and capture the status in the same command list (&& TRAIN_RC=0 || TRAIN_RC=$? — a trailing || true resets $? to 0 and is the same lie), then assert the artifact exists rather than trusting rc=0.
Separately, the ensure: block got an unplanned stress test. Build 2 was killed at 31m38s by someone running sudo systemctl restart docker on the host, destroying the worker mid-task. After recovery, gpu/unclaimed/gpu-lock was present and gpu/claimed/ held only .gitkeep. The sequential, never-in_parallel lock release survived having its worker shot out from under it.
Friction, pain points, surprises
The stock engine image cannot do GPU, and the failure is a bare exit code: 1. Dagger's executor hardcodes an OCI prestart hook at the literal path /usr/bin/nvidia-container-runtime-hook on every nested exec with GPUs enabled. registry.dagger.io/engine:v0.21.10 doesn't ship it, so container creation fails — even for an exec whose only command is exit 0. The real error (error running prestart hook #0: fork/exec ... no such file or directory) is only visible deeper in the engine log. Two obvious escapes are dead ends: upstream publishes no GPU image variant past v0.9.10 (v0.21.10-gpu is a 404), and the engine image is Wolfi-based with no package manager — apk, apt-get, dnf, yum all absent, so nothing can be added at runtime. The fix is a derived image that unpacks the toolkit debs with dpkg-deb -x and grafts the binaries plus libnvidia-container*, filling only genuinely-missing non-core shared deps (libcap, libseccomp). glibc core must be excluded deliberately — overwriting Wolfi's libc with Ubuntu's breaks dagger itself.
The Python SDK snake-cases experimentalWithAllGPUs inconsistently. v0.21.10's codegen emits experimental_with_all_gp_us — the acronym splits. We resolve it at runtime across candidate spellings and raise loudly if none match, because a GPU stage that quietly falls back to CPU is worse than a loud failure.
Dagger's TUI renders a braille spinner frame per frame into a non-TTY Concourse log. Thousands of lines, burying the one actual error. NO_COLOR=1 and DAGGER_PROGRESS=plain are load-bearing, not cosmetic.
Wrapping a stage in Dagger silently severs egress. This is the structural finding and it matters more than the cache numbers. Build 6 produced a genuine 46 MB checkpoint inside the Dagger container and zero bytes reached the store — across all three MinIO buckets (110 objects) nothing has an mtime on or after the run date; the newest object predates the runs by two days. Two compounding causes: reconstruct() is typed -> str and ends in .stdout(), so the container filesystem holding /out is discarded and only the log text escapes; and the dagger-call Concourse task declares no outputs: at all, so even with a put: appended there'd be no directory to read. Egress stops being "declare an output, add a put:" and becomes an explicit export step the module author has to remember. Build 6 reported exit 0, with a real artifact, having published nothing. Concourse's own put: is fine — both builds claimed and unclaimed gpu-lock through the pool resource, build 7 releasing on its failure path — but the Dagger pipelines contain exactly two put: steps and both of them are the lock.
Concourse interpolates ((var)) in a task config before the container starts, and POSIX arithmetic looks exactly like a var. $((T1-T0)) in an inline task script kills the build with failed to interpolate task config: undefined vars: T1-T0. Move the script to a repo file that gets fetched rather than interpolated.
The cache is enormous and grows. 0 → 30 GB on the cold run, 30 → 96 GB on the warm one: 66 GB of growth in a single build, on a disk already 71% full. Because it lives as a declared Concourse task cache volume (not a container root overlay), fly clear-task-cache is sufficient to release it — but you will be clearing it, regularly.
ARG above the first FROM is invisible inside build stages. Classic Dockerfile trap, hit here on the derived engine image: the versioned COPY paths interpolated to empty. Redeclare per stage. Also, the Wolfi engine image has no bash, so exec > >(tee ...) dies — plain sh only in task scripts.
Should we adopt it under Concourse?
No. The thing Dagger was being evaluated for — skipping the 14-minute CUDA compile — did not happen once in the warm run, and the attempt made every stage slower than it had been with no cache at all before corrupting its own content store. Against that, the costs are concrete: a privileged engine, a hand-maintained derived image pinned to the host driver's toolkit version, an SDK whose GPU method name has to be probed at runtime, a 66 GB/run cache on a disk that doesn't have it, and an egress path that fails silently with a green build.
Scope the GPU support honestly too: experimentalWithAllGPUs is marked experimental, API subject to change or removal. And the parts of Concourse that make the lab work have no Dagger equivalent we'd want — there's no cross-run exclusive-lock primitive (dagger#4675 is still unshipped, which is why gpu-lock stays a Concourse pool resource), no scheduler or trigger model, and no versioned resource/put:. Dagger was only ever a candidate for the inside of a task, and the inside of a task is the one place it didn't pay.
What to keep from the exercise: the derived engine image (it works, it's documented, and it de-risks a future look at this), the exit-status idiom, and the artifact-not-exit-code assertion. Not the dependency.