Jeff (firelex)
Evaluated 2026-10-11 under task t_5fd75cdd (jeff-systemone-pipeline.yml, build #7) as a lightweight, fast System-One decision provider for browser navigation on blades '68.
Verdict: recommended for adoption. With a 2.0 GiB VRAM footprint and 46.9 ms warm latency, Jeff is the only candidate that coexists with active desktop GPU tenants while delivering calibrated decision confidence out of the box.
Goals
Test whether Jeff (mstrasser/jeff-base + mstrasser/jeff-adapter-nav rev v1.3, upstream 3720e7c6) can serve the canonical POST /v1/systemone decision contract under Concourse GPU-lock on the lab 3090, fitting the 24 GB card with substantial headroom, while producing sane choices and calibrated confidence on a browser login form.
Effectiveness
N=1 smoke test against a login state (email-field, sign-in-button, cancel-button, goal "sign in as alice@example.com").
| Metric | Measured |
|---|---|
Serves /v1/systemone |
Yes — HTTP 200, documented choice shape |
| VRAM attributable | 2,066 MiB (~8.4% of the 24 GB card, 21,776 MiB free headroom) |
| Warm latency | 46.9 ms median over 9 warm requests (46.5–48.3 ms; 41.2 ms on base model) |
| First request latency | 29.5 s (Triton JIT CUDA compilation) |
| Model cold load | 10.0 s (base + nav adapter) |
| Decision sanity | Picked email-field at 0.980 (confidence 0.970), sign-in-button 0.017, cancel-button 0.003 |
The decision distribution is well-calibrated and concentrated: cancel-button (which abandons the goal) is suppressed, and email-field is prioritized ahead of the sign-in button. The untouched base model also made the sane choice (0.931 email-field, confidence 0.896), with the adapter sharpening the margin.
What made it effective
- Minimal VRAM Footprint: At 2,066 MiB, Jeff leaves ~21.8 GB free on a 24 GB card, easily coexisting with active desktop applications and concurrent system workloads.
- In-Task Lifecycle: Because it loads in 10 seconds, Jeff runs as a direct subprocess of the Concourse task holding
gpu-lockand terminates with it. Unlike AutoJev-27B, it does not require a permanent host-side service or lease-watchdog shim just to answer questions. - Clean Tooling: Standard PyPI
torch==2.14.0bundled its own CUDA runtime; no custom vLLM forks, no patches, and no gated Hugging Face weights.
Friction / pain points / surprises
- Triton Runtime C Compiler Requirement: Triton compiles CUDA kernels on first launch. On a minimal container image, requests failed with
"Failed to find C compiler"while/healthstill returned 200ready(a 500 error behind a green healthcheck). Addingbuild-essentialto the runtime container resolved it. - 39-Second Cold First Request: The first forward pass pays the Triton compilation cost. Any production deployment needs a warmup request before serving live agent decisions.
- uv Floor: Jeff's
pyproject.tomlpinsrequired-version >=0.12.19; older uv binaries refused to sync. - No Upstream Dockerfile: Had to author a custom container build in the pipeline repo.