WebJev-35B-A3B (lexmount)
Evaluated 2026-10-10 under tasks t_d48f6ebb and t_945b23bc (feature/webjev-gpu-resident-smoke) as a browser-specialized decision model on blades '68.
Verdict: viable for dedicated "focus time" under exclusive lock; unviable as an ambient co-tenant. Specialized 35B architecture with high accuracy on browser DOMs. Its 20.2 GiB quant demands an idle 24 GB card, precluding coexistence with background desktop tasks or parallel GenOps pipelines. Adopting it means treating agent runs as scheduled exclusive blocks under Concourse gpu-lock rather than an ambient background service.
Goals
Determine whether WebJev-35B-A3B (Q4_K_M GGUF, 20,186 MiB weights) can be served under llama.cpp (b11541) via a host-side serving window shim under Concourse GPU-lock, measuring full offload latency and VRAM limits on the RTX 3090.
Effectiveness
The pipeline and serving rig were built, containerized, and verified end-to-end (webjev-pipeline.yml, builds #2–#4).
| Metric | Measured |
|---|---|
Serves /v1/systemone |
Yes — through Lexmount adapter on top of llama-server |
| VRAM footprint (device peak) | 20,872 MiB (~84.9% of 24,576 MiB card, leaving 3,704 MiB headroom) |
| Latency on GPU (distinct pages) | 222.2 ms median (p95 233.6 ms) across 10 distinct pages |
| Latency on GPU (replayed prompt) | 96.2 ms median (p95 102.2 ms) via prompt cache |
| Choice accuracy | Picked email on 10 of 10 distinct pages with confidence 0.949–0.969 |
The test rig verified end-to-end under Concourse gpu-lock: it opened the serving window with full GPU offload (-ngl 99, context 8192), executed 20 timed decision requests across distinct DOMs and cached replays, captured real VRAM usage and headroom, and cleanly released the lock.
What made it effective
- High Decision Accuracy: Correctly identified the target input field on 10 distinct variations of the page structure with high, calibrated confidence (0.95+).
- Fast Full-GPU Inference: Offloaded completely to CUDA, distinct page decisions dropped from 2.66 s on CPU down to 222.2 ms median.
- llama.cpp Prompt Caching: Replaying requests with shared prefixes yielded sub-100 ms latency (96.2 ms median), providing fast step-to-step throughput on multi-turn trajectories on the same page.
Friction / pain points / surprises
- Heavyweight VRAM Demands: At 20,872 MiB peak device VRAM (20.2 GiB weights + KV context), WebJev leaves 3,704 MiB spare (~15% margin) on a 24 GB RTX 3090. It is strictly an exclusive-tenant model; running concurrent GPU tasks or heavy desktop applications alongside it risks OOM.
- The Prompt Cache Benchmark Trap: An initial measurement script replayed a single request body 10 times and reported ~300 ms "decisions" on CPU. That measured cache lookups and 1 sampled token rather than real inference. When tested on distinct pages, true decision latency on CPU rose tenfold to 2.66 seconds.
- Dockerfile Glitch:
libgomp1was omitted from the Debian runtime packages, crashing the llama.cpp server on its first container startup until patched. - Alpine Lacks NVIDIA Devices: Running the window measurement task on
python:3.12-alpinefailed because Alpine lacksnvidia-smiand does not trigger the NVIDIA container toolkit device hook; had to switch tonvidia/cuda.
The "Focus Time" Concession
Ruling WebJev out purely because it cannot coexist with ambient workloads assumes an agent decision model must run as an always-on background daemon.
If browser-specialized accuracy and reasoning from a 35B model are necessary for complex, hostile web flows that 0.8B models fumble, dedicating the GPU for batch "focus time" is a viable operational compromise:
- Concourse coordination: Holding
gpu-lockguarantees sequential safety. GenOps pipelines and image experiments simply queue while an agent session runs, preventing GPU out-of-memory crashes without manual juggling. - Dedicated Tenancy: Running WebJev in focus windows requires an unencumbered GPU where the full ~22 GiB can be allocated without competing host or desktop graphics processes.