SemIf / OpenJev (TheoLeeCJ)
Evaluated 2026-10-10 under task t_ad1cc8be (semif-spike/semif-systemone-smoke, build #4) as a candidate System-One decision provider on blades '68.
Verdict: viable to build on, not viable to deploy today. Solid 4B decision model with sub-100 ms latency, but upstream ships no HTTP server, forcing an ad-hoc adapter layer, and exports uncalibrated option scores.
Goals
Test whether SemIf (formerly OpenJev, based on Qwen3.5-4B) can serve the POST /v1/systemone decision contract under Concourse GPU-lock on the lab 3090, assessing VRAM consumption, decision speed, and integration friction.
Effectiveness
N=1 smoke test against the login form state.
| Metric | Measured |
|---|---|
Serves /v1/systemone |
Yes — through a custom ~200-line adapter |
| VRAM attributable | ~8.4 GiB (278 MiB baseline -> 8,700 MiB peak, 8,616 MiB resident) |
| Warm latency | 64.8 ms median steady-state (3 requests; 1,023 ms first request) |
| Model load | 201.6 s cold inside the task container |
| Choice | Picked signin at probability 0.9886 (email 0.0110, cancel 0.0004), 163 tokens |
Decision latency was strong (64.8 ms) for a 4-billion parameter autoregressive model, and memory (8.7 GiB peak) left 15.8 GiB headroom on the idle card.
What made it effective
- Clean Schema Mapping: SemIf's native JSONL row schema (
{id, state, question, options:[{id, description}]}) maps 1:1 to Aristodromos decisions with the goal folded into the question. - Shared Mode Potential: Upstream supports
--mode shared, prefilling page state once across multiple evaluation criteria—useful if an agent poses several parallel queries against one DOM snapshot.
Friction / pain points / surprises
- Upstream Ships No Server: SemIf is strictly a Python library and a CLI (
semif-score). There is no HTTP API or daemon in the repo. To test it against our API contract, we had to author a ~200-linehttp.serverwrapper in the pipeline script (semif_systemone_smoke.py). Adopting it permanently requires maintaining our own service wrapper. - Uncalibrated Confidence: Upstream explicitly notes in its response:
The returned score (0.9886) reflects relative logit margins across options, not calibrated true-positive probability."probability_status": "conditional option score; uncalibrated as decision confidence" - 204-Second Cold Load: Pulling and loading the 4B weights inside an ephemeral task took over 3 minutes. Practical deployment requires a persistent host service.
- Missing Git in PyTorch Image: Pytorch base images lack
git, causingpip install git+https://...to fail after 11 minutes of heavy dependency downloading. We resolved this by downloading the pinned tarball release instead.