Two local LLMs drive Coresmith end-to-end

Coresmith is the agentic RTL-to-GDS pipeline behind coresmith.ai. It normally calls Claude for every uarch / RTL / testbench / debug step. As a stress test we swapped Claude out for two local LLMs served by vLLM on a single RTX PRO 6000 — Gemma 4 31B and Qwen 3.6 27B — and ran them against two real blocks: a 3-stage pipelined micro (mcu3) and the variable-length coding stage of an H.264 image codec. The two models failed in different and instructive ways. This is a write-up of what we measured.

Setup #

Everything below is reproducible from the public Coresmith engine code on a single RunPod RTX PRO 6000 Blackwell (96 GB). vLLM 0.20.2 serves either Gemma 4 31B (BF16, with --reasoning-parser gemma4 and --tool-call-parser gemma4) or Qwen 3.6 27B (BF16, --reasoning-parser qwen3 and --tool-call-parser qwen3_xml). Tools (Read / Write / Edit / Grep / Bash) come from the open-source opencode CLI. We monkey-patched Coresmith's ClaudeLLM wrapper to subprocess opencode run --model local/<model> instead of claude — same prompt templates, same LangGraph pipeline, same Verilator + cocotb + Yosys + Sky130 PDK.

Round 1 — a 3-stage pipelined MCU (mcu3) #

Single-block tier-1 design. Custom 16-bit ISA. The model has to author the uarch spec, write the Verilog, generate a cocotb testbench, run it under Verilator, and ship a netlist through Yosys.

Both models produced RTL that passed lint. Gemma's testbench failed on first run (a registered-output read timing mismatch) and the pipeline's DebugAgent correctly classified it as a TB bug, fed the diagnosis back, and Gemma re-generated a passing TB on attempt 2. Qwen's pipeline never reached a passing TB before the watchdog killed it.

Full mcu3 telemetry, including the WaveDrom render of Gemma's actual dump.vcd: orbitality.com/dashboard/socmate-llm-bench/ (password protected — the dashboard side of the bench).

Round 2 — A real H.264 codec block #

mcu3 is a textbook block. We wanted to know how the same models behave on a task with sharper correctness constraints — something where a one-bit RTL error wouldn't pass lint, wouldn't fail sim against a fuzzy testbench, but would visibly corrupt an output image.

The task: implement expgolomb_enc, the variable-length coding stage of an H.264-style intra image codec. Bit-compatible with the VLC.expgolomb_enc function from PyH264. AXI-Stream both sides, signed 16-bit coefficient in, 32-bit MSB-aligned codeword + 6-bit length out, an err_o pulse for codewords that would exceed 32 bits.

Encoding rule:

value 0           → '1'
positive value N  → codenum = 2·N − 1
negative value N  → codenum = 2·|N|
emit (count_bits − 1) zeros, then bin(codenum + 1)
                  where count_bits = bit-length of (codenum + 1)

The roundtrip harness: take a 128×128 luma PNG, run DCT + quantize in Python on each 4×4 block, stream every coefficient through the Verilator-simulated expgolomb_enc.v, concatenate per-cycle m_tdata[31 : 32 − m_tlen] outputs into a bitstream, feed that bitstream back through PyH264's decoder + dequant + IDCT, save the reconstructed PNG. If the LLM-authored RTL is wrong by a single bit, the decoder either crashes on an illegal codeword or reconstructs garbage.

Gemma 4 — fast, wrong, then handheld to pass #

Gemma authored the spec, RTL, and TB scaffolding in ~3 minutes. The RTL lint-passed first try. Two testbench self-repair loops failed (both due to missing scaffolding — the auto-generated TB literally never started a clock), so we hand-patched the TB. With the clock running, two of four tests passed; the other two surfaced inconsistencies between Gemma's golden-model code and Gemma's own RTL.

We then drove the actual PNG-derived coefficients through the Verilator-simulated DUT and found two arithmetic-width bugs in Gemma's RTL:

Bug #1 — shifted by L instead of 2·L − 1

The codeword length is len = 2·L − 1 where L = bit_length(X). Gemma computed len_comb correctly but then MSB-aligned the value by shifting by 32 − L, dropping the leading-zero prefix. 2,075 mismatches on a 16,384-coefficient run.

- assign data_comb = (err_comb) ? 32'h0 : (32'(X) << (32 - L_comb));
+ assign data_comb = (err_comb) ? 32'h0 : (32'(X) << (32 - len_comb));

Bug #2 — s_tdata declared unsigned, compared as signed

After the shift fix, 502 mismatches remained — all for val = −1. The codenum-conversion logic compared s_tdata > 16'sd0 but the port was declared [15:0] (unsigned), so every negative coefficient (0xFFFF … 0x8000) was treated as a huge positive, the leading-one detector landed on bit 16, and err_o fired for every negative input. Explicit $signed cast fixed it.

- assign codenum = (s_tdata == 16'sd0) ? 17'd0 :
-                  (s_tdata > 16'sd0)  ? (17'd0 | {s_tdata, 1'b0}) - 17'd1 :
-                                       (17'd0 | {(-s_tdata), 1'b0});
+ wire signed [15:0] sdata = $signed(s_tdata);
+ assign codenum = (sdata == 16'sd0) ? 17'd0 :
+                  (sdata > 16'sd0)  ? (17'd0 | {sdata, 1'b0}) - 17'd1 :
+                                       (17'd0 | {(-sdata), 1'b0});

After both fixes: 16,384 / 16,384 coefficients matched the golden codeword bit-for-bit. The 25,350-bit hardware bitstream decodes cleanly to a 22.6 dB PSNR reconstruction at 1.55 bpp:

The 22.6 dB is the lossy QP-26 quantize step, not an RTL artefact. Both bugs share a common root: Gemma reasoned about the math correctly in prose — its uarch_spec.md §3.2 describes the encoding exactly right — but lost precision when translating to bit-vector widths. The auto-generated testbench's own golden model had the same shift bug, so it could never have caught bug #1 either. Only the end-to-end PNG roundtrip surfaced it.

Qwen 3.6 — wrote a 466-line spec, but the watchdog killed it first #

Qwen behaved like a more deliberate engineer. Where Gemma streamed a 5 KB spec in 3 minutes and shipped wrong RTL, Qwen spent its 20-minute budget on archaeology and never delivered any text at all.

What Qwen actually did before being killed:

• glob **/* the entire socmate/ tree to find related files
• glob **/*expgolomb* — located examples/expgolomb_enc/expgolomb_enc_model.py
• 8× read calls on the golden model, the YAML block-spec, the example mcu3.md, and several engine source files
• Ran the PyH264 golden model in a bash tool call and verified its output: OK — golden matches PyH264 reference table
• Listed the existing uarch_specs directory to match prior formatting conventions
• Emitted a 24,202-character reasoning block (about 4 minutes of generation) writing the spec inside its hidden think channel
• Followed up with a write tool call whose argument dict appeared to come through as {} — empty path, empty content
• Coresmith's subprocess timeout fired

So why did Qwen fail? It's not the prompt length.

We were initially convinced the system prompt was too long. The numbers say otherwise.

Qwen missed by about 2.7 minutes, not by an order of magnitude. The prompt isn't too long. Qwen's reasoning is too long, at a generation speed that's too slow, for a timeout that's too tight.

Proof point: the on-disk spec file arch/uarch_specs/expgolomb_enc.md appeared on disk with mtime 10:43:47 — 9 minutes after the parent Coresmith process killed its subprocess at 10:34:19. The opencode subprocess wasn't in the killed process tree, so it kept draining vLLM and eventually wrote the file. The "empty write tool call" we saw in opencode's storage DB is the snapshot of a tool call mid-flight: Coresmith took its stdout slice at the 20-minute mark before opencode had finished parsing the call's argument block.

And the file Qwen eventually wrote is gorgeous. 466 lines, every required section, an ASCII block diagram, a per-stage datapath narrative with explicit bit widths and codenum + 1 overflow analysis (it correctly notes that N = −32768 is the only 16-bit signed input that triggers err_o), a registered output timing contract, a WaveDrom timing diagram, edge-case enumeration, a Verilog interface stub, and a JSON summary. Gemma's spec was 155 lines and missed half of that detail.

Gemma vs Qwen — the RTL diff #

For the mcu3 block both models produced RTL, so we can actually compare. (For the codec block, Qwen never got past UARCH due to the timeout, so there's no Qwen RTL to diff against Gemma's.)

Lessons for local-model deployment #

Two failure modes worth keeping in mind:

• Gemma 4 is fast and confidently wrong in ways an LLM judge can't catch without running the code. Lint passes, the auto-generated testbench's golden model agrees with the broken RTL, and you only find the shift error when a real PNG comes out scrambled.
• Qwen 3.6 is careful and right, but the toolchain leaks its output. 27K tokens of hidden reasoning at 25 tokens/sec doesn't fit inside a 20-minute subprocess timeout — and when the subprocess gets killed mid-tool-call, the audit log lies about what happened. The file does eventually land on disk, 9 minutes after the parent process gave up.

Three concrete fixes, in increasing cost:

1. Bump the timeout from 1 200 s → 2 400 s. One-line change in orchestrator/langchain/agents/socmate_llm.py. Gives Qwen its 23-minute run time with headroom. Doesn't fix the underlying generation rate.
2. Disable the thinking channel via extra_body={"chat_template_kwargs": {"enable_thinking": false}}. Cuts output by ~80%. But Qwen's "reasoning out loud" is exactly what produces its 466-line spec depth — turning it off moves Qwen down toward Gemma's quality tier.
3. Cap output per phase with a sensible max_tokens budget — say 8 192 for UARCH gen, 16 384 for RTL. Forces the model to commit faster without losing reasoning entirely. Best long-term fix.

And on the design side, both bugs that broke Gemma's codec RTL were signedness / width mismatches. Those are exactly the cases that a structural lint pass (Verilator -Wall already warns on, but doesn't error on, WIDTHEXPAND) could catch before sim. Plausibly the next Coresmith DebugAgent rule.

Update — Qwen wins the rerun #

This post originally ended at "Qwen DNF, blame the watchdog." After publishing we tried what the conclusion suggested — a configurable timeout, a single prompt directive, and a max_tokens cap — and Qwen produced a working RTL with zero hand-fixes. Same PNG roundtrip, same 22.6 dB PSNR, byte-identical bitstream to Gemma's hand-patched version. The framing was wrong; what follows are the receipts.

What we changed (≈15 LOC total)

1. New helper orchestrator/_timeouts.py with a scaled(base, env=None) wrapper, read by every timeout site in the pipeline. Single env var SOCMATE_TIMEOUT_MULTIPLIER scales them all at once. Set to 3.0 for the Qwen rerun, so the 20-min opencode subprocess cap became 60 min and the per-phase budgets followed proportionally.
2. Six-line "reasoning budget" stanza at the top of prompts/rtl_generator.md: "You MUST call the Write tool within this response. Do not keep reasoning indefinitely."
3. maxOutputTokens: 16384 on the Qwen model in ~/.config/opencode/opencode.json as a hard safety net.

Rerun outcome

The two cocotb fails are TB-side, not RTL-side

test_known_vectors for N=−1 expects 0xC0000000 — which is what Qwen's uarch spec's worked-example table says — but the RTL emits 0x60000000, which is what Qwen's spec's formula yields, and is the correct PyH264 codeword. The spec had a single transcription error in one row of a 7-row verification table. The RTL implemented the formula correctly and ignored the typo. The TB inherited the typo as a golden value.

This is the symmetric inverse of Gemma's failure mode: Gemma's TB and RTL both had the same bug (same shift formula), so they agreed wrongly and lint+sim passed. Qwen's TB and RTL disagree, and the disagreement points to the spec's typo, not to a real design defect. The random-stress test (256 random inputs) and the exhaustive sweep (−1024..1023, 2048 values) both PASS — every value Qwen's RTL emits matches PyH264's VLC.expgolomb_enc bit-for-bit.

Net read

Qwen wasn't failing the design. It was failing the watchdog. Given the same wall-clock budget Gemma needed, Qwen ships cleaner RTL than Gemma — explicit 17-bit signed datapath catching the −32768 corner case, single-bit-tap error detection, correct shift formula. Zero hand-fixes vs Gemma's two arithmetic-width bugs. The 5× wall-clock cost buys you a design that doesn't silently embed bugs synthesizable-but-wrong gate netlists carry forward.

Artifacts #

The dashboard side of this bench, with every pipeline_events.jsonl, every UARCH spec, every generated TB, every Yosys netlist, and a WaveDrom render of the Gemma dump.vcd:

• orbitality.com/dashboard/socmate-llm-bench/ — three tabs, Round 1 Gemma / Round 1 Qwen DNF / Round 2 codec (password-gated, ping us for the cred)
• facebookexperimental/socmate — engine source
• github.com/balbekov/PyH264 — reference codec used as the golden model