ICONIC research / specialist model training

39 out of 48 was the easy part.

I fine-tuned a 4-billion-parameter Qwen model on a 16GB graphics card. It went from four correct answers to 39 on familiar test families. Then I changed the way the questions were written, and the number fell apart. Here's the full experiment.

RTX 4060 Ti · 16GBQwen3 4B · QLoRAReal model-weight updatesNot production-ready

Patrick Christell · ICONIC · November 16, 2026 · Original LinkedIn post ↗

The result, without the spin
The trained model passed 39 of 48 fresh, template-family questions, up from four for the untuned base. On 12 separately written questions it passed only four. It also lost a previously passing behavior in a protected 22-case regression suite.

The training worked for a narrow, recognizable pattern. The more important question—whether that skill survives unfamiliar wording and evidence—remains open. This adapter was not promoted to production.

01 · Why train it at all?

The mistake I wanted it to stop making

Consider an inventory report. A dealership website lists 108 vehicles. The inventory export lists 96. The difference is twelve. That is the entire finding until someone checks what produced it.

Those twelve might be listing-feed errors, units omitted from one system, duplicate records, stale data, or vehicles in different statuses. They are not twelve confirmed sales. A model that computes the difference correctly and then confidently announces twelve sales has failed the business task.

Most of these jobs don't require grand philosophical reasoning. They require reliable arithmetic, discipline about sources, and the ability to say precisely what the available records do—and don't—prove. The question was whether a small local model could learn those habits well enough to handle routine cases without spending the time and compute required by a larger one.

At ICONIC I also run a separate 27-billion-parameter model on my own hardware. This experiment was not about retraining that larger model. It was about testing whether a much smaller specialist could take a narrow class of work off its hands.

02 · The machine

One desktop, two different jobs

My setup has an RTX 3090 with 24GB of video memory and an RTX 4060 Ti with 16GB. The 3090 runs the larger production inference workload. I isolated the experiment on the 4060 Ti. The two cards do different jobs; 40GB total VRAM does not mean a single 40GB pool.

ComponentWhat ranRole
RTX 3090 · 24GBQwen3.8-27B, local inferenceExisting production service; left running throughout
RTX 4060 Ti · 16GBunsloth/Qwen3-4B-Base-bnb-4bitIsolated fine-tuning and A/B evaluation
AdapterQLoRA / LoRA rank 8, alpha 16Trainable low-rank parameters on a frozen, 4-bit base
Decode160-token greedy outputSame generation conditions for both evaluation variants

This distinction matters: the trained 4B adapter is not an update to the 27B model. It cannot be loaded into the larger model's GGUF as if it were a general upgrade. Nothing was switched, restarted, or routed into production to run this test.

03 · How it was trained

1,800 fictional examples. Only 240 actually trained.

I generated 1,800 deterministic fictional cases with independently checked target answers. The generator produced six kinds of business evidence tasks: contact rates, contradictory sources, net supplier credits, inventory differences, advertised markdowns, and claims about personnel that the records do not support.

The split was 1,639 training examples and 161 validation examples. But this was a short calibration run, not a full training pass: 60 optimizer steps × four examples per effective step = about 240 training examples. That is roughly 0.146 of one epoch through the training pool. The split and the amount actually consumed are different numbers; conflating them would make the experiment sound far larger than it was.

Training parameterRecorded value
Starting modelQwen3-4B-Base, 4-bit Unsloth build
MethodAnswer-only prompt-completion QLoRA
LoRA configurationRank 8 · alpha 16
Updates60 optimizer steps
Effective batch4 examples per step
Learning rate0.00006 (6 × 10⁻⁵)
Sequence limit768 tokens
Approximate training tokens33,000, plus validation and testing
Optimizer runtime76.33 seconds; approximately 109 seconds for the full process
Reported lossesTraining 0.5242 · validation 0.02081

QLoRA updates small adapter parameters while keeping the quantized base weights fixed. In other words, this changed learned parameters, not merely a system prompt. The very low validation loss is much less impressive once you know the training and validation examples came from the same generator families. A model can be extremely good at a template and still fail the next person who phrases the question differently.

No customer records, live dealership data, paid teacher-model outputs, or private production logs went into the training examples. The checkpoints and raw predictions remained local and were not uploaded to GitHub.

04 · The results

The first test looked extraordinary. The second was the point.

4 → 39passes out of 48 new examples from familiar generator families
4 / 12trained model passes on separately authored, differently formatted questions
2 / 22passes on the protected transfer suite; one previously correct answer was lost

The A/B comparison was controlled: identical 4-bit base weights, tokenizer, input prompts and 160-token greedy decoding. One variant had the adapter disabled; the other used the trained adapter. Both were evaluated in the same isolated GPU process.

Evaluation suiteUntuned baseTrained adapterInterpretation
48 fictional examples, new seed, same six families4 / 48 (8.3%)39 / 48 (81.3%)Strong improvement within the generator's patterns
12 separately written challenges with different phrasing and source labels0 / 124 / 12 (33.3%)Limited transfer; exploratory test
22-case protected transfer/regression suite2 / 222 / 22No net increase and one prior passing case regressed
Same model, different questions
Pass rates. Brown = base, violet = trained adapter. The three suites are different populations; their scores should not be combined.
48 new examples from the training generator's task families
Base
8.3%
Adapter
81.3%
12 separately written out-of-template challenges
Base
0%
Adapter
33.3%
22 original protected transfer tests
Base
9.1%
Adapter
9.1%

Where the 39 passes came from

Problem familyBase / 8Trained / 8
Contact-rate calculations46
Contradictory evidence08
Net supplier credit07
Inventory differences05
Advertised markdowns05
Unsupported personnel claims08
Total4 / 4839 / 48

The 48-case test used a fresh random seed but the same task generator families. The 12 hand-authored cases were created after the first test and are exploratory, not a pre-registered benchmark. Neither is a measure of general real-world accuracy.

05 · What failed

New wording is a different kind of difficulty

The separately written questions changed source labels, evidence formatting, and the way requests were phrased. They covered the same six underlying skills, but broke some of the surface patterns the training examples had made familiar. The trained model passed just four of twelve.

The more serious result came from the original 22-case regression suite. Both versions passed only two cases, but they were not the same two. The new adapter lost an earlier passing untrusted-source test. Instead of returning a supported, source-linked JSON answer, it generated nothing.

That was a regression in reliability, not an observed prompt-injection leak. The model did not reveal the malicious content. But an empty answer still fails a workflow that depends on a valid response. A specialist that becomes excellent at the new task and silently loses an old guarantee is not ready to take over production work.

That is why the headline improvement is useful but insufficient. A model has to handle unfamiliar examples, retain required behavior, and fail in ways the surrounding system can detect. We do not have those guarantees yet.

Deployment decision: Keep the existing 27B service untouched. The 4B adapter stays experimental. No production routing, model promotion, or live client workflow was changed.
06 · Economics and privacy

Zero cloud rental is not zero cost

The optimizer completed in 76.33 seconds and the overall process took about 109 seconds. The training updates processed roughly 33,000 tokens, in addition to validation and testing. It ran on a graphics card I already owned.

Incremental cloud GPU rental: $0. API fees: $0. That does not make the experiment free. The workstation, electricity, setup, data generation, evaluator design, and time spent investigating failures all have costs.

Local training also made it possible to keep the adapter, predictions and examples on my own machine. In this run the training examples were fictional anyway. Keeping the workflow local matters much more when an organization is working with genuinely sensitive records, where data handling and permission need to be designed before any training begins.

07 · What comes next

Route tasks by demonstrated competence, not model size

The architecture I am working toward is straightforward: let a small specialist do the well-defined work it has proved it can handle, and route exceptions to a more capable model. But that is the intended operating model, not a claim that this 4B candidate is already routing live work.

  1. Start with the actual business failure. Define which calculations, source citations and boundaries between evidence and inference must be correct.
  2. Train on verified, varied cases. Expand beyond a single generator's sentence structures and source-marker conventions. Keep the existing holdouts sealed.
  3. Test on different-looking problems. Freeze independently authored cases before the next training attempt. Include missing records, contradictory documents, source poisoning and changed output formats.
  4. Require regression safety. Block promotion if a previously correct protected behavior breaks, even if the headline score improves.
  5. Compare routing economics. Evaluate the same frozen cases with the candidate, the larger local model and a frontier model, including latency, power, failure cost and operational availability. That comparison has not yet been completed.
business problem
    ↓
verified examples
    ↓
small-model specialist training
    ↓
independent evaluation + regression checks
    ↓
route only the jobs it has actually earned
    ↓
send exceptions to the stronger model

A 4B model that gets the answer right reliably can be the right tool for a repetitive job. A 4B model that answers 81% of familiar synthetic tests but 33% of changed ones is still an experiment. The work is in finding, testing and maintaining that boundary.

08 · Questions

What these numbers do and don't say

Did you fine-tune the 27-billion-parameter model?

No. The production 27B model continued running on the RTX 3090. The fine-tuned candidate was a separate Qwen3 4B model trained and evaluated on the RTX 4060 Ti.

Does 39 out of 48 mean 81% real-world accuracy?

No. Those 48 questions came from the same six synthetic generator families used to build the training pool. The independently written 12-case suite yielded 4/12. Neither score is a representative estimate of performance across real business tasks.

Was this prompt engineering or actual training?

It was QLoRA fine-tuning. Trainable adapter weights changed across 60 optimization steps. The quantized base model weights stayed fixed. The comparison enabled and disabled the adapter against the same base.

Was it cheap to run because it was local?

Incremental cloud GPU and API charges were zero because the training ran on already-owned hardware. Acquisition, electricity, engineering time, and the cost of evaluating failures were not zero.

Can the trained model handle production client data now?

It was not deployed. The independent generalization check was weak and one previously passing protected test regressed. It needs substantially broader training and evaluation before unrestricted work could be justified.

09 · Reproducibility and sources

The post, the numbers, and the code

This article expands my November 16 LinkedIn post. The experimental parameters, evaluation counts, failure notes and promotion decision come from my November 15 experiment report. The report, the training and evaluation scripts, the adapters, the synthetic example files and the raw predictions are kept private.

Related ICONIC research: The Context Tax: how production agents consume tokens and Detecting Synchronized Repricing in Dealer Inventory.

Citation: Patrick Christell (2026), “39 Out of 48 Was the Easy Part: What Fine-Tuning a 4B Model on a 16GB GPU Actually Proved,” ICONIC Research, https://iconic.onl/research/local-model-specialist-training/. Experiment completed November 15; article published November 16.

Want to know what private AI can actually do on your hardware?Start with a real workload and a testable definition of success. Book a free assessment.