Research library

EXPERIMENTAL PROGRESS / TRIAL 001

When Fine-Tuning Made a 5B Model Worse

A bounded study of instruction tuning, regression and claim qualification.

Revised evidence review · Publication draft · Not peer reviewed

Unresolved audit identities

The source requires reconciliation of bundle, case and manifest identities. The reported numerical result is presented with that limitation; definitive audit provenance is not established.

Abstract

A one-epoch, assistant-only QLoRA intervention on a locally inspected K2 Horizon checkpoint reduced exact-pass performance from 238/300 (79.33%) to 106/300 (35.33%) on a frozen, generated instruction-following confirmation set. The reported paired change was −44.00 percentage points, with a reported 95% paired bootstrap interval of −49.67 to −38.33 pp.

Training loss decreased while the confirmation result regressed. The source supports a negative result for this intervention and task contract, not universal model deterioration or a diagnosis of its cause. The reported Osmium Audit v0.1 result is narrower than full product qualification, and its documented provenance requires reconciliation.

Source PDF, p. 1

Question and predeclared endpoint

Did this specific fine-tuning intervention improve exact instruction adherence on the declared task distribution? The reported positive threshold was +2 percentage points. A negative result was admissible.

The primary endpoint was the untouched 300-case confirmation comparison. Sixty development cases were separate from that endpoint. The same tokenizer, high-reasoning chat template, greedy decoding, batch size and 256-token generation ceiling were reported for both conditions.

An unfinished response, missing final-answer delimiter or execution error counted as failure under the frozen contract. The score therefore concerns completion and exact adherence under that budget, rather than unconstrained general task-solving ability.

Source PDF, p. 3

Model, data and experimental setup

Recorded setup from the revised case study.

FieldReported value
Checkpoint labelIFM/K2-Horizon-3.7B in the local card and path.
Inspected local checkpoint5,058,255,360 unique parameters across 36 BF16 safetensor shards. “5B” describes this inspected checkpoint; it does not authenticate every public K2 Horizon variant.
Training dataset3,000 deterministically selected reviewed English assistant conversations from OpenAssistant/oasst1, pinned revision fdf72ae0827c1cda404aff25b6603abec9e3399b.
InterventionOne epoch, assistant-only NF4 QLoRA; rank 16, alpha 32, dropout 0.05; q_proj and v_proj.
OptimisationAdamW, learning rate 2×10−4, cosine decay, 3% warmup; microbatch 1, accumulation 8; sequence maximum 1,024; training seed 4021001.
Completion375 optimiser steps; reported mean loss 1.4824 over the first 20 steps and 1.4036 over the final 20.
Execution environmentReported local Windows 11 execution: i7-12700K, RTX 3080 Ti 12 GiB, 32 GiB RAM; Python 3.11.16, PyTorch 2.11.0+cu128, Transformers 5.17.0, PEFT 0.18.0, bitsandbytes 0.50.2.

Lexical exact and thresholded near-duplicate screening against 360 development and confirmation prompts reported no matches. This does not establish semantic decontamination or absence of base-model pretraining exposure. The recorded dataset licence declaration was not an independent rights audit.

Source PDF, p. 2

Primary confirmation results

Exact-pass rates by category. Reproduced from the source PDF’s summary table; 50 cases per category. No raw item-level dataset is included in this website package.
Exact-pass rates by categoryOriginal and fine-tuned rates from 50 cases per category. Full numerical values are in the following table. The scale runs from zero to 100 percent.0%25%50%75%100%Arithmetic100%0%Fact extraction96%94%JSON construction100%100%Multi-constraint aggregation88%0%Ordered extraction82%18%String reversal10%0%Original: light bars · Fine-tuned: grey bars · 50 cases per category

Numerical results from the source report. The primary metric is not replaced by post-hoc diagnostics.

CategoryOriginalFine-tunedChange
Arithmetic50/50 · 100%0/50 · 0%-100 pp
Fact extraction48/50 · 96%47/50 · 94%-2 pp
JSON construction50/50 · 100%50/50 · 100%+0 pp
Multi-constraint aggregation44/50 · 88%0/50 · 0%-88 pp
Ordered extraction41/50 · 82%9/50 · 18%-64 pp
String reversal5/50 · 10%0/50 · 0%-10 pp
All cases238/300 · 79.33%106/300 · 35.33%−44.00 pp

All 300 paired confirmation cases are retained in the reported comparison. The source records 16 baseline truncations and 13 baseline execution/format errors, versus one candidate truncation and zero candidate execution errors; those diagnostic categories may overlap and are not additional case exclusions.

Source PDF, p. 4

Paired outcomes and uncertainty

Paired outcome matrix: the same 300 items in both conditions.

Original conditionCandidate passCandidate failTotal
Original pass104134238
Original fail26062
Total106194300

The original passed 134 items that the candidate failed; the candidate passed two items that the original failed. Both passed 104, and both failed 60.

Reported paired change and uncertainty. The interval is reproduced from the PDF, not recalculated from unavailable raw records.
Reported paired change and 95 percent bootstrap intervalFine-tuned minus original: minus 44 percentage points. Reported 95 percent paired bootstrap interval: minus 49.67 to minus 38.33 percentage points. Zero means no change.-60-50-40-30-20-100−44.00 ppNo changeCandidate minus original · percentage points

The report specifies 20,000 paired bootstrap resamples, seed 6172026, and an exact two-sided McNemar value of p = 2.14 × 10−37. The interval reflects item-sampling uncertainty under the analysed set, not variability across training seeds or new task populations.

Source PDF, p. 5

Post-hoc diagnostics and examples

The source’s later log recount reports matching 300-case IDs, the same 238 and 106 pass totals and the same paired matrix. These are the source review’s internal record checks, not a new verification performed for this website.

All 300 candidate raw responses reportedly begin with the reasoning-closing marker </ifm|think>. None of the original raw responses begins that way. This is an observed change in response behaviour, not proof of its causal role.

Selected examples reported in the revised post-hoc review.

DiagnosticCase identityReported observation
Strict format, correct numberconfirm-arithmetic-027The original returned 1350. The candidate returned (31 + 23) * 25 = 54 * 25 = 1350. The number was correct, but the required answer-only format was violated.
Substantive calculation errorconfirm-multi_constraint-014Both outputs were valid JSON; the candidate supplied sum_odd: 101 where the expected value was 85.
A candidate-only improvementconfirm-fact_extraction-030The original supplied the day as the string "4"; the candidate supplied numeric 4 and passed the deterministic validator.

Retrospective diagnostics must not substitute for the frozen primary score. Reconstructed output-token totals of 37,211 versus 6,192 are not saved generation-time token IDs. The shorter candidate runtime does not establish faster per-token inference.

Source PDF, p. 6

What the audit interface actually checked

The reported Osmium Audit v0.1 determination was NOT_SUPPORTED for the bounded improvement claim. The recorded scope covered intake schema, artifact inventory and hashes, declared comparison identities and supplied point-score direction.

It did not inspect raw predictions, calculate the paired interval or p-value, test contamination, independently qualify evaluator validity, reproduce training, or invoke the wider Osmium implementation. Statistical analysis came from separate experiment scripts.

A separate synthetic evaluator-version mismatch returned INSUFFICIENT_EVIDENCE. That demonstrates a narrow declared-identity gate, not detection of hidden evaluator changes or arbitrary semantic contamination.

Source PDF, p. 7

Limitations and unresolved audit provenance

Publication draft — reconciliation required

The source reports conflicting bundle hashes, case identities and evidence-manifest identities across its audit records. Those discrepancies must be reconciled against the native audit record, immutable bundle and creation history before a definitive audit-provenance assertion.

This website preserves that unresolved status. It does not present the audit trace as independently authenticated, and does not silently treat reported document hashes as proof that the reports identify the same bytes.

Other limits include one training seed, a narrow generated test distribution, lexical contamination screening, possible base-model pretraining exposure, and a 256-token generation ceiling. The frozen validators, evaluator implementation, checkpoint tokenizer, original model, adapter and native audit bundle are not available in this website package for independent rerun. Model and dataset rights were not independently audited. The trial is not peer reviewed or third-party certified.

Source PDF, p. 8

Maximum defensible conclusion

For the specific inspected checkpoint, one-epoch training intervention, generated confirmation set and exact-output generation contract reported here, exact-pass performance regressed by 44 percentage points. The evidence does not establish a general decline in all model capabilities or identify the cause.

Training completion, decreasing loss, valid local records and a defensible improvement claim are different questions. The reported result failed the predeclared progress hypothesis; it does not qualify every capability of Osmium or Adamantine.

Source and downloadable summary materials

The original revised PDF is preserved byte-for-byte. The CSV, JSON and vector figures below are explicitly transcribed summaries, not the raw 300-pair prediction logs or an audit bundle.

Source PDF SHA-256: 96b71239c778e35cfd2d33f39e142997b2fdc957fe32a55ae750b17072adea49

Suggested citation: Viridian (2026). When Fine-Tuning Made a 5B Model Worse: Experimental Progress Trial 001. Revised evidence review, publication draft. Study date: 13 October 2026. Website summary added 17 October 2026.

See the PDF evidence register on pages 9–10 for the underlying source inventory and publication-clearance checklist. Listed source artifacts are not represented as downloadable if they were not supplied.

Related: Osmium’s distinct product scope, Adamantine, and the full research library.