Abstract
A one-epoch, assistant-only QLoRA intervention on a locally inspected K2 Horizon checkpoint reduced exact-pass performance from 238/300 (79.33%) to 106/300 (35.33%) on a frozen, generated instruction-following confirmation set. The reported paired change was −44.00 percentage points, with a reported 95% paired bootstrap interval of −49.67 to −38.33 pp.
Training loss decreased while the confirmation result regressed. The source supports a negative result for this intervention and task contract, not universal model deterioration or a diagnosis of its cause. The reported Osmium Audit v0.1 result is narrower than full product qualification, and its documented provenance requires reconciliation.
Source PDF, p. 1Question and predeclared endpoint
Did this specific fine-tuning intervention improve exact instruction adherence on the declared task distribution? The reported positive threshold was +2 percentage points. A negative result was admissible.
The primary endpoint was the untouched 300-case confirmation comparison. Sixty development cases were separate from that endpoint. The same tokenizer, high-reasoning chat template, greedy decoding, batch size and 256-token generation ceiling were reported for both conditions.
An unfinished response, missing final-answer delimiter or execution error counted as failure under the frozen contract. The score therefore concerns completion and exact adherence under that budget, rather than unconstrained general task-solving ability.
Source PDF, p. 3Model, data and experimental setup
Recorded setup from the revised case study.
| Field | Reported value |
|---|---|
| Checkpoint label | IFM/K2-Horizon-3.7B in the local card and path. |
| Inspected local checkpoint | 5,058,255,360 unique parameters across 36 BF16 safetensor shards. “5B” describes this inspected checkpoint; it does not authenticate every public K2 Horizon variant. |
| Training dataset | 3,000 deterministically selected reviewed English assistant conversations from OpenAssistant/oasst1, pinned revision fdf72ae0827c1cda404aff25b6603abec9e3399b. |
| Intervention | One epoch, assistant-only NF4 QLoRA; rank 16, alpha 32, dropout 0.05; q_proj and v_proj. |
| Optimisation | AdamW, learning rate 2×10−4, cosine decay, 3% warmup; microbatch 1, accumulation 8; sequence maximum 1,024; training seed 4021001. |
| Completion | 375 optimiser steps; reported mean loss 1.4824 over the first 20 steps and 1.4036 over the final 20. |
| Execution environment | Reported local Windows 11 execution: i7-12700K, RTX 3080 Ti 12 GiB, 32 GiB RAM; Python 3.11.16, PyTorch 2.11.0+cu128, Transformers 5.17.0, PEFT 0.18.0, bitsandbytes 0.50.2. |
Lexical exact and thresholded near-duplicate screening against 360 development and confirmation prompts reported no matches. This does not establish semantic decontamination or absence of base-model pretraining exposure. The recorded dataset licence declaration was not an independent rights audit.
Source PDF, p. 2Primary confirmation results
Both reported conditions are shown.
Numerical results from the source report. The primary metric is not replaced by post-hoc diagnostics.
| Category | Original | Fine-tuned | Change |
|---|---|---|---|
| Arithmetic | 50/50 · 100% | 0/50 · 0% | -100 pp |
| Fact extraction | 48/50 · 96% | 47/50 · 94% | -2 pp |
| JSON construction | 50/50 · 100% | 50/50 · 100% | +0 pp |
| Multi-constraint aggregation | 44/50 · 88% | 0/50 · 0% | -88 pp |
| Ordered extraction | 41/50 · 82% | 9/50 · 18% | -64 pp |
| String reversal | 5/50 · 10% | 0/50 · 0% | -10 pp |
| All cases | 238/300 · 79.33% | 106/300 · 35.33% | −44.00 pp |
All 300 paired confirmation cases are retained in the reported comparison. The source records 16 baseline truncations and 13 baseline execution/format errors, versus one candidate truncation and zero candidate execution errors; those diagnostic categories may overlap and are not additional case exclusions.
Source PDF, p. 4Paired outcomes and uncertainty
Paired outcome matrix: the same 300 items in both conditions.
| Original condition | Candidate pass | Candidate fail | Total |
|---|---|---|---|
| Original pass | 104 | 134 | 238 |
| Original fail | 2 | 60 | 62 |
| Total | 106 | 194 | 300 |
The original passed 134 items that the candidate failed; the candidate passed two items that the original failed. Both passed 104, and both failed 60.
The report specifies 20,000 paired bootstrap resamples, seed 6172026, and an exact two-sided McNemar value of p = 2.14 × 10−37. The interval reflects item-sampling uncertainty under the analysed set, not variability across training seeds or new task populations.
Source PDF, p. 5Post-hoc diagnostics and examples
The source’s later log recount reports matching 300-case IDs, the same 238 and 106 pass totals and the same paired matrix. These are the source review’s internal record checks, not a new verification performed for this website.
All 300 candidate raw responses reportedly begin with the reasoning-closing marker </ifm|think>. None of the original raw responses begins that way. This is an observed change in response behaviour, not proof of its causal role.
Selected examples reported in the revised post-hoc review.
| Diagnostic | Case identity | Reported observation |
|---|---|---|
| Strict format, correct number | confirm-arithmetic-027 | The original returned 1350. The candidate returned (31 + 23) * 25 = 54 * 25 = 1350. The number was correct, but the required answer-only format was violated. |
| Substantive calculation error | confirm-multi_constraint-014 | Both outputs were valid JSON; the candidate supplied sum_odd: 101 where the expected value was 85. |
| A candidate-only improvement | confirm-fact_extraction-030 | The original supplied the day as the string "4"; the candidate supplied numeric 4 and passed the deterministic validator. |
Retrospective diagnostics must not substitute for the frozen primary score. Reconstructed output-token totals of 37,211 versus 6,192 are not saved generation-time token IDs. The shorter candidate runtime does not establish faster per-token inference.
Source PDF, p. 6What the audit interface actually checked
The reported Osmium Audit v0.1 determination was NOT_SUPPORTED for the bounded improvement claim. The recorded scope covered intake schema, artifact inventory and hashes, declared comparison identities and supplied point-score direction.
It did not inspect raw predictions, calculate the paired interval or p-value, test contamination, independently qualify evaluator validity, reproduce training, or invoke the wider Osmium implementation. Statistical analysis came from separate experiment scripts.
A separate synthetic evaluator-version mismatch returned INSUFFICIENT_EVIDENCE. That demonstrates a narrow declared-identity gate, not detection of hidden evaluator changes or arbitrary semantic contamination.
Limitations and unresolved audit provenance
The source reports conflicting bundle hashes, case identities and evidence-manifest identities across its audit records. Those discrepancies must be reconciled against the native audit record, immutable bundle and creation history before a definitive audit-provenance assertion.
This website preserves that unresolved status. It does not present the audit trace as independently authenticated, and does not silently treat reported document hashes as proof that the reports identify the same bytes.
Other limits include one training seed, a narrow generated test distribution, lexical contamination screening, possible base-model pretraining exposure, and a 256-token generation ceiling. The frozen validators, evaluator implementation, checkpoint tokenizer, original model, adapter and native audit bundle are not available in this website package for independent rerun. Model and dataset rights were not independently audited. The trial is not peer reviewed or third-party certified.
Source PDF, p. 8Maximum defensible conclusion
For the specific inspected checkpoint, one-epoch training intervention, generated confirmation set and exact-output generation contract reported here, exact-pass performance regressed by 44 percentage points. The evidence does not establish a general decline in all model capabilities or identify the cause.
Training completion, decreasing loss, valid local records and a defensible improvement claim are different questions. The reported result failed the predeclared progress hypothesis; it does not qualify every capability of Osmium or Adamantine.
Source and downloadable summary materials
The original revised PDF is preserved byte-for-byte. The CSV, JSON and vector figures below are explicitly transcribed summaries, not the raw 300-pair prediction logs or an audit bundle.
Source PDF SHA-256: 96b71239c778e35cfd2d33f39e142997b2fdc957fe32a55ae750b17072adea49
Suggested citation: Viridian (2026). When Fine-Tuning Made a 5B Model Worse: Experimental Progress Trial 001. Revised evidence review, publication draft. Study date: 13 October 2026. Website summary added 17 October 2026.
See the PDF evidence register on pages 9–10 for the underlying source inventory and publication-clearance checklist. Listed source artifacts are not represented as downloadable if they were not supplied.
Related: Osmium’s distinct product scope, Adamantine, and the full research library.