7.13 – 7.19 Weekly Update
Summary
My motivation for interning here is to find strategies for better post-training of code-agent models, both with and without a teacher model, namely self-improvement and distillation. I'm currently exploring multiple experimental settings and hypotheses in parallel, identifying the most promising direction, and then investigating it in greater depth.
1. SWE-OPD full-infrastructure evaluation (§1). This week I completed the SWE-OPD training and inference infrastructure and verified that the model trained with OPD improves performance on both SWE-Bench Verified and SWE-Bench Pro.
2. Self-distillation Training (35B-A3B) (§2). Following "Embarrassingly Simple Self-Distillation" — shape the sampling distribution with temperature and top-k / top-p truncation alone — I test whether the recipe carries over to agentic SWE tasks. The self-sampling stage is complete, and the SFT-based self-training stage is now running.
3. Math TIR (tool-integrated reasoning) validation platform (§3). A SWE trajectory takes 25–40 minutes, so the contribution of each improvement is hard to see. I use math TIR for initial validation of the method instead — a 60–100× speedup — to sweep the OPD design space first, then transfer the conclusions back to SWE.
4. Self-in-context distillation experiments (§4). In this setting, the teacher is replaced with a frozen snapshot of the student model, together with injected hints. I will initially use a strong critic model to generate the hints and, if the approach proves effective, gradually transition hint generation to the student itself. I completed the configuration alignment and compared the resulting training dynamics.
1OPD: full infra evaluation
The OPD-trained model beats the 9B base on both benchmarks — SWE-bench Verified (495) and SWE-bench Pro (266) — which demonstrates the correctness of the current infra.
Setup. Qwen3.5-9B student distilled from a standalone 27B teacher, minimal harness, no hint
injection, TP2 × CP2 × DP4 = 16 across 4 training nodes plus 1 teacher node, context 65,536 / generation
12,288, max_tokens_per_gpu 38,912.
1.1 Evaluation results
| Model | Training | Verified 495 | Pro 266 | Avg. turns (V/P) | Abort |
|---|---|---|---|---|---|
| 9B base | none | 38.4% net 40.5% | 20.7% net 22.9% | 162 / 176 | 26 / 26 |
| OPD-plain-i38 | minimal harness | 42.2% net 42.7% | 24.8% net 25.3% | 101 / 120 | 6 / 5 |
1.2 What the numbers say
The gain shows up on both benchmarks and survives pairing. Verified improves by 3.8pp (38.4 → 42.2) and Pro by 4.1pp (20.7 → 24.8). Instance by instance, OPD uniquely solves 50 problems on Verified against the base model's 38, and 17 against 12 on Pro. Combined McNemar significance is marginal (p ≈ 0.14) with limited n, but two benchmarks moving the same direction plus a net per-instance gain is a real signal rather than evaluation noise.
1.3 Why OPD suits long-horizon tasks
A natural worry about applying on-policy distillation to SWE work is cost: trajectories are long, so computing teacher log-probs over them sounds prohibitive. In practice it is not, because OPD is a per-turn teacher–student distillation. The context may be long, but the generated span that actually participates in the OPD computation at each turn is short.
This follows from how multi-turn SWE harnesses are normally built: the think tokens of each turn are not carried into the next turn's input. The next turn sees only the previous turn's tool selection, not the reasoning behind it — otherwise retaining think content at every turn would make the trajectory balloon. But those think tokens are model-generated and must still be trained on, so the implementation proceeds one turn at a time.
2Self-distillation Training (35B-A3B)
Testing whether sampling-distribution shaping alone — temperature plus top-k / top-p truncation — can self-distill a gain on agentic SWE tasks, with no external teacher involved. Training is running.
Apple's paper, "Embarrassingly Simple Self-Distillation Improves Code Generation," shows that a remarkably simple recipe — shaping the exploration distribution and the resulting token distribution by training on self-generated rollouts with appropriate temperature and truncation — can improve code generation on its own.
If the student's own rollouts turn out not to be good enough, there is a fallback: let a teacher intervene in the student's trajectories, either by teacher selection or by teacher re-rollout.
2.1 Choosing the rollout temperature
Eleven temperatures were swept on 100 SWE-bench Verified instances at top-k 10, on a clean H100 curve.
| Temperature | pass@1 | Timeouts |
|---|---|---|
| 0.7 | 53% | 3 |
| 0.8 | 59% | 3 |
| 0.9 | 58% | 0 |
| 1.0 | 59% | 4 |
| 1.1 — selected | 57% | 4 |
| 1.2 | 53% | 4 |
| 1.4 | 46% | 6 |
| 1.6 | 21% | 15 |
| 1.8 | 6% | 7 |
| 2.0 | 1% | 3 |
| 2.2 | 1% | 8 |
1.1 was chosen to preserve rollout diversity. 0.8 and 1.0 score slightly higher (59%), but 1.1 gives up almost nothing (57%) while leaving more exploration in the distribution — which is precisely the lever the self-distillation recipe depends on. Quality collapses beyond 1.4.
2.2 Training data
| Item | Value |
|---|---|
| Rollout source | 6,650 SWE problems / 5,124 Docker images |
| Dataset | 10k self-sampled trajectories |
| Design decision | Train on both positive and negative trajectories — not rejection sampling. |
| Sampling | Temperature 1.1 / top_k 10 / top_p 0.95 — same spec as Apple's paper |
2.3 Error masking
Failed steps stay in the trajectory, so the model still sees them and keeps its error-recovery context. What changes is the loss: the single assistant turn whose action itself was invalid does not contribute. The masked share comes to 2.45%.
| Tier | Condition | Turns | Masked |
|---|---|---|---|
| Tier 1 | No valid action — no tool_call, or hallucinated tool name / arguments | 20,729 | yes |
| Tier 2 | Command syntax error — rc ≠ 0 with command-not-found or syntax-error | 867 | yes |
| Excluded | Legitimate command that failed — a test not passing, a genuine traceback | 10.4% of turns | no |
The exclusion is the important line. A command that runs correctly and reveals a real failure is an error-recovery signal worth learning from; only turns that were never valid actions get zeroed out.
2.4 Training
Running on 2 MI300 nodes under slime Megatron: 2 epochs, global batch size 16, lr 1e-5 cosine, bf16, loss taken only on assistant think and tool_call spans.
Evaluation is pending compute. Once the rollout work frees capacity, base 35B and SFT 35B will be run against Verified 495 + Pro 266, minimal harness only, temperature 1.0.
3Math TIR as a faster ablation platform
A math tool-integrated-reasoning setup generates a trajectory 60–100× faster than SWE while keeping the OPD teacher mechanism byte-for-byte identical — which makes it the right place to answer the open design questions.
The difficulty with iterating on OPD inside SWE is that three expensive properties are entangled: the task needs a CPU-heavy sandbox cluster that is prone to falling over, trajectories routinely exceed 100 turns, and the reward comes from noisy sandbox evaluation. When a run improves or fails to improve, it is hard to attribute that to the OPD signal itself. Math TIR removes all three while changing nothing about the distillation mechanism.
| Dimension | SWE-OPD (current) | Math TIR-OPD (proposed) |
|---|---|---|
| Sandbox | docker / E2B container cluster; CPU-heavy, fragile | in-process subprocess — no containers, nothing remote |
| Trajectory length | often > 100 turns | ≤ 16 tool calls |
| Reward | sandbox eval, noisy | boxed answer — verifiable and clean |
Because the OPD loss and the teacher query are unchanged, the confounders drop out and the OPD signal itself can be isolated — the KL coefficient, the teacher–student gap, early stopping, ReOPD prefix scheduling. Conclusions then transfer back to SWE.
3.1 Measured cost difference
| Metric | Qwen3.5-9B · SWE (minimal) | Qwen3-4B-SFT · math TIR | Ratio |
|---|---|---|---|
| Tool-call turns | ~101–162 | 3.5 | ~30–46× fewer |
| Trajectory length | ~24k tokens | ~3.6k tokens | ~7× shorter |
| Time per trajectory | ~25–40 min ~14s/turn = 9.5s model + 4.5s sandbox | ~24s | ~60–100× faster |
With limited compute, that ratio is the whole argument: settle the OPD design space where a trajectory costs 24 seconds, then carry the answers back to the setting where one costs half an hour.
3.2 Qwen3-4B TIR baseline
| Run | aime24 | aime25 | aime26 | amc23 | hmmt25 | hmmt26 | AVG |
|---|---|---|---|---|---|---|---|
| base, reasoning only | 38.3 | 42.5 | 52.5 | 84.4 | 26.7 | 20.5 | 44.1 |
| base, TIR | 47.9 | 51.2 | 60.8 | 96.9 | 34.2 | 21.2 | 52.0 |
| SFT, reasoning only | 31.2 | 38.3 | 40.8 | 84.4 | 17.5 | 13.6 | 37.7 |
| SFT, TIR | 23.3 | 26.7 | 25.8 | 81.9 | 17.5 | 12.1 | 31.2 |
The base TIR score is misleading on its own. 52.0 against 44.1 for reasoning-only looks like tool use helping, but in many of those runs the model never actually calls a tool. Scoring well is not the same as having learned TIR; genuine interleaved tool-use reasoning is a capability that has to be trained in with RL.
The only openly available model we could find that has learned this interleaved behaviour through RL is from the Qwen2.5 generation. Because its distillation data comes from that older generation, quality is lower and SFT costs accuracy — 37.7 reasoning-only and 31.2 TIR, both below base. That does not block the experiment: the Re-OPD paper works the same way, and we are following their setting.
3.3 Status
Waiting on the teacher SFT, which should complete within a day. After that the OPD and OPSD ablations will likely run here first rather than on SWE, simply because SWE is too slow for the compute currently available.
4OPSD: hint-bearing feedback
The other form self-distillation can take: the student is 35B-A3B, the teacher is its own frozen initial snapshot, and the difference between them is manufactured by injecting a hint.
The motivating observation is that an RL reward is a scalar and carries no direction. It says an attempt was worse without indicating how. Replacing it with hint-bearing feedback, trained through a feedback-distillation loss, turns that into directional supervision. The intended progression is to hold a critic model fixed while it generates hints, then swap the hint source over to the student itself, and eventually let the critic improve on its own.
4.1 Open questions
- Feedback must not contain the answer — otherwise the signal degenerates into supervision.
- Hint quality matters a great deal, which raises the harder question of how hints should be generated at all.
- Passing everything in causes the model to lose focus. Should hints be chunked?
- Could the divergence between the student's trajectory and the teacher's serve as the hint?
- How should the summarization step be designed?
4.2 Two-stage plan
Stage one, in progress: establish whether hints written by a genuinely strong teacher — GPT-5.6-terra in the current experiments — actually help the student train on this task at all.
Stage two: work out how to write good hints, and then whether the student model can be brought to write them itself. If that lands, it closes the loop into iterative self-improvement.
4.3 Configuration against OPD
| Dimension | OPD (9B + 27B) | OPSD (35B-A3B) | Same |
|---|---|---|---|
| Distillation type | cross — 9B student ← 27B teacher | self — 35B student ← own frozen snapshot | ✗ |
| Student | Qwen3.5-9B | Qwen3.5-35B-A3B | ✗ |
| Teacher | Qwen3.6-27B | Qwen3.5-35B-A3B | ✗ |
| Hint | none | GPT-5.6-terra, one per trajectory | ✗ |
| Non-expert parallelism | TP2 × CP2 × DP4 = 16 | TP2 × CP2 × DP4 = 16 | ✓ |
| Nodes | 4 training + 1 teacher | 4 training + 1 teacher | ✓ |
| ctx / gen | 65,536 / 12,288 | 65,536 / 12,288 | ✓ |
| max_tokens_per_gpu | 38,912 | 38,912 | ✓ |
4.4 Training dynamics
| Metric | OPD 9B | OPSD 35B | Reading |
|---|---|---|---|
| step_time | ~72 min | ~68 min | essentially the same |
| rollout_time | 42 min | 36 min | OPSD slightly faster |
| train_time | 31 min | 29 min | comparable |
| actor_train_time | 25 min | 22 min | comparable |
| effective tok/gpu/s | 25.3 | 14.3 | OPD higher — 9B, longer outputs |
| num_turns / traj | 57 | 59 | the same |
| tokens_per_turn | 282 | 106 | 35B emits much less per turn |
| opd_reverse_kl | 0.122 | 0.004 | the substantive difference — see below |
| student_gt_teacher_frac | 0.65 | 0.46 | — |
The reverse-KL gap is the finding worth sitting with. At 0.122 against 0.004 the two runs differ by roughly 30×, which is exactly what self-distillation predicts: a student and its own frozen snapshot start out very nearly identically distributed, so there is almost no divergence to measure. The implication is that OPSD's effective supervision does not come from a capability gap at all — it comes from the distributional difference the injected hint creates. That makes hint quality the central variable rather than a detail, and connects directly to the open questions in §4.1.