When Does RL Actually Help Fine-Tuning? A Difficulty-Controlled Study on Structured Generation

The uncomfortable question

Reinforcement learning is often the finishing move of the modern fine-tuning stack: run SFT first, then add RL (GRPO, PPO, DPO) to squeeze out the last few points. In practice the return is wildly inconsistent — sometimes a real jump, sometimes nothing after a burned GPU budget. The folk rule “RL helps when the task is hard” is directionally right but too vague to budget against: it doesn’t say how much, which fields, or how to check in advance. This post turns that intuition into a decision rule you can apply before committing to an RL run.

TL;DR. In a controlled text-to-JSON experiment, GRPO added only +0.0111 absolute after SFT on a straightforward formulation where target attributes were explicitly provided. On a harder formulation requiring attributes to be inferred and normalized from prose, the weighted score increased from 0.6983 to 0.7498 — +0.0515 absolute, or approximately +7.4% relative. The largest gains occurred in Semantic-IoU (SIoU)-scored attribute fields, while cosine-scored free-text fields improved less. The practical takeaway is that low post-SFT performance alone is not enough to justify RL: first confirm that SFT has plateaued, that the missing information is recoverable, and that the reward can provide meaningful sequence-level credit. Fully reproducible on public ABO data.

The setup: one task, two difficulties

The task is a large class of enterprise fine-tuning jobs: turn semi-structured product data into a fixed 8-field JSON object. Four fields are free-text editorial descriptions scored by cosine similarity; four are sparse attribute lists (category, color, material, style — surfaced in the schema as inference-type tags like vibe, quick_semantic, trend_signals, and overall_vibe) scored by Semantic-IoU (SIoU) (fuzzy set overlap). The reported metric is the weighted aggregate of these eight per-field scores. GRPO uses the same underlying cosine and SIoU scoring functions as evaluation; during training both arms apply the same reward shaping (threshold annealing and per-field weighting), while final test reporting uses a fixed SIoU threshold (τ = 0.7) and equal field weights.

We use Amazon Berkeley Objects (ABO) (public, CC BY 4.0) as a stand-in for a proprietary product catalog, and build two arms of the same task:

• Easy — the target attributes (productType / color / material) are handed to the model in the prompt, so the gold is copyable. SFT can reformat its way to a high score; RL should have little to add.
• Hard — those same label lines are hidden in prose (the info still lives in the product name and bullets), so the gold must be inferred and normalized, not copied.

Both arms use the same Qwen3-32B base and the identical SFT-then-GRPO training recipe — same LoRA config, learning rates, sampling settings, reward functions, reward curriculum (threshold annealing), and per-field weighting. The only thing that changes is whether the target attributes are exposed in the prompt or must be inferred from prose. Train and test are disjoint product sets, so the reported gains are measured on held-out items the model did not see in training.

What the two arms look like (same product)

A representative pair makes the difficulty knob concrete — the rows are the same item, differing only in whether the structured attributes are exposed:

Easy — attributes exposed (gold is copyable):

Every gold token already appears in the prompt, so SFT can reformat its way to a high score — RL has little to add.

Hard — the same three label lines are hidden (gold must be inferred):

The attributes still live in the product name (“yellow gold”, “earrings”), so the model must locate and normalize them from prose instead of copying a labeled field — and the free-text gold is rephrased so the bullet-copy shortcut is closed too. That is where SFT leaves more headroom and GRPO produces a larger improvement.

The headline result

Two things stand out. First, the observed SFT ceiling depends strongly on task formulation: exposing the attributes lets SFT learn a largely copy-and-format transformation, reaching 0.8885; hiding them drops it to 0.6983 because the model can no longer copy. Second, the GRPO gain is much larger on the hard formulation (+0.0515) than the easy one (+0.0111).

The pattern in one line: headroom is necessary but not sufficient — GRPO helped most where SFT had plateaued on recoverable, inference-heavy fields and the task-level reward provided direct semantic credit. In this experiment the size of the GRPO gain was strongly associated with the headroom remaining after SFT, particularly for inference-oriented fields.

Figure 1. Weighted held-out scores before and after GRPO. The easy formulation improves by +0.0111 absolute, while the hard formulation improves by +0.0515 absolute, equivalent to approximately +7.4% relative.

Where the gain lands: it’s the field type, not just the score

If the pattern holds, the hard-arm gain should concentrate in the SIoU list fields — the ones that require inference — and touch the already-strong cosine fields less. That is what we observe.

Notice the contrast. On the hard arm, the four cosine free-text fields also have headroom (rich_semantic 0.788, trend 0.819), yet GRPO produced smaller and less consistent gains there (e.g., trend +0.039, quick_summary +0.024). The four SIoU inference fields, sitting at 0.47–0.67 after SFT, are where the largest gains land: quick_semantic +0.105, vibe +0.101, trend_signals +0.061, overall_vibe +0.052.

Figure 2. Per-field GRPO-minus-SFT differences. The largest hard-arm improvements occur in SIoU-scored attribute fields, especially quick_semantic and vibe.

Why? Standard SFT optimizes token-level likelihood against one reference output. Although it can generalize beyond exact memorization, the objective does not directly assign sequence-level partial credit based on the downstream SIoU or cosine metric. GRPO instead evaluates sampled outputs with the task-level grader and reinforces higher-scoring generations. This gives the optimizer a direct signal for recovering semantically correct attributes even when the generated sequence differs from the reference wording. In this experiment, GRPO’s gains concentrated where the missing information was recoverable from the input but poorly served by token-level imitation — the hard-arm SIoU fields.

A pre-flight check: should you run RL at all?

You don’t need to run RL to estimate whether it will help. Because GRPO’s gains here concentrated where SFT left recoverable headroom, a short SFT-baseline inspection is informative. But mind the trap: a low score is necessary, not sufficient. The check is three cheap steps:

1. Always run SFT first. RL isn’t a replacement for SFT — it comes after it. So the question is never “SFT vs RL,” but “after SFT, is it worth spending more on RL?” SFT is relatively cheap, so do it first.
2. Score the SFT checkpoint per-field on a held-out test set (not just the aggregate). This is nearly zero-cost — a single inference pass plus grading.
3. Apply the checklist below to each low-scoring field.

When to consider RL. Consider RL when SFT has genuinely plateaued, important fields remain below target, the missing information is recoverable from the input, sampled outputs can meaningfully vary, and a reliable task-level reward can distinguish partial improvements. In this study that reward is SIoU on attribute lists — a concrete choice for this task, not a universal requirement.

The point is that a low score alone is not enough. GRPO helped here specifically where the missing information was recoverable and the task-level reward could credit partial progress. A low free-text (cosine) score, a noisy label, and an unconverged SFT are different problems with different fixes, and GRPO should not be assumed to solve them automatically.

Figure 3. Exploratory relationship between post-SFT field score and subsequent GRPO gain. Each point is one field on one arm: x = its post-SFT score (inspectable on your own checkpoint), y = the GRPO gain it received. Color = arm (orange = hard, blue = easy); shape = field type (circle = inference / SIoU, square = free-text / cosine). In this experiment the largest gains occur among low-scoring SIoU attribute fields, while free-text fields generally show smaller gains over the observed score range. (Headroom size alone still correlates with gain, pooled R²≈0.74 — the field-type axis is what a headroom-only reading misses.)

A prerequisite: make sure you’ve actually maxed out SFT

The whole diagnostic assumes the post-SFT score reflects a genuine ceiling, not an under-trained run. Hyperparameters absolutely matter here: if SFT is left under-tuned, its “headroom” is illusory — RL will appear to help when a better-tuned SFT would have closed the same gap for far less compute. So before you read any field as “stranded,” confirm SFT has converged:

• Train to a plateau, then measure. Push epochs/steps until the held-out score stops improving. A still-rising validation curve means you’re measuring an unfinished SFT, not headroom.
• Sweep the few hyperparameters that move the needle — learning rate, epochs, and (for LoRA) rank — and keep the best-validation checkpoint. You don’t need an exhaustive grid; you need evidence the val metric is no longer sensitive to these.
• Sanity-check the copyable fields. Fields whose answer is directly copyable from the input should saturate near 1.0. If even those are low, SFT is under-trained — fix that first; recovering copyable content is SFT’s job, not RL’s.

In this study both arms share one SFT recipe (Qwen3-32B, LoRA rank 32, lr 5e-4) that we confirmed can saturate: on the easy arm it reaches 0.8885 with many test items at exact 1.0 — direct evidence the recipe isn’t leaving easy points unclaimed. Because the identical recipe (SFT, GRPO, and reward shaping) sits under both arms, the difference in GRPO gain is attributable to task difficulty rather than to a change in the training setup

What we can conclude — and what remains open

Shows: in this experiment, GRPO’s post-SFT gain was strongly associated with the headroom remaining after SFT, particularly for inference-oriented fields — small on the easy formulation (+0.0111) and larger on the hard one (+0.0515), concentrated in the SIoU attribute fields.

Does not establish: that this holds unchanged for hard-verifiable rewards (math, code) or other task families. The gold is a rule-derived silver label, not a human-preference target. The easy +0.0111 is a small gain, and the reported numbers use a 48-example held-out subset with a single seed, so we do not treat it as a significance test; the hard +0.0515 is larger and lands on the predicted field-level structure, but additional seeds and a larger test set would be needed to quantify uncertainty.

Confounded factors. In this study, inference requirement, output format, and grader type are partly coupled: the four hidden attribute fields are list-valued and scored with SIoU. The experiment identifies a useful empirical pattern but does not fully disentangle these factors. The hard formulation also removes explicit attribute lines and reduces direct lexical overlap with the target output, so it changes more than a single microscopic difficulty knob. The GRPO recipe (its reward curriculum and per-field weights) was tuned on the harder task and applied unchanged to both arms; holding it fixed keeps the arm-to-arm comparison clean, but the recipe is not independently optimized for the easy arm.

What’s next

• Extend to hard-verifiable rewards (math-correctness, code-execution) to test whether the observed headroom–gain relationship holds under other reward types — the most important open question.
• Explore a predictor for whether the post-SFT per-field distribution can forecast the GRPO gain.
• Sweep more difficulty points (a graded reveal-fraction) to trace the full curve.

Reproduce it on public data (CC BY 4.0)

Everything runs on the public Amazon Berkeley Objects dataset (Collins et al., CVPR 2022; abo-listings text metadata only) — no proprietary data, checkpoints, or code required. The study is specified end-to-end below, and because evaluation uses the same cosine and SIoU graders (at a fixed threshold τ = 0.7 and equal field weights), the headline numbers can be verified from a set of model predictions with no fine-tuning access at all; only retraining the models needs the preview.

1. Get the data. Download the ABO listings metadata (public, no credentials):

aws s3 cp –no-sign-request –recursive s3://amazon-berkeley-objects/listings/ abo-listings/listings/

2. Build the two arms. Filter to fashion listings and map each product’s real attributes (brand, product_type, color, material, style, pattern, bullet points, keywords) into the 8-field editorial JSON — four free-text fields + four attribute-list fields. Build both arms aligned by product_id, differing by the difficulty knob: easy exposes the productType / color / material lines in the prompt (gold is copyable); hard hides those label lines (the words still live in the product name and bullets) and rephrases the free-text gold so the copy shortcut is closed. Split disjointly by product_id (4,800 / 600 / 600 train / val / test). Because of inference and grading cost, the numbers in this post are computed on a fixed 48-example subset of the 600-example test split (seed 42), used for every checkpoint (both arms, SFT and GRPO).
3. Verify the numbers (no special access). The evaluation grader is fully specified in the Appendix: cosine similarity on the four free-text fields + SIoU (embedding overlap at τ = 0.7, text-embedding-3-small) on the four list fields, averaged with equal weights into weighted_overall. Implement it and score predictions on that fixed 48-example evaluation subset drawn from the held-out test split; you should land within small-sample range of Easy ≈ 0.89, Hard SFT ≈ 0.70 → GRPO ≈ 0.75.
4. Rerun the training (needs preview access). The fine-tuning itself uses the Interactive Post-Training API (currently in private preview). The base model, SFT configuration, GRPO configuration, and reward shaping are identical across arms; only the data (attributes exposed vs. inferred) changes:

• • SFT: Qwen3-32B LoRA r32, lr 5e-4, batch 32, 1 epoch, 4,800 train examples.
  • GRPO (from the SFT checkpoint): lr 2e-5, temperature 1.0, group_size 8, groups_per_batch 8, importance-sampling loss, 64 steps, seed 42. Both arms anneal the SIoU threshold 0.5→0.7 and up-weight the inference-type list fields (overall_vibe/trend_signals ×3, vibe ×2, quick_semantic ×1.5).

Score both checkpoints with the same evaluation grader (τ = 0.7, equal weights) and compare the SFT vs GRPO deltas. This is a custom reward on public data, not a preset recipe.

Bottom line

Run and tune SFT first. Then inspect held-out performance by field rather than relying only on an aggregate score. When SFT has saturated on a copy-and-format task, RL may offer little additional value. When important, recoverable attributes remain difficult and a reliable task-level reward can distinguish partial semantic improvements, RL becomes a more promising investment. These results establish that pattern for one structured-generation task; testing whether it generalizes to math, code, preference optimization, and other reward types remains future work.

Appendix — Scoring metrics

This is the evaluation grader used to report all test metrics in this post. Each of the eight fields is scored in [0, 1] by one of two functions, then averaged into weighted_overall. Training additionally applies reward shaping (threshold annealing and per-field weights, the same on both arms); the numbers reported here use the fixed evaluation configuration below (τ = 0.7, equal weights). All embeddings use text-embedding-3-small; emb(·) is the embedding of a normalized (lowercased, whitespace-collapsed) string.

1. Cosine fields (quick_summary, rich_semantic, summary, trend) — free-text descriptions. Gold and prediction are each reduced to one string, embedded, and compared:

2. Semantic-IoU fields (quick_semantic, trend_signals, overall_vibe, vibe) — sparse attribute lists. Let G, P be the gold and predicted lists. Embed every item, greedily match pairs whose cosine ≥ τ (neither already matched), and let m be the number of matched pairs (the semantic intersection |G ∩ P|):

This is the Jaccard index |G ∩ P| / |G ∪ P|, but intersection is defined by embedding similarity ≥ τ (τ = 0.7) instead of exact string match — so “sneakers” matches “shoes” without being identical.

3. Aggregation. With per-field scores sₖ and equal weights (wₖ = 1 over all 8 fields at scoring time):

The denominator is fixed at the full field count (8), so a missing or malformed field contributes 0 to the numerator but still counts in the denominator — the model can’t inflate its score by emitting only the easy fields, and output that fails to parse as JSON scores 0.

In one line: cosine fields ask “is the sentence semantically close?”; SIoU fields ask “how much do the two attribute sets overlap, allowing synonyms?” The hard task’s headroom lives almost entirely in the second question — which is exactly where GRPO delivers its gain.

References

This work uses the Amazon Berkeley Objects (ABO) dataset (Collins et al., ABO: Dataset and Benchmarks for Real-World 3D Object Understanding, CVPR 2022, arXiv:2110.06199). Data © Amazon.com, used under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). We use only the abo-listings text-metadata subset (not the 3D assets), which we reshaped and adapted into the text-to-JSON task described above.

Disclaimer

© 2026 Microsoft Corporation. All rights reserved. This document is provided “as-is.” Information and views expressed in this document, including URL and other internet website references, may change without notice. You bear the risk of using it.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

Some examples are for illustration only and are fictitious. No real association is intended or inferred.