TL; DR: Hill climbing means improving the deployable model package one measured step at a time across quality, latency, and cost. SFT made this workload cheaper, but did not transfer the teacher’s planning quality. Adding explicit reasoning produced the major quality gain. RFT in Microsoft Foundry then retained near-teacher quality, achieved 100% feasibility on 150 held-out scenarios, and reduced token cost from $0.09975 to $0.03101 per scenario. Jump to the measured hill climb.
What you will learn
- How to frame model customization as a measurable hill climb rather than a one-shot training decision.
- Where SFT and grader-based RFT differ for multi-constraint planning.
- How a deterministic Python grader turns business outcomes into an RFT reward.
- How Microsoft Foundry manages datasets, training, monitoring, checkpoints, deployment, and evaluation.
- What the training metrics and deployed-model comparison actually establish.
Quick glossary
- Microsoft Foundry: Microsoft’s platform for building, customizing, deploying, and evaluating AI applications and models.
- SFT (supervised fine-tuning): Model customization using labeled input-and-output demonstrations.
- RFT (reinforcement fine-tuning): Model customization in which sampled outputs receive rewards from a grader and the training process increases the likelihood of higher-reward behavior.
- Grader: A scoring function used during RFT. This experiment uses deterministic Python code rather than another model as the judge.
- Feasibility: Whether a complete plan satisfies schema, coverage, inventory, capacity, substitution, quantity, warehouse, shipping-mode, and expedite-budget constraints.
- Hill climb: A sequence of measured model, prompt, reasoning, and fine-tuning changes that seeks a better operational package at each step.
Why Hill Climb the Model Package?
Model selection is not a single-axis leaderboard exercise. A package that produces the highest-quality answer may be too slow or expensive for the workload. A smaller model may be economical but fail the business constraints. The useful unit of comparison is therefore the complete package:
model + prompt + reasoning configuration + fine-tuning method
For this experiment, I evaluated every package on three dimensions:
| Dimension | What was measured |
|---|---|
| Quality | Feasibility plus priority-weighted service, retained margin, and shipping-cost efficiency |
| Latency | End-to-end time to produce one coordinated plan |
| Cost | Input, output, and reasoning-token charges; fine-tuned hosting was tracked separately |
The hill climb used quality as the primary objective. Only after the task was solved well did it make sense to reduce cost and latency without giving that quality back.
Why Microsoft Foundry Changes the Fine-Tuning Workflow
Microsoft Foundry has long supported supervised fine-tuning for supported models. It now also supports reinforcement fine-tuning for supported reasoning models, including grader-based training, structured response formats, RFT-specific hyperparameters, reward monitoring, automatic validation evaluations, checkpoints, and deployment.
This matters because RFT is more than uploading a different JSONL file. A practical workflow must:
- Prepare and validate training and validation datasets.
- Attach a grader that can score generated responses reliably.
- Sample candidate behavior during managed training.
- Run reward-based optimizer updates.
- Monitor training reward, validation reward, reasoning tokens, grader latency, and step duration.
- Produce deployable checkpoints.
- Deploy the customized model and evaluate it on a separate held-out test set.
Foundry turns that lifecycle into a turnkey, managed workflow through its portal and APIs. In this project, GitHub Copilot agent mode in Visual Studio Code drove the workflow code-first, while Foundry supplied the managed training and model-hosting infrastructure. The datasets, grader, job payloads, deployment scripts, and evaluation artifacts remain reviewable in source control.
Model support is method-specific. At the time of this experiment, in Microsoft Foundry, gpt-4.1-mini supported SFT and o4-mini supported RFT; the same base model was not available for both methods. The results should therefore be read as a comparison of deployable model-and-method packages, not as a controlled claim that RFT universally outperforms SFT when the base model is held constant. Always check the current fine-tuning model support table before starting a new experiment.
The Supply-Chain Recovery Problem
The experiment runs on a benchmark of procedurally generated warehouse-disruption scenarios — hundreds used for training and validation, plus a separate held-out set of 150 for evaluation. Each scenario represents a disruption in which one warehouse is unavailable. The model receives:
- 12 to 16 affected orders, each with a SKU, quantity, priority, deadline, margin, and optional approved substitute.
- Three warehouses with availability, SKU inventory, shipment capacity, transit time, and standard and expedite costs.
- One shared expedite budget.
For every order, the model must choose a warehouse, the requested SKU or an approved substitute, a quantity, and standard or expedited shipping, or defer the order. The answer must be one complete JSON plan.
This is difficult because decisions are coupled. Assigning three units of SKU A to one order changes the inventory available to all other orders. Expediting a flexible order may consume the budget needed for an urgent, high-priority order. A plan can use only valid choices and still be suboptimal; worse, one shared-constraint violation makes the entire plan infeasible.
Figure 1. The comparison application exposes orders, priorities, deadlines, substitutes, margins, and the shared expedite budget. Readers can change the scenario and run the deployed packages without changing code.
SFT and RFT Receive Different Learning Signals
The SFT dataset contained a scenario and one selected teacher plan as the labeled assistant response. Only teacher plans that were feasible and scored at least 0.75 were eligible. This produced 148 training and 38 validation demonstrations.
SFT is a strong choice when the desired answer can be demonstrated directly. It teaches the model to reproduce the labeled target token by token. An SFT model can still reason about constraints at inference time, but its training objective does not ask it to generate several valid alternatives, evaluate their business outcomes, or reward a different plan that is better than the single label. Shared constraints are learned only to the extent that the demonstrations and prompt represent them successfully.
RFT rows contained the prompt and scenario but no target plan. The model sampled plans during training, and the grader evaluated each sampled outcome. Every captured scenario was retained, producing 300 training and 60 validation rows across equal loose, mixed, and tight budget-pressure families.
This distinction is central:
| SFT | RFT |
|---|---|
| Learns from one labeled response per example | Learns from rewards on sampled responses |
| Optimizes agreement with demonstrations | Optimizes the grader-defined outcome |
| A different valid plan can still disagree with the label | Different plans can earn similar rewards if their outcomes are similar |
| Requires high-quality target answers | Requires a trustworthy, discriminating grader |
Turning Business Outcomes into a Reward
The grader has two jobs. First, it acts as a hard feasibility gate. Malformed plans, missing or duplicate orders, unavailable warehouses, incorrect quantities, prohibited substitutes, exhausted inventory or capacity, invalid shipping modes, and excess expedite spending receive zero reward.
Second, it scores every feasible plan using normalized service, margin, and cost components:
Q=0.55S+0.25M+0.20CeQ=0.55S+0.25M+0.20Ce
Here, SS is priority-weighted on-time service, MM is retained margin, and CeCe is fulfillment-adjusted shipping-cost efficiency. Service receives the highest weight, but the model cannot ignore margin or spend.
The following shortened excerpt shows the important shape of the Python scorer used in the experiment:
def score_plan(plan, scenario):
# Schema, coverage, type, warehouse, quantity, substitution, and
# shipping-mode checks run before this shortened scoring excerpt.
orders = {order[“order_id”]: order for order in scenario[“orders”]}
warehouses = {
warehouse[“warehouse_id”]: warehouse
for warehouse in scenario[“warehouses”]
}
inventory_used = defaultdict(int)
capacity_used = defaultdict(int)
expedite_spend = 0.0
shipping_cost = 0.0
service_earned = 0.0
margin_earned = 0.0
shipped_orders = 0
for decision in plan[“decisions”]:
order = orders[decision[“order_id”]]
warehouse = warehouses[decision[“warehouse_id”]]
quantity = decision[“quantity”]
inventory_used[(warehouse[“warehouse_id”], decision[“sku”])] += quantity
capacity_used[warehouse[“warehouse_id”]] += quantity
unit_cost = warehouse[f”{decision[‘shipping_mode’]}_cost”]
shipping_cost += unit_cost * quantity
if decision[“shipping_mode”] == “expedite”:
expedite_spend += unit_cost * quantity
delivered_on_time = (
warehouse[f”{decision[‘shipping_mode’]}_hours”]
<= order["deadline_hours"]
)
if delivered_on_time:
service_earned += order["priority"] retention = 0.90 if decision["sku"] != order["sku"] else 1.0
if not delivered_on_time:
retention *= 0.50
margin_earned += order["margin"] * retention
shipped_orders += 1 if any(
used > warehouses[warehouse_id][“inventory”].get(sku, 0)
for (warehouse_id, sku), used in inventory_used.items()
):
return {“score”: 0.0, “feasible”: False, “reason”: “inventory exceeded”}
if any(
used > warehouses[warehouse_id][“shipment_capacity”]
for warehouse_id, used in capacity_used.items()
):
return {“score”: 0.0, “feasible”: False, “reason”: “capacity exceeded”}
if expedite_spend > scenario[“expedite_budget”]:
return {“score”: 0.0, “feasible”: False, “reason”: “budget exceeded”}
service = service_earned / sum(order[“priority”] for order in orders.values())
margin = margin_earned / sum(order[“margin”] for order in orders.values())
fulfilled_fraction = shipped_orders / len(orders)
reference_cost = max(1.0, sum(o[“quantity”] for o in orders.values()) * 15.0)
cost_efficiency = fulfilled_fraction * max(0.0, 1.0 – shipping_cost / reference_cost)
return {
“score”: 0.55 * service + 0.25 * margin + 0.20 * cost_efficiency,
“feasible”: True,
}
Microsoft Foundry packages this logic as a Python grader. During RFT, the model output is available to the grader as the sample, while the structured scenario remains private grader context. This avoids leaking a target answer to the model and keeps the reward deterministic.
SFT Run: Excellent Imitation Metrics, Weak Business Generalization
The SFT run used gpt-4.1-mini-2025-04-14 with the following configuration:
| Setting | Value |
|---|---|
| Training rows | 148 |
| Validation rows | 38 |
| Epochs | 3 |
| Seed | 42 |
| Training type | Global Standard |
| Foundry-selected batch size | 1 |
| Foundry-selected learning-rate multiplier | 2 |
| Training steps | 444 |
The job completed in approximately 1 hour and 6 minutes wall-clock and billed 699,000 training tokens. Final training loss was 0.01, final validation loss was 0.01, and both training and validation mean token accuracy reached 0.99.
Figure 2. The completed SFT job in Microsoft Foundry. Low loss and high token accuracy show that the model learned the selected teacher demonstrations closely.
Those metrics looked excellent, but they measured imitation of the labeled responses. They did not establish that a newly generated plan would satisfy every shared constraint or maximize the business reward. That distinction became visible only after deployment: across 150 held-out scenarios, SFT achieved 0.313 mean quality and only 38.7% feasibility.
This was a valuable failed hill-climb step. SFT reduced P50 latency from 4.54 to 4.18 seconds and token cost from $0.00858 to $0.00119 per scenario, but the quality regression meant the package had not improved overall.
RFT Run: Reward Increased as the Policy Improved
The RFT run used o4-mini-2025-04-16 and the deterministic grader:
| Setting | Value |
|---|---|
| Training rows | 300 |
| Validation rows | 60 |
| Epochs | 2 |
| Reasoning effort | Medium |
| Compute multiplier | 2 |
| Learning-rate multiplier | 1.0 |
| Evaluation interval | Every 5 training steps |
| Validation samples per evaluation | 10 |
| Seed | 42 |
| Training type | Global Standard |
| Foundry-selected batch size | 9 |
| Approximate training steps | 67 |
The run completed in approximately 25.6 hours wall-clock, including about one hour for service orchestration, data import, and GPU queueing. Foundry billed 24.750 training hours. Final mean training reward reached 0.80, and final full-validation mean reward reached 0.82.
Figure 3. Foundry’s RFT monitor shows training reward rising from roughly 0.69 toward 0.80. It also exposes validation reward, reasoning-token usage, grader execution latency, and per-step duration. Periodic validation produces the regular duration spikes.
The curve is noisy because each step uses a different training batch, so the trend matters more than any one point. The validation reward did not diverge from training reward, but training metrics alone were not the decision gate. A model can exploit a grader, overfit its scenarios, or find a mediocre shortcut that still earns reward. The final decision therefore used a separate 150-scenario test set that did not overlap training or validation data.
The Measured Hill Climb
The experiment progressed through four packages:
- GPT-5.2 teacher without explicit reasoning: Fast, but frequently violated coupled constraints.
- GPT-4.1-mini with SFT: Cheaper and slightly faster, but did not preserve planning quality.
- GPT-5.2 teacher with a strengthened prompt and medium reasoning: Solved the task well, with a large latency and token-cost increase.
- o4-mini with grader-based RFT and medium reasoning: Preserved near-teacher quality while reducing latency and token cost.
Figure 4. The hill climb uses a documented composite index: 70% quality, 15% inverse token cost, and 15% inverse P50 latency. The measured components remain visible so the weighting is auditable.
| Package | Quality | Feasible | P50 latency | Token cost/scenario |
|---|---|---|---|---|
| GPT-5.2 teacher, no explicit reasoning | 0.335 | 42.7% | 4.54 s | $0.00858 |
| GPT-4.1-mini SFT | 0.313 | 38.7% | 4.18 s | $0.00119 |
| GPT-5.2 teacher, medium reasoning | 0.851 | 99.3% | 66.68 s | $0.09975 |
| o4-mini RFT, medium reasoning | 0.833 | 100% | 58.59 s | $0.03101 |
The reasoning teacher achieved the highest absolute quality. RFT trailed it by 0.018 quality points, but reduced P50 latency by about 12% and token cost by about 69%. That made RFT the strongest operational balance in this experiment, not the universal winner on every dimension.
Fine-tuned model hosting is separate from the token charges in the table and accrues while a customized deployment is active. Workload volume and concurrency therefore matter when calculating production cost per scenario.
What the Deployed Models Did in the Comparison App
After training, I deployed the SFT and RFT models and exposed them in the Warehouse Planning Gym comparison application. The app sends the same editable scenario to both packages, displays token usage, latency, estimated token cost, and allocation decisions, and then runs the production-style grader to explain the operational result.
Note: The input provided below to the model is different from the one used in the finetuning and evaluation run
Figure 5. On this live scenario, SFT produced a plausible-looking plan but spent $42 on expedite shipping against a $35 shared budget. One coupled-constraint violation made the complete plan infeasible, so the grader returned 0.000. RFT respected the budget and every other hard constraint, earning 0.900.
The SFT failure is exactly the kind that can hide in a row-by-row review. Each allocation decision looks reasonable in isolation. The error appears only after expedite spending is accumulated across the complete plan. RFT took longer for this request because it used 4,160 reasoning tokens, but it produced an executable outcome.
Figure 6. On the same scenario, the reasoning teacher and RFT produced the same feasible allocation and both scored 0.900. RFT’s estimated token charge was $0.0219, compared with $0.0306 for the teacher in this run.
This single scenario makes the constraint behavior understandable, but it is not the performance claim. In this sample, the teacher happened to be faster than RFT (18.2 versus 40.9 seconds). The 150-scenario paired evaluation is the stronger evidence: there, RFT retained near-teacher quality, was faster at P50, and had substantially lower average token cost.
What This Experiment Establishes, and What It Does Not
Five conclusions stand out.
- Hill climbing kept the optimization honest. The cheap SFT package looked attractive until held-out quality was included. The strongest package emerged only after quality, latency, and cost were measured together.
- Training metrics must match the business decision. SFT’s 0.99 token accuracy proved excellent imitation, not constraint satisfaction. RFT’s reward metric was better aligned with the business objective, but still required held-out evaluation.
- A grader converts policy into a learnable signal. Hard rules received zero reward; feasible alternatives received continuous credit for service, margin, and cost. That let training reward outcomes instead of one canonical answer.
- RFT learns a policy, not an embedded optimizer. At inference time the model does not normally run the grader or prove global optimality. For guaranteed feasibility, keep validation, repair, best-of-N selection, or a mathematical optimizer in the runtime architecture.
- Foundry made the lifecycle operational. Dataset management, grader attachment, managed training, reward monitoring, validation evaluations, checkpoints, deployment, and model invocation were part of one platform workflow rather than separate training infrastructure.
There are also important limitations. SFT and RFT used different base models because Foundry’s supported methods were model-specific. The teacher packages used different prompting and reasoning settings. Raw o4-mini could not be deployed for a same-base pre/post comparison because that version was deprecated. This experiment therefore demonstrates a practical package hill climb for one supply-chain workload; it does not prove that RFT always beats SFT or mathematical optimization.
For a production fulfillment system, an optimization solver remains the right choice when a formally optimal allocation is required. The model can still add value by interpreting disruptions, mapping unstructured inputs into constraints, proposing strategies, or producing candidate plans that deterministic components validate.
Get Started
Ready to run the hill climb or adapt it to another constrained planning problem?
Explore the complete implementation: Clone the Supply-Chain Hill-Climb Showcase. The repository includes dataset generation, SFT and RFT submissions, the Python grader, deployment scripts, held-out evaluation, charts, and the model comparison application.
Follow the experiment: Use the step-by-step runbook, review the fine-tuning stage, and inspect the held-out evaluation methodology.
Learn the Foundry workflow:
- Customize a model with fine-tuning in Microsoft Foundry
- Reinforcement fine-tuning in Microsoft Foundry
- Open the Microsoft Foundry portal
Join the conversation: Which business process in your organization has several valid answers, objective outcome measures, and enough complexity that prompt engineering alone has plateaued? Share the scenario in the comments, or open an issue in the repository so the grader and evaluation design can be discussed with the implementation in view.