To a lot of people, getting started with teaching AI new capabilities seems very intimidating. But the truth is that it’s a lot easier than it seems, and I’m going to break down why that is step by step here.
I used a 4-billion-parameter model, NVIDIA’s Nemotron 3 Nano 4B, on my RTX 4090. I taught it to go from essentially no ability to use Blender to creating scenes and arranging them properly most of the time, with about three and a half hours of GRPO training on my single consumer desktop GPU. All of the code is in my repo, including the tasks, the reward, the Blender sandbox, the NeMo Gym environment and the configs for this run.
Before training, the model invented Blender functions that don’t exist, got object sizes wrong by half, and sometimes built the whole scene and then deleted it. After 600 training steps, it built the requested scene correctly 99.8% of the time on the kinds of tasks it practised. On a kind of task it never saw during training, it went from 10.8% to 76.4%.
When you’re post-training a base model, the key to making efficient gains is being able to create an environment with a reward that is easily measurable and easy to improve on, given the environment the model is operating in. Most of what follows is about how we built that reward.
How GRPO works, and why the reward matters
GRPO (group relative policy optimization) is a reinforcement learning method for post-training a model. In this case it trains an adapter, a small set of extra weights called a LoRA that sits on top of the base model, which gives that model a difference in behavior, a specific capability or a specific output mode, without retraining the whole thing.
The task we’re evaluating here is a model’s ability to use Blender. We noticed recently that frontier models have shown tremendous capability at recreating scenes in Blender. Personally, I’ve been wondering how much of that is based on reinforcement learning, and how much of it is based on the sheer size of this new class of models. A 4B model is a good way to poke at the first half of that question.
The experiment puts Nemotron 3 Nano 4B in an environment where it writes Blender Python. We give it a prompt describing the scene we want to create, and it writes a script that builds it. So we’re going from text to scene, essentially. Each prompt names a few cubes and spheres with exact sizes and positions. The model trained on two kinds of task: stack, where it piles two or three cubes on top of each other, and next to, where it places a cube or sphere touching one face of a cube. We held back a third kind, between, where an object goes exactly halfway between two cubes, so we could see whether the model learned anything general.
This type of reward is easy to measure, specifically because in 3D modeling we can determine exactly how close the model gets to a specific scene. As long as the scene we specify is specific enough, the model should be able to infer the meaning and place each object correctly. Our golden examples are the outlines where each shape should exist if the model places it correctly, and that’s how the reward comes into play: we can measure how far away each shape is from the outline where we want it to be. In practice, every requirement (the right name, the right size, resting on the ground, in the right position) becomes a check that passes if the object lands within 3 cm of its outline.
To keep the grading honest, the model’s script runs in a locked-down copy of Blender with no network, a read-only filesystem and a 40-second limit. Then a second, clean copy of Blender opens the saved scene and does the measuring, so the model can’t grade itself. Before training, we tried a set of faked results against the checker, and they all scored zero.
We’re using GRPO here because it learns by comparing multiple attempts at the same prompt. For every prompt, it samples a number of different answers, depending on how we configure it. In this case we’re sampling eight different answers, and we’re scoring them using the mechanism I just described: how far away the shapes are from the outlines we authored. Every answer in the group is then compared with the average of the group. Answers above the average are made more likely, and answers below the average are made less likely, and this is where the feedback signal comes from. The size of each nudge is the difference from the average, divided by how spread out the group’s scores are.
Understanding how each attempt in the group differs from the group’s average gives us signal on how much learning can actually occur during our rollouts. If all eight attempts get the same score, every difference is zero, and that group teaches the model nothing.
So the essential component to GRPO, and to getting this type of hill-climbing, comparison-based learning, is that the reward is detailed and able to give partial credit. It lets us determine how close we are to the ultimate goal, or how far we are from it, and GRPO will then optimize for reaching the best score possible by favoring the kinds of answers that scored above average during our runs.
A group of eight only teaches if its scores differ.
Switch the reward. GRPO scores each sample against the rest of its group, so a group where everyone ties produces no update at all.
Samples above the group's average get pushed up and the rest get pushed down. The two crashes are discouraged hardest, and the near miss is encouraged most.
The eight samples are illustrative. The two numbers above this note are measured: 200 tasks of the trained types, 8 samples each, on the base model at temperature 0.7. The partial-credit figure is rounded.
Why a pass/fail reward doesn’t work here
The obvious reward for this task would be pass/fail: 1 if the scene passes every check, and 0 if it doesn’t. That doesn’t work here, because the base model starts out so badly. Before training, it passed every check on just 1.8% of its attempts at the trained task types. Across 200 tasks with 8 attempts each, only 10.5% of tasks had even one attempt that passed. So with a pass/fail reward, about nine groups out of ten would have been eight zeros, and GRPO would have had almost nothing to learn from.
The base model failed in a few recurring ways. More than a third of its scripts crashed outright, usually because it made up functions that don’t exist in Blender.
bpy.data.mesh, bpy.data.cube and bpy.data.sphere, none of which exist, so it crashes before building anything.
How we set up partial credit
So instead, the reward gives credit for getting closer. It has three parts:
- 0.1 for running at all. A script that runs without crashing is better than one that doesn’t, and with more than a third of scripts crashing, this alone gives GRPO something to compare.
- 0.5 times the share of checks passed. Every correct name, size and position is worth something on its own, so a scene that gets 5 of 7 checks right scores higher than one that only gets 2.
- 0.4 for passing everything. A near miss with 6 of 7 checks scores 0.53, and a full pass scores 1.0. The bonus makes sure a correct scene is always worth clearly more than an almost-correct one, so the model doesn’t settle for collecting partial credit.
When we measured this on the base model, the average reward came out to about 0.22, and the share of tasks where the 8 attempts got different scores went from 10.5% under pass/fail to essentially 100%.
Checking the reward before training
Because of all that, I added a check that runs before any training starts. It evaluates the untrained model and looks at two numbers. The average reward has to be between 0.15 and 0.85, because if it’s too low there’s nothing to build on, and if it’s too high there isn’t much left to learn. And at least 40% of tasks have to produce different scores within their group of 8. If either of those fails, the pipeline stops, and the fix is to change the tasks (make them easier, or do a short supervised warm-up first) instead of burning a night of GPU time on a reward the model can’t climb.
Sampling temperature turned out to matter a lot too. Temperature controls how random the model’s answers are. At 1.0, which is a common default, the base model’s average reward dropped from 0.22 to 0.052, mostly from syntax errors and made-up functions. My first pilot run, before I moved to NeMo Gym, used that default and learned nothing, so I switched to 0.7. Sampling at 0.7 gives answers that are varied enough to compare, but not so random that almost all of them crash.
Running it in NeMo Gym
NeMo Gym is NVIDIA’s library for packaging a task and its reward as an environment that a trainer can call. I wrapped the scene checker as a Gym environment and trained with NeMo RL’s GRPO, using NVIDIA’s prebuilt nemo-rl:v0.7.0 container, which comes with Gym 0.4.0. The environment hands each script off to a sandbox service running on the host machine, so the training container never needs access to Docker itself.
The training settings were pretty standard: 600 steps of 4 prompts with 8 samples each, a learning rate of 1e-5, temperature 0.7, no KL penalty, and a LoRA adapter of rank 32 on the attention and Mamba input layers.
Getting all of this to fit on a 24 GB card took three changes:
- Switching to the Megatron backend. NeMo RL’s default backend keeps a full-precision copy of every weight, including the frozen ones, which is about 15 GiB for a 4B model, and it ran out of memory. The Megatron backend keeps the frozen model in half precision and only the small adapter in full precision.
- Turning off full-layer activation checkpointing. With it on, the adapter got no gradient at all, and there was no error. The gradient norm just read 0.0 and the reward stayed flat. A good reward doesn’t do anything if no gradient actually reaches the weights, so now I run a 5-step pilot first and check that the gradient norm is above zero.
- Shrinking the NCCL buffers. Even on a single GPU, Megatron sets up about nine NCCL communicators, and by default each one reserved around 0.38 GiB. Smaller buffers and a few NCCL settings brought that down from about 3.5 GiB to 0.11 GiB.
After that, training took 3 hours and 31 minutes, at about 20 seconds per step.
Results
The average training reward started around 0.21, passed 0.95 around step 150, and settled at 0.99. On the trained task types, a single attempt passed every check 51.7% of the time by step 100 and 98.4% of the time by step 200.
The held-out between tasks climbed more slowly: 39.1% at step 100, 62.2% at step 200, and a peak of 78.6% at step 400, before ending at 76.4%. Crashes on those tasks fell from 35.6% of scripts to 0.5%. And by the end, if you gave the trained model eight tries at a held-out task, it solved all 200 of them, compared to 54.5% for the base model with the same eight tries.
The model climbs, and the slope runs out.
Top: how often a single sample passes every check. Bottom: how many training prompts still produced different scores within their group of 8.
NeMo RL 0.7 with a NeMo Gym environment on one RTX 4090, 600 steps of 4 prompts × 8 samples. Held-out peaked at 78.6% at step 400; the 500–600 dip is within noise.
Most of the held-out improvement came from skills the trained tasks already taught: working out a position from another object’s center, and setting an object’s height so it sits on the ground. Between needs both of those. Almost all of the mistakes that are left are the same one: in 360 of the 370 held-out scenes that still failed, the middle object wasn’t resting on the ground.
When the signal runs out
A prompt only teaches the model something when its 8 attempts score differently. For the first 80 steps, every prompt did. Between steps 150 and 300, only 8% did, and after step 300 it was down to 2.3%. The model had essentially climbed to the top of what these tasks could reward, and most groups were now eight perfect scores, which teach the model just as little as eight zeros. You can see that drop in the bottom half of Figure 02.
That means training longer on these same tasks won’t help much. The next step is harder tasks for the reward to measure, like rotations, more objects, or relationships that chain together.
One thing I can’t fully explain yet is that the held-out score kept rising from 62% to 79% between steps 200 and 400, even though only about one prompt in 20 still had any signal.
Making sure it didn’t cheat
A model that controls the scene could try to cheat, by writing the measurement file itself, exiting early, or messing with Blender. The setup is built to prevent that. The script and the measurement run in separate Blender processes, and the faked results we tried before training all scored zero. After training, I read through the 20 highest-scoring samples, and they’re plain, correct scripts, with none of them matching our search for tampering patterns.
What I’d take away
The trainer does matter, and I ran the same experiment with TRL as well, but that comparison is a story for another post. For this run, the biggest thing that took the model from crashing on more than a third of its scripts to solving the trained tasks was getting the reward right before training started:
- Give credit for each check separately, plus a bonus for passing everything.
- Before training, measure how many groups actually get different scores, and stop if too few do.
- Keep watching that same number during training. When it drops toward zero, the model has solved the tasks and needs harder ones.
This is one training run, on one model, with one held-out task type that reuses skills from training, so it shows near transfer and I haven’t tested anything further than that. But it does back up what I said at the start: teaching a small model a new capability is a lot easier than it seems. This one took a task we could measure, a reward with partial credit, and a few hours on one GPU. If you want to try it yourself, the repo has the code and configs I used.