How Aesthetic RL Taught a 35B Code Model to Paint Watercolours

When a 35B Model Learns to Paint with Code

A viral video of language-model watercolours now has an open, reproducible engineering recipe.

On 23 August, artist and researcher Surya Narreddi posted footage of a model writing JavaScript through the p5.brush library to paint loose, handmade-looking watercolours.

The clip gathered more than 1.5 million views before Hugging Face published a technical walkthrough on 3 September 2026 that strips the idea down to open components.

That new Hugging Face report details how to train a Qwen3.5-35B-A3B model with TRL and OpenEnv so it can paint a peach hibiscus by generating roughly 150 lines of code.

The entire pipeline — reference pool, environment, scorer model, training scripts, and trained adapters — runs on the same platform’s Jobs and Spaces infrastructure and is published in one collection.

This matters because the reward is not a unit test or a known answer.

It is an aesthetic preference model, which means the training signal is taste.

Inside the Reward Function: Taste as a Trainable Signal

The environment wraps everything between the policy model and the reward calculation.

It includes the p5.brush JavaScript library, the restrictive system prompt, a headless Chromium renderer, and a gate that rejects sketches that fail to compile, bypass the library, or cheat the scorer.

p5.brush simulates a physical medium: pigment bleeds beyond shape edges, paper texture adds resistance, and flow fields drag brushwork across the canvas.

The library exposes 47 methods, but the training prompt allows only 10.

That tight allowlist forces the model to paint with filled shapes and lets the library’s bleed effect carry the watercolour style.

One deliberate addition from the engineer behind the reproduction is a single instruction: paint each petal two or three times, first a large pass and then a smaller, more opaque layer inside it.

The reward function mixes four terms with unequal weights.

• Gate — 0.05 weight; verifies the sketch compiles, paints something, and does not cheat.
• Length — 0.05 weight; softly encourages longer code snippets.
• Pairwise judge — 0.60 weight; compares the candidate against four references from the pool.
• HPSv3 — 0.30 weight; an open 7B aesthetic preference model that scores how much a person would prefer the rendered image.

The pairwise judge is a Qwen3-VL-30B-A3B-Instruct vision model called through inference providers.

It sees the candidate painting next to four randomly selected references, evaluates both presentation orders against written criteria such as bleeds, translucent washes, and soft edges, and returns the share of comparisons the candidate wins.

Because its only standard is the hand-rated pool, that share encodes the curator’s personal taste.

HPSv3, by contrast, is an average of many people’s preferences, trained on large-scale human pairwise choices.

The reference pool contains 178 paintings divided into two tiers: love and okay.

All were generated by four open-weight models writing p5.brush sketches, each based on openly licensed hibiscus photos.

Every render was rated by hand, and only 178 made the cut.

The tier split is not cosmetic; the pairwise judge draws half its references from each tier so weak early policies still get winnable comparisons.

No human-made painting appears in the pool, a limitation the report acknowledges.

Three training runs explored how much weight the pairwise judge should carry.

• Judge-led — 0.60 judge, 0.30 HPSv3, stopped at step 110.
• HPS-led — 0.30 judge, 0.60 HPSv3, stopped at step 110.
• HPS-only — 0.00 judge, 0.90 HPSv3, stopped at step 60 as a validation run.

The more weight the pairwise judge carried, the lower the initial reward and the noisier the climb.

Mean group reward moved from 0.45 to 0.72 for judge-led, from 0.57 to 0.82 for hps-led, and from 0.58 to 0.71 for hps-only.

The key finding is not just that all three learned, but what they learned.

In hps-only, most of the improvement came from making bad paintings rare; the quality of the good ones barely moved.

Adding the pairwise judge pushed the model to get closer to the pool, which doubled paint coverage and raised the best-of-step quality in both judge runs.

The report also notes that the model ignored an explicit instruction to produce fifteen to thirty filled shapes; the actual mean stayed between seven and nine because the reward never reinforced that sentence.

Infrastructure failures were quietly entering the reward as zeros.

Across all runs, about 1.5 percent of rollouts were affected, and the worst run reached 5.2 percent.

The fix was to return None instead of zero so the rollout is excluded from the group, and a websocket caching bug in OpenEnv was fixed and submitted upstream.

Visual RL Moves from Verifiable Rewards to Aesthetic Judgment

The watercolour reproduction is not an isolated curiosity.

A paper published on arXiv under identifier 2511.00391v3 describes VinciCoder, a two-stage multimodal code generation pipeline that also uses reinforcement learning with a visual reward.

VinciCoder first applies supervised fine-tuning on 1.6 million multimodal code samples and then runs Group Relative Policy Optimization on 42,000 RL samples across chart-to-code, web-to-HTML, image-to-SVG, image-to-LaTeX, and chemical image tasks.

Its reward combines a coarse-to-fine visual reward weighted at 0.9 with a language-alignment reward weighted at 0.1.

The visual reward compares DINOv2 embeddings of a global thumbnail and local patches, and any failed render receives a visual reward of zero.

That objective is closer to fidelity than taste: it rewards structural similarity to a target image, not a curator’s preference.

The watercolour project flips this by making the pairwise judge the dominant term.

Its reward function assigns 0.60 weight to a judge that compares against a hand-rated pool and only 0.30 to the average-preference HPSv3 model.

The difference matters because it changes what the model optimizes: rather than matching a reference, it learns to produce output that a specific human finds beautiful.

VinciCoder’s ablations reinforce why a prior matters.

The paper reports that applying visual RL directly to a base model helped on chart-to-code and design-to-code tasks but degraded performance on chemical drawing.

It also warns that visual rewards can be gamed in unfamiliar domains without supervised fine-tuning first.

The watercolour project had a similar safeguard: the base Qwen3.5-35B-A3B model already generated valid p5.brush sketches, so the RL phase only had to steer style rather than teach syntax from scratch.

Community tutorials on verifiable-reward code RL still emphasize format extraction, compile checks, and unit test pass rates as the dominant reward stack.

They also warn that subprocess execution is not a security sandbox, a concern that applies equally to rendering untrusted JavaScript in a headless browser.

The watercolour environment handles that by running inside a Docker Space with a gate that rejects direct p5 calls or text-on-canvas cheats.

Cost remains the sharpest barrier.

The watercolour run needed a single H200 for up to 34 hours, but 70 to 80 percent of each 15-to-18-minute step went to software rendering in a CPU-only Space.

VinciCoder’s RL phase used 16 GPUs for policy training and four more for reward scoring, a footprint that puts visual RL beyond most individual experimenters.

Still, the open release of the watercolour recipe, pool, and trained adapters lowers the practical threshold for teams that want to build aesthetic RL for creative code.

The Bottleneck Is Human Judgment

The watercolour project proves that a reward signal built from 178 hand-rated images can steer a code model’s style, but it also exposes the real limit: someone has to decide what belongs in the pool.

For teams building reinforcement learning pipelines that need to scale beyond a single curated pool, programmatic SEO AI automation is how Andres SEO Expert approaches the infrastructure — contact us.