RL scheduler

Hand-written PPO for a render queue

Role: Simulator, algorithm, evaluation
Tools: PyTorch
PPO
NumPy
Status: Functional

Policy learning over 2,000 iterations.

less weighted tardiness than FIFO; ATC, the reference, stays ahead (+78.2%): +76.7%
test scenarios, unseen seeds: 300
of CPU training: 3.2 min
state inputs, 256 × 256 MLP: 196

The problem: Images and videos arrive continuously with different durations, priorities and deadlines: in what order should they render?
For whom: Studio Flow’s render queue; a weighted-tardiness problem, NP-hard and classic in scheduling.
The outcome: A PPO policy trained in 3.2 min on CPU: +76.7% vs FIFO over 300 unseen scenarios.

Context

Images and videos arrive continuously with different durations, priorities and deadlines. Minimising total weighted tardiness is a classic NP-hard problem: a learned policy can take the whole queue state into account.

What I built

Simulator calibrated on durations actually measured by the GPU bench (P06).
Hand-written PPO: actor-critic, clipped ratio, entropy 0.02.
Fair comparison against five heuristics, including ATC, the field reference.
Integration point for Studio Flow: rank(tasks).

Architecture 5 blocks

01 · InputSimulatorPoisson arrivals, 0.85 load, measured durations (P06)
02 · ToolState24 tasks × 8 features + 4 globals = 196 inputs
03 · ModelPPOActor-critic, clipped ratio, entropy 0.02
04 · ToolBenchmark300 unseen scenarios vs FIFO, SPT, EDD, WSPT, ATC
05 · Outputclasser(taches)Greedy policy, Studio Flow integration point

Pipeline 4 steps

SimulatorPoisson arrivals, load 0.85.
PPO2,000 iterations.
Benchmark300 unseen scenarios.
Integrationrank(tasks) entry point for Studio Flow, deterministic decisions.

Decisions the choice, the reason, the alternative ruled out

D1
Calibrate the simulator on durations actually measured by the GPU bench (median image: 25 s).
The policy learns on a realistic queue with no dataset to buy.
Ruled outArbitrary durations.
D2
Add ATC, the field’s reference rule, to the benchmark.
A complete benchmark against the field’s reference: the learned policy finishes 3rd of 6, behind ATC and EDD. The result is published as is.
Ruled outComparing only against FIFO.
D3
Scale rewards by ×0.001 (one unit = 1,000 s).
To stabilise PPO training.
D4
Evaluate in streaming mode, the training mode.
The policy is trained for continuous arrivals: the regime of a production render queue.
Ruled outA static-burst evaluation, outside the target regime.
D5
A greedy policy at the integration point.
Deterministic, replayable decisions, with no sampling. The rule in use stays the one that wins the benchmark.

Results

Policy	Mean weighted tardiness	p90	Gain vs FIFO
FIFO	2493	5652	—
SPT (shortest first)	675	1506	+72.9%
WSPT	682	1438	+72.6%
EDD (deadline)	571	1206	+77.1%
ATC (reference)	544	1114	+78.2%
P08 (RL)	581	1167	+76.7%

Ranking: ATC, EDD, P08 (RL), SPT, WSPT, FIFO. The learned policy only replaces a simple rule if it beats it; that is not the case yet.

Deliverables 4

Trained policy (politique.pt), replayable.
Report, CSV summary and per-scenario results.
Learning and comparison charts.
Reproducible scenario (seed 2026): the policy ties the best, 57 = EDD = ATC.

Environment

Hardware and measurement setup

Trained in 3.2 min on CPU, no GPU.

Reproduce README commands

Private code: read access under NDA, or a 30-minute live demo. Request access →

PY=~/Documents/ComfyUI/.venv/bin/python
$PY ordonnanceur/entrainer.py --iterations 2000   # ≈ 3 min CPU → resultats/
$PY ordonnanceur/demonstration.py

Outputs: rapport.md, resume.csv, per-scenario results, charts and politique.pt.
Simulator calibrated on the GPU bench durations (P06).

Gallery 1

RL policy vs heuristics, 300 scenarios: ATC stays ahead.

What this project means for you

The same know-how, on your problem.

Optimising a production queue

Renders, processing, tickets: I model your queue, measure the simple rules first, and keep learning only if it beats them.

A simulator before production

Calibrated on your real durations, it lets you try a policy without touching production.

A fair benchmark

The method includes the best known heuristic: if a simple rule is enough, I will tell you.

Scope a project →The company offer