PLAYABLE TRAINER — SERVER RACKS · AI TRAINING · AGENT LOOPS · SAMPLING
8 racks × 16 B200-class GPUs = 128 accelerators. Real thermal model: heat builds and sheds over time — nothing changes instantly. Pump speed and coolant setpoint actually drive the sim.
FAILURE INJECTION is the lesson: real floors break. A fan failure halves one rack's cooling. A leak drops loop pressure and cooling efficiency everywhere until you reset. Watch junction temps climb toward the real throttle point — 83 °C, not before.
Cold aisle ≈ coolant inlet · Hot aisle runs 15–25 °C above it under load.
Faster pump = more flow = lower junction temps, but pump power raises PUE. Try 1,200 RPM under full load and watch Rack 04.
Colder coolant buys thermal headroom and costs energy. Real direct-to-chip loops run warm (25–32 °C supply) on purpose — free cooling beats refrigeration.
Inference predicts. Training learns. Run gradient descent on a small transformer run: forward pass → loss → backward pass → optimizer step, thousands of times. Your knobs decide whether it converges, crawls, or blows up.
A batch of text flows through the network. Every weight participates. Out come predictions for the next token at every position.
Predictions are scored against the real next tokens (cross-entropy). Loss ≈ 4.6 means "very wrong"; ≈ 1.0 means the model mostly knows what's coming.
Backpropagation applies the chain rule layer by layer, computing how much each weight contributed to the error — its gradient.
w ← w − lr × ∇w
Each weight nudges against its gradient. Too big a nudge (high lr) and training diverges. Too small and it takes forever.
Pick a learning rate and press Run. Goal for Mission 3: get loss under 1.50 without a divergence.
The single most important knob in training. Past ~0.03 on this run, gradients explode and loss goes NaN. That's a real failure mode, not a bug.
Bigger batch = smoother, less noisy gradients and higher throughput — but each step costs more compute.
Each GPU trains on its own shard of the batch, then all GPUs average gradients in an NCCL all-reduce ring. More GPUs = faster steps, with growing sync overhead.
Lower precision = more throughput per watt. Production runs mix: BF16/FP8 math with FP32 master weights.
Perceive → Reason → Act (tool) → Observe → Reflect → repeat. Failure injection is ON: tools fail here, like they do in production. Watch the agent notice, reflect, and retry instead of pretending everything worked.
Telemetry ingestion & environment state
Chain-of-thought decomposition on a scratchpad
Bash, Python, Modbus, sensor/actuator calls
Check the result — including failures — and re-plan
Store the lesson so the next incident starts smarter
// Agent idle. Pick a scenario and run the cycle. One tool call in each run is rigged to fail first — find out whether the agent recovers.
// Waiting for tool dispatch. Exit 0 = success. Exit 1 = failure — and failures are information.
This is exactly how a language model picks the next word: logits ÷ temperature → softmax → top-K → top-P nucleus → sample. Every slider genuinely reshapes the distribution.
T→0: greedy, always the top word. T→2: flat, chaotic, creative.
Keep only the K most likely candidates (this vocab has 10).
Keep the smallest set whose probabilities sum to P. The rest are pruned (dimmed).
P(token) = exp(logit ÷ T) ÷ Σ exp(logits ÷ T) — with max-subtraction for numerical stability, exactly as production inference code does it.
Causal mask: a token can only attend to itself and earlier tokens — the future is blacked out. That's what makes generation autoregressive.
SIREN networks use sin(ω₀x) activations instead of ReLU, so they can represent fine detail — and their derivative is another SIREN (a cosine). The same math shapes the alarm synth below.
ω₀ scales the input frequency. High ω₀ = fine detail, harder to train. The SIREN paper's default is 30.
Overtones summed into the waveform: Σ sin(k·ω₀x)/k. More harmonics = richer, more piercing alarm.
A SIREN fed (x, y) coordinates can memorize an entire image or sound field in its weights — that's an implicit neural representation.
The physical layer under everything above. These relays are wired into the floor sim: kill the aux pump and cooling suffers; engage overdrive and Rack loads (and heat) climb.
481.2 V
— A
Check it yourself: P = V × A × √3 × PF. At 480 V, the floor's real draw lands near 260 A — not thousands.
— °C
Off = pump capped at 2,000 RPM.
On = racks run +12% load. Heat follows.
On = InfiniBand fabric partitioned; training all-reduce stalls (check the Training Lab throughput).
Real EPO cuts power AND cooling. Loads die instantly — but junction temps first coast UP on residual heat, then decay slowly. Try it and watch.
Four jobs. Each one teaches a real system. Score updates live in the header.
On the Machine Room tab, hit Simulate Load Spike, then keep every rack under the 83 °C throttle point until the sim clock passes 1:00 — using pump speed, setpoint, boost, and per-rack throttling. No shortcuts: temps move at thermal speed.
Inject a fan failure or coolant leak and bring the floor back to NOMINAL without any rack shutting down (95 °C).
In the Training Lab, get loss under 1.50 and save a checkpoint. Then, if you're feeling brave, find the learning rate where it diverges — once. Knowing where the cliff is matters.
Run any Agent Loop scenario to completion — including recovering from the injected tool failure. An agent that can't handle a failed tool call isn't an agent, it's a demo.
| GPU power (B200-class) | ≈ 1,000 W per GPU at full load. 128 GPUs ≈ 128 kW of IT load, ≈ 165–175 kW with CPUs, networking, and storage. If a sim shows ~11 kW per GPU, it's wrong by 10× — always sanity-check watts per GPU. |
|---|---|
| Throttle point | Junction (die) temperature throttles around 83–90 °C on H100/B200-class parts. ~52 °C figures you may see elsewhere are coolant or case temps — a different sensor entirely. |
| Cooling | Production AI halls use chilled water and direct-to-chip liquid cold plates. Liquid nitrogen is for overclocking records, not production. PUE 1.1–1.3 is realistic; 1.09 is best-in-class on a good day. |
| Bus math | Three-phase power: P = V × A × √3 × power factor. 170 kW at 480 V ≈ 230–260 A. Check any dashboard against this before trusting it. |
| Emergency Power Off | EPO kills compute and cooling together. Temperatures coast up briefly on residual heat, then fall slowly. Anything that instantly freezes a rack to room temp is fiction. |
| Interconnect | Training clusters move gradients over InfiniBand / NVLink fabrics (400–800 Gbps per link). The all-reduce sync is why adding GPUs stops scaling linearly. |
Machine Room is a first-order thermal model: each rack's junction temperature chases an equilibrium set by load, coolant temperature, and flow, with a time constant — which is why recovery takes sim-minutes, not a click. Training Lab runs a stylized but honest gradient-descent loop: loss really does fall exponentially-ish with steps, really does explode past a learning-rate cliff, and checkpoints really are the only thing standing between you and restarting a multi-day run. Agent Loop is the ReAct pattern used by real agent frameworks, with the part demos usually hide: tools fail, and the loop has to notice. Next-Word is production sampling math, unmodified. SIREN is the actual activation and derivative from the Sitzmann et al. paper, driving a real oscillator.
Play order if you're new: Next-Word (5 min) → Training Lab (Mission 3) → Machine Room (Missions 1–2) → Agent Loop (Mission 4) → SIREN whenever your ears are ready.