open-8b: an auditable model, in training

Today we’re opening the training run of open-8b, an 8-billion-parameter language model, while it is still in progress. You can see the run here. Every step’s state is committed as it happens, and anyone can replay a trained step on their own hardware and compare it with the record.

open-1b was the first model with proof of how it was trained: a public fingerprint of every step, and a tool that lets anyone check those fingerprints on their own machine. It answered the question of whether auditable training is possible at all. It is, on everyday hardware.

open-8b is built to answer the next question: whether an auditable model can be good enough to do real work. It is five times the size of open-1b, trained on two and a half times the tokens, and is being trained to become the judge model for Delphi, our information market platform. This means it will be responsible for settling markets with real customer funds once post-training is done.

Good enough also has to mean affordable. Training open-1b so that every step could be replayed cost 5.5x the GPU-hours of a standard run. open-8b is training at about 1.3x.

From possible to useful

When open-1b launched, we described model transparency as a ladder. Open weights hand you the cake. An open recipe hands you the cake and the recipe card. An auditable model like open-1b hands you the cake, the recipe, and invites you into the kitchen.

A model you can verify is still theoretical until it is doing a job where verification matters. For us, that use case is in Delphi. Information markets need a judge: something that reads the evidence and decides what happened. People who lose money to a judge's ruling will want to know what the judge is, how it was trained, and whether anything was slipped into it. A closed model cannot answer that. An auditable one can, from the first training step through to the ruling.

We’re training open-8b because we want to use it, rather than to prove a point.

There are countless use cases where organizations are trusting closed models with operations involving money or sensitive information. Delphi will be the first to use an auditable model for this type of operation, but it can’t be the last.

What you can watch

The run page at open.gensyn.ai shows open-8b's pretraining as the trainer reports it, live. The model is a 32-layer Llama-3-style decoder, 8B parameters, trained natively in int8 (W8A8) on 48 GPUs, with a budget of 1T tokens over about 201,756 steps. A checkpoint is written every 500 steps.

A public run surfaces things as they happen. Around 300B tokens, for instance, the model's multiple-choice scores on the base benchmarks caught up with its cloze scores, the kind of moment we could normally only share in a paper months later.

The same stack that made open-1b reproducible trains open-8b. RepOps makes every operation land on the same bits on a consumer NVIDIA card, an x86 or ARM CPU, or an Apple M processor as it does on the cluster. The data is a canonical stream that depends only on the seed and the corpus, and the collectives are deterministic, so a single device can replay what 48 GPUs did. The details are in the open-1b post, with many performance optimizations landing for the latest training run.

An audit is still simple: pick a step, load the checkpoint before it, replay the step, hash the result, compare. The difference is that the step you pick may have been trained this morning.

The cost of auditability, one model later

When we launched open-1b we were transparent about the cost: about 5% model FLOPs utilisation, and 5.5× the GPU-hours of the same recipe in PyTorch at bf16. We also said we could scale it.

Before open-8b started, we trained the same 8B architecture for 100B tokens in plain PyTorch at bf16, as a control. That run sustained 32% MFU. open-8b sustains 24% over wall-clock time. For the same tokens, that is about 1.3× the GPU-hours of a standard run. At 8B shapes, that takes auditability from 8.5 months of GPU time to under two.

Additionally, open-1b trained natively in int8, and its loss drifted above the bf16 baseline as training went on. Switching from per-tensor to per-token quantisation, with stochastic rounding, closed that gap: reproducible int8 training now tracks the bf16 trajectory almost exactly. open-8b trains with weights and activations in int8 from the first step, with no measured loss of quality against the bf16 control on perplexity or the OLMES suite.

Int8 pretraining that is bitwise reproducible across hardware and matches bf16 quality is not something anyone had shown before, and it is what makes an 8B step replayable on a laptop.

Most of the speed-up came from a few weeks of kernel work done in agentic optimisation loops: point an agent at a measured bottleneck, let it iterate until it hits a target speed-up, and hold it to hard constraints, including never breaking the numerical contract.

Auditable training still costs more than a standard run, but we don't think 30% is the floor: the kernels have had the attention, the training loop hasn't yet. The gap is already small enough, though, that the question for any lab is no longer whether it can afford to train auditably. It is whether it can afford not to.

Audit it yourself

Open the board at open.gensyn.ai, claim a step and replay it: about 2 hours on an H100, or 13 on a 48 GB MacBook Pro, with a 96 GB checkpoint to download first. As with open-1b, auditing is voluntary. There are no payments, no tokens, no points and no future rewards. The reward is your name on the public record of a model that is headed for real work, and the knowledge that when it rules on a market, you were one of the people who verified the record.

What’s next

open-1b showed that a lab can say how a model was made and let the world check. open-8b is meant to show that such a model can be good enough to use.

Check out the training run now, and audit it if you want to verify.