How they use Nebius

Pleias pretrained its Nano 1.2B preview model on TractoAI using Nebius AI’s ISEG GPU cluster. Its model card reports 192 H100 GPUs over five days. The training code used TractoAI’s fork of Nanotron. The training schedule comprised 518,000 steps with a batch size of 1,024, covering nearly five trillion tokens over three epochs. Pleias estimated training emissions at four tonnes CO2eq. Pleias also continuously pretrained its Nano RAG variant on TractoAI using the ISEG cluster, with a retrieval dataset containing about 45 billion tokens.

Pleias converted tokenized training data into TractoAI tables, built a Nanotron dataset adapter and adapted file operations, including checkpoint saving. It used tractorun to deploy and coordinate distributed training automatically with fault tolerance. Pleias also generated synthetic RAG and instruction training data with larger models through a Map-Reduce based TractoAI method.

For Common Corpus, Pleias credited Nebius AI and Tracto AI among the organizations supporting storage and processing. Its release article describes a multilingual training dataset containing more than two trillion permissibly licensed tokens. For Common Corpus 2, Pleias processed 11 million scientific articles from OpenAlex, representing about 175 billion tokens, using PDF parsing models running on Nebius.