ReadPaper Blog
A Sovereign, Open-Source Foundation Model for German and English
The paper introduces Soofi S 30B-A3B, a sovereign, open-source German-English foundation model designed to combine strong benchmark performance, auditable training transparency, and efficient long-context serving. Its approach uses a hybrid Mamba-Transformer Mixture-of-Experts architecture, a deliberately German-weighted pretraining curriculum of roughly 27 trillion tokens, and a release plan that includes weights, checkpoints, data accounting, hyperparameters, and training and evaluation code.
Source: A Sovereign, Open-Source Foundation Model for German and English
Why Build Soofi S?
The paper frames Soofi S as a response to three practical gaps in current open foundation models: incomplete openness, weak German coverage, and inefficient long-context deployment. It argues that many nominally open releases are weight-only artifacts that omit the data mixtures, recipes, and training decisions needed for reproduction or audit. It also identifies German as underrepresented in general-purpose multilingual models, especially relative to the language's economic and scientific importance. The deployment problem is technical as well as linguistic: in high-concurrency generation, memory bandwidth and Transformer key-value cache growth can dominate cost as context length increases. Soofi S 30B-A3B is therefore presented as a model intended to be open in substance, strong in both German and English, and structurally efficient for long-context serving.
The Three Gaps
The introduction makes clear that Soofi S is not only a capability project but also a reproducibility project. The authors contrast their release plan with reports that disclose little beyond aggregate token counts, emphasizing full per-source and per-language token accounting, tokenizer-exact consumed-token statistics, and reproducible construction scripts where licenses permit. The paper also states that the German:English:code mixing ratio and its rationale are part of the documented recipe, reflecting a deliberate attempt to correct English-centric data allocation. On the systems side, the paper explains that dense full-attention models incur rising cache costs because every decoded token must revisit both model weights and accumulated attention state. This motivates an architecture whose per-sequence state remains small and near-constant even as contexts extend into tens or hundreds of thousands of tokens.
The Core Trick
Section 2 describes Soofi S 30B-A3B Base as adopting the Nemotron 3 Nano reference architecture rather than inventing a bespoke model family. The resulting network has 52 layers: 23 Mamba-2 sequence-mixing layers, 23 granular Mixture-of-Experts layers with shared experts, and 6 Grouped-Query Attention layers distributed through depth. Although the model totals about 31.6 billion parameters, only about 3.2 billion are active per token, or about 3.6 billion including embeddings. The Mamba-2 backbone carries most sequence mixing with a fixed-size recurrent state, while only the six GQA layers maintain a KV cache. The paper argues that this design gives Soofi S the capacity profile of a 30B-class model while keeping inference closer to the active-parameter and cache footprint of a much smaller model.
What The Paper Shows
The empirical claims combine training scale, benchmark comparisons, and serving measurements. The paper reports pretraining on approximately 26.68 to 27 trillion tokens with German deliberately up-weighted, using a three-phase German-English data curriculum and a Warmup-Stable-Decay optimization schedule. In evaluation against 16 open models of comparable or larger active size, Soofi S is reported to match dense 14B to 27B models on aggregate English and German benchmarks. The abstract further states that Soofi S achieves the best code aggregates in both languages among 17 open base models and obtains the highest English and German evaluation scores among fully open models in the comparison, ahead of OLMo 3 32B and Apertus 70B. For long-context serving, Figure 1 reports that aggregate decode throughput per GPU remains essentially flat from 4K to 256K context and reaches about 8-9 times the throughput of dense 14B to 24B models at 40K context and batch 32.
Why It Matters
The paper's broader implication is that sovereignty, openness, and production efficiency can be treated as linked design requirements rather than trade-offs. Soofi S was trained end-to-end on the German Industrial AI Cloud, an HPC-scale AI infrastructure operated by Deutsche Telekom in Munich and funded through the German Federal Ministry for Economic Affairs and Energy. The release plan includes model weights, selected intermediate checkpoints, hyperparameters, training code, evaluation code, and detailed data accounting, with data-construction artifacts released under permissive licenses where source licenses allow. The authors position this as an audit-friendly alternative to weight-only openness and as a European sovereign baseline that remains competitive with larger or denser international models. The paper therefore presents Soofi S as both a technical artifact and an infrastructure statement: a German-English foundation model designed for transparent inspection, reproducible study, and efficient long-context deployment.
