@missedminutes

Key points

  • 280B MoE with only 14B active per token, pretrained from scratch on 33.5T tokens
  • Pure base model with no mid- or post-training, enabling domain-specific adaptation
  • Only 6 of 43 layers keep token-level access to full context; other 37 squeeze every 128 tokens into a single cached entry, keeping KV cache small while supporting up to 1M tokens for fine-tuning and inference
  • Multi-Router MoE: multiple routers select experts together without increasing activated expert compute or communication costs
  • ECHO: multi-token prediction runs through the model's own final layer, eliminating need for separate MTP module
  • Early evals show strong results: MMLU-Pro 78.4 vs DeepSeek-V4-Flash-Base's 69.2; HumanEval+ 82.3 vs 69.5; LiveCodeBench 40.1 vs 31.9

First posted on the Missed Minutes channel.