If there’s one concept to take from this article, it is that this. Start right here except there is a selected, measured motive to not. That is the place most groups should start their testing. Quantisation is what makes self-hosted LLMs practical for most groups. S. The M variants use a extra sophisticated quantisation scheme that preserves high quality at negligible dimension cost. A sensible word: do not benchmark quantisation on generic benchmarks alone. A Q4 model might rating identically to Q8 on normal benchmarks however produce noticeably worse output for a particular area, or it may be completely indistinguishable. The 70B competes with earlier frontier models on many benchmarks. DeepSeek R1 at 671B parameters, with round 37B energetic via MoE, competes with the best proprietary fashions on reasoning tasks. Fine-tuning on proprietary information.
Capable enough for general duties, quick enough for actual-time use, small sufficient to run on a single shopper GPU.