LLM EXPLORER
 LLM Explorer: A Curated Large Language Model Directory and Analytics
Dark Theme
  LLM  List    LLM Hosting    LLM Leaderboards    Blog    Newsfeed    Advertise

Mpt 1B Redpajama 200B by anas-awadalla

 »  All LLMs  »  anas-awadalla  »  Mpt 1B Redpajama 200B   URL Share it on

  2956   
  Arxiv:2108.12409   Arxiv:2205.14135   Arxiv:2302.13971   Autotrain compatible   Custom code Dataset:togethercomputer/redpa...   Mosaic gpt   Pytorch   Region:us
Model Card on HF 🤗: https://huggingface.co/anas-awadalla/mpt-1b-redpajama-200b 

Mpt 1B Redpajama 200B Benchmarks

ARC: 25.77 vs 96.7 (so35)-73.4%
HellaSwag: 26.08 vs 95.3 (gpt4)-72.6%
MMLU: 24.5 vs 88.3 (so35)-72.3%
TruthfulQA: 47.57 vs 59 (gpt4)-19.4%
WinoGrande: 50.36 vs 87.5 (gpt4)-42.4%
LLME Score: 0.15527
nn.n% — How the model compares to the reference models: Anthropic Sonnet 3.5 ("so35"), GPT-4o ("gpt4o") or GPT-4 ("gpt4").
Mpt 1B Redpajama 200B (anas-awadalla/mpt-1b-redpajama-200b)
🌟 Advertise your project 🚀

Mpt 1B Redpajama 200B Parameters and Internals

Model Type 
decoder-only transformer
Additional Notes 
This model requires `trust_remote_code=True` due to using a custom model architecture `MosaicGPT`. Training efficiency features like FlashAttention and ALIBI are included.
Training Details 
Data Sources:
RedPajama Common Crawl, C4, RedPajama GitHub, RedPajama Wikipedia, RedPajama Books, RedPajama Arxiv, RedPajama StackExchange
Data Volume:
200B tokens
Context Length:
2048
Training Time:
~ half a day
Hardware Used:
440 A100-40GBs
Model Architecture:
modified decoder-only transformer with 24 layers, 16 attention heads, width 2048, using ALiBi, QK LayerNorm, and no biases.
Input Output 
Performance Tips:
Use `attn_impl='triton'` with `bfloat16` for optimized performance.
LLM NameMpt 1B Redpajama 200B
Repository 🤗https://huggingface.co/anas-awadalla/mpt-1b-redpajama-200b 
Model Size1b
Required VRAM5.2 GB
Updated2025-09-15
Maintaineranas-awadalla
Model Typemosaic_gpt
Model Files  5.2 GB
Model ArchitectureMosaicGPT
Licenseapache-2.0
Model Max Length2048
Transformers Version4.27.4
Tokenizer ClassGPTNeoXTokenizer
Vocabulary Size50432
Torch Data Typefloat32

Best Alternatives to Mpt 1B Redpajama 200B

Best Alternatives
Context / RAM
Downloads
Likes
Mpt 1B Redpajama 200B0K / 5.2 GB25292
Mpt 1B Redpajama 200B Dolly0K / 5.2 GB13177

Rank the Mpt 1B Redpajama 200B Capabilities

🆘 Have you tried this model? Rate its performance. This feedback would greatly assist ML community in identifying the most suitable model for their needs. Your contribution really does make a difference! 🌟

Instruction Following and Task Automation  
Factuality and Completeness of Knowledge  
Censorship and Alignment  
Data Analysis and Insight Generation  
Text Generation  
Text Summarization and Feature Extraction  
Code Generation  
Multi-Language Support and Translation  

What open-source LLMs or SLMs are you in search of? 51369 in total.

Email us: info@extractum.io. Our Privacy Policy  | Terms and Conditions  | Suggest an improvement.
Our Social Media →  
Original data from HuggingFace, OpenCompass and various public git repos.
Release v20241124
  Support LLM Explorer