MobileLLM 1B is an open-source language model by facebook. Features: 1b LLM, VRAM: 2.1GB, Context: 2K, License: cc-by-nc-4.0, LLM Explorer Score: 0.18.
MobileLLM integrated several key techniques such as SwiGLU activation function, deep and thin architectures, embedding sharing, and grouped-query attention.
Training Details
Data Sources:
Publicly available online data.
Data Volume:
1T tokens
Context Length:
2000
Hardware Used:
32 NVIDIA A100 80G GPUs
Model Architecture:
MobileLLM is an auto-regressive language model leveraging an optimized transformer architecture with techniques such as SwiGLU activation function, deep and thin architectures, embedding sharing, and grouped-query attention.