Pythia 160M is an open-source language model by EleutherAI. Features: 160m LLM, VRAM: 0.4GB, Context: 2K, License: apache-2.0, LLM Explorer Score: 0.22, Arc: 22.8, HellaSwag: 30.3, MMLU: 25, GSM8K: 0.2.
Research on behavior, functionality, limitations of large language models
Primary Use Cases:
Controlled scientific experiments
Limitations:
Not suitable for human-facing interactions, English language-only models, unsuitable for generating text in other languages, Not fine-tuned for genre prose or commercial chatbots
Considerations:
Conduct risk and bias assessment if fine-tuning; evaluate risks before deployment
Additional Notes
Pythia model suite renamed in January 2023 for clarity
Supported Languages
English (Native)
Training Details
Data Sources:
The Pile, 22 diverse sources including arXiv, CommonCrawl, Project Gutenberg, YouTube subtitles, GitHub
Data Volume:
299,892,736,000 tokens
Model Architecture:
GPT-NeoX
Responsible Ai Considerations
Fairness:
Documented biases with regards to gender, religion, and race (as per Pile paper).
Input Output
Input Format:
String of text for next token prediction.
Accepted Modalities:
text
Output Format:
String (one token at a time)
Release Notes
Version:
Current Release
Date:
January 2023
Notes:
Pyhtia-160M retrained to address hyperparameter discrepancies