| Model Type | | auto-regressive language model, transformer architecture |
|
| Use Cases |
| Areas: | | research on large language models |
|
| Applications: | | question answering, natural language understanding, reading comprehension |
|
| Primary Use Cases: | | understanding capabilities and limitations of current language models, developing improvement techniques |
|
| Limitations: | | not intended for downstream applications without risk evaluation |
|
| Considerations: | | Model can produce harmful content or incorrect information. |
|
|
| Supported Languages | | en (>20 languages including English, but primarily trained on English) |
|
| Training Details |
| Data Sources: | | CCNet (67%), C4 (15%), GitHub (4.5%), Wikipedia (4.5%), Books (4.5%), ArXiv (2.5%), Stack Exchange (2%) |
|
|
| Responsible Ai Considerations |
| Fairness: | | The model might exhibit biases related to gender, religion, race, sexual orientation, age, nationality, disability, and physical appearance, reflecting biases from the training data. |
|
| Transparency: | | The model's performance and evaluation metrics are detailed in published resources. |
|
| Accountability: | | Meta AI's FAIR team is responsible for the model's outputs. |
|
| Mitigation Strategies: | | Data filtering was performed using a Kneser-Ney language model and a fastText linear classifier to remove offensive content. |
|
|
| Release Notes |
| Version: | |
| Date: | |
| Notes: | | Initial release of the model. |
|
|
|