Model Type | auto-regressive, transformer |
|
Use Cases |
Primary Use Cases: | Research on large language models including question answering, natural language understanding, evaluating biases, toxic and harmful content generations |
|
Limitations: | Model is foundational and should not be used on downstream applications without further risk evaluation and mitigation. |
|
|
Supported Languages | en (primarily, with support for bg, ca, cs, da, de, es, fr, hr, hu, it, nl, pl, pt, ro, ru, sl, sr, sv, uk) |
|
Training Details |
Data Sources: | CCNet, C4, GitHub, Wikipedia, Books, ArXiv, Stack Exchange |
|
Model Architecture: | Transformer-based auto-regressive |
|
|
Safety Evaluation |
Ethical Considerations: | Data used to train the model could be offensive, harmful, and biased as it was collected from the web. |
|
|
Responsible Ai Considerations |
Fairness: | Evaluated on RAI datasets for biases concerning gender, religion, race, sexual orientation, age, nationality, disability, physical appearance and socio-economic status. |
|
Mitigation Strategies: | Filtered data from the Web using a Kneser-Ney language model and a fastText linear classifier. |
|
|