GPT J 6B is an open-source language model by EleutherAI. Features: 6b LLM, VRAM: 24.2GB, License: apache-2.0, LLM Explorer Score: 0.15, Arc: 41.4, HellaSwag: 67.5, MMLU: 26.8, GSM8K: 3.
Autoregressive, Transformer, Causal Language Model
Use Cases
Areas:
Research
Primary Use Cases:
Generating text from prompts
Limitations:
Not suitable for use without fine-tuning and supervision for downstream tasks., Not fit for human-facing interactions without moderation.
Considerations:
Users should evaluate risks with using the model in particular contexts.
Additional Notes
The model requires fine-tuning to improve its use in specific applications such as chatbots.
Supported Languages
English (High)
Training Details
Data Sources:
EleutherAI's the Pile dataset
Methodology:
Autoregressive language modeling with cross-entropy loss.
Context Length:
2048
Hardware Used:
TPU v3-256 pod
Model Architecture:
The model consists of 28 layers. Each layer has one feedforward block and one self-attention block. The model has a dimension of 4096 and a feedforward dimension of 16384.
Responsible Ai Considerations
Fairness:
The Pile dataset, used for training, contains biases which might reflect in the model's output.
Transparency:
The model is intended for research and transparency in experimental settings.
Accountability:
Users must supervise the use of the model to ensure suitable output.
Mitigation Strategies:
It is recommended to have human curation or filtering of outputs before deployment.
Input Output
Input Format:
Text
Accepted Modalities:
Text
Output Format:
Generated Text
Performance Tips:
Ensure monitoring for quality and undesired outputs.