Phi 2 is an open-source language model by microsoft. Features: 2.8b LLM, VRAM: 5.6GB, Context: 2K, License: mit, LLM Explorer Score: 0.32, Arc: 61, HellaSwag: 74.9, MMLU: 57.9, GSM8K: 55.
Generate Inaccurate Code and Facts, Limited Scope for code, Unreliable Responses to Instruction, Language Limitations, Potential Societal Biases, Toxicity, Verbosity
Additional Notes
Phi-2 is intended for QA, chat, and code purposes. Model-generated text/code should be treated as a starting point. Users should be cautious when employing these models in applications.
Supported Languages
en (Standard English)
Training Details
Data Sources:
Phi-1.5, NLP synthetic texts, filtered websites
Data Volume:
250B tokens
Methodology:
Transformer-based model with next-word prediction objective
Context Length:
2048
Training Time:
14 days
Hardware Used:
96xA100-80G GPUs
Model Architecture:
Transformer-based model with next-word prediction objective
Input Output
Accepted Modalities:
text
Performance Tips:
Phi-2 has an attention overflow issue (with FP16). If encountering this issue, enable/disable autocast on the PhiAttention.forward() function.