ALMA 13B Pretrain is an open-source language model by haoranxu. Features: 13b LLM, VRAM: 52.1GB, Context: 4K, License: mit, LLM Explorer Score: 0.14, Arc: 56.9, HellaSwag: 80.2, MMLU: 50.3, GSM8K: 8.9.
ALMA 13B Pretrain Parameters and Internals
Model Type Use Cases
Areas: Applications: Primary Use Cases: Translating Chinese to English
Considerations: For usage with LoRA models.
Additional Notes ALMA-R incorporates Contrastive Preference Optimization (CPO) for improved performance.
Supported Languages Monolingual Data (Multi-lingual), Parallel Data (High-quality parallel data)
Training Details
Data Sources: 20B monolingual tokens, high-quality parallel data, triplet preference data
Data Volume: 20B tokens for 7B model and 12B tokens for 13B model
Methodology: Two-step fine-tuning with monolingual data followed by parallel data; Further optimized with Contrastive Preference Optimization (CPO)
Context Length: Model Architecture: LLaMA based, fine-tuned with LoRA
Input Output
Input Format: Accepted Modalities: Output Format: Performance Tips: Use LoRA models together with Base Models for intended performance.
Release Notes
Version: Date: Notes: Full-weight Fine-tune LLaMA-2-7B on 12B monolingual tokens and then LoRA fine-tune on human-written parallel data
Version: Date: Notes: Further LoRA fine-tuning upon ALMA-13B-LoRA with contrastive preference optimization
Rank the ALMA 13B Pretrain capabilities
Have you tried this model? Rate its performance — this feedback helps the ML community find the right model for their needs.
Instruction Following and Task Automation
Factuality and Completeness of Knowledge
Censorship and Alignment
Data Analysis and Insight Generation
Text Generation
Text Summarization and Feature Extraction
Code Generation
Multi-Language Support and Translation
Quantized Models of the ALMA 13B Pretrain
Best Alternatives to ALMA 13B Pretrain
Expand