LLM EXPLORER 60,783 MODELS INDEXED

Llama 2 70B Longlora 32K by Yukang

By Yukang · 6 downloads
HF Repo ↗ Copy URL

Llama 2 70B Longlora 32K is an open-source language model by Yukang. Features: 70b LLM, VRAM: 0.1GB, LLM Explorer Score: 0.09.

  Arxiv:2309.12307   Endpoints compatible   Llama   Lora   Region:us
Model Card on HF 🤗: https://huggingface.co/Yukang/Llama-2-70b-longlora-32k 

Llama 2 70B Longlora 32K Parameters and Internals

Model Type 
Language Model, Text Generation
Additional Notes 
LongLoRA extends models' context while retaining their original architectures, and is compatible with most existing techniques.
Training Details 
Data Sources:
LongQA dataset
Methodology:
Improved fine-tuning using sparse local attention and LoRA for context extension
Hardware Used:
8x A100 machine
Model Architecture:
Retains original architecture, compatible with techniques like FlashAttention-2
Release Notes 
Version:
1.0
Date:
2023
Notes:
Efficient fine-tuning approach using sparse local attention and improved LoRA.
LLM NameLlama 2 70B Longlora 32K
Repository 🤗https://huggingface.co/Yukang/Llama-2-70b-longlora-32k 
Model Size70b
Required VRAM0.1 GB
Updated2026-07-03
MaintainerYukang
Model Files  0.1 GB   1.1 GB
Model ArchitectureAutoModelForCausalLM
Is Biasednone
PEFT TypeLORA
LoRA ModelYes
PEFT Target Modulesq_proj|k_proj|v_proj|o_proj
LoRA Alpha16
LoRA Dropout0
R Param8

Best Alternatives to Llama 2 70B Longlora 32K

Best Alternatives
Context / RAM
Downloads
Likes
Llama 3.1 Tango 70B0K / 3 GB68
LLama3 70B SWE LLM0K / 16.1 GB31
Limarpv3 Llama2 70B Qlora0K / 1.7 GB18112
Limarpv3 Llama2 70B Qlora0K / 1.7 GB162
Llama 2 70B Chat Longlora 32K0K / 0.1 GB69
Nous Hermes Llama2 70B0K / 138 GB14484
... 70M Instruct Orca Chkpt 640000K / 0.2 GB81
NorskGPT Llama 3 70B Adapter0K / 0.2 GB67
Llama 3 70B Tagengo0K / 141.9 GB31
...3 70B Instruct Uncensored Lora0K / 0.8 GB83
Note: green Score (e.g. "73.2") means that the model is better than Yukang/Llama-2-70b-longlora-32k.