LLM EXPLORER 60,645 MODELS INDEXED

Qwen2 72B Instruct FP8 by neuralmagic

By neuralmagic · 524 downloads
HF Repo ↗ Copy URL

Qwen2 72B Instruct FP8 is an open-source language model by neuralmagic. Features: 72b LLM, VRAM: 75.5GB, Context: 32K, License: other, Instruction-Based, LLM Explorer Score: 0.13.

  Autotrain compatible   Conversational   Endpoints compatible   Fp8   Instruct   Qwen2   Region:us   Safetensors   Sharded   Tensorflow   Vllm
Model Card on HF 🤗: https://huggingface.co/neuralmagic/Qwen2-72B-Instruct-FP8 

Qwen2 72B Instruct FP8 Parameters and Internals

Model Type 
Assistant-like chat model
Use Cases 
Areas:
Commercial and Research Use
Applications:
Assistant-like chat
Primary Use Cases:
Assistant-like chat
Limitations:
Use in languages other than English, Use in any manner that violates applicable laws or regulations (including trade compliance laws)
Additional Notes 
This model has been optimized using FP8 quantization, reducing memory requirements while maintaining performance. The quantization method was achieved using AutoFP8 and calibration samples from ultrachat.
Supported Languages 
English (Intended for commercial and research use)
Input Output 
Input Format:
Text
Accepted Modalities:
Text
Output Format:
Text
Release Notes 
Version:
1.0
Date:
6/8/2024
Notes:
Model quantized to FP8 for improved efficiency with vLLM.
LLM NameQwen2 72B Instruct FP8
Repository 🤗https://huggingface.co/neuralmagic/Qwen2-72B-Instruct-FP8 
Model Size72b
Required VRAM75.5 GB
Updated2025-04-08
Maintainerneuralmagic
Model Typeqwen2
Instruction-BasedYes
Model Files  4.9 GB: 1-of-16   4.8 GB: 2-of-16   4.9 GB: 3-of-16   4.8 GB: 4-of-16   4.9 GB: 5-of-16   4.8 GB: 6-of-16   4.9 GB: 7-of-16   4.8 GB: 8-of-16   4.9 GB: 9-of-16   4.8 GB: 10-of-16   4.9 GB: 11-of-16   4.8 GB: 12-of-16   4.9 GB: 13-of-16   4.8 GB: 14-of-16   4.9 GB: 15-of-16   2.7 GB: 16-of-16
Model ArchitectureQwen2ForCausalLM
Licenseother
Context Length32768
Model Max Length32768
Transformers Version4.41.2
Tokenizer ClassQwen2Tokenizer
Padding Token<|endoftext|>
Vocabulary Size152064
Torch Data Typebfloat16
Errorsreplace

Quantized Models of the Qwen2 72B Instruct FP8

Model
Likes
Downloads
VRAM
Qwen2 72B Instruct 8bit14877 GB

Best Alternatives to Qwen2 72B Instruct FP8

Best Alternatives
Context / RAM
Downloads
Likes
EVA Qwen2.5 72B V0.2128K / 146 GB15922
AceInstruct 72B128K / 146 GB8818
...n2.5 72B 2x Instruct TIES V1.0128K / 146.1 GB801
...2B PT BR Instruct Experimental128K / 146 GB200
... BI Qwen2.5 72B PT BR Instruct128K / 146 GB50
...A Abliterated TIES Qwen2.5 72B128K / 146.1 GB81
72B Qwen2.5 Kunou V1128K / 146 GB7726
EVA Qwen2.5 72B V0.1128K / 146 GB1719
EVA Qwen2.5 72B V0.0128K / 146 GB105
Qwen2 72B Instruct32K / 145.5 GB61318718
Note: green Score (e.g. "73.2") means that the model is better than neuralmagic/Qwen2-72B-Instruct-FP8.