What are the hardware requirements for ReluFalcon 40B?

ReluFalcon 40B requires approximately 167.1 GB of VRAM. Quantized variants may run on less VRAM; see the Quantized Models section on this page.

Who developed ReluFalcon 40B and how large is it?

ReluFalcon 40B is developed by SparseLLM, a model with 40b parameters. The model is published as open weights on Hugging Face and indexed on LLM Explorer with full benchmark history.

Where can I download or evaluate ReluFalcon 40B?

ReluFalcon 40B is hosted on Hugging Face and linked from this page. LLM Explorer also lists quantized variants and similar alternatives if available.

ReluFalcon 40B by SparseLLM — VRAM 167.1GB

Name: ReluFalcon 40B
Author: SparseLLM

ReluFalcon 40B is an open-source language model by SparseLLM. Features: 40b LLM, VRAM: 167.1GB, License: apache-2.0, LLM Explorer Score: 0.09.

Arxiv:2110.01786 Arxiv:2310.04564 Custom code En Endpoints compatible Falcon Pytorch Region:us Sharded

Model Card on HF 🤗: https://huggingface.co/SparseLLM/ReluFalcon-40B

ReluFalcon 40B Parameters and Internals

Model Type

Sparse Computation, Sparse Large Language Model, ReLU Activation

Use Cases

Areas:

Research, Commercial applications

Limitations:

May produce inaccurate, biased or objectionable responses, Fine-tuning tested primarily in English, unknown results in other languages

Considerations:

Developers should perform safety testing and tuning before deploying applications of the model.

Additional Notes

This is a fine-tuned version using ReLU activation to enhance sparse LLM efficiency.

Supported Languages

English (tested)

Training Details

Data Sources:

Wikipedia, Pile, StackOverflow

Data Volume:

5 billion tokens

Methodology:

Joint optimization on language modeling and knowledge distillation objectives. ReLU activation applied through fine-tuning techniques.

Hardware Used:

40x NVIDIA A100 GPUs with 80GB memory

Input Output

Accepted Modalities:

text

LLM Name	ReluFalcon 40B
Repository 🤗	https://huggingface.co/SparseLLM/ReluFalcon-40B
Model Size	40b
Required VRAM	167.1 GB
Updated	2026-08-07
Maintainer	SparseLLM
Model Type	falcon
Model Files	9.2 GB: 1-of-18 9.8 GB: 2-of-18 9.2 GB: 3-of-18 9.8 GB: 4-of-18 9.2 GB: 5-of-18 9.8 GB: 6-of-18 9.2 GB: 7-of-18 9.8 GB: 8-of-18 9.2 GB: 9-of-18 9.8 GB: 10-of-18 9.2 GB: 11-of-18 9.8 GB: 12-of-18 9.2 GB: 13-of-18 9.8 GB: 14-of-18 9.2 GB: 15-of-18 9.8 GB: 16-of-18 9.2 GB: 17-of-18 5.9 GB: 18-of-18
Supported Languages	en
Model Architecture	FalconForCausalLM
License	apache-2.0
Model Max Length	1024
Transformers Version	4.33.2
Is Biased	0
Tokenizer Class	PreTrainedTokenizerFast
Vocabulary Size	65024
Torch Data Type	float32

Rank the ReluFalcon 40B capabilities

Have you tried this model? Rate its performance — this feedback helps the ML community find the right model for their needs.

Instruction Following and Task Automation
Factuality and Completeness of Knowledge
Censorship and Alignment
Data Analysis and Insight Generation
Text Generation
Text Summarization and Feature Extraction
Code Generation
Multi-Language Support and Translation

Best Alternatives to ReluFalcon 40B

Best Alternatives	Context / RAM	Downloads	Likes
... Falcon 40B Instruct 4 Bit Bnb	2K / 23.9 GB	8	0
Tiny Random Falcon 40B	2K / 0 GB	368	0
Openbuddy Falcon 40B V16.1 4K	2K / 82.6 GB	82	1
Falcon 40B	0K / 83.6 GB	26343	2437
Falcon 40B Instruct	0K / 83.6 GB	5569	1176
Tiny Random Falcon 40B	0K / 0.2 GB	700	0
Falcon 40B Megacode2	0K / 82.5 GB	5	1
...lcon 40B Ft Alpaca Dolly Dutch	0K / 82.5 GB	40	4
...slessMegaCoder Falcon 40B Mini	0K / 82.5 GB	122	2
Falcon 40B Megacode2 Oasst	0K / 82.5 GB	11	6

Note: green Score (e.g. "73.2") means that the model is better than SparseLLM/ReluFalcon-40B.