
Falcon
Overview
Falcon is a class of causal decoder-only models built by TII. The largest Falcon checkpoints
have been trained on >=1T tokens of text, with a particular emphasis on the RefinedWeb
corpus. They are made available under the Apache 2.0 license.
Falcon's architecture is modern and optimized for inference, with multi-query attention and support for efficient
attention variants like FlashAttention. Both 'base' models trained only as causal language models as well as
'instruct' models that have received further fine-tuning are available.
Falcon models are (as of 2023) some of the largest and most powerful open-source language models,
and consistently rank highly in the OpenLLM leaderboard.
Converting custom checkpoints