text
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
new_line
new_line
text
Paper: 
Paper: 
resource
https://huggingface.co/papers/2404.03323
paper
2404.03323
https://huggingface.co/papers/2404.03323
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning (2404.03323)
new_line
new_line
text
Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models. 
Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models. 
new_line
new_line
text
Key concepts:
Key concepts:
new_line
text
– Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations. 
– Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations. 
new_line
text
– Sparse $\ell_1$ regularization.
– Sparse $\ell_1$ regularization.
new_line
text
– Contrastive loss for inner layers of the model.
– Contrastive loss for inner layers of the model.
new_line
new_line
text
Methodology:
Methodology:
new_line
text
The approach consists of three main steps:
The approach consists of three main steps:
new_line
text
– Create a set of concepts based on the labels of the dataset.
– Create a set of concepts based on the labels of the dataset.
new_line
text
– Supply a multi-modal encoder with CBL.
– Supply a multi-modal encoder with CBL.
new_line
text
– Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.
– Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.
new_line
new_line
text
Results and Analysis:
Results and Analysis:
new_line
text
The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.
The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.
new_line
new_line
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning

Paper: https://huggingface.co/papers/2404.03323

Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models. 

Key concepts:
– Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations. 
– Sparse $\ell_1$ regularization.
– Contrastive loss for inner layers of the model.

Methodology:
The approach consists of three main steps:
– Create a set of concepts based on the labels of the dataset.
– Supply a multi-modal encoder with CBL.
– Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.

Results and Analysis:
The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.


🚀
Andron00e tit33 EffyOsvin
👀
Andron00e EffyOsvin
🔥
EffyOsvin