sigridjineth
/

colbert-small-korean-20241212

PyTorch

Korean

bert

colbert

korean

Model card Files Files and versions Community

sigridjineth commited on 24 days ago

Commit

c687746

•

1 Parent(s): a88f03e

Update README.md

Browse files

Files changed (1) hide show

README.md +11 -11

README.md CHANGED Viewed

@@ -20,17 +20,17 @@ Compared to other ColBERT-based models tested (`colbert-ir/colbertv2.0` and `ans
 ## Model Comparison
 The [AutoRAG Benchmark](https://github.com/Marker-Inc-Korea/AutoRAG-example-korean-embedding-benchmark) serves as both the evaluation dataset and the toolkit for reporting these metrics.
-| Model                                     | top_k | F1     | MRR     | NDCG    | Notes                                                                           |
-|-------------------------------------------|-------|--------|---------|---------|---------------------------------------------------------------------------------|
-| colbert-ir/colbertv2.0                    | 1     | 0.2456 | 0.2456  | 0.2456  | Low initial performance.                                                        |
-|                                           | 3     | 0.3596 | 0.4459  | 0.5158  | Shows notable improvement at top_k=3.                                           |
-|                                           | 5     | 0.3596 | 0.4459  | 0.5158  | Similar to top_k=3, no further MRR/NDCG gains.                                  |
-| answerai/answerai-colbert-small-v1        | 1     | 0.2193 | 0.2193  | 0.2193  | Lower performance at top_k=1.                                                   |
-|                                           | 3     | 0.3596 | 0.4240  | 0.4992  | Improved performance at top_k=3, but MRR/NDCG still behind colbertv2.0.          |
-|                                           | 5     | 0.3596 | 0.4240  | 0.4992  | Same as top_k=3, no additional metrics gain.                                    |
-| sigridjineth/colbert-small-korean-20241212| 1     | 0.3772 | 0.3772  | 0.3772  | Highest F1 at top_k=1 among the three models.                                   |
-|                                           | 3     | 0.3596 | **0.5278** | **0.5769** | Slight F1 drop vs. top_k=1, but MRR/NDCG significantly surpass both competitors. |
-|                                           | 5     | 0.3596 | 0.5278  | 0.5769  | Same as top_k=3, maintaining high MRR/NDCG.                                     |
 ## Usage

 ## Model Comparison
 The [AutoRAG Benchmark](https://github.com/Marker-Inc-Korea/AutoRAG-example-korean-embedding-benchmark) serves as both the evaluation dataset and the toolkit for reporting these metrics.
+| Model                                     | top_k | F1     | MRR     | NDCG    |
+|-------------------------------------------|-------|--------|---------|---------|
+| colbert-ir/colbertv2.0                    | 1     | 0.2456 | 0.2456  | 0.2456  |
+|                                           | 3     | 0.3596 | 0.4459  | 0.5158  |
+|                                           | 5     | 0.3596 | 0.4459  | 0.5158  |
+| answerai/answerai-colbert-small-v1        | 1     | 0.2193 | 0.2193  | 0.2193  |
+|                                           | 3     | 0.3596 | 0.4240  | 0.4992  |
+|                                           | 5     | 0.3596 | 0.4240  | 0.4992  |
+| sigridjineth/colbert-small-korean-20241212| 1     | 0.3772 | 0.3772  | 0.3772  |
+|                                           | 3     | 0.3596 | **0.5278** | **0.5769** |
+|                                           | 5     | 0.3596 | 0.5278  | 0.5769  |                             |
 ## Usage