Hugging Face's logo

sshleifer
/
distilbart-cnn-12-6

Summarization
Transformers
PyTorch
JAX
Rust
English
bart
text2text-generation
Inference Endpoints
Model card Files Community
20
distilbart-cnn-12-6 / README.md
autoevaluator's picture
autoevaluator HF staff
Add evaluation results on the samsum config and test split of samsum
8ee0cc2
preview code
|
raw
history blame
3.98 kB
metadata
language: en
license: apache-2.0
tags:
  - summarization
datasets:
  - cnn_dailymail
  - xsum
thumbnail: https://huggingface.co/front/thumbnails/distilbart_medium.png
model-index:
  - name: sshleifer/distilbart-cnn-12-6
    results:
      - task:
          type: summarization
          name: Summarization
        dataset:
          name: samsum
          type: samsum
          config: samsum
          split: test
        metrics:
          - type: rouge
            value: 29.1424
            name: ROUGE-1
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOGFlMzY0OTJmOGU3ZTMyM2IwNmYxZjQwMDZiMzZmN2YwNmRkM2VmOTRmYzZjNmNhZjhjNTMxMjExMTUxNTEwOSIsInZlcnNpb24iOjF9.V-HAE31Uq09ebIHK43uC2XCaPJCRHF7BhIMY5fwYITijp_kKxFCCmlNWp1vwwBOhq-6Mu9N1WfasLgWuoZCXCA
          - type: rouge
            value: 10.2812
            name: ROUGE-2
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzU5NTZmZjI0OTA5N2EyNTU1NjI2YTk1YTczN2M2ZjMwZTMxNGNkOGNiMzk0Njc0MzAzZGIwYzIzZGY1OWYwYyIsInZlcnNpb24iOjF9.XR0a42MvfG3D4Zdth74UBSARMIxhnLNntWOoh-JAJ3fIuGDNF6sn2Nfpy_Gu4wsq3dCcCrhgjROLQuwiFT4MBg
          - type: rouge
            value: 21.8105
            name: ROUGE-L
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2EzNzNiYzcxMjIzMDgxMmFkYTAxNjFlMDgyYzg5OTU3NGM1YjkwZjZmNTRhYjA2YmI5ODU1OWM4OWFlYjk1ZCIsInZlcnNpb24iOjF9.okc2azHN0tKAeTMd88UxuLoNQk1oJFa1mu9lJ9tt7GHZiMHXrV6Mbr_7hVq1t6XctWnQsqa6OS5GeCxpYRqiAA
          - type: rouge
            value: 26.9236
            name: ROUGE-LSUM
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMmU4ZjVlM2NkNzhlMjg4Nzg4MGI3YTMzNjJjYjM1N2ZhZmQyYzEzZGU0ZjUzZWRjMjdkNDcyMjE3YmE1NTQyNiIsInZlcnNpb24iOjF9.A3jPSOgtSvTfB1mkXjL-lKCREWu5vEovEVfFpK_nIj6bSmCkr80mvTDxT5k1Pz9QSo1hw-3bu_68GFx1b98_Cg
          - type: loss
            value: 3.155787706375122
            name: loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZTIwNjU0ZTZhZmE5NDBmN2ViNTFjYjI2NmJjN2UxZjdmZDlkNjlkNWUzZDI5MzQ0NTAwOWNhMWEzZTdhYmRkYSIsInZlcnNpb24iOjF9.Uq9A9C0oks9ryKPlde3vRUFf1cqsPY6jLsYxnuf5WE4vvtwMmzFgh9nrCLvQJr5pDBcEyoC4r5ES4LChVV7yDw
          - type: gen_len
            value: 69.6838
            name: gen_len
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiN2UwOTVmYjcxZTBmMGUxNGZjNzE4ZTYxYmY2ZTczMTU2ZWI1ZjU1MWEwNDNhODJjOWVkMGU1NTRiZjY0NGQ5MCIsInZlcnNpb24iOjF9.aIoYZIk_fI5URZ86zIgNdFQKq0pBnTQSGi3-RnNvqTu8ZDjKkJTrt7pJxT6a3l0wGmxni4SaeS9n74NvrQwlBA

Usage

This checkpoint should be loaded into BartForConditionalGeneration.from_pretrained. See the BART docs for more information.

Metrics for DistilBART models

Model Name MM Params Inference Time (MS) Speedup Rouge 2 Rouge-L
distilbart-xsum-12-1 222 90 2.54 18.31 33.37
distilbart-xsum-6-6 230 132 1.73 20.92 35.73
distilbart-xsum-12-3 255 106 2.16 21.37 36.39
distilbart-xsum-9-6 268 136 1.68 21.72 36.61
bart-large-xsum (baseline) 406 229 1 21.85 36.50
distilbart-xsum-12-6 306 137 1.68 22.12 36.99
bart-large-cnn (baseline) 406 381 1 21.06 30.63
distilbart-12-3-cnn 255 214 1.78 20.57 30.00
distilbart-12-6-cnn 306 307 1.24 21.26 30.59
distilbart-6-6-cnn 230 182 2.09 20.17 29.70