
# Dataset Card for PubMed

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** : [https://www.nlm.nih.gov/databases/download/pubmed_medline.html]()
- **Documentation:** : [https://www.nlm.nih.gov/databases/download/pubmed_medline_documentation.html]()
- **Repository:**
- **Paper:**
- **Leaderboard:**
- **Point of Contact:** [National Center for Biotechnology Information](mailto:info@ncbi.nlm.nih.gov)

### Dataset Summary

PubMed comprises more than 36 million citations for biomedical literature from MEDLINE, life science journals, and online books. Citations may include links to full-text content from PubMed Central and publisher web sites.

NLM produces a baseline set of PubMed citation records in XML format for download on an annual basis. The annual baseline is released in December of each year.
- Last Updated December 15, 2023

Each day, NLM produces update files that include new, revised, and deleted citations.

Source: https://ftp.ncbi.nlm.nih.gov/pubmed/README.txt

### Supported Tasks and Leaderboards

[More Information Needed]

### Languages

- English

## Dataset Structure

Bear in mind the data comes from XML that have various tags that are hard to reflect
in a concise JSON format. Tags and list are kind of non "natural" to XML documents
leading this library to make some choices regarding data. "Journal" info was dropped
altogether as it would have led to many fields being empty all the time.

The hierarchy is also a bit unnatural but the choice was made to keep as close as
possible to the original data for future releases that may change schema from NLM's side.

Author has been kept and contains either "ForeName", "LastName", "Initials", or "CollectiveName".
(All the fields will be present all the time, but only some will be filled)

### Data Instances

```json
{
    "MedlineCitation": {
        "PMID": 0,
        "DateCompleted": {"Year": 0, "Month": 0, "Day": 0},
        "NumberOfReferences": 0,
        "DateRevised": {"Year": 0, "Month": 0, "Day": 0},
        "Article": {
            "Abstract": {"AbstractText": "Some abstract (can be missing)" },
            "ArticleTitle": "Article title",
            "AuthorList": {"Author": [
                {"FirstName": "John", "ForeName": "Doe", "Initials": "JD", "CollectiveName": ""}
                {"CollectiveName": "The Manhattan Project", "FirstName": "", "ForeName": "", "Initials": ""}
            ]},
            "Language": "en",
            "GrantList": {
                "Grant": [],
            },
            "PublicationTypeList": {"PublicationType": []},
        },
        "MedlineJournalInfo": {"Country": "France"},
        "ChemicalList": {"Chemical": [{
            "RegistryNumber": "XX",
            "NameOfSubstance": "Methanol"
        }]},
        "CitationSubset": "AIM",
        "MeshHeadingList": {
            "MeshHeading": [],
        },
    },
    "PubmedData": {
        "ArticleIdList": {"ArticleId": "10.1002/bjs.1800650203"},
        "PublicationStatus": "ppublish",
        "History": {"PubMedPubDate": [{"Year": 0, "Month": 0, "Day": 0}]},
        "ReferenceList": [{"Citation": "Somejournal", "CitationId": 01}],
    },
}
```

### Data Fields

Main Fields will probably interest people are:

- "MedlineCitation" > "Article" > "AuthorList" > "Author"
- "MedlineCitation" > "Article" > "Abstract" > "AbstractText"
- "MedlineCitation" > "Article" > "Article Title"
- "MedlineCitation" > "ChemicalList" > "Chemical"
- "MedlineCitation" > "NumberOfReferences"

### Data Splits

There are no splits in this dataset. It is given as is.

## Dataset Creation

### Curation Rationale

The use of "Medline" in an element name does not mean the record represents a citation from a MEDLINE-selected journal. When the NLM DTDs and XML elements were first created, MEDLINE records were the only data exported. Now NLM exports citations other than MEDLINE records. To minimize unnecessary disruption to users of the data, NLM has retained the original element names (e.g., MedlineCitation, MedlineJournalInfo, MedlineTA).

Policies affecting data creation have evolved over the years. Some PubMed records are added or revised well after the cited article was first published. In these cases, on occasion an element that had not yet been created when the article was published may appear on the record. For example, the Abstract element was not created until 1975, but some records published before 1975 but added to PubMed after 1975 contain <Abstract>. It is also possible that an element may be treated differently from the way it would have been treated had the record been created or maintained near the time the article was published. For example, the number of <Author> occurrences can diverge from the policies stated in the NLM author indexing policy (https://pubmed.ncbi.nlm.nih.gov/help/#author-indexing-policy). Lastly, as of October 2016, the publisher of the original article has the capability to edit the PubMed record’s citation data, with the exception of MeSH data, using the PubMed Data Management system. PubMed record data for older citations, therefore, may contain data for elements that didn’t exist when the citation was created.

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[National Library of Medicine Terms and Conditions](https://www.nlm.nih.gov/databases/download/terms_and_conditions.html)

Downloading PubMed data from the National Library of Medicine FTP servers indicates your acceptance of the following Terms and Conditions. No charges, usage fees or royalties are paid to NLM for these data.

#### PubMed Specific Terms:

NLM freely provides PubMed data. Please note some abstracts may be protected by copyright.

#### General Terms and Conditions

Users of the data agree to:
- acknowledge NLM as the source of the data in a clear and conspicuous manner,
- NOT use the PubMed wordmark or the PubMed logo in association or in connection with user's or any other party's product or service.
- NOT adopt, use, or seek to register any mark or trade name confusingly similar to or suggestive of the PubMed wordmark or PubMed logo
- NOT to indicate or imply that NLM/NIH/HHS has endorsed its products/services/applications.

Users who republish or redistribute the data (services, products or raw data) agree to:
- maintain the most current version of all distributed data, or
- make known in a clear and conspicuous manner that the products/services/applications do not reflect the most current/accurate data available from NLM.

These data are produced with a reasonable standard of care, but NLM makes no warranties express or implied, including no warranty of merchantability or fitness for particular purpose, regarding the accuracy or completeness of the data. Users agree to hold NLM and the U.S. Government harmless from any liability resulting from errors in the data. NLM disclaims any liability for any consequences due to use, misuse, or interpretation of information contained or not contained in the data.

NLM does not provide legal advice regarding copyright, fair use, or other aspects of intellectual property rights. See the NLM Copyright page:  https://www.nlm.nih.gov/web_policies.html#copyright

NLM reserves the right to change the type and format of its machine-readable data. NLM will take reasonable steps to inform users of any changes to the format of the data before the data are distributed via the announcement section or subscription to email and RSS updates.

The PubMed wordmark and the PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

### Citation Information

[Courtesy of the U.S. National Library of Medicine](https://www.nlm.nih.gov/databases/download/terms_and_conditions.html).

### Contributions

Thanks to [@Narsil](https://github.com/Narsil) for adding this dataset.

task_categories:text-generation task_categories:fill-mask task_categories:text-classification task_ids:language-modeling task_ids:masked-language-modeling task_ids:text-scoring task_ids:topic-classification annotations_creators:crowdsourced language_creators:crowdsourced multilinguality:monolingual size_categories:10M<n<100M source_datasets:original language:en license:other citation-estimation region:us
{"annotations_creators": ["crowdsourced"], "language_creators": ["crowdsourced"], "language": ["en"], "license": ["other"], "multilinguality": ["monolingual"], "size_categories": ["10M<n<100M"], "source_datasets": ["original"], "task_categories": ["text-generation", "fill-mask", "text-classification"], "task_ids": ["language-modeling", "masked-language-modeling", "text-scoring", "topic-classification"], "paperswithcode_id": "pubmed", "pretty_name": "PubMed", "tags": ["citation-estimation"], "dataset_info": [{"config_name": "2024", "features": [{"name": "MedlineCitation", "struct": [{"name": "PMID", "dtype": "int32"}, {"name": "DateCompleted", "struct": [{"name": "Year", "dtype": "int32"}, {"name": "Month", "dtype": "int32"}, {"name": "Day", "dtype": "int32"}]}, {"name": "NumberOfReferences", "dtype": "int32"}, {"name": "DateRevised", "struct": [{"name": "Year", "dtype": "int32"}, {"name": "Month", "dtype": "int32"}, {"name": "Day", "dtype": "int32"}]}, {"name": "Article", "struct": [{"name": "Abstract", "struct": [{"name": "AbstractText", "dtype": "string"}]}, {"name": "ArticleTitle", "dtype": "string"}, {"name": "AuthorList", "struct": [{"name": "Author", "sequence": [{"name": "LastName", "dtype": "string"}, {"name": "ForeName", "dtype": "string"}, {"name": "Initials", "dtype": "string"}, {"name": "CollectiveName", "dtype": "string"}]}]}, {"name": "Language", "dtype": "string"}, {"name": "GrantList", "struct": [{"name": "Grant", "sequence": [{"name": "GrantID", "dtype": "string"}, {"name": "Agency", "dtype": "string"}, {"name": "Country", "dtype": "string"}]}]}, {"name": "PublicationTypeList", "struct": [{"name": "PublicationType", "sequence": "string"}]}]}, {"name": "MedlineJournalInfo", "struct": [{"name": "Country", "dtype": "string"}]}, {"name": "ChemicalList", "struct": [{"name": "Chemical", "sequence": [{"name": "RegistryNumber", "dtype": "string"}, {"name": "NameOfSubstance", "dtype": "string"}]}]}, {"name": "CitationSubset", "dtype": "string"}, {"name": "MeshHeadingList", "struct": [{"name": "MeshHeading", "sequence": [{"name": "DescriptorName", "dtype": "string"}, {"name": "QualifierName", "dtype": "string"}]}]}]}, {"name": "PubmedData", "struct": [{"name": "ArticleIdList", "sequence": [{"name": "ArticleId", "sequence": "string"}]}, {"name": "PublicationStatus", "dtype": "string"}, {"name": "History", "struct": [{"name": "PubMedPubDate", "sequence": [{"name": "Year", "dtype": "int32"}, {"name": "Month", "dtype": "int32"}, {"name": "Day", "dtype": "int32"}]}]}, {"name": "ReferenceList", "sequence": [{"name": "Citation", "dtype": "string"}, {"name": "CitationId", "dtype": "int32"}]}]}], "splits": [{"name": "train", "num_bytes": 54723097181, "num_examples": 36555430}], "download_size": 45202943276, "dataset_size": 54723097181}]}
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_categories-text-classification #task_ids-language-modeling #task_ids-masked-language-modeling #task_ids-text-scoring #task_ids-topic-classification #annotations_creators-crowdsourced #language_creators-crowdsourced #multilinguality-monolingual #size_categories-10M<n<100M #source_datasets-original #language-English #license-other #citation-estimation #region-us 


# Dataset Card for PubMed

## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions

## Dataset Description

- Homepage: : [URL
- Documentation: : [URL
- Repository:
- Paper:
- Leaderboard:
- Point of Contact: National Center for Biotechnology Information

### Dataset Summary

PubMed comprises more than 36 million citations for biomedical literature from MEDLINE, life science journals, and online books. Citations may include links to full-text content from PubMed Central and publisher web sites.

NLM produces a baseline set of PubMed citation records in XML format for download on an annual basis. The annual baseline is released in December of each year.
- Last Updated December 15, 2023

Each day, NLM produces update files that include new, revised, and deleted citations.

Source: URL

### Supported Tasks and Leaderboards



### Languages

- English

## Dataset Structure

Bear in mind the data comes from XML that have various tags that are hard to reflect
in a concise JSON format. Tags and list are kind of non "natural" to XML documents
leading this library to make some choices regarding data. "Journal" info was dropped
altogether as it would have led to many fields being empty all the time.

The hierarchy is also a bit unnatural but the choice was made to keep as close as
possible to the original data for future releases that may change schema from NLM's side.

Author has been kept and contains either "ForeName", "LastName", "Initials", or "CollectiveName".
(All the fields will be present all the time, but only some will be filled)

### Data Instances



### Data Fields

Main Fields will probably interest people are:

- "MedlineCitation" > "Article" > "AuthorList" > "Author"
- "MedlineCitation" > "Article" > "Abstract" > "AbstractText"
- "MedlineCitation" > "Article" > "Article Title"
- "MedlineCitation" > "ChemicalList" > "Chemical"
- "MedlineCitation" > "NumberOfReferences"

### Data Splits

There are no splits in this dataset. It is given as is.

## Dataset Creation

### Curation Rationale

The use of "Medline" in an element name does not mean the record represents a citation from a MEDLINE-selected journal. When the NLM DTDs and XML elements were first created, MEDLINE records were the only data exported. Now NLM exports citations other than MEDLINE records. To minimize unnecessary disruption to users of the data, NLM has retained the original element names (e.g., MedlineCitation, MedlineJournalInfo, MedlineTA).

Policies affecting data creation have evolved over the years. Some PubMed records are added or revised well after the cited article was first published. In these cases, on occasion an element that had not yet been created when the article was published may appear on the record. For example, the Abstract element was not created until 1975, but some records published before 1975 but added to PubMed after 1975 contain <Abstract>. It is also possible that an element may be treated differently from the way it would have been treated had the record been created or maintained near the time the article was published. For example, the number of <Author> occurrences can diverge from the policies stated in the NLM author indexing policy (URL Lastly, as of October 2016, the publisher of the original article has the capability to edit the PubMed record’s citation data, with the exception of MeSH data, using the PubMed Data Management system. PubMed record data for older citations, therefore, may contain data for elements that didn’t exist when the citation was created.

### Source Data

#### Initial Data Collection and Normalization



#### Who are the source language producers?



### Annotations

#### Annotation process



#### Who are the annotators?



### Personal and Sensitive Information



## Considerations for Using the Data

### Social Impact of Dataset



### Discussion of Biases



### Other Known Limitations



## Additional Information

### Dataset Curators



### Licensing Information

National Library of Medicine Terms and Conditions

Downloading PubMed data from the National Library of Medicine FTP servers indicates your acceptance of the following Terms and Conditions. No charges, usage fees or royalties are paid to NLM for these data.

#### PubMed Specific Terms:

NLM freely provides PubMed data. Please note some abstracts may be protected by copyright.

#### General Terms and Conditions

Users of the data agree to:
- acknowledge NLM as the source of the data in a clear and conspicuous manner,
- NOT use the PubMed wordmark or the PubMed logo in association or in connection with user's or any other party's product or service.
- NOT adopt, use, or seek to register any mark or trade name confusingly similar to or suggestive of the PubMed wordmark or PubMed logo
- NOT to indicate or imply that NLM/NIH/HHS has endorsed its products/services/applications.

Users who republish or redistribute the data (services, products or raw data) agree to:
- maintain the most current version of all distributed data, or
- make known in a clear and conspicuous manner that the products/services/applications do not reflect the most current/accurate data available from NLM.

These data are produced with a reasonable standard of care, but NLM makes no warranties express or implied, including no warranty of merchantability or fitness for particular purpose, regarding the accuracy or completeness of the data. Users agree to hold NLM and the U.S. Government harmless from any liability resulting from errors in the data. NLM disclaims any liability for any consequences due to use, misuse, or interpretation of information contained or not contained in the data.

NLM does not provide legal advice regarding copyright, fair use, or other aspects of intellectual property rights. See the NLM Copyright page:  URL

NLM reserves the right to change the type and format of its machine-readable data. NLM will take reasonable steps to inform users of any changes to the format of the data before the data are distributed via the announcement section or subscription to email and RSS updates.

The PubMed wordmark and the PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.



Courtesy of the U.S. National Library of Medicine.

### Contributions

Thanks to @Narsil for adding this dataset.

# Dataset Card for PubMed ## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: : [URL
- Documentation: : [URL
- Repository:
- Paper:
- Leaderboard:
- Point of Contact: National Center for Biotechnology Information ### Dataset Summary

PubMed comprises more than 36 million citations for biomedical literature from MEDLINE, life science journals, and online books. Citations may include links to full-text content from PubMed Central and publisher web sites.

NLM produces a baseline set of PubMed citation records in XML format for download on an annual basis. The annual baseline is released in December of each year.
- Last Updated December 15, 2023

Each day, NLM produces update files that include new, revised, and deleted citations.

Source: URL ### Supported Tasks and Leaderboards ### Languages

- English ## Dataset Structure

Bear in mind the data comes from XML that have various tags that are hard to reflect
in a concise JSON format. Tags and list are kind of non "natural" to XML documents
leading this library to make some choices regarding data. "Journal" info was dropped
altogether as it would have led to many fields being empty all the time.

The hierarchy is also a bit unnatural but the choice was made to keep as close as
possible to the original data for future releases that may change schema from NLM's side.

Author has been kept and contains either "ForeName", "LastName", "Initials", or "CollectiveName".
(All the fields will be present all the time, but only some will be filled) ### Data Instances ### Data Fields

Main Fields will probably interest people are:

- "MedlineCitation" > "Article" > "AuthorList" > "Author"
- "MedlineCitation" > "Article" > "Abstract" > "AbstractText"
- "MedlineCitation" > "Article" > "Article Title"
- "MedlineCitation" > "ChemicalList" > "Chemical"
- "MedlineCitation" > "NumberOfReferences" ### Data Splits

There are no splits in this dataset. It is given as is. ## Dataset Creation ### Curation Rationale

The use of "Medline" in an element name does not mean the record represents a citation from a MEDLINE-selected journal. When the NLM DTDs and XML elements were first created, MEDLINE records were the only data exported. Now NLM exports citations other than MEDLINE records. To minimize unnecessary disruption to users of the data, NLM has retained the original element names (e.g., MedlineCitation, MedlineJournalInfo, MedlineTA).

Policies affecting data creation have evolved over the years. Some PubMed records are added or revised well after the cited article was first published. In these cases, on occasion an element that had not yet been created when the article was published may appear on the record. For example, the Abstract element was not created until 1975, but some records published before 1975 but added to PubMed after 1975 contain <Abstract>. It is also possible that an element may be treated differently from the way it would have been treated had the record been created or maintained near the time the article was published. For example, the number of <Author> occurrences can diverge from the policies stated in the NLM author indexing policy (URL Lastly, as of October 2016, the publisher of the original article has the capability to edit the PubMed record’s citation data, with the exception of MeSH data, using the PubMed Data Management system. PubMed record data for older citations, therefore, may contain data for elements that didn’t exist when the citation was created. ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators ### Licensing Information

National Library of Medicine Terms and Conditions

Downloading PubMed data from the National Library of Medicine FTP servers indicates your acceptance of the following Terms and Conditions. No charges, usage fees or royalties are paid to NLM for these data. #### PubMed Specific Terms:

NLM freely provides PubMed data. Please note some abstracts may be protected by copyright. #### General Terms and Conditions

Users of the data agree to:
- acknowledge NLM as the source of the data in a clear and conspicuous manner,
- NOT use the PubMed wordmark or the PubMed logo in association or in connection with user's or any other party's product or service.
- NOT adopt, use, or seek to register any mark or trade name confusingly similar to or suggestive of the PubMed wordmark or PubMed logo
- NOT to indicate or imply that NLM/NIH/HHS has endorsed its products/services/applications.

Users who republish or redistribute the data (services, products or raw data) agree to:
- maintain the most current version of all distributed data, or
- make known in a clear and conspicuous manner that the products/services/applications do not reflect the most current/accurate data available from NLM.

These data are produced with a reasonable standard of care, but NLM makes no warranties express or implied, including no warranty of merchantability or fitness for particular purpose, regarding the accuracy or completeness of the data. Users agree to hold NLM and the U.S. Government harmless from any liability resulting from errors in the data. NLM disclaims any liability for any consequences due to use, misuse, or interpretation of information contained or not contained in the data.

NLM does not provide legal advice regarding copyright, fair use, or other aspects of intellectual property rights. See the NLM Copyright page:  URL

NLM reserves the right to change the type and format of its machine-readable data. NLM will take reasonable steps to inform users of any changes to the format of the data before the data are distributed via the announcement section or subscription to email and RSS updates.

The PubMed wordmark and the PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.



Courtesy of the U.S. National Library of Medicine. ### Contributions

Thanks to @Narsil for adding this dataset.
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_categories-text-classification #task_ids-language-modeling #task_ids-masked-language-modeling #task_ids-text-scoring #task_ids-topic-classification #annotations_creators-crowdsourced #language_creators-crowdsourced #multilinguality-monolingual #size_categories-10M<n<100M #source_datasets-original #language-English #license-other #citation-estimation #region-us 
 # Dataset Card for PubMed ## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: : [URL
- Documentation: : [URL
- Repository:
- Paper:
- Leaderboard:
- Point of Contact: National Center for Biotechnology Information ### Dataset Summary

PubMed comprises more than 36 million citations for biomedical literature from MEDLINE, life science journals, and online books. Citations may include links to full-text content from PubMed Central and publisher web sites.

NLM produces a baseline set of PubMed citation records in XML format for download on an annual basis. The annual baseline is released in December of each year.
- Last Updated December 15, 2023

Each day, NLM produces update files that include new, revised, and deleted citations.

Source: URL ### Supported Tasks and Leaderboards ### Languages

- English ## Dataset Structure

Bear in mind the data comes from XML that have various tags that are hard to reflect
in a concise JSON format. Tags and list are kind of non "natural" to XML documents
leading this library to make some choices regarding data. "Journal" info was dropped
altogether as it would have led to many fields being empty all the time.

The hierarchy is also a bit unnatural but the choice was made to keep as close as
possible to the original data for future releases that may change schema from NLM's side.

Author has been kept and contains either "ForeName", "LastName", "Initials", or "CollectiveName".
(All the fields will be present all the time, but only some will be filled) ### Data Instances ### Data Fields

Main Fields will probably interest people are:

- "MedlineCitation" > "Article" > "AuthorList" > "Author"
- "MedlineCitation" > "Article" > "Abstract" > "AbstractText"
- "MedlineCitation" > "Article" > "Article Title"
- "MedlineCitation" > "ChemicalList" > "Chemical"
- "MedlineCitation" > "NumberOfReferences" ### Data Splits

There are no splits in this dataset. It is given as is. ## Dataset Creation ### Curation Rationale

The use of "Medline" in an element name does not mean the record represents a citation from a MEDLINE-selected journal. When the NLM DTDs and XML elements were first created, MEDLINE records were the only data exported. Now NLM exports citations other than MEDLINE records. To minimize unnecessary disruption to users of the data, NLM has retained the original element names (e.g., MedlineCitation, MedlineJournalInfo, MedlineTA).

Policies affecting data creation have evolved over the years. Some PubMed records are added or revised well after the cited article was first published. In these cases, on occasion an element that had not yet been created when the article was published may appear on the record. For example, the Abstract element was not created until 1975, but some records published before 1975 but added to PubMed after 1975 contain <Abstract>. It is also possible that an element may be treated differently from the way it would have been treated had the record been created or maintained near the time the article was published. For example, the number of <Author> occurrences can diverge from the policies stated in the NLM author indexing policy (URL Lastly, as of October 2016, the publisher of the original article has the capability to edit the PubMed record’s citation data, with the exception of MeSH data, using the PubMed Data Management system. PubMed record data for older citations, therefore, may contain data for elements that didn’t exist when the citation was created. ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators ### Licensing Information

National Library of Medicine Terms and Conditions

Downloading PubMed data from the National Library of Medicine FTP servers indicates your acceptance of the following Terms and Conditions. No charges, usage fees or royalties are paid to NLM for these data. #### PubMed Specific Terms:

NLM freely provides PubMed data. Please note some abstracts may be protected by copyright. #### General Terms and Conditions

Users of the data agree to:
- acknowledge NLM as the source of the data in a clear and conspicuous manner,
- NOT use the PubMed wordmark or the PubMed logo in association or in connection with user's or any other party's product or service.
- NOT adopt, use, or seek to register any mark or trade name confusingly similar to or suggestive of the PubMed wordmark or PubMed logo
- NOT to indicate or imply that NLM/NIH/HHS has endorsed its products/services/applications.

Users who republish or redistribute the data (services, products or raw data) agree to:
- maintain the most current version of all distributed data, or
- make known in a clear and conspicuous manner that the products/services/applications do not reflect the most current/accurate data available from NLM.

These data are produced with a reasonable standard of care, but NLM makes no warranties express or implied, including no warranty of merchantability or fitness for particular purpose, regarding the accuracy or completeness of the data. Users agree to hold NLM and the U.S. Government harmless from any liability resulting from errors in the data. NLM disclaims any liability for any consequences due to use, misuse, or interpretation of information contained or not contained in the data.

NLM does not provide legal advice regarding copyright, fair use, or other aspects of intellectual property rights. See the NLM Copyright page:  URL

NLM reserves the right to change the type and format of its machine-readable data. NLM will take reasonable steps to inform users of any changes to the format of the data before the data are distributed via the announcement section or subscription to email and RSS updates.

The PubMed wordmark and the PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.



Courtesy of the U.S. National Library of Medicine. ### Contributions

Thanks to @Narsil for adding this dataset.
passage: TAGS
#task_categories-text-generation #task_categories-fill-mask #task_categories-text-classification #task_ids-language-modeling #task_ids-masked-language-modeling #task_ids-text-scoring #task_ids-topic-classification #annotations_creators-crowdsourced #language_creators-crowdsourced #multilinguality-monolingual #size_categories-10M<n<100M #source_datasets-original #language-English #license-other #citation-estimation #region-us 
# Dataset Card for PubMed## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions## Dataset Description

- Homepage: : [URL
- Documentation: : [URL
- Repository:
- Paper:
- Leaderboard:
- Point of Contact: National Center for Biotechnology Information### Dataset Summary

PubMed comprises more than 36 million citations for biomedical literature from MEDLINE, life science journals, and online books. Citations may include links to full-text content from PubMed Central and publisher web sites.

NLM produces a baseline set of PubMed citation records in XML format for download on an annual basis. The annual baseline is released in December of each year.
- Last Updated December 15, 2023

Each day, NLM produces update files that include new, revised, and deleted citations.

Source: URL### Supported Tasks and Leaderboards### Languages

- English passage: ## Dataset Structure

Bear in mind the data comes from XML that have various tags that are hard to reflect
in a concise JSON format. Tags and list are kind of non "natural" to XML documents
leading this library to make some choices regarding data. "Journal" info was dropped
altogether as it would have led to many fields being empty all the time.

The hierarchy is also a bit unnatural but the choice was made to keep as close as
possible to the original data for future releases that may change schema from NLM's side.

Author has been kept and contains either "ForeName", "LastName", "Initials", or "CollectiveName".
(All the fields will be present all the time, but only some will be filled)### Data Instances### Data Fields

Main Fields will probably interest people are:

- "MedlineCitation" > "Article" > "AuthorList" > "Author"
- "MedlineCitation" > "Article" > "Abstract" > "AbstractText"
- "MedlineCitation" > "Article" > "Article Title"
- "MedlineCitation" > "ChemicalList" > "Chemical"
- "MedlineCitation" > "NumberOfReferences"### Data Splits

There are no splits in this dataset. It is given as is.## Dataset Creation### Curation Rationale

The use of "Medline" in an element name does not mean the record represents a citation from a MEDLINE-selected journal. When the NLM DTDs and XML elements were first created, MEDLINE records were the only data exported. Now NLM exports citations other than MEDLINE records. To minimize unnecessary disruption to users of the data, NLM has retained the original element names (e.g., MedlineCitation, MedlineJournalInfo, MedlineTA).

Policies affecting data creation have evolved over the years. Some PubMed records are added or revised well after the cited article was first published. In these cases, on occasion an element that had not yet been created when the article was published may appear on the record. For example, the Abstract element was not created until 1975, but some records published before 1975 but added to PubMed after 1975 contain <Abstract>. It is also possible that an element may be treated differently from the way it would have been treated had the record been created or maintained near the time the article was published. For example, the number of <Author> occurrences can diverge from the policies stated in the NLM author indexing policy (URL Lastly, as of October 2016, the publisher of the original article has the capability to edit the PubMed record’s citation data, with the exception of MeSH data, using the PubMed Data Management system. PubMed record data for older citations, therefore, may contain data for elements that didn’t exist when the citation was created.### Source Data passage: #### Initial Data Collection and Normalization#### Who are the source language producers?### Annotations#### Annotation process#### Who are the annotators?### Personal and Sensitive Information## Considerations for Using the Data### Social Impact of Dataset### Discussion of Biases### Other Known Limitations## Additional Information### Dataset Curators### Licensing Information

National Library of Medicine Terms and Conditions

Downloading PubMed data from the National Library of Medicine FTP servers indicates your acceptance of the following Terms and Conditions. No charges, usage fees or royalties are paid to NLM for these data.#### PubMed Specific Terms:

NLM freely provides PubMed data. Please note some abstracts may be protected by copyright.