# Dataset Card for "ArtifactAI/arxiv_s2orc_parsed"


## Dataset Description

https://huggingface.co/datasets/ArtifactAI/arxiv_s2orc_parsed


### Dataset Summary

ArtifactAI/arxiv_s2orc_parsed is a subset of the [AllenAI S2ORC dataset](https://github.com/allenai/s2orc), a general-purpose corpus for NLP and text mining research over scientific papers, 
The dataset is filtered strictly for ArXiv papers, including the full text for each paper. Github links have been extracted from each paper to aid in the development of [ArtifactAI/arxiv_python_research_code](https://huggingface.co/datasets/ArtifactAI/arxiv_python_research_code)

### How to use it
```python
from datasets import load_dataset

ds = load_dataset("ArtifactAI/arxiv_s2orc_parsed", split="train")

# dataset streaming (will only download the data as needed)
ds = load_dataset("ArtifactAI/arxiv_s2orc_parsed", streaming=True, split="train")
```

## Dataset Structure
### Data Instances
Each data instance corresponds to one file. The content of the file is in the `text` feature, and other features provide some metadata.
### Data Fields
- `title` (sequence): list of titles.
- `author` (sequence): list of authors.
- `authoraffiliation` (sequence): list of institution affiliations for each author.
- `venue`: (integer): paper publication venue.
- `doi`: (float): paper doi.
- `pdfurls`: (integer): url link to the paper.
- `corpusid`: (int): corpus ID as defined by s2orc.
- `arxivid`: (int): arxiv paper id.
- `pdfsha`: (string): unique pdf hash.
- `text`: (string): full text of the arxiv paper.
- github_urls: (sequence): list of github urls referenced within the text

### Data Splits

The dataset has no splits and all data is loaded as train split by default.

## Additional Information

### Dataset Curators
Matthew Kenney, Artifact AI, matt@artifactai.com

### Citation Information
```
@misc{arxiv_s2orc_parsed,
    title={arxiv_s2orc_parsed},
    author={Matthew Kenney},
    year={2023}
}
```
task_categories:text-generation task_categories:zero-shot-classification size_categories:10B<n<100B language:en region:us
{"language": ["en"], "size_categories": ["10B<n<100B"], "task_categories": ["text-generation", "zero-shot-classification"], "pretty_name": "arxiv_s2orc_parsed", "dataset_info": {"features": [{"name": "title", "sequence": "string"}, {"name": "author", "sequence": "string"}, {"name": "authoraffiliation", "sequence": "string"}, {"name": "venue", "sequence": "string"}, {"name": "abstract", "dtype": "string"}, {"name": "doi", "dtype": "string"}, {"name": "pdfurls", "sequence": "string"}, {"name": "corpusid", "dtype": "int64"}, {"name": "arxivid", "dtype": "string"}, {"name": "pdfsha", "dtype": "string"}, {"name": "text", "dtype": "string"}, {"name": "github_urls", "sequence": "string"}], "splits": [{"name": "train", "num_bytes": 89132091867, "num_examples": 1671614}], "download_size": 35993359504, "dataset_size": 89132091867}}
TAGS
#task_categories-text-generation #task_categories-zero-shot-classification #size_categories-10B<n<100B #language-English #region-us 

# Dataset Card for "ArtifactAI/arxiv_s2orc_parsed"


## Dataset Description

URL


### Dataset Summary

ArtifactAI/arxiv_s2orc_parsed is a subset of the AllenAI S2ORC dataset, a general-purpose corpus for NLP and text mining research over scientific papers, 
The dataset is filtered strictly for ArXiv papers, including the full text for each paper. Github links have been extracted from each paper to aid in the development of ArtifactAI/arxiv_python_research_code

### How to use it


## Dataset Structure
### Data Instances
Each data instance corresponds to one file. The content of the file is in the 'text' feature, and other features provide some metadata.
### Data Fields
- 'title' (sequence): list of titles.
- 'author' (sequence): list of authors.
- 'authoraffiliation' (sequence): list of institution affiliations for each author.
- 'venue': (integer): paper publication venue.
- 'doi': (float): paper doi.
- 'pdfurls': (integer): url link to the paper.
- 'corpusid': (int): corpus ID as defined by s2orc.
- 'arxivid': (int): arxiv paper id.
- 'pdfsha': (string): unique pdf hash.
- 'text': (string): full text of the arxiv paper.
- github_urls: (sequence): list of github urls referenced within the text

### Data Splits

The dataset has no splits and all data is loaded as train split by default.

## Additional Information

### Dataset Curators
Matthew Kenney, Artifact AI, matt@URL



# Dataset Card for "ArtifactAI/arxiv_s2orc_parsed" ## Dataset Description

URL ### Dataset Summary

ArtifactAI/arxiv_s2orc_parsed is a subset of the AllenAI S2ORC dataset, a general-purpose corpus for NLP and text mining research over scientific papers, 
The dataset is filtered strictly for ArXiv papers, including the full text for each paper. Github links have been extracted from each paper to aid in the development of ArtifactAI/arxiv_python_research_code ### How to use it ## Dataset Structure ### Data Instances
Each data instance corresponds to one file. The content of the file is in the 'text' feature, and other features provide some metadata. ### Data Fields
- 'title' (sequence): list of titles.
- 'author' (sequence): list of authors.
- 'authoraffiliation' (sequence): list of institution affiliations for each author.
- 'venue': (integer): paper publication venue.
- 'doi': (float): paper doi.
- 'pdfurls': (integer): url link to the paper.
- 'corpusid': (int): corpus ID as defined by s2orc.
- 'arxivid': (int): arxiv paper id.
- 'pdfsha': (string): unique pdf hash.
- 'text': (string): full text of the arxiv paper.
- github_urls: (sequence): list of github urls referenced within the text ### Data Splits

The dataset has no splits and all data is loaded as train split by default. ## Additional Information ### Dataset Curators
Matthew Kenney, Artifact AI, matt@URL
TAGS
#task_categories-text-generation #task_categories-zero-shot-classification #size_categories-10B<n<100B #language-English #region-us 
 # Dataset Card for "ArtifactAI/arxiv_s2orc_parsed" ## Dataset Description

URL ### Dataset Summary

ArtifactAI/arxiv_s2orc_parsed is a subset of the AllenAI S2ORC dataset, a general-purpose corpus for NLP and text mining research over scientific papers, 
The dataset is filtered strictly for ArXiv papers, including the full text for each paper. Github links have been extracted from each paper to aid in the development of ArtifactAI/arxiv_python_research_code ### How to use it ## Dataset Structure ### Data Instances
Each data instance corresponds to one file. The content of the file is in the 'text' feature, and other features provide some metadata. ### Data Fields
- 'title' (sequence): list of titles.
- 'author' (sequence): list of authors.
- 'authoraffiliation' (sequence): list of institution affiliations for each author.
- 'venue': (integer): paper publication venue.
- 'doi': (float): paper doi.
- 'pdfurls': (integer): url link to the paper.
- 'corpusid': (int): corpus ID as defined by s2orc.
- 'arxivid': (int): arxiv paper id.
- 'pdfsha': (string): unique pdf hash.
- 'text': (string): full text of the arxiv paper.
- github_urls: (sequence): list of github urls referenced within the text ### Data Splits

The dataset has no splits and all data is loaded as train split by default. ## Additional Information ### Dataset Curators
Matthew Kenney, Artifact AI, matt@URL
passage: TAGS
#task_categories-text-generation #task_categories-zero-shot-classification #size_categories-10B<n<100B #language-English #region-us 
# Dataset Card for "ArtifactAI/arxiv_s2orc_parsed"## Dataset Description

URL### Dataset Summary

ArtifactAI/arxiv_s2orc_parsed is a subset of the AllenAI S2ORC dataset, a general-purpose corpus for NLP and text mining research over scientific papers, 
The dataset is filtered strictly for ArXiv papers, including the full text for each paper. Github links have been extracted from each paper to aid in the development of ArtifactAI/arxiv_python_research_code### How to use it## Dataset Structure### Data Instances
Each data instance corresponds to one file. The content of the file is in the 'text' feature, and other features provide some metadata.### Data Fields
- 'title' (sequence): list of titles.
- 'author' (sequence): list of authors.
- 'authoraffiliation' (sequence): list of institution affiliations for each author.
- 'venue': (integer): paper publication venue.
- 'doi': (float): paper doi.
- 'pdfurls': (integer): url link to the paper.
- 'corpusid': (int): corpus ID as defined by s2orc.
- 'arxivid': (int): arxiv paper id.
- 'pdfsha': (string): unique pdf hash.
- 'text': (string): full text of the arxiv paper.
- github_urls: (sequence): list of github urls referenced within the text### Data Splits

The dataset has no splits and all data is loaded as train split by default.## Additional Information### Dataset Curators
Matthew Kenney, Artifact AI, matt@URL