
# Hacker News corpus, 2007-Nov 2022

## Dataset Description

### Dataset Summary

**Dataset Name:** Hacker News Full Corpus (2007 - November 2022)

**Description:**

> NOTE: I am not affiliated with Y Combinator.

This dataset is a July 2023 snapshot of YCombinator's [BigQuery dump](https://console.cloud.google.com/marketplace/details/y-combinator/hacker-news) of the entire archive of posts and comments made on Hacker News. It contains posts from Hacker News' inception in 2007 through to November 16, 2022, when the BigQuery database was last updated.

The dataset does not incorporate any modifications or filtering - it is a raw dump from the original dataset provided by YCombinator. Hence, it retains the same structure and fields as the original BigQuery table, serving as a ready-to-use resource for conducting large-scale data analysis or training language models. 

All credit for the original data collection and maintenance goes to YCombinator and the original post and comment authors. This version of the dataset has been prepared for convenience and ease of use within the HuggingFace ecosystem, especially for those interested in offline usage or who prefer not to use Google Cloud.

Please bear in mind that this dataset is a snapshot and will probably not be updated. For the latest data, consider accessing the live data directly from the official [Hacker News API](https://github.com/HackerNews/API), potentially using [Anant Narayanan's scripts](https://www.kix.in/2023/05/05/hacker-news-chatgpt-plugin/#downloading-the-dataset).

Please use responsibly, respecting all relevant terms of use and privacy considerations inherent in the data.


### Languages

English

## Dataset Structure

### Data Fields

| fullname    | mode     | type      | description                                                  |
| ----------- | -------- | --------- | ------------------------------------------------------------ |
| title       | NULLABLE | STRING    | Story title                                                  |
| url         | NULLABLE | STRING    | Story url                                                    |
| text        | NULLABLE | STRING    | Story or comment text                                        |
| dead        | NULLABLE | BOOLEAN   | Is dead?                                                     |
| by          | NULLABLE | STRING    | The username of the item's author.                           |
| score       | NULLABLE | INTEGER   | Story score                                                  |
| time        | NULLABLE | INTEGER   | Unix time                                                    |
| timestamp   | NULLABLE | TIMESTAMP | Timestamp for the unix time                                  |
| type        | NULLABLE | STRING    | Type of details (comment, comment_ranking, poll, story, job, pollopt) |
| id          | NULLABLE | INTEGER   | The item's unique id.                                        |
| parent      | NULLABLE | INTEGER   | Parent comment ID                                            |
| descendants | NULLABLE | INTEGER   | Number of story or poll descendants                          |
| ranking     | NULLABLE | INTEGER   | Comment ranking                                              |
| deleted     | NULLABLE | BOOLEAN   | Is deleted?                                                  |

## Dataset Creation

### Curation Rationale

This dataset provides a snapshot of the Hacker News posts and comments archive, sourced from YCombinator's open data, to enable easy and direct access without the need for a Google Cloud account or BigQuery interface, and without putting undue strain on the HN API. It aims to simplify the data acquisition process, promoting its use within the HuggingFace ecosystem for various tasks including analysis, trend prediction, sentiment studies, and language model training. By minimizing barriers to access, this dataset encourages a wider usage, fostering innovation in natural language processing and related fields.

### Annotations

### Personal and Sensitive Information

This dataset has not undergone specific checks for personally identifiable information (PII); hence, it's possible that some may exist within the data. However, as the data source is publicly available and shared by YCombinator, any potential PII present is already part of the public domain.

## Considerations for Using the Data

### Social Impact of Dataset

The collective wisdom and perspectives captured in the posts and comments of this Hacker News dataset represent a unique gift from YCombinator and countless contributors worldwide; it is part of the common heritage of humanity. The potential insights to be gleaned and the future knowledge to be generated, especially through the training of language models on this corpus, can provide unbounded new perspectives, enriching our understanding and potential solutions to complex issues. It is a testament to the power of shared knowledge and open dialogue in shaping the world. While there is a risk that some may use language models trained on this dataset for disinformation purposes, it's worth noting that the misuse of technology is a challenge that predates this dataset. The proverbial horse of misused technology has long since left the barn; our focus now must be on harnessing this shared intellectual legacy responsibly for the common good.

### Discussion of Biases

Given that Hacker News is a technology-focused platform with a largely self-selected user base, the content and perspectives found within this dataset may lean towards technology, entrepreneurship, and related fields, often reflecting the views and biases of this specific community. As such, users should be aware that analysis drawn from this data may not fully represent a balanced, global perspective and might contain inherent biases towards topics and viewpoints that are overrepresented in the Hacker News community.

## Additional Information

### Licensing Information

In the absence of an explicit license for the upstream BigQuery dataset, this dataset uses the same MIT license as the Hacker News API. The upstream terms of use are reproduced here:

> This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source - [https://github.com/HackerNews/API ](https://github.com/HackerNews/API) - and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.
size_categories:10M<n<100M language:en license:mit region:us
{"language": ["en"], "license": "mit", "size_categories": ["10M<n<100M"], "pretty_name": "Hacker News posts and comments, 2007-Nov 2022"}
TAGS
#size_categories-10M<n<100M #language-English #license-mit #region-us 

Hacker News corpus, 2007-Nov 2022
=================================


Dataset Description
-------------------


### Dataset Summary


Dataset Name: Hacker News Full Corpus (2007 - November 2022)


Description:



> 
> NOTE: I am not affiliated with Y Combinator.
> 
> 
> 


This dataset is a July 2023 snapshot of YCombinator's BigQuery dump of the entire archive of posts and comments made on Hacker News. It contains posts from Hacker News' inception in 2007 through to November 16, 2022, when the BigQuery database was last updated.


The dataset does not incorporate any modifications or filtering - it is a raw dump from the original dataset provided by YCombinator. Hence, it retains the same structure and fields as the original BigQuery table, serving as a ready-to-use resource for conducting large-scale data analysis or training language models.


All credit for the original data collection and maintenance goes to YCombinator and the original post and comment authors. This version of the dataset has been prepared for convenience and ease of use within the HuggingFace ecosystem, especially for those interested in offline usage or who prefer not to use Google Cloud.


Please bear in mind that this dataset is a snapshot and will probably not be updated. For the latest data, consider accessing the live data directly from the official Hacker News API, potentially using Anant Narayanan's scripts.


Please use responsibly, respecting all relevant terms of use and privacy considerations inherent in the data.


### Languages


English


Dataset Structure
-----------------


### Data Fields



Dataset Creation
----------------


### Curation Rationale


This dataset provides a snapshot of the Hacker News posts and comments archive, sourced from YCombinator's open data, to enable easy and direct access without the need for a Google Cloud account or BigQuery interface, and without putting undue strain on the HN API. It aims to simplify the data acquisition process, promoting its use within the HuggingFace ecosystem for various tasks including analysis, trend prediction, sentiment studies, and language model training. By minimizing barriers to access, this dataset encourages a wider usage, fostering innovation in natural language processing and related fields.


### Annotations


### Personal and Sensitive Information


This dataset has not undergone specific checks for personally identifiable information (PII); hence, it's possible that some may exist within the data. However, as the data source is publicly available and shared by YCombinator, any potential PII present is already part of the public domain.


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


The collective wisdom and perspectives captured in the posts and comments of this Hacker News dataset represent a unique gift from YCombinator and countless contributors worldwide; it is part of the common heritage of humanity. The potential insights to be gleaned and the future knowledge to be generated, especially through the training of language models on this corpus, can provide unbounded new perspectives, enriching our understanding and potential solutions to complex issues. It is a testament to the power of shared knowledge and open dialogue in shaping the world. While there is a risk that some may use language models trained on this dataset for disinformation purposes, it's worth noting that the misuse of technology is a challenge that predates this dataset. The proverbial horse of misused technology has long since left the barn; our focus now must be on harnessing this shared intellectual legacy responsibly for the common good.


### Discussion of Biases


Given that Hacker News is a technology-focused platform with a largely self-selected user base, the content and perspectives found within this dataset may lean towards technology, entrepreneurship, and related fields, often reflecting the views and biases of this specific community. As such, users should be aware that analysis drawn from this data may not fully represent a balanced, global perspective and might contain inherent biases towards topics and viewpoints that are overrepresented in the Hacker News community.


Additional Information
----------------------


### Licensing Information


In the absence of an explicit license for the upstream BigQuery dataset, this dataset uses the same MIT license as the Hacker News API. The upstream terms of use are reproduced here:



> 
> This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source - URL - and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.
> 
> 
> 



### Dataset Summary


Dataset Name: Hacker News Full Corpus (2007 - November 2022)


Description:



> 
> NOTE: I am not affiliated with Y Combinator.
> 
> 
> 


This dataset is a July 2023 snapshot of YCombinator's BigQuery dump of the entire archive of posts and comments made on Hacker News. It contains posts from Hacker News' inception in 2007 through to November 16, 2022, when the BigQuery database was last updated.


The dataset does not incorporate any modifications or filtering - it is a raw dump from the original dataset provided by YCombinator. Hence, it retains the same structure and fields as the original BigQuery table, serving as a ready-to-use resource for conducting large-scale data analysis or training language models.


All credit for the original data collection and maintenance goes to YCombinator and the original post and comment authors. This version of the dataset has been prepared for convenience and ease of use within the HuggingFace ecosystem, especially for those interested in offline usage or who prefer not to use Google Cloud.


Please bear in mind that this dataset is a snapshot and will probably not be updated. For the latest data, consider accessing the live data directly from the official Hacker News API, potentially using Anant Narayanan's scripts.


Please use responsibly, respecting all relevant terms of use and privacy considerations inherent in the data. ### Languages


English


Dataset Structure
----------------- ### Data Fields



Dataset Creation
---------------- ### Curation Rationale


This dataset provides a snapshot of the Hacker News posts and comments archive, sourced from YCombinator's open data, to enable easy and direct access without the need for a Google Cloud account or BigQuery interface, and without putting undue strain on the HN API. It aims to simplify the data acquisition process, promoting its use within the HuggingFace ecosystem for various tasks including analysis, trend prediction, sentiment studies, and language model training. By minimizing barriers to access, this dataset encourages a wider usage, fostering innovation in natural language processing and related fields. ### Annotations ### Personal and Sensitive Information


This dataset has not undergone specific checks for personally identifiable information (PII); hence, it's possible that some may exist within the data. However, as the data source is publicly available and shared by YCombinator, any potential PII present is already part of the public domain.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


The collective wisdom and perspectives captured in the posts and comments of this Hacker News dataset represent a unique gift from YCombinator and countless contributors worldwide; it is part of the common heritage of humanity. The potential insights to be gleaned and the future knowledge to be generated, especially through the training of language models on this corpus, can provide unbounded new perspectives, enriching our understanding and potential solutions to complex issues. It is a testament to the power of shared knowledge and open dialogue in shaping the world. While there is a risk that some may use language models trained on this dataset for disinformation purposes, it's worth noting that the misuse of technology is a challenge that predates this dataset. The proverbial horse of misused technology has long since left the barn; our focus now must be on harnessing this shared intellectual legacy responsibly for the common good. ### Discussion of Biases


Given that Hacker News is a technology-focused platform with a largely self-selected user base, the content and perspectives found within this dataset may lean towards technology, entrepreneurship, and related fields, often reflecting the views and biases of this specific community. As such, users should be aware that analysis drawn from this data may not fully represent a balanced, global perspective and might contain inherent biases towards topics and viewpoints that are overrepresented in the Hacker News community.


Additional Information
---------------------- ### Licensing Information


In the absence of an explicit license for the upstream BigQuery dataset, this dataset uses the same MIT license as the Hacker News API. The upstream terms of use are reproduced here:



> 
> This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source - URL - and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.
> 
> 
>
TAGS
#size_categories-10M<n<100M #language-English #license-mit #region-us 
 ### Dataset Summary


Dataset Name: Hacker News Full Corpus (2007 - November 2022)


Description:



> 
> NOTE: I am not affiliated with Y Combinator.
> 
> 
> 


This dataset is a July 2023 snapshot of YCombinator's BigQuery dump of the entire archive of posts and comments made on Hacker News. It contains posts from Hacker News' inception in 2007 through to November 16, 2022, when the BigQuery database was last updated.


The dataset does not incorporate any modifications or filtering - it is a raw dump from the original dataset provided by YCombinator. Hence, it retains the same structure and fields as the original BigQuery table, serving as a ready-to-use resource for conducting large-scale data analysis or training language models.


All credit for the original data collection and maintenance goes to YCombinator and the original post and comment authors. This version of the dataset has been prepared for convenience and ease of use within the HuggingFace ecosystem, especially for those interested in offline usage or who prefer not to use Google Cloud.


Please bear in mind that this dataset is a snapshot and will probably not be updated. For the latest data, consider accessing the live data directly from the official Hacker News API, potentially using Anant Narayanan's scripts.


Please use responsibly, respecting all relevant terms of use and privacy considerations inherent in the data. ### Languages


English


Dataset Structure
----------------- ### Data Fields



Dataset Creation
---------------- ### Curation Rationale


This dataset provides a snapshot of the Hacker News posts and comments archive, sourced from YCombinator's open data, to enable easy and direct access without the need for a Google Cloud account or BigQuery interface, and without putting undue strain on the HN API. It aims to simplify the data acquisition process, promoting its use within the HuggingFace ecosystem for various tasks including analysis, trend prediction, sentiment studies, and language model training. By minimizing barriers to access, this dataset encourages a wider usage, fostering innovation in natural language processing and related fields. ### Annotations ### Personal and Sensitive Information


This dataset has not undergone specific checks for personally identifiable information (PII); hence, it's possible that some may exist within the data. However, as the data source is publicly available and shared by YCombinator, any potential PII present is already part of the public domain.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


The collective wisdom and perspectives captured in the posts and comments of this Hacker News dataset represent a unique gift from YCombinator and countless contributors worldwide; it is part of the common heritage of humanity. The potential insights to be gleaned and the future knowledge to be generated, especially through the training of language models on this corpus, can provide unbounded new perspectives, enriching our understanding and potential solutions to complex issues. It is a testament to the power of shared knowledge and open dialogue in shaping the world. While there is a risk that some may use language models trained on this dataset for disinformation purposes, it's worth noting that the misuse of technology is a challenge that predates this dataset. The proverbial horse of misused technology has long since left the barn; our focus now must be on harnessing this shared intellectual legacy responsibly for the common good. ### Discussion of Biases


Given that Hacker News is a technology-focused platform with a largely self-selected user base, the content and perspectives found within this dataset may lean towards technology, entrepreneurship, and related fields, often reflecting the views and biases of this specific community. As such, users should be aware that analysis drawn from this data may not fully represent a balanced, global perspective and might contain inherent biases towards topics and viewpoints that are overrepresented in the Hacker News community.


Additional Information
---------------------- ### Licensing Information


In the absence of an explicit license for the upstream BigQuery dataset, this dataset uses the same MIT license as the Hacker News API. The upstream terms of use are reproduced here:



> 
> This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source - URL - and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.
> 
> 
>
passage: TAGS
#size_categories-10M<n<100M #language-English #license-mit #region-us 
### Dataset Summary


Dataset Name: Hacker News Full Corpus (2007 - November 2022)


Description:



> 
> NOTE: I am not affiliated with Y Combinator.
> 
> 
> 


This dataset is a July 2023 snapshot of YCombinator's BigQuery dump of the entire archive of posts and comments made on Hacker News. It contains posts from Hacker News' inception in 2007 through to November 16, 2022, when the BigQuery database was last updated.


The dataset does not incorporate any modifications or filtering - it is a raw dump from the original dataset provided by YCombinator. Hence, it retains the same structure and fields as the original BigQuery table, serving as a ready-to-use resource for conducting large-scale data analysis or training language models.


All credit for the original data collection and maintenance goes to YCombinator and the original post and comment authors. This version of the dataset has been prepared for convenience and ease of use within the HuggingFace ecosystem, especially for those interested in offline usage or who prefer not to use Google Cloud.


Please bear in mind that this dataset is a snapshot and will probably not be updated. For the latest data, consider accessing the live data directly from the official Hacker News API, potentially using Anant Narayanan's scripts.


Please use responsibly, respecting all relevant terms of use and privacy considerations inherent in the data.### Languages


English


Dataset Structure
-----------------### Data Fields



Dataset Creation
---------------- passage: ### Curation Rationale


This dataset provides a snapshot of the Hacker News posts and comments archive, sourced from YCombinator's open data, to enable easy and direct access without the need for a Google Cloud account or BigQuery interface, and without putting undue strain on the HN API. It aims to simplify the data acquisition process, promoting its use within the HuggingFace ecosystem for various tasks including analysis, trend prediction, sentiment studies, and language model training. By minimizing barriers to access, this dataset encourages a wider usage, fostering innovation in natural language processing and related fields.### Annotations### Personal and Sensitive Information


This dataset has not undergone specific checks for personally identifiable information (PII); hence, it's possible that some may exist within the data. However, as the data source is publicly available and shared by YCombinator, any potential PII present is already part of the public domain.


Considerations for Using the Data
---------------------------------### Social Impact of Dataset


The collective wisdom and perspectives captured in the posts and comments of this Hacker News dataset represent a unique gift from YCombinator and countless contributors worldwide; it is part of the common heritage of humanity. The potential insights to be gleaned and the future knowledge to be generated, especially through the training of language models on this corpus, can provide unbounded new perspectives, enriching our understanding and potential solutions to complex issues. It is a testament to the power of shared knowledge and open dialogue in shaping the world. While there is a risk that some may use language models trained on this dataset for disinformation purposes, it's worth noting that the misuse of technology is a challenge that predates this dataset. The proverbial horse of misused technology has long since left the barn; our focus now must be on harnessing this shared intellectual legacy responsibly for the common good.### Discussion of Biases


Given that Hacker News is a technology-focused platform with a largely self-selected user base, the content and perspectives found within this dataset may lean towards technology, entrepreneurship, and related fields, often reflecting the views and biases of this specific community. As such, users should be aware that analysis drawn from this data may not fully represent a balanced, global perspective and might contain inherent biases towards topics and viewpoints that are overrepresented in the Hacker News community.


Additional Information
----------------------