license:cc-by-nc-4.0 region:us
---
license: cc-by-nc-4.0
---

Dataset preprocessed from https://huggingface.co/datasets/Yukang/LongAlpaca-12k.

This contains 1000 samples that have a minimum length of 16k tokens.

## Script to reproduce

```python
from datasets import load_dataset
from transformers import AutoTokenizer
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

# Load the dataset and tokenizer
data = load_dataset("Yukang/LongAlpaca-12k")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", trust_remote_code=True)

def filter_function(batch):
    # Separate each round of conversation and concatenate them into single strings
    conversation_strs = [f'{instruction}\n\n{output}' for instruction, output in zip(batch['instruction'], batch['output'])]
    
    # Tokenize the strings without truncation
    tokens = tokenizer(conversation_strs, truncation=False, return_length=True)
    
    # Return True for examples where the token count exceeds max
    return [length > 16384 for length in tokens['length']]

# Note that I've added a "keep" key to the return dictionary
filtered_data = data.filter(filter_function, batched=True, batch_size=1000)

# Convert to Pandas DataFrame
df = pd.DataFrame(filtered_data['train'])

# Sample 1k rows
sampled_df = df.sample(n=1000, random_state=1)

# Convert the Pandas DataFrame to a PyArrow Table
table = pa.table(sampled_df)

# Save the table as a Parquet file
pq.write_table(table, 'data.parquet')

```