
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# clm_gpt2_scifi_webnovel2

This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on the None dataset.

## trained using custom dataset
```python
from datasets import Dataset

# Function to split text into chunks of 128 words
def split_text_into_chunks(text, chunk_size=128):
    words = text.split()
    chunks = [words[i:i+chunk_size] for i in range(0, len(words), chunk_size)]
    return [" ".join(chunk) for chunk in chunks]

# Read the text file
with open("train_clm_scifi_webnovel_data.txt", "r", encoding="latin-1") as file:
    text = file.read()

# Replace newline characters with double newline characters
text = text.replace("\n", "\\n")

# Split text into chunks
text_chunks = split_text_into_chunks(text)

# Create a dictionary for the dataset
dataset_dict = {"text": text_chunks}

# Create a Hugging Face dataset
dataset = Dataset.from_dict(dataset_dict)

dataset.push_to_hub("faizalnf1800/scifi_webnovel_preprocessed")

from datasets import load_dataset

dataset = load_dataset('faizalnf1800/scifi_webnovel_preprocessed')

# Define the custom pre-processing function
def preprocess_text(example):
    text = example['text']
    
    # Replace '\\n' with '\n'
    text = text.replace('\\n', '\n')
    
    # Replace '\\' with ''
    text = text.replace('\\', '')

    example['text'] = text
    return example

# Apply the pre-processing function using dataset.map()
dataset = dataset.map(preprocess_text)

dataset.push_to_hub("faizalnf1800/scifi_webnovel_preprocessed")
```
```python
from datasets import load_dataset

dataset = load_dataset("faizalnf1800/scifi_webnovel_preprocessed")

max_seq_length = 128
num_proc = 4

def tokenize_function(examples):
    # Remove empty lines
    examples["text"] = [line for line in examples["text"] if len(line) > 0 and not line.isspace()]
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=max_seq_length,
    )

tokenized_dataset = dataset.map(
    tokenize_function,
    batched=True,
    num_proc=num_proc,
    remove_columns=["text"],
)

# Main data processing function that will concatenate all texts from
# our dataset and generate chunks of max_seq_length.
def group_texts(examples):
    # Concatenate all texts.
    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    # We drop the small remainder, we could add padding if the model supported it instead of this drop,
    # you can customize this part to your needs.
    total_length = (total_length // max_seq_length) * max_seq_length
    # Split by chunks of max_len.
    result = {
        k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]
        for k, t in concatenated_examples.items()
    }
    return result

# Note that with `batched=True`, this map processes 1,000 texts together,
# so group_texts throws away a remainder for each of those groups of 1,000 texts.
# You can adjust that batch_size here but a higher value might be slower to preprocess.

tokenized_dataset = tokenized_dataset.map(
    group_texts,
    batched=True,
    num_proc=num_proc,
)
```

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.3
- training_steps: 180

### Training results



### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.0+cu118
- Datasets 2.15.0
- Tokenizers 0.15.0

{"license": "mit", "tags": ["generated_from_trainer"], "base_model": "gpt2", "model-index": [{"name": "clm_gpt2_scifi_webnovel2", "results": []}]}
transformers tensorboard safetensors gpt2 text-generation generated_from_trainer base_model:gpt2 license:mit autotrain_compatible endpoints_compatible has_space text-generation-inference region:us