training_guide.md

Quick training guide

Combine it with this guide, it's really helpful!

Fine-Tune ViT for Image Classification with Hugging Face Transformers

Start

pip install transformers datasets

Preparing the data:

Your data shouldn't look like this:

{
    "file_name": "train/aeae3547df6be819a42dcbb83e65586fd6deb424f134375c1dbc00188b37e2bf.jpeg", 
    "labels":  ["general", "furina (genshin impact)", "1girl", "ahoge", "bangs", "bare shoulders", ...]
}

But it should look more like this:

{
    "file_name": "train/aeae3547df6be819a42dcbb83e65586fd6deb424f134375c1dbc00188b37e2bf.jpeg", 
    "labels": ["0", "3028", "4", "702", "8", "9", "382", ...]
}

Where the labels should be represented as a list of integers (or anything you define as a number) that correspond to the tags you want to train with – essentially, they're the IDs of the labels.

Loading labels and their IDs:

import csv

with open("labels.csv", "r", encoding="utf-8") as f:
    reader = csv.reader(f)
    l = [row for row in reader]
    header = l[0]  # tag_id,name,category
    rows = l[1:]

id2labels = {}
labels2id = {}

for row in rows:
    id2labels[str(row[0])] = row[1]
    labels2id[row[1]] = str(row[0])

Where labels.csv is a file containing labels and their respective IDs.

Load dataset:

from datasets import load_dataset
dataset = load_dataset("./vit_dataset")

Congratulations! You've completed the toughest challenge. Why, you ask? Training this model took me a whole week just to gather and label the data.

Preprocess:

from transformers import ViTImageProcessor
import torch
model_name_or_path = 'google/vit-base-patch16-224-in21k'
processor = ViTImageProcessor.from_pretrained(model_name_or_path)

def transform(example_batch):
    inputs = processor([x for x in example_batch['image']], return_tensors='pt')

    inputs['labels'] = []
    inputs['label_names'] = [[id2labels[tagid] for tagid in x] for x in example_batch['labels']]
    
    for x in example_batch['labels']:
        x : list
        one_hot = [0 for x in range(0, len(labels2id.items()))]
        for index in x:
            one_hot[int(index)] = 1

        inputs['labels'] += [one_hot]


    return inputs
prepared_dataset = dataset.with_transform(transform)

Well, this code might not look pretty, but it gets the job done! As for the images (inputs), we resize them to 224x224 and flatten them out. Now, for the labels (target), we're transforming them into a multi-hot format. Why, you ask? Because I like it that way, and it's simple.

Training

These parts are relatively simple so I'll go quickly.

• Load dataset:

from torch.utils.data import DataLoader

batch_size = 16

def collate_fn(batch):
    data = {
        'pixel_values': torch.stack([x['pixel_values'] for x in batch]),
        'labels': torch.stack([torch.tensor(x['labels']) for x in batch]),
        'label_names' : [x['label_names'] for x in batch]
    }

    return data

train_dataloader = DataLoader(prepared_dataset['train'], collate_fn=collate_fn, batch_size=batch_size)

eval_dataloader = DataLoader(prepared_dataset['test'], collate_fn=collate_fn, batch_size=1)

• Initialize the model:

from transformers import ViTForImageClassification, ViTConfig

configuration  = ViTConfig(
    num_labels=len(id2labels.items()),
    id2label=id2labels,
    label2id=labels2id)
model = ViTForImageClassification(config=configuration)

Setup train:

device = torch.device('cuda')
test_steps = 5000
epochs = 50
mix_precision = torch.float16
global_steps = 0

optimizer = torch.optim.AdamW(model.parameters(), lr=0.0001)
lr_scheduler = torch.optim.lr_scheduler.LinearLR(optimizer=optimizer)

Test and Evaluation:

import torch
from transformers.modeling_outputs import ImageClassifierOutput

def test(eval_dataloader : DataLoader, model : ViTForImageClassification, device, t=0.7):
    batchs = list(iter(eval_dataloader))
    batch = batchs[0]

    with torch.no_grad():
        pixel_values = batch['pixel_values'].to(device=device)
        labels = batch['labels'].to(device=device, dtype=torch.float)

        outputs : ImageClassifierOutput = model(pixel_values=pixel_values)

        logits = outputs.logits
        sigmod = torch.nn.Sigmoid()
        logits : torch.FloatTensor = sigmod(logits)
        predictions = []

        for idx, p in enumerate(logits[0]):
            if p > t:
                predictions.append((model.config.id2label[idx], p.item()))

    print(f"label_names : {batch['label_names'][0]}")
    print(f"predictions : {predictions}")

def eval(eval_dataloader : DataLoader, model : ViTForImageClassification, device, t=0.7):

    result = {
        "eval_predictions" : 0,
        "eval_loss" : 0,
        "total_predictions" : 0,
        "total_loss" : 0
    }

    for batch in eval_dataloader:
        pixel_values = batch['pixel_values'].to(device=device)
        labels = batch['labels'].to(device=device, dtype=torch.float)
        label_names = batch['label_names'][0]

        prediction = 0
        with torch.no_grad():
        
            outputs : ImageClassifierOutput = model(pixel_values=pixel_values, labels=labels)

        logits = outputs.logits
        loss = outputs.loss
        predictions = []

        for idx, p in enumerate(logits[0]):
            if p > t:
                predictions.append(model.config.id2label[idx])
        
        for p in predictions:
            if p in label_names:
                prediction += 1 / len(label_names)
        
        result['total_predictions'] += prediction
        result['total_loss'] += loss.item()

    result['eval_predictions'] = result['total_predictions'] / len(eval_dataloader)
    result['eval_loss'] = result['total_loss'] / len(eval_dataloader)
    print(result)

Train:

import tqdm
from transformers.modeling_outputs import ImageClassifierOutput

process_bar = tqdm.tqdm(total=epochs * len(train_dataloader))

for e in range(1, epochs + 1):
    model.train()

    total_loss = 0

    for idx, (batch) in enumerate(train_dataloader):

        pixel_values = batch['pixel_values'].to(device=device)
        labels = batch['labels'].to(device=device, dtype=torch.float)
        
        with torch.autocast(device_type=str(device), dtype=mix_precision):
            outputs : ImageClassifierOutput = model(pixel_values=pixel_values, labels=labels)
        
        loss = outputs.loss
        total_loss += loss.detach().float()

        loss.backward()
        if torch.isnan(loss):
            assert False, "NaN detection."

        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()

        process_bar.update(1)
        process_bar.desc = f"{model.config.problem_type} - Epoch: {e}/{epochs}"
        process_bar.set_postfix({'loss' : f'{loss.item():.5f}', "train_loss" : total_loss.item() / len(train_dataloader)})

        if global_steps % test_steps == 0 and global_steps > 1:
            model.eval()
            process_bar.desc = f"Evalute - Epoch: {e}/{epochs}"
            eval(eval_dataloader=eval_dataloader, model=model, device=device, t=0.3)
            test(eval_dataloader, model, device, 0.3)
            model.train()

        global_steps += 1

Thank you for reading through all this verbose stuff. Of course, all the code above is impromptu; there might be some inconsistencies. Your contributions are highly appreciated.