Finetuning clip can be done locally with decent results (even if you are GPU poor).

<p>

this is the journal of me following clipfinetune I have already posted this on medium but trying to slowly migrate my stuff here</p>
<pre><code class="language-python"><span class="hljs-keyword">import</span> os
<span class="hljs-keyword">import</span> datasets
<span class="hljs-keyword">from</span> dataclasses <span class="hljs-keyword">import</span> dataclass, field
<span class="hljs-keyword">from</span> typing <span class="hljs-keyword">import</span> <span class="hljs-type">Optional</span>
<span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt
<span class="hljs-keyword">import</span> requests
<span class="hljs-keyword">import</span> random
<span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">import</span> torch
<span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset
<span class="hljs-keyword">from</span> PIL <span class="hljs-keyword">import</span> Image
<span class="hljs-keyword">from</span> torchvision.io <span class="hljs-keyword">import</span> ImageReadMode, read_image
<span class="hljs-keyword">from</span> torchvision.transforms <span class="hljs-keyword">import</span> CenterCrop, ConvertImageDtype, Normalize, Resize
<span class="hljs-keyword">from</span> torchvision.transforms.functional <span class="hljs-keyword">import</span> InterpolationMode
<span class="hljs-keyword">from</span> pdb <span class="hljs-keyword">import</span> set_trace

<span class="hljs-keyword">import</span> transformers
<span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> (
    VisionTextDualEncoderProcessor,
    VisionTextDualEncoderModel,
    AutoImageProcessor,
    AutoModel,
    AutoTokenizer,
    HfArgumentParser,
    Trainer,
    TrainingArguments,
    set_seed,
)
<span class="hljs-keyword">from</span> transformers.trainer_utils <span class="hljs-keyword">import</span> get_last_checkpoint
<span class="hljs-keyword">from</span> transformers.utils <span class="hljs-keyword">import</span> check_min_version, send_example_telemetry
<span class="hljs-keyword">from</span> transformers.utils.versions <span class="hljs-keyword">import</span> require_version

<span class="hljs-keyword">import</span> warnings
warnings.filterwarnings(<span class="hljs-string">'ignore'</span>, category=UserWarning, module=<span class="hljs-string">'torchvision'</span>)

<span class="hljs-comment"># Will error if the minimal version of Transformers is not installed. Remove at your own risks.</span>
check_min_version(<span class="hljs-string">"4.31.0.dev0"</span>)
require_version(<span class="hljs-string">"datasets&gt;=1.8.0"</span>, <span class="hljs-string">"To fix: pip install -r examples/pytorch/contrastive-image-text/requirements.txt"</span>)
</code></pre>
<pre><code>c:\Users\kevol\anaconda3\envs\pytorch\lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
</code></pre>
<p>Image Encoder for our CLIP is clip-vit-base-patch32 and Text Encoder is roberta-base</p>
<pre><code class="language-python">model = VisionTextDualEncoderModel.from_vision_text_pretrained(
    <span class="hljs-string">"openai/clip-vit-base-patch32"</span>, <span class="hljs-string">"roberta-base"</span>
)

tokenizer = AutoTokenizer.from_pretrained(<span class="hljs-string">"roberta-base"</span>)
image_processor = AutoImageProcessor.from_pretrained(<span class="hljs-string">"openai/clip-vit-base-patch32"</span>)
processor = VisionTextDualEncoderProcessor(image_processor, tokenizer)

model.save_pretrained(<span class="hljs-string">"clip-roberta"</span>)
processor.save_pretrained(<span class="hljs-string">"clip-roberta"</span>)
</code></pre>
<pre><code>config.json: 100%|██████████| 4.19k/4.19k [00:00&lt;00:00, 1.18MB/s]
c:\Users\kevol\anaconda3\envs\pytorch\lib\site-packages\huggingface_hub\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\kevol\.cache\huggingface\hub\models--openai--clip-vit-base-patch32. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
pytorch_model.bin: 100%|██████████| 605M/605M [00:20&lt;00:00, 29.5MB/s] 
config.json: 100%|██████████| 481/481 [00:00&lt;00:00, 41.7kB/s]
c:\Users\kevol\anaconda3\envs\pytorch\lib\site-packages\huggingface_hub\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\kevol\.cache\huggingface\hub\models--roberta-base. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
model.safetensors: 100%|██████████| 499M/499M [00:16&lt;00:00, 30.0MB/s] 
Some weights of RobertaModel were not initialized from the model checkpoint at roberta-base and are newly initialized: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
The projection layer and logit scale weights `['visual_projection.weight', 'text_projection.weight', 'logit_scale']` are newly initialized. You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
vocab.json: 100%|██████████| 899k/899k [00:00&lt;00:00, 2.38MB/s]
merges.txt: 100%|██████████| 456k/456k [00:00&lt;00:00, 1.84MB/s]
tokenizer.json: 100%|██████████| 1.36M/1.36M [00:00&lt;00:00, 10.6MB/s]
preprocessor_config.json: 100%|██████████| 316/316 [00:00&lt;00:00, 100kB/s]
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
</code></pre>
<p>We define two argument classes: ModelArguments and HfArgumentParser. This allows us to utilize Hugging Face’s HfArgumentParser in conjunction with the default TrainingArguments from Hugging Face.</p>
<pre><code class="language-python"><span class="hljs-meta">@dataclass</span>
<span class="hljs-keyword">class</span> <span class="hljs-title class_">ModelArguments</span>:
    <span class="hljs-string">"""</span>
<span class="hljs-string">    Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch.</span>
<span class="hljs-string">    """</span>

    model_name_or_path: <span class="hljs-built_in">str</span> = field(
        metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"Path to pretrained model or model identifier from huggingface.co/models"</span>},
    )
    config_name: <span class="hljs-type">Optional</span>[<span class="hljs-built_in">str</span>] = field(
        default=<span class="hljs-literal">None</span>, metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"Pretrained config name or path if not the same as model_name"</span>}
    )
    tokenizer_name: <span class="hljs-type">Optional</span>[<span class="hljs-built_in">str</span>] = field(
        default=<span class="hljs-literal">None</span>, metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"Pretrained tokenizer name or path if not the same as model_name"</span>}
    )
    image_processor_name: <span class="hljs-built_in">str</span> = field(default=<span class="hljs-literal">None</span>, metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"Name or path of preprocessor config."</span>})
    cache_dir: <span class="hljs-type">Optional</span>[<span class="hljs-built_in">str</span>] = field(
        default=<span class="hljs-literal">None</span>, metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"Where do you want to store the pretrained models downloaded from s3"</span>}
    )
    model_revision: <span class="hljs-built_in">str</span> = field(
        default=<span class="hljs-string">"main"</span>,
        metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"The specific model version to use (can be a branch name, tag name or commit id)."</span>},
    )
    use_fast_tokenizer: <span class="hljs-built_in">bool</span> = field(
        default=<span class="hljs-literal">True</span>,
        metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."</span>},
    )
    use_auth_token: <span class="hljs-built_in">bool</span> = field(
        default=<span class="hljs-literal">False</span>,
        metadata={
            <span class="hljs-string">"help"</span>: (
                <span class="hljs-string">"Will use the token generated when running `huggingface-cli login` (necessary to use this script "</span>
                <span class="hljs-string">"with private models)."</span>
            )
        },
    )
</code></pre>
<pre><code class="language-python"><span class="hljs-meta">@dataclass</span>
<span class="hljs-keyword">class</span> <span class="hljs-title class_">DataTrainingArguments</span>:
    <span class="hljs-string">"""</span>
<span class="hljs-string">    Arguments pertaining to what data we are going to input our model for training and eval.</span>
<span class="hljs-string">    """</span>
    dataset_name: <span class="hljs-type">Optional</span>[<span class="hljs-built_in">str</span>] = field(
        default=<span class="hljs-literal">None</span>, metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"The name of the dataset to use (via the datasets library)."</span>}
    )
    data_dir: <span class="hljs-type">Optional</span>[<span class="hljs-built_in">str</span>] = field(default=<span class="hljs-literal">None</span>, metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"The data directory containing input files."</span>})
    image_column: <span class="hljs-type">Optional</span>[<span class="hljs-built_in">str</span>] = field(
        default=<span class="hljs-string">"image_path"</span>,
        metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"The name of the column in the datasets containing the full image file paths."</span>},
    )
    caption_column: <span class="hljs-type">Optional</span>[<span class="hljs-built_in">str</span>] = field(
        default=<span class="hljs-string">"caption"</span>,
        metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"The name of the column in the datasets containing the image captions."</span>},
    )
    max_seq_length: <span class="hljs-type">Optional</span>[<span class="hljs-built_in">int</span>] = field(
        default=<span class="hljs-number">128</span>,
        metadata={
            <span class="hljs-string">"help"</span>: (
                <span class="hljs-string">"The maximum total input sequence length after tokenization. Sequences longer "</span>
                <span class="hljs-string">"than this will be truncated, sequences shorter will be padded."</span>
            )
        },
    )
    overwrite_cache: <span class="hljs-built_in">bool</span> = field(
        default=<span class="hljs-literal">False</span>, metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"Overwrite the cached training and evaluation sets"</span>}
    )
    preprocessing_num_workers: <span class="hljs-type">Optional</span>[<span class="hljs-built_in">int</span>] = field(
        default=<span class="hljs-literal">None</span>,
        metadata={<span class="hljs-string">"help"</span>: <span class="hljs-string">"The number of processes to use for the preprocessing."</span>},
    )
</code></pre>
<pre><code class="language-python">args_dict = {<span class="hljs-string">'output_dir'</span>: <span class="hljs-string">'./clip-roberta-finetuned'</span>,
 <span class="hljs-string">'model_name_or_path'</span>: <span class="hljs-string">'./clip-roberta'</span>,
 <span class="hljs-string">'data_dir'</span>: <span class="hljs-string">'./data'</span>,
 <span class="hljs-string">'dataset_name'</span>: <span class="hljs-string">'arampacha/rsicd'</span>,
 <span class="hljs-string">'image_column'</span>: <span class="hljs-string">'image'</span>,
 <span class="hljs-string">'caption_column'</span>: <span class="hljs-string">'captions'</span>,
 <span class="hljs-string">'remove_unused_columns'</span>: <span class="hljs-literal">False</span>,
 <span class="hljs-string">'per_device_train_batch_size'</span>: <span class="hljs-number">64</span>,
 <span class="hljs-string">'per_device_eval_batch_size'</span>: <span class="hljs-number">64</span>,
 <span class="hljs-string">'learning_rate'</span>: <span class="hljs-number">5e-05</span>,
 <span class="hljs-string">'warmup_steps'</span>: <span class="hljs-number">0</span>,
 <span class="hljs-string">'weight_decay'</span>: <span class="hljs-number">0.1</span>,
 <span class="hljs-string">'overwrite_output_dir'</span>: <span class="hljs-literal">True</span>,
 <span class="hljs-string">'push_to_hub'</span>: <span class="hljs-literal">False</span>}

parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
model_args, data_args, training_args = parser.parse_dict(args_dict)
</code></pre>
<pre><code class="language-python"><span class="hljs-comment"># Dataset</span>
<span class="hljs-keyword">class</span> <span class="hljs-title class_">Transform</span>(torch.nn.Module):
    <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, image_size, mean, std</span>):
        <span class="hljs-built_in">super</span>().__init__()
        self.transforms = torch.nn.Sequential(
            Resize([image_size], interpolation=InterpolationMode.BICUBIC, antialias=<span class="hljs-literal">True</span>),
            CenterCrop(image_size),
            ConvertImageDtype(torch.<span class="hljs-built_in">float</span>),
            Normalize(mean, std),
        )
    <span class="hljs-keyword">def</span> <span class="hljs-title function_">forward</span>(<span class="hljs-params">self, x</span>) -&gt; torch.Tensor:
        <span class="hljs-string">"""`x` should be an instance of `PIL.Image.Image`"""</span>
        <span class="hljs-keyword">with</span> torch.no_grad():
            x = self.transforms(x)
        <span class="hljs-keyword">return</span> x
</code></pre>
<pre><code class="language-python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">collate_fn</span>(<span class="hljs-params">examples</span>):
    pixel_values = torch.stack([example[<span class="hljs-string">"pixel_values"</span>] <span class="hljs-keyword">for</span> example <span class="hljs-keyword">in</span> examples])
    input_ids = torch.tensor([example[<span class="hljs-string">"input_ids"</span>] <span class="hljs-keyword">for</span> example <span class="hljs-keyword">in</span> examples], dtype=torch.long)
    attention_mask = torch.tensor([example[<span class="hljs-string">"attention_mask"</span>] <span class="hljs-keyword">for</span> example <span class="hljs-keyword">in</span> examples], dtype=torch.long)
    <span class="hljs-keyword">return</span> {
        <span class="hljs-string">"pixel_values"</span>: pixel_values,
        <span class="hljs-string">"input_ids"</span>: input_ids,
        <span class="hljs-string">"attention_mask"</span>: attention_mask,
        <span class="hljs-string">"return_loss"</span>: <span class="hljs-literal">True</span>,
    }
</code></pre>
<pre><code class="language-python">dataset = datasets.load_dataset(<span class="hljs-string">"arampacha/rsicd"</span>)
</code></pre>
<pre><code>Downloading metadata: 100%|██████████| 1.03k/1.03k [00:00&lt;?, ?B/s]
Downloading data: 100%|██████████| 419M/419M [01:28&lt;00:00, 4.73MB/s]
Downloading data: 100%|██████████| 55.1M/55.1M [00:10&lt;00:00, 5.43MB/s]
Downloading data: 100%|██████████| 51.6M/51.6M [00:09&lt;00:00, 5.18MB/s]
Downloading data files: 100%|██████████| 3/3 [01:48&lt;00:00, 36.25s/it]
Extracting data files: 100%|██████████| 3/3 [00:00&lt;00:00, 117.53it/s]
Generating train split: 100%|██████████| 8734/8734 [00:02&lt;00:00, 3624.40 examples/s]
Generating test split: 100%|██████████| 1093/1093 [00:00&lt;00:00, 3410.02 examples/s]
Generating valid split: 100%|██████████| 1094/1094 [00:00&lt;00:00, 4414.00 examples/s]
</code></pre>
<pre><code class="language-python">dataset
</code></pre>
<pre><code>DatasetDict({
    train: Dataset({
        features: ['filename', 'captions', 'image'],
        num_rows: 8734
    })
    test: Dataset({
        features: ['filename', 'captions', 'image'],
        num_rows: 1093
    })
    valid: Dataset({
        features: ['filename', 'captions', 'image'],
        num_rows: 1094
    })
})
</code></pre>
<pre><code class="language-python"><span class="hljs-comment"># Seeing an example</span>
<span class="hljs-keyword">def</span> <span class="hljs-title function_">show_images</span>(<span class="hljs-params">dset, num_images=<span class="hljs-number">8</span>, without_caption=<span class="hljs-literal">True</span>,num_columns=<span class="hljs-number">2</span>,img_size=(<span class="hljs-params"><span class="hljs-number">4</span>, <span class="hljs-number">4</span></span>)</span>):
    num_rows = -(-num_images // num_columns)  <span class="hljs-comment"># Ceiling division</span>
    fig = plt.figure(figsize=(img_size[<span class="hljs-number">0</span>] * num_columns, img_size[<span class="hljs-number">1</span>] * num_rows))

    _<span class="hljs-built_in">list</span> = <span class="hljs-built_in">list</span>(<span class="hljs-built_in">range</span>(<span class="hljs-built_in">len</span>(dset)))
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(num_images):
        index = _<span class="hljs-built_in">list</span>[i]
        ax = fig.add_subplot(num_rows, num_columns, i+<span class="hljs-number">1</span>)
        image = dset[index][<span class="hljs-string">'image'</span>]
        plt.imshow(image)
        
        <span class="hljs-comment"># Set title as the first caption</span>
        <span class="hljs-keyword">if</span> without_caption:
            caption = dset[index][<span class="hljs-string">'captions'</span>][<span class="hljs-number">0</span>]
            ax.set_title(caption, fontsize=<span class="hljs-number">10</span>)
        
        <span class="hljs-comment"># Remove axis</span>
        plt.axis(<span class="hljs-string">'off'</span>)
    plt.tight_layout()
    plt.subplots_adjust(wspace=<span class="hljs-number">0.5</span>, hspace=<span class="hljs-number">0.01</span>)  <span class="hljs-comment"># Adjust these values as needed</span>

    plt.show()
</code></pre>
<pre><code class="language-python">show_images(dataset[<span class="hljs-string">'train'</span>], num_images=<span class="hljs-number">4</span>, without_caption=<span class="hljs-literal">True</span>)
</code></pre>
<p><a href="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_13_0.png" rel="nofollow"><img alt="png" src="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_13_0.png"/></a></p>
<pre><code class="language-python">tokenizer = AutoTokenizer.from_pretrained(
    model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer
)
</code></pre>
<pre><code class="language-python">image_processor = AutoImageProcessor.from_pretrained(
    model_args.image_processor_name <span class="hljs-keyword">or</span> model_args.model_name_or_path,
    cache_dir=model_args.cache_dir,
    revision=model_args.model_revision,
    use_auth_token=<span class="hljs-literal">True</span> <span class="hljs-keyword">if</span> model_args.use_auth_token <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
)

model = AutoModel.from_pretrained(
    model_args.model_name_or_path,
    cache_dir=model_args.cache_dir,
    revision=model_args.model_revision,
    use_auth_token=<span class="hljs-literal">True</span> <span class="hljs-keyword">if</span> model_args.use_auth_token <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
)
config = model.config
</code></pre>
<pre><code class="language-python">set_seed(training_args.seed)
</code></pre>
<pre><code class="language-python">image_transformations = Transform(
    config.vision_config.image_size, image_processor.image_mean, image_processor.image_std
)
image_transformations = torch.jit.script(image_transformations)
</code></pre>
<pre><code class="language-python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">tokenize_captions</span>(<span class="hljs-params">examples</span>):
    captions = [example[<span class="hljs-number">0</span>] <span class="hljs-keyword">for</span> example <span class="hljs-keyword">in</span> examples[data_args.caption_column]]
    text_inputs = tokenizer(captions, max_length=data_args.max_seq_length, padding=<span class="hljs-string">"max_length"</span>, truncation=<span class="hljs-literal">True</span>)
    examples[<span class="hljs-string">"input_ids"</span>] = text_inputs.input_ids
    examples[<span class="hljs-string">"attention_mask"</span>] = text_inputs.attention_mask
    <span class="hljs-keyword">return</span> examples

<span class="hljs-keyword">def</span> <span class="hljs-title function_">transform_images</span>(<span class="hljs-params">examples</span>):
    images = [torch.tensor(np.array(image)).permute(<span class="hljs-number">2</span>, <span class="hljs-number">0</span>, <span class="hljs-number">1</span>) <span class="hljs-keyword">for</span> image <span class="hljs-keyword">in</span> examples[data_args.image_column]]
    examples[<span class="hljs-string">"pixel_values"</span>] = [image_transformations(image) <span class="hljs-keyword">for</span> image <span class="hljs-keyword">in</span> images]
    <span class="hljs-keyword">return</span> examples

<span class="hljs-keyword">def</span> <span class="hljs-title function_">filter_corrupt_images</span>(<span class="hljs-params">examples</span>):
    <span class="hljs-string">"""remove problematic images"""</span>
    valid_images = []
    <span class="hljs-keyword">for</span> image_file <span class="hljs-keyword">in</span> examples[data_args.image_column]:
        <span class="hljs-keyword">try</span>:
            Image.<span class="hljs-built_in">open</span>(image_file)
            valid_images.append(<span class="hljs-literal">True</span>)
        <span class="hljs-keyword">except</span> Exception:
            valid_images.append(<span class="hljs-literal">False</span>)
    <span class="hljs-keyword">return</span> valid_images
</code></pre>
<pre><code class="language-python">train_dataset = dataset[<span class="hljs-string">"train"</span>]
train_dataset = train_dataset.<span class="hljs-built_in">map</span>(
    function=tokenize_captions,
    batched=<span class="hljs-literal">True</span>,
    num_proc=data_args.preprocessing_num_workers,
    load_from_cache_file=<span class="hljs-keyword">not</span> data_args.overwrite_cache,
    desc=<span class="hljs-string">"Running tokenizer on train dataset"</span>,
)
train_dataset.set_transform(transform_images)
</code></pre>
<pre><code>Running tokenizer on train dataset: 100%|██████████| 8734/8734 [00:01&lt;00:00, 6126.48 examples/s]
Parameter 'transform'=&lt;function transform_images at 0x0000019A476F09D0&gt; of the transform datasets.arrow_dataset.Dataset.set_format couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
</code></pre>
<pre><code class="language-python">train_dataset
</code></pre>
<pre><code>Dataset({
    features: ['filename', 'captions', 'image', 'input_ids', 'attention_mask'],
    num_rows: 8734
})
</code></pre>
<pre><code class="language-python">eval_dataset = dataset[<span class="hljs-string">"valid"</span>]
eval_dataset = eval_dataset.<span class="hljs-built_in">map</span>(
    function=tokenize_captions,
    batched=<span class="hljs-literal">True</span>,
    num_proc=data_args.preprocessing_num_workers,
    load_from_cache_file=<span class="hljs-keyword">not</span> data_args.overwrite_cache,
    desc=<span class="hljs-string">"Running tokenizer on validation dataset"</span>,
)
eval_dataset.set_transform(transform_images)
</code></pre>
<pre><code>Running tokenizer on validation dataset: 100%|██████████| 1094/1094 [00:00&lt;00:00, 3151.84 examples/s]
</code></pre>
<pre><code class="language-python">train_dataset, eval_dataset
</code></pre>
<pre><code>(Dataset({
     features: ['filename', 'captions', 'image', 'input_ids', 'attention_mask'],
     num_rows: 8734
 }),
 Dataset({
     features: ['filename', 'captions', 'image', 'input_ids', 'attention_mask'],
     num_rows: 1094
 }))
</code></pre>
<pre><code class="language-python">processor =  VisionTextDualEncoderProcessor(image_processor, tokenizer)
</code></pre>
<pre><code class="language-python"><span class="hljs-comment"># example from hugging face</span>
urls = [
    <span class="hljs-string">"http://images.cocodataset.org/val2017/000000039769.jpg"</span>,
    <span class="hljs-string">"https://farm3.staticflickr.com/2674/5850229113_4fe05d5265_z.jpg"</span>,
]
images = [Image.<span class="hljs-built_in">open</span>(requests.get(url, stream=<span class="hljs-literal">True</span>).raw) <span class="hljs-keyword">for</span> url <span class="hljs-keyword">in</span> urls]
inputs = processor(
    text=[<span class="hljs-string">"a photo of a cat"</span>], images=images, return_tensors=<span class="hljs-string">"pt"</span>, padding=<span class="hljs-literal">True</span>
)
inputs[<span class="hljs-string">'input_ids'</span>] = inputs[<span class="hljs-string">'input_ids'</span>].cuda()
inputs[<span class="hljs-string">'attention_mask'</span>] = inputs[<span class="hljs-string">'attention_mask'</span>].cuda()
inputs[<span class="hljs-string">'pixel_values'</span>] = inputs[<span class="hljs-string">'pixel_values'</span>].cuda()
model = model.cuda()
outputs = model(**inputs)
logits_per_image = outputs.logits_per_image 
</code></pre>
<pre><code class="language-python">logits_per_image
</code></pre>
<pre><code>tensor([[-0.4522],
        [ 0.7341]], device='cuda:0', grad_fn=&lt;PermuteBackward0&gt;)
</code></pre>
<pre><code class="language-python">images[<span class="hljs-number">0</span>]
</code></pre>
<p><a href="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_26_0.png" rel="nofollow"><img alt="png" src="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_26_0.png"/></a></p>
<pre><code class="language-python">images[<span class="hljs-number">1</span>]
</code></pre>
<p><a href="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_27_0.png" rel="nofollow"><img alt="png" src="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_27_0.png"/></a></p>
<pre><code class="language-python"><span class="hljs-comment"># base model prediction</span>
np.random.seed(<span class="hljs-number">0</span>)
indices = np.random.choice(<span class="hljs-built_in">len</span>(dataset[<span class="hljs-string">'valid'</span>]), <span class="hljs-number">8</span>, replace=<span class="hljs-literal">False</span>)
patches = dataset[<span class="hljs-string">'valid'</span>].select(indices.tolist())
</code></pre>
<pre><code class="language-python">show_images(patches, <span class="hljs-number">8</span>, without_caption=<span class="hljs-literal">False</span>, num_columns=<span class="hljs-number">4</span>,img_size=(<span class="hljs-number">3</span>, <span class="hljs-number">3</span>))
</code></pre>
<p><a href="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_29_0.png" rel="nofollow"><img alt="png" src="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_29_0.png"/></a></p>
<pre><code class="language-python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">show_result</span>(<span class="hljs-params">model, patches, text, top_n = <span class="hljs-number">3</span></span>):
    images = [patch[<span class="hljs-string">'image'</span>] <span class="hljs-keyword">for</span> patch <span class="hljs-keyword">in</span> patches]
    inputs = processor(text=[text], images=images, return_tensors=<span class="hljs-string">"pt"</span>, padding=<span class="hljs-literal">True</span>)
    inputs[<span class="hljs-string">'input_ids'</span>] = inputs[<span class="hljs-string">'input_ids'</span>].cuda()
    inputs[<span class="hljs-string">'attention_mask'</span>] = inputs[<span class="hljs-string">'attention_mask'</span>].cuda()
    inputs[<span class="hljs-string">'pixel_values'</span>] = inputs[<span class="hljs-string">'pixel_values'</span>].cuda()
    
    model = model.cuda()
    outputs = model(**inputs)
    logits_per_image = outputs.logits_per_image
    sorted_idx = (torch.sort(logits_per_image, dim=<span class="hljs-number">0</span>, descending=<span class="hljs-literal">True</span>)[<span class="hljs-number">1</span>][:,<span class="hljs-number">0</span>]).tolist()
    sorted_idx = sorted_idx[:top_n]
    patches_sorted = patches.select(sorted_idx)
    show_images(patches_sorted, num_images=<span class="hljs-built_in">len</span>(patches_sorted), without_caption=<span class="hljs-literal">False</span>, num_columns=<span class="hljs-number">1</span>, img_size=(<span class="hljs-number">3</span>,<span class="hljs-number">3</span>))
</code></pre>
<pre><code class="language-python">show_result(model, patches, <span class="hljs-string">'more than one plane'</span>)
</code></pre>
<p><a href="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_31_0.png" rel="nofollow"><img alt="png" src="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_31_0.png"/></a></p>
<p>This was my first time training using HF trainer, feels a little overwhelming but I feel like If i do more and more after I get a good gpu, it will be easier than normal finetuning</p>
<pre><code class="language-python"><span class="hljs-comment"># 8. Initalize our trainer</span>
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=collate_fn,
)

<span class="hljs-comment"># 9. Training</span>
train_result = trainer.train()
trainer.log_metrics(<span class="hljs-string">"train"</span>, train_result.metrics)
metrics = trainer.evaluate()
trainer.log_metrics(<span class="hljs-string">"eval"</span>, metrics)
</code></pre>
<pre><code>  0%|          | 0/411 [00:00&lt;?, ?it/s]c:\Users\kevol\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py:1501: UserWarning: operator () profile_node %376 : int = prim::profile_ivalue(%out_dtype.1)
 does not have profile information (Triggered internally at C:\cb\pytorch_1000000000000\work\third_party\nvfuser\csrc\graph_fuser.cpp:108.)
  return forward_call(*args, **kwargs)
100%|██████████| 411/411 [1:19:45&lt;00:00, 11.64s/it]


{'train_runtime': 4785.2548, 'train_samples_per_second': 5.476, 'train_steps_per_second': 0.086, 'train_loss': 1.8650288454227495, 'epoch': 3.0}
***** train metrics *****
  epoch                    =        3.0
  train_loss               =      1.865
  train_runtime            = 1:19:45.25
  train_samples_per_second =      5.476
  train_steps_per_second   =      0.086


100%|██████████| 18/18 [00:20&lt;00:00,  1.16s/it]

***** eval metrics *****
  epoch                   =        3.0
  eval_loss               =     3.8813
  eval_runtime            = 0:00:22.34
  eval_samples_per_second =     48.959
  eval_steps_per_second   =      0.806
</code></pre>
<pre><code class="language-python">show_result(model, patches, <span class="hljs-string">'green trees'</span>)
</code></pre>
<p><a href="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_34_0.png" rel="nofollow"><img alt="png" src="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_34_0.png"/></a></p>
<pre><code class="language-python">show_result(model, patches, <span class="hljs-string">'airplane'</span>)
</code></pre>
<p><a href="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_35_0.png" rel="nofollow"><img alt="png" src="https://raw.githubusercontent.com/KevorkSulahian/Non-Quant-DL/main/clip_finetune_files/clip_finetune_35_0.png"/></a></p>
<p>Honestly not the best but I also didn't train for long, the result or numbers aren't the important part here but rather I wanted to focus on how I can finetune</p>
