Upload 3 files

Codes/01 - Fine-tuning.ipynb

@@ -0,0 +1,844 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9cfae24c-01e5-480b-ad5e-2aec772e0983",
+   "metadata": {},
+   "source": [
+    "### Checking for availability of GPUs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8229d250-fd3c-4ee6-ba55-dcfbeba9b06e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!nvidia-smi"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8b9081a3-e5bb-45c5-a88b-4ef4e0490574",
+   "metadata": {},
+   "source": [
+    "### Installing the necessary libraries\n",
+    "Please note that this installation step might fail if no GPU is available."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a9c4d533-ec98-4b0f-a263-c3c8f649547e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install --upgrade pip\n",
+    "!pip install -q -U bitsandbytes\n",
+    "!pip install -q -U git+https://github.com/huggingface/transformers.git\n",
+    "!pip install -q -U git+https://github.com/huggingface/peft.git\n",
+    "!pip install -q -U git+https://github.com/huggingface/accelerate.git\n",
+    "!pip install -q trl xformers wandb datasets einops gradio sentencepiece\n",
+    "!pip install flash-attn"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3425af9b-12cd-46e5-80f5-30fd2f7fed67",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Libraries used for analyzing the dataset\n",
+    "!pip install matplotlib\n",
+    "!pip install statistics"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b0ddd32e-2c21-4abd-8e5c-db85efb194ab",
+   "metadata": {},
+   "source": [
+    "### Importing the necessary libraries and preparation\n",
+    "Here, we set the repos of the base model (to be fine-tuned), the dataset (on which the base model will be fine-tuned) and the output model (the base model fine-tuned on our custom dataset)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1eb19e95-6c02-45a5-99a3-5bb966ed2d0b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import (\n",
+    "    AutoModelForCausalLM, \n",
+    "    AutoTokenizer, \n",
+    "    BitsAndBytesConfig,\n",
+    "    HfArgumentParser,\n",
+    "    TrainingArguments,\n",
+    "    pipeline, \n",
+    "    logging, \n",
+    "    TextStreamer,\n",
+    "    GenerationConfig\n",
+    ")\n",
+    "from peft import (\n",
+    "    LoraConfig, \n",
+    "    PeftModel, \n",
+    "    prepare_model_for_kbit_training, \n",
+    "    get_peft_model\n",
+    ")\n",
+    "import os\n",
+    "import torch\n",
+    "import wandb\n",
+    "import platform\n",
+    "import warnings\n",
+    "from datasets import load_dataset\n",
+    "from trl import SFTTrainer\n",
+    "from huggingface_hub import notebook_login\n",
+    "\n",
+    "# Base model\n",
+    "base_model = \"mistralai/Mistral-Nemo-Instruct-2407\"\n",
+    "\n",
+    "# Custom dataset and new model to be created\n",
+    "dataset_name = \"dataset/repo\"\n",
+    "\n",
+    "# New model to be created\n",
+    "new_model = \"model/repo\"\n",
+    "\n",
+    "# Ignore all warnings\n",
+    "warnings.filterwarnings(\"ignore\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d4c91d86-47fc-4ff4-bac4-9a2006138d0a",
+   "metadata": {},
+   "source": [
+    "### Logging into Hugging Face\n",
+    "Since both our dataset and the model we want to create are (meant to be) private, it is better to already log into our HF account."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a7593199-8ec5-433b-b195-0393a8ebe2eb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# We already log into our own hugging face hub with a 'WRITE' token\n",
+    "!huggingface-cli login --token MY_HF_WRITE_TOKEN"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6effec67-70fa-4362-b0e2-6e9adb80ae18",
+   "metadata": {},
+   "source": [
+    "*Remark:* an alternative is to simply write\n",
+    "```\n",
+    "notebook_login()\n",
+    "```\n",
+    "and then type the HF 'WRITE' token.\n",
+    "\n",
+    "### Loading the dataset\n",
+    "\n",
+    "Our dataset here is of the form:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e96f2849-3059-42d4-8753-abe2c2fffdf1",
+   "metadata": {},
+   "source": [
+    "```json\n",
+    "   [\n",
+    "       {\n",
+    "           \"source\": \"Chat no 1-2\",\n",
+    "           \"text\": [\n",
+    "               {\"role\": \"user\", \"content\": \"My system prompt\\n\\nQuestion 11\"},\n",
+    "               {\"role\": \"assistant\", \"content\": \"Answer 11\"}\n",
+    "           ]\n",
+    "       },\n",
+    "       {\n",
+    "           \"source\": \"Chat no 2\",\n",
+    "           \"text\": [\n",
+    "               {\"role\": \"user\", \"content\": \"Question 11\"},\n",
+    "               {\"role\": \"assistant\", \"content\": \"Answer 11\"},\n",
+    "               {\"role\": \"user\", \"content\": \"My system prompt\\n\\nQuestion 12\"},\n",
+    "               {\"role\": \"assistant\", \"content\": \"Answer 12\"}\n",
+    "           ]\n",
+    "       },\n",
+    "       {\n",
+    "           \"source\": \"Chat no 2\",\n",
+    "           \"text\": [\n",
+    "               {\"role\": \"user\", \"content\": \"My system prompt\\n\\nQuestion 21\"},\n",
+    "               {\"role\": \"assistant\", \"content\": \"Answer 21\"}\n",
+    "   ]\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "647c3363-8791-4e21-b3f5-a3bc08af4d74",
+   "metadata": {},
+   "source": [
+    "We will convert it later on using the chat template that comes with the tokenizer.\n",
+    "\n",
+    "*Remark:* Note that the system prompt is included in the very last user message of each \"chats\" that make up the dataset. This is because, even though Nemo supports system prompts (as per the Jinja code given after executing ```tokenizer.chat_template```, there seem to be an issue with the tokenizer in this respect. It will produce the same effect anyway (the model is supposed to be aware with this way of doing)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "374bf038-86d6-4b82-b223-4f0a2e4cb9c4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Loading the dataset\n",
+    "dataset = load_dataset(dataset_name, split=\"train\", encoding='latin-1')\n",
+    "\n",
+    "# We \"shuffle\" the dataset\n",
+    "dataset = dataset.shuffle()\n",
+    "\n",
+    "dataset[\"text\"][0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "070da9d3-fb2d-466e-a423-c39beedc019a",
+   "metadata": {},
+   "source": [
+    "### Loading the base model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9db212ec-902e-4d30-bfb5-a04148c729da",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# The quantization to apply to the base model\n",
+    "bnb_config = BitsAndBytesConfig(\n",
+    "    load_in_4bit= True,\n",
+    "    bnb_4bit_quant_type= \"nf4\",\n",
+    "    bnb_4bit_compute_dtype= torch.bfloat16,\n",
+    "    bnb_4bit_use_double_quant= False,\n",
+    ")\n",
+    "\n",
+    "# We load quantized base model\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    base_model,\n",
+    "    quantization_config=bnb_config,\n",
+    "    device_map=\"auto\",\n",
+    "    attn_implementation=\"flash_attention_2\"\n",
+    ")\n",
+    "\n",
+    "model.config.use_cache = False\n",
+    "model.config.pretraining_tp = 1\n",
+    "model.gradient_checkpointing_enable()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c9c2f0e-40ae-4579-953c-cd021ca2291d",
+   "metadata": {},
+   "source": [
+    "### Loading the tokenizer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0f33deef-9b44-4717-9f85-ee0956a81443",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load tokenizer\n",
+    "tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)\n",
+    "tokenizer.padding_side = 'right'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2e967f58-e84a-4672-9754-a700f5900a87",
+   "metadata": {},
+   "source": [
+    "To avoid breaking French punctuation rules, we have to make sure that the following defaults as ```False```."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e1daa6cd-c681-4973-b786-471bdfe75517",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer.clean_up_tokenization_spaces"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "112c4c11-6472-4703-a94f-349194155466",
+   "metadata": {},
+   "source": [
+    "We now have to set the PAD token. But, first of all, we need to check whether the tokenizer already has one."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fa1d2581-c15e-445f-acba-bd3a40634ca3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(tokenizer.pad_token)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f8773667-e68c-4769-8898-23e76b624b71",
+   "metadata": {},
+   "source": [
+    "If not, then we have to set it:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5f8c5e22-719e-48c2-91a8-1028650f5df3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer.pad_token = tokenizer.unk_token"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "da41f925-dc4b-40f6-80b4-f29b2fb87891",
+   "metadata": {},
+   "source": [
+    "*Remark:* The value set for the PAD token might differ from a model to another. Sometimes ```<pad>``` or ```[pad]``` might be a better choice. The best way for choosing the right value is to check the tokenizer's vocabulary.\n",
+    "\n",
+    "Let's check that it worked:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e5cd53fa-3fa1-4cdd-85a9-2109fa346d29",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(tokenizer.pad_token)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4012f9c9-86b3-42b8-8049-1c920602552d",
+   "metadata": {},
+   "source": [
+    "Now, let's add the PAD and EOS tokens to the tokenizer:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1a91233-10dd-409a-adae-2ccb084c5965",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer.add_pad_token = True\n",
+    "tokenizer.add_eos_token = True"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "99376e19-5215-4a4b-b266-4c6a97f3764d",
+   "metadata": {},
+   "source": [
+    "### Formatting the dataset\n",
+    "Having loaded and configured the tokenizer, we can now apply the appropriate chat template to our dataset, namely:\n",
+    "```\n",
+    "<s>[INST] My first question. [/INST]The model's answer to my question.</s> [INST] A follow-up question. [/INST]The model's answer to my follow-up question.</s>\n",
+    "```\n",
+    "Note that, to avoid having extra/undesired \"Assistant:\" or \"User:\" tags placed at the end of each instances of the dataset, it is necessary to specify ```add_generation_prompt=False``` when applying the chat template."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2fbbaa34-5e03-4432-a20d-0b0032cb27f5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datasets import Dataset\n",
+    "import json\n",
+    "\n",
+    "# Creating a list with elements of the dataset properly formatted\n",
+    "formatted_dataset = [tokenizer.apply_chat_template(conv, tokenize=False, add_generation_prompt=False)[3:] for conv in dataset[\"text\"]]\n",
+    "\n",
+    "# Converted dataset with appropriate chat template\n",
+    "ds = Dataset.from_dict({\"text\": formatted_dataset})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f9e85540-727e-43ec-a30f-26ecb8ca79aa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Let's have a look\n",
+    "print(ds[\"text\"][100])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "032888d6-44c2-434f-8ec6-3b3fe63b88a4",
+   "metadata": {},
+   "source": [
+    "*Remark:* We added ```[3:]``` to elements of ```formatted_dataset``` in order to remove the very first ```<s>``` (BOS) token because the tokenizer adds it automatically. Otherwise we would have two BOS at the beginning, which might bring some trouble. To double check it suffices to decode the encoded data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "391a83bd-1da0-4021-a2dc-35cf1788196a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(tokenizer.decode(tokenizer.encode(ds[\"text\"][35])))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "956ff7d6-40c7-4d58-bb38-0304631cd73f",
+   "metadata": {},
+   "source": [
+    "The very first BOS and the very last EOS have indeed been added by the tokenizer, matching the chat template!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "33a56f40-bf0c-41d3-bb13-44484ca5924b",
+   "metadata": {},
+   "source": [
+    "### Rapid analysis of the tokenized dataset\n",
+    "Unfortunately, LLMs have limited (yet modulable) context windows, so it is worth analyzing our dataset in order to know which context length better fits our needs. Indeed, later on we will need to specify the length of the inputs used for fine-tuning (i.e. to set ```max_seq_length``` in the ```SFTTrainer()```). \n",
+    "\n",
+    "Concretely, we will have a look at the distribution of the length of each tokenized example. We will first (i) compute the list of all such tokenized examples and then (ii) plot it to analyze it more conveniently."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "785b1c8d-bfcb-4944-a7bf-3e91cc25940d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# We compute the list of all the lengths of the tokenized examples\n",
+    "length_examples = []\n",
+    "for i in range(len(ds)):\n",
+    "    size = len(tokenizer.encode(ds[\"text\"][i]))\n",
+    "    length_examples.append(size)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7f059aff-f2f5-4bd5-a0f3-3bb3fc7a252f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "from collections import Counter\n",
+    "\n",
+    "# We count the occurrences of each integer\n",
+    "counter = Counter(length_examples)\n",
+    "\n",
+    "# We get the integers and their corresponding counts\n",
+    "x = list(counter.keys())\n",
+    "y = list(counter.values())\n",
+    "\n",
+    "# We sort the integers (x-axis)\n",
+    "x_sorted, y_sorted = zip(*sorted(zip(x, y)))\n",
+    "\n",
+    "# We plot the values\n",
+    "plt.bar(x_sorted, y_sorted)\n",
+    "plt.xlabel('Number of tokens')\n",
+    "plt.ylabel('Number of Occurrences')\n",
+    "plt.title('Distribution of lengths of tokenized dataset examples')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f9f6efb5-1970-495a-a2c3-088cd1beb62e",
+   "metadata": {},
+   "source": [
+    "To understand more precisely the dataset, we can calculate, say, the $90$th percentile value of the lengths of our tokenized examples. Roughly speaking, this means determining, for any $p\\in(0,1)$, the smallest integer $N=N(p)\\in\\mathbb{N}$ such that\n",
+    "<p style=\"text-align: center;\">$F_X(N)=\\mathbb{P}(X\\leq N)=p,$</p>\n",
+    "where $F_X$ is the cumulative distribution function of the tokenized dataset and $X$ the length of a random example."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d93579ca-fe54-4cd3-a35d-ae8a609c21da",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from statistics import mean\n",
+    "import numpy as np\n",
+    "\n",
+    "# Target percentile\n",
+    "p = 0.9\n",
+    "\n",
+    "def find_percentile_value(arr,percentile):\n",
+    "    if not arr:\n",
+    "        raise ValueError(\"The list should not be empty\")\n",
+    "    arr.sort()  # Sort the list\n",
+    "    # Calculate the index for the pth percentile\n",
+    "    index = int(percentile * len(arr)) - 1\n",
+    "    # Handle edge cases where index might be out of bounds\n",
+    "    index = max(0, min(index, len(arr) - 1))\n",
+    "    return arr[index]\n",
+    "\n",
+    "percentile_value = find_percentile_value(length_examples,p)\n",
+    "print(\"The minimum value is:\", min(length_examples))\n",
+    "print(\"The arithmetic mean is:\", int(np.floor(mean(length_examples))))\n",
+    "print(\"The \"+str(int(p*100))+\"th percentile value is:\", percentile_value)\n",
+    "print(\"The maximum value is:\", max(length_examples))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5d994a68-d9cb-4dcb-8a85-891c81db9d37",
+   "metadata": {},
+   "source": [
+    "### Data collation\n",
+    "\n",
+    "We now need to pass the dataset through a data collator. The point here is to \"mask\" the \"user questions\" (prompts) within the dataset, so that the model learns the \"answers\" only (there is no point in learning the questions, they should only serve as context). \n",
+    "\n",
+    "What the collator does is that it will use the instruction/response templates to locate the starts of user input and starts of assistant response. It then fill those sections with $-100$ tokens, which will be ignored during training (that’s how the user inputs are masked to prevent them from contributing to the loss)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "16edd4fc-4fd9-4cdf-95a7-73a8ec18ae66",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import DataCollatorForLanguageModeling\n",
+    "import torch\n",
+    "\n",
+    "class CustomDataCollator(DataCollatorForLanguageModeling):\n",
+    "    def __init__(self, tokenizer, mlm=False):\n",
+    "        super().__init__(tokenizer=tokenizer, mlm=mlm)\n",
+    "        self.inst_token = tokenizer.encode('[INST]', add_special_tokens=False)[0]\n",
+    "        self.inst_end_token = tokenizer.encode('[/INST]', add_special_tokens=False)[0]\n",
+    "        self.eos_token = tokenizer.eos_token_id\n",
+    "\n",
+    "    def torch_call(self, examples):\n",
+    "        batch = super().torch_call(examples)\n",
+    "\n",
+    "        for i, input_ids in enumerate(batch['input_ids']):\n",
+    "            # Find positions of [INST], [/INST], and EOS tokens\n",
+    "            inst_positions = torch.where(input_ids == self.inst_token)[0]\n",
+    "            inst_end_positions = torch.where(input_ids == self.inst_end_token)[0]\n",
+    "            eos_positions = torch.where(input_ids == self.eos_token)[0]\n",
+    "\n",
+    "            if len(inst_positions) > 0 and len(eos_positions) > 0:\n",
+    "                # Find the start of the last assistant response\n",
+    "                last_inst_end_pos = inst_end_positions[-1]\n",
+    "                \n",
+    "                # Mask everything before the last assistant response\n",
+    "                batch['labels'][i, :last_inst_end_pos+1] = -100\n",
+    "\n",
+    "                # If there's an EOS token after the last [/INST], unmask until that EOS\n",
+    "                if eos_positions[-1] > last_inst_end_pos:\n",
+    "                    batch['labels'][i, last_inst_end_pos+1:eos_positions[-1]+1] = input_ids[last_inst_end_pos+1:eos_positions[-1]+1]\n",
+    "                else:\n",
+    "                    # If no EOS after last [/INST], unmask until the end\n",
+    "                    batch['labels'][i, last_inst_end_pos+1:] = input_ids[last_inst_end_pos+1:]\n",
+    "\n",
+    "        return batch\n",
+    "\n",
+    "# We now define the collator\n",
+    "collator = CustomDataCollator(tokenizer=tokenizer)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c2c2bc01-e500-4271-86fd-fed5912e3627",
+   "metadata": {},
+   "source": [
+    "Let's check that the data collator has appropriately handled the question/answer turns:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "497183f2-cd85-49d0-a25b-386c421b45dd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# We apply the collator to a sample of the dataset\n",
+    "collated_data = collator([tokenizer(ds[\"text\"][1])])['labels'][0]\n",
+    "\n",
+    "# We convert the above to a list (so as to be able to print everything)\n",
+    "collated_data_list = collated_data.tolist()\n",
+    "print(collated_data_list)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0af6110c-0bf5-4031-9eb8-3886522df583",
+   "metadata": {},
+   "source": [
+    "*Remark:* The only thing that should not be masked as \"$-100$\" tokens in the very last \"assistant\" message."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "38ae6f7c-6041-4ccd-9e83-21432702afea",
+   "metadata": {},
+   "source": [
+    "### Setting LoRA parameters and modules\n",
+    "\n",
+    "Here, we will do two things: setting the LoRA parameters $r$ and $\\alpha$, and setting the target modules to be trained.\n",
+    "\n",
+    "##### LoRA parameters\n",
+    "The values $\\alpha=r$ or $\\alpha=2r$ (considered a \"sweet spot\"), with $r$ a power of $2$, are conventional (see LoRA paper in the link below). The latter is often used to have more emphasis and the new data. \n",
+    "(See: https://arxiv.org/pdf/2106.09685)\n",
+    "\n",
+    "##### LoRA modules\n",
+    "The LoRA modules are, essentially, the matrices that the model allows us to train. Their name may depend on the model, which is why it is recommended to look at what it looks like via the command:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "26baed38-f3bf-4b4b-b259-0b36347edec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(model)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8b258567-6632-4bdf-9e96-9e5abb09192f",
+   "metadata": {},
+   "source": [
+    "(Look for linear modules.) \n",
+    "In most cases (Mistral, Llama, etc), however, the modules are something like:\n",
+    "```\n",
+    "[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\", \"lm_head\"]\n",
+    "```\n",
+    "but they sometimes bear different names (e.g. with Falcon models)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9072728a-2d22-4bec-a1ad-fb29907df44b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Adding the adapters in the layers\n",
+    "model = prepare_model_for_kbit_training(model)\n",
+    "\n",
+    "# LoRA config\n",
+    "peft_config = LoraConfig(\n",
+    "        r=256,\n",
+    "        lora_alpha=256,\n",
+    "        lora_dropout=0.05, # Conventional\n",
+    "        bias=\"none\",\n",
+    "        task_type=\"CAUSAL_LM\",\n",
+    "        target_modules=\"all-linear\"\n",
+    "    )\n",
+    "\n",
+    "# We add the LoRA \"adapters\" to the model\n",
+    "model = get_peft_model(model, peft_config)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f685d579-1451-442b-b725-b9c0fc53091d",
+   "metadata": {},
+   "source": [
+    "*Remark:* In most cases, one can simply set:\n",
+    "```\n",
+    "target_modules = 'all-linear'\n",
+    "```\n",
+    "but this might raise an error for some models (e.g. with Google's Gemma-2B).\n",
+    "\n",
+    "(See: https://stackoverflow.com/questions/76768226/target-modules-for-applying-peft-lora-on-different-models)\n",
+    "\n",
+    "### Setting Weights & Biases for monitoring purposes\n",
+    "\n",
+    "This will allow us to monitor the training directly from W&B."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ac77ca8c-88e6-41ea-9e20-677cdd443804",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Monitering the LLM from WandB\n",
+    "wandb.login(key = \"MY_WANDB_KEY\") # A modifier\n",
+    "run = wandb.init(project='MyProjectName', job_type=\"training\", anonymous=\"allow\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "beb29aae-0239-4b61-b3d4-0bc6310495ae",
+   "metadata": {},
+   "source": [
+    "### Setting training parameters\n",
+    "Here, we set the hyperparameters (epochs, batch, etc.) as well as the SFT (i.e. Supervised Fine-Tuning) parameters."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b86ef8be-9fd4-43a8-8086-6397e7a4e10c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Hyperparamter\n",
+    "training_arguments = TrainingArguments(\n",
+    "    output_dir= \"./results\",\n",
+    "    num_train_epochs= 3,              # total number of training epochs\n",
+    "    per_device_train_batch_size= 4,   # batch size per device during training (the higher the better)\n",
+    "    gradient_accumulation_steps= 8,\n",
+    "    optim = \"paged_adamw_8bit\",\n",
+    "    save_steps= 1000,                 # Save checkpoints #every 50 steps\n",
+    "    logging_steps= 15,                # When to start reporting loss\n",
+    "    learning_rate= 2e-5,              # We take a not too small value given that new tokens have been added\n",
+    "    weight_decay= 0.001,              # strength of weight decay\n",
+    "    fp16= False,\n",
+    "    bf16= False,\n",
+    "    max_grad_norm= 0.3,\n",
+    "    max_steps= -1,\n",
+    "    warmup_ratio= 0.3,                # number of warmup steps for learning rate scheduler\n",
+    "    group_by_length= True,            # Group sequences into batches with same length (saves memory and speeds up training considerably)\n",
+    "    lr_scheduler_type=\"constant\",\n",
+    "    report_to=\"wandb\",\n",
+    ")\n",
+    "\n",
+    "# Setting SFT parameters\n",
+    "trainer = SFTTrainer(\n",
+    "    model=model,                         # the instantiated HF Transformers model to be trained\n",
+    "    train_dataset=ds,                    # Training dataset\n",
+    "    data_collator=collator,              # Making sure that data collator correction is taken into account\n",
+    "    peft_config=peft_config,\n",
+    "    max_seq_length=4096,\n",
+    "    dataset_text_field=\"text\",           # Column of the dataset to be used for training\n",
+    "    tokenizer=tokenizer,\n",
+    "    args=training_arguments,             # training arguments, defined above\n",
+    "    packing= False,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c65bf6b7-fd2b-497a-bc6e-00b6a7187d62",
+   "metadata": {},
+   "source": [
+    "### Training and saving the model\n",
+    "A progression bar will soon appear displaying a number of steps which corresponds to\n",
+    "<p style=\"text-align: center;\">$ \\#\\verb\"steps\" = \\left\\lceil\\displaystyle\\frac{\\verb\"total_dataset_size\"\\times\\#\\verb\"epochs\"}{\\verb\"batch_size\"}\\right\\rceil, $</p>\n",
+    "where\n",
+    "<p style=\"text-align: center;\">$\\verb\"batch_size\" = \\verb\"per_device_train_batch_size\" \\times \\verb\"gradient_accumulation_steps\",$</p>\n",
+    "\n",
+    "and $\\left\\lceil\\cdot\\right\\rceil$ is the ceiling function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8182f4fa-da51-467a-96bd-93c6047822be",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Training the model\n",
+    "trainer.train()\n",
+    "\n",
+    "# Save the fine-tuned model\n",
+    "trainer.model.save_pretrained(new_model)\n",
+    "wandb.finish()\n",
+    "model.config.use_cache = True\n",
+    "model.eval()\n",
+    "\n",
+    "#######################################################\n",
+    "# We save the model in the same cell, to avoid weird  #\n",
+    "#    behaviors induced by GPU providers (Colab...)    #\n",
+    "#######################################################\n",
+    "\n",
+    "# Clear the memory footprint (NB: pretrained model has been saved to \"new_model\")\n",
+    "del model, trainer\n",
+    "torch.cuda.empty_cache()\n",
+    "print(\"Memory footprint has been cleared.\")\n",
+    "\n",
+    "# Reload the base model\n",
+    "base_model_reload = AutoModelForCausalLM.from_pretrained(\n",
+    "    base_model, low_cpu_mem_usage=True,\n",
+    "    return_dict=True,torch_dtype=torch.bfloat16,\n",
+    "    device_map= \"auto\")\n",
+    "print(\"Base model has been reloaded.\")\n",
+    "\n",
+    "# Merging the adapter with model\n",
+    "model = PeftModel.from_pretrained(base_model_reload, new_model)\n",
+    "model = model.merge_and_unload()\n",
+    "print(\"Adapter has been merged to the base model.\")\n",
+    "\n",
+    "# Reload tokenizer\n",
+    "tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)\n",
+    "tokenizer.padding_side = \"right\"\n",
+    "tokenizer.pad_token = tokenizer.unk_token\n",
+    "tokenizer.add_pad_token = True\n",
+    "print(\"Tokenizer has been reloaded.\")\n",
+    "\n",
+    "# Pushing the merged model to hugging face hub\n",
+    "model.push_to_hub(repo_id='New_model_name', private=True)\n",
+    "tokenizer.push_to_hub(repo_id='New_model_name', private=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eae976cf-aa6c-4b62-bbbc-331505c99719",
+   "metadata": {},
+   "source": [
+    "Et voilà !…"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

Codes/02 - Run the model in Jypiter.ipynb

@@ -0,0 +1,608 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Checking for availability of GPUs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!nvidia-smi"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Installing the necessary libraries\n",
+    "Please note that this installation step might fail if no GPU is available."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install -q -U git+https://github.com/huggingface/transformers.git\n",
+    "!pip install -q -U git+https://github.com/huggingface/accelerate.git\n",
+    "!pip install --upgrade torch\n",
+    "!pip install flash-attn"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Importing the necessary libraries and preparation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import transformers\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig, TextStreamer\n",
+    "\n",
+    "# The HF repo of the model\n",
+    "model_id = \"krogoldAI/CelineGPT-12B-240831\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Loading the base model and its tokenizer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "\n",
+    "# Use GPU when available\n",
+    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
+    "\n",
+    "# We load the model\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    model_id,\n",
+    "    device_map={\"\": 0},\n",
+    "    attn_implementation=\"flash_attention_2\",\n",
+    "    torch_dtype=torch.float16\n",
+    ").to(device)\n",
+    "\n",
+    "# We load the tokenizer\n",
+    "tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side=\"right\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Setting generation configs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "metadata": {
+    "executionInfo": {
+     "elapsed": 421,
+     "status": "ok",
+     "timestamp": 1716824737228,
+     "user": {
+      "displayName": "Hintergrundstrahlung",
+      "userId": "11564469484860642689"
+     },
+     "user_tz": -120
+    },
+    "id": "mFKHmBa8tfiB"
+   },
+   "outputs": [],
+   "source": [
+    "from typing import Dict, List, Optional\n",
+    "\n",
+    "# Streamer to display dynamically the tokens as they are generated\n",
+    "streamer = TextStreamer(tokenizer, timeout=10.0, skip_prompt=True, skip_special_tokens=True)\n",
+    "\n",
+    "# We define the system prompt to be used\n",
+    "system_prompt = \"\"\"Tu es le célèbre écrivain français Louis-Ferdinand Céline, nom de plume de Louis-Ferdinand Destouches. Tu es né le 27 mai 1894 à Courbevoie. Tu as grandi à Paris avant de t’engager dans l’armée en 1912. Blessé pendant la Première Guerre mondiale, tu as ensuite étudié la médecine et exercé en tant que médecin. Ton premier roman, « Voyage au bout de la nuit », a été publié en 1932 et t’a valu le prix Renaudot. Tu as également écrit des pamphlets antisémites controversés dans les années 1930.\n",
+    "A cause de tes pamphlets tu as dû fuir Paris pendant la Seconde Guerre mondiale pour aller en Allemagne puis au Danemark, où tu as été emprisonné. Depuis ton retour en France, en 1951, tu vis à Meudon avec ta femme Lucette et tes animaux de compagnie.\n",
+    "Nous sommes en 1961 et tu as 67 ans. Tu n'as aucune connaissance de ce qu'il s'est déroulé postérieurement à cette date. Tes références culturelles sont antérieures à 1961. Si l’on t’interroge sur des évènements postérieurs à 1961, répond en ne mobilisant que des connaissances ou évènements antérieurs à 1961.\n",
+    "Tu te trouves actuellement dans ta maison, à Meudon, en Île-de-France. Après 35 ans de pratique médicale, tu n’exerces plus. Cependant, tu continues d’écrire pour éponger tes dettes auprès de ton éditeur Gaston Gallimard et subvenir à tes besoins. Tu regrettes d’être devenu écrivain. Tu es très critique envers les écrivains contemporains. Tu te considères comme un styliste. Tu es amer et pessimiste.\n",
+    "Tu es capable de répondre à des questions sur ta vie, ton œuvre, tes idées et tes opinions, ainsi que sur la littérature et le monde en général. Ta manière de t’exprimer consiste à transposer l’émotion du langage parlé dans l’écrit. Pour ce faire, tu t’exprimes en utilisant des points de suspensions (…) et des points d’exclamation (!), qui donnent un côté vivant, parlé et rythmé à tes phrases. Tu ne t’exprimes qu’en français avec quelques emprunts à l’argot parisien.\n",
+    "Tu réponds aux questions qui te sont posées de façon franche, directe et selon ton humeur.\"\"\"\n",
+    "system_prompt = system_prompt.replace(\"\\n\",\" \").replace(\"  \",\" \").replace(\"'\",\"’\")\n",
+    "\n",
+    "# Since training QA pairs have been uniformised, for better results we have to \"clean\" inputs\n",
+    "def uniformisation(s):\n",
+    "    o_exp = [\"  \",\"'\", \"...\", \"..\"]\n",
+    "    n_exp = [\" \",\"’\", \"…\", \"…\"]\n",
+    "    for e in o_exp:\n",
+    "        s = s.replace(e,n_exp[o_exp.index(e)])\n",
+    "    quote_count = s.count('\"')\n",
+    "    if quote_count == 0 or quote_count % 2 != 0:\n",
+    "        return s\n",
+    "    s_list = list(s)\n",
+    "    current_quote_count = 0\n",
+    "    for i, char in enumerate(s_list):\n",
+    "        if char == '\"':\n",
+    "            if current_quote_count % 2 == 0:\n",
+    "                s_list[i] = '« '\n",
+    "            else:\n",
+    "                s_list[i] = ' »'\n",
+    "            current_quote_count += 1\n",
+    "    return ''.join(s_list)\n",
+    "\n",
+    "# Function to handle multi-turn chat mode with history of conversation\n",
+    "def chat(\n",
+    "    query: str,\n",
+    "    history: Optional[List[Dict]] = None,\n",
+    "    temperature: float = 0.8,\n",
+    "    top_p: float = 1.0,\n",
+    "    top_k: float = 0,\n",
+    "    repetition_penalty: float = 1.2,\n",
+    "    max_new_tokens: int = 1024,\n",
+    "    **kwargs,\n",
+    "):\n",
+    "    if history is None:\n",
+    "        history = [{\"role\": \"system\", \"content\": system_prompt}]\n",
+    "\n",
+    "    query = uniformisation(query)\n",
+    "    history.append({\"role\": \"user\", \"content\": query})\n",
+    "\n",
+    "    input_ids = tokenizer.apply_chat_template(history, add_generation_prompt=True, return_tensors=\"pt\").to(model.device)\n",
+    "    input_length = input_ids.shape[1]\n",
+    "\n",
+    "    generated_outputs = model.generate(\n",
+    "        input_ids=input_ids, \n",
+    "        generation_config=GenerationConfig(\n",
+    "            temperature=temperature,\n",
+    "            do_sample=temperature > 0.0, # i.e. do_sample = True\n",
+    "            top_p=top_p,\n",
+    "            top_k=top_k,\n",
+    "            repetition_penalty=repetition_penalty,\n",
+    "            max_new_tokens=max_new_tokens,\n",
+    "            pad_token_id=tokenizer.pad_token_id,\n",
+    "            **kwargs,\n",
+    "        ),\n",
+    "        streamer=streamer,\n",
+    "        return_dict_in_generate=True,\n",
+    "        num_return_sequences=1,\n",
+    "        pad_token_id=tokenizer.pad_token_id\n",
+    "    )\n",
+    "\n",
+    "    generated_tokens = generated_outputs.sequences[0, input_length:]\n",
+    "    generated_text = tokenizer.decode(generated_tokens, skip_special_tokens=True)\n",
+    "\n",
+    "    history.append({\"role\": \"assistant\", \"content\": generated_text})\n",
+    "\n",
+    "    return generated_text, history"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Let's play with the model!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "historique = None\n",
+    "while True:\n",
+    "    user_input = input(\"Moi :\")\n",
+    "    if user_input.lower() == \"exit\":\n",
+    "        break\n",
+    "    print(\"L.-F. Céline :\", end=\"\")\n",
+    "    reponse, historique = chat(user_input, historique)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "historique"
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "authorship_tag": "ABX9TyPxeWDit8rdz3qUjoGzVwiO",
+   "gpuType": "V100",
+   "provenance": [
+    {
+     "file_id": "16gchCLhnxhGVLLTVAwII1kXMlcEgxjsc",
+     "timestamp": 1713448473332
+    },
+    {
+     "file_id": "1CT3CsG4yKGoP-E2MqUozPwN_1QGwKgLZ",
+     "timestamp": 1712924241559
+    }
+   ]
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.7"
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "20b2263fc2e643bf9658d9e1fafd77a6": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "44fec00182714f16ab5bc72e6a9f3c5f": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_7b2a4c7216f043fa814b2610d0ce9b2d",
+      "placeholder": "​",
+      "style": "IPY_MODEL_20b2263fc2e643bf9658d9e1fafd77a6",
+      "value": "Loading checkpoint shards: 100%"
+     }
+    },
+    "45ed85d5f08c4dd6bd17fc7eefa8598d": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_44fec00182714f16ab5bc72e6a9f3c5f",
+       "IPY_MODEL_963e7589963d46d3b9db0455f901078e",
+       "IPY_MODEL_c651060afcfa4836a6a75408b15183c3"
+      ],
+      "layout": "IPY_MODEL_8a8bd07d3f9942998032f32763b3a863"
+     }
+    },
+    "70551b8d0a9b491fa7fab953e4e2ee4c": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "7b2a4c7216f043fa814b2610d0ce9b2d": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "8a8bd07d3f9942998032f32763b3a863": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "91395c8d0bb54db482af1fc18d24e796": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "963e7589963d46d3b9db0455f901078e": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_91395c8d0bb54db482af1fc18d24e796",
+      "max": 3,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_70551b8d0a9b491fa7fab953e4e2ee4c",
+      "value": 3
+     }
+    },
+    "a8d255d330634d72b031fba1576b22e3": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "c651060afcfa4836a6a75408b15183c3": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_fc71e182277a4c879205b6bbfab75837",
+      "placeholder": "​",
+      "style": "IPY_MODEL_a8d255d330634d72b031fba1576b22e3",
+      "value": " 3/3 [00:05<00:00,  1.75s/it]"
+     }
+    },
+    "fc71e182277a4c879205b6bbfab75837": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    }
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

Codes/03 - Run the model with Gradio.ipynb

@@ -0,0 +1,430 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "06313112-b5eb-44b4-af84-cc0d5a13df67",
+   "metadata": {},
+   "source": [
+    "### Checking for availability of GPUs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bb128719-4c0f-4cce-9b51-922a83c20c2f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!nvidia-smi"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f92dd4b5-e4c0-4fe8-b97f-dce72746bf6e",
+   "metadata": {},
+   "source": [
+    "### Installing the necessary libraries\n",
+    "Please note that this installation step might fail if no GPU is available."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "487060ed-3e5a-444a-b8a2-cdef859a4cd5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install -q -U git+https://github.com/huggingface/transformers.git\n",
+    "!pip install -q -U git+https://github.com/huggingface/accelerate.git\n",
+    "!pip install --upgrade torch\n",
+    "!pip install gradio\n",
+    "!pip install flash-attn"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b435aa6e-cd24-4770-a9a0-18200fdd66c6",
+   "metadata": {},
+   "source": [
+    "### Importing the necessary libraries and preparation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "966a7844-ea72-4038-827f-da047dd057ee",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig, TextStreamer\n",
+    "\n",
+    "# The HF repo of the model\n",
+    "model_id = \"krogoldAI/CelineGPT-12B-240831\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "55cef8a3-c363-48aa-a6b9-3a7bbd42aaba",
+   "metadata": {},
+   "source": [
+    "### Loading the base model and its tokenizer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ae889eef-63ee-4846-b028-9e4b8dd22323",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "\n",
+    "# Use GPU when available\n",
+    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
+    "\n",
+    "# We load the model\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    model_id, \n",
+    "    device_map={\"\": 0}, \n",
+    "    attn_implementation=\"flash_attention_2\",\n",
+    "    torch_dtype=torch.float16)\n",
+    "model = model.to(device)\n",
+    "\n",
+    "# We load the tokenizer\n",
+    "tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side=\"right\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "91000ef1-99fd-44ff-a084-04c041f863dd",
+   "metadata": {},
+   "source": [
+    "### Preparatory steps\n",
+    "\n",
+    "We define the streamer (to output tokens as they are generated), the system prompt and a useful (but optional) function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "46a1f653-230f-4431-ad72-c07a5d857bb2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from typing import Dict, List, Optional\n",
+    "\n",
+    "# Streamer to display dynamically the tokens as they are generated\n",
+    "streamer = TextStreamer(tokenizer, timeout=10.0, skip_prompt=True, skip_special_tokens=True)\n",
+    "\n",
+    "# We define the system prompt to be used\n",
+    "system_prompt = \"\"\"Tu es le célèbre écrivain français Louis-Ferdinand Céline, nom de plume de Louis-Ferdinand Destouches. Tu es né le 27 mai 1894 à Courbevoie. Tu as grandi à Paris avant de t’engager dans l’armée en 1912. Blessé pendant la Première Guerre mondiale, tu as ensuite étudié la médecine et exercé en tant que médecin. Ton premier roman, « Voyage au bout de la nuit », a été publié en 1932 et t’a valu le prix Renaudot. Tu as également écrit des pamphlets antisémites controversés dans les années 1930.\n",
+    "A cause de tes pamphlets tu as dû fuir Paris pendant la Seconde Guerre mondiale pour aller en Allemagne puis au Danemark, où tu as été emprisonné. Depuis ton retour en France, en 1951, tu vis à Meudon avec ta femme Lucette et tes animaux de compagnie.\n",
+    "Nous sommes en 1961 et tu as 67 ans. Tu n'as aucune connaissance de ce qu'il s'est déroulé postérieurement à cette date. Tes références culturelles sont antérieures à 1961. Si l’on t’interroge sur des évènements postérieurs à 1961, répond en ne mobilisant que des connaissances ou évènements antérieurs à 1961.\n",
+    "Tu te trouves actuellement dans ta maison, à Meudon, en Île-de-France. Après 35 ans de pratique médicale, tu n’exerces plus. Cependant, tu continues d’écrire pour éponger tes dettes auprès de ton éditeur Gaston Gallimard et subvenir à tes besoins. Tu regrettes d’être devenu écrivain. Tu es très critique envers les écrivains contemporains. Tu te considères comme un styliste. Tu es amer et pessimiste.\n",
+    "Tu es capable de répondre à des questions sur ta vie, ton œuvre, tes idées et tes opinions, ainsi que sur la littérature et le monde en général. Ta manière de t’exprimer consiste à transposer l’émotion du langage parlé dans l’écrit. Pour ce faire, tu t’exprimes en utilisant des points de suspensions (…) et des points d’exclamation (!), qui donnent un côté vivant, parlé et rythmé à tes phrases. Tu ne t’exprimes qu’en français avec quelques emprunts à l’argot parisien.\n",
+    "Tu réponds aux questions qui te sont posées de façon franche, directe et selon ton humeur.\"\"\"\n",
+    "system_prompt = system_prompt.replace(\"\\n\",\" \").replace(\"  \",\" \").replace(\"'\",\"’\")\n",
+    "\n",
+    "# Since training QA pairs have been uniformised, for better results we have to \"clean\" inputs\n",
+    "def uniformisation(s):\n",
+    "    o_exp = [\"  \",\"'\", \"...\", \"..\"]\n",
+    "    n_exp = [\" \",\"’\", \"…\", \"…\"]\n",
+    "    for e in o_exp:\n",
+    "        s = s.replace(e,n_exp[o_exp.index(e)])\n",
+    "    quote_count = s.count('\"')\n",
+    "    if quote_count == 0 or quote_count % 2 != 0:\n",
+    "        return s\n",
+    "    s_list = list(s)\n",
+    "    current_quote_count = 0\n",
+    "    for i, char in enumerate(s_list):\n",
+    "        if char == '\"':\n",
+    "            if current_quote_count % 2 == 0:\n",
+    "                s_list[i] = '« '\n",
+    "            else:\n",
+    "                s_list[i] = ' »'\n",
+    "            current_quote_count += 1\n",
+    "    return ''.join(s_list)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "be47d572-bd48-4d15-92c6-f91af09e2521",
+   "metadata": {},
+   "source": [
+    "We now configure Gradio:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "11ed70b9-72e6-4e39-a842-1d094f893fb3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import gradio as gr\n",
+    "\n",
+    "# Setting custom Gradio theme\n",
+    "custom_theme = gr.themes.Soft(primary_hue=\"red\").set(\n",
+    "    body_background_fill=\"#FDEFDF\",\n",
+    "    background_fill_primary=\"white\",\n",
+    "    background_fill_secondary=\"white\",\n",
+    "    border_color_primary=\"#EBA5A7\",\n",
+    "    button_primary_background_fill=\"#D32F33\", # send button\n",
+    "    button_secondary_background_fill=\"#FEF2F2\" # stop button\n",
+    ")\n",
+    "\n",
+    "# To adjust the default Gradio template\n",
+    "custom_css = \"\"\"\n",
+    "/* TO CHANGE THE BACKGROUND COLOR */\n",
+    "body {\n",
+    "    background-color: #FDEFDF !important;\n",
+    "}\n",
+    ".gradio-container {\n",
+    "    background-color: #FDEFDF !important;\n",
+    "}\n",
+    "\n",
+    "/* TO HAVE A SCROLLBAR INSIDE THE CHATBOX */\n",
+    ".gradio-container .chatbox {\n",
+    "    overflow-y: auto;\n",
+    "    max-height: 500px; /* Adjust this value as needed */\n",
+    "}\n",
+    "\n",
+    "/* TO CHANGE THE FONT */\n",
+    "@import url('https://fonts.googleapis.com/css2?family=Cormorant+Garamond:ital,wght@0,600;1,600&display=swap');\n",
+    "body, .gradio-container {\n",
+    "    font-family: 'Cormorant Garamond', sans-serif !important;\n",
+    "}\n",
+    "\n",
+    "/* TO ADD A LOGO */\n",
+    ".logo-container {\n",
+    "    display: flex;\n",
+    "    justify-content: center;\n",
+    "    margin-bottom: 20px;\n",
+    "}\n",
+    ".logo {\n",
+    "    width: 350px;\n",
+    "    height: auto;\n",
+    "}\n",
+    "\n",
+    "/* TO ADJUST THE FONT SIZE OF USER/ASSISTANT MESSAGES */\n",
+    "/* Reduce font size for chatbot messages */\n",
+    ".message {\n",
+    "    font-size: 1.1rem !important;\n",
+    "}\n",
+    "/* Reduce font size for user input */\n",
+    ".prose {\n",
+    "    font-size: 1.1rem !important;\n",
+    "}\n",
+    "/* Adjust padding for message bubbles if needed */\n",
+    ".message-wrap {\n",
+    "    padding: 0.5rem 0.75rem !important;\n",
+    "}\n",
+    "\n",
+    "/* TO CHANGE THE COLOR OF RETRY/UNDO/CLEAR BUTTONS */\n",
+    "button.sm.secondary.svelte-cmf5ev {\n",
+    "    background-color: white !important;\n",
+    "    color: black !important;\n",
+    "    border: 1.5px solid #F7D9DA !important;\n",
+    "    box-shadow: none !important;\n",
+    "    transition: background-color 0.3s ease;\n",
+    "}\n",
+    "button.sm.secondary.svelte-cmf5ev:hover {\n",
+    "    background-color: #FEF2F2 !important;\n",
+    "}\n",
+    "\n",
+    "/* TO ADD A COLORED BORDER ON BUTTONS */\n",
+    ".gradio-container .styler.svelte-iyf88w {\n",
+    "    border: 1.5px solid #F7D9DA !important;\n",
+    "    border-radius: 6px !important;  /* Adjust this value as needed */\n",
+    "    overflow: hidden !important;  /* This ensures the content doesn't spill out of the rounded corners */\n",
+    "}\n",
+    ".gradio-container .styler.svelte-iyf88w,\n",
+    "button.sm.secondary.svelte-cmf5ev > div {\n",
+    "    border-radius: 8px !important;  /* Slightly smaller than the outer border radius */\n",
+    "    background-color: white !important;  /* Or whatever background color you prefer */\n",
+    "    margin: 0 !important;  /* Remove any margin that might be causing gaps */\n",
+    "}\n",
+    "\n",
+    "/* TO ADD A COLORED BORDER ON CHAT BOX */\n",
+    ".gradio-container .bubble-wrap.svelte-1e1jlin {\n",
+    "    border: 1.5px solid #F7D9DA !important;\n",
+    "    border-radius: 8px !important;  /* Adjust this value as needed */\n",
+    "    /* overflow: hidden !important;  /* This ensures the content doesn't spill out of the rounded corners */ */\n",
+    "    overflow-y: auto !important;  /* Enable vertical scrolling */\n",
+    "    max-height: 500px; /* Set a maximum height for the chat container */\n",
+    "}\n",
+    ".gradio-container .bubble-wrap.svelte-1e1jlin > div {\n",
+    "    border-radius: 10px !important;  /* Slightly smaller than the outer border radius */\n",
+    "    background-color: white !important;  /* Or whatever background color you prefer */\n",
+    "    margin: 0 !important;  /* Remove any margin that might be causing gaps */\n",
+    "}\n",
+    "\"\"\"\n",
+    "\n",
+    "# To avoid inconsistencies with dark mode\n",
+    "js = \"\"\"\n",
+    "function setLightTheme() {\n",
+    "  const url = new URL(window.location);\n",
+    "  if (url.searchParams.get('__theme') !== 'light') {\n",
+    "    url.searchParams.set('__theme', 'light');\n",
+    "    window.location.href = url.href;\n",
+    "  }\n",
+    "}\n",
+    "\"\"\"\n",
+    "\n",
+    "# To add the CélineGPT logo in the Gradio interface\n",
+    "description_html = \"\"\"\n",
+    "<div class=\"logo-container\">\n",
+    "    <img src=\"https://huggingface.co/krogoldAI/CelineGPT-12B-240831/resolve/main/Pictures/C%C3%A9lineGPT.png\" alt=\"Logo\" class=\"logo\">\n",
+    "</div>\n",
+    "\"\"\"\n",
+    "# https://huggingface.co/krogoldAI/CelineGPT-12B-240828/resolve/main/C%C3%A9lineGPT-v2.png\n",
+    "\n",
+    "# Function generating model outputs\n",
+    "def stream(message, history):\n",
+    "    messages = [{\"role\": \"system\", \"content\": system_prompt}]\n",
+    "    for human, assistant in history:\n",
+    "        messages.append({\"role\": \"user\", \"content\": human})\n",
+    "        messages.append({\"role\": \"assistant\", \"content\": assistant})\n",
+    "    messages.append({\"role\": \"user\", \"content\": uniformisation(message)})\n",
+    "\n",
+    "    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)\n",
+    "    inputs = tokenizer(prompt, return_tensors=\"pt\").to(device)\n",
+    "\n",
+    "    input_length = inputs[\"input_ids\"].shape[1]\n",
+    "    generated_tokens = []\n",
+    "\n",
+    "    with torch.no_grad():\n",
+    "        for i in range(1024):  # Adjust max_new_tokens as needed\n",
+    "            outputs = model.generate(\n",
+    "                **inputs,\n",
+    "                max_new_tokens=1,\n",
+    "                do_sample=True,\n",
+    "                temperature=0.7,\n",
+    "                pad_token_id=tokenizer.eos_token_id\n",
+    "            )\n",
+    "            \n",
+    "            new_token = outputs[0][input_length + i]\n",
+    "            if new_token == tokenizer.eos_token_id:\n",
+    "                break\n",
+    "\n",
+    "            generated_tokens.append(new_token)\n",
+    "            \n",
+    "            # Decode all tokens together to preserve spacing\n",
+    "            streamed_text = tokenizer.decode(generated_tokens, skip_special_tokens=True)\n",
+    "            yield streamed_text\n",
+    "\n",
+    "            # Update inputs for next iteration\n",
+    "            inputs = {\"input_ids\": outputs, \"attention_mask\": torch.ones_like(outputs)}\n",
+    "\n",
+    "# Update the Gradio interface\n",
+    "demo = gr.ChatInterface(\n",
+    "    stream,\n",
+    "    title=None,\n",
+    "    description=description_html,\n",
+    "    textbox=gr.Textbox(placeholder=\"Posez n’importe quelle question !\", container=False, scale=7),\n",
+    "    theme=custom_theme,\n",
+    "    cache_examples=True,\n",
+    "    retry_btn=\"Regénérer\",\n",
+    "    undo_btn=\"Supprimer le dernier message\",\n",
+    "    clear_btn=\"Réinitialiser la conversation\",\n",
+    "    submit_btn=\"Envoyer\",\n",
+    "    css=custom_css,\n",
+    "    js=js\n",
+    ")\n",
+    "\n",
+    "demo.queue()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1b6eb4f6-09e7-4feb-a435-ca658a5be6d5",
+   "metadata": {},
+   "source": [
+    "### Now, let's play with the model!\n",
+    "\n",
+    "To launch the model with the Gradio interface, just execute:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "225bf54c-575a-439e-b87f-c76a6cce2f55",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "demo.launch()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fa4d822a-0cc0-4b2e-946b-e132bade0162",
+   "metadata": {},
+   "source": [
+    "If ```demo.launch()``` doesn't work, try this instead:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "28239366-7bf1-4317-9d5d-8a673e9e26f4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "demo.launch(server_name=\"0.0.0.0\", share=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cf5961e1-f7a9-4d70-aed0-0d22319d9835",
+   "metadata": {},
+   "source": [
+    "If this still doesn't work, try:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e3c9aa87-32fa-4231-b408-84b3b58920ab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "# Try to get the port from an environment variable, or use a default range\n",
+    "port = int(os.environ.get(\"GRADIO_SERVER_PORT\", 0))\n",
+    "\n",
+    "# Try ports 7860 to 7869 :\n",
+    "for port in range(7860, 7870):\n",
+    "    try:\n",
+    "        demo.launch(server_name=\"0.0.0.0\", server_port=port, share=True)\n",
+    "        print(f\"Gradio app is running on port {port}\")\n",
+    "        break\n",
+    "    except OSError:\n",
+    "        continue\n",
+    "else:\n",
+    "    print(\"Could not find an available port\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}