<kaggle_start><data_title>PlantVillage Dataset<data_description>Human society needs to increase food production by an estimated 70% by 2050 to feed an expected population size that is predicted to be over 9 billion people. Currently, infectious diseases reduce the potential yield by an average of 40% with many farmers in the developing world experiencing yield losses as high as 100%. The widespread distribution of smartphones among crop growers around the world with an expected 5 billion smartphones by 2020 offers the potential of turning the smartphone into a valuable tool for diverse communities growing food. One potential application is the development of mobile disease diagnostics through machine learning and crowdsourcing. Here we announce the release of over 50,000 expertly curated images on healthy and infected leaves of crops plants through the existing online platform PlantVillage. We describe both the data and the platform. These data are the beginning of an on-going, crowdsourcing effort to enable computer vision approaches to help solve the problem of yield losses in crop plants due to infectious diseases.<data_name>plantvillage-dataset
<code># # Knowledge Distillation
# **Author:** [Aminu Musa]
# **Date created:** 2022/09/01
# **Last modified:** 2020/10/17
# **Description:** Lighweight Plant Disease Detection Model for Embedded Device Using Knowledge Distillation Technique.
# This work is from a paper titled "Low-Power Deep Learning Model for Plant Disease Detection for Smart-Hydroponics Using Knowledge Distillation Techniques" link to the paper: https://www.mdpi.com/1605040
# the Techniques of Knowledge Distillation was a from a paper by Geofrey Hinton.https://arxiv.org/abs/1503.02531
# #   **INTRODUCTION**
# In this work We tried to implement Knowledge Distillation(KD) technique as proposed by Geofrey Hinton in thier paper titled  "Distilling the Knowledge in a Neural Network" the technique was adopted in our paper called Lighweight Plant Disease Detection Model for Embedded Device with the aim of achieving a lighter CNN model for plant disease detection.
# * **Goal**:
# Our aim is to developed a deep learning model that will aid African farmers in detecting diseases that are affecting thier crops using thier smartphones.the start of the art CNN models are not suitable for deployment on smartphones due to thier sizes and huge number of parametres. We therefore, compressed the model using KD to reduce model parameters while keeping the performance. we use open source plant disease detection dataset organized by PlantVIllage to train and validate our model.
# * **Contribution:**
# To obtain a lighweight model that will be suitable for deployment on embedded devices such as smartphone
# # 1- Uplaoding necessary libraries
import os
import pickle
import zipfile
import random
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import RMSprop
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from shutil import copyfile
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (
    BatchNormalization,
    Conv2D,
    MaxPooling2D,
    Activation,
    Flatten,
    Dropout,
    Dense,
)
from tensorflow.keras import backend as K
from sklearn.preprocessing import LabelBinarizer
from keras.preprocessing import image
import matplotlib.pyplot as plt


# #  2- How Does Knowledge Distillation Works?
# Knowledge distillation is a technique of model compression that employs teacher-student learning. In this technique, a large, cumbersome pre-trained model called the teacher model is trained first on a large dataset. The knowledge learned by the teacher model is transferred (distilled) to a shallow lightweight model called the student model, whereby the student mimics the performance of the teacher model despite its shallow nature. The student model size is relatively reduced without sacrificing accuracy. Figure.1 presents the architecture of the knowledge distillation technique.
# ![image.png](attachment:b9b79385-2927-499c-b3fd-fb74d22d7d33.png)
# # 3- Creating Distiller Class
# ## Construct `Distiller()` class
# This custom class`Distiller()` was used to overrides the  original Model methods namely:`train_step`, `test_step`,
# and `compile()`.
# * **Train_step:** A training step is one gradient update. In one step batch_size many examples are processed. An epoch consists of one full cycle through the training data. This is usually many steps. As an example, if you have 2,000 images and use a batch size of 10 an epoch consists of 2,000 images / (10 images / step) = 200 steps
# * **Compile method:** It specifies a loss, metrics, and an optimizer. To train a model with fit(), you need to specify a loss function, an optimizer, and optionally, some metrics to monitor. We can pass these variables to the model as arguments to the compile() method.
# In order to use the distiller, we need:
# - A trained teacher model
# - A student model to train
# - A student loss function on the difference between student predictions and ground-truth
# - A distillation loss function, along with a `temperature`, on the difference between the
# soft student predictions and the soft teacher labels
# - An `alpha` factor to weight the student and distillation loss
# - An optimizer for the student and (optional) metrics to evaluate performance
# In the `train_step` method, we perform a forward pass of both the teacher and student,
# calculate the loss with weighting of the `student_loss` and `distillation_loss` by `alpha` and
# `1 - alpha`, respectively, and perform the backward pass. Note: only the student weights are updated,
# and therefore we only calculate the gradients for the student weights.
# In the `test_step` method, we evaluate the student model on the provided dataset.
class Distiller(keras.Model):
    def __init__(self, student, teacher):
        super(Distiller, self).__init__()
        self.teacher = teacher
        self.student = student

    def compile(
        self,
        optimizer,
        metrics,
        student_loss_fn,
        distillation_loss_fn,
        alpha=0.1,
        temperature=3,
    ):
        """Configure the distiller.
        Args:
            optimizer: Keras optimizer for the student weights
            metrics: Keras metrics for evaluation
            student_loss_fn: Loss function of difference between student
                predictions and ground-truth
            distillation_loss_fn: Loss function of difference between soft
                student predictions and soft teacher predictions
            alpha: weight to student_loss_fn and 1-alpha to distillation_loss_fn
            temperature: Temperature for softening probability distributions.
                Larger temperature gives softer distributions.
        """
        super(Distiller, self).compile(optimizer=optimizer, metrics=metrics)
        self.student_loss_fn = student_loss_fn
        self.distillation_loss_fn = distillation_loss_fn
        self.alpha = alpha
        self.temperature = temperature

    def train_step(self, data):
        # Unpack data
        x, y = data
        # Forward pass of teacher
        teacher_predictions = self.teacher(x, training=False)
        with tf.GradientTape() as tape:
            # Forward pass of student
            student_predictions = self.student(x, training=True)
            # Compute losses
            student_loss = self.student_loss_fn(y, student_predictions)
            distillation_loss = self.distillation_loss_fn(
                tf.nn.softmax(teacher_predictions / self.temperature, axis=1),
                tf.nn.softmax(student_predictions / self.temperature, axis=1),
            )
            loss = self.alpha * student_loss + (1 - self.alpha) * distillation_loss
        # Compute gradients
        trainable_vars = self.student.trainable_variables
        gradients = tape.gradient(loss, trainable_vars)
        # Update weights
        self.optimizer.apply_gradients(zip(gradients, trainable_vars))
        # Update the metrics configured in `compile()`.
        self.compiled_metrics.update_state(y, student_predictions)
        # Return a dict of performance
        results = {m.name: m.result() for m in self.metrics}
        results.update(
            {"student_loss": student_loss, "distillation_loss": distillation_loss}
        )
        return results

    def test_step(self, data):
        # Unpack the data
        x, y = data
        # Compute predictions
        y_prediction = self.student(x, training=False)
        # Calculate the loss
        student_loss = self.student_loss_fn(y, y_prediction)
        # Update the metrics.
        self.compiled_metrics.update_state(y, y_prediction)
        # Return a dict of performance
        results = {m.name: m.result() for m in self.metrics}
        results.update({"student_loss": student_loss})
        return results

    def call(self, data, training=False):
        # You don't need this method for training.
        # So just pass.
        pass


import splitfolders

splitfolders.ratio(
    "../input/plantvillage-dataset/color",
    output="output",
    seed=1337,
    ratio=(0.8, 0.1, 0.1),
)


# # 4- Flowing Images from directory
# In step 4, the traning and validation datasets are flown from the directory to a variables called train_generator and validation_genearator. we also specify the batch size and resize all the images to 250 by 250px.
TRAINING_DIR = "./output/train"
train_datagen = ImageDataGenerator(rescale=1.0 / 255)
train_generator = train_datagen.flow_from_directory(
    TRAINING_DIR, class_mode="categorical", batch_size=250, target_size=(250, 250)
)
VALIDATION_DIR = "./output/test"
validation_datagen = ImageDataGenerator(rescale=1.0 / 255.0)
validation_generator = validation_datagen.flow_from_directory(
    VALIDATION_DIR, class_mode="categorical", batch_size=250, target_size=(250, 250)
)


# # Sample Images From the Dataset
# The code below defines a method plotImages which defined how to  plot sample images from the dataset using matplotlib library and imshow() method.  The Imagedatagenerator output array train_generator was used as input array to the method and then  five images  were selected from it.
def plotImages(images_arr):
    fig, axes = plt.subplots(1, 5, figsize=(20, 20))
    axes = axes.flatten()
    for img, ax in zip(images_arr, axes):
        ax.imshow(img)
    plt.tight_layout()
    plt.show()


augmented_images = [train_generator[0][0][0] for i in range(5)]
plotImages(augmented_images)


# # 5- Getting Categorical Labels
# We obtained the the labels associated with each images that is all the directories name from our dataset and assign it to a varaible called train_y
train_y = train_generator.classes

# # 6- Converting categorical labels to binary otherwise called (Encoding)
# Label Binarizer is an SciKit Learn class that accepts Categorical data as input and returns an Numpy array. Unlike Label Encoder, it encodes the data into dummy variables indicating the presence of a particular label or not. Encoding make column data using Label Binarizer
label_binarizer = LabelBinarizer()
image_labels = label_binarizer.fit_transform(train_y)
pickle.dump(label_binarizer, open("label_transform.pkl", "wb"))
n_classes = len(label_binarizer.classes_)

# ## 7- Create student and teacher models
# Initialy, we create a teacher model and a smaller student model using keras. Both models are
# convolutional neural networks and created using `Sequential()`,
# but could be any Keras model.
# Create the teacher
teacher = keras.Sequential(
    name="teacher",
)
inputShape = (250, 250, 3)
chanDim = -1
if K.image_data_format() == "channels_first":
    inputShape = (3, 250, 250)
    chanDim = 1
teacher.add(Conv2D(32, (3, 3), padding="same", input_shape=inputShape))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization(axis=chanDim))
teacher.add(MaxPooling2D(pool_size=(3, 3)))
teacher.add(Dropout(0.25))
teacher.add(Conv2D(64, (3, 3), padding="same"))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization(axis=chanDim))
teacher.add(Conv2D(64, (3, 3), padding="same"))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization(axis=chanDim))
teacher.add(MaxPooling2D(pool_size=(2, 2)))
teacher.add(Dropout(0.25))
teacher.add(Conv2D(128, (3, 3), padding="same"))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization(axis=chanDim))
teacher.add(Conv2D(128, (3, 3), padding="same"))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization(axis=chanDim))
teacher.add(MaxPooling2D(pool_size=(2, 2)))
teacher.add(Dropout(0.25))
teacher.add(Flatten())
teacher.add(Dense(1024))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization())
teacher.add(Dropout(0.5))
teacher.add(Dense(n_classes))
# Create the student
student = keras.Sequential(
    [
        keras.Input(shape=(250, 250, 3)),
        layers.Conv2D(16, (3, 3), strides=(2, 2), padding="same"),
        layers.LeakyReLU(alpha=0.2),
        layers.MaxPooling2D(pool_size=(2, 2), strides=(1, 1), padding="same"),
        layers.Conv2D(32, (3, 3), strides=(2, 2), padding="same"),
        layers.LeakyReLU(alpha=0.2),
        layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding="same"),
        layers.Flatten(),
        layers.Dense(n_classes),
    ],
    name="student",
)
# Clone student for later comparison
student_scratch = keras.models.clone_model(student)

# # 8- Displaying the model summary
# the custom method 'summary' allows us to diplay the architecture of the models described above, this allows us to know thw number parameters avaible in our defined model.
teacher.summary()
student.summary()

# # 9- Compiling Teacher Model for training
# The teacher model is first compile using custom compile method to achieve better accucaracy and learn more information
# Train teacher as usual
teacher.compile(
    optimizer=keras.optimizers.Adam(),
    loss=keras.losses.CategoricalCrossentropy(from_logits=True),
    metrics=["acc"],
)

# ## 10- Train the teacher
# During training Step, the model sifts through preexisting data and draws conclusions based on what it “thinks” the data represents. Every time it comes to an incorrect conclusion, that result is fed back to the system so that it “learns” from its mistake.
# This process makes connections between the artificial neurons stronger over time and increases the likelihood that the system will make accurate predictions in the future. As it’s presented with novel data, the DNN should be able to categorize and analyze new and possibly more complex information. Ultimately, it will continue to learn from its encounters and become more intuitive over time.
# * ** to train a model a fit() method is called on the model, fit takes arguments such as trainig data, validation data, epoch, train step and valudation step.
#
teacher_history = teacher.fit(
    train_generator, validation_data=validation_generator, epochs=5
)

# # 11- Plotting Teacher Model Accuracy and loss Curves
# * **Accruacy curve:** The accuracy curve describes the accuracy of test and validation sets. The X-axis is the iteration step. The Y-axis is the prediction accuracy. The blue curve is the accuracy on a batch of training examples from the training set, the orange curve is the accuracy for the test set.
# * **Loss curve:** One of the most used plots to debug a neural network is a Loss curve during training. It gives us a snapshot of the training process and the direction in which the network learns
import matplotlib.pyplot as plt

acc = teacher_history.history["acc"]
val_acc = teacher_history.history["val_acc"]
loss = teacher_history.history["loss"]
val_loss = teacher_history.history["val_loss"]
print(range(1, len(acc) + 1))
epochs = range(1, len(acc) + 1)
# Train and validation accuracy
plt.plot(epochs, acc, "b", label="Training accurarcy")
plt.plot(epochs, val_acc, "r", label="Validation accurarcy")
plt.title("Training and Validation accurarcy")
plt.legend()
plt.figure()
# Train and validation loss
plt.plot(epochs, loss, "b", label="Training loss")
plt.plot(epochs, val_loss, "r", label="Validation loss")
plt.title("Training and Validation loss")
plt.legend()
plt.show()

# ## 12 Distilling teacher knowledge to student Model
# We have already trained the teacher model, and we only need to initialize a
# `Distiller(student, teacher)` instance, `compile()` it with the desired losses,
# hyperparameters and optimizer, and distill the teacher to the student.
# Here we are trying to calculate the general loss by conputing KL divergence between student loss and distillation using the following relation.
# Dist_loss=  1/m ∑_(j=0)^m〖(2t^2 αD_KL (p^j,q^j )-(1-α)∑_(i=1)^cyi〗^(j  )  log⁡(1-〖yhat〗^j)〗
# where:
# P: is the soft labels from the teacher network
# q: is the softmax scores of student model
# Dkl: is the Kullback- Liebler (KL) divergence between p and q
# α: is the relative importance of the teacher’s guidance
# The distillation loss was minimized using the KL divergence between soft labels of a teacher p, and student model score q
#
# Initialize and compile distiller
distiller = Distiller(student=student, teacher=teacher)
distiller.compile(
    optimizer=keras.optimizers.Adam(),
    metrics=["acc"],
    student_loss_fn=keras.losses.CategoricalCrossentropy(from_logits=True),
    distillation_loss_fn=keras.losses.KLDivergence(),
    alpha=0.1,
    temperature=10,
)
# Distill teacher to student
student_history = distiller.fit(
    train_generator, validation_data=validation_generator, epochs=3
)

# # Plotting Student model Accuracy and loss curves
# to investigate the performance of the student model for comparison with teacher model, an accaucry and validation curve was plotted below.
import matplotlib.pyplot as plt

acc = student_history.history["acc"]
val_acc = student_history.history["val_acc"]
loss = student_history.history["distillation_loss"]
val_loss = student_history.history["student_loss"]
print(range(1, len(acc) + 1))
epochs = range(1, len(acc) + 1)
# Train and validation accuracy
plt.plot(epochs, acc, "b", label="Training accurarcy")
plt.plot(epochs, val_acc, "r", label="Validation accurarcy")
plt.title("Training and Validation accurarcy")
plt.legend()
plt.figure()
# Train and validation loss
plt.plot(epochs, loss, "b", label="Training loss")
plt.plot(epochs, val_loss, "r", label="Validation loss")
plt.title("Training and Validation loss")
plt.legend()
plt.show()

# ## Train student from scratch for comparison
# We can also train an equivalent student model from scratch without the teacher, in order
# to evaluate the performance gain obtained by knowledge distillation.
# Train student as doen usually
student_scratch.compile(
    optimizer=keras.optimizers.Adam(),
    loss=keras.losses.CategoricalCrossentropy(from_logits=True),
    metrics=["acc"],
)
# Train and evaluate student trained from scratch.
student_scratch.fit(train_generator, epochs=1)
std_history = student_scratch.evaluate(validation_generator)
<|endoftext|>
[{"Id": 113042082, "ScriptId": 29154796, "ParentScriptVersionId": NaN, "ScriptLanguageId": 9, "AuthorUserId": 4773887, "CreationDate": "12/05/2022 22:39:12", "VersionNumber": 2.0, "Title": "Lightweight Model of Disease Detection", "EvaluationDate": "12/05/2022", "IsChange": true, "TotalLines": 408.0, "LinesInsertedFromPrevious": 108.0, "LinesChangedFromPrevious": 0.0, "LinesUnchangedFromPrevious": 300.0, "LinesInsertedFromFork": NaN, "LinesDeletedFromFork": NaN, "LinesChangedFromFork": NaN, "LinesUnchangedFromFork": NaN, "TotalVotes": 3}]
[{"Id": 158070707, "KernelVersionId": 113042082, "SourceDatasetVersionId": 658267}]
[{"Id": 658267, "DatasetId": 277323, "DatasourceVersionId": 677630, "CreatorUserId": 3478941, "LicenseName": "CC BY-NC-SA 4.0", "CreationDate": "09/01/2019 11:52:26", "VersionNumber": 3.0, "Title": "PlantVillage Dataset", "Slug": "plantvillage-dataset", "Subtitle": "Dataset of diseased plant leaf images and corresponding labels", "Description": "Human society needs to increase food production by an estimated 70% by 2050 to feed an expected population size that is predicted to be over 9 billion people. Currently, infectious diseases reduce the potential yield by an average of 40% with many farmers in the developing world experiencing yield losses as high as 100%. The widespread distribution of smartphones among crop growers around the world with an expected 5 billion smartphones by 2020 offers the potential of turning the smartphone into a valuable tool for diverse communities growing food. One potential application is the development of mobile disease diagnostics through machine learning and crowdsourcing. Here we announce the release of over 50,000 expertly curated images on healthy and infected leaves of crops plants through the existing online platform PlantVillage. We describe both the data and the platform. These data are the beginning of an on-going, crowdsourcing effort to enable computer vision approaches to help solve the problem of yield losses in crop plants due to infectious diseases.", "VersionNotes": "PlantVillage Dataset", "TotalCompressedBytes": 0.0, "TotalUncompressedBytes": 2184925679.0}]
[{"Id": 277323, "CreatorUserId": 3478941, "OwnerUserId": 3682811.0, "OwnerOrganizationId": NaN, "CurrentDatasetVersionId": 658267.0, "CurrentDatasourceVersionId": 677630.0, "ForumId": 288696, "Type": 2, "CreationDate": "07/26/2019 10:40:16", "LastActivityDate": "07/26/2019", "TotalViews": 107780, "TotalDownloads": 13555, "TotalVotes": 322, "TotalKernels": 45}]
# # Knowledge Distillation
# **Author:** [Aminu Musa]
# **Date created:** 2022/09/01
# **Last modified:** 2020/10/17
# **Description:** Lighweight Plant Disease Detection Model for Embedded Device Using Knowledge Distillation Technique.
# This work is from a paper titled "Low-Power Deep Learning Model for Plant Disease Detection for Smart-Hydroponics Using Knowledge Distillation Techniques" link to the paper: https://www.mdpi.com/1605040
# the Techniques of Knowledge Distillation was a from a paper by Geofrey Hinton.https://arxiv.org/abs/1503.02531
# #   **INTRODUCTION**
# In this work We tried to implement Knowledge Distillation(KD) technique as proposed by Geofrey Hinton in thier paper titled  "Distilling the Knowledge in a Neural Network" the technique was adopted in our paper called Lighweight Plant Disease Detection Model for Embedded Device with the aim of achieving a lighter CNN model for plant disease detection.
# * **Goal**:
# Our aim is to developed a deep learning model that will aid African farmers in detecting diseases that are affecting thier crops using thier smartphones.the start of the art CNN models are not suitable for deployment on smartphones due to thier sizes and huge number of parametres. We therefore, compressed the model using KD to reduce model parameters while keeping the performance. we use open source plant disease detection dataset organized by PlantVIllage to train and validate our model.
# * **Contribution:**
# To obtain a lighweight model that will be suitable for deployment on embedded devices such as smartphone
# # 1- Uplaoding necessary libraries
import os
import pickle
import zipfile
import random
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import RMSprop
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from shutil import copyfile
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (
    BatchNormalization,
    Conv2D,
    MaxPooling2D,
    Activation,
    Flatten,
    Dropout,
    Dense,
)
from tensorflow.keras import backend as K
from sklearn.preprocessing import LabelBinarizer
from keras.preprocessing import image
import matplotlib.pyplot as plt


# #  2- How Does Knowledge Distillation Works?
# Knowledge distillation is a technique of model compression that employs teacher-student learning. In this technique, a large, cumbersome pre-trained model called the teacher model is trained first on a large dataset. The knowledge learned by the teacher model is transferred (distilled) to a shallow lightweight model called the student model, whereby the student mimics the performance of the teacher model despite its shallow nature. The student model size is relatively reduced without sacrificing accuracy. Figure.1 presents the architecture of the knowledge distillation technique.
# ![image.png](attachment:b9b79385-2927-499c-b3fd-fb74d22d7d33.png)
# # 3- Creating Distiller Class
# ## Construct `Distiller()` class
# This custom class`Distiller()` was used to overrides the  original Model methods namely:`train_step`, `test_step`,
# and `compile()`.
# * **Train_step:** A training step is one gradient update. In one step batch_size many examples are processed. An epoch consists of one full cycle through the training data. This is usually many steps. As an example, if you have 2,000 images and use a batch size of 10 an epoch consists of 2,000 images / (10 images / step) = 200 steps
# * **Compile method:** It specifies a loss, metrics, and an optimizer. To train a model with fit(), you need to specify a loss function, an optimizer, and optionally, some metrics to monitor. We can pass these variables to the model as arguments to the compile() method.
# In order to use the distiller, we need:
# - A trained teacher model
# - A student model to train
# - A student loss function on the difference between student predictions and ground-truth
# - A distillation loss function, along with a `temperature`, on the difference between the
# soft student predictions and the soft teacher labels
# - An `alpha` factor to weight the student and distillation loss
# - An optimizer for the student and (optional) metrics to evaluate performance
# In the `train_step` method, we perform a forward pass of both the teacher and student,
# calculate the loss with weighting of the `student_loss` and `distillation_loss` by `alpha` and
# `1 - alpha`, respectively, and perform the backward pass. Note: only the student weights are updated,
# and therefore we only calculate the gradients for the student weights.
# In the `test_step` method, we evaluate the student model on the provided dataset.
class Distiller(keras.Model):
    def __init__(self, student, teacher):
        super(Distiller, self).__init__()
        self.teacher = teacher
        self.student = student

    def compile(
        self,
        optimizer,
        metrics,
        student_loss_fn,
        distillation_loss_fn,
        alpha=0.1,
        temperature=3,
    ):
        """Configure the distiller.
        Args:
            optimizer: Keras optimizer for the student weights
            metrics: Keras metrics for evaluation
            student_loss_fn: Loss function of difference between student
                predictions and ground-truth
            distillation_loss_fn: Loss function of difference between soft
                student predictions and soft teacher predictions
            alpha: weight to student_loss_fn and 1-alpha to distillation_loss_fn
            temperature: Temperature for softening probability distributions.
                Larger temperature gives softer distributions.
        """
        super(Distiller, self).compile(optimizer=optimizer, metrics=metrics)
        self.student_loss_fn = student_loss_fn
        self.distillation_loss_fn = distillation_loss_fn
        self.alpha = alpha
        self.temperature = temperature

    def train_step(self, data):
        # Unpack data
        x, y = data
        # Forward pass of teacher
        teacher_predictions = self.teacher(x, training=False)
        with tf.GradientTape() as tape:
            # Forward pass of student
            student_predictions = self.student(x, training=True)
            # Compute losses
            student_loss = self.student_loss_fn(y, student_predictions)
            distillation_loss = self.distillation_loss_fn(
                tf.nn.softmax(teacher_predictions / self.temperature, axis=1),
                tf.nn.softmax(student_predictions / self.temperature, axis=1),
            )
            loss = self.alpha * student_loss + (1 - self.alpha) * distillation_loss
        # Compute gradients
        trainable_vars = self.student.trainable_variables
        gradients = tape.gradient(loss, trainable_vars)
        # Update weights
        self.optimizer.apply_gradients(zip(gradients, trainable_vars))
        # Update the metrics configured in `compile()`.
        self.compiled_metrics.update_state(y, student_predictions)
        # Return a dict of performance
        results = {m.name: m.result() for m in self.metrics}
        results.update(
            {"student_loss": student_loss, "distillation_loss": distillation_loss}
        )
        return results

    def test_step(self, data):
        # Unpack the data
        x, y = data
        # Compute predictions
        y_prediction = self.student(x, training=False)
        # Calculate the loss
        student_loss = self.student_loss_fn(y, y_prediction)
        # Update the metrics.
        self.compiled_metrics.update_state(y, y_prediction)
        # Return a dict of performance
        results = {m.name: m.result() for m in self.metrics}
        results.update({"student_loss": student_loss})
        return results

    def call(self, data, training=False):
        # You don't need this method for training.
        # So just pass.
        pass


import splitfolders

splitfolders.ratio(
    "../input/plantvillage-dataset/color",
    output="output",
    seed=1337,
    ratio=(0.8, 0.1, 0.1),
)


# # 4- Flowing Images from directory
# In step 4, the traning and validation datasets are flown from the directory to a variables called train_generator and validation_genearator. we also specify the batch size and resize all the images to 250 by 250px.
TRAINING_DIR = "./output/train"
train_datagen = ImageDataGenerator(rescale=1.0 / 255)
train_generator = train_datagen.flow_from_directory(
    TRAINING_DIR, class_mode="categorical", batch_size=250, target_size=(250, 250)
)
VALIDATION_DIR = "./output/test"
validation_datagen = ImageDataGenerator(rescale=1.0 / 255.0)
validation_generator = validation_datagen.flow_from_directory(
    VALIDATION_DIR, class_mode="categorical", batch_size=250, target_size=(250, 250)
)


# # Sample Images From the Dataset
# The code below defines a method plotImages which defined how to  plot sample images from the dataset using matplotlib library and imshow() method.  The Imagedatagenerator output array train_generator was used as input array to the method and then  five images  were selected from it.
def plotImages(images_arr):
    fig, axes = plt.subplots(1, 5, figsize=(20, 20))
    axes = axes.flatten()
    for img, ax in zip(images_arr, axes):
        ax.imshow(img)
    plt.tight_layout()
    plt.show()


augmented_images = [train_generator[0][0][0] for i in range(5)]
plotImages(augmented_images)


# # 5- Getting Categorical Labels
# We obtained the the labels associated with each images that is all the directories name from our dataset and assign it to a varaible called train_y
train_y = train_generator.classes

# # 6- Converting categorical labels to binary otherwise called (Encoding)
# Label Binarizer is an SciKit Learn class that accepts Categorical data as input and returns an Numpy array. Unlike Label Encoder, it encodes the data into dummy variables indicating the presence of a particular label or not. Encoding make column data using Label Binarizer
label_binarizer = LabelBinarizer()
image_labels = label_binarizer.fit_transform(train_y)
pickle.dump(label_binarizer, open("label_transform.pkl", "wb"))
n_classes = len(label_binarizer.classes_)

# ## 7- Create student and teacher models
# Initialy, we create a teacher model and a smaller student model using keras. Both models are
# convolutional neural networks and created using `Sequential()`,
# but could be any Keras model.
# Create the teacher
teacher = keras.Sequential(
    name="teacher",
)
inputShape = (250, 250, 3)
chanDim = -1
if K.image_data_format() == "channels_first":
    inputShape = (3, 250, 250)
    chanDim = 1
teacher.add(Conv2D(32, (3, 3), padding="same", input_shape=inputShape))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization(axis=chanDim))
teacher.add(MaxPooling2D(pool_size=(3, 3)))
teacher.add(Dropout(0.25))
teacher.add(Conv2D(64, (3, 3), padding="same"))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization(axis=chanDim))
teacher.add(Conv2D(64, (3, 3), padding="same"))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization(axis=chanDim))
teacher.add(MaxPooling2D(pool_size=(2, 2)))
teacher.add(Dropout(0.25))
teacher.add(Conv2D(128, (3, 3), padding="same"))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization(axis=chanDim))
teacher.add(Conv2D(128, (3, 3), padding="same"))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization(axis=chanDim))
teacher.add(MaxPooling2D(pool_size=(2, 2)))
teacher.add(Dropout(0.25))
teacher.add(Flatten())
teacher.add(Dense(1024))
teacher.add(Activation("relu"))
teacher.add(BatchNormalization())
teacher.add(Dropout(0.5))
teacher.add(Dense(n_classes))
# Create the student
student = keras.Sequential(
    [
        keras.Input(shape=(250, 250, 3)),
        layers.Conv2D(16, (3, 3), strides=(2, 2), padding="same"),
        layers.LeakyReLU(alpha=0.2),
        layers.MaxPooling2D(pool_size=(2, 2), strides=(1, 1), padding="same"),
        layers.Conv2D(32, (3, 3), strides=(2, 2), padding="same"),
        layers.LeakyReLU(alpha=0.2),
        layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding="same"),
        layers.Flatten(),
        layers.Dense(n_classes),
    ],
    name="student",
)
# Clone student for later comparison
student_scratch = keras.models.clone_model(student)

# # 8- Displaying the model summary
# the custom method 'summary' allows us to diplay the architecture of the models described above, this allows us to know thw number parameters avaible in our defined model.
teacher.summary()
student.summary()

# # 9- Compiling Teacher Model for training
# The teacher model is first compile using custom compile method to achieve better accucaracy and learn more information
# Train teacher as usual
teacher.compile(
    optimizer=keras.optimizers.Adam(),
    loss=keras.losses.CategoricalCrossentropy(from_logits=True),
    metrics=["acc"],
)

# ## 10- Train the teacher
# During training Step, the model sifts through preexisting data and draws conclusions based on what it “thinks” the data represents. Every time it comes to an incorrect conclusion, that result is fed back to the system so that it “learns” from its mistake.
# This process makes connections between the artificial neurons stronger over time and increases the likelihood that the system will make accurate predictions in the future. As it’s presented with novel data, the DNN should be able to categorize and analyze new and possibly more complex information. Ultimately, it will continue to learn from its encounters and become more intuitive over time.
# * ** to train a model a fit() method is called on the model, fit takes arguments such as trainig data, validation data, epoch, train step and valudation step.
#
teacher_history = teacher.fit(
    train_generator, validation_data=validation_generator, epochs=5
)

# # 11- Plotting Teacher Model Accuracy and loss Curves
# * **Accruacy curve:** The accuracy curve describes the accuracy of test and validation sets. The X-axis is the iteration step. The Y-axis is the prediction accuracy. The blue curve is the accuracy on a batch of training examples from the training set, the orange curve is the accuracy for the test set.
# * **Loss curve:** One of the most used plots to debug a neural network is a Loss curve during training. It gives us a snapshot of the training process and the direction in which the network learns
import matplotlib.pyplot as plt

acc = teacher_history.history["acc"]
val_acc = teacher_history.history["val_acc"]
loss = teacher_history.history["loss"]
val_loss = teacher_history.history["val_loss"]
print(range(1, len(acc) + 1))
epochs = range(1, len(acc) + 1)
# Train and validation accuracy
plt.plot(epochs, acc, "b", label="Training accurarcy")
plt.plot(epochs, val_acc, "r", label="Validation accurarcy")
plt.title("Training and Validation accurarcy")
plt.legend()
plt.figure()
# Train and validation loss
plt.plot(epochs, loss, "b", label="Training loss")
plt.plot(epochs, val_loss, "r", label="Validation loss")
plt.title("Training and Validation loss")
plt.legend()
plt.show()

# ## 12 Distilling teacher knowledge to student Model
# We have already trained the teacher model, and we only need to initialize a
# `Distiller(student, teacher)` instance, `compile()` it with the desired losses,
# hyperparameters and optimizer, and distill the teacher to the student.
# Here we are trying to calculate the general loss by conputing KL divergence between student loss and distillation using the following relation.
# Dist_loss=  1/m ∑_(j=0)^m〖(2t^2 αD_KL (p^j,q^j )-(1-α)∑_(i=1)^cyi〗^(j  )  log⁡(1-〖yhat〗^j)〗
# where:
# P: is the soft labels from the teacher network
# q: is the softmax scores of student model
# Dkl: is the Kullback- Liebler (KL) divergence between p and q
# α: is the relative importance of the teacher’s guidance
# The distillation loss was minimized using the KL divergence between soft labels of a teacher p, and student model score q
#
# Initialize and compile distiller
distiller = Distiller(student=student, teacher=teacher)
distiller.compile(
    optimizer=keras.optimizers.Adam(),
    metrics=["acc"],
    student_loss_fn=keras.losses.CategoricalCrossentropy(from_logits=True),
    distillation_loss_fn=keras.losses.KLDivergence(),
    alpha=0.1,
    temperature=10,
)
# Distill teacher to student
student_history = distiller.fit(
    train_generator, validation_data=validation_generator, epochs=3
)

# # Plotting Student model Accuracy and loss curves
# to investigate the performance of the student model for comparison with teacher model, an accaucry and validation curve was plotted below.
import matplotlib.pyplot as plt

acc = student_history.history["acc"]
val_acc = student_history.history["val_acc"]
loss = student_history.history["distillation_loss"]
val_loss = student_history.history["student_loss"]
print(range(1, len(acc) + 1))
epochs = range(1, len(acc) + 1)
# Train and validation accuracy
plt.plot(epochs, acc, "b", label="Training accurarcy")
plt.plot(epochs, val_acc, "r", label="Validation accurarcy")
plt.title("Training and Validation accurarcy")
plt.legend()
plt.figure()
# Train and validation loss
plt.plot(epochs, loss, "b", label="Training loss")
plt.plot(epochs, val_loss, "r", label="Validation loss")
plt.title("Training and Validation loss")
plt.legend()
plt.show()

# ## Train student from scratch for comparison
# We can also train an equivalent student model from scratch without the teacher, in order
# to evaluate the performance gain obtained by knowledge distillation.
# Train student as doen usually
student_scratch.compile(
    optimizer=keras.optimizers.Adam(),
    loss=keras.losses.CategoricalCrossentropy(from_logits=True),
    metrics=["acc"],
)
# Train and evaluate student trained from scratch.
student_scratch.fit(train_generator, epochs=1)
std_history = student_scratch.evaluate(validation_generator)
