## Exercise 1 (4 points): Basic TensorFlow Computations

TensorFlow is a software library for numerical operations which in its core uses the concept of data flow graphs. In such a graph, we can define mathematical operations and let tensor objects flow through the graph to compute an output. The graph representation is especially useful to determine which parts of the code can be executed in parallel (including devices like a GPU). In TensorFlow, this is done automatically without active intervention by the user.

In this first exercise, we want to get a basic idea of how such a data flow graph works, which objects we can use and how we define the operations. For this, we use the simple example function
\begin{equation} f(x) = a(x=1)^2 + bx \end{equation}<br/>
with the constants 𝑎 = 2 and 𝑏 = 4. For the test script, please pay also attention to Table 1.

1. [Python] Our first goal is to minimize Equation 1 numerically. For this, we need to define a graph which computes 𝑓 (𝑥) and treats 𝑥 as a variable so that we can apply gradient descent to optimize it.

    a) Let’s begin with the construction phase.
    
        i. Define a graph for 𝑓 (𝑥). 𝑎 and 𝑏 are constants (tf.constant) and 𝑥 should be treated as a variable (tf.Variable) with the initial value 𝑥0 = 0. The latter is important for our goals since variables maintain their state (their current value) and can be changed by other operations, i.e. 𝑥0 is the starting point for gradient descent.
    
        ii. An optimizer is defined by an operation which changes the variables according to a target function by computing the gradients. Define this operation which applies one step of gradient descent (tf.train.GradientDescentOptimi zer) to our target function 𝑓 (𝑥). Use a learning rate of 𝜂 = 0.05.
    
    b) It is now time to enter the execution phase. We need an active session to perform any operations in our graph. It is common to create a session enclosed in a with block so that the session is closed automatically when leaving the block. The session object also determines the lifetime of the variables as their state is only maintained during the currently active session.
    
        i. Even though we specified a value for our variable 𝑥, we still need to explicitly initialize it. Since it would be a bit laborious to initialize all variables manually, there is the global initializer operation tf.global_variables_initializer() which automatically initializes all variables in the graph. Execute this operation (tf.Operation objects have a run() method which you can use). 

        ii. Now, apply 30 steps of gradient descent by first executing the training operation and then retrieving the current values for 𝑓 (𝑥∗) and 𝑥∗. Show these values in each epoch. To stay efficient, you should calculate both values in the same graph run, i.e. do not use the eval() method of your tensors. If implemented correctly, the values in the last iteration should be close to the analytic solution 𝑓 (−2) = −6.

2. [Python] Let’s plot our function 𝑓 (𝑥) together with the minimum found. For this, we need
to evaluate 𝑓 (𝑥) over a predefined range. 𝑥 has now a different purpose than before: we
want to plug in arbitrary values instead of using it as a variable subject to optimization.
a) Implement Equation 1 again in a new graph (reset the default one via tf.reset_d
efault_graph()) but this time defining 𝑥 as a placeholder instead of a variable
(tf.placeholder). A placeholder is a node representing an arbitrary tensor which
is defined later in the execution phase with concrete values. In the construction
phase, they only have a type and a defined shape. If one of the shape components
is not known in advance (like here), it is possible to use None instead (the shape is
then set during the execution phase).
b) In a new session, evaluate 𝑓 (𝑥) for 50 values in the range [−6; 2]. You need the
feed_dict argument to specify the values for the placeholder 𝑥.
c) Plot 𝑓 (𝑥) in the defined range and highlight your minimum 𝑓 (𝑥∗) found previously
in the plot.

3. [Pen and Paper] We found the minimum of our function 𝑓 (𝑥) without ever specifying any gradient. This is possible because TensorFlow implements reverse-mode autodiff and we now want to go through the steps by calculating the derivative 𝑓 ′(2) = 16 via Table 2. The basis is the graph of our function as shown in Figure 1.

    a) Calculate the forward phase by computing the output of each node 𝑛𝑖 for the input 𝑥 = 2.

    b) Calculate the backward phase by computing ∂𝑓/∂𝑛𝑖, i.e. the derivative of each node 𝑛𝑖 with respect to the target function 𝑓 (𝑥). For this, you need the chain rule 
\begin{equation} \frac{\partial f}{\partial n_i} = \sum_{n \in parent(n_i)} \frac{\partial f}{\partial n} \frac{\partial n}{\partial n_i}\end{equation}
where the sum iterates over all parent nodes on which the node 𝑛𝑖 depends on (e.g. parent($n_5$) = {$n_6$}). Specify also the parents for the other nodes.

4. [Pen and Paper] Which operations in the graph of Figure 1 can be executed in parallel
(theoretically)?

## Exercise 2 (6 points): Neural Network in TensorFlow [Python]

Even though TensorFlow is a library to built data flow graphs suitable for general purposes, it is mostly used to implement neural networks. In this exercise, we want to see how this works for a simple neural network consisting only of a single hidden layer. There are multiple ways of defining the architecture of a network. Here, we are building it from scratch and in later
exercises we are going to use helper functions which hide some of the logic. 

We are using a small classification problem based on the wine dataset1 as an example. It classifies three different wines based on 𝑑 = 13 different (chemical) features for 𝑛 = 178 instances. The dataset has a very good class discrimination so that the neural network should not have any problems with it.

Please use the file TensorFlowNeuralNetwork.ipynb attached to this exercise as the basis for your implementation. For the test script, please pay also attention to Table 3.
1. The dataset is included in sklearn and you can load it via the load_wine function of the sklearn.datasets package.

2. We need to split the dataset so that we can use one part to train and another to evaluate our network. You can use the function sklearn.model_selection.train_test_s
plit for this job (with a test size of 30 %).

3. The feature dimensions have highly different ranges and it is hence a good idea to standardize our train and test sets (sklearn.preprocessing.StandardScaler). Make
sure you scale your test data only with the parameters extracted from the training data.

4. We have now all ingredients together to build our neural network which is also sketched in Figure 2. In the following, use named scopes whenever you think it is suited and add names to your nodes.

    a) Create the input layer by defining placeholders for the input data and the labels.
        • The input tensor should have a dimension of 𝑚 × 𝑑 whereas 𝑚 denotes the batch size and does not have to be specified yet, i.e. it can be set to None.
        • The tensor storing the class labels is a vector with a length equal to the batch size 𝑚.

    b) Define a hidden layer with 60 neurons which essentially compute the output 𝐻 = tanh(𝑋𝑊1 + 𝒃1) based on the input 𝑋, the weight matrix 𝑊1 and the bias vector 𝒃1. We want to learn 𝑊1 and 𝒃1 so they have to be defined as variables and initialized properly. 𝒃1 can be set to zero but 𝑊1 should be initialized according to a truncated normal distribution with an appropriate standard deviation <br/><br/>
\begin{equation} \sigma = \sqrt{\frac{2}{n_{in} + n_{out}}} \end{equation}
$n_{in}$ and $n_{out}$ denote the number of input and output connections of each neuron, respectively

    c) Define the output layer in a similar way but this time using linear transfer functions2 \begin{equation}y = HW_2 + b_2\end{equation} Note: the order in which the variables are defined matters for the test script so please define first 𝑊1 and 𝒃1 before you define 𝑊2 and 𝒃2.
    
    d) We need an error function to train our network. Since we are dealing with a classification problem, the cross-entropy function is a good choice. You can use $tf.nn.sparse\_softmax\_cross\_entropy\_with\_logits$ for this task. The function accepts the labels and the logits directly. Do not apply the softmax function yourself or convert the labels to one-hot vectors.

    e) The cross-entropy calculates an error value for each instance. Calculate the average over the complete batch dimension (tf.reduce_mean) to arrive at a scalar value.
    
    f) Like in Exercise 1, we need a training operation which adjusts the variables in the graph according to our error function. Define this operation but this time using tf.train.AdamOptimizer (default parameters are fine). This is a more advanced optimization technique than classical gradient descent.

    g) It is helpful to track the error as well as the accuracy on the training set during the learning process. For this, define a tensor which holds the accuracy
\begin{equation}acc = \frac{1}{m} \sum_{i=1}^m correct_i \hspace{0.5cm} with \hspace{0.5cm} correct_i = \left\{
        \begin{array}{ll}
            1, & argmax(y_i) = T_i\\
            0, & \quad else
        \end{array}
    \right. \end{equation}
over all 𝑚 instances in the current batch. 𝒚𝑖 denotes the output vector containing the logit value for each class of the 𝑖-th instance and 𝑇𝑖 ∈ {0, 1, 2} denotes the corresponding label.

    h) TensorFlow ships with a visualization tool called TensorBoard which can be used to explore the network and its result. To use it, we need to write some data about our model to a log folder which can then later be read by TensorBoard.
```python
now = datetime.utcnow().strftime('%Y-%m-%d %H;%M;%S')
logdir = '{}/run-{}/'.format('tf_logs', now)
file_writer = tf.summary.FileWriter(logdir, tf.get_default_graph())
```
which creates a log folder and already includes basic information about the graph. The file_writer object can later be used to add more information to the log. Don’t forget to close the file_writer at the end.

5. We can now enter the execution phase to train and evaluate our network.

    a) Run an operation which initializes all variables (similar to Exercise 1).

    b) Use 50 epochs to train the network. For simplicity, you can use full batches, i.e. supply all training instances at once (𝑛 = 𝑚). After each training step (in a separate graph run), evaluate the model on the training set by calculating the loss and the accuracy.
    
    c) We can include the evaluation metrics to the log folder so that TensorBoard can plot the result. The error and the loss are both simple scalar values which can be added to the log via
```python    
loss_summary = tf.Summary(value=[tf.Summary.Value(tag='loss', simple_value=loss_value)])
accuracy_summary = tf.Summary(value=[tf.Summary.Value(tag='accuracy', simple_value=acc_value)])
file_writer.add_summary(loss_summary, epoch)
file_writer.add_summary(accuracy_summary, epoch)
```
This first creates a binary string (optimized representation used by TensorFlow) and
then adds it to the log folder via the file_writer object.

    d) After finishing with the training, evaluate the network on the test set. For this, compute the accuracy as well as the confusion matrix. For the latter, you need to retrieve the corresponding logits and then create a vector storing the predicted labels which can be compared with the ground truth.

6. Explore your network in TensorBoard. You can open it by executing
```python
tensorboard --logdir tf_logs
```
on the command line in the working directory where the Jupyter notebook is located.

f8847bca7258e0f8d5521a0b0ad21d926b817730
ipynb
Jupyter Notebook
10_intro_to_tensorflow/basic_tensorflow.ipynb
jhinga-la-la/pattern-recognition-course
7ad4f70b2c427f3c37f59f47768b90371873823c
Apache-2.0
10_intro_to_tensorflow/basic_tensorflow.ipynb
jhinga-la-la/pattern-recognition-course
7ad4f70b2c427f3c37f59f47768b90371873823c
Apache-2.0
10_intro_to_tensorflow/basic_tensorflow.ipynb
jhinga-la-la/pattern-recognition-course
7ad4f70b2c427f3c37f59f47768b90371873823c
Apache-2.0
Qwen/Qwen-72B
1. YES
2. YES