# Métricas para avaliação de modelos

A escolha da métrica a ser otimizada depende da aplicação!

#### Métricas para regressão

MSE (mean squared error)

$
\begin{align}
\frac{1}{n} \sum_{i=1}^{n}(y_i - \hat{y}_i)^2
\end{align}
$

MAE (mean absolute error)

$
\begin{align}
\frac{1}{n} \sum_{i=1}^{n}(y_i - \hat{y}_i)
\end{align}
$

#### Métricas para classificação

Sejam:

$
\begin{align}
TP = true \ positives \\
TN = true \ negatives \\
FP = false \ positives \ (erro \ tipo \ I)\\
FN = false \ negatives \ (erro \ tipo \ II)
\end{align}
$

Precisão

$
\begin{align}
\frac{TP}{TP+FP}
\end{align}
$

Recall / sensibilidade

$
\begin{align}
\frac{TP}{TP+FN}
\end{align}
$

Acurácia

$
\begin{align}
\frac{TP+TN}{TP+TN+FP+FN}
\end{align}
$


F1 score: média harmônica de precisão e recall

$
\begin{align}
2 \cdot \frac{precision \cdot recall}{precision + recall}
\end{align}
$

## Validação cruzada

Consiste em particionar o dataset em k subconjuntos e treinar o modelo k vezes, usando um subconjunto diferente para validação por vez e os k-1 restantes para treino.

Vamos usar uma função auxiliar que recebe uma instância de um modelo, a quantidade de partições $k$, a matriz de *features* $X$, o vetor de *targets* $y$ e uma função que computa a métrica desejada.

Experimente variar $k$ e a função de métrica nos exemplos abaixo: classificação de flores e regressão de valores de casas.


```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.datasets import load_boston, load_iris
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import accuracy_score, recall_score, f1_score, mean_absolute_error, mean_squared_error
from sklearn.model_selection import KFold


def train_model(model, k, X, y, score_func):
    kfolds = KFold(k)
    scores = []
    for train_idx, test_idx in kfolds.split(X, y):
        X_train, y_train = X[train_idx], y[train_idx]
        X_test, y_test = X[test_idx], y[test_idx]
        model = model.fit(X_train, y_train)
        y_hat = model.predict(X_test)
        scores.append(score_func(y_test, y_hat))
    print('scores: ', np.round(scores,4), '\nmean:    ', np.round(np.mean(scores),4))
```

#### Classificação de flores

Dataset Iris.


```python
data_iris = load_iris()
df_iris = pd.DataFrame(data_iris.data, columns=data_iris.feature_names)

X_iris = df_iris.values
y_iris = data_iris.target
cols_iris = data_iris.feature_names

df_iris.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>sepal length (cm)</th>
      <th>sepal width (cm)</th>
      <th>petal length (cm)</th>
      <th>petal width (cm)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>5.1</td>
      <td>3.5</td>
      <td>1.4</td>
      <td>0.2</td>
    </tr>
    <tr>
      <th>1</th>
      <td>4.9</td>
      <td>3.0</td>
      <td>1.4</td>
      <td>0.2</td>
    </tr>
    <tr>
      <th>2</th>
      <td>4.7</td>
      <td>3.2</td>
      <td>1.3</td>
      <td>0.2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4.6</td>
      <td>3.1</td>
      <td>1.5</td>
      <td>0.2</td>
    </tr>
    <tr>
      <th>4</th>
      <td>5.0</td>
      <td>3.6</td>
      <td>1.4</td>
      <td>0.2</td>
    </tr>
  </tbody>
</table>
</div>



Podemos visualizar as *features* duas a duas, como no *snippet* abaixo. Experimente variar as *features* (i.e. `X_iris[:,0]` e `X_iris[:,1]`). Não se esqueça de também trocar as labels do gráfico.


```python
plt.scatter(X_iris[:,0], X_iris[:,1], c=y_iris, alpha=0.7)
plt.xlabel(cols_iris[0])
plt.ylabel(cols_iris[1])
```


```python
lr_iris = LogisticRegression(solver='lbfgs', max_iter=200, multi_class='multinomial')
train_model(lr_iris, 5, X_iris, y_iris, accuracy_score)
```

    scores:  [1.     1.     0.8667 0.9333 0.8333] 
    mean:     0.9267


#### Regressão de valores de casas

Dataset *Boston Housing*.


```python
data_boston = load_boston()
df_boston = pd.DataFrame(data_boston.data, columns=data_boston.feature_names)

X_boston = df_boston.values
y_boston = data_boston.target
cols_boston = data_boston.feature_names

df_boston.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>CRIM</th>
      <th>ZN</th>
      <th>INDUS</th>
      <th>CHAS</th>
      <th>NOX</th>
      <th>RM</th>
      <th>AGE</th>
      <th>DIS</th>
      <th>RAD</th>
      <th>TAX</th>
      <th>PTRATIO</th>
      <th>B</th>
      <th>LSTAT</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.00632</td>
      <td>18.0</td>
      <td>2.31</td>
      <td>0.0</td>
      <td>0.538</td>
      <td>6.575</td>
      <td>65.2</td>
      <td>4.0900</td>
      <td>1.0</td>
      <td>296.0</td>
      <td>15.3</td>
      <td>396.90</td>
      <td>4.98</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.02731</td>
      <td>0.0</td>
      <td>7.07</td>
      <td>0.0</td>
      <td>0.469</td>
      <td>6.421</td>
      <td>78.9</td>
      <td>4.9671</td>
      <td>2.0</td>
      <td>242.0</td>
      <td>17.8</td>
      <td>396.90</td>
      <td>9.14</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.02729</td>
      <td>0.0</td>
      <td>7.07</td>
      <td>0.0</td>
      <td>0.469</td>
      <td>7.185</td>
      <td>61.1</td>
      <td>4.9671</td>
      <td>2.0</td>
      <td>242.0</td>
      <td>17.8</td>
      <td>392.83</td>
      <td>4.03</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.03237</td>
      <td>0.0</td>
      <td>2.18</td>
      <td>0.0</td>
      <td>0.458</td>
      <td>6.998</td>
      <td>45.8</td>
      <td>6.0622</td>
      <td>3.0</td>
      <td>222.0</td>
      <td>18.7</td>
      <td>394.63</td>
      <td>2.94</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.06905</td>
      <td>0.0</td>
      <td>2.18</td>
      <td>0.0</td>
      <td>0.458</td>
      <td>7.147</td>
      <td>54.2</td>
      <td>6.0622</td>
      <td>3.0</td>
      <td>222.0</td>
      <td>18.7</td>
      <td>396.90</td>
      <td>5.33</td>
    </tr>
  </tbody>
</table>
</div>




```python
plt.scatter(X_boston[:,0], X_boston[:,1], alpha=0.7)
plt.xlabel(cols_boston[0])
plt.ylabel(cols_boston[1])
```


```python
lr_boston = LinearRegression()
train_model(lr_boston, 10, X_boston, y_boston, mean_absolute_error)
```

    scores:  [2.2069 2.8968 2.7867 4.5985 4.1099 3.5647 2.6697 9.6564 5.0227 2.5373] 
    mean:     4.0049


ec73f550860d9e506eefd5ec4883d5764455dada
ipynb
Jupyter Notebook
aulas/aula05_01_metricas.ipynb
flaviobp/introducao-aprendizado-maquina
87c2f57352a03dd39e2507a6f6ff33f5dd280aa9
MIT
aulas/aula05_01_metricas.ipynb
flaviobp/introducao-aprendizado-maquina
87c2f57352a03dd39e2507a6f6ff33f5dd280aa9
MIT
aulas/aula05_01_metricas.ipynb
flaviobp/introducao-aprendizado-maquina
87c2f57352a03dd39e2507a6f6ff33f5dd280aa9
MIT
Qwen/Qwen-72B
1. YES
2. YES