<kaggle_start><data_title>mlcourse.ai<data_description>Open Machine Learning Course [mlcourse.ai](http://mlcourse.ai/) is designed to perfectly balance theory and practice; therefore, each topic is followed by an assignment with a deadline in a week. You can also take part in several Kaggle Inclass competitions held during the course and write your own tutorials. The next session starts on **October 1, 2018**. Fill in [this form](https://docs.google.com/forms/d/1_pDNuVHwBxV5wuOcdaXoxBZneyAQcqfOl4V2qkqKbNQ/) to participate. More info in [GitHub repo](https://github.com/Yorko/mlcourse.ai). 

### Outline
This is the list of published articles on [medium.com](https://medium.com/open-machine-learning-course) (English), [habr.com](https://habr.com/company/ods/blog/344044/) (Russian), and [jqr.com](https://www.jqr.com/) (Chinese). See Kernels of this Dataset for the same material in English. 

1. Exploratory Data Analysis with Pandas [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-1-exploratory-data-analysis-with-pandas-de57880f1a68)  [ru](https://habrahabr.ru/company/ods/blog/322626/), [cn](https://www.jqr.com/article/000079), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-1-exploratory-data-analysis-with-pandas)
2. Visual Data Analysis with Python [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-2-visual-data-analysis-in-python-846b989675cd)  [ru](https://habrahabr.ru/company/ods/blog/323210/), [cn](https://www.jqr.com/article/000086), Kaggle Kernels: [part1](https://www.kaggle.com/kashnitsky/topic-2-visual-data-analysis-in-python), [part2](https://www.kaggle.com/kashnitsky/topic-2-part-2-seaborn-and-plotly)
3. Classification, Decision Trees and k Nearest Neighbors [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-3-classification-decision-trees-and-k-nearest-neighbors-8613c6b6d2cd), [ru](https://habrahabr.ru/company/ods/blog/322534/), [cn](https://www.jqr.com/article/000139), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-3-decision-trees-and-knn)
4. Linear Classification and Regression [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-4-linear-classification-and-regression-44a41b9b5220), [ru](https://habrahabr.ru/company/ods/blog/323890/), [cn](https://www.jqr.com/article/000175), Kaggle Kernels: [part1](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-1-ols), [part2](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-2-classification), [part3](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-3-regularization), [part4](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-4-more-of-logit), [part5](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-5-validation)
5. Bagging and Random Forest [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-5-ensembles-of-algorithms-and-random-forest-8e05246cbba7), [ru](https://habrahabr.ru/company/ods/blog/324402/), [cn](https://www.jqr.com/article/000241), Kaggle Kernels: [part1](https://www.kaggle.com/kashnitsky/topic-5-ensembles-part-1-bagging), [part2](https://www.kaggle.com/kashnitsky/topic-5-ensembles-part-2-random-forest), [part3](https://www.kaggle.com/kashnitsky/topic-5-ensembles-part-3-feature-importance)
6. Feature Engineering and Feature Selection [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-6-feature-engineering-and-feature-selection-8b94f870706a), [ru](https://habrahabr.ru/company/ods/blog/325422/), [cn](https://www.jqr.com/article/000249), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-6-feature-engineering-and-feature-selection)
7. Unsupervised Learning: Principal Component Analysis and Clustering [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-7-unsupervised-learning-pca-and-clustering-db7879568417), [ru](https://habrahabr.ru/company/ods/blog/325654/), [cn](https://www.jqr.com/article/000336), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-7-unsupervised-learning-pca-and-clustering)
8. Vowpal Wabbit: Learning with Gigabytes of Data [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-8-vowpal-wabbit-fast-learning-with-gigabytes-of-data-60f750086237), [ru](https://habrahabr.ru/company/ods/blog/326418/), [cn](https://www.jqr.com/article/000348), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-8-online-learning-and-vowpal-wabbit)
9. Time Series Analysis with Python, part 1 [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-9-time-series-analysis-in-python-a270cb05e0b3), [ru](https://habrahabr.ru/company/ods/blog/327242/), [cn](https://www.jqr.com/article/000450). Predicting future with Facebook Prophet, part 2 [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-9-part-3-predicting-the-future-with-facebook-prophet-3f3af145cdc), [cn](https://www.jqr.com/article/000598) Kaggle Kernels: [part1](https://www.kaggle.com/kashnitsky/topic-9-part-1-time-series-analysis-in-python), [part2](https://www.kaggle.com/kashnitsky/topic-9-part-2-time-series-with-facebook-prophet)
10. Gradient Boosting [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-10-gradient-boosting-c751538131ac), [ru](https://habrahabr.ru/company/ods/blog/327250/), [cn](https://www.jqr.com/article/000573), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-10-gradient-boosting)

### Assignments
Each topic is followed by an assignment. See demo versions in this Dataset. Solutions will be discussed in the upcoming run of the course.  

### Kaggle competitions
1. Catch Me If You Can: Intruder Detection through Webpage Session Tracking. [Kaggle Inclass](https://www.kaggle.com/c/catch-me-if-you-can-intruder-detection-through-webpage-session-tracking2)
2. How good is your Medium article? [Kaggle Inclass](https://www.kaggle.com/c/how-good-is-your-medium-article/)

### Rating
Throughout the course we are maintaining a student rating. It takes into account credits scored in assignments and Kaggle competitions. Top students (according to the final rating) will be listed on a special Wiki page.

### Community
Discussions between students are held in the **#mlcourse_ai** channel of the OpenDataScience Slack team. Fill in [this form](https://drive.google.com/open?id=1_pDNuVHwBxV5wuOcdaXoxBZneyAQcqfOl4V2qkqKbNQ) to get an invitation (to be sent in September 2018). The form will also ask you some personal questions, don't hesitate

### Collaboration
You can publish Kernels using this Dataset. But please respect others' interests: don't share solutions to assignments and well-performing solutions for Kaggle Inclass competitions. If you notice any typos/errors in course material, please open an [Issue](https://github.com/Yorko/mlcourse.ai/issues) or make a pull request in course [repo](https://github.com/Yorko/mlcourse.ai).<data_name>mlcourse
<code>#
#
# ## [mlcourse.ai](https://mlcourse.ai) - Open Machine Learning Course
# Authors: [Maria Sumarokova](https://www.linkedin.com/in/mariya-sumarokova-230b4054/), and [Yury Kashnitsky](https://www.linkedin.com/in/festline/). Translated and edited by Gleb Filatov, Aleksey Kiselev, [Anastasia Manokhina](https://www.linkedin.com/in/anastasiamanokhina/), [Egor Polusmak](https://www.linkedin.com/in/egor-polusmak/), and [Yuanyuan Pao](https://www.linkedin.com/in/yuanyuanpao/). All content is distributed under the [Creative Commons CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license.
# #  Assignment #3 (demo)
# ##   Decision trees with a toy task and the UCI Adult dataset
# Same assignment as a [Kaggle Kernel](https://www.kaggle.com/kashnitsky/a3-demo-decision-trees) + [solution](https://www.kaggle.com/kashnitsky/a3-demo-decision-trees-solution). Fill in the answers in the [web-form](https://docs.google.com/forms/d/1wfWYYoqXTkZNOPy1wpewACXaj2MZjBdLOL58htGWYBA/edit).
# Let's start by loading all necessary libraries:
from matplotlib import pyplot as plt

plt.rcParams["figure.figsize"] = (10, 8)
import seaborn as sns
import numpy as np
import pandas as pd
from sklearn.preprocessing import LabelEncoder
import collections
from sklearn.model_selection import GridSearchCV
from sklearn import preprocessing
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from ipywidgets import Image
from io import StringIO
import pydotplus  # pip install pydotplus


# ### Part 1. Toy dataset "Will They? Won't They?"
# Your goal is to figure out how decision trees work by walking through a toy problem. While a single decision tree does not yield outstanding results, other performant algorithms like gradient boosting and random forests are based on the same idea. That is why knowing how decision trees work might be useful.
# We'll go through a toy example of binary classification - Person A is deciding whether they will go on a second date with Person B. It will depend on their looks, eloquence, alcohol consumption (only for example), and how much money was spent on the first date.
# #### Creating the dataset
# Create dataframe with dummy variables
def create_df(dic, feature_list):
    out = pd.DataFrame(dic)
    out = pd.concat([out, pd.get_dummies(out[feature_list])], axis=1)
    out.drop(feature_list, axis=1, inplace=True)
    return out


# Some feature values are present in train and absent in test and vice-versa.
def intersect_features(train, test):
    common_feat = list(set(train.keys()) & set(test.keys()))
    return train[common_feat], test[common_feat]


features = ["Looks", "Alcoholic_beverage", "Eloquence", "Money_spent"]


# #### Training data
df_train = {}
df_train["Looks"] = [
    "handsome",
    "handsome",
    "handsome",
    "repulsive",
    "repulsive",
    "repulsive",
    "handsome",
]
df_train["Alcoholic_beverage"] = ["yes", "yes", "no", "no", "yes", "yes", "yes"]
df_train["Eloquence"] = ["high", "low", "average", "average", "low", "high", "average"]
df_train["Money_spent"] = ["lots", "little", "lots", "little", "lots", "lots", "lots"]
df_train["Will_go"] = LabelEncoder().fit_transform(["+", "-", "+", "-", "-", "+", "+"])
df_train = create_df(df_train, features)
df_train

# #### Test data
df_test = {}
df_test["Looks"] = ["handsome", "handsome", "repulsive"]
df_test["Alcoholic_beverage"] = ["no", "yes", "yes"]
df_test["Eloquence"] = ["average", "high", "average"]
df_test["Money_spent"] = ["lots", "little", "lots"]
df_test = create_df(df_test, features)
df_test
# Some feature values are present in train and absent in test and vice-versa.
y = df_train["Will_go"]
df_train, df_test = intersect_features(train=df_train, test=df_test)
df_train
df_test

# #### Draw a decision tree (by hand or in any graphics editor) for this dataset. Optionally you can also implement tree construction and draw it here.
# 1\. What is the entropy $S_0$ of the initial system? By system states, we mean values of the binary feature "Will_go" - 0 or 1 - two states in total.
# you code here

# 2\. Let's split the data by the feature "Looks_handsome". What is the entropy $S_1$ of the left group - the one with "Looks_handsome". What is the entropy $S_2$ in the opposite group? What is the information gain (IG) if we consider such a split?
# you code here

# #### Train a decision tree using sklearn on the training data. You may choose any depth for the tree.
# you code here

# #### Additional: display the resulting tree using graphviz. You can use pydot or [web-service](https://www.coolutils.com/ru/online/DOT-to-PNG) dot2png.
# you code here

# ### Part 2. Functions for calculating entropy and information gain.
# Consider the following warm-up example: we have 9 blue balls and 11 yellow balls. Let ball have label **1** if it is blue, **0** otherwise.
balls = [1 for i in range(9)] + [0 for i in range(11)]

#
# Next split the balls into two groups:
#
# two groups
balls_left = [1 for i in range(8)] + [0 for i in range(5)]  # 8 blue and 5 yellow
balls_right = [1 for i in range(1)] + [0 for i in range(6)]  # 1 blue and 6 yellow


# #### Implement a function to calculate the Shannon Entropy
def entropy(a_list):
    # you code here
    pass


# Tests
print(entropy(balls))  # 9 blue и 11 yellow
print(entropy(balls_left))  # 8 blue и 5 yellow
print(entropy(balls_right))  # 1 blue и 6 yellow
print(entropy([1, 2, 3, 4, 5, 6]))  # entropy of a fair 6-sided die


# 3\. What is the entropy of the state given by the list **balls_left**?
# 4\. What is the entropy of a fair dice? (where we look at a dice as a system with 6 equally probable states)?
# information gain calculation
def information_gain(root, left, right):
    """root - initial data, left and right - two partitions of initial data"""
    # you code here
    pass


# 5\. What is the information gain from splitting the initial dataset into **balls_left** and **balls_right** ?
def best_feature_to_split(X, y):
    """Outputs information gain when splitting on best feature"""
    # you code here
    pass


# #### Optional:
# - Implement a decision tree building algorithm by calling **best_feature_to_split** recursively
# - Plot the resulting tree
# ### Part 3. The "Adult" dataset
# #### Dataset description:
# [Dataset](http://archive.ics.uci.edu/ml/machine-learning-databases/adult) UCI Adult (no need to download it, we have a copy in the course repository): classify people using demographical data - whether they earn more than \$50,000 per year or not.
# Feature descriptions:
# - **Age** – continuous feature
# - **Workclass** –  continuous feature
# - **fnlwgt** – final weight of object, continuous feature
# - **Education** –  categorical feature
# - **Education_Num** – number of years of education, continuous feature
# - **Martial_Status** –  categorical feature
# - **Occupation** –  categorical feature
# - **Relationship** – categorical feature
# - **Race** – categorical feature
# - **Sex** – categorical feature
# - **Capital_Gain** – continuous feature
# - **Capital_Loss** – continuous feature
# - **Hours_per_week** – continuous feature
# - **Country** – categorical feature
# **Target** – earnings level, categorical (binary) feature.
# #### Reading train and test data
data_train = pd.read_csv("../../data/adult_train.csv", sep=";")
data_train.tail()
data_test = pd.read_csv("../../data/adult_test.csv", sep=";")
data_test.tail()
# necessary to remove rows with incorrect labels in test dataset
data_test = data_test[
    (data_test["Target"] == " >50K.") | (data_test["Target"] == " <=50K.")
]
# encode target variable as integer
data_train.loc[data_train["Target"] == " <=50K", "Target"] = 0
data_train.loc[data_train["Target"] == " >50K", "Target"] = 1
data_test.loc[data_test["Target"] == " <=50K.", "Target"] = 0
data_test.loc[data_test["Target"] == " >50K.", "Target"] = 1

# #### Primary data analysis
data_test.describe(include="all").T
data_train["Target"].value_counts()
fig = plt.figure(figsize=(25, 15))
cols = 5
rows = np.ceil(float(data_train.shape[1]) / cols)
for i, column in enumerate(data_train.columns):
    ax = fig.add_subplot(rows, cols, i + 1)
    ax.set_title(column)
    if data_train.dtypes[column] == np.object:
        data_train[column].value_counts().plot(kind="bar", axes=ax)
    else:
        data_train[column].hist(axes=ax)
        plt.xticks(rotation="vertical")
plt.subplots_adjust(hspace=0.7, wspace=0.2)

# #### Checking data types
data_train.dtypes
data_test.dtypes

# As we see, in the test data, age is treated as type **object**. We need to fix this.
data_test["Age"] = data_test["Age"].astype(int)

# Also we'll cast all **float** features to **int** type to keep types consistent between our train and test data.
data_test["fnlwgt"] = data_test["fnlwgt"].astype(int)
data_test["Education_Num"] = data_test["Education_Num"].astype(int)
data_test["Capital_Gain"] = data_test["Capital_Gain"].astype(int)
data_test["Capital_Loss"] = data_test["Capital_Loss"].astype(int)
data_test["Hours_per_week"] = data_test["Hours_per_week"].astype(int)

# #### Fill in missing data for continuous features with their median values, for categorical features with their mode.
# choose categorical and continuous features from data
categorical_columns = [
    c for c in data_train.columns if data_train[c].dtype.name == "object"
]
numerical_columns = [
    c for c in data_train.columns if data_train[c].dtype.name != "object"
]
print("categorical_columns:", categorical_columns)
print("numerical_columns:", numerical_columns)
# we see some missing values
data_train.info()
# fill missing data
for c in categorical_columns:
    data_train[c].fillna(data_train[c].mode()[0], inplace=True)
    data_test[c].fillna(data_train[c].mode()[0], inplace=True)
for c in numerical_columns:
    data_train[c].fillna(data_train[c].median(), inplace=True)
    data_test[c].fillna(data_train[c].median(), inplace=True)
# no more missing values
data_train.info()

# We'll dummy code some categorical features: **Workclass**, **Education**, **Martial_Status**, **Occupation**, **Relationship**, **Race**, **Sex**, **Country**. It can be done via pandas method **get_dummies**
data_train = pd.concat(
    [data_train[numerical_columns], pd.get_dummies(data_train[categorical_columns])],
    axis=1,
)
data_test = pd.concat(
    [data_test[numerical_columns], pd.get_dummies(data_test[categorical_columns])],
    axis=1,
)
set(data_train.columns) - set(data_test.columns)
data_train.shape, data_test.shape

# #### There is no Holland in the test data. Create new zero-valued feature.
data_test["Country_ Holand-Netherlands"] = 0
set(data_train.columns) - set(data_test.columns)
data_train.head(2)
data_test.head(2)
X_train = data_train.drop(["Target"], axis=1)
y_train = data_train["Target"]
X_test = data_test.drop(["Target"], axis=1)
y_test = data_test["Target"]

# ### 3.1 Decision tree without parameter tuning
# Train a decision tree **(DecisionTreeClassifier)** with a maximum depth of 3, and evaluate the accuracy metric on the test data. Use parameter **random_state = 17** for results reproducibility.
# you code here
# tree =
# tree.fit

# Make a prediction with the trained model on the test data.
# you code here
# tree_predictions = tree.predict
# you code here
# accuracy_score

# 6\. What is the test set accuracy of a decision tree with maximum tree depth of 3 and **random_state = 17**?
# ### 3.2 Decision tree with parameter tuning
# Train a decision tree **(DecisionTreeClassifier, random_state = 17).** Find the optimal maximum depth using 5-fold cross-validation **(GridSearchCV)**.
tree_params = {"max_depth": range(2, 11)}
locally_best_tree = GridSearchCV  # you code here
locally_best_tree.fit
# you code here

# Train a decision tree with maximum depth of 9 (it is the best **max_depth** in my case), and compute the test set accuracy. Use parameter **random_state = 17** for reproducibility.
# you code here
# tuned_tree =
# tuned_tree.fit
# tuned_tree_predictions = tuned_tree.predict
# accuracy_score

# 7\. What is the test set accuracy of a decision tree with maximum depth of 9 and **random_state = 17**?
# ### 3.3 (Optional) Random forest without parameter tuning
# Let's take a sneak peek of upcoming lectures and try to use a random forest for our task. For now, you can imagine a random forest as a bunch of decision trees, trained on slightly different subsets of the training data.
# Train a random forest **(RandomForestClassifier)**. Set the number of trees to 100 and use **random_state = 17**.
# you code here
# rf =
# rf.fit # you code here

# Make predictions for the test data and assess accuracy.
# you code here

# ### 3.4 (Optional) Random forest with parameter tuning
# Train a random forest **(RandomForestClassifier)**. Tune the maximum depth and maximum number of features for each tree using **GridSearchCV**.
# forest_params = {'max_depth': range(10, 21),
#                 'max_features': range(5, 105, 20)}
# locally_best_forest = GridSearchCV # you code here
# locally_best_forest.fit # you code here

# Make predictions for the test data and assess accuracy.
# you code here
<|endoftext|>
[{"Id": 14064907, "ScriptId": 1215902, "ParentScriptVersionId": NaN, "ScriptLanguageId": 9, "AuthorUserId": 178864, "CreationDate": "05/12/2019 07:28:41", "VersionNumber": 7.0, "Title": "A3 (demo). Decision trees", "EvaluationDate": "05/12/2019", "IsChange": true, "TotalLines": 359.0, "LinesInsertedFromPrevious": 14.0, "LinesChangedFromPrevious": 0.0, "LinesUnchangedFromPrevious": 345.0, "LinesInsertedFromFork": NaN, "LinesDeletedFromFork": NaN, "LinesChangedFromFork": NaN, "LinesUnchangedFromFork": NaN, "TotalVotes": 2}]
[{"Id": 9170963, "KernelVersionId": 14064907, "SourceDatasetVersionId": 117853}]
[{"Id": 117853, "DatasetId": 32132, "DatasourceVersionId": 128405, "CreatorUserId": 178864, "LicenseName": "CC BY-NC-SA 4.0", "CreationDate": "10/07/2018 17:25:09", "VersionNumber": 16.0, "Title": "mlcourse.ai", "Slug": "mlcourse", "Subtitle": "Open Machine Learning Course by OpenDataScience", "Description": "Open Machine Learning Course [mlcourse.ai](http://mlcourse.ai/) is designed to perfectly balance theory and practice; therefore, each topic is followed by an assignment with a deadline in a week. You can also take part in several Kaggle Inclass competitions held during the course and write your own tutorials. The next session starts on **October 1, 2018**. Fill in [this form](https://docs.google.com/forms/d/1_pDNuVHwBxV5wuOcdaXoxBZneyAQcqfOl4V2qkqKbNQ/) to participate. More info in [GitHub repo](https://github.com/Yorko/mlcourse.ai). \n\n### Outline\nThis is the list of published articles on [medium.com](https://medium.com/open-machine-learning-course) (English), [habr.com](https://habr.com/company/ods/blog/344044/) (Russian), and [jqr.com](https://www.jqr.com/) (Chinese). See Kernels of this Dataset for the same material in English. \n\n1. Exploratory Data Analysis with Pandas [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-1-exploratory-data-analysis-with-pandas-de57880f1a68)  [ru](https://habrahabr.ru/company/ods/blog/322626/), [cn](https://www.jqr.com/article/000079), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-1-exploratory-data-analysis-with-pandas)\n2. Visual Data Analysis with Python [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-2-visual-data-analysis-in-python-846b989675cd)  [ru](https://habrahabr.ru/company/ods/blog/323210/), [cn](https://www.jqr.com/article/000086), Kaggle Kernels: [part1](https://www.kaggle.com/kashnitsky/topic-2-visual-data-analysis-in-python), [part2](https://www.kaggle.com/kashnitsky/topic-2-part-2-seaborn-and-plotly)\n3. Classification, Decision Trees and k Nearest Neighbors [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-3-classification-decision-trees-and-k-nearest-neighbors-8613c6b6d2cd), [ru](https://habrahabr.ru/company/ods/blog/322534/), [cn](https://www.jqr.com/article/000139), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-3-decision-trees-and-knn)\n4. Linear Classification and Regression [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-4-linear-classification-and-regression-44a41b9b5220), [ru](https://habrahabr.ru/company/ods/blog/323890/), [cn](https://www.jqr.com/article/000175), Kaggle Kernels: [part1](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-1-ols), [part2](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-2-classification), [part3](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-3-regularization), [part4](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-4-more-of-logit), [part5](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-5-validation)\n5. Bagging and Random Forest [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-5-ensembles-of-algorithms-and-random-forest-8e05246cbba7), [ru](https://habrahabr.ru/company/ods/blog/324402/), [cn](https://www.jqr.com/article/000241), Kaggle Kernels: [part1](https://www.kaggle.com/kashnitsky/topic-5-ensembles-part-1-bagging), [part2](https://www.kaggle.com/kashnitsky/topic-5-ensembles-part-2-random-forest), [part3](https://www.kaggle.com/kashnitsky/topic-5-ensembles-part-3-feature-importance)\n6. Feature Engineering and Feature Selection [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-6-feature-engineering-and-feature-selection-8b94f870706a), [ru](https://habrahabr.ru/company/ods/blog/325422/), [cn](https://www.jqr.com/article/000249), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-6-feature-engineering-and-feature-selection)\n7. Unsupervised Learning: Principal Component Analysis and Clustering [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-7-unsupervised-learning-pca-and-clustering-db7879568417), [ru](https://habrahabr.ru/company/ods/blog/325654/), [cn](https://www.jqr.com/article/000336), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-7-unsupervised-learning-pca-and-clustering)\n8. Vowpal Wabbit: Learning with Gigabytes of Data [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-8-vowpal-wabbit-fast-learning-with-gigabytes-of-data-60f750086237), [ru](https://habrahabr.ru/company/ods/blog/326418/), [cn](https://www.jqr.com/article/000348), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-8-online-learning-and-vowpal-wabbit)\n9. Time Series Analysis with Python, part 1 [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-9-time-series-analysis-in-python-a270cb05e0b3), [ru](https://habrahabr.ru/company/ods/blog/327242/), [cn](https://www.jqr.com/article/000450). Predicting future with Facebook Prophet, part 2 [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-9-part-3-predicting-the-future-with-facebook-prophet-3f3af145cdc), [cn](https://www.jqr.com/article/000598) Kaggle Kernels: [part1](https://www.kaggle.com/kashnitsky/topic-9-part-1-time-series-analysis-in-python), [part2](https://www.kaggle.com/kashnitsky/topic-9-part-2-time-series-with-facebook-prophet)\n10. Gradient Boosting [uk](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-10-gradient-boosting-c751538131ac), [ru](https://habrahabr.ru/company/ods/blog/327250/), [cn](https://www.jqr.com/article/000573), [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-10-gradient-boosting)\n\n### Assignments\nEach topic is followed by an assignment. See demo versions in this Dataset. Solutions will be discussed in the upcoming run of the course.  \n\n### Kaggle competitions\n1. Catch Me If You Can: Intruder Detection through Webpage Session Tracking. [Kaggle Inclass](https://www.kaggle.com/c/catch-me-if-you-can-intruder-detection-through-webpage-session-tracking2)\n2. How good is your Medium article? [Kaggle Inclass](https://www.kaggle.com/c/how-good-is-your-medium-article/)\n\n### Rating\nThroughout the course we are maintaining a student rating. It takes into account credits scored in assignments and Kaggle competitions. Top students (according to the final rating) will be listed on a special Wiki page.\n\n### Community\nDiscussions between students are held in the **#mlcourse_ai** channel of the OpenDataScience Slack team. Fill in [this form](https://drive.google.com/open?id=1_pDNuVHwBxV5wuOcdaXoxBZneyAQcqfOl4V2qkqKbNQ) to get an invitation (to be sent in September 2018). The form will also ask you some personal questions, don't hesitate\n\n### Collaboration\nYou can publish Kernels using this Dataset. But please respect others' interests: don't share solutions to assignments and well-performing solutions for Kaggle Inclass competitions. If you notice any typos/errors in course material, please open an [Issue](https://github.com/Yorko/mlcourse.ai/issues) or make a pull request in course [repo](https://github.com/Yorko/mlcourse.ai).", "VersionNotes": "add beauty.csv", "TotalCompressedBytes": 211086036.0, "TotalUncompressedBytes": 53878942.0}]
[{"Id": 32132, "CreatorUserId": 178864, "OwnerUserId": 178864.0, "OwnerOrganizationId": NaN, "CurrentDatasetVersionId": 205932.0, "CurrentDatasourceVersionId": 217276.0, "ForumId": 40462, "Type": 2, "CreationDate": "06/18/2018 06:56:16", "LastActivityDate": "06/18/2018", "TotalViews": 267148, "TotalDownloads": 39410, "TotalVotes": 1585, "TotalKernels": 523}]
#
#
# ## [mlcourse.ai](https://mlcourse.ai) - Open Machine Learning Course
# Authors: [Maria Sumarokova](https://www.linkedin.com/in/mariya-sumarokova-230b4054/), and [Yury Kashnitsky](https://www.linkedin.com/in/festline/). Translated and edited by Gleb Filatov, Aleksey Kiselev, [Anastasia Manokhina](https://www.linkedin.com/in/anastasiamanokhina/), [Egor Polusmak](https://www.linkedin.com/in/egor-polusmak/), and [Yuanyuan Pao](https://www.linkedin.com/in/yuanyuanpao/). All content is distributed under the [Creative Commons CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license.
# #  Assignment #3 (demo)
# ##   Decision trees with a toy task and the UCI Adult dataset
# Same assignment as a [Kaggle Kernel](https://www.kaggle.com/kashnitsky/a3-demo-decision-trees) + [solution](https://www.kaggle.com/kashnitsky/a3-demo-decision-trees-solution). Fill in the answers in the [web-form](https://docs.google.com/forms/d/1wfWYYoqXTkZNOPy1wpewACXaj2MZjBdLOL58htGWYBA/edit).
# Let's start by loading all necessary libraries:
from matplotlib import pyplot as plt

plt.rcParams["figure.figsize"] = (10, 8)
import seaborn as sns
import numpy as np
import pandas as pd
from sklearn.preprocessing import LabelEncoder
import collections
from sklearn.model_selection import GridSearchCV
from sklearn import preprocessing
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from ipywidgets import Image
from io import StringIO
import pydotplus  # pip install pydotplus


# ### Part 1. Toy dataset "Will They? Won't They?"
# Your goal is to figure out how decision trees work by walking through a toy problem. While a single decision tree does not yield outstanding results, other performant algorithms like gradient boosting and random forests are based on the same idea. That is why knowing how decision trees work might be useful.
# We'll go through a toy example of binary classification - Person A is deciding whether they will go on a second date with Person B. It will depend on their looks, eloquence, alcohol consumption (only for example), and how much money was spent on the first date.
# #### Creating the dataset
# Create dataframe with dummy variables
def create_df(dic, feature_list):
    out = pd.DataFrame(dic)
    out = pd.concat([out, pd.get_dummies(out[feature_list])], axis=1)
    out.drop(feature_list, axis=1, inplace=True)
    return out


# Some feature values are present in train and absent in test and vice-versa.
def intersect_features(train, test):
    common_feat = list(set(train.keys()) & set(test.keys()))
    return train[common_feat], test[common_feat]


features = ["Looks", "Alcoholic_beverage", "Eloquence", "Money_spent"]


# #### Training data
df_train = {}
df_train["Looks"] = [
    "handsome",
    "handsome",
    "handsome",
    "repulsive",
    "repulsive",
    "repulsive",
    "handsome",
]
df_train["Alcoholic_beverage"] = ["yes", "yes", "no", "no", "yes", "yes", "yes"]
df_train["Eloquence"] = ["high", "low", "average", "average", "low", "high", "average"]
df_train["Money_spent"] = ["lots", "little", "lots", "little", "lots", "lots", "lots"]
df_train["Will_go"] = LabelEncoder().fit_transform(["+", "-", "+", "-", "-", "+", "+"])
df_train = create_df(df_train, features)
df_train

# #### Test data
df_test = {}
df_test["Looks"] = ["handsome", "handsome", "repulsive"]
df_test["Alcoholic_beverage"] = ["no", "yes", "yes"]
df_test["Eloquence"] = ["average", "high", "average"]
df_test["Money_spent"] = ["lots", "little", "lots"]
df_test = create_df(df_test, features)
df_test
# Some feature values are present in train and absent in test and vice-versa.
y = df_train["Will_go"]
df_train, df_test = intersect_features(train=df_train, test=df_test)
df_train
df_test

# #### Draw a decision tree (by hand or in any graphics editor) for this dataset. Optionally you can also implement tree construction and draw it here.
# 1\. What is the entropy $S_0$ of the initial system? By system states, we mean values of the binary feature "Will_go" - 0 or 1 - two states in total.
# you code here

# 2\. Let's split the data by the feature "Looks_handsome". What is the entropy $S_1$ of the left group - the one with "Looks_handsome". What is the entropy $S_2$ in the opposite group? What is the information gain (IG) if we consider such a split?
# you code here

# #### Train a decision tree using sklearn on the training data. You may choose any depth for the tree.
# you code here

# #### Additional: display the resulting tree using graphviz. You can use pydot or [web-service](https://www.coolutils.com/ru/online/DOT-to-PNG) dot2png.
# you code here

# ### Part 2. Functions for calculating entropy and information gain.
# Consider the following warm-up example: we have 9 blue balls and 11 yellow balls. Let ball have label **1** if it is blue, **0** otherwise.
balls = [1 for i in range(9)] + [0 for i in range(11)]

#
# Next split the balls into two groups:
#
# two groups
balls_left = [1 for i in range(8)] + [0 for i in range(5)]  # 8 blue and 5 yellow
balls_right = [1 for i in range(1)] + [0 for i in range(6)]  # 1 blue and 6 yellow


# #### Implement a function to calculate the Shannon Entropy
def entropy(a_list):
    # you code here
    pass


# Tests
print(entropy(balls))  # 9 blue и 11 yellow
print(entropy(balls_left))  # 8 blue и 5 yellow
print(entropy(balls_right))  # 1 blue и 6 yellow
print(entropy([1, 2, 3, 4, 5, 6]))  # entropy of a fair 6-sided die


# 3\. What is the entropy of the state given by the list **balls_left**?
# 4\. What is the entropy of a fair dice? (where we look at a dice as a system with 6 equally probable states)?
# information gain calculation
def information_gain(root, left, right):
    """root - initial data, left and right - two partitions of initial data"""
    # you code here
    pass


# 5\. What is the information gain from splitting the initial dataset into **balls_left** and **balls_right** ?
def best_feature_to_split(X, y):
    """Outputs information gain when splitting on best feature"""
    # you code here
    pass


# #### Optional:
# - Implement a decision tree building algorithm by calling **best_feature_to_split** recursively
# - Plot the resulting tree
# ### Part 3. The "Adult" dataset
# #### Dataset description:
# [Dataset](http://archive.ics.uci.edu/ml/machine-learning-databases/adult) UCI Adult (no need to download it, we have a copy in the course repository): classify people using demographical data - whether they earn more than \$50,000 per year or not.
# Feature descriptions:
# - **Age** – continuous feature
# - **Workclass** –  continuous feature
# - **fnlwgt** – final weight of object, continuous feature
# - **Education** –  categorical feature
# - **Education_Num** – number of years of education, continuous feature
# - **Martial_Status** –  categorical feature
# - **Occupation** –  categorical feature
# - **Relationship** – categorical feature
# - **Race** – categorical feature
# - **Sex** – categorical feature
# - **Capital_Gain** – continuous feature
# - **Capital_Loss** – continuous feature
# - **Hours_per_week** – continuous feature
# - **Country** – categorical feature
# **Target** – earnings level, categorical (binary) feature.
# #### Reading train and test data
data_train = pd.read_csv("../../data/adult_train.csv", sep=";")
data_train.tail()
data_test = pd.read_csv("../../data/adult_test.csv", sep=";")
data_test.tail()
# necessary to remove rows with incorrect labels in test dataset
data_test = data_test[
    (data_test["Target"] == " >50K.") | (data_test["Target"] == " <=50K.")
]
# encode target variable as integer
data_train.loc[data_train["Target"] == " <=50K", "Target"] = 0
data_train.loc[data_train["Target"] == " >50K", "Target"] = 1
data_test.loc[data_test["Target"] == " <=50K.", "Target"] = 0
data_test.loc[data_test["Target"] == " >50K.", "Target"] = 1

# #### Primary data analysis
data_test.describe(include="all").T
data_train["Target"].value_counts()
fig = plt.figure(figsize=(25, 15))
cols = 5
rows = np.ceil(float(data_train.shape[1]) / cols)
for i, column in enumerate(data_train.columns):
    ax = fig.add_subplot(rows, cols, i + 1)
    ax.set_title(column)
    if data_train.dtypes[column] == np.object:
        data_train[column].value_counts().plot(kind="bar", axes=ax)
    else:
        data_train[column].hist(axes=ax)
        plt.xticks(rotation="vertical")
plt.subplots_adjust(hspace=0.7, wspace=0.2)

# #### Checking data types
data_train.dtypes
data_test.dtypes

# As we see, in the test data, age is treated as type **object**. We need to fix this.
data_test["Age"] = data_test["Age"].astype(int)

# Also we'll cast all **float** features to **int** type to keep types consistent between our train and test data.
data_test["fnlwgt"] = data_test["fnlwgt"].astype(int)
data_test["Education_Num"] = data_test["Education_Num"].astype(int)
data_test["Capital_Gain"] = data_test["Capital_Gain"].astype(int)
data_test["Capital_Loss"] = data_test["Capital_Loss"].astype(int)
data_test["Hours_per_week"] = data_test["Hours_per_week"].astype(int)

# #### Fill in missing data for continuous features with their median values, for categorical features with their mode.
# choose categorical and continuous features from data
categorical_columns = [
    c for c in data_train.columns if data_train[c].dtype.name == "object"
]
numerical_columns = [
    c for c in data_train.columns if data_train[c].dtype.name != "object"
]
print("categorical_columns:", categorical_columns)
print("numerical_columns:", numerical_columns)
# we see some missing values
data_train.info()
# fill missing data
for c in categorical_columns:
    data_train[c].fillna(data_train[c].mode()[0], inplace=True)
    data_test[c].fillna(data_train[c].mode()[0], inplace=True)
for c in numerical_columns:
    data_train[c].fillna(data_train[c].median(), inplace=True)
    data_test[c].fillna(data_train[c].median(), inplace=True)
# no more missing values
data_train.info()

# We'll dummy code some categorical features: **Workclass**, **Education**, **Martial_Status**, **Occupation**, **Relationship**, **Race**, **Sex**, **Country**. It can be done via pandas method **get_dummies**
data_train = pd.concat(
    [data_train[numerical_columns], pd.get_dummies(data_train[categorical_columns])],
    axis=1,
)
data_test = pd.concat(
    [data_test[numerical_columns], pd.get_dummies(data_test[categorical_columns])],
    axis=1,
)
set(data_train.columns) - set(data_test.columns)
data_train.shape, data_test.shape

# #### There is no Holland in the test data. Create new zero-valued feature.
data_test["Country_ Holand-Netherlands"] = 0
set(data_train.columns) - set(data_test.columns)
data_train.head(2)
data_test.head(2)
X_train = data_train.drop(["Target"], axis=1)
y_train = data_train["Target"]
X_test = data_test.drop(["Target"], axis=1)
y_test = data_test["Target"]

# ### 3.1 Decision tree without parameter tuning
# Train a decision tree **(DecisionTreeClassifier)** with a maximum depth of 3, and evaluate the accuracy metric on the test data. Use parameter **random_state = 17** for results reproducibility.
# you code here
# tree =
# tree.fit

# Make a prediction with the trained model on the test data.
# you code here
# tree_predictions = tree.predict
# you code here
# accuracy_score

# 6\. What is the test set accuracy of a decision tree with maximum tree depth of 3 and **random_state = 17**?
# ### 3.2 Decision tree with parameter tuning
# Train a decision tree **(DecisionTreeClassifier, random_state = 17).** Find the optimal maximum depth using 5-fold cross-validation **(GridSearchCV)**.
tree_params = {"max_depth": range(2, 11)}
locally_best_tree = GridSearchCV  # you code here
locally_best_tree.fit
# you code here

# Train a decision tree with maximum depth of 9 (it is the best **max_depth** in my case), and compute the test set accuracy. Use parameter **random_state = 17** for reproducibility.
# you code here
# tuned_tree =
# tuned_tree.fit
# tuned_tree_predictions = tuned_tree.predict
# accuracy_score

# 7\. What is the test set accuracy of a decision tree with maximum depth of 9 and **random_state = 17**?
# ### 3.3 (Optional) Random forest without parameter tuning
# Let's take a sneak peek of upcoming lectures and try to use a random forest for our task. For now, you can imagine a random forest as a bunch of decision trees, trained on slightly different subsets of the training data.
# Train a random forest **(RandomForestClassifier)**. Set the number of trees to 100 and use **random_state = 17**.
# you code here
# rf =
# rf.fit # you code here

# Make predictions for the test data and assess accuracy.
# you code here

# ### 3.4 (Optional) Random forest with parameter tuning
# Train a random forest **(RandomForestClassifier)**. Tune the maximum depth and maximum number of features for each tree using **GridSearchCV**.
# forest_params = {'max_depth': range(10, 21),
#                 'max_features': range(5, 105, 20)}
# locally_best_forest = GridSearchCV # you code here
# locally_best_forest.fit # you code here

# Make predictions for the test data and assess accuracy.
# you code here
