<kaggle_start><code># # **Titanic survival prediction with Decision Tree**
# Hello there, this notebook will go through the process of my data preprocessing approach and building a decision tree using sklearn. What you can expect from this notebook:
# 1. Feature enginneering with pipeline
# 2. Building a simple decision tree using sklearn
# 3. Improving our model with cost-complexity pruning using sklearn
# I will be importing libraries from different sections so you can understand what each libraries are correspond to, feel free to leave any kind of comments if you see any problems/improvements that I can make, it will be very much appreciated!
import pandas as pd
import numpy as np

train_df = pd.read_csv("/kaggle/input/titanic/train.csv")
test_df = pd.read_csv("/kaggle/input/titanic/test.csv")
train_df.head()

# I have already done my EDA on a seperate notebook,
# https://www.kaggle.com/code/crxxom/titanic-survival-rate-interactive-dashboard-eda which is an interactive tableau dashboard you can interact with the find valuable insight, feel free to go check it out!
# # **Data Preprocessing and feature selection**
# I am going to build a simple pipeline to preprocess the dataframe. What you could expect in the following section:
# 1. Imputing Age, Embarked and Fare
# 2. 'Companion' feature based on the 'SibSp' and 'Parch' Feature
# 3. 'Title' feature based on the 'Name' feature
# 4. Binarizing Sex, Companion and Title
# 5. One hot encoding of Embarked feature
# 6. Dropping features
# libraries we will be using in this section
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
import math


# The general structure of building a pipeline
class Name_of_your_estimator(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        # Your code
        return  # what you are going to return


# ### 1. Impute (NaN handling)
# Here we are imputing age using the mean, imputing embarked using mode, imputing fare using mean
class Imputer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        # Impute Age
        imputer = SimpleImputer(strategy="mean")
        X["Age"] = imputer.fit_transform(X["Age"].values.reshape(-1, 1))[:, 0]
        # Impute Embarked
        imputer = SimpleImputer(strategy="most_frequent")
        X["Embarked"] = imputer.fit_transform(X["Embarked"].values.reshape(-1, 1))[:, 0]
        # Impute Fare
        imputer = SimpleImputer(strategy="mean")
        X["Fare"] = imputer.fit_transform(X["Fare"].values.reshape(-1, 1))[:, 0]
        return X


# ### 2. 'Companion' feature
# Basically we are creating a companion feature to classify between people travelling alone and people travelling with a companion.
class Companion(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        X["companion"] = X["SibSp"] + X["Parch"]
        return X.drop(["SibSp", "Parch"], axis=1)


# ### 3. 'Title' feature
# By extracting the title from the Name column, we classify people with special titles (eg. Master)
class TitleFeature(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        lr = ["Mr", "Mrs", "Miss", "Ms", "Mlle"]  # Ms and Mlle both means Miss

        # Basically we classify people with special titles and people that do not
        def rank_title(x):
            if x in lr:
                return "LR"
            else:
                return "HR"

        X["title"] = (
            X["Name"]
            .str.split(",", expand=True)[1]
            .str.split(".", expand=True)[0]
            .str.strip()
        )
        X["title"] = X["title"].apply(rank_title)
        return X


# ### 4. Binarize
# Since most ML model do not read string values, we need to binarize the values with 0 and 1.
class FeatureBinarize(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        # Gender binarize
        gender_dct = {"male": 0, "female": 1}
        X["Sex"] = [gender_dct[g] for g in X["Sex"]]
        # Title binarize - Classify low rank and high rank
        title_dct = {"LR": 0, "HR": 1}
        X["title"] = [title_dct[t] for t in X["title"]]

        # Companion binarize
        def is_alone(x):
            if x > 0:
                return 1
            else:
                return 0

        X["companion"] = X["companion"].apply(is_alone)
        return X


# ### 5. One hot encoding
# 'One-hot encoding in machine learning is the **conversion of categorical information** into a format that may be fed into machine learning algorithms to improve prediction accuracy. One-hot encoding is a common method for dealing with categorical data in machine learning.'
# From my understanding, basically why we don't simply plug in 1,2,3,4 for 4 different categories to label them (just an example) is because the 'distance' between 1 and 2 is smaller between 1 and 4, and it will be more likely that the model will cluster category 1 with category 2, but it doesn't make sense in this scenario, so we use one hot encoding to deal with categorical information in this case. Note, there are a lot of different methods other than one hot encoding, also please let me know if my understanding on this topic is wrong, thanks!
class FeatureEncoder(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        encoder = OneHotEncoder()
        matrix = encoder.fit_transform(X["Embarked"].values.reshape(-1, 1)).toarray()
        column_name = ["C", "Q", "S"]
        for i in range(len(matrix.T)):
            X[column_name[i]] = matrix.T[i]
        return X.drop(["Embarked"], axis=1)


# ### 6. Dropping features
# We will be dropping Name, PassengerId, Ticket and Cabin from our dataset since they are either irrelavent or too many NaN data (Cabin).
class FeatureDropper(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        return X.drop(["Name", "PassengerId", "Ticket", "Cabin"], axis=1)


# ### Extra: Unit adjusting
class UnitAdjust(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        X["Age"] = X["Age"].apply(lambda y: math.floor(y))
        X["Fare"] = X["Fare"].apply(lambda f: round(f, 1))
        return X


# ### Run the dataset through the pipeline
pipe = Pipeline(
    [
        ("imputer", Imputer()),
        ("title", TitleFeature()),
        ("companion", Companion()),
        ("binarize", FeatureBinarize()),
        ("encoder", FeatureEncoder()),
        ("dropper", FeatureDropper()),
        ("unit", UnitAdjust()),
    ]
)
train_ab = pipe.fit_transform(train_df.copy())
train_ab.head()

# And we scale it using StandardScaler. (Feel free to let me know if there are better ways to do it)
from sklearn.preprocessing import StandardScaler

X = train_ab.drop(["Survived"], axis=1)
y = train_ab["Survived"]
scaler = StandardScaler()
train = scaler.fit_transform(X)
y = y.to_numpy()

# # **Decision Tree Model Building**
# Libraries we will be using
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import plot_tree
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import ConfusionMatrixDisplay

# ### 1. Building the preliminary classification tree
X_train, X_test, y_train, y_test = train_test_split(train, y, random_state=42)
clf_dt = DecisionTreeClassifier(random_state=42)
clf_dt = clf_dt.fit(X_train, y_train)

# I found an interesting article of why we set random_state to 42, https://stackoverflow.com/questions/49147774/what-is-random-state-in-sklearn-model-selection-train-test-split-example.
# We can visualize our decision tree with the plot_tree library we just imported.
plt.figure(figsize=(15, 7.5))
plot_tree(
    clf_dt,
    filled=True,
    rounded=True,
    class_names=["Dead", "Survived"],
    feature_names=X.columns,
)

# Damn that's a pretty huge tree, we can also see the performance of the tree with a confusion matrix.
ConfusionMatrixDisplay.from_estimator(
    clf_dt, X_test, y_test, display_labels=["Dead", "Survived"]
)

# We can calculate that the sensitivity of the model is (108/(108+24)) = 81.8%, while the specificity of the model is (65/(65+26)) = 71.4%, the model is relatively better at correctly predicting people that are dead. (Intuitively it might be better to swap the dead and survived position in the matrix since we are trying to find the survival rate, but it's similar)
# ### 2. Cost-complexity pruning with cross validation
# We will apply cost complexity pruning to reduce the overfitting which will occur frequently when it comes to decision tree. If you don't know what pruning is, I highly recommend you to watch this video https://www.youtube.com/watch?v=D0efHEJsfHo&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&index=50 which discuss the use of pruning in decision tree. In fact, a lot of the code that I used building this model is from another video from this series!
path = clf_dt.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas = path.ccp_alphas
ccp_alphas = ccp_alphas[:-1]  # remove the maximum alpha value from the dataset
alpha_loop_value = []
for ccp_alpha in ccp_alphas:
    clf_dt = DecisionTreeClassifier(random_state=0, ccp_alpha=ccp_alpha)
    scores = cross_val_score(clf_dt, X_train, y_train, cv=5)
    alpha_loop_value.append([ccp_alpha, np.mean(scores), np.std(scores)])
alpha_results = pd.DataFrame(
    alpha_loop_value, columns=["alpha", "mean_accuracy", "std"]
)
alpha_results.sort_values(by="mean_accuracy", ascending=False)
alpha_results.plot(x="alpha", y="mean_accuracy", yerr="std", marker="o", linestyle="--")
optimal_ccp_alpha = alpha_results.iloc[63, 0]

# Now that we get the optimal alpha value for the model, we can build our modified decision tree!
clf_dt_pruned = DecisionTreeClassifier(random_state=42, ccp_alpha=optimal_ccp_alpha)
clf_dt_pruned = clf_dt_pruned.fit(X_train, y_train)
plt.figure(figsize=(15, 7.5))
plot_tree(
    clf_dt_pruned,
    filled=True,
    rounded=True,
    class_names=["Dead", "Survived"],
    feature_names=X.columns,
)
ConfusionMatrixDisplay.from_estimator(
    clf_dt_pruned, X_test, y_test, display_labels=["Dead", "Survived"]
)

# We notice that the sensitivity worsen a bit while the specificity have improved.
# # **Submission**
# We first run our test set through our pipeline.
pipe = Pipeline(
    [
        ("imputer", Imputer()),
        ("title", TitleFeature()),
        ("companion", Companion()),
        ("binarize", FeatureBinarize()),
        ("encoder", FeatureEncoder()),
        ("dropper", FeatureDropper()),
        ("unit", UnitAdjust()),
    ]
)
test_ab = pipe.fit_transform(test_df.copy())
test_ab.head()
scaler = StandardScaler()
X_test = scaler.fit_transform(test_ab)

# Then we run our filtered dataset to our pruned decision tree.
Y_pred = clf_dt_pruned.predict(X_test)
Y_pred

# Check if the predicted output fullfill the requirement.
Y_pred.shape

# And now we only need to put our results in a dataframe for submission!
submission = pd.DataFrame(
    {"PassengerId": test_df.PassengerId.values, "Survived": Y_pred}
)
submission.to_csv("Titanic_DT.csv", index=False)
<|endoftext|>
[{"Id": 132007580, "ScriptId": 39373078, "ParentScriptVersionId": NaN, "ScriptLanguageId": 9, "AuthorUserId": 12917908, "CreationDate": "06/02/2023 16:54:28", "VersionNumber": 3.0, "Title": "Prediction with Decision Tree", "EvaluationDate": "06/02/2023", "IsChange": true, "TotalLines": 304.0, "LinesInsertedFromPrevious": 1.0, "LinesChangedFromPrevious": 0.0, "LinesUnchangedFromPrevious": 303.0, "LinesInsertedFromFork": NaN, "LinesDeletedFromFork": NaN, "LinesChangedFromFork": NaN, "LinesUnchangedFromFork": NaN, "TotalVotes": 2}]
# # **Titanic survival prediction with Decision Tree**
# Hello there, this notebook will go through the process of my data preprocessing approach and building a decision tree using sklearn. What you can expect from this notebook:
# 1. Feature enginneering with pipeline
# 2. Building a simple decision tree using sklearn
# 3. Improving our model with cost-complexity pruning using sklearn
# I will be importing libraries from different sections so you can understand what each libraries are correspond to, feel free to leave any kind of comments if you see any problems/improvements that I can make, it will be very much appreciated!
import pandas as pd
import numpy as np

train_df = pd.read_csv("/kaggle/input/titanic/train.csv")
test_df = pd.read_csv("/kaggle/input/titanic/test.csv")
train_df.head()

# I have already done my EDA on a seperate notebook,
# https://www.kaggle.com/code/crxxom/titanic-survival-rate-interactive-dashboard-eda which is an interactive tableau dashboard you can interact with the find valuable insight, feel free to go check it out!
# # **Data Preprocessing and feature selection**
# I am going to build a simple pipeline to preprocess the dataframe. What you could expect in the following section:
# 1. Imputing Age, Embarked and Fare
# 2. 'Companion' feature based on the 'SibSp' and 'Parch' Feature
# 3. 'Title' feature based on the 'Name' feature
# 4. Binarizing Sex, Companion and Title
# 5. One hot encoding of Embarked feature
# 6. Dropping features
# libraries we will be using in this section
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
import math


# The general structure of building a pipeline
class Name_of_your_estimator(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        # Your code
        return  # what you are going to return


# ### 1. Impute (NaN handling)
# Here we are imputing age using the mean, imputing embarked using mode, imputing fare using mean
class Imputer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        # Impute Age
        imputer = SimpleImputer(strategy="mean")
        X["Age"] = imputer.fit_transform(X["Age"].values.reshape(-1, 1))[:, 0]
        # Impute Embarked
        imputer = SimpleImputer(strategy="most_frequent")
        X["Embarked"] = imputer.fit_transform(X["Embarked"].values.reshape(-1, 1))[:, 0]
        # Impute Fare
        imputer = SimpleImputer(strategy="mean")
        X["Fare"] = imputer.fit_transform(X["Fare"].values.reshape(-1, 1))[:, 0]
        return X


# ### 2. 'Companion' feature
# Basically we are creating a companion feature to classify between people travelling alone and people travelling with a companion.
class Companion(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        X["companion"] = X["SibSp"] + X["Parch"]
        return X.drop(["SibSp", "Parch"], axis=1)


# ### 3. 'Title' feature
# By extracting the title from the Name column, we classify people with special titles (eg. Master)
class TitleFeature(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        lr = ["Mr", "Mrs", "Miss", "Ms", "Mlle"]  # Ms and Mlle both means Miss

        # Basically we classify people with special titles and people that do not
        def rank_title(x):
            if x in lr:
                return "LR"
            else:
                return "HR"

        X["title"] = (
            X["Name"]
            .str.split(",", expand=True)[1]
            .str.split(".", expand=True)[0]
            .str.strip()
        )
        X["title"] = X["title"].apply(rank_title)
        return X


# ### 4. Binarize
# Since most ML model do not read string values, we need to binarize the values with 0 and 1.
class FeatureBinarize(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        # Gender binarize
        gender_dct = {"male": 0, "female": 1}
        X["Sex"] = [gender_dct[g] for g in X["Sex"]]
        # Title binarize - Classify low rank and high rank
        title_dct = {"LR": 0, "HR": 1}
        X["title"] = [title_dct[t] for t in X["title"]]

        # Companion binarize
        def is_alone(x):
            if x > 0:
                return 1
            else:
                return 0

        X["companion"] = X["companion"].apply(is_alone)
        return X


# ### 5. One hot encoding
# 'One-hot encoding in machine learning is the **conversion of categorical information** into a format that may be fed into machine learning algorithms to improve prediction accuracy. One-hot encoding is a common method for dealing with categorical data in machine learning.'
# From my understanding, basically why we don't simply plug in 1,2,3,4 for 4 different categories to label them (just an example) is because the 'distance' between 1 and 2 is smaller between 1 and 4, and it will be more likely that the model will cluster category 1 with category 2, but it doesn't make sense in this scenario, so we use one hot encoding to deal with categorical information in this case. Note, there are a lot of different methods other than one hot encoding, also please let me know if my understanding on this topic is wrong, thanks!
class FeatureEncoder(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        encoder = OneHotEncoder()
        matrix = encoder.fit_transform(X["Embarked"].values.reshape(-1, 1)).toarray()
        column_name = ["C", "Q", "S"]
        for i in range(len(matrix.T)):
            X[column_name[i]] = matrix.T[i]
        return X.drop(["Embarked"], axis=1)


# ### 6. Dropping features
# We will be dropping Name, PassengerId, Ticket and Cabin from our dataset since they are either irrelavent or too many NaN data (Cabin).
class FeatureDropper(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        return X.drop(["Name", "PassengerId", "Ticket", "Cabin"], axis=1)


# ### Extra: Unit adjusting
class UnitAdjust(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        X["Age"] = X["Age"].apply(lambda y: math.floor(y))
        X["Fare"] = X["Fare"].apply(lambda f: round(f, 1))
        return X


# ### Run the dataset through the pipeline
pipe = Pipeline(
    [
        ("imputer", Imputer()),
        ("title", TitleFeature()),
        ("companion", Companion()),
        ("binarize", FeatureBinarize()),
        ("encoder", FeatureEncoder()),
        ("dropper", FeatureDropper()),
        ("unit", UnitAdjust()),
    ]
)
train_ab = pipe.fit_transform(train_df.copy())
train_ab.head()

# And we scale it using StandardScaler. (Feel free to let me know if there are better ways to do it)
from sklearn.preprocessing import StandardScaler

X = train_ab.drop(["Survived"], axis=1)
y = train_ab["Survived"]
scaler = StandardScaler()
train = scaler.fit_transform(X)
y = y.to_numpy()

# # **Decision Tree Model Building**
# Libraries we will be using
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import plot_tree
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import ConfusionMatrixDisplay

# ### 1. Building the preliminary classification tree
X_train, X_test, y_train, y_test = train_test_split(train, y, random_state=42)
clf_dt = DecisionTreeClassifier(random_state=42)
clf_dt = clf_dt.fit(X_train, y_train)

# I found an interesting article of why we set random_state to 42, https://stackoverflow.com/questions/49147774/what-is-random-state-in-sklearn-model-selection-train-test-split-example.
# We can visualize our decision tree with the plot_tree library we just imported.
plt.figure(figsize=(15, 7.5))
plot_tree(
    clf_dt,
    filled=True,
    rounded=True,
    class_names=["Dead", "Survived"],
    feature_names=X.columns,
)

# Damn that's a pretty huge tree, we can also see the performance of the tree with a confusion matrix.
ConfusionMatrixDisplay.from_estimator(
    clf_dt, X_test, y_test, display_labels=["Dead", "Survived"]
)

# We can calculate that the sensitivity of the model is (108/(108+24)) = 81.8%, while the specificity of the model is (65/(65+26)) = 71.4%, the model is relatively better at correctly predicting people that are dead. (Intuitively it might be better to swap the dead and survived position in the matrix since we are trying to find the survival rate, but it's similar)
# ### 2. Cost-complexity pruning with cross validation
# We will apply cost complexity pruning to reduce the overfitting which will occur frequently when it comes to decision tree. If you don't know what pruning is, I highly recommend you to watch this video https://www.youtube.com/watch?v=D0efHEJsfHo&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&index=50 which discuss the use of pruning in decision tree. In fact, a lot of the code that I used building this model is from another video from this series!
path = clf_dt.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas = path.ccp_alphas
ccp_alphas = ccp_alphas[:-1]  # remove the maximum alpha value from the dataset
alpha_loop_value = []
for ccp_alpha in ccp_alphas:
    clf_dt = DecisionTreeClassifier(random_state=0, ccp_alpha=ccp_alpha)
    scores = cross_val_score(clf_dt, X_train, y_train, cv=5)
    alpha_loop_value.append([ccp_alpha, np.mean(scores), np.std(scores)])
alpha_results = pd.DataFrame(
    alpha_loop_value, columns=["alpha", "mean_accuracy", "std"]
)
alpha_results.sort_values(by="mean_accuracy", ascending=False)
alpha_results.plot(x="alpha", y="mean_accuracy", yerr="std", marker="o", linestyle="--")
optimal_ccp_alpha = alpha_results.iloc[63, 0]

# Now that we get the optimal alpha value for the model, we can build our modified decision tree!
clf_dt_pruned = DecisionTreeClassifier(random_state=42, ccp_alpha=optimal_ccp_alpha)
clf_dt_pruned = clf_dt_pruned.fit(X_train, y_train)
plt.figure(figsize=(15, 7.5))
plot_tree(
    clf_dt_pruned,
    filled=True,
    rounded=True,
    class_names=["Dead", "Survived"],
    feature_names=X.columns,
)
ConfusionMatrixDisplay.from_estimator(
    clf_dt_pruned, X_test, y_test, display_labels=["Dead", "Survived"]
)

# We notice that the sensitivity worsen a bit while the specificity have improved.
# # **Submission**
# We first run our test set through our pipeline.
pipe = Pipeline(
    [
        ("imputer", Imputer()),
        ("title", TitleFeature()),
        ("companion", Companion()),
        ("binarize", FeatureBinarize()),
        ("encoder", FeatureEncoder()),
        ("dropper", FeatureDropper()),
        ("unit", UnitAdjust()),
    ]
)
test_ab = pipe.fit_transform(test_df.copy())
test_ab.head()
scaler = StandardScaler()
X_test = scaler.fit_transform(test_ab)

# Then we run our filtered dataset to our pruned decision tree.
Y_pred = clf_dt_pruned.predict(X_test)
Y_pred

# Check if the predicted output fullfill the requirement.
Y_pred.shape

# And now we only need to put our results in a dataframe for submission!
submission = pd.DataFrame(
    {"PassengerId": test_df.PassengerId.values, "Survived": Y_pred}
)
submission.to_csv("Titanic_DT.csv", index=False)
