<kaggle_start><data_title>Grid Loss Prediction Dataset<data_description>### Context

A power grid transports the electricity from power producers to the consumers. But all that is produced is not delivered to the customers. Some parts of it are lost in either transmission or distribution. In Norway, the grid companies are responsible for reporting this grid loss to the institutes responsible for national transmission networks. They have to nominate the expected loss day ahead to the market so that the electricity price can be decided.

The physics of grid losses are well understood and can be calculated quite accurately given the grid configuration. Still, as these are not known or changes all the time, calculating grid losses is not straight forward. 


### Content

Grid loss is directly correlated with the total amount of power in the grid, which is also known as the grid load.

We provide data for three different grids from Norway that are owned by Tensio (Previously Trønderenergi Nett).

Features:
In this dataset, we provide the hourly values of all the features we found relevant for predicting the grid loss. 

For each of the grids, we have:
1. Grid loss: historical measurements of grid loss in MWh
2. Grid load: historical measurements of grid load in MWh
3. Temperature forecast in Kelvin
4. Predictions using the Prophet model in MWh
5. Trend, daily, weekly and yearly components of the grid loss, also from the Prophet Model.

Other than these grid specific features, we provide:
1. Calendar features: year, season, month, week, weekday, hour, in the cyclic form (see Notes 1.) and whether it is a holiday or not.
2. Incorrect data: whether the data was marked incorrect by the experts, in retrospective. We recommend removing this data before training your model.
3. Estimated demand in Trondheim: predicted demand for electricity in Trondheim, a big city in the middle of Norway, in MWh (see Note 2.)

We have split the dataset into two parts: training and testing set.

Training set:
This file (train.csv) contains two years of data (December 2017 to November 2019). All the features mentioned above are provided for this duration.

Test set:
This file (test.csv) contains six months of data (December 2019 to May 2020). All the features from training data are provided for the test set as well. Occasionally, some of the features could be missing.

Additionally, we provide a copy of test dataset (test\_backfilled\_missing\_features.csv) where the missing features are backfilled.


Note:
1. Calendar features are cyclic in nature. If we encode the weekdays (Monday to Sunday) as 0 to 6, we find that while Sunday and Monday are next to each other, the distance between their embeddings does not reflect it. To reflect this cyclic nature of the calendar features, we created cyclic calendar features based on cosine and sine which together place the highest and lowest value of the features close to each other in the feature space. 
2. We don't have an estimate of demand for all the grids. We used the demand predictions for Trondheim, the biggest city closest to the three grids.
3. The grid load is directly proportional to the grid loss. While we don't have predictions for grid load, but since we have historical measurements for them, it makes sense to predict it and use it as a feature for predicting the grid loss.
4. While the Prophet Model did not perform nicely as a prediction tool for our dataset, we found it useful to include its prediction and other components as features in our model.
5. Grid 3 has less training data than grid 1 and grid 2.
6. We published our solution. For more details, please refer to:

Dalal, N., Mølnå, M., Herrem, M., Røen, M., & Gundersen, O. E. (2020). Day-Ahead Forecasting of Losses in the Distribution Network. In AAAI (pp. 13148-13155).

Bibtex format for citation:

@incollection{dalal2020a,
  author = {Dalal, N. and Mølnå, M. and Herrem, M. and Røen, M. and Gundersen, O.E.},
  date = {2020},
  title = {Day-Ahead Forecasting of Losses in the Distribution Network},
  pages = {13148–13155},
  language = {en},
  booktitle = {AAAI}
}


### Challenges

Working with clean and processed data often hides the complexity of running the model in deployment. Some of the challenges we had while predicting grid loss in deployment are:

1. Day-ahead predictions: We need to predict the grid loss for the next day before 10 am the current day at an hourly resolution i.e on 10:00 May 26, 2020, we need to predict the grid loss on May 27, 2020, from midnight to 23:00 May 28, 2020, at an hourly resolution (24 values) for each grid.
2. Delayed measurements: We don't receive the measured values of load and loss immediately. We receive them 5 days after. Sometimes, there can be additional delays for a few more days. While grid loss and load are provided for the test data set as well, DO NOT USE them as features, unless they are 6 days old i.e while predicting grid loss for 27th January 2020, you can use the grid loss values will 20th January 2020. Using grid loss or grid load data after that date is unfair and will be discarded.
3. Missing data: Sometimes, we don't receive some of the features. For example, weather client might be out of service. You should make sure that your model should work even when some features are unavailable/missing.
4. Incorrect data: There have been times when the measurements we received were incorrect, by a big margin. They have been marked in the dataset in the incorrect_data column. It is recommended to remove those data points before you start analysing the data.
5. Less training data: For one of the grids, grid 3, we only have a few months of data.
6. Changes in the grids: Grid structures can keep changing. Sometimes new big consumers are added, or small grids can be merged into big ones.<data_name>grid-loss-time-series-dataset
<code># # Grid loss demonstration and Persistence model
# This notebook will give a demo for how to use the dataset. The pupose of this notebook is to provide the persistence model for predicting the grid loss.
# **Persistence model:**
# Persistence model is a well used baseline time series perdiction. It assumes that time series data does not change rapidly from day to day. Under this assumption, the persistence models 'predicts' that the values in the future will be a repitition of the last observed values. In this project, persistence model assumes that the grid loss for today will be same as the grid loss last week, the same day. Hence, in the model implementation, training is not needed. For prediction, it just returns the last week's measured grid loss values.
# As you will notice, Persistence model is not the best model. Feel free to explore other features and beat its performance with your models.
import numpy as np  # linear algebra
import pandas as pd  # data processing, CSV file I/O (e.g. pd.read_csv)
import datetime as dt  # date time
import matplotlib.pyplot as plt  # for plotting

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory
import os

for dirname, _, filenames in os.walk("/kaggle/input"):
    for filename in filenames:
        print(os.path.join(dirname, filename))
# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All"
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session

# For this demonstration, we used the test.csv file from the dataset. Persistence model was tested on this whole file in this demo, but feel free to play around the start date, duration and end date. We predicted grid loss for grid 1 here.
## Add dates and grid number you are interested in
start_date = dt.datetime(2019, 12, 1)
end_date = dt.datetime(2020, 5, 31)
grid_nr = 1
grid_col = f"grid{grid_nr}-loss"
# Load the test dataset. Training dataset is not needed here as Persistence model just returns the grid loss values from the week before, hence no training is needed.
test_data = pd.read_csv("../input/grid-loss-time-series-dataset/test.csv", index_col=0)
x_test = test_data[grid_col]
display(x_test.head())


# Persistence model was defined below. To use the model in the similar fashion as other sklearn models, we used the same prototype for our model.
# Defining the Persistence model, in line with other sklearn models
class PersistenceModel:
    def __init__(self):
        pass

    def train(self, x_train, y_train):
        # No training needed
        pass

    def predict(self, x_test):
        # returns the values shifted back by 7 days (i.e. 24*7 hourly values)
        return x_test.shift(24 * 7)


## Other helful functions
# Calculating the model performance. MAE, RMSE and MAPE are calculated.
def calculate_error(pred, target):
    target = target.loc[pred.index[0] : pred.index[-1]]
    metrics = {
        "mae": np.mean(np.abs(pred - target)),  # Mean absolute error
        "rmse": np.sqrt(np.mean((pred - target) ** 2)),  # Root mean squared error
        "mape": 100 * np.sum(np.abs(pred - target)) / np.sum(target),
    }  # Mean absolute percentage error
    return metrics


# Visualizing the target and predictions
def plot_predictions(pred, target):
    target.plot(figsize=(30, 10), label="target", linewidth=2)
    pred.plot(label="prediction", linewidth=2)
    plt.title("Persistence model performance", fontsize=20)
    plt.xlabel("Date and time", fontsize=18)
    plt.ylabel("Grid loss", fontsize=18)
    plt.xticks(fontsize=14)
    plt.legend()


# Initializing the model
model = PersistenceModel()
# Returns the last week's values
y_test = model.predict(x_test)
# Check model's performance
error_metrics = calculate_error(x_test, y_test)
print(f"Model performance for predicting loss for grid {grid_nr} is: {error_metrics}")
# Visualize the performance
plot_predictions(y_test, x_test)
<|endoftext|>
[{"Id": 42711809, "ScriptId": 11740565, "ParentScriptVersionId": NaN, "ScriptLanguageId": 9, "AuthorUserId": 1203660, "CreationDate": "09/15/2020 07:29:31", "VersionNumber": 2.0, "Title": "Grid loss demo and persistence model", "EvaluationDate": "09/15/2020", "IsChange": false, "TotalLines": 93.0, "LinesInsertedFromPrevious": 0.0, "LinesChangedFromPrevious": 0.0, "LinesUnchangedFromPrevious": 93.0, "LinesInsertedFromFork": NaN, "LinesDeletedFromFork": NaN, "LinesChangedFromFork": NaN, "LinesUnchangedFromFork": NaN, "TotalVotes": 0}]
[{"Id": 52282979, "KernelVersionId": 42711809, "SourceDatasetVersionId": 1487717}]
[{"Id": 1487717, "DatasetId": 746802, "DatasourceVersionId": 1521634, "CreatorUserId": 5773405, "LicenseName": "CC BY-SA 4.0", "CreationDate": "09/14/2020 14:35:19", "VersionNumber": 2.0, "Title": "Grid Loss Prediction Dataset", "Slug": "grid-loss-time-series-dataset", "Subtitle": "A time series dataset for predicting loss in three electrical grids in Norway", "Description": "### Context\n\nA power grid transports the electricity from power producers to the consumers. But all that is produced is not delivered to the customers. Some parts of it are lost in either transmission or distribution. In Norway, the grid companies are responsible for reporting this grid loss to the institutes responsible for national transmission networks. They have to nominate the expected loss day ahead to the market so that the electricity price can be decided.\n\nThe physics of grid losses are well understood and can be calculated quite accurately given the grid configuration. Still, as these are not known or changes all the time, calculating grid losses is not straight forward. \n\n\n### Content\n\nGrid loss is directly correlated with the total amount of power in the grid, which is also known as the grid load.\n\nWe provide data for three different grids from Norway that are owned by Tensio (Previously Tr\u00f8nderenergi Nett).\n\nFeatures:\nIn this dataset, we provide the hourly values of all the features we found relevant for predicting the grid loss. \n\nFor each of the grids, we have:\n1. Grid loss: historical measurements of grid loss in MWh\n2. Grid load: historical measurements of grid load in MWh\n3. Temperature forecast in Kelvin\n4. Predictions using the Prophet model in MWh\n5. Trend, daily, weekly and yearly components of the grid loss, also from the Prophet Model.\n\nOther than these grid specific features, we provide:\n1. Calendar features: year, season, month, week, weekday, hour, in the cyclic form (see Notes 1.) and whether it is a holiday or not.\n2. Incorrect data: whether the data was marked incorrect by the experts, in retrospective. We recommend removing this data before training your model.\n3. Estimated demand in Trondheim: predicted demand for electricity in Trondheim, a big city in the middle of Norway, in MWh (see Note 2.)\n\nWe have split the dataset into two parts: training and testing set.\n\nTraining set:\nThis file (train.csv) contains two years of data (December 2017 to November 2019). All the features mentioned above are provided for this duration.\n\nTest set:\nThis file (test.csv) contains six months of data (December 2019 to May 2020). All the features from training data are provided for the test set as well. Occasionally, some of the features could be missing.\n\nAdditionally, we provide a copy of test dataset (test\\_backfilled\\_missing\\_features.csv) where the missing features are backfilled.\n\n\nNote:\n1. Calendar features are cyclic in nature. If we encode the weekdays (Monday to Sunday) as 0 to 6, we find that while Sunday and Monday are next to each other, the distance between their embeddings does not reflect it. To reflect this cyclic nature of the calendar features, we created cyclic calendar features based on cosine and sine which together place the highest and lowest value of the features close to each other in the feature space. \n2. We don't have an estimate of demand for all the grids. We used the demand predictions for Trondheim, the biggest city closest to the three grids.\n3. The grid load is directly proportional to the grid loss. While we don't have predictions for grid load, but since we have historical measurements for them, it makes sense to predict it and use it as a feature for predicting the grid loss.\n4. While the Prophet Model did not perform nicely as a prediction tool for our dataset, we found it useful to include its prediction and other components as features in our model.\n5. Grid 3 has less training data than grid 1 and grid 2.\n6. We published our solution. For more details, please refer to:\n\nDalal, N., M\u00f8ln\u00e5, M., Herrem, M., R\u00f8en, M., & Gundersen, O. E. (2020). Day-Ahead Forecasting of Losses in the Distribution Network. In AAAI (pp. 13148-13155).\n\nBibtex format for citation:\n\n@incollection{dalal2020a,\n  author = {Dalal, N. and M\u00f8ln\u00e5, M. and Herrem, M. and R\u00f8en, M. and Gundersen, O.E.},\n  date = {2020},\n  title = {Day-Ahead Forecasting of Losses in the Distribution Network},\n  pages = {13148\u201313155},\n  language = {en},\n  booktitle = {AAAI}\n}\n\n\n### Challenges\n\nWorking with clean and processed data often hides the complexity of running the model in deployment. Some of the challenges we had while predicting grid loss in deployment are:\n\n1. Day-ahead predictions: We need to predict the grid loss for the next day before 10 am the current day at an hourly resolution i.e on 10:00 May 26, 2020, we need to predict the grid loss on May 27, 2020, from midnight to 23:00 May 28, 2020, at an hourly resolution (24 values) for each grid.\n2. Delayed measurements: We don't receive the measured values of load and loss immediately. We receive them 5 days after. Sometimes, there can be additional delays for a few more days. While grid loss and load are provided for the test data set as well, DO NOT USE them as features, unless they are 6 days old i.e while predicting grid loss for 27th January 2020, you can use the grid loss values will 20th January 2020. Using grid loss or grid load data after that date is unfair and will be discarded.\n3. Missing data: Sometimes, we don't receive some of the features. For example, weather client might be out of service. You should make sure that your model should work even when some features are unavailable/missing.\n4. Incorrect data: There have been times when the measurements we received were incorrect, by a big margin. They have been marked in the dataset in the incorrect_data column. It is recommended to remove those data points before you start analysing the data.\n5. Less training data: For one of the grids, grid 3, we only have a few months of data.\n6. Changes in the grids: Grid structures can keep changing. Sometimes new big consumers are added, or small grids can be merged into big ones.\n\n\n### Acknowledgements\n\nWe wouldn't be here without the help of others. We would like to thank Tensio for allowing us to make their grid data public in the interest of open science and research. We would also like to thank the AI group in NTNU for strong collaborations and scientific discussions.\nIf you use this dataset, please cite the following paper:\nDalal, N., M\u00f8ln\u00e5, M., Herrem, M., R\u00f8en, M., & Gundersen, O. E. (2020). Day-Ahead Forecasting of Losses in the Distribution Network. In AAAI (pp. 13148-13155).", "VersionNotes": "Split the data into train and test", "TotalCompressedBytes": 0.0, "TotalUncompressedBytes": 0.0}]
[{"Id": 746802, "CreatorUserId": 1203660, "OwnerUserId": 5773405.0, "OwnerOrganizationId": NaN, "CurrentDatasetVersionId": 1546931.0, "CurrentDatasourceVersionId": 1581682.0, "ForumId": 761702, "Type": 2, "CreationDate": "06/30/2020 08:44:25", "LastActivityDate": "06/30/2020", "TotalViews": 6191, "TotalDownloads": 433, "TotalVotes": 10, "TotalKernels": 3}]
# # Grid loss demonstration and Persistence model
# This notebook will give a demo for how to use the dataset. The pupose of this notebook is to provide the persistence model for predicting the grid loss.
# **Persistence model:**
# Persistence model is a well used baseline time series perdiction. It assumes that time series data does not change rapidly from day to day. Under this assumption, the persistence models 'predicts' that the values in the future will be a repitition of the last observed values. In this project, persistence model assumes that the grid loss for today will be same as the grid loss last week, the same day. Hence, in the model implementation, training is not needed. For prediction, it just returns the last week's measured grid loss values.
# As you will notice, Persistence model is not the best model. Feel free to explore other features and beat its performance with your models.
import numpy as np  # linear algebra
import pandas as pd  # data processing, CSV file I/O (e.g. pd.read_csv)
import datetime as dt  # date time
import matplotlib.pyplot as plt  # for plotting

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory
import os

for dirname, _, filenames in os.walk("/kaggle/input"):
    for filename in filenames:
        print(os.path.join(dirname, filename))
# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All"
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session

# For this demonstration, we used the test.csv file from the dataset. Persistence model was tested on this whole file in this demo, but feel free to play around the start date, duration and end date. We predicted grid loss for grid 1 here.
## Add dates and grid number you are interested in
start_date = dt.datetime(2019, 12, 1)
end_date = dt.datetime(2020, 5, 31)
grid_nr = 1
grid_col = f"grid{grid_nr}-loss"
# Load the test dataset. Training dataset is not needed here as Persistence model just returns the grid loss values from the week before, hence no training is needed.
test_data = pd.read_csv("../input/grid-loss-time-series-dataset/test.csv", index_col=0)
x_test = test_data[grid_col]
display(x_test.head())


# Persistence model was defined below. To use the model in the similar fashion as other sklearn models, we used the same prototype for our model.
# Defining the Persistence model, in line with other sklearn models
class PersistenceModel:
    def __init__(self):
        pass

    def train(self, x_train, y_train):
        # No training needed
        pass

    def predict(self, x_test):
        # returns the values shifted back by 7 days (i.e. 24*7 hourly values)
        return x_test.shift(24 * 7)


## Other helful functions
# Calculating the model performance. MAE, RMSE and MAPE are calculated.
def calculate_error(pred, target):
    target = target.loc[pred.index[0] : pred.index[-1]]
    metrics = {
        "mae": np.mean(np.abs(pred - target)),  # Mean absolute error
        "rmse": np.sqrt(np.mean((pred - target) ** 2)),  # Root mean squared error
        "mape": 100 * np.sum(np.abs(pred - target)) / np.sum(target),
    }  # Mean absolute percentage error
    return metrics


# Visualizing the target and predictions
def plot_predictions(pred, target):
    target.plot(figsize=(30, 10), label="target", linewidth=2)
    pred.plot(label="prediction", linewidth=2)
    plt.title("Persistence model performance", fontsize=20)
    plt.xlabel("Date and time", fontsize=18)
    plt.ylabel("Grid loss", fontsize=18)
    plt.xticks(fontsize=14)
    plt.legend()


# Initializing the model
model = PersistenceModel()
# Returns the last week's values
y_test = model.predict(x_test)
# Check model's performance
error_metrics = calculate_error(x_test, y_test)
print(f"Model performance for predicting loss for grid {grid_nr} is: {error_metrics}")
# Visualize the performance
plot_predictions(y_test, x_test)
