Efficiently Format Big DataFrame for Ingestion into Time Series Transformer
I am trying to use a Time Series Transformer on a huge multivariate dataset of 101,049,768 rows.The dataset is structured as follows:MONTH: goes from ‘2021-12-01’ to ‘2023-04-01’; with 17 unique dates being the first date of each month in that intervalMSISDN: includes “5,944,104” unique phone number; which are repeated for each monthTOT_OUTG_REV: which includes the total revenue (a real number) generated by the MSISDN in the particular MONTH variableTOT_DATA_REV: which includes the total data revenue (a real number) generated by the MSISDN in the particular MONTH variable; this is the target variableNB_OUTG_SMS: which includes the number of SMS (an int) sent by MSISDNCOMMUNE: which includes the city (a categorical variable) the MSISDN is located inOCCUPATION: which includes the profession (a categorical variable) of the MSISDNREG_DATE: which includes the registration date of the MSISDNHere is a sample of the code I wrote to format the datasetCODE:pers_loc_rev_sample_hf = []
start_date = pers_loc_rev["COHORT_MONTH"].min()
for i, msisdn in enumerate(pers_loc_rev_apr23_msisdn[8470:]):
    REDACTED_TOKEN = {}
    REDACTED_TOKEN["MSISDN"] = msisdn
    REDACTED_TOKEN["START_DATE"] = pers_loc_rev[pers_loc_rev["MSISDN"]==msisdn]["COHORT_MONTH"].min()
    REDACTED_TOKEN["TARGET"] = pers_loc_rev[pers_loc_rev["MSISDN"]==msisdn]["TOT_DATA_REV"].values
    REDACTED_TOKEN["FEAT_STATIC_CAT"] = []
    REDACTED_TOKEN["FEAT_DYNAMIC_REAL"] = []
    for dyn_col in dynamic_real_cols:
        REDACTED_TOKEN["FEAT_DYNAMIC_REAL"].append(pers_loc_rev[pers_loc_rev["MSISDN"]==msisdn][dyn_col].values)
    
    for stat_col in static_cat_cols:
        REDACTED_TOKEN["FEAT_STATIC_CAT"].append(pers_loc_rev[pers_loc_rev["MSISDN"]==msisdn][stat_col].values)
    if i==10000 : break   
    pers_loc_rev_sample_hf.append(REDACTED_TOKEN)from datasets import Dataset, Features, Value, Sequence

features  = Features(
    {    
        "START_DATE": Value("timestamp[s]"),
        "TARGET": Sequence(Value("float32")),
        "FEAT_STATIC_CAT": Sequence(Value("string")),
        "FEAT_DYNAMIC_REAL": Sequence(Sequence(Value("float32"))),
        "MSISDN": Value("string"),
    }
)
# pers_loc_rev_sample_hf
dataset = Dataset.from_list(pers_loc_rev_sample_hf, features=features)This code is extremely slow for a 101,049,768 rows dataframe, does anyone have a recommendation on how to do these efficient?