\section{Introduction}
With the rapid development of online payment, credit card fraud also continues to climb to new heights, which leads to losses totalling billions of US dollars each year [1]. To address this issue, banks and online payment companies turn to anti-fraud systems to detect illegitimate transactions. Among numerous techniques considered, machine learning (ML) and deep learning (DL) are attracting much attention due to their powerful predictive ability as a fraud detection system [2][3]. Generally, for ML/DL-based methods, models' input features are card transaction data such as the identity of card holders and amount of funds, while outputs are confidence scores that form a probability space for determination of genuine/fraudulent transactions.

Credit card fraud detection, however, is not a typical classification task since fraudulent transactions are extremely rare out of all transactions. Specifically, the dataset is highly skewed, which often causes models to perform poorly when encountering rare cases. There have been some works directly applying traditional ML-based methods [4][5][6][7] where a large dataset is inessential. In terms of DL-based approaches, attempts at high quality data augmentation [8] and the use of key features [9][10] have been made, while few studies [11] regard this problem as anomaly detection which naturally avoids the imbalance problem and has larger room for improvement.

Another crucial topic for DL-based credit card fraud detection systems is the model explainability [12], a research hotspot in the ML community. Considering the black-box property of neural networks, it is desirable to give an explanation of why a transaction is detected as fraudulent for official reports in court. Though various techniques for different levels and perspectives of explanations have been proposed [13][14][15][16][17], investigation into their applications of anomaly detection methods are lacking.

In this paper, we make the first attempt to leverage a novel anomaly detection framework [18] along with a LIME-based [13] explaining module in the realm of credit card fraud detection. This anomaly detection framework is adopted for its promising performance on detecting irregular images, while the LIME is chosen because of its focus on a single instance of interest which might be the fraudulent transaction in this case. The detection module is adjusted to match the properties of tabular financial data and experiments on comparison with other baseline methods are carried out. LIME explainers are displayed in different parts of the detection module to monitor and inspect the whole prediction pipeline. In summary, the contributions of this work is twofold:
\begin{enumerate}
    \item Modifying and utilizing a novel yet simple anomaly detection architecture to cope with the credit card fraud detection problem, achieving state-of-the-art performance in comparison with other cutting-edge or iconic methods.
    \item Applying LIME to give explanations to different input-output relations for a certain transaction of interest in the detection module.
\end{enumerate}

\section{Related Work}
This paper puts effort into the imbalance issue of credit card fraud detection dataset and the interpretability of utilized anomaly detection framework. In this section, we shall discuss several benchmark methods for imbalanced classification and explainable AI.
\subsection{Imbalanced Classification}
Datasets for detection of fraudulent transactions are inherently skewed because normal cases significantly outnumber dishonest ones. The class imbalance problem greatly increases the difficulty of neural network training. There are two main solutions: (1) manipulating data to balance the dataset and (2) using algorithms like weighted loss function or one-class classification (OCC).

The first approach includes undersampling of majority class and oversampling of minority class. The former deletes excessive data points and the latter generates synthetic data that have similar distributions of real data in minority class, which is much more complicated yet widely adopted. For data synthesis, SMOTE [19] is the most common technique which utilizes k-nearest neighbors algorithm to produce samples close to the minor-class data in the feature space. GAN-based [20] upsampling models [8] have also been broadly studied due to its impressive performance in generating vivid images.

In terms of algorithm-based approaches, weighting of majority and minority class is usually implemented by weighted loss functions. Among them, Focal Loss [21] is one of the most iconic works which encourages networks to down-weight easy examples and focus on learning hard negative examples. One-class classification, also known as anomaly detection from application point of view, merely uses the majority class during training. Conventional ML anomaly detection methods mainly include linear models, such as OCSVM [22], proximity-based models, such as OCNN [23], and probabilistic models, such as COPOD [24]. OCSVM is SVM trained in an unsupervised manner, trying to find a hyperplane that well-encloses the only class in the training set and excludes potential outliers. OCNN is the one-class k-nearest neighbors method which calculates the averaged distance to k nearest data points as the outlier score. COPOD is a copula-based non-parametric statistical approach that can directly provide some decision explanations.

AnoGAN [25] first utilizes adversarial training, where the generator learns to produce samples with distributions similar to major-class data (inliers). As a result, the discriminator could tell whether the input is from a minor class (outliers) by comparing its distribution with that of generated samples. OCAN [11] uses a novel LSTM-AutoEncoder for input feature representation and then conducts the prediction process similar to AnoGAN. These two models both apply a generator to produce fake features sampling from a noise. However, as an intrinsic defect of GAN, the upsampling generation process is unstable. Reference [18] replaces the original noise-to-sample generator with an AutoEncoder for sample reconstruction, which is adopted by many following works [26][27]. For example, Both the GANomaly [26] and Skip-GANomaly [27] have an AutoEncoder as parts of the generator. The former introduces an extra encoder for latent space restriction, while the latter features the use of skip-connections.

\subsection{Explainable AI}
Explainable AI (XAI) is an emerging field in the AI community which aims at allowing human users to comprehend and trust model outputs. Though this domain is relatively new, broad and comprehensive studies [12] have been done. This paper categorizes these methods based on the scope of explanations: (1) local interpretability and (2) global interpretability.

Local interpretability refers to explanations of decisions a model makes about an instance. Methods in this category either visualize or plot the importance of each input feature to the final output. The saliency map [14], which has long been present and used in local explanations, is a gradient-based visualization approach that calculates each pixel's attribution. LIME [13] proposes a concrete framework for implementations of local surrogate models. Given an instance of interest and a dataset, LIME first generates new data points by perturbing the dataset and getting the black-box predictions for these new data points. Subsequently, a simple transparent model, such as a decision tree, is trained on this dataset, which then should be a good approximation of the target model around this instance. Therefore, analysis on this explainable proxy model could be easily carried out to provide human users with a clear view on how each input feature of this particular instance leads to final outputs made by the black-box model.

Global interpretability answers how model parameters affect the final prediction. It explains the whole model rather than a certain input instance. Activation Maximization (AM) methods [15][16] put forward visualization of neural networks, which is done by finding the input probing image that maximally activates certain neurons with gradient ascent. In particular, reference [16] proposes to use generative models, where the generated images are fed into the target model for classification. We could then examine what kinds of noises and corresponding generated images lead to a certain prediction result. Following works [17] have achieved image-level fine-grained visual explanations for convolutional neural networks (CNN).

\begin{figure}[t]
    \centering
    \includegraphics[width=0.45\textwidth]{structure}
    \caption{Structure of our method.}
\end{figure}

\section{Methodology}
The proposed framework comprises two modules: (1) the anomaly detection model and (2) the model explainers. The former's architecture is mainly derived from [18], which consists of an AutoEncoder for input-output reconstruction and a fully-connected classifier for fraud detection. The two networks are trained in an adversarial and unsupervised manner so as to cope with the class imbalance issue, and some adjustments are applied on account of specific properties of this credit card fraud detection task. The model explainers provide users with a clear perspective on how the network output is influenced by input features. We choose LIME to analyze a single transaction instance of interest. The overview of the proposed structure is shown in Fig 1.

\subsection{Anomaly Detection Model - AutoEncoder}
Capable of reconstructing and denoising input samples [18][26][27], AutoEncoders have been widely used in novelty detection tasks recently. In this paper, the target and non-target class are the genuine and fraudulent transaction, respectively. For the low-dimensional tabular financial transaction data, we adopt an AutoEncoder for reconstruction. That is, no extra Gaussian noise is added in the input.

The model is composed of fully-connected layers. Batch Normalization (BN) and Rectified Linear Unit (ReLU) are utilized to ensure stable gradient flows. Given that fraudulent transactions are extremely rare, we only train the AutoEncoder with genuine transactions. Specifically, the model is trained to preserve the distribution of target-class inputs, while the outliers, namely, non-target class, will be naturally mapped to an uncertain point in the latent space and fail to be reconstructed due to their absence from the training set. To achieve this goal, two loss functions are presented:
\begin{equation}

    L_\mathcal{R} = \|\mathcal{R}(X) - X)\|_2,
\end{equation}
\begin{equation}
    L_\mathcal{R}^{GAN} = -log{\ \mathcal{C}(\mathcal{R}(X))}.
\end{equation}
Equation (1) is the reconstruction loss, which is decided to be L2 loss in the next section. R denotes the reconstructor, that is, the AutoEncoder. X is the transaction data. Equation (2), which is a part of adversarial loss, is the binary cross entropy that guides the AutoEncoder to produce outputs that have the same distribution as inputs and subsequently confuse the discriminator. C stands for the Classifier. The overall training objective of the AutoEncoder is then:
\begin{equation}
    \min(\E_{X \sim pt}[\|\mathcal{R}(X) - X)\|_2 -log{\ \mathcal{C}(\mathcal{R}(X))}]),
\end{equation}
where pt denotes the distribution of genuine transactions.

\subsection{Anomaly Detection Model - Classifier}
The classifier is a simple Multilayer Perceptron (MLP) that aims to separate original transaction data from the reconstructed one. The output of this model is a single value representing the probability of target class. As a result, the binary cross entropy loss for the Classifier can be as follows:
\begin{equation}
    L_{\mathcal{C}}^{GAN} = -log{\ \mathcal{C}(X)} - log{(1- \mathcal{C}(\mathcal{R}(X)))}.
\end{equation}
The adversarial training objective in the whole anomaly detection model can then be summarized as:
\begin{equation}
    \begin{split}
        \min_{\mathcal{R}}\max_{\mathcal{C}}(\E_{X \sim pt}[log{\ \mathcal{C}(X)} + log{(1- \mathcal{C}(\mathcal{R}(X)))}]).
    \end{split}
\end{equation}
It is notable that the training procedure is done when the AutoEncoder and classifier reach the Nash equilibrium and the reconstruction loss is small. This means the reconstruction process is performed well and the classifier outputs around 0.5 on both original and reconstructed features, which indicates that their distributions are too close to be distinguished. In this situation, it is simple and straightforward to determine by a threshold whether the instance of interest is a fraudulent transaction, which is mapped to an unknown distribution when passing through the AutoEncoder and leads to a prediction value far from 0.5 at the classifier. The threshold is set to be 0.7 in this work.

\subsection{Model Explainer}
In terms of the explainability of our one-class fraud detection framework, we are particularly interested in the interpretations of (1) how the reconstruction stage is affected by genuine/fraudulent transaction data, (2) how the consequent reconstructed features bring about final prediction results, and (3) the overall credit card fraud detection module. Therefore, three LIME-based explainers, which are simple white-box models, are employed to interpret the three different input-output relations. According to their functions, models are named as AE explainer, C explainer, and general explainer, respectively, as demonstrated in Fig 1.

As a model-agnostic technique, LIME only requires the target model to be a classifier or regressor. Thus, the C and general classification explainer can be directly trained with samples acquired by sampling from the distribution of the given dataset, while the vector output of AutoEncoder obstructs this way for the AE explainer. To address this problem, we make use of L2-norm to compute a single value of reconstruction error as the label for the training of the AE regression explainer. This is equivalent to (1), while we describe the formula here to clarify the calculation of label:
\begin{equation}
    label = \frac{\sum\limits_{i = 1}^n{(\mathcal{R}(X)_i - X_i)^2}}{n},
\end{equation}
where the character n is the number of features. With this step, the AE explainer is turned to an explainable regression model whose outputs denote L2 reconstruction error.

\setlength{\arrayrulewidth}{0.5pt}
\setlength{\tabcolsep}{25pt}
\renewcommand{\arraystretch}{1.5}

\begin{table*}[t]
    \centering
    \caption{Credit card fraud detection results on accuracy, precision, recall, F1-score, and Matthews correlation coefficient}
    \begin{tabular}{c c c c c c}
    \hline
    Methods & Accuracy & Precision & Recall & F1-score & MCC \\
    \hline
    OCSVM & 0.8898 & 0.9204 & 0.8673 & 0.8931 & 0.7811 \\
    OCNN & 0.9000 & 0.9122 & 0.8904 & 0.9012 & 0.8002 \\
    COPOD & 0.8388 & 0.7224 & 0.9415 & 0.8175 & 0.6967 \\
    AutoEncoder & 0.8000 & \textbf{0.9245} & 0.7402 & 0.8221 & 0.6195 \\
    OCAN & 0.8806 & 0.8061 & \textbf{0.9472} & 0.8710 & 0.7698 \\
    Ours & \textbf{0.9061} & 0.9216 & 0.8878 & \textbf{0.9044} & \textbf{0.8128} \\

    \hline
    \end{tabular}

\end{table*}

\section{Experiments}
\subsection{Experimental setup}
\subsubsection{Datasets}
We test the performance of our anomaly detection framework on a benchmark credit card fraud detection dataset\footnote{https://www.kaggle.com/mlg-ulb/creditcardfraud}. The dataset contains 284,807 credit card transactions, collected in Europe during a 2-day period in September 2013. There are only 492 fraudulent cases (minority class, labeled as 1) accounting for 0.172\% and the other 234315 cases are genuine transactions (majority class, labeled as 0), which indicates the dataset is highly imbalanced. Each transaction has 30 features, 28 of which are principal components obtained from PCA and the other two are time and amount. The 28 features are listed as V1 to V28 and further information is not provided because of confidentiality issues. Time feature denotes the elapsed time between the current and first transaction, which is considered in temporal-based models [9][10]. Amount feature is the amount of money in the transaction, and it is used in cost-sensitive learning. We only utilize the 28 features in this work as in [11]. Since the data have been normalized before PCA, we do not clean them again before the training process. Since the original dataset is not split into training and testing sets, we select 490 out of 492 fraudulent cases and 490 out of 234315 genuine cases to generate a well-balanced testing set, and the remaining 233825 genuine cases form the training set for our model. All the baselines are tested on this testing set, while training sets for different methods may differ.

\subsubsection{anomaly detection methods}
We compare our framework with five baseline models: OCSVM, OCNN, COPOD, vanilla AutoEncoder, and OCAN. For the first four baselines, we use implementations provided in PyOD [28], which is a renowned high-quality toolkit for novelty detection. OCAN is not included in this package, so we retrieve its open-source original implementation\footnote{https://github.com/PanpanZheng/OCAN}. In terms of training parameters, OCSVM, COPOD, AutoEncoder, and OCAN are trained with 700 genuine cases. OCNN has no training stage, but an evaluation set is required for hyper-parameter tuning. Thus, we randomly pick 25 genuine and 25 fraudulent cases for evaluation and subsequently determine k = 5. All hyper-parameters are manually adjusted to reach the best performances if no default values are provided by the original paper or source code.

Our model is trained on the whole training set for 2 epochs. The batch size is 4096, and the learning rate is 2e-4. Weight initialization is applied to ensure a fine starting point. The threshold is 0.7, and a classifier output above this value is identified as an illegal transaction.

\subsubsection{LIME}
We use LIME's open-source implementations\footnote{https://github.com/marcotcr/lime} to build the three explainers. The AE explainer is a regression model, where the label is defined in (6). The D and general explainer are classifiers, where the label is the output of the fraud detection module.

Firstly, we select a single fraudulent case from the testing set. Secondly, 5000 data points are produced by sampling from the distribution of the class-balanced testing set, and their corresponding outputs by our anomaly detection model are used as labels. Finally, these data points and labels serve as the training set to build a simple explainable model, into which we can feed the original fraudulent case to obtain its prediction explanation.

\subsection{Results}
\subsubsection{Anomaly Detection Performance}
The performance measures include accuracy, precision, recall, F1-score, and Matthews correlation coefficient (MCC). Particularly, F1-score and MCC are balanced binary classification measures that take the entire confusion matrix into account. Precision indicates how likely a case detected asf fraudulent is indeed illegitimate. Recall denotes the proportion of fraudulent cases that are successfully detected. The testing results are displayed in Table 1.

It is remarkable that our method achieves 0.9061 accuracy, 0.9044 F1-score, and 0.8128 MCC, higher than any other baselines. This indicates that our model reaches the best performance if we consider precision and recall simultaneously. For precision only, AutoEncoder sacrifices recall to reach the highest precision (0.9245), resulting in a lower accuracy, F1-score, and MCC. However, this merely outperforms our result (0.9216), by a small margin. For recall only, OCAN and COPOD obtain promising results (0.9472 and 0.9415, respectively). Nevertheless, they fail to reach a balance between precision and recall. If we compare OCSVM, OCNN, and our method, which have relatively balanced precision and recall, our method almost outperforms the other two models in all indices except for recall, where OCNN is only slightly better.

We also compare different loss functions to determine the most suitable form for reconstruction. Three candidates are L1, SmoothL1, and L2 loss. For simplicity, we finally choose L2 loss because it results in the best MCC, as shown in Table 2.

\begin{table}[h]
    \centering
    \caption{MCC of different reconstruction losses}
    \begin{tabular}{ | c | c | }
        \hline
           Loss Function & MCC \\
        \hline
        L1  & 0.7990 \\
        \hline
        SmoothL1 & 0.7970 \\
        \hline
        L2  & \textbf{0.8128} \\
        \hline
    \end{tabular}
\end{table}

Finally, we plot the receiver operating characteristic (ROC) curve and calculate the area under curve (AUC) in Fig 2. ROC and AUC show the classifier performance at various thresholds and serve as a criterion for model robustness. The AUC value ranges from 0 to 1. Specifically, 0.5 (such as the diagonal in Fig 2) denotes a naive classifier and 1 indicates a perfect classifier. Our method achieves an AUC of 0.9434.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.3\textwidth]{AUROC}
    \caption{ROC and AUC of our model}
\end{figure}

\subsubsection{Model Interpretation}
Fig 3., 4., and 5. display interpretations of a fraudulent transaction by the AE, C, and general explainers, respectively. The left sub-figure of each figure (Fig 3(a), 4(a), 5(a)) is the output value of the explainer. The middle bar chart (Fig 3(b), 4(b), 5(b)) shows the contribution of each feature value to the explainer output. The right table (Fig 3(c), 4(c), 5(c)) lists down the value of each feature. For clarity, we only plot the top six important features out of 28 features.

\begin{figure}[H]
     \centering
     \begin{subfigure}[b]{0.17\textwidth}
         \centering
         \includegraphics[width=\textwidth]{AE_predicted}
         \caption{}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.2\textwidth}
         \centering
         \includegraphics[width=\textwidth]{AE_explanation}
         \caption{}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.08\textwidth}
         \centering
         \includegraphics[width=\textwidth]{AE_table}
         \caption{}
     \end{subfigure}
        \caption{Interpretations by the AE explainer}
        \label{fig:three graphs}
\end{figure}

We first elaborate on Fig 3. Since the input transaction is fraudulent, it is mapped to an unknown distribution rather than being perfectly reconstructed, as discussed in the previous section. This leads to a large reconstructed error up to 156.35. Then, by disturbing the fraudulent input, it is found that V1 possesses the highest influence among all features. If V1 rises from -16.60, as listed in the right feature value table, to -2.73, as shown in the middle chart, the predicted value will be reduced to 156.35 - 23.92 = 132.43.

\begin{figure}[H]
     \centering
     \begin{subfigure}[b]{0.17\textwidth}
         \centering
         \includegraphics[width=\textwidth]{D_predicted}
         \caption{}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.2\textwidth}
         \centering
         \includegraphics[width=\textwidth]{D_explanation}
         \caption{}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.08\textwidth}
         \centering
         \includegraphics[width=\textwidth]{D_table}
         \caption{}
     \end{subfigure}
        \caption{Interpretations by the C explainer}
        \label{fig:three graphs}
\end{figure}

Fig 4. shows how each feature of the reconstructed input affects the classifier prediction. If V19 decreases from 1.63 to 0.58, the classifier will reduce its confidence of fraudulent transaction from 0.59 to 0.53. Observing Fig 3(c) and 4(c), we can find that feature values of the reconstructed inputs are significantly different from that of the original input. This matches the high reconstruction error shown in Fig 3.

\begin{figure}[H]
     \centering
     \begin{subfigure}[b]{0.17\textwidth}
         \centering
         \includegraphics[width=\textwidth]{AED_predicted}
         \caption{}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.2\textwidth}
         \centering
         \includegraphics[width=\textwidth]{AED_explanation}
         \caption{}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.08\textwidth}
         \centering
         \includegraphics[width=\textwidth]{AED_table}
         \caption{}
     \end{subfigure}
        \caption{Interpretations by the general explainer}
        \label{fig:three graphs}
\end{figure}

Fig 5. is the overall analysis. The explainer is a single transparent model, where inputs are original transactions and outputs are predictions. It approximates the entire fraud detection module and the provided analysis can be interpreted as in Fig 3. and Fig 4.

\section{Conclusion}
This work proposes to leverage an adversarially trained anomaly detection model for credit card fraud detection problem. Experimental results show that this framework outperforms other iconic or state-of-the-art baseline models. Furthermore, LIME is applied to investigate input-output relations of this fraud detection model and analyses of an instance of interest are presented, providing a clear view on how each input feature influences the final prediction. Future work will focus on interpretability of other unsupervised or semi-supervised methods in the application area.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi


\section{Introduction}
\label{sect:Introduction}

Data is considered a key production factor, comparable in importance to labour, capital, and infrastructure. Companies are often in need of data they do not possess, or cannot collect directly. Therefore, general purpose\footnote{See for example DAWEX, Azure Data Catalog, or AWS Data Exchange} and domain specific\footnote{See for example, Openprise, Lotame PDX (marketing), Qlik (business intelligence), or Battlefin (investment information)} data marketplaces (DMs) have appeared with the purpose of building a business of mediation between data selling and data buying \emph{companies}. Leading data management platforms and innovative startups\footnote{See for example Snowflake, Cognite, Carto, and Openprise} are also introducing marketplace functionalities into their products. Finally, personal information management systems (PIMS) have answered the call of recent legislative developments in personal data protection by offering data control, portability, and monetization services for \emph{individuals}.

Designing and building a successful DM calls for solving a plethora of technology, business, and economics challenges in the context of a complex two-sided market (see \cite{Armstrong06, Rochet06, Rysman09} for an exposition). According to our survey of more than 75 real-world data marketplaces, the most common bootstrapping strategy is for a DM to spend effort and money to attract a sufficient set of data sellers, and then try to convince as many buyers as possible to start purchasing these datasets. Therein lie two fundamental problems: (1) the \emph{dataset pricing problem} for data sellers, and (2) the \emph{dataset purchasing problem} for data buyers.

Recent theoretical work on the intersection between computer science and economics has looked into those problems, and has proposed solution concepts and algorithms for them \cite{Agarwal19, Chen19, Chawla19, Koutris15}. For data sellers, selecting efficient prices requires knowing the level of competition with other data sellers, the willingness to pay of buyers, potential customer lock-ins and other information that affects prices in digital and non-digital markets. For buyers, the problem of selecting which datasets to buy (problem (2)), given the prices set by sellers (problem (1)), can be further broken down into 2 interrelated subproblems: (2.a) compute how useful these datasets will be to their AI/ML algorithms, something that can be captured by various accuracy metrics, and (2.b) compute how such accuracy can be converted into monetary gains (via e.g., improved sales, acquisition of new customers, retention of existing ones, etc.). Sub-problem (2.b) is probably the easiest of the two challenges faced by buyers, since most companies are able to gather historical data about the impact of things like recommendation quality on actual sales \cite{Brovman16}. Subproblem (2.a), on the other hand, is inherently more challenging, since buyers need to have access to the data before they can compute their value for their AI/ML task, but such access is only granted \emph{after} a dataset purchase has taken place -- a chicken and egg problem essentially. (2.a) is further exacerbated when the buyer can/has to buy more than one dataset in order to improve the accuracy of its AI/ML algorithm. With $N$ available datasets, a buyer has $O(2^N)$ data purchase options each one with a cost equal to the sum of individual dataset prices and a value defined by the maximum accuracy of AI/ML algorithm operating over the aggregate data.

In theoretical works, the value of any subset of datasets for a data buyer is considered as known \textit{a priori} \cite{Shen16}. In reality, however, things are completely different \cite{Hubert18}. In almost all the 75 DMs that we have surveyed, data sellers provide only a description of their datasets, a price, sometimes an outdated sample, and buyers have to make purchase decision with that information alone. Few of these DM (e.g., Dawex, Airbloc, Wibson or Databroker) also allow buyers to make offers (bids) for data when sellers do not indicate a fixed price, or if they are willing to pay something below the asking price. This case suffers as well from the fundamental problem (2.a) of not knowing the value of a dataset before purchasing it.
\vspace{2pt}

\noindent \textbf{Our contribution:} In this paper we show how to solve the \emph{dataset purchasing problem} for data buyers in a way that approximates the efficiency of an optimal full information solution, but in a way that is implementable in practice with real-world DMs. Our main contribution is a family of dataset purchase algorithms that we call "Try Before You Buy" (or TBYB) that allow data buyers to identify the best datasets to buy with only $O(N)$ information about the accuracy of AI/ML algorithms on individual datasets, instead of $O(2^N)$ information used by an optimal strategy using full information. Effectively, TBYB needs to know only the accuracy of an AI/ML on \emph{individual} datasets, and with this information it can approximate the optimal \emph{combination} of datasets that maximizes the profit of the buyer, i.e., the difference between the value extracted from the datasets minus the cost of purchasing them. The accuracy of individual datasets can either be precomputed by the DM or the data sellers, and be made available as part of the dataset description (e.g., for some common AI/ML algorithms). Another alternative is for the DM to use recently developed ``sandboxed'' environments that allow data buyers to experiment versions of the data without being able to copy or extract them (hence the ``Try'' part on the algorithm's name; Otonomo, Advaneo, Caruso or Battlefin are examples of marketplaces that implement such functionality). Overall, with TBYB our objective is to increase the efficiency of buying datasets online from DMs. We believe that this is key for allowing both DMs and the data offer side to grow, as well.
\vspace{2pt}

\noindent \textbf{Our findings:} We compare the performance of TBYB against several heuristics that do not use information about the value of a datasets for the particular AI/ML task at hand, as well as against an optimal solution that uses full information. We start with a synthetic evaluation and then validate our conclusions using real-world spatio-temporal data and a use case in predicting demand for taxi rides in metropolitan areas \cite{Andres20}. Our findings are as follows:
\begin{itemize}
    \item TBYB remains close to the optimal for a wide range of parameters, whereas its performance gap against the heuristics increases with the catalog size.
    \item TBYB is almost optimal when buying more data is yielding a progressively diminishing return in value for the buyer (i.e., when the value function of the buyer is concave). With convex function, it becomes increasingly difficult for TBYB to match the optimal performance. It's performance gap with the heuristics, however, is maintained.
    \item When the asking price of datasets does not correlate with their actual value for the buyer, the performance advantage of TBYB over the heuristic becomes maximal. When the pricing of data follows their value for buyers, the performance of TBYB is still superior but the gap with the heuristics becomes smaller.
\end{itemize}

Overall, our work demonstrates that near optimal dataset purchasing is realistic in practice and that it could be implemented relatively easy by real-world data marketplaces.

\section{Marketplace Model \& Definitions}
\label{sect:Definitions}

Existing DMs typically list the datasets that they make available and provide for each one a description and a price. In our case we will assume that the DM also provides for each dataset its \emph{accuracy} over a range of common AI/ML tasks. This list cannot/need not be exhaustive, nor it needs to capture all the specificities of the particular AI/ML algorithm that the buyer intends to use. The intention is merely to provide the buyer with a hint, even an \emph{approximate} one, regarding the accuracy that he should expect if he buys a particular dataset.

If, on the other hand, a buyer would like to know before buying a dataset the \emph{exact} performance of his algorithm on the data, then the following two options exist. The buyer could submit a description of the task for which he needs data, so that the DM returns a list of candidate sellers, or, alternatively, she could just go over the data catalog and select the best candidates manually. In both cases, the DM can provide a sandboxed environment in which the buyer can submit his \emph{exact} algorithm and get an \emph{exact} answer in terms of the achieved accuracy over each candidate dataset, without being able to see/copy the raw data.

To model any of the above cases (see figure \ref{Fig:model}), we will denote by $\mathcal{S}$ the set of suitable sellers for the AI/ML task of a particular buyer. We will denote by $d(s)$ the dataset offered by seller $s \in \mathcal{S}$, by $p(s)$ its \emph{price}, and by $a(d(s))$ the \emph{accuracy} that the buyer's AI/ML task can achieve if trained by $d(s)$. Similarly, for a subset of the sellers $S\in \mathcal{S}$, we will denote by $d(S)$ their aggregated dataset, and by $a(d(S))$ the maximum accuracy that can be achieved using all or a subset of the data in $d(S)$. We will also introduce the \emph{value function} $v(a)$ of the buyer, that indicates the (monetary) value that the buyer can achieve when his AI/ML algorithm gets to an accuracy of $a$. In Sect.~\ref{subsec:TheorSensitivityMUP} we will look at both concave and convex $v(\cdot)$ functions.

\begin{figure}
\centering
\includegraphics[width=0.8\textwidth]{Model.jpg}
\caption{Reference DM model}
\label{Fig:model}
\end{figure}

When datasets are sequentially bought, we will append a subscript to identify the round to the notation we already defined. Hence, $S_n \subset \mathcal{S}$ will refer to the set of eligible datasets in round $n$. We will denote by $P_n$ the set of data already under the buyer's control in round $n$. As a result, she is able to achieve an accuracy equal to $a_n=max_{S'\in 2^{P_n}} a(S')$ and a value $v_n=v(a_n)$.

Buyers will always look forward to optimizing their profit, hence a greedy buyer will decide to purchase a dataset $d(s)$, if its marginal value exceeds its cost, i.e. $v(a(d(s|P))) \geq {p(s)}$. In the event that such a value is unknown, the buyer must estimate it, and assume the risk that the purchase may not fulfil expectations.

\section{Data Purchase Strategies}
\label{sect:Purchase algorithms}

In this section we will present a series of data purchase strategies that cover the spectrum from full information, i.e., knowing the accuracy over any subset of the available data, to having no information about accuracy, as is currently the case in most DMs. In between the two extremes, lies our proposed algorithm called TBYB, that runs only on accuracy information of individual datasets.

\subsection{Optimal purchase under full information}
\label{subsec:fullinformation}
In this case, the buyer knows $a(d(S))$ for any subset $S\subseteq 2^\mathcal{S}$. This allows for an optimal purchase $\mathcal{S}^\star$ that maximizes the profit, i.e., the difference between the value that the buyer extracts from the data, and the cost paid to purchase them:

\begin{equation}
\label{eq:optimalfull}
    \mathcal{S}^\star =  arg\,max_{S\in 2^\mathcal{S}}  \left(v(a(d(S)))-\sum_{s \in S} p(s) \right),
\end{equation}

subject to $v(a(d(\mathcal{S^\star}))) \geq \sum_{s\in \mathcal{S}^\star} p(s)$.

Such a full information scenario is optimal from a buyer's perspective, but not scalable nor practical: a DM would need to compute the accuracy of each AI/ML algorithm over $2^{|\mathcal{S}|}$ combinations of eligible datasets.

\subsection{Try Before You Buy (TBYB)}
\label{subsec:TBYB}
For our proposal, we assume that the DM provides the buyer with the accuracy of her algorithm on individual datasets, but not on combinations of them. The algorithm is sequential and greedy in nature, and can run for up to $|\mathcal{S}|$ iterations. We will consider two versions.

\subsubsection{Stand-alone version - S-TBYB}
\label{subsec:S-TBYB}
The marketplace provides $a(d(s))$ on all $s\in\mathcal{S}$. Then the algorithm starts buying datasets in descending order of \emph{expected profit} until a stopping condition is reached. For the first dataset, the profit is not expected but exact, so the best dataset is bought provided $v(a(d(s)))- p(s) \geq -\lambda \cdot v(a^\star)$, where:
\begin{enumerate}
    \item the best accuracy $a^\star \leq 1$ that can be delivered by the marketplace. Either the data marketplace provides data buyers with this information, or the buyer makes her best guess, and
    \item the risk parameter $\lambda$ models the maximum relative admissible loss the buyer is willing to assume in each operation. The risk assumed every round will be bounded by $\lambda$ times the potential value of the sourcing operation which, in round n, is equal to $v(a^\star) - v_n$. For example, $\lambda = 0.1$ means that the buyer will buy a new dataset $s$ if its price is lower than the marginal value she expects to get plus $10\%$ of the maximum value that she could add by buying new data.
\end{enumerate}

In some sourcing problems, the marginal value of new data increases as more information is bought. In such a setting, buyers may be required to assume some temporary losses when acquiring the first datasets, in the hope that they provide additional accuracy, and become profitable when fused together with other data.

\paragraph{n-th iteration}The buyer will proceed as follows:
\begin{itemize}
        \item Identify the best possible dataset $s^\star \in S_n$ such that:
        \begin{equation}
            s^\star =  arg\,max_{s\in S_n}  \left( v(a(d(s)))- p(s) \right)
        \end{equation}
        \item Purchase $s^\star$ if its estimated marginal value exceeds its price, and a risk threshold that depends on the remaining value she expects to get out of the operation, i.e., if $v(E\{a(s^\star \cup S_n)\}) - v_n - p(s^\star) \geq -\lambda \cdot (v(a^\star) - v_n)$
        \item If the buy condition is met then, $s^\star$ is added to the set of controlled datasets: $P_{n+1} = P_n \cup d(s^\star)$ and the next round starts
        \item else if no dataset in $S$ meets this requirement, then the process stops
\end{itemize}

To estimate $E\{a(s^\star \cup S_n)\}$ the buyer could use the following information:

\begin{enumerate}
    \item The price and accuracy pairs $<p(s),v(s)>$ for all individual datasets $s\in S$
    \item The accuracy of every possible combination of already purchased datasets, i.e., the $v(S')$ for all $S' \subseteq 2^{P_n}$
\end{enumerate}

This estimation must be tailored to each specific problem, and it turns out to be non-trivial. We estimate the relative added accuracy of $s^\star$ by multiplying its individual accuracy $a(s^\star)$, and the ratio of the marginal contribution and the individual accuracy of the last purchased dataset:
\begin{equation}
\label{eq:ExpectedAccuracyEstimation}
    E\{a(s^\star \cup S_n)\} = \frac{a_n - a_{n-1}}{a(P_n - P_{n-1})} \cdot a(s^\star) \cdot (a^\star - a_n).
\end{equation}

\subsubsection{Assisted version - A-TBYB}
\label{subsec:A-TBYB}
In this case, we will assume the buyer is allowed to ask the marketplace \textit{every round} for the marginal accuracy of any eligible datasets given the data she already owns.

\paragraph{n-th iteration}The purchase process will be the following:
\begin{enumerate}
    \item Ask the marketplace for complementary datasets $S_n \subseteq \mathcal{S}$, and $a(d(s)|P_n),$ $\forall s \in S_n$ given the task $(\mathcal{M}, a)$ and $P_n$
    \item If $S_n \neq \emptyset$: \begin{itemize}
        \item Identify the best possible dataset $s^\star \in S_n$ such that:
        \begin{equation}
            s^\star =  arg\,max_{s\in S_n}  \left( v(a(d(s \cup P_n)))- p(s) \right)
        \end{equation}
        \item Buy provided $v(a(d(P_n \cup d(s^\star)))) - v_n - p(s^\star) \geq -\lambda \cdot (v(a^\star) - v(a(P_n)))$
        \item If the buy condition is met then, $s^\star$ is added to the set of controlled datasets: $P_{n+1} = P_n \cup d(s^\star)$ and the next round starts
        \item Else if the buy condition is not met, then the process stops
    \end{itemize}
\end{enumerate}

As a result, if the marketplace is asked to compute the marginal accuracy for every remaining dataset every round, the model will be processed a maximum if $\sum^{r-1}_{i=0}{|S|-i}$ times for $r$ rounds. To prevent abuses from buyers, a marketplace implementing this solution could set up a maximum limit of trials for a certain task. Such a limit may be updated as the buyer purchases data.

\subsection{Buying without trying}
\label{subsec:BuyingWithoutTrying}
\subsubsection{Volume-based purchasing}
\label{subsec:volumePurchasing}
Most commercial marketplaces provide buyers with a description of datasets, their metadata, source, procedure used to collect them, etc. Oftentimes, the volume of data in a particular dataset (e.g. nº observations or samples) is used as the deciding figure of merit for choosing among different offers. Let $vol(s)$ denote the volume of dataset $s$, used as merit figure by the following volume-based purchasing heuristic:

\paragraph{n-th iteration}We will assume that a greedy buyer would select the dataset $s^\star \in S_n$ with the highest $vol(s) / p(s)$ ratio every round. However, it is not possible to know the accuracy it will yield to the specific problem. We assume a conservative condition for the algorithm to decide to purchase $s^\star$, specifically:

\begin{equation}
    p(s^*) \leq -\lambda \cdot (v(a^\star) - v_n),
\end{equation}

which assumes that even in the worst case, where the purchase does not improve $(\mathcal{M},a)$ accuracy at all, the maximum relative admissible loss is not exceeded in the operation.

\subsubsection{Price-based purchasing}
\label{subsec:PriceBasedPurchasing}
It may happen that the marketplace just publishes the list of suitable datasets $\mathcal{S}$, and their prices. This setting resembles real situations where information about data offer is insufficient or misleading to the buyer's purposes. We assume such a buyer would randomly select among datasets whose price is lower than their maximum relative admissible loss.

\paragraph{n-th iteration}The buyer will randomly select one of the datasets $S_n \subseteq \mathcal{S}$ such that, $\forall s \in S_n, p(s) \leq - \lambda \cdot (v(a^\star) - v_n)$. If $S_n = \emptyset$ then the process stops.

\section{Performance evaluation with synthetic data}
\label{sect:TheoreticalEvaluation}
We will use synthetic data to evaluate the performance of the different purchase strategies of Section \ref{sect:Purchase algorithms} across a wide range of parameters. Our synthetic model is easy to reproduce, captures a wide range of parameters, and allows us to extract useful insights about the relative performance of different data purchase strategies. As we will show later in Sect.~\ref{sect:ValidationWithData}, our conclusions from this section are also validated by results with real data.

\subsection{Synthetic model description}
\label{subsec:TheoreticalModel}

To simplify the evaluation, we will assume that the value for the data buyer will be equal to the accuracy $a$. Hence, the maximum value that a buyer can extract from data is equal to 1, which occurs when the accuracy of its AI/ML algorithm trained on the purchased data, becomes 1 (i.e., 100\%). We will denote as Total Cost of Data (TCOD), the cost of buying all the available datasets in $\mathcal{S}$, i.e., $TCOD=\sum_{s\in \mathcal{S}} p(s)$. Therefore, when $TCOD<1$, then the buyer is guaranteed to make a profit, independently of the data purchase strategy used. In the more interesting case of $TCOD\geq 1$, the buyer needs to select which datasets to buy carefully, to avoid ending up with a loss.

Having equated value with accuracy, we will also need to connect the datasets bought, with the achieved accuracy. For a data buyer that buys datasets $S\subseteq \mathcal{S}$, the value will be given by the following expression:

\begin{equation}
    v(a(d(S))) = a(d(S)) = \left( \frac{\sum_{s_i \in S}{DI^{i}}} {\sum_{s_i \in \mathcal{S}}{DI^{i}}} \right) ^{MUP},
\end{equation}

where:
\begin{itemize}
    \item \textbf{MUP is the Marginal Utility Profile parameter} that controls the concaveness/convexity of $v$ from 0 to 1 as more data is bought. When $MUP<1$, then buying additional datasets will have a decreasing marginal utility in terms of accuracy, and hence value for the buyer, both of which will be concave with respect to the amount of data bought. On the other hand, with $MUP>1$, the marginal contribution of new data sources will be increasing as more datasets are bought, making $v(\cdot)$ and $a(\cdot)$ convex. Finally, $MUP=1$ means that all datasets yield the same accuracy if they are bought first, and the same incremental change if they are bought second, and so forth.
    \item \textbf{DI is the Data Interchangeability parameter} that controls the relative importance of different datasets in $\mathcal{S}$. Setting DI equal to 1, amounts to making all datasets fully interchangeable. Therefore, in this case, it only matters how many datasets are bought, but not which ones. For $DI>1$ and $MUP=1$, dataset $s_i$ becomes DI times more important than dataset $s_{i-1}$, $1\leq i\leq |\mathcal{S}|$. Effectively, for $DI\neq 1$ what matters is not only how many datasets are bought, but also which ones.
\end{itemize}

The last element of our synthetic model has to do with how we set the prices of individual datasets. We will consider the following pricing schemes:

\begin{itemize}
    \item \textbf{All datasets having the same price}, i.e., $p(s)=$TCOD$/|\mathcal{S}|$, $\forall s\in \mathcal{S}$.
    \item \textbf{Datasets having random prices} drawn from a uniform distribution in $[0,1]$ and scaled to add up to TCOD.
    \item \textbf{Datasets having a price that reflects their importance} captured by their Shapley value within a coalition of $\mathcal{S}$ datasets that achieve a total value equal to $a(d(\mathcal{S}))$ (see works such as \cite{Ghorbani19, Paraschiv19} for a justification and explanation about hot to use the Shapley value with aggregate datasets).
\end{itemize}

\subsection{Results for different marginal utility profiles}
\label{subsec:TheorSensitivityMUP}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{MUPSensitivityRandomPrice.jpg}
\caption{Profit (\% of the optimal) vs. TCOD for different MUP and value-unrelated prices}
\label{Fig:MUPSensitivity Random Price}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{SamplePurchasingSequencesTheoretical.jpg}
\caption{Purchase sequences for MUP = 1 and TCOD = 1.5 (left), and 3 (right)}
\label{Fig:PurchasingSequencesTheoretical}
\end{figure}

Next we will compare optimal purchasing under complete information, with TBYB, and the value-agnostic heuristics across a range of parameter values for capturing things like the total cost of datasets (TCOD), the marginal utility of buying more datasets (MUP), their relative value (DI), and their relative price. Our main evaluation metric will be the profit for a data buyer, i.e., the value extracted from the data minus the cost paid to obtain them. As stated in the introduction, guaranteeing that buyers obtain a healthy profit from data is vital for bootstrapping the nascent data marketplace sector. In the future we will also examine seller side profits and social welfare. Of course doing the latter makes sense for already bootstrapped markets. It also requires modeling complex market dynamics such as competition, dynamic pricing, etc. that go beyond the scope of the current work. Whenever randomization is used, e.g., in pricing, or in some of the value-agnostic heuristics, we report average values from 50 executions.

The first parameters that we examine are TCOD and MUP, assuming some datasets are more important than others ($DI = 2$). Obviously, as data become more expensive (higher TCOD), all the strategies, including the optimal one, yield a smaller profit for buyers. What we are interested to study, therefore, is the relative performance of different strategies under different TCODs and MUPs. Figure \ref{Fig:MUPSensitivity Random Price} shows that A-TBYB matches the optimal purchasing for both concave (MUP=0.5, left subplot) and linear (MUP=1, middle subplot) value profiles, across the entire range of TCOD values. For convex value profiles (MUP=3, right subplot), A-TBYB ceases to be optimal, but remains the best performing strategy.

What is even more interesting, is the performance of the much simpler to implement S-TBYB, which stays above 90\% of the optimal for concave and linear MUPs, when the value agnostic heuristics drops below 50\% with TCOD>1 and even leads to losses (see MUP=1 results). Under convex value profiles (MUP=3), all strategies yield a lower performance, since reaching a higher accuracy (and therefore buyer value) requires buying more datasets, which, in turn, eats away the profit margins for buyers. Even in these cases, S-TBYB yields a profit and avoids loss.

To explain why TBYB outperforms the value unaware heuristics, we plot in Fig.~\ref{Fig:PurchasingSequencesTheoretical} a series of ``\textit{purchase sequences}'', demonstrating the evolution of profit with the number of datasets purchased (by different algorithms). As shown in the plot, TBYB algorithms buy both the most valuable datasets (they achieve higher profits from the first round), and the right number of them (they stop buying before profits decrease). On the other hand, value unaware heuristics overbuy, and randomly select datasets they can afford according to their risk appetite, which generally lead to lower profits, or even loses especially for risk-prone buyers.

\subsection{The effect of data interchangeability}
\label{subsec:TheorSensitivityDI}
To find out how TBYB is affected by the interchangeability of datasets, we have run a set of simulations for different values of the parameter DI. Figure~\ref{Fig:DISensitivity} shows three different plots of the relative profit of different purchase algorithms for different DI values under MUP=1. The subplot on the left depicts results for perfectly interchangeable datasets (DI = 1), whereas the next two show cases of datasets that are increasingly less interchangeable (DI=2 and DI=3).

\begin{figure}
\centering
\includegraphics[width=\textwidth]{DISensitivity.jpg}
\caption{Profit for purchase algorithms (MUP = 1, linear) with different DI values}
\label{Fig:DISensitivity}
\end{figure}

These plots show that the performance benefits of TBYB over the heuristics increase when different datasets have different value in terms of the accuracy they can achieve, both alone, as well as combined with other datasets. This happens, of course, because the advantage of knowing the value of data before buying them, gets diminished when datasets are almost interchangeable. In reality, as we will show in the next section, real world datasets are not interchangeable, which means that value unaware heuristics will not be able to match the performance of TBYB.

\subsection{The effect of data pricing}
\label{subsec:TheorSensitivityPricing}
In this section, we look at the role of dataset pricing on the performance of TBYB. Our main interest is to see what happens when the price of a dataset is proportional to its value for an AI/ML algorithm, and when it is not. The former we create via the Shapley value method discussed in Sect.~\ref{subsec:TheoreticalModel}. The latter we model it in two ways: with datasets that have the same price but yield different accuracy, and with datasets that have randomly distributed prices and different accuracy.

Figure \ref{Fig:PricingSensitivity} shows the results of our purchase algorithms for different pricing models. In every case, A-TBYB matches the optimal. Pricing data based on their real value for AI/ML algorithms reduces the gap of S-TBYB vs. price-based purchasing, although S-TBYB still outperforms price-based purchasing for the same level of risk. Notice, however, that pricing data in accordance to their actual value for buyers, requires knowing the value function of each buyer, something that buyers, of course, have no incentive to disclose to sellers. Even if they did, different buyers may have different value functions, so, in general, it cannot be expected that the price of a dataset will follow its value for different buyers that may be using it with different AI/ML algorithms, and having different value functions.\footnote{Notice that to simplify our synthetic evaluation we have assumed that buyer value follows the accuracy achieved by each dataset. This, of course, need not apply in the real world, since different buyers may have radically different value functions that translate accuracy into monetary worth.}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{PricingSensivity.jpg}
\caption{Profit for different pricing methodologies (MUP = 1, DI = 2)}
\label{Fig:PricingSensitivity}
\end{figure}

\subsection{Summary}
\label{subsec:TheorConclusions}
Table \ref{tab:ParametersImpact} summarizes the impact of parameters in the performance gap between TBYB and price-based purchasing. In summary, TBYB, even in its simplest, stand-alone version, always outperforms the value unaware heuristics, especially in the most realistic scenarios involving high TCOD, concave value functions, non-interchangeable datasets, and pricing that does not follow value. In a good part of the parameter space TBYB is very close to the performance of optimal purchasing that uses full information.

\begin{table}
  \caption{Impact of parameters on the gap between TBYB and price-based purchasing}
  \label{tab:ParametersImpact}
  \scriptsize
  \begin{tabular}{p{1.2cm}p{5cm}p{6cm}}
    \hline
    Parameter&Impact&Explanation\\
    \hline
    \texttt{TCOD}&The higher TCOD, the more valuable TBYB&More difficult for other strategies to find the right datasets in terms of price - value to buy \\
    \texttt{MUP}&The higher MUP, the more difficult to find the optimal. TBYB loses effectiveness but still outperforms other algorithms&TBYB buys more valuable datasets, minimizes temporary losses and limits risk for buyers, since it allows for a better estimation of expected marginal value of datasets\\
    \texttt{DI}&The less interchangeable datasets are, the more advantage of using TBYB&With perfectly interchangeable datasets, TBYB only improves the estimation of marginal utility as information increases \\
    \texttt{Pricing}&TBYB gap with price-based purchasing narrows when prices are not related to value&Price-based purchasing works better if value is embedded in price\\
    \hline
  \end{tabular}
\end{table}

\section{Validation with real data}
\label{sect:ValidationWithData}

The synthetic model of the previous section limits the ways in which two or more datasets may mix, and impact the accuracy of an AI/ML algorithm. It allows only for concave/convex mixing with equal (interchangeable, DI=1) or unequal (non-interchangeable, DI>1) contributions to accuracy from the different datasets. In reality, however, different datasets may mix in much more complex ways, that cannot be represented by any parameter setting of the above model. For example, a certain dataset $d_i$ can be very useful if combined with another dataset $d_j$, but not so useful if combined with others that individually yield the same accuracy as $d_j$ does. To verify our conclusions from the previous section, we tested the performance of different data purchase strategies using real spatio-temporal data, in a use case that involves forecasting demand for taxi rides in a city.

Furthermore, in this section we expand our performance evaluation by introducing a new data pricing scheme, and a new data purchase strategy:
\begin{enumerate}
    \item \textbf{Volume-based pricing}. In this case the price of a dataset becomes proportional to its volume. In our use case, volume will correspond to the number of  drivers in the company.
    \item \textbf{Volume-based purchasing}. We will test the performance of a new heuristic that seeks to purchase that largest possible dataset in terms of volume for a given price.
\end{enumerate}

According to an internal survey covering more than 75 companies in the data economy, pricing and purchasing data by volume is a commonly extended practice in data trading. Therefore we compare TBYB against those practices as well.

\subsection{Use case description}
\label{subsec:ValidationUseCaseDescription}
We will assume that a data buyer is looking to purchase datasets for training a multiseasonal SARIMA forecasting model with the purpose of forecasting, at an  hourly timescale, the demand for taxi rides in different districts of Chicago City, for the weeks to come. To achieve this objective, a number of taxi companies will be assumed to be the data sellers, that release historical data on past trips that they have provided in the \textit{observation period} or $T_o$. Such datasets are publicly available, thanks to reporting obligations that such companies have to fulfil towards the local authorities \cite{TaxiTrips19}. The accuracy of the forecasting algorithm is quantified by how accurate the algorithm can predict real demand observed in a \textit{control period} or $T_c$. Our model is able to accommodate any sequence similarity metric in order to compare predicted vs. real demand in $T_c$.

\subsection{Dataset description}
\label{subsec:ValidationDatasetDescription}
From the above-mentioned repository \cite{TaxiTrips19}, we have obtained 11.1 MM rides corresponding to the first 8 months of 2019. These rides are included in 15 datasets that correspond to the 15 largest taxi companies in the city (servicing 94\% of the total demand), plus a hypothetical 16th company that aggregates all the rides reported by the remaining smaller companies. These will be our 16 data sellers according to our problem formulation.

We computed the exact Shapley value of data from each company to the forecasting accuracy achieved by the multiseasonal SARIMA model in predicting the demand in the second half of April using taxi rides from the previous six weeks for training ($T_o = $ Mar. 4th - Apr. 14th and $T_c = $ Apr. 15th - 28th). We deliberately chose to predict the taxi demand of a medium size district of the city (community area 11, Jefferson Park), where data from several companies is needed in order to achieve a good prediction accuracy. As a result, the Shapley values are very different for each source (standard deviation = 76\% of the average). Moreover, these were found to be weakly correlated with the number of licenses of each company ($R^2 = 0.54397$), because big companies usually concentrate in other areas of the city.

Unlike what happened in the theoretical use case, the maximum accuracy the marketplace can give using all the information is $a^\star = 0.896294$ in this case. As in the synthetic case, we will assume that the economic value of a prediction is equal to its accuracy.

\subsection{Empirical results}
\label{subsec:ValidationResults}
We have simulated all purchase algorithms for different TCOD, pricing models, and $\lambda$ parameters. Figure~\ref{Fig:ResultsChicagoD11} (a) shows that both A-TBYB and S-TBYB achieve above 90\% of the optimal buyer's profit under value-unrelated dataset pricing. The results are in line with the ones we obtained using synthetic data.

Regarding volume-based purchasing, it proved to outperform price-based purchasing, but only when TCOD is low (< 5). This is because value and volume are not tightly correlated in this case, hence buying by volume does not lead necessarily to higher accuracy.

Looking at Fig.~\ref{Fig:ResultsChicagoD11} (b) we see the corresponding results under volume-based prices. In this case, profit reduces faster than in the case of value-unrelated prices as TCOD grows, since valuable datasets are set higher prices. Still TBYB outperforms buying without trying algorithms, since it selects cheaper and more valuable datasets.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{D11Results.jpg}
\caption{Profit for different purchase strategies when prices are (a) unrelated to value, and (b) related to volume}
\label{Fig:ResultsChicagoD11}
\end{figure}

Figure \ref{Fig:PurchasingSequencesD11} shows an average purchase sequence to understand why TBYB works for volume-related prices. TBYB improves price-based purchasing both by selecting the best datasets, and stopping the purchase process before profit gets diminished. This feature is especially relevant when the offer is wide in comparison to the value they provide (TCOD $>> 1$). Picking datasets based on volume did not improve price-based purchasing in this specific example.

\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{D11Sequence2.jpg}
\caption{Sample purchase sequence for volume-based pricing and TCOD = 3.3}
\label{Fig:PurchasingSequencesD11}
\end{figure}

\section{Related work}
\label{sect:RelatedWork}
Different efforts have been done by the research community in order to identify the challenges of existing data marketplaces and define new business models \cite{Fernandez20}. In particular, some ML/AI oriented data marketplace proposals are trying to mechanize in a marketplace the process that some niche data an AI service providers are already bringing to the market. Most AI/ML-oriented theoretical data marketplace platforms leverage a data valuation framework similar to what we propose in this paper \cite{Agarwal19,Chen19}. In general, such marketplaces train buyers’ models in a neutral platform by feeding them with their data, and ask for a price depending on the accuracy (and, thus the value) they provide. They even suggest that marketplaces should return a trained algorithm instead of bulk data. The more you pay, the higher the accuracy you get in any case.

To the best of our knowledge, there is no practical nor commercial implementation of those designs yet, although some digital service providers (SigOpt, Comet.ml), provide algorithm optimization services.

Furthermore, some researchers have studied the dynamics of a data marketplace \cite{Moor19}, and proposed mechanisms that prevent sellers and buyers to postpone their arrival to the marketplace or misreport their costs or values. However, lots of challenges and research issues are still open around data trading and pricing \cite{Pei20, Yang19, Shen16}.

As data provided to buyers usually benefits from combining different sources, it becomes very relevant the problem of how to fairly split the payment for a transaction among all the sources that contributed to the traded data. Existing marketplaces usually calculate this through simple heuristics, such as the data volume or the number of sources involved. However, simple heuristics are not necessarily tied to the utility of data, and consequently could be considered unfair by sellers. To address this challenge, researchers resort to well-known concepts of game theory to split the revenue. Most papers  propose using the Shapley value \cite{Shapley52} for such a task \cite{Agarwal19, Ghorbani19, Paraschiv19}, whereas others propose to use the core \cite{Yan20}. We have used the Shapley value and its approximation algorithms (see \cite{Jia19_2, Castro09, Ghorbani19}) to price datasets according to their value.

\section{Conclusion and future work}
\label{Conclusion}
TBYB was shown to provide near optimal data buyers' profits under a wide range of parameters and data. Used with off-the-shelf AI/ML algorithms, or more complex ones in a sandboxed ``try before you buy'' infrastructure, can make TBYB a practical, high performance alternative to value-unaware purchasing, which is currently the normal in real-world DMs. Helping buyers achieve higher profits would thus help in bootstrapping and growing the currently nascent data marketplace economy.

We are currently working on developing fully functional prototype of TBYB to be used by real users. This will enable us to test arbitrary user-provided algorithms and models. Of course, building a functional DM goes beyond the scope of this paper. It involves additional aspects not covered here such as dynamic data pricing, protection against arbitrage, price discrimination, and lots of engineering, scalability, and security challenges that will be the focus of forthcoming works.


\section{Introduction}

M87 is a supergiant elliptical galaxy with an active  nucleus (AGN). It is classified as a radio galaxy (RDG) of type Fanaroff-Riley I (FR-I) and is located in the Virgo Cluster at a distance of 16.7 $\pm$ 0.2 Mpc \cite{mei2007}. Its prominent jet was first discovered in 1918 \cite{curtis1918} and has been studied in multiple wavelengths and scales \cite{blandford2019}.  The Event Horizon Telescope (EHT) observed the supermassive black hole (SMBH)  M87*, which engines the AGN of M87. producing an image of its shadow \cite{ehtI2019}. This result was used to constrain the black hole mass  \cite{ehtVI2019}, spin and recently, the magnetic field structure near the event horizon \cite{ehtVIII2021}. \\

M87 is a well-established MeV, GeV and TeV gamma-ray source \cite{abdo2009,magic2020}. It was the first RDG detected in the TeV gamma-ray range and some TeV flares have been observed \cite{aharonian2006,albert2008,abramowski2012}. In non-flaring state, this source has also shown complicated gamma-ray spectral and flux variability \cite{benkhali2019}. The zone where this emission is produced is not well determined, being the inner jet or core its most likely origin \cite{abramowski2012,benkhali2019}. Other candidates are the jet feature HST-I and the SMBH vicinity \cite{abramowski2012,benkhali2019}. The physical mechanism that produces this emission is not known either. It is commonly accepted that an one-zone synchrotron self Compton (SSC) scenario is not enough to explain this emission i.e, \citep{fraija2016,georganopoulos2005}. This is supported by the evidence of a spectral turnover at energies of $\sim 10$ GeV, which could be produced by the transition between  two different emission processes \cite{benkhali2019}. One of the proposed alternatives to explain this emission are the lepto-hadronic scenarios, which explain the  spectral energy distribution (SED) combining leptonic models with photo-hadronic interactions \cite{fraija2016,sahu2015}.\\

The High Energy Water Cherenkov (HAWC) gamma-ray observatory marginally detected this source at $E>0.5$ TeV \cite{albert2021}. This long-term 1523 days observation represents a good constraint of the average TeV emission of M87. In this work we fit a lepto-hadronic model to a SED built to  include the HAWC observations for the first time. \\

\section{Data}

An average SED of M87 was constructed using historical archive data \cite{morabito1986,morabito1988,junor1995,lee2008,lonsdale1998,doeleman2012,biretta1991,perlman2001,sparks1996,marshall2002,wong2017,abdo2009}. We also included \textit{Fermi}-LAT observations from the 4FGL catalog \cite{abdollahi2020}. As it was mentioned above, HAWC data from \cite{albert2021} were used to consider the VHE emission. These observations were already corrected by an Extra-galactic background light (EBL) absorption model \cite{dominguez2011}. Therefore, we do not consider this effect during the fitting process.

\section{Model and Methodology}
The model that we use postulates an electron population contained in spherical region in the inner jet \cite{finke2008}. The spectral electron distribution (as a function  of $\gamma^\prime$, the  electron comoving Lorentz factor) is a broken  power law given by Equation \ref{eq:Ne} :

\begin{equation}
    N_e(\gamma^\prime)\propto \begin{cases} \gamma^{\prime^{-p_1}} \text{for } \gamma^\prime<\gamma_c^\prime \\
     \gamma^{\prime^{-p_2}} \text{for } \gamma^\prime>\gamma_c^\prime

    \end{cases},
    \label{eq:Ne}
\end{equation}

where $p_1$,$p_2$ correspond to the power law indices, and $\gamma^\prime_c$ to the break Lorentz factor, which is one of the fitting  parameters of the model. Other two model parameters are B, the magnetic field intensity and $D$, the Doppler factor of the emission zone. The one-zone SSC scenario explains the SED range from radio to X-rays as synchrotron emission produced by electrons moving in  the magnetic field. A second energy component, from X-rays to gamma rays, is produced when  electrons Compton scatter synchrotron photons \cite{finke2008}. \\

The proton population in the emission zone is assumed to have a single power law spectral distribution (Equation \ref{eq:Np}):

\begin{equation}
\label{eq:Np}
N_p(\gamma_p^\prime) \propto \gamma_p^{\prime-\alpha},
\end{equation}

where $\alpha_p$ is the proton spectral index, which is a fitting parameter of the model. This scenario postulates that gamma-ray emission is produced in particle cascades generated by interaction between SSC  photons and accelerated protons \cite{sahu2019}. The other fitting parameter of this model is a normalization constant $A_\gamma$ \cite{sahu2019}. \\

The methodology consisted of developing a Python code to fit the emission model to the SED. First, the one-zone SSC model \cite{finke2008} was fitted to the data between radio and MeV-GeV gamma rays. Then, the photo-hadronic \cite{sahu2019} component was added to fit the VHE emission.  The best fit values of the model parameters were obtained with chi-square minimization and errors were estimated with Monte Carlo simulations. \\

\section{Results and Discussion}

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{plotSSC_ICRC.png}
    \caption{Spectral Energy Distribution (SED) of the RDG M87 with the SSC model fit. Archive data points are shown black. The  violet curve corresponds to the synchrotron component and  the blue curve to the inverse Compton one.  The red curve corresponds to the total emission. The HAWC 1$\sigma$ error band is shown in light blue.  }
    \label{fig:SSC}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{plotHAWC_ICRC.png}
    \caption{Spectral Energy Distribution (SED) of the RDG M87, including HAWC data, with the lepto-hadronic model fit.  Archive data points are shown black.  The  violet curve corresponds to the synchrotron component,  the blue curve to the inverse Compton component and the green curve to photo-hadronic one. The red curve corresponds to the total emission. The HAWC 1$\sigma$ error band is shown in light blue. }
    \label{fig:ph}
\end{figure}

\begin{table}[]
    \centering
    \begin{tabular}{c c c} \hline
           Parameter& & Value \\ \hline
          Magnetic Field intensity (mG) & $B$&$ 46 \pm 3$  \\
  Doppler Factor & $D$ & $ 4.3\pm 0.2$\\ \hline
  \textbf{Electron spectral parameters} \\ \hline
  Broken  PL index & $p_1$ &$1.52 ^{+0.02}_{-0.01} $\\
 Broken PL index & $p_2$ & $ 3.53 \pm 0.02 $\\
    Break Lorentz factor ($\times 10^3$) & $\gamma_c^\prime$ &$ 3.80 ^{+0.07}_{-0.05} $   \\ \hline
   \textbf{Photo-hadronic Component} \\ \hline
    Proton spectral index & $\alpha$ & $3.0 \pm 0.2 $ \\
    Normalization & $\log(A_\gamma)$ & $-0.5\pm 0.2  $ \\ \hline
    $\chi^2_\nu$(d.o.f) & & 25.8 (22) \\ \hline
    \end{tabular}
    \caption{Best fit values of the fitting parameters}
    \label{tab:SSC}
\end{table}

The best fit model for the SSC scenario is shown in Figure \ref{fig:SSC}. At first glance, it seems that the VHE spectral region shows a spectral hardening which cannot be explained with the SSC components. This agrees with the spectral turnover  reported by \cite{benkhali2019},  which would explain that the VHE SSC flux prediction is well below the HAWC observed spectrum.\\

The best fit model for the entire lepto-hadronic scenario is shown in Figure \ref{fig:ph}. As can be seen, this component is necessary to fit both  GeV and TeV  gamma-ray observations. A slight spectral hardening is seen at $E\sim10$ GeV, which agrees with the results obtained in \cite{benkhali2019}. The fit results are shown in Table \ref{tab:SSC}. \\

A similar spectral hardening is also observed in the gamma-ray emitter RDG Centaurus A  \cite{sahakyan2013}.  The existence of an additional emission component in TeV emitter RDGs could be expected due to their low Doppler factors ($D \lesssim 10$)\cite{rieger2018}, compared to those of blazars whose gamma-ray emission is enhanced by their substantial  Doppler-boosting.  Moreover, the possible detection of neutrino emission from blazars suggests that similar mechanisms may be present in other types of AGNs  \cite{aartsen2018}. \\

The goal of this  work is explaining the VHE emission of M87, but specific features in the rest of the bands need a more detailed analysis \cite{algaba2021}.  Multi-zone models are probably needed, as well as some specific data sets. In case of the mm bands, the recent EHT results could be very helpful.\\

HAWC data correspond to the first long-term continuous TeV observation of this source. Previous air Cherenkov campaigns constrained very well the average TeV emission, however they could be affected by short-term spectral and flux variations. For example, in the MAGIC two-year campaign reported in \cite{acciari2020} no evidence of the gamma-ray spectral turnover was found. However, this campaign was coincident with a high activity period reported by \cite{benkhali2019}, where the spectral turnover seems to have temporarily disappeared. A more detailed analysis would be needed to know if these short term variations could be explained with the photo-hadronic scenario.\\

HAWC is still continuously taken data, which can be used in a near future to improve the results of this analysis, as well as to test other physical scenarios.

\subsection{Summary and conclusions}

M87 is a giant RDG that emits in gamma rays up to TeV bands. The physical mechanism that produces the VHE emission has yet to be determined. The lepto-hadronic scenario explains this emission with photo-hadronic interactions, whereas the rest of the broadband  SED is attributed to a leptonic mechanism. The HAWC Collaboration recently reported a marginal detection of this source that can be used to constrain its average VHE emission, which can be used to test possible physical scenarios. In this work we fit a lepto-hadronic model to a  SED which includes the results from HAWC.  We obtained best  values for the fitting parameters, which are the mean magnetic field intensity ($B=46\pm3$ mG), the Doppler factor ($D=4.3\pm 0.2$), the electron spectral parameters ($p_1=1.52^{+0.02}_{-0.01},p_2=3.53\pm0.02,\gamma_c^\prime=3.80^{+0.07}_{-0.05} \times 10^{3}$) and the photo-hadronic parameters ($\alpha=3.0\pm0.2$, $\log(A_\gamma)=-0.5\pm0.2$). We concluded that this scenario could explain the M87 VHE emission, including some spectral features like a possible turnover at $\sim 10$ GeV.

 \acknowledgments
 We acknowledge the support from: the US National Science Foundation (NSF); the US Department of Energy Office of High-Energy Physics; the Laboratory Directed Research and Development (LDRD) program of Los Alamos National Laboratory; Consejo Nacional de Ciencia y Tecnolog\'ia (CONACyT), M\'exico, grants 271051, 232656, 260378, 179588, 254964, 258865, 243290, 132197, A1-S-46288, A1-S-22784, c\'atedras 873, 1563, 341, 323, Red HAWC, M\'exico; DGAPA-UNAM grants IG101320, IN111716-3, IN111419, IA102019, IN110621, IN110521; VIEP-BUAP; PIFI 2012, 2013, PROFOCIE 2014, 2015; the University of Wisconsin Alumni Research Foundation; the Institute of Geophysics, Planetary Physics, and Signatures at Los Alamos National Laboratory; Polish Science Centre grant, DEC-2017/27/B/ST9/02272; Coordinaci\'on de la Investigaci\'on Cient\'ifica de la Universidad Michoacana; Royal Society - Newton Advanced Fellowship 180385; Generalitat Valenciana, grant CIDEGENT/2018/034; Chulalongkorn University’s CUniverse (CUAASC) grant; Coordinaci\'on General Acad\'emica e Innovaci\'on (CGAI-UdeG), PRODEP-SEP UDG-CA-499; Institute of Cosmic Ray Research (ICRR), University of Tokyo, H.F. acknowledges support by NASA under award number 80GSFC21M0002. We also acknowledge the significant contributions over many years of Stefan Westerhoff, Gaurang Yodh and Arnulfo Zepeda Dominguez, all deceased members of the HAWC collaboration. Thanks to Scott Delay, Luciano D\'iaz and Eduardo Murrieta for technical support.