\section{PanJoin} \label{sec:pan}
\subsection{Architecture}  \label{sec:archtect}
PanJoin supports a sliding window theta join over two data streams (Stream $S$ and Stream $R$ in the following). PanJoin processes incoming data at two levels of parallelism: the \textit{architecture level}, or node level, where multiple worker nodes generate partial results in parallel, and \textit{thread level}, where each worker node executes with multiple threads.

\begin{figure}[!htb]
    \centering
    \begin{minipage}[b]{.40\textwidth}
  \centering
  \includegraphics[width=\textwidth]{diagrams/Sigmod2019_-_sys_arch.pdf}
  \caption{System architecture.}
  \label{fig:sys_arch}
    \end{minipage}\hfill%
    \begin{minipage}[b]{0.40\textwidth}
  \centering
  \includegraphics[width=\textwidth]{diagrams/Sigmod2019_-_sys_process.pdf}
  \caption{Process procedure. The buffer of Stream $S$ (blue) is full and is processed.}
  \label{fig:sys_process}
    \end{minipage} \hfill%
\end{figure}

The architecture of PanJoin is shown in Figure \ref{fig:sys_arch}. There are several worker nodes and one manager node. Each worker node has some number of subwindows. The subwindows of the same stream are chronologically chained and allocated in a round robin way among the worker nodes, and there is no communication between worker nodes, which is different from (Low Latency) Handshake Join \cite{teubner2011soccer, roy2014low}. The manager node manages the topology and the locations of all subwindows. The manager node also preprocesses incoming raw data and distributes the preprocessed data together with processing commands to the worker nodes. The processing commands includes: \textsl{create} subwindow, \textsl{insert} tuples, \textsl{probe} subwindow, and \textsl{expire} subwindow.
To generate correct commands, the manager node needs to collect the running status from every worker node. This information is sent together with the result data from the worker nodes to the manager node with very low overhead (several bits). The two most important messages are whether the oldest subwindow is empty and whether the newest subwindow is full. Additionally, because the manager node needs to distribute data to worker nodes, to improve network and processor utilization, PanJoin can process more than one tuple packed as a batch simultaneously, which is referred to as \textit{batch mode}.

The high-level algorithm of PanJoin implements a five-step procedure, as demonstrated in Figure \ref{fig:sys_process}. Each of the steps is defined as follows:

\begin{enumerate}[label=Step \arabic*]

\item \textit{Collecting}: the manager node collects tuples and places them into two independent buffers, one for each stream.

\item \textit{Preprocessing}: The manager node retrieves the joining field from the incoming tuples and other fields that are necessary for the theta join. In batch mode, the manager node sorts the tuples by the values of the joining field. Then, the manager node decides whether to create a new subwindow if the current newest subwindow is full, as well as whether the oldest subwindows have to expire tuples. Subsequently, the manager node generates processing commands and packs the commands with the incoming tuples into messages.

\item \textit{Sending}: The manager node sends the messages to the worker nodes and ensures that the \textsl{probe} commands are sent to all the worker nodes with nonempty subwindow(s) of the opposite stream. In Figure \ref{fig:sys_process}, when the system is processing tuples from Stream $S$, the second worker node does not have a nonempty subwindow of stream $R$. Thus, it has no need to perform probing, and it will not receive a \textsl{probe} command. The message sent to the newest subwindow has an \textsl{insert} command, and the message sent to the oldest subwindow has an \textsl{expire} command. If necessary, the manager node sends a node with a \textsl{create} command to create a new subwindow.

\item \textit{Processing}: The worker nodes receive messages and perform the processing commands.

\item \textit{Feedback}: The worker nodes send the probing result (optional, depending on the topology of the processing nodes, e.g., worker nodes may directly forward their result to the nodes that process the ``select'' operation) and their running status back to the manager node.

\end{enumerate}

Note that the manager node can perform Steps 1 and 2 in parallel with Steps 4 and 5 when the worker nodes are processing the data. We can also add one or several prefilter nodes ahead of the manager node to retrieve the joining field from the incoming tuples. In our preliminary experiment, the manager node can process more than 300M tuples per second, and the main bottleneck is in Step 4 in the worker nodes and network communication between the manager and worker nodes. To accelerate Step 4, we introduce three data structures to manage tuples inside a subwindow: \textit{Range Partition Table} (RaP-Table), \textit{Wide B$^+$-Tree} (WiB$^+$-Tree), and \textit{Buffered Indexed Sort} (BI-Sort). RaP-Table and WiB$^+$-Tree perform well when batch size and selectivity is small, while BI-Sort runs faster when batch size or selectivity is large. WiB$^+$-Tree is slower but more powerful than RaP-Table because it can handle increasing values. Users can choose one of these data structures for a subwindow, where the chosen data structure serves as an ``index'' and further divides the subwindow into partitions. Therefore, in the probing step, rather than scanning the whole window, the worker node only needs to check the tuples in a limited number of partitions, which significantly reduces the number of comparisons and improves the system throughput by several orders of magnitude. The following subsections provide detailed discussions about these data structures.

\begin{comment}

PanJoin supports sliding window theta join over data streams. PanJoin handles the three major challenges of stream join processing by range partitioning the tuples in the sliding window of each stream. Using range partitioning brings several advantages. First, it naturally accelerates non-equi-join. By comparing the boundaries of join conditions, it is easy to select the partitions that store the possible tuples that may satisfy the join condition. Instead of scanning the whole window, we only need to check the tuples in the selected partitions, which significantly reduces the amount of comparisons. Second, it also accelerates equi-join. We can consider range partitioning as a special form of hash partitioning. Third, when the partition is smaller than a threshold, the overhead of preparing the data and post-comparison processing will exceed the time spent on the actual comparison. In other words, the necessary number of partitions is limited to a certain value. This value depends on many factors, such as the input parsing time, the partitioning time, the memory access delay, tuple storage time, and other overheads during the join processing. This feature enables us to design a joiner with limited resource requirement, as the size of the partition table is limited. It also allows us to deploy our joiner to a heterogeneous computing platform, e.g., an FPGA.

The main difficulty to use range partitioning is the adaptiveness of the range partitioner to data skew. Because our target system processes real-time data, there is strong possibility that the distribution of data values varies frequently and unpredictably. Example situation can be breaking news on Twitter, a new buzzword spreading across the Internet, or some repetitive data generated by random errors in a sensor network. Those events can result in unbalanced partitioning outcomes, i.e., some partitions receive more tuples than average, which can significantly reduce the performance since it takes more time to process the tuples joining with those unbalanced partitions. It also brings about difficulties in storage; because the size of every partition is not predictable, we need to reserve more space for each partition than the average partition size. In the worst case, the size of the reserved storage for each partition can equal the window size, which is not acceptable for a system with a large number of partitions.

To handle the skewed data, PanJoin samples the incoming tuples frequently. The time interval between two sampling actions can be configured to be shorter than the period of filling the entire window with incoming data. During each sampling action, PanJoin analyzes the histograms that are generated during this interval and calculate a new partition table for the next interval. In this way, we utilize a critical feature of stream processing that has been overlooked: the tuples in a small time interval share the same information on the value distribution with many of their successors. In practice, we can calculate an approximate variance from the histograms before generating the partition table. When the data distribution changes, the variance is above a predefined threshold, and we compute the new partition table. When the distribution stays unchanged, we can retrieve this information from the variance which is under the threshold and reuse the current partition table for the next interval.

\begin{figure}

  \centering
  \includegraphics[width=0.5\textwidth]{diagrams/Sigmod2018_-_Overview2-3.pdf}
  \caption{Overview of a sliding window in PanJoin.}
  \label{fig:overview_algorithm}
\end{figure}

Since the data distribution may change during each interval, in the same window, the tuples adhering to different partition tables need to be managed separately. PanJoin realizes this by dividing the entire sliding window into subwindows based on the arrival time of incoming tuples, as shown in Figure \ref{fig:overview_algorithm}. Each subwindow holds the partitioned tuples within a time interval and an associated header which includes a partition table and several run-time histograms. There is also a global table called \textit{expiration table} dealing with expiration for the entire window. The subwindows are organized as a queue: every new tuple is inserted into the newest subwindow, and the oldest tuple is expired from the oldest subwindow. When the new tuple is inserted, the header of the newest subwindow is also updated. When the newest subwindow is full, a new subwindow is created and becomes the newest subwindow, and its partition table is calculated based on the histograms in its predecessor. Every time a new tuple arrives, the \textit{expiration table} checks if there are some tuple to be expired and let the oldest subwindow execute expiration if needed. When the oldest subwindow is empty, it is popped from the subwindow queue, and its successor becomes the oldest subwindow.

Our algorithm implements a similar methodology used in RBSNLJ \cite{ya2006indexed}. We manage the tuples in each subwindow by range partitioning instead of a red-black tree. Different from handshake join \cite{teubner2011soccer, roy2014low}, each tuple is inserted into a fixed subwindow and it will not be bypassed through the data flow, which means that each tuple will not be compared with all the tuples in the window of the partner stream. Compared with SplitJoin \cite{najafi2016splitjoin}, ScaleJoin \cite{gulisano2016scalejoin} and CellJoin \cite{gedik2009celljoin}, we add additional tuple management (range partitioning) inside each subwindow to reduce comparison during the probing phase of join processing.

\end{comment}

\subsection{RaP-Table}

RaP-Table range partitions the tuples in a subwindow. The partition boundaries, which are called \textit{splitters}, are stored in the \textit{partition table}. To find the target partition during insertion and probing, RaP-Table performs a binary search on the partition table.
The main challenge for using range partitioning is handling the skew in real-time data, i.e., the data may unevenly distribute among the partitions according to the current splitters such that the subwindow needs more time to scan some large partitions to obtain the join result. In addition, data skew introduces another challenge for storing the tuples because RaP-Table also attempts to store the tuples continuously in the same partition to accelerate probing, and skewed data make it difficult to predict the proper size of each partition to allocate in memory.

RaP-Table provides a solution to the two challenges by implementing an \textit{adjustment algorithm} and a data storage structure called \textit{Linked List Adaptive Table} (LLAT).

\subsubsection{Adjustment Algorithm} \label{adjust_algo}

When a new subwindow is created, it receives a new partition table calculated based on the sampling information in its predecessor. The sampling information includes three histograms: the tuple count $count_i$, the maximum tuple value $max_i$, and the minimum tuple value $min_i$ of each partition.
The main idea is to scan the histogram of the tuple count to find an approximate range for each splitter and use two other histograms to calculate a new value for each splitter.

\begin{figure*}[!htb]
    \centering
    \begin{minipage}[b]{.26\textwidth}
  \centering
  \includegraphics[width=\textwidth]{diagrams/Sigmod2018_-_Adjust1.pdf}
  \caption{Scan the count histogram.}
  \label{fig:adjust1}
    \end{minipage}\hfill%
    \begin{minipage}[b]{0.26\textwidth}
  \centering
  \includegraphics[width=\textwidth]{diagrams/Sigmod2019_-_rap_worst.pdf}
  \caption{A adjustment worst case.}
  \label{fig:adjust2}
    \end{minipage} \hfill%
    \begin{minipage}[b]{0.26\textwidth}
  \centering
  \includegraphics[width=\textwidth]{diagrams/Sigmod2018_-_LLAT-2.pdf}
  \caption{An LLAT with $P = 4$.}
  \label{fig:llat}
    \end{minipage}
\end{figure*}

First, we calculate the prefix sums $sum_i = sum_{i-1} + count_i$ of the tuple count histogram and the \textit{balancing indicator} $bal_i = N / P \cdot i $, where $N$ is the total tuple count and $P$ is the partition count. The value of $bal_i$ is the ideal value of $sum_i$ if the tuples are evenly partitioned. In the example shown in Figure \ref{fig:adjust1} where $N = 16$ and $P = 4$, we have $bal_2 = 8$, which means that the previous 2 partitions should have 8 tuples in the ideal case. For any two partitions $i$ and $j$, if $bal_j \in (sum_{i-1}, sum_i]$, we know that the new $j^{th}$ splitter $s^{new}_j$ should be the value of a tuple stored in the $i^{th}$ partition, i.e., $s^{new}_j \in [min_i, max_i]$. For example, in Figure \ref{fig:adjust1}, $bal_2 = 8 \in (sum_2, sum_3] = (5, 10]$, which indicates that the $s^{new}_2$ should be a tuple value stored in the $3^{rd}$ partition, i.e., $s^{new}_2 \in [min_3, max_3]$.

Then, we compute the value of $s^{new}_j$. Since we do not have more information about the distribution inside a partition, we assume that they follow a uniform distribution. Thus, according to the feature of a uniform distribution, the value of $s^{new}_j$ should be:
$$
s^{new}_j = \frac{bal_j - sum_{i-1}}{count_i} \cdot (max_i - min_i)
$$
The calculation of the new splitters can be performed in a single loop. We present the pseudocode in Algorithm \ref{alg:adjust}. For each iteration, we compute the prefix sum $sum_i$ (Line 4). Because there may be more than one $bal_j$ that satisfies $bal_j \in (sum_{i-1}, sum_i]$, once we find a valid $bal_j$ (Line 5), we use a while loop to check all possible $bal_j$ values (Line 6) and  calculate the new splitters (Line 7). The time complexity of the algorithm is $\mathcal{O}(P+P) = \mathcal{O}(P)$ because we basically generate and scan the two arrays ($sum_i$ and $bal_i$) of size P only one time.

\begin{algorithm}[ht]
\KwData{Histogram $count_i$, $max_i$, $min_i$, $i$ in $1...P$}
\KwResult{New splitters $s^{new}$}
\Begin{
$sum_0 \longleftarrow 0$; $bal_1 \longleftarrow N / P$; $i \longleftarrow 0$; $j \longleftarrow 0$\;
\While{$sum_i < N$}{
	$i \longleftarrow i + 1$; $sum_{i} = sum_{i-1} + count_i$\;
    \If{$bal_j \in (sum_{i-1}, sum_i]$}{
      \While{$bal_j <= sum_i$}{
        $s^{new}_j \longleftarrow \frac{bal_j - sum_{i-1}}{count_i} \cdot (max_i - min_i)$\;
        $j \longleftarrow j + 1$; $bal_{j} \longleftarrow bal_{j-1} + N / P $\;
      }
    }
}
}
\caption{Splitter Adjustment}
\label{alg:adjust}
\end{algorithm}

For this algorithm, we have constructed a worst case scenario, as shown in Figure \ref{fig:adjust2}. In this case, before calling the adjustment algorithm, all of the data are inserted into one partition (for convenience, let us assume that it is the $1^{st}$ partition). The data are distributed inside that partition as follows: the largest value equals the splitter $s_1$, the second largest value equals $s_1/P$, the third largest value equals $s_1/P^2$, and so forth. Now, because $min_1 = 0$, $max_1 = s_1$, $bal_i = N/P \cdot i$, $count_1 = N$ and $sum_0 = 0$, the algorithm provides $s^{new}_1 = s_1/P$, and other splitters $s^{new}_i = s_1 \cdot i/P$. However, in this way, all updated partitions are still empty, except the $P^{th}$ partition holds one tuple and the $1^{st}$ partition holds the remaining tuples. The same situation will repeat after another adjustment and will end once the the range $[min_1, max_1]$ is not divisible by $P$. If we assume that the values are 32-bit integers, then the maximum number of adjustments is $\lceil log_{P}2^{32}\rceil$. If $P = 1024$, then $\lceil log_{P}2^{32}\rceil = 4$. If $P = 65536$, then $\lceil log_{P}2^{32}\rceil = 3$. Thus, the algorithm needs only a few calls to adjust the splitters. In Section \ref{sec:eva_rap}, we show that for some commonly used distributions, the algorithm can converge in only 1-3 times of adjustment.

The size of a histogram equals the partition count $P$. RaP-Table uses three histograms (count, maximum value, and minimum value) and a partition table, which are four arrays that hold $4P$ elements in total. An element has the same size as a tuple value. Since the partition count is considerably smaller than the subwindow size $N_{Sub}$, e.g., $P=64K$ and $N_{Sub}=8M$, the overhead of memory allocation for those arrays is negligible ($64K * 4 / 8M < 0.04$).

\begin{comment}

Suppose we have a case that the data is evenly distributed in the value range of 32-bit integers, and the distribution is suddenly narrowed down to a small range $R$. In the case of $P$ partitions in the subwindow, $R$ can be as small as $P$ (if $R < P$, the data is not partition-able by $P$ partitions). Figure \ref{fig:adjust2} shows the worst case, where only one tuple has a value at one extremum, and the values of the remaining tuples concentrate on other extremum. Because the algorithm assumes that the data follows a uniform distribution, one tuple is partitioned into one partition and the remaining $N_{sub}-1$ tuples are partitioned into another partition, which means that we have to adjust the latter partition again. The worst case can repeat recursively for several times until the value range $R$ of the tuples in one partition is equal to $P$, when the algorithm can compute the splitters in the correct way. From the figure, we can see that this procedure takes $log_{P}R_{sub}$ iterations, where $R_{sub}$ is equal to $2^{32}$ when the values are 32-bit integers. If $P = 1000$, $\lceil log_{P}R_{sub} \rceil = 4$. Thus, the algorithm needs only a few iterations to adjust the splitters.

\begin{figure}

  \centering
  \includegraphics[width=0.40\textwidth]{diagrams/Sigmod2018_-_Adjust3.pdf}
  \caption{A global worst case for the adjustment algorithm.}
  \label{fig:adjust3}
\end{figure}

However, in the real-time use cases, the distribution can be arbitrary. The worse case scenario could be as shown in Figure \ref{fig:adjust3}. In this case, $P-2$ partitions hold values concentrated on one extremum, and two partitions have values that are distributed among the rest of the data range. In other words, the algorithm needs to adjust two partitions with the worst case scenario discussed in the previous paragraph. Similarly, it needs $log_2[R_{sub}-(P-2)]$ iterations to get the balanced result. When $(P-2)\ll R_{sub}$, we have $log_2[R_{sub}-(P-2)] \approx log_2R_{sub}$. Given $R_{sub} = 2^{32}$, the algorithm needs 32 iterations to converge. Nevertheless, in Section \ref{sec:adjust}, we show that for some commonly used distributions, the algorithm can converge in only 1-2 iterations.

Note that for each iteration, we need to repartition the tuples inside the subwindow; the total time complexicity for this operation will be $N_{sub} \cdot log_2R_{sub}$. Thus, we introduce a strategy called \textit{preliminary sampling}, meaning that we sample and calculate the new splitters when the subwindow receives a small number of tuples $N$ where $N \ll N_{sub}$. We can trigger preliminary sampling by calculating the mean absolute error (MAE) of the count histogram. If the MAE is larger than a threshold, we can either calculate the new partition table or dynamically shrink the subwindow size and allocate a new subwindow. Because MAE is easy to compute on FPGA, this strategy is hardware friendly.

\end{comment}

\subsubsection{LLAT}

\begin{comment}

For each sliding window, there are three kinds of data we need to store: the global expiration table, the subwindows' headers, and the partitioned tuples in each subwindow. In the following subsections, we discuss each of the categories in detail.

\subsubsection{Global Expiration Table} \label{sec:expire}

The global expiration table stores the partition index of every tuple in the window. The partition index is generated by the partition function of the subwindow the tuple belongs to. The expiration table stores those indexes in order of the arrival time of the tuples. Each time a new tuple arrives, it is not only processed by the subwindow but sends its partition index to the expiration table to add a new entry for itself as well. When the window is full, every time a new tuple is inserted, the expiration table pops its oldest entry and sends it to the oldest subwindow. Because the tuples of each partition inside the subwindow are also saved in order of their arrival, once the subwindow gets the partition index, it expires the oldest tuple stored in that partition.

The global expiration table requires the ability to store entries in order of their arrival. For a count-based sliding window, it can be implemented as a circular buffer. For a time-based sliding window, we can realize it by a queue. However, for a time-based sliding window, each entry also needs to store a timestamp for each tuple. When a new tuple arrives, it scans one or more tuples in the tail and pops out all the tuples with an expired timestamp. Note that expired tuples may come from more than one subwindow.

\subsubsection{Subwindow Header}

The storage of the headers is trivial. We save the partition table and histograms as arrays. We have three histograms: one that keeps the highest value of each partition, one that preserves the lowest value of each partition, and one that saves the number of tuples in each partition. The histograms are updated every time a tuple arrives in the subwindow.

\subsubsection{Partitioned Tuples}

There are three requirements that a good storage strategy needs to satisfy to store the range-partitioned tuples. First, it must be adaptive to the data skew. Because the distribution is unknown, it must be able to deal with the most extreme condition without a high overhead, that is, all the tuples belong to a partition with an unpredictable index. Second, it should keep the memory access pattern continuous. We do not expect a strategy that allocates random addresses for each tuple in a chunk of data and links them together as a conventional linked list, which can significantly reduce the performance for comparison. Third, it should be trivial to apply on FPGA. That is the reason why we do not consider the dynamically size-adaptive containers in the standard C++ libraries, e.g., vector or list.

\end{comment}

RaP-Table requires a storage strategy that keeps the memory access pattern continuous and adapts to the data skew.
To meet these requirements, we designed a data structure called \textit{Linked List Adaptive Table} (LLAT). Figure \ref{fig:llat} illustrates the structure of an LLAT. Each LLAT has $2P$ entries, where $P$ is the partition count. The first $P$ entries, the \textit{normal entries}, are used for all partitions, with one entry per partition. The last $P$ entries are called \textit{reserved entries} which store skewed data. The LLAT has a global pointer \textsl{PtrG} that points to the first unused reserved entry. Each entry has an array that holds $(N_{Sub} / P) \cdot \sigma$ tuples belonging to the same partition, where $N_{Sub}$ is the subwindow size and $\sigma$ is a threshold larger than 1. We suggest that $\sigma$ should be approximately 1.10-1.25 to handle the small skew in the real-time data. Each entry has two data pointers: \textsl{Head} and \textsl{Tail}. Similar to the case in a linked list, each entry also has a \textsl{Next} pointer referring to an entry in the LLAT.

The insertion and deletion operation is defined as follows. Initially, in every entry, the \textsl{Head} and the \textsl{Tail} point to the zero position of its array. The \textsl{PtrG} is set to $P$. Once a tuple obtains its partition index $i$, it is inserted at the partition's \textsl{Head}, which is then increased by 1. If an entry is full after an insertion, its \textsl{Next} pointer will point to the entry that \textsl{PtrG} refers to, then \textsl{PtrG} is increased by 1. In this way, we create a linked list for an unbalanced partition. Once an unbalanced partition receives another new tuple, LLAT will go through its linked list and insert the tuple at the last entry. The deletion is straightforward. When the LLAT receives the expired request with a value of a partition index, it increases the \textsl{Tail} of the corresponding entry by 1. If the entry is empty (its \textsl{Head} equals its \textsl{Tail}), LLAT will check the entry indicated by the \textsl{Next} pointer. This process continues iteratively until LLAT finds a nonempty entry and expires the tuple.

It is easy to prove that $2P$ entries are enough for all the cases. Suppose that we have a case in which $2P$ entries are not enough. When this occurs, there must be at least $P$ entries that are full (so they require $P$ reserved entries). Because $\sigma$ is larger than 1, it means that the subwindow has at least $(N_{Sub} / P) \cdot \sigma \cdot P = N_{Sub} \cdot \sigma > N_{Sub}$ tuples, which is not possible because a subwindow has at most $N_{Sub}$ tuples.

The storage overhead of LLAT can be calculated as
$(N_{Sub} / P \cdot \sigma) \cdot 2P / N_{Sub}-1$. When $\sigma$ is 1.25, the overhead equals 150\%, meaning that we need additionally 1.5 times more space to store all the data, or the space utilization is 40\%, which is acceptable in practice.

\subsubsection{Known Issues}

RaP-Table has a limitation in handling incoming data that have increasing values. For example, the id field of the tuples generally contains increasing values. Because RaP-Table generates the new partition table based on the data pattern in the previous subwindow, the new partition table is not able to partition newly arrived tuples evenly but delivers them into one or few partitions. For example, if the value range of the previous subwindow is $[0, 1000]$, the generated new partition table will attempt to partition the values between $[0, 1000]$. If the following data has a value range of $[1000, 2000]$, the new partition table becomes powerless.

To address this issue, we designed a new method based on B$^+$-tree called \textit{Wide B$^+$-Tree} (WiB$^+$-Tree), a slower but more powerful data structure, which is shown in the following subsection.

\subsection{WiB$^+$-Tree}

\textit{Wide B$^+$-Tree} (WiB$^+$-Tree) is a data structure adapted from regular B$^+$-tree; therefore, it naturally adapts to skewed data. A leaf node in WiB$^+$-Tree is designed similar to a partition in RaP-Table, while internal nodes are used as the partition table to index the leaf nodes. The major differences between a WiB$^+$-Tree and a B$^+$-tree is that the leaf nodes have different configurations than the internal nodes in a WiB$^+$-Tree.

First, a leaf node has more elements than internal nodes. We would like to keep the size of the internal nodes small enough such that the internal nodes remain in the CPU cache.
An internal node with 64 elements of 32-bit integers needs at least 256 Bytes of memory, which means that a modern CPU core is able to hold 4096 nodes in its private 1 MB L2 cache or approximately 85,000 nodes in the shared 22 MB L3 cache, which are enough for a WiB$^+$-Tree to index a large subwindow (larger than 1M).

Second, the elements in the leaf nodes are unsorted, unlike the internal nodes where all the elements are sorted. We sort the leaf node only when we need to split it into two nodes.
Suppose that the width of a full leaf node is $W$; if we keep the order inside the leaf node, the time complexity of obtaining a full node (inserting $W$ elements in that node) is $\mathcal{O}(W^2)$, while a sort operation at the time of node splitting has a complexity of only $\mathcal{O}(WlogW)$.
In our preliminary experiment, we found that a WiB$^+$-Tree with unsorted leaf nodes commonly has 3-5 times less insertion time than a WiB$^+$-Tree with sorted leaf nodes.

Third, no internal node has duplicate elements. All the tuples with the same value will be inserted into the same leaf node, which may cause some leaf nodes to have more elements than their maximum width. To efficiently store the extra tuples, we use LLAT to organize all the tuples stored in all the leaf nodes. A slight difference between the LLAT in WiB$^+$-Tree and RaP-Table is that the \textsl{PtrG} pointer initially points to the first partition, since there is no leaf node serving as a partition when the subwindow is empty. Each leaf node holds a pointer to an entry in the LLAT,  while the other information of the leaf node, such as node size, can be held in the CPU cache.

\subsection{BI-Sort}\label{sec:bisort}

After we designed RaP-Table and WiB$^+$-Tree, we found that both data structures sort the subwindow at a coarse level. RaP-Table behaves similar to a bucket sort, while WiB$^+$-Tree indexes those ``buckets" with a heterogeneous B$^+$-Tree. Thus, we design a data structure that genuinely sorts the entire subwindow at a fine level.

There are two challenges in designing a sorted-based data structure. First, we have to spend $\mathcal{O}(N)$ time to insert a new tuple into the correct position. Although we can find the address in $\mathcal{O}(logN)$ time through a binary search, we still need $\mathcal{O}(N)$ time to shift the tuples larger than the inserted tuple to the new addresses.
Second, in the probing operation, even though we can find the target tuple in $\mathcal{O}(logN)$ time with a binary search, each step of the binary search has a memory access that can be treated as a random memory access. Although the first few accesses may hit the cache, when $N$ becomes large, e.g., 8M = $2^{23}$, most of the memory accesses have a considerable delay, which slows the probing operation.

BI-Sort overcomes the first challenge with an \textit{insertion buffer}. When a new tuple arrives, it is first inserted into the insertion buffer. The insertion buffer has a limited size $B$, and all the data are unsorted. The new tuple will remain in the buffer until the buffer is full, then the buffer will be sorted and merged into the subwindow data that are stored as a sorted array called the \textit{main array} in the memory. Merging two sorted arrays requires $\mathcal{O}(M+B)$ time ($M$ is the size of the main array), which is shared by $B$ tuples. Therefore, we can significantly reduce the insertion time such that the insertion will not be the bottleneck of the stream join processing. When the subwindow is probed, both the tuples in the main array and the insertion buffer will be both probed. Since the insertion buffer has a limited size, it can be stored in the cache to have a limited time cost of a linear scan.

To address the second challenge, BI-Sort adds an \textit{index array} for the main array. The index array is updated immediately after the insertion buffer is merged into the main array. The index array samples the value of every $M/P$ tuple in the main array, where $P$ is the size of the index array, i.e., the partition count. The memory space between two adjacent indexed tuples is called a \textit{partition}. When $P$=64K, the index of 32-bit integer needs only 256 KB, which can easily be stored in the L2 private cache. When BI-Sort needs to perform a binary search on the main array, it first performs the binary search on the index to find the target partition, and then it performs a binary or linear search (batch mode, discussed in the following subsection) inside the partition. Note that usually $P << M$; therefore, the time complexity of merging the main array together with updating the index is $\mathcal{O}(M+B+P) = \mathcal{O}(M+B)$. Thus, updating index has little effect on the performance.

In conclusion, to overcome the disadvantages of the traditional sort-based solution, we add a buffer and an index to a ``merge sort''-based data structure, which is named \textit{Buffered Indexed Sort}.
BI-Sort has the smallest storage overhead among the three data structures: the size of the main array is the same size as the subwindow, while RaP-Table and WiB$^+$-Tree use LLAT that needs more space than BI-Sort. With the same number of partitions, the index array and the insertion buffer of BI-Sort also require less space than the partition table and histograms in RaP-Table, as well as the tree nodes in WiB$^+$-Tree.
BI-Sort is also the simplest algorithm, which allows us to implement it on FPGA.
Furthermore, because BI-Sort saves the tuples in order, the theta join result of a probing tuple can be represented as a $<\mathsf{id_{start}}, \mathsf{id_{end}}>$ record (with a label \textit{not} for condition $\neq$), where $\mathsf{id_{start}}$ is the index of the main array where the result starts and $\mathsf{id_{end}}$ is the index where the result ends. If the join condition has more than one band, e.g., $a \in [b-5, b+5] \lor a \in [b+20, b+35]$, BI-Sort uses the same number of records as the bands to represent the result. Therefore, when the selectivity is very high (e.g., most of the tuple values in the stream are the same), RaP-Table and WiB$^+$-Tree need to copy a large amount of tuples into the result, while BI-Sort simply returns index records. In this way, BI-Sort can significantly reduce the size of the feedback message to the manager node. To support this, the manager node has to maintain a mirror of the main array of every subwindow. The main array in the manager node will perform the same insertion as in the corresponding worker node. In our preliminary experiment, there is no performance overhead to keep a mirror of each subwindow, since the insertion in the manager node is performed in parallel with the newest worker node.

\begin{comment}

Compare to a tree structure, LLAT offers two advantages summarized in the two following paragraphs.

First, it keeps the memory access pattern continuous. In each entry of an LLAT, the tuples are saved continuously as an array. Those tuples share only one random memory access penalty. In a balanced tree, each node is added based on its created time. Thus, the memory location of those nodes is continuous in the aspect of the arriving time of tuples instead of their logical relationship. Furthermore, their logical relationships are frequently changed during insertion of nodes. Thus, in practice, each read or write to a tree node is a random memory access. Because our sliding window can be large (millions of tuples), there is not enough space to hold all the trees of every subwindow in the CPU cache. Therefore, we have to perform a large amount of random access to main memory during the comparison phase of join processing. Suppose the access penalty is 200 cycles. During this time, we can linearly scan 200 tuples in a fully pipelined fashion on CPU and may scan more if the implementation is empowered by SIMD! Thus, when the selectivity is higher than a threshold (in this example is $1/200=0.5\%$), it is even better to use nested-loop join instead of a tree-based indexed join.

Second, it makes hardware design simple. Similar to CPU, an FPGA chip also has limited internal memory resources, called Block RAMs (BRAMs). A typical size of total BRAM is around 1.6-8.6MB on the newest Xilinx FPGA \cite{xilinx2017ultrascalep}, which means that an FPGA can hold less than 1.1 million tuples if each tuple has a 32-bit value and 32-bit key. Recent studies have presented a tree implementation that keeps a maximum number of 219, or 512K nodes, on an FPGAs with a large capacity of BRAMs \cite{qu2013high}. Apparently, we can not store large trees on the FPGA chip. If we put trees in the external memory, we will encounter the same problem we have discussed above. Besides, implementing trees is not trivial on FPGA. However, PanJoin only needs a few pointers, while tuples are stored in the external memory. In the real implementation, those pointers are configured as integers that serve as offsets of the entries. Thus, it is straightforward to implement LLAT on FPGA.

Note that for every insertion, the LLAT has to go through the entire linked list of one partition to find the destination entry. This process will slow down the performance when most of the tuples concentrate on a few partitions. A trick is to maintain a table that saves the id of all the final entries for each partition. Thus, for the insertion, only once access is necessary. Similarly, a table for expiration is also required to reduce the processing time.

\end{comment}

\subsection{Batch Mode}

\begin{figure}
  \centering
  \includegraphics[width=.40\textwidth]{diagrams/Sigmod2019_-_rebounding_bi.pdf}
  \caption{Rebounding binary search.}
  \label{fig:rebounding}
\end{figure}

In PanJoin, the manager node needs to send incoming data to the worker nodes. To fully utilize the network bandwidth, PanJoin supports \textit{batch mode} where a batch of tuples will be processed simultaneously. In addition to batching the tuples at the architecture level, all three of the data structures support batch mode for insertion and probing.

RaP-Table uses a modified partitioning algorithm for  insertion and probing in batch mode. Rather than regular binary search, RaP-Table uses a \textit{rebounding binary search}, shown in Figure \ref{fig:rebounding}. Rebounding binary search has two phases: forward phase and backward phase. Starting with an initial element in the sorted array, the algorithm goes in one direction in the forward phase, then  turns back to backward phase if it reaches a value larger than the target value, and then performs a regular binary search. The step length is doubled after each comparison in the forward phase and halved in the backward phase. Because all the tuples in the batch are presorted by the manager node, the target partition for the next tuple must have an index larger or equal to the target partition of the current tuple. Therefore, during insertion and probing, RaP-Table uses rebounding binary search to find the target partitions of the batched incoming tuples : each tuple starts it search from the partition id of its previous tuple, which is faster than performing a ``complete'' binary search per tuple. In this manner, the partition phase only accounts for less than 10\% of the total execution time of the insertion and probing operation, which is competitive to a hash-based solution, e.g., SHJ. When probing the subwindow, the tuples with the same target partition can share the same memory access, which also notably improves the performance.

When traversing the tree, WiB$^+$-Tree starts with the node path in the tree of the previous batch tuple, backtracks to the upper levels, and then goes downwards to the correct leaf node if necessary. During probing, similar to RaP-Table, WiB$^+$-Tree first finds the target partitions for all the batch tuples, and then it scans the partitions to improve the performance. Both RaP-Table and WiB$^+$-Tree use a nested loop (inner: related batch tuples, outer: partition tuples) to scan a target partition of several related batch tuples.

BI-Sort first compares the sizes of the batch and the insertion buffer before actual insertion. If the batch is larger than the buffer, BI-Sort directly merges the batch into the main array. Otherwise, it places all the batch tuples into the insertion buffer. While performing probing, BI-Sort first uses the rebounding binary search to find the target partitions of the batch tuples, and then it scans the corresponding partitions. Because the tuples in the partition and related batch tuples are both sorted, the result index of tuple $t_i$ must be larger than or equal to the result index of $t_{i+1}$. Thus, rather than a nested loop, BI-Sort can simply scan the partition with a merged-like loop: if the current partition tuple is smaller than the join condition of the current batch tuple, we increase the iterator of the partition tuples; otherwise, we increase the iterator of the batch tuples. In this way, each partition tuple is accessed only once: thus, BI-Sort can achieve higher performance than the other two data structures.
We illustrate this algorithm by using equi-join as an example. In RaP-Table and WiB$^+$-Tree, because the partition is unsorted, a partition tuple in partition $P$ needs to be joined with all the batch tuple with the same target partition as $P$. In BI-Sort, when a batch tuple $t_1$ reaches to a partition tuple with a larger value than $t_1$, it stops probing and this partition tuple becomes the start of the next batch tuple $t_2$. Here, we assume $t_1$ has larger value than $t_2$. Otherwise, $t_2$ will have the same value as $t_1$ (batch is sorted), then BI-Sort copies the join result from $t_1$ to $t_2$. In this way, a partition tuple in partition $P$ only needs to be joined with a single batch tuple. Therefore,  Although RaP-Table and WiB$^+$-Tree can sort their partitions after the subwindow is full and stays unchanged, this can cause some extra overhead and may significantly reduce the performance in some cases. Thus, in this paper, all the partitions in RaP-Table and WiB$^+$-Tree remain unsorted.

In practice, the manager node sets two conditions of maximum collecting time and maximum tuple count for batch mode: either of the two conditions is satisfied, the manager node packs all the collected tuples into a batch and starts processing.

\subsection{Equi-Join vs. Non-Equi-Join} \label{sec:join_con}

The three data structures each have different strategies for equi-join and non-equi-join to probe partitions.

\subsubsection{Equi-join} In RaP-Table and WiB$^+$-Tree, each probing tuple has a single target partition. In BI-Sort, a probing tuple may have multiple target partitions because tuples with the same value may be stored across several partitions. Therefore, in BI-Sort, the rebounding binary search provides the target partition $P_i$ with the lowest id $i$. Then, BI-Sort converts the current equi-join $x=v$ into a non-equi-join as $x \in [v, v^+)$, where $v^+$ means the smallest value larger than $v$.

\subsubsection{Non-equi-join} Except for the condition $\neq$, which can be performed by a equi-join with a filtering operation, PanJoin will calculate the upper bound and lower bound of the band of the join condition for each probing tuple. Then, three data structure probes the target partitions $P_{low}$ of the lower bounds, and the target partition $P_{up}$ of the upper bounds if $P_{low} \neq P_{up}$. Additionally, RaP-Table and WiB$^+$-Tree copy the partitions between $P_{low}$ and $P_{up}$, while BI-Sort can skip this procedure if it chooses $<\mathsf{id_{start}}, \mathsf{id_{end}}>$ as its result format.

\subsection{Miscellaneous Implementation Decisions}

\subsubsection{Expiration} PanJoin expires an entire subwindow (the oldest one) instead of several tuples in the oldest subwindow. Therefore, none of the three data structures currently has a deletion operation (LLAT has, but RaP-Table does not). Since there will be a number of subwindows for a stream and all of them are probed in parallel, an extra subwindow will not cause much overhead. While probing the oldest subwindow, we employ a filtering operation to remove expired tuples in the result.

\subsubsection{Count-based Window vs. Time-based Window vs. Out-of-Order Window} A time-based window needs extra fields in each tuple to save the event time or arrived time, depending on the application requirements. There is no difference in using the three data structures. The running status of each worker node saved in the manager node is slightly different: for a count-based window, the manager node monitors the tuple count in each subwindow, while for a time-based window, the manager node also saves the time fields of the oldest and newest tuple in each subwindow. These fields are used for expiring the subwindow: when all the tuples are older than the watermark, the entire subwindow is expired. This mechanism can also handle out-of-order tuples. In addition, when there are a few late arrived tuples in a subwindow, the subwindow can put them into a small buffer instead of its data structure so that when there is a probing request, it does not need to probe the whole data structure to find the join result.

\subsubsection{Multithreading in Worker Nodes} When we implement the three data structure in batch mode, we find that a single thread on CPU cannot fully utilize the memory bandwidth. Therefore, in batch mode, all of the three data structures perform probing with multithreading. The workload is divided based on how many partitions each thread need to probe. In addition, BI-Sort also performs insertion with multiple threads, where BI-Sort tries to balance the size of each piece of the partially merged main array.

\section{Evaluation} \label{sec:eva}

In this section, we first show the analytical performance of the three data structures on CPU and of BI-Sort on FPGA. Then, we show the throughput of PanJoin as a whole system and the comparisons with other stream join solutions. All the throughput axes in the following figures are in \textbf{log} scale.

\subsection{Analytical Evaluation} \label{sec:ana_eva}

\begin{figure*}[!htb]
\captionsetup[subfigure]{justification=centering}
    \centering
    \begin{subfigure}[b]{0.32\textwidth}

\centering
\includegraphics[width=\textwidth]{figures/rap_insert_1.tikz}
\caption{Insertion, $N_{sub}$ = 8M.}
\label{fig:rap_insert_1}

    \end{subfigure}%
    \begin{subfigure}[b]{0.32\textwidth}

\centering
\includegraphics[width=\textwidth]{figures/rap_insert_2.tikz}
\caption{Insertion, $P$ = 64K.}
\label{fig:rap_insert_2}

    \end{subfigure}%
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/rap_probe_1.tikz}
\caption{Equi-join probing, selectivity $S$= 1.}
\label{fig:rap_probe_1}

    \end{subfigure}
        \begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/rap_probe_2.tikz}
\caption{Equi-join probing, $N_{sub}$ = 8M \\ and batch size $N_{Bat}$ = 32K.}
\label{fig:rap_probe_2}

\end{subfigure}
\centering
 \begin{subfigure}[b]{0.32\textwidth}
 \includegraphics[width=\textwidth]{figures/rap_probe_3.tikz}
\caption{Non-equi-join probing, $N_{sub}$ = 8M  \\ and batch size $N_{Bat}$ = 32K.}
\label{fig:rap_probe_3}
 \end{subfigure}
  \begin{subfigure}[b]{0.32\textwidth}
  \centering
   \includegraphics[width=\textwidth]{figures/rap_adjust.tikz}
\caption{Adjustment algorthm. $N_{sub}$ = 510K for Youtube and $N_{sub}$ = 8M for the rest.}
\label{fig:rap_adjust}
   \end{subfigure}
   \caption{Performance of RaP-Table.}
      \label{fig:rap}
\end{figure*}

We implemented the three data structures in C++17. The program is tested on one node of a high-performance cluster released in 2018. The node has two 20-core Intel\textsuperscript{\textregistered} Xeon\textsuperscript{\textregistered} Gold 6148 processors and 192 GB DDR4 memory with a total bandwidth of 150 GB/s. We evaluate the performance of the insertion and probing operation for each data structure by measuring the throughput of input tuples (unit: tuples per second) on the host subwindow of the data structure. Each input tuple has a format of $<\mathsf{key}, \mathsf{value}>$, where $\mathsf{key}$ and $\mathsf{value}$ are both 32-bit integers. We use band join to test the performance of PanJoin processing a non-equi-join. The band join for Stream S is defined as:
$$
\bm{\mathsf{WHERE}}\; \mathsf{s.value}\; \bm{\mathsf{BETWEEN}}\; \mathsf{r.value} - \epsilon\; \bm{\mathsf{AND}}\; \mathsf{r.value} + \epsilon
$$
and vice versa for Stream R, where $\epsilon$ is used to control the selectivity.
The throughput is mainly influenced by the following four factors: subwindow size $N_{Sub}$, batch size $N_{Bat}$, partition count $P$, partition size $N_P = N_{Sub}/P$, and selectivity $S$ which is defined as the average number of matching tuples in a full subwindow per probing tuple. We use count-based windows to present the throughput because it is easier to show the correlation between throughput and the factors it relies on, while changing the input rate of time-based windows will change both the window size and the throughput. Some data structures have their features, which are also presented in the following part of this section. RaP-Table and WiB$^+$-Tree perform insertion with 1 thread, and BI-Sort performs insertion with 8 threads. The probe operations of all data structures are paralleled with 8 threads when $N_{Bat} \geqslant 128$ and are executed with 1 thread when $N_{Bat} < 128$.

\subsubsection{RaP-Table on CPU} \label{sec:eva_rap}

We first show the insertion throughput with a fixed subwindow size $N_{Sub}$=8M in Figure \ref{fig:rap_insert_1}. A larger batch size provides better performance because tuples in the same partition can share memory accesses to the LLAT, and fewer partitions (smaller $P$) save time during binary search on the partition table. When $N_{Bat}$=8M, RaP-Table reaches its peak throughput (approximately 100M tuples/s). In Figure \ref{fig:rap_insert_2} with a fixed $P$, we observe a slight but not significant advantage of smaller subwindow size $N_{Sub}$, which proves that the insertion throughput of RaP-Table mainly relies on the scan of the partition table and memory access to the LLAT.

Figure \ref{fig:rap_probe_1} presents the throughput of equi-join with a selectivity $S$=1, meaning that on average, a probing tuple matches one tuple in the subwindow. Here, smaller $N_P$ provides better performance because fewer tuples are accessed per probing tuple. Additionally, a smaller subwindow runs faster because it has a smaller partition table to scan. When $S$ increases, as shown in Figure \ref{fig:rap_probe_2}, the throughput begins to decrease. Meanwhile, when $S$ is small, a larger partition count $P$ provides higher throughput since it corresponds to a smaller $N_P$. The performance of non-equi-join probing shown in Figure \ref{fig:rap_probe_3} is similar to the equi-join probing: smaller $S$ and larger $P$ provides higher throughput.

In Figure \ref{fig:rap_adjust}, we also present the performance, i.e. the normalized MAE (mean average error),  under the multimodal normal distributions with legends ``N(normalized $\sigma$, modal count, $P$)'', uniform distributions with legends ``U(normalized range, modal count, $P$)''， and a real dataset of Youtube videos (first file, depth 4) \cite{chen2008youtube}. In the Youtube data, the values (view counts) follow a rank-size distribution, where 99\% fall in the 1\% of the data range or 0.01\% of the range of a 32-bit integer. Initially, the partition table assumes that the value is evenly distributed among the range of a 32-bit integer.
We can observe that subwindows with smaller $P$ require fewer iterations to converge and deliver more balanced outcomes (with a lower MAE). Under each distribution, RaP-Table is able to converge in 3 iterations, which proves our statement in Section \ref{adjust_algo}.

\begin{comment}

We first explore the throughput of insertion in RaP-Table. In Figure \ref{fig:rap_insert_1} with a subwindow size of 8M, different lines show the average insertion throughputs using left y-axis. We can observe that increasing the batch size can improve the insertion performance significantly. With the same batch size, larger partition count can slow down the performance because RaP-Table has a larger partition table to process. However, when the batch size gets larger, the throughput difference gets smaller, which shows the effectiveness of the rebounding binary search in the batch mode of RaP-Table. The bars in Figure \ref{fig:rap_insert_1} shows the throughput of inserting the last batch before the subwindow is full, using the right y-axis. This data is important because the subwindow can be a bottleneck if the last insertion is slow. We can see that when the batch size is smaller than 128, the throughput is even lower than 10000. Therefore, when the input rate is high, we need a large batch size to handle the incoming data. We also explore the relationship between the throughput and the subwindow size. In Figure \ref{fig:rap_insert_2}, with a fixed partition count, larger batch size still gives higher throughput, while the subwindow size does not affect the performance much. Note that in Figure \ref{fig:rap_insert_2}, for each line, the batch size must be smaller than $N_{Sub}$; otherwise the subwindow is overfilled.

For probing, we first explore equi-join. Figure \ref{fig:rap_probe_1} gives the throughput of equi-join with a selectivity of 1, meaning that on average a probing tuple matches one tuple in the subwindow. This figure shows two series of data: a series with average partition size $N_P = 64$ and another with $N_P = 512$, where $N_P = N_{Sub}/P$. Therefore, a bigger subwindow size $N_{Sub}$ corresponds to a bigger partition count $P$. We can see that a large batch size can significantly improve the throughput by as much as three orders of magnitude. When batch size gets larger, a smaller average partition size $N_P$ gives higher throughput because each probing tuple needs to compare with $N_P$ tuples in the target partition. With the same $N_P$ and batch size, smaller subwindow runs faster, since it has fewer partitions to probe and a smaller partition table to visit. When the selectivity $S$ increases, as shown in Figure \ref{fig:rap_probe_2}, the throughput starts to drop  when selectivity is larger than 1K ($2^{10}$).
When $S$ is small, a larger partition count $P$ gives higher throughput, since it corresponds to a smaller $N_P$, the number of the total tuples for each probing tuple to compare.
Note that when selectivity $S$ becomes larger than 1K ($2^{10}$), $N_{Sub}/S < P$, meaning that the number of non-empty partitions is equal to $N_{Sub}/S$ (we are testing equi-join), and the number of tuples in a non-empty partition is $S$. In these case, the throughput does not differ among the given $P$.
The performance of non-equi-join probing (here we use band-join with different selectivities) shown in Figure \ref{fig:rap_probe_3} is similar to the equi-join probing, except that there is a throughput drop in each line, e.g., $P$=256K at selectivity $S=2^6$. In this case, $S > N_P$, meaning that RaP-Table needs to probe both the partition with the lower bound of the join condition and the partition with the upper bound of the join condition, as well as copy the middle partitions to the result. Comparing with Figure \ref{fig:rap_probe_2} and Figure \ref{fig:rap_probe_3}, we can observe that non-equi-join is slower than equi-join by approximately 30\% when selectivity is small, due to the extra check of the partition table for both the lower bound and upper bound of the join condition for each tuple, where in equi-join RaP-Table checks the partition table for the tuple value only once per tuple. However, with a larger selectivity, non-equi-join is faster than equi-join, because in non-equi-join RaP-Table does not really compare the value in the middle partitions but instead copies them straightly to the result, which is faster than reading and comparing the same amount of tuples as being done in the equi-join probing.

We also test the adjustment algorithm under normal distributions, uniform distributions and a real data set of Youtube videos \cite{chen2008youtube, chen2008youtubedata}. We measure the performance by the normalized mean absolute error (MAE) of tuple count in every partition of the subwindow. The normalized value is the raw MAE divided by the average partition size $N_P$. Figure \ref{fig:rap_adjust} shows the normalized MAE under different distributions. The thin solid lines present performance under the multimodal normal distributions with legends ``(normalized $\sigma$, the number of modals, partition count $P$)''. We can see that the normalized MAE quickly reduces to 0.02 - 0.06 after 2-3 iterations of adjustment, which can be considered as a balanced status. Here, subwindows with smaller $P$ (blue or green lines) have smaller MAE (0.1-0.24) than those with larger $P$ (red or orange lines, with MAE around 0.5). Furthermore, a smaller $\sigma = \frac{1}{60000}$ causes 1 extra iteration than a larger $\sigma = \frac{1}{600}$ to reach the balance status. The number of modals does not affect the performance much. The thin dashed lines present throughput under the multimodal uniform distributions with legends ``(normalized range, the number of modals, partition count $P$)''. Here, we only use one range value $\frac{1}{6000}$ to illustrate the performance as we do not find much difference in performance among different ranges. Similar to the situation under the normal distributions, subwindows with smaller $P$ have smaller MAE (around 0.1) than those with larger $P$ (around 0.5). For the Youtube data, we use the view count of 510K Youtube videos stored in the depth 4 of the first file in the dataset \cite{chen2008youtube, chen2008youtubedata}. We tested the performance with 2 different $P=128$ and $P=512$, shown as bold solid lines in Figure \ref{fig:rap_adjust}. We do not use large $P$ because the data cannot be divided by $P \geqslant 1024$. The data follows a rank-size distribution, where 99\% falls in the 1\% of the data range, or 0.01\% of the value range of a 32-bit integer. Though the data is highly skewed, RaP-Table only needs 1-3 iterations to adjust, where smaller $P$ gives smaller MAE than larger $P$.

\end{comment}

\subsubsection{WiB$^+$-Tree on CPU}

\begin{figure*}[!htb]
\captionsetup[subfigure]{justification=centering}
    \centering
    \begin{subfigure}[b]{0.32\textwidth}

\centering
\includegraphics[width=\textwidth]{figures/wib_insert_1.tikz}
\caption{Insertion, $N_{sub}$ = 8M.}
\label{fig:wib_insert_1}

    \end{subfigure}%
    \begin{subfigure}[b]{0.32\textwidth}

\centering
\includegraphics[width=\textwidth]{figures/wib_insert_2.tikz}
\caption{Insertion, $P$ = 64K.}
\label{fig:wib_insert_2}

    \end{subfigure}%
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/wib_probe_1.tikz}
\caption{Equi-join probing, selectivity $S$ = 1.}
\label{fig:wib_probe_1}

    \end{subfigure}
        \begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/wib_probe_2.tikz}
\caption{Equi-join probing, $N_{sub}$ = 8M \\ and batch size $N_{Bat}$ = 32K.}
\label{fig:wib_probe_2}

\end{subfigure}
 \begin{subfigure}[b]{0.32\textwidth}
 \includegraphics[width=\textwidth]{figures/wib_probe_3.tikz}
\caption{Non-equi-join probing, $N_{sub}$ = 8M  \\ and batch size $N_{Bat}$ = 32K.}
\label{fig:wib_probe_3}
 \end{subfigure}
 \centering
 \begin{subfigure}[b]{0.32\textwidth}
 \includegraphics[width=\textwidth]{figures/wib_comp.tikz}
\caption{Speedup of WiB$^+$-Tree over B-Tree. $N_{Sub}$ = 8M, $P$=128K.}
\label{fig:wib_comp}
 \end{subfigure}
   \caption{Performance of WiB$^+$-Tree.}
      \label{fig:wib}
\end{figure*}

We use the same metrics as for RaP-Table to test the performance of WiB$^+$-Tree. Here, the partition count $P$ is the number of leaf nodes in the tree. The number is approximate because the tree structure may vary with different randomly generated data. Similar to RaP-Table, in Figure \ref{fig:wib_insert_1} and Figure \ref{fig:wib_insert_2}, we can also observe the benefit brought by a large batch size for insertion operations.  However, with a fixed $N_{Sub}$, the insertion throughput of WiB$^+$-Tree is not sensitive to the value of $P$, as shown in Figure \ref{fig:wib_insert_1}. This is because we do not retain the tuples sorted in the leaf node; therefore the size of leaf node (=$N_{Sub} / P$) does not affect the throughput much. Figure \ref{fig:wib_insert_2} shows that with a fixed $P$, larger subwindows ($>$512K) normally perform slower because the LLAT is too large to fit in the cache, which leads to more memory accesses and lower throughput.

In Figure \ref{fig:wib_probe_1}, we can observe a similar  impact of $N_P$ on the probing throughput of WiB$^+$-Tree to RaP-Table. Figure \ref{fig:wib_probe_2} and Figure \ref{fig:wib_probe_3} also show the similar throughput decreases when the selectivity is large. Both WiB$^+$-Tree to RaP-Table can reach an ideal throughput when $P>$16K.

Figure \ref{fig:wib_comp} shows the speedup of WiB$^+$-Tree over a regular B-Tree implemented by Google \cite{btree2011google}, where $N_{Sub}$ = 8M, $P$=128K, and $S$ varies from 1 to 16K. The insertion speedup is 3.5-4.5x because WiB$^+$-Tree does not sort the leaf nodes. When the batch size $N_{Bat}$ is larger, the speedup of equi-join and non-equi-join probing becomes higher (up to 2.8x for equi-join and 1.3x for non-equi-join), which shows the efficiency of batch mode. Here, we prove the correctness of our design rationale.

\subsubsection{BI-Sort on CPU}

\begin{figure*}[!htb]
\captionsetup[subfigure]{justification=centering}
    \centering
    \begin{subfigure}[b]{0.32\textwidth}

\centering
\includegraphics[width=\textwidth]{figures/bis_insert_1.tikz}
\caption{Insertion, $N_{sub}$ = 8M.}
\label{fig:bis_insert_1}

    \end{subfigure}%
    \begin{subfigure}[b]{0.32\textwidth}

\centering
\includegraphics[width=\textwidth]{figures/bis_insert_2.tikz}
\caption{Insertion, $P$ = 64K.}
\label{fig:bis_insert_2}

    \end{subfigure}%
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/bis_probe_1.tikz}
\caption{Equi-join probing, selectivity $S$ = 1.}
\label{fig:bis_probe_1}

    \end{subfigure}
        \begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/bis_probe_2.tikz}
\caption{Equi-join probing, $N_{sub}$ = 8M \\ and batch size $N_{Bat}$ = 32K.}
\label{fig:bis_probe_2}

\end{subfigure}
\centering
 \begin{subfigure}[b]{0.32\textwidth}
 \includegraphics[width=\textwidth]{figures/bis_probe_3.tikz}
\caption{Non-equi-join probing, $N_{sub}$ = 8M  \\ and batch size $N_{Bat}$ = 32K.}
\label{fig:bis_probe_3}
 \end{subfigure}
  \begin{subfigure}[b]{0.32\textwidth}
  \centering
   \includegraphics[width=\textwidth]{figures/bis_buffer.tikz}
\caption{Insertion throughput for different buffer size, $N_{Sub}$ = 64K.}
\label{fig:bis_buffer}
   \end{subfigure}
   \caption{Performance of BI-Sort.}
      \label{fig:bis}
\end{figure*}

We use the same metrics as RaP-Table and  WiB$^+$-Tree to test the performance of BI-Sort. The default size of the insertion buffer is 1K. Figure \ref{fig:bis_insert_1} and Figure \ref{fig:bis_insert_2} show a significant impact of the insertion buffer: the throughput remains at the same value when $N_{Bat} < $1K. Figure \ref{fig:bis_insert_1} shows insertion of BI-Sort is not sensitive to $P$ because the index array is very small and requires little computation time compared to the main array. However, Figure \ref{fig:bis_insert_2} shows that a larger $N_{Sub}$ has a considerably slower throughput when $N_{Bat}$ is small. In these cases, main array is so large that merging the insertion buffer or tuple batch is costly. The only solution is to increase $N_{Bat}$ to amortize the merging time.

As shown in Figure \ref{fig:bis_probe_1}, $N_P$ does not greatly affect on BI-Sort's probing throughput except when $N_{Bat}$ is small. Figure \ref{fig:bis_probe_2} and Figure \ref{fig:bis_probe_3} show the main difference and advantage of BI-Sort compared with RaP-Table and WiB$^+$-Tree: the probing throughput of BI-Sort is not sensitive to selectivity $S$ because BI-Sort only attempts to find the indices of the join result in the subwindow rather than copying the real tuples to the result. For equi-join shown in Figure \ref{fig:bis_probe_2}, as $S$ increases, each tuple may need to probe two partitions instead of one. Thus, there is a throughput decrease when $P$ is sufficiently large such that $N_P$ is smaller than $S$, whereas Figure \ref{fig:bis_probe_3} has no decreases because during non-equi-join, BI-Sort always checks the upper and lower bounds of join conditions per probing tuple. Figure \ref{fig:bis_probe_2} and \ref{fig:bis_probe_3} also tell us that a larger $P$ improves the throughput because more tuples share the same memory access to the same target partition.

In Figure \ref{fig:bis_buffer}, we use $N_{sub}$=64K to illustrate the impact of buffer size on handling inputs with a small batch size. As shown, the throughput with a large buffer size is 2-3 orders of magnitude larger than with a small buffer size. However, because the buffer is unsorted, the tuple batch needs to perform a nest-loop-join with the buffer during the probing operation. Therefore, we do not recommend a large buffer size which can reduce the system performance.

\subsubsection{Comparison}

\begin{figure}[!htb]
\centering
\captionsetup[subfigure]{justification=centering}
    \begin{subfigure}[t]{0.22\textwidth}
        \centering
 \includegraphics[width=\textwidth]{figures/com_insert.tikz}
\caption{Insertion.}
\label{fig:com_insert}
 \end{subfigure}\hfill%
  \begin{subfigure}[t]{0.22\textwidth}
  \centering
   \includegraphics[width=\textwidth]{figures/com_probe.tikz}
\caption{Probing.}
\label{fig:com_probe}
   \end{subfigure}
   \caption{Performance comparisons, $N_{Sub}$=8M and $P$=64K.}
   \label{fig:com}
\end{figure}

We compare the throughput of insertion and non-equi-join probing with the configuration $N_{Sub}$=8M and $P$=64K. For insertion shown in \ref{fig:com_insert} , BI-Sort outperforms the other two only when batch size is larger than 64K. When $N_{Bat} <$64K, the insertion of BI-Sort becomes the bottleneck of the system. For non-equi-join probing with $N_{Bat} =$32K shown in \ref{fig:com_probe}, when selectivity $S<$1K ($2^{10}$),  RaP-Table is 1.7-2.4x over WiB$^+$-Tree and BI-Sort is about 1.6-7x over RaP-Table. When $S$ is large, BI-Sort can be 100x over the other two because it is not sensitive to $S$, as we have stated before. Therefore, we suggest the following strategy: if $N_{Bat}$ is large or $S$ is large, we should choose BI-Sort; otherwise, if the tuple values does not vary too often and do not gradually increase, we can use RaP-Table, and if not, we should use WiB$^+$-Tree.

\subsection{BI-Sort on FPGA}

\begin{figure}[!htb]
\centering
\captionsetup[subfigure]{justification=centering}
    \begin{subfigure}[t]{0.22\textwidth}
        \centering
 \includegraphics[width=\textwidth]{figures/fpga_insert.tikz}
\caption{Insertion.}
\label{fig:fpga_insert}
 \end{subfigure}\hfill%
  \begin{subfigure}[t]{0.22\textwidth}
  \centering
   \includegraphics[width=\textwidth]{figures/fpga_probe.tikz}
\caption{Probing.}
\label{fig:fpga_probe}
   \end{subfigure}
   \caption{Performance of BI-Sort on FPGA.}
   \label{fig:fpga}
\end{figure}

\input{tables/fpga_util}

We implement our FPGA subwindow on a Terasic DE5a-Net FPGA Development Kit which contains an Arria 10 (10AX115N\-2F45E1SG) FPGA and two channels of DDR3-1066 4 GB memory. We present the performance comparison of the FPGA version with the 3 data structures on CPU in Figure \ref{fig:fpga}, where $N_{Sub}$=8M and $P$=64K. There are 8 mergers and 8 probers on the FPGA. More mergers or probers can not provide better performance because these 16 units are enough to fully utilize the memory bandwidth on FPGA. Figure \ref{fig:fpga_insert} presents the insertion throughput: when $N_{Bat}$ becomes larger than 64K, BI-Sort on FPGA is faster than RaP-Table and WiB$^+$-Tree on CPU. BI-Sort on FPGA is approximately 0.4x-0.6x over the throughput of BI-Sort on CPU because the DDR3 memory (2 channels) used by FPGA is 8.8x slower than the DDR4 memory (6 channels) used by CPU. However, the energy efficiency ratio (EER) on FPGA is approximate 4x over that on CPU, shown as bars using the right y-axis. This result occurs because the TDP (Thermal Design Power) of the CPU is 150 W and we use 8 out of 20 cores on CPU such that the power of the CPU version is 60 W, while the power of the FPGA solution is only 7.9W. For equi-join probing with $S$=1 show in Figure \ref{fig:fpga_probe},  when $N_{Bat}$ is larger than 64K, BI-Sort on FPGA provides better throughput than RaP-Table and WiB$^+$-Tree on CPU. It also provides an EER larger than 1 over BI-Sort on CPU. Therefore, when $N_{Bat}$ is large enough, BI-Sort on FPGA becomes an excellent choice when both throughput and power are concidered. Table \ref{tab:fpga} shows that our FPGA solution uses 42\% of the logic resources and 31\% of the internal block memory (BRAMs), which means that we can puts more processing units for other operations.

\subsection{System Performance}

\begin{figure*}[!htb]
\captionsetup[subfigure]{justification=centering}
    \centering
    \begin{subfigure}[t]{0.22\textwidth}

\centering
\includegraphics[width=\textwidth]{figures/sys_per_1.tikz}
\caption{Equi-join: $W$=128M, $S$=1, $P$=64.}
\label{fig:sys_per_1}

    \end{subfigure}\hfill%
    \begin{subfigure}[t]{0.22\textwidth}

\centering
\includegraphics[width=\textwidth]{figures/sys_per_2.tikz}
\caption{Non-equi-join: $W$=128M, $N_{Bat}$ =32K, $P$=64.}
\label{fig:sys_per_2}

    \end{subfigure}\hfill%
\begin{subfigure}[t]{0.22\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/sys_per_3.tikz}
\caption{Receive overhead. }
\label{fig:sys_per_3}

    \end{subfigure}\hfill
        \begin{subfigure}[t]{0.22\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/sys_per_4.tikz}
\caption{$W$=1G, $N_{Bat}$ = 8M, $P$=64K.}
\label{fig:sys_per_4}

\end{subfigure}

    \begin{subfigure}[b]{0.22\textwidth}
        \centering
 \includegraphics[width=\textwidth]{figures/sys_com_1.tikz}
\caption{System comparison: $W$=8M.}
\label{fig:sys_com_1}
 \end{subfigure}\hfill%
  \begin{subfigure}[b]{0.22\textwidth}
  \centering
   \includegraphics[width=\textwidth]{figures/sys_com_2.tikz}
\caption{System comparison: $W$=128M.}
\label{fig:sys_com_2}
   \end{subfigure}\hfill%
       \begin{subfigure}[b]{0.22\textwidth}
        \centering
 \includegraphics[width=\textwidth]{figures/sys_storm_1.tikz}
\caption{PanJoin in Storm: throughput comparison.}
\label{fig:sys_storm_1}
 \end{subfigure}\hfill%
  \begin{subfigure}[b]{0.22\textwidth}
  \centering
   \includegraphics[width=\textwidth]{figures/sys_storm_2.tikz}
\caption{PanJoin in Storm: throughput ratio.}
\label{fig:sys_storm_2}
   \end{subfigure}
   \label{fig:sys}
   \caption{System performance of PanJoin.}
\end{figure*}

We test the complete PanJoin on a cluster. Each node has the same configuration as discussed in Section \ref{sec:ana_eva}. Every two nodes are connected with InfiniBand with a bandwidth of 100 Gb/s (12.5 GB/s). We first compare the throughput of equi-join with selectivity $S$=1, as shown in Figure \ref{fig:sys_per_1}. The window size $W$=128M for both streams. Each stream has 16 worker nodes and there is one subwindow per node, i.e., $N_{Sub}$=128M$/16$=8M. We find that PanJoin with BI-Sort has the highest throughput up to 28.9M/s. At this input rate, we can process 1G input tuples within 34.6 seconds. We also observe that RaP-Table is faster than WiB$^+$-Tree by approximate 1.5x when $N_{Bat}>$64K. Figure \ref{fig:sys_per_1} also suggests that if the input rate is high, we should use a large batch size such that the system can handle the data. Figure \ref{fig:sys_per_2} shows the system performance of non-equi-join with a smaller batch size ($N_{Bat}$=32K). We can observe the throughput decreases on RaP-Table and WiB$^+$-Tree when the selectivity $S$ becomes larger. Note that the throughput of RaP-Table and WiB$^+$-Tree are nearly identical because under this configuration the overhead of data transmission between manger and workers is more than 60\% of the processing time, which amortizes the throughput differences inside subwindows. Therefore, their lines are overlapped in the figure. Figure \ref{fig:sys_per_3} shows the overhead of data transmission from workers to the manager (not manager to workers). When the selectivity becomes larger than $2^{10}$ (1K), the result receiving overhead for RaP-Table and WiB$^+$-Tree can be more than 90\% of the processing time. For this reason, we marked result receiving as an optional operation in Step 5 of the system architecture (Section \ref{sec:archtect}). We also evaluate non-equi-join with large window $W$=1G processed by different numbers of worker nodes, as shown in Figure \ref{fig:sys_per_4}, where we use lines for selectivity $S$=1 and bars for $S$=64. Here, adding more worker nodes does not always gives better performance because the network communication overhead is dominating the execution time. Still, the system with BI-Sort provide a throughput of approximately 10M/s ($10^7$), where the other two data structures are 1-10x slower when $S$=1 and 10-100x slower when $S$=64

We compare PanJoin with Low Latency Handshake Join and SplitJoin (ScaleJoin), which are adapted to support large $N_{Bat}=$32K and more subwindows/nodes. We consider SplitJoin and ScaleJoin together because their architectures are similar. With window size $W$=8M and 16 worker nodes in Figure \ref{fig:sys_com_1}, the speedup of BI-Sort is more than 1000x over Low Latency Handshake Join and SplitJoin (ScaleJoin), while RaP-Table and WiB$^+$-Tree is more than 100x when the selectivity is small, and RaP-Table is approximate 1.5x over WiB$^+$-Tree. When the selectivity is larger than 4K, RaP-Table and WiB$^+$-Tree have the same throughput as Handshake Join and SplitJoin (ScaleJoin). When $W$ increases to 128 in Figure \ref{fig:sys_com_2}, the speedup of BI-Sort increases to approximately 5000x, while the RaP-Table's and WiB$^+$-Tree's speedup are approximately 100x-5000x when $S\leqslant$1K and 2-100x when $S>$1K. In this way, we show how powerful PanJoin is as an integrated design compared with the existing stream join solutions, which use a nested-loop join inside their subwindows/nodes.

We also attempt to integrate PanJoin into Apache Storm \cite{toshniwal2014storm}. Because the system overhead is too large compared with the pure PanJoin solution, we compare the performance of integrated PanJoin with a system where every subwindow is empty, i.e., the join processor does nothing but receive the input tuples and discard them afterward. Figure \ref{fig:sys_storm_1} shows the absolute throughput and Figure \ref{fig:sys_storm_2} shows the throughput ratio of PanJoin to the empty system, where we set $N_{Sub}$=1M. Here, throughput ratio is always approximately 90 percent, which proves that PanJoin works well with Storm and is fast enough for an existing stream processing engine.

\begin{comment}

\subsection{Throughput on CPU}

We implemented our algorithm in C++11. The program is tested on a 2013 released notebook with a Core i5 4258U 2.4 GHz dual-core processor and 8GB DDR3L 1600 memory. We choose this notebook because its processor has better performance than several desktop processors we have tested. Furthermore, we would like to show how robust our algorithm can be under imperfect conditions.

The algorithm runs with one thread. We have tested the multi-threaded version, and its performance was much slower than the single-threaded version. This is because the algorithm processes one incoming tuple with several threads. When the throughput is very high (>100K tuples/s), the cost of creating and managing the threads is much higher than the actual processing. An alternative way is to process multiple tuples in parallel like \textit{micro-batching} in Apache Spark. However, because the achieved throughput is already very high, in this work, we focus on the single-threaded approach.

We have tested our algorithm on both equi-join and non-equi-join. In this evaluation, we use the configuration of uniform distribution as the ideal case to test the peak performance of our algorithm. The non-uniform cases are shown in Section \ref{sec:adjust} to illustrate the effectiveness of our adjustment algorithm under unevenly distributed data. Note that the input data must be created by a random data generator rather than an arithmetic progression to mimic a uniform distribution. This is because a arithmetic progression will force the algorithm to have an excellent memory locality (a series of tuples will be stored in the same local partition and probe the same partner partition), which can not simulate the real-time situation when the values of tuples follow a random pattern. In our experiment, we find that arithmetic sequences lead to 2-3x better performance than random data. This number happens to be the bandwidth ratio of two adjacent levels of caches.

\subsubsection{Equi-Join}

\begin{figure}
\centering
\includegraphics[width=0.45\textwidth]{figures/throughput_cpu_4m.tikz}
\caption{Throughput comparison under different subwindow counts and partition counts. The window size is 4M.}
\label{fig:throu_cpu_4m}
\end{figure}

\begin{figure*}
\centering
\includegraphics[width=1.0\textwidth]{figures/throughput_cpu.tikz}
\caption{Throughput comparison under different window sizes and partition counts. The subwindow count is 1+1.}
\label{fig:throu_cpu_all}
\end{figure*}

There are four parameters that can affect the throughput: the \textit{window size}, the number of partitions of each stream (\textit{partition count}), the number of subwindows in each stream (\textit{subwindow count}), and the \textit{data type} of tuple values. Figure \ref{fig:throu_cpu_4m} shows the throughput of our algorithm when the window size is 4M (2M for each stream) and the data type is a 32-bit unsigned integer. We choose 4M because it is slightly larger than the window tested in handshake join (a combined input rate 2000-3000 tuples/s $\cdot$ 15 min window = 1.8M-2.7M) \cite{teubner2011soccer, roy2014low}. We also divided the window into between 1 and 16 subwindows. During runtime, neither the oldest nor the newest subwindows is full, therefore an extra subwindow is needed, denoted as $+1$ in the figure. We can see that the peak throughput reduces when the \textit{subwindow count} increases. This is because the algorithm needs to probe more subwindows, during which it has to probe the partition table $N$ times if the subwindow count is $N$. For a fixed \textit{subwindow count}, the throughput increases with the growth of \textit{partition count} to a peak value and decreases when the \textit{partition count} is too large. When the \textit{partition count} is large, the algorithm needs to spend more time to probe the partition tables in both local stream and partner stream, hence the performance reduces.

Figure \ref{fig:throu_cpu_all} shows how the performance changes when the \textit{window size} increases. Here we set the \textit{subwindow count} as $1+1$ to compare the maximum throughput for each case. The peak throughput reduces as the window size increases. The reason is that even though the partition sizes are equal, the \textit{partition count} increases with the growth of \textit{window size}, meaning that the partition tables become larger. Thus, the algorithm takes more time to probe the partition tables. In addition, as the size of the partition tables grow, it becomes increasingly likely that the CPU cache is not able to hold the tables. Because the algorithm uses binary search to probe the table, it may generate more cache misses in these cases. From the figure, we can also observe that the algorithm performs well even when the \textit{window size} is very large. For a \textit{window size} of 256M (128M per stream), it can process more than 200K tuples per second. This window size is equal to a 21-minute window when the combined input rate is 200K tuples/s. Even when we reduce the partition count to 2K, we can still process 15469 tuples per second or 7734 tuples per second per stream. If the input rate per stream is 7.8K tuples/s, each stream (128M) can hold a window of 4.6 hours. Consider the fact that there are 6000-7800 tweets posted per second in the whole world \cite{tweet2017persecond, sayce2017persecond}, our algorithm is very powerful if the input data is partition-able (for integer, the value range is larger than the \textit{partition count}) by the number of partitions given in the figure.

\begin{figure}
\centering
\includegraphics[width=0.45\textwidth]{figures/throughput_cpu_float.tikz}
\caption{Throughput comparison under different data type and window size. The subwindow count is 1+1.}
\label{fig:throu_cpu_float}
\end{figure}

Figure \ref{fig:throu_cpu_float} shows the throughput comparison with different data types of tuple values. Here we present the highest throughput for each given window size. We can see that using a floating type slightly decreases the performance by 10-25\% when the window size is larger than 2M. Thus, our algorithm can perform well with more complex data types.

\subsubsection{Non-equi-join}

\begin{figure}
\centering
\includegraphics[width=0.45\textwidth]{figures/throughput_cpu_non_equal.tikz}
\caption{Throughput comparison for non-equi-join under different selectivity and window size. The subwindow count is 1+1.}
\label{fig:throu_cpu_non_equal}
\end{figure}

We evaluate our algorithm using band join with different selectivities from $\frac{1}{250000}$ to $\frac{1}{25}$. Here the data type of tuple values is 32-bit unsigned integer. To process band join, the algorithm scans the two boundary partitions and copies all the tuples of internal partitions into the result array. Figure \ref{fig:throu_cpu_non_equal} shows the throughput with different \textit{window size}s and the y-axis is set to a logarithmic scale. In this figure, we present the maximum throughput for each given window size. When the selectivity is $\frac{1}{250000}$ and the \textit{window size} is 4M, the throughput is higher than 500K tuples per second, which is two orders of magnitude higher than handshake join. We can also see that the throughput drops significantly when the selectivity increases. However, when the selectivity is lower than $\frac{1}{250}$, our algorithm can still process more than 5500 tuples per second with a \textit{window size} of 256M.

\subsection{Adjustment Algorithm}\label{sec:adjust}

\begin{figure}
\centering
\includegraphics[width=0.45\textwidth]{figures/adjust_uni.tikz}
\caption{Adjustment accuracy under uniform distribution}
\label{fig:adjust_uni}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=0.45\textwidth]{figures/adjust_normal.tikz}
\caption{Adjustment accuracy under normal distribution}
\label{fig:adjust_normal}
\end{figure}

We tested our adjust algorithm on both uniform and normal distribution. Here we use the configuration \textit{window size} = 4M, \textit{subwindow count} = 1+1, and \textit{data type} is 32-bit unsigned integer. We measure the converge accuracy with the mean absolute error (MAE) of the count histogram (which saves how many tuples are stored in each partition) divided by the average number of tuples a partition should hold (the ideal partition size, acquired by subwindow size divided by partition count). This ratio denotes the adjustment accuracy of how unbalanced the number of tuples of each partition could be compared with the ideal case. Initially, we assume the data is evenly distributed among the range of 32-bit unsigned integer. Its accuracy is also calculated from the data that is generated by the random function of C++.

Figure \ref{fig:adjust_uni} shows the adjustment accuracy  under multi-modal uniform distributions with different configurations. The configuration is denoted as a vector (data range, modal, partition count). The data range represents the ratio of shrunk new range with the original range, meaning that how narrow the new distribution becomes. We can see that no matter how narrow the new distribution becomes, or how big the modal is, the algorithm needs only one iteration to reach a low accuracy, and becomes stable afterward. This accuracy happens to be the accuracy we obtained from the original distribution with a given partition count (0.022 or 1K partitions and 0.091 for 16K partitions), thus our algorithm converges fast. Note that the dashed line represents a special case where the range is $\frac{1}{60000}$, modal is 1, and partition count is 16K. In this case, each partition holds only $0xFFFFFFFF \cdot \frac{1}{60000} \div 16K \approx 4$ kinds of values. We find that if a large number of partitions concentrates on such a narrow range, the value of accuracy ratio will increase, meaning that the adjusted result is more unbalanced. Thus, if the data range is too narrow, a better practice is to reduce the partition count, if the throughput is still acceptable for the system.

\input{tables/compare}

Figure \ref{fig:adjust_normal} shows the adjustment accuracy under multi-modal normal distributions. The configuration is denoted as a vector ($\sigma$, modal, partition count). Here, the value of $\sigma$ we show is not the real $\sigma$ value we use, but the normalized $\sigma_{nor}$ which is the ratio $\frac{\sigma_{real}}{0xFFFFFFFF}$. We can see that when the $\sigma$ is large ($>=\frac{1}{600}$), the algorithm converges after only one iteration. When $\sigma$ is very small ($\frac{1}{60000}$), it needs two iterations, and then reaches  a stable accuracy ratio that is close to the ratio obtained from the original ideal distribution. Similarly, the number of modals does not affect the performance much.

Therefore, from the cases we studied, we conclude that our algorithm adjust well to data skew.

\subsection{Performance on FPGA}

\begin{figure}
\centering
\includegraphics[width=0.45\textwidth]{figures/throughput_fpga.tikz}
\caption{Throughput of FPGA solution under different window sizes. subwindow count is set to 1+1 and partition count to 4096.}
\label{fig:through_fpga_slow}
\end{figure}

\input{tables/fpga_util}

We implement our system on a Alpha-Data ADM-PCIE-8K5 board \cite{alpha20178k5}. It has a Xilinx Kintex Ultrascale XCKU115-FLVA1517E-2 FPGA. The board is equipped with 2x banks of 8G DDR4-2400 as the external memory of the FPGA. Figure \ref{fig:through_fpga_slow} shows the throughput of PanJoin on FPGA. Because the current memory controller supports 4GB memory, our system can not hold a window size larger than 128M. The maximum \textit{partition count} that can be configured on this FPGA is 4K. We can see that the throughput on FPGA is about 3-30\% of the throughput on CPU show in Figure \ref{fig:throu_cpu_all}. This is because the current AXI Interconnect module constraints the number of burst read transactions into a very small number (around 2). Nevertheless, the core can still process more than 12K tuple per second if the \textit{window size} is 128M. Consider that the energy consumption of our FPGA is only 3.8W (our CPU consumes 28W), the energy efficiency of our FPGA solution is high. In addition, the resource utilization shown in Table \ref{tab:util} tells us that there are still free areas on FPGA that can be configured into other processing units.

\subsection{Comparison with Other Solutions} \label{sec:comp}

We compared our algorithm with several stream join implementations shown in Table \ref{tab:compare}. Because the other solutions are either deployed on a cluster or some specified hardware, we compare the performance of our single-threaded solution with the performances of the other solutions shown in their original papers. For our algorithm, we set the same input format as CellJoin \cite{gedik2009celljoin}, which are also used in other solutions. The input data value range is [1, 10000] and the selectivity is $\frac{1}{250000}$. Here the partition table is two dimensions, each for one attribute to be compared. We can see that our algorithm is 22-144x faster than the other solutions (not including handshake join with index) when the partition count is set as 2K, In addition, if the data is divisible by 32K partitions, our algorithm is 44-283x faster than the others. In this way, we show the power of range partition to reduce the comparison during join processing. In addition, though our algorithm runs only one core, we still outperform indexed handshake join by 0.3x (with a partition count of 2K) or 1.5x (with a partition count of 32K). Because in our design, the subwindows are independent with each other. Thus, it is easy to extend our algorithm on multi-node systems. In this way, we can increase the parallelism of our algorithm to achieve better performance in the future work.

\end{comment}
\section{Conclusions} \label{sec:con}

In this paper, we present a stream join solution called PanJoin that has high throughput, supports non-equi-join, and adapts to skewed data. We present the three new data structures for subwindows to manage their data and provide a strategy that users can choose. Our evaluation proved that the performance of PanJoin is more than three orders of magnitude higher than several recently proposed stream join solutions. The limitation of PanJoin on large window sizes is the network bandwidth. Since InfiniBand is one of the fastest solutions on the market, we expect that the future network technology will solve this limitation.

\section{BI-Sort on FPGA} \label{sec:imp}

Because BI-Sort is simple enough to implement a hardware version, we attempt to build a worker node with BI-Sort on FPGA by Intel OpenCL in order to benefit from the two major advantages of FPGA: high throughput and low energy consumption. The system architecture is shown in Figure \ref{fig:fpga_arch}. The system has an insertion engine and a probing engine. Both of these engines can access the data buffered in the external memory on the FPGA board. The data can either come from the Internet I/O port that connects to the manager node or come from the host computer, depending on the configuration of the system.

\begin{figure}[!htb]
    \centering
    \begin{minipage}[b]{.40\textwidth}
  \centering
  \includegraphics[width=\textwidth]{diagrams/Sigmod2019_-_fpga_arch.pdf}
  \caption{System architecture of FPGA solution.}
  \label{fig:fpga_arch}
    \end{minipage}
    \centering
    \begin{minipage}[b]{.20\textwidth}
  \centering
  \includegraphics[width=\textwidth]{diagrams/Sigmod2019_-_merger.pdf}
  \caption{Merger.}
  \label{fig:merger}
    \end{minipage}
    \begin{minipage}[b]{0.20\textwidth}
  \centering
  \includegraphics[width=\textwidth]{diagrams/Sigmod2019_-_prober.pdf}
  \caption{Prober.}
  \label{fig:prober}
    \end{minipage} \hfill%
\end{figure}

The insertion engine has several \textsf{merger}s that merge the insertion buffer into the main array.
The structure of a merger is shown in Figure \ref{fig:merger}, where there are two input stream ports, one output stream port, and a comparator. Here, we implement continuous memory access as a data stream to utilize the memory bandwidth. Initially, the merger reads one tuple from each input stream. Then, in each iteration, it compares the two tuples, chooses the smaller one to write into the output stream, and reads one tuple from the chosen stream. When all the mergers finish their work, an \textsf{indexer} will generate the index array.

During probing, the \textsf{boundary generator} inside the probing engine first calculates the upper bound and lower bound of the join condition per probing tuple. Then, the \textsf{partitioner}s find the target partition(s) of each probing tuple. Subsequently, the \textsf{prober}s probe partitions and generate the result. The design of a prober is shown in Figure \ref{fig:prober}, which is similar to a merger. It has an input port for target partitions, an input port for bounds (upper or lower bounds), an output port for results, and a comparator. Initially, it reads a partition tuple and a bound. Then, in each iteration, if the value of the partition tuple matches the join condition, it writes $i$ into the result and reads the next bound. If the tuple exceeds the join condition, it reads the next bound. Otherwise, it reads the next tuple.

\begin{comment}

\begin{figure}

  \centering
  \includegraphics[width=0.50\textwidth]{diagrams/Sigmod2018_-_fpga_arch.pdf}
  \caption{System architecture.}
  \label{fig:arch}
\end{figure}

\begin{figure}

  \centering
  \includegraphics[width=0.50\textwidth]{diagrams/Sigmod2018_-_fpga_core.pdf}
  \caption{An example of a \textit{join core} which has two input streams, S and R. Each stream has four sub-windows. The two \textit{stream modules} have the same content.}
  \label{fig:core}
\end{figure}

\subsection{Overview}

We divided our FPGA implementation into three parts: input, processing, and communication. Figure \ref{fig:arch} shows the system architecture of our implementation. Their functionalities, high-level architectures, and design rationales are described as follows:

\subsubsection*{Input}

The input part is responsible for generating input data and dispatching it to the corresponding \textit{Stream Module} inside the processing part. For a system with two streams, there are two \textit{data generator}s. A \textit{data generator} can create tuples that follow different value distributions, e.g., uniform distribution or normal distribution.

\subsubsection*{Processing}

The processing part consists of a \textit{join core} that performs our stream join algorithm on the input data streams. It has $M$ input stream data ports, one output stream data port, and an I/O port for DDR4 memory data. Here $M$ is the number of input streams.

\subsubsection*{Communication}

The communication part has an \textit{AXI Interconnect}, a \textit{DDR4 memory controller} and a \textit{MicroBlaze} processor. The main function of the \textit{AXI Interconnect} is to interconnect the memory port of the \textit{join core} and the data port of the \textit{DDR4 memory controller}, as well as distribute the control signal from the \textit{MicroBlaze} processor. The \textit{MicroBlaze} processor is a softcore CPU that serves as a control unit for the system. It can initiate the \textit{data generators} and the \textit{join core}. It can also communicate with the host PC so that we can monitor the system and its performance at runtime.

\subsection{Join Core}

The join core has several \textit{stream module}s and a \textit{memory prober}, as shown in Figure \ref{fig:core}. Each \textit{stream module} is used for managing the data in its corresponding stream. These stream modules are responsible for processing the incoming tuples for storage and sending the probing requests to their partner \textit{stream module}(s) who will then probe its stream for this tuple. When there are memory requests (memory write during storage and memory reads during probing), the \textit{stream module} will deliver those requests to the \textit{memory prober}, which performs the actual communication with other modules in the system.

Inside each \textit{stream module}, there are three histograms implemented as Block-RAMs. The histograms are only for the current newest sub-window. We do not save the histograms for other sub-windows because they are not used after the generation of a new partition table by the \textit{splitter adjuster}. There is also a hardware version of our LLAT data structure for each sub-window. However, this hardware LLAT does not hold any data: the tuples are stored in the external memory. It only saves the value of the \texttt{head} and \texttt{tail} pointers, which are implemented as offset integers stored in Block-RAMs. When there is a memory operation, the \textit{memory requester} will send the request to the \textit{memory prober}. The \textit{expiration table} performs the same function as a \textit{global expiration table} that has been described in Section \ref{sec:expire}.

We implemented the \textit{join core} and the \textit{data generators} using High-Level Synthesis (HLS). HLS is a technology that converts C/C++ code into a hardware description at the register-transfer level (RTL), such as Verilog or VHDL. In other words, by using HLS, we only need to write C/C++ code and let the IDE tool compile the software into hardware. The hardware code is derived from software code. We changed the code related to the external memory access for the tuple data. This part of the code accounts for 5.5\% of the total code.
\end{comment}

\section{Related Work} \label{sec:rel}

In this section, we mainly consider stream join and its hardware acceleration. We do not include non-streaming cases because it is not straightforward to map these works directly to stream join processing: traditional join processes a large batch of data at once, while stream join has to produce results on the fly.
\textit{Symmetric Hash Join} (SHJ) \cite{wilschut1993dataflow} is one of the first proposals for bounded input stream-like data. It continuously updates its two hash tables (one for each relation) that stores the incoming tuples while it receives tuples of each relation in turn.
SHJ uses hash functions to reduce the comparison for equi-join.
It also has an extension called XJoin \cite{urhan2001dynamic} to handle disk-resident data.
XJoin becomes a basis for many later designed algorithms, such as \textit{Rate-based Progressive Join} (RPJ) \cite{tao2005rpj} which uses a smarter memory replacement algorithm.
After XJoin,
Kang et al. presented the first formalized 3-step paradigm for window-based stream join \cite{kang2003evaluating}.

Red Black indexing tree based Symmetric Nested Loop Join (RBSNLJ) \cite{ya2006indexed} is one of the first works on processing non-equi stream joins. RBSNLJ partitions the sliding window into subwindows based on the arrival time of tuples. The complex data structures (e.g., the red-black trees in RBSNLJ) are built in subwindows. Only the data structure in the newest subwindow is updated at runtime and the other subwindows reuse their data structures for probing operation.
In this way, the cost of updating the complex data structure is significantly reduced, which increases the overall performance. However, RBSNLJ does not explore how to process a subwindow in parallel. The red-black tree is also unnecessarily complicated and generates a large number of random memory accesses during processing, compared with the data structures used in PanJoin.

To reduce comparisons during probing, some research works focus on increasing the parallelism of join processing. CellJoin \cite{gedik2009celljoin} divides a sliding window into subwindows based on the arrival time of tuples, which is similar to RBSNLJ. CellJoin maps subwindows to processing cores on heterogeneous multicore Cell processors. Because CellJoin implements nested-loop-based join processing, its performance is limited to several thousand tuples per second on a 15-minutes window (several million tuples in total). Nevertheless, we believe CellJoin can easily be extended to support our PanJoin by adding our specially designed data structures into subwindows.

(Low Latency) Handshake Join \cite{teubner2011soccer, roy2014low} parallelizes the join processing by maintaining a bidirectional data flow similar to a doubly linked list. Each node (subwindow) in the flow is mapped to a processing core. New tuples flow from the starting node to the ending node and join with the tuples of the opposite stream saved in all bypassed node. The performance of handshake join is also constrained by its nested-loop-based join processing.

SplitJoin \cite{najafi2016splitjoin} parallelizes join processing in a more straightforward manner. Rather than forwarding tuples through the data flow, SplitJoin stores each tuple in a fixed node. It also splits storage and probing into two separate steps, where probing can be further divided into several independent processes that run in parallel. BiStream \cite{lin2015scalable} processes stream join in a similar fashion and is applied on large-scale distributed stream join systems. BiStream also implements indexes to accelerate equi-join and non-equi-join. Another similar work called ScaleJoin first merges all incoming tuples into one stream, and then it distributes them to \textit{processing threads} (PTs) \cite{gulisano2016scalejoin}. Each tuple is dispatched to all PTs but is stored in only one PT in a round robin fashion. However, (Low Latency) Handshake Join, SplitJoin and ScaleJoin lack discussions on the data management inside the subwindows (nodes or PTs) while PanJoin provides three new data structures for users to chose. Their subwindows also update and expire the tuples frequently, which is inefficient for a complex data structure to manage. PanJoin solves this problem by providing an architecture with a highly simplified expiration mechanism.

Recently FPGAs have received increasingly more attention in accelerating stream processing. Pioneering works show the tremendous potential of FPGAs in stream processing \cite{hagiescu2009computing, woods2010complex, sadoghi2012multi}.
Handshake Join has an FPGA implementation \cite{teubner2011soccer}. ScaleJoin \cite{gulisano2016scalejoin} also has an FPGA version \cite{kritikakis2016fpga} that uses 4 Virtex-6 FPGAs.
In addition to a pure stream join processor, M. Najafi et al. suggest a more general FPGA-based stream processing architecture, Flexible Query Processor (FQP) \cite{najafi2015configurable}, which perform stream join similar to Handshake Join. FQP introduces OP-Block, a connectable stream processing unit, for constructing a Flexible Query Processor to process complex queries. Each OP-Block can be configured as a join core, a projection filter or a selection filter. FQP maps stream processing queries to a chain of OP-Blocks, thus handshake join can be realized as a chain of OP-Blocks which are configured as join cores. Similar to handshake join, in FQP each tuple is passed from the first partition stored in the first OP-Block (near input) to the next partition (OP-Block). Therefore FQP also dynamically partitions the window based on the arrival time of each tuple. To collects result, each OP-Blocks is connected to a result filter which inserts the result into a result aggregation buffer, and then the result is output to the result stream. FQP implements nested-loop join inside each OP-Block. Therefore, as a pioneer work, the peak throughput of FQP is limited. Here, both FQP and the FPGA version of Handshake Join use internal memory on FPGA to store the tuples, which limits the maximum window size to several thousand, where PanJoin's subwindow on FPGA stores data in the external DDR3 memory which decides the maximum size of the window.

\begin{comment}

\subsection{Hardware Acceleration on Relational Database Processing}

In addition to stream processing, many other works explore the usage of FPGAs in relational database processing. \cite{teubner2013data} presents a detailed survey about data processing on FPGAs. \cite{mueller2009data} compares FPGAs and different kinds of CPUs on processing basic database queries and shows that FPGA has better performance and lower energy consumption. This work also introduces a query compiler that translates queries into an FPGA implementation \cite{mueller2009streams}.

A. Putnam et al. proposed an FPGA-based accelerator for large-scale data center services \cite{putnam2014reconfigurable}. It is the first paper that illustrates how costumed FPGAs succeed in a commercial database engine. On FPGA there is a custom multi-core processor with 60 cores that process Free Form Expressions (FFEs). Each processor core is implemented as an 8-stage pipeline. The onboard CPU processes each query and dispatches jobs to the FPGA. The system contains 1632 server notes and has been evaluated by the Bing ranking engine applications, showing an improvement of 95\% in throughput and reduces the tail latency by 29\%.

J. Casper et al. implements selector, merge join core, and merge sort algorithm on FPGAs, and combines them to construct a sort merge join processor \cite{casper2014hardware}. In their work, the selector or the merge join core consists of an array of comparators and input/output control modules. They also implement a sorting tree to realize the merge sort algorithm. The system consists of 4 FPGA working in parallel and 4 DDR3 RAMs running at 400MHz with a total bandwidth of 153.6GB/s. The memory bandwidth utilization varies from 5 to 25 percent.

L. Wu et al. proposed the first hardware accelerator for
range partitioning called \textit{Hardware-Accelerated Range Partitioner} (HARP) \cite{wu2014hardware}. This framework can seamlessly integrate existing hardware and software by plugging an ASIC HARP block into a CPU core. Each HARP can pull and push data through memory controller and compare the input key in a highly pipelined way. The partitioner is programmed by software with extra instructions, including
setting values for each splitter (boundary).

There are many other data processing solutions on hardware. Q100 proposed by L. Wu et al. \cite{wu2014q100} is a relational database query processor which contains a collection of heterogeneous ASIC tiles. It is the first DPU (database processing unit) that targets database applications. A dedicated FPGA accelerator for Large Semantic Web Database was proposed in \cite{werner2013hardware} and reached a speed up for Symmetric Hash Join between 1.3-10.2x compared to the CPU-based counterparts. GROUP BY aggregation on FPGA was also explored in \cite{woods2014ibex} which achieved improvements around 7-11x. \cite{halstead2015fpga} presented an FPGA-based multi-threading solution for in-memory hash join and claimed that the throughput is 2-3.4x faster than multicore solutions. \cite{istvan2015hash} described a new hash table design on FPGA, which is implemented without a large overflowed buffer and outperforms a CPU-based solution by a factor of 10. Recent works also explored  CPU-FPGA heterogeneous architectures to accelerate pattern matching and data partitioning \cite{sidler2017accelerating, kara2017fpga}.

\end{comment}
\section{Introduction}

Stream processing techniques are used in many modern applications to analyze data in real time. In stream processing, stream join is a commonly used operator for relating information in different streams. Many researchers \cite{gulisano2016scalejoin, teubner2011soccer, ananthanarayanan2013photon, lin2015scalable} consider stream join to be one of the most critical and expensive operations in stream processing.

The definition of stream join is extended from the relational (theta) join. A commonly used model is sliding-window-based stream join (henceforth referred to as \textit{stream join} in this paper), where each stream maintains a sliding window and each incoming tuple from one stream is joined with the sliding window(s) of other stream(s). Stream join has been intensively studied over the past two decades \cite{kang2003evaluating, xie2007survey, teubner2011soccer, roy2014low, vitorovic2016squall, najafi2016splitjoin, lin2015scalable, gulisano2016scalejoin}. The main challenge in processing a stream join is producing results at runtime with high throughput, particularly when the window size is large and the input data rate is high. Recent works \cite{teubner2011soccer, roy2014low, najafi2016splitjoin, gulisano2016scalejoin} attempted to address this challenge by dividing a stream window into several subwindows and assigning them to multiple processors or join cores that work in parallel. In this way, their solutions can handle several thousand input tuples per second with a window size of several million. The limitation of these works is that they mainly parallelize stream join at the architectural level: there are few discussions on how to parallelize subwindow internally or design a data structure to accelerate the processing. Furthermore, a recent report from LinkedIn \cite{noghabi2017samza} shows a real-world requirement to handle input rates of 3.5K-150K per second, which is higher than the maximum throughput of the aforementioned works.

\begin{comment}

The third challenge is to add support for non-equi-join. Non-equi-join, especially band join, is useful in many stream processing applications. However, it is poorly supported by the popular stream processing platforms. All of the three most popular platforms, Apache Storm\cite{joinstorm2017doc}, Apache Spark Streaming\cite{joinspark2017doc} and Apache Flink DataStream \cite{joinflink2017doc}, only provide functions for equi-join. Apache Kafka Streams also only supports limited modes of equi-join \cite{joinkafka2017doc}.

To achieve high performance (high throughput and low latency), we can try to partition the sliding window into a serious of subwindows. The tuple then only needs to join with one or several subwindows, which reduces the required comparison for each tuple. This approach also increases the parallelism of stream join. Therefore, a partition method that supports non-equi-join could help us overcome the challenges in designing a stream joiner.

L. Wu et al. have listed three properties a good partitioner should have \cite{wu2014hardware}: \textit{ordered partitions}, whereby there is an order among the value ranges of partitions, thus the input data is sorted in a coarse way; \textit{record order preservation}, whereby the order inside a partition appears the same as the arrival order of tuples; and \textit{skew tolerance}, whereby the partitioner maintains the same throughput regardless of the distribution of tuples among partitions. In this paper, we add a further requirement for \textit{skew tolerance}: the partitioner can distribute the tuple evenly among partitions regardless of the data skew.

There are two partitioning methods commonly used by database system: \textit{hash partitioning} and \textit{range partitioning} \cite{oracledb2017doc}. Both of them satisfy \textit{record order preservation} if the tuples are saved in their arrival order. Hash partitioning is adaptive to data skew since a good hash function can evenly distribute the data regardless of its distribution (assuming the portion of repetitive values is small). However, it does not save the ordering of values among partitions. It is difficult to determine whether a partition contains values that are larger or smaller than a given range. Thus, it can not be used to accelerate non-equi-join. On the other hand, range partitioning holds the ordering information, but it does not adapt to data skew with a fixed partition table. Therefore, if we choose range partitioning to support non-equi-join, we need to find a method that updates the partition table at runtime and separates the result generated by different partition tables.

Since the partition table needs to be updated online, we can take a snapshot of the sliding window in a given period. The snapshot contains the values of all tuples in the current window. We can use this information to create a range partition table that can evenly partition the current window and use the new table for the next period. By analyzing the histograms among current partitions, we designed an algorithm that gives an approximate estimation of the new partition table which partitions the window in a near even fashion. We show that we can get the new partition table after $log_PR$ iterations (P is the partition count, R is the range of tuple values calculated by $Value_{max} - Value_{min}$). For example, given $P=1000$ and $R = 2^{32}-1$ (range of a 32bit integer), we only need a maximum of 4 iterations to calculate the new table. This algorithm also works on floating point values where the precision is close to commonly used values (like 0.01 in currency representation).

Furthermore, we studied the features of stream join and found an important characteristic we should use: given a time period when the value distribution of incoming tuples is unchanged, we can use a few early arriving tuples as samples to construct the partition table for the tuples arriving later. Thus, instead of analyzing the histograms of the entire sliding window, we can divide the window into subwindows based on the arrival time (or order) of tuples and build new partition tables for each hop (subwindow). New tuples are inserted into the newest subwindow and a new partition table is constructed once the newest subwindow is full, while oldest tuples expire from the oldest subwindow and other subwindows stay unchanged. This mechanism is close to the idea used in \cite{ya2006indexed}, where complex data structures (like trees) are built based on subwindows: only the data structure of the newest subwindow is updated, while the data structures of other subwindows stay unchanged and are retained for the use of probing.

\begin{figure}

  \centering
  \includegraphics[width=0.5\textwidth]{diagrams/Sigmod2018_-_Overview.pdf}
  \caption{Overview of PanJoin.}
  \label{fig:overview}
\end{figure}
\end{comment}

In this paper, we first propose a new architecture generalized from an algorithm named Red Black indexing tree based Symmetric Nested Loop Join (RBSNLJ) \cite{ya2006indexed}. In the new architecture, the window is divided into subwindows based on the arrival time of tuples, and subwindows are chained similar to a circular buffer. The new tuples are only inserted into the newest subwindow, and the oldest subwindow(s) is(are) expired as a whole, while the middle subwindows remain unchanged. This architecture is suitable for implementing a complex data structure such as a tree inside each subwindow, since expiration is highly simplified and the processing overhead caused by the remaining expired tuples is decreased by the number of subwindows. We then propose three new data structures that are specifically designed for the subwindows of this architecture. Each of the data structures has advantages over the others under different categories of configuration, e.g., different subwindow sizes or selectivity values. The main idea of these data structures is to further divide a subwindow into several partitions based on the tuple value. In this way, we can reduce a considerable amount of memory access for probing operation compared with the nested-loop join used in the related works \cite{teubner2011soccer, najafi2016splitjoin, gulisano2016scalejoin}, thereby significantly accelerating the overall stream join by more than 1000x. The three data structures can also process requests with multiple threads, and we design some mechanisms for our data structures to handle highly skewed data, where the tuple value is not evenly distributed among the value range. Therefore, we name our algorithm PanJoin (Partitioned Adaptive uNiformization join), which partitions and parallelizes a sliding window at both the architectural level and the subwindow level, as well as adaptively manages highly skewed data to achieve performance as good as managing data following a uniform distribution. Furthermore, since FPGA is commercially applicable in several cloud computing service providers (e.g., IBM, Amazon and Microsoft) and becomes increasingly popular, we decide to implement the most hardware-friendly data structure, named BI-Sort, on FPGA to achieve high throughput and high energy efficiency.

Our experiment shows that PanJoin can handle an input rate of 10M-28M tuples per second for a window size of 8M-1G on a cluster, which is more than 1000x faster than the recently proposed solutions \cite{teubner2011soccer, roy2014low, najafi2016splitjoin, gulisano2016scalejoin}.  Our data structures adapt well to highly skewed data from the real world. The subwindows on FPGA also provide energy-efficient performance compared to the subwindows on CPU.

The contributions of this paper are four-fold:
\begin{enumerate}
\item A new stream join architecture which significantly simplifies expiration operation and avoids communication between worker nodes.
\item Three novel data structures which remarkably reduce the comparisons in probing opeartion over nested-loop join by more than three orders of magnitude.
\item Several innovative data structures (e.g., Linked List Adaptive Table) and algorithms to implement the three new data structures or provide efficient storage strategy.
\item An FPGA solution which has more than 4x energy efficiency over the corresponding CPU solution.
\end{enumerate}

The organization of this paper is as follows: Section \ref{sec:rel} shows and discusses the related works; Section \ref{sec:pan} presents the PanJoin algorithm; Section \ref{sec:imp} presents the implementation of PanJoin on FPGA; Section \ref{sec:eva} shows the performance results of PanJoin.

\section{Headings in Appendices}