\section{INTRODUCTION}
In recent years, great achievements have been made in 3D area \cite{pon2020object,wang2020pointtracknet,battrawy2019lidar,wei2019conditional,qi2017pointnet++,griffiths2020finding}. Beyond 2D perception \cite{deng2009imagenet,simonyan2014very,huang2017densely,ren2015faster,liu2016ssd,he2017mask,li2017fully,wei2018quantization}, 3D scene understanding is indispensable and important because of wide real-world appliations. As one of hot 3D topics , 3D object detection \cite{qin2019monogrnet, li2019stereo, li2019gs3d,srivastava2019learning,simonelli2019disentangling,wang2019pseudo,ku2019improving,brazil2019m3d,liu2019deep,qi2019deep} focuses on the problem of detecting objects' tight bounding boxes in 3D space. It has attracted more and more attention due to eager demand in computer vision and robotics, \emph{e.g. }  autonomous driving, robot navigation and augmented reality. Point clouds are one of the key 3D representations, which consist of points in 3D space. Since point clouds provide rich geometric information, many 3D object detection methods \cite{ku2018joint, yi2020segvoxelnet, zhou2019iou, chen2019fast, yan2018second, liang2019multi,shi2019pv}  use them as inputs.

Thanks to the development of deep learning, many approaches \cite{ mousavian20173d,yan2018second, liang2019multi,shi2019pv} adopt neural networks for 3D object detection, which achieve remarkable performances on various benchmarks \cite{dai2017scannet,geiger2012we,song2015sun}. Despite high accuracy, these learning-based methods require large amounts of 3D groundtruth. However, annotating these 3D data is a complex task demanding accurate quality control. The labeling process is expensive and time-consuming even utilizing crowd-sourcing systems (\emph{e.g. } MTurk).  Compared with 3D labels, Tang \emph{et al.} \cite{tang2019transferable} mention that labeling 2D bounding boxes can be 3-16 times faster. Benefitting from large-scale 2D datasets, such as ImageNet \cite{deng2009imagenet} and MS COCO \cite{lin2014microsoft}, we can also obtain accurate 2D bounding boxes from high-performance 2D detectors. In addition, with the improvement of 3D sensors, it becomes much easier to get high-quality raw 3D training data. Thus, a valuable direction is exploring how to leverage 2D bounding boxes along with sparse point clouds from LiDAR for 3D object detection.

\begin{figure}[tb]
	\centering
	\includegraphics[width=0.85\linewidth]{intro.jpg}
	\caption{An overview of our proposed framework. In our algorithm, we only need 2D labels along with sparse point clouds from LiDAR. We find that semantic information in 2D bounding boxes and 3D structure information in point clouds are enough for 3D vehicle detection.}
	\label{fig:overview}
	\vspace{-6mm}
\end{figure}

 In this work, we propose frustum-aware geometric reasoning (FGR) for weakly supervised 3D vehicle detection. Fig. \ref{fig:overview} illustrates the overview of our method. Our framework only leverages 2D bounding boxes annotations and sparse point clouds. It consists of two components: coarse 3D segmentation and 3D bounding box estimation. In coarse 3D segmentation stage, we first employ RANdom SAmple Consensus (RANSAC) \cite{fischler1981random} to remove the ground part and extract the frustum area for each object. To tackle cases with different point densities, we propose a context-aware adaptive region growing method that leverages contextual information and adjusts the threshold adaptively. The aim of 3D bounding box estimation stage is to calculate 3D bounding boxes of segmented vehicle point clouds.  More specifically, we design an anti-noise framework to estimate bounding rectangles in Bird's Eye View (BEV) and localize the key vertex.  Then, the key vertex along with its edges are used to intersect the frustum. Based on 2D bounding boxes, our pipeline can generate pseudo 3D labels , which are finally used to train a 3D detector. To evaluate our method, we conducted experiments on KITTI Object Detection Benchmark \cite{geiger2012we}. Experimental results show that FGR achieves comparable performance with fully supervised methods and can be applied as a 3D annotator given 2D labels. The code is available at {\small \url{https://github.com/weiyithu/FGR}}.

\section{Related Work}
\noindent \textbf{LiDAR-based 3D Object Detection:}
Obtaining high-quality 3D data, LiDAR-based methods \cite{ku2018joint, yi2020segvoxelnet, zhou2019iou, chen2019fast, yan2018second, liang2019multi, shi2019pv}  are able to predict more accurate results than image-based methods. AVOD \cite{ku2018joint} adopts RPN to generate 3D proposals, which leverages a detection network to perform accurate oriented 3D bounding box regression and category classification. Lang \emph{et al.} \cite{lang2019pointpillars} develop a well-designed encoder called PointPillars to learn the representation of point clouds organized in vertical columns. Similar to PointPillars, PointRCNN \cite{shi2019pointrcnn} also utilizes PointNet \cite{qi2017pointnet} as the backbone. It consists of two stages and directly uses point clouds to produce 3D proposals. Following \cite{shi2019pointrcnn}, to better refine proposals, STD \cite{yang2019std} designs a sparse to dense strategy.  Beyond PointRCNN \cite{shi2019pointrcnn}, Part-$A^2$ \cite{shi2020points} has the part-aware stage and the part-aggregation stage, which can better leverages free-of-charge part supervisions from 3D labels. Compared with two-stage detectors, single-stage detectors \cite{yang20203dssd, he2020structure} are more efficient. 3DSSD \cite{yang20203dssd} adopts a delicate box prediction network including a candidate generation layer and an anchor-free regression head to balance speed and accuracy. SA-SSD \cite{he2020structure} leverages the structure information of 3D point clouds with an auxiliary network .

\noindent \textbf{Frustum-based 3D Object Detection:}
Some of 3D object detection methods \cite{shen2019frustum,qi2018frustum,wang2019frustum} first extract frustum area which is tightly fitted with 2D bounding boxes. In particular, F-PointNet \cite{qi2018frustum} predicts 3D bounding boxes from the points in frustums and achieves efficiency as well as high recall for small objects. It can also handle strong occlusion or cases with very sparse points. Beyond F-PointNet, F-ConvNet \cite{wang2019frustum} aggregates point-wise features as frustum level feature vectors and arrays these aggregated  features as the feature map. It also applies FCN to fuse frustum-level features and refine features over a reduced 3D space. Frustum VoxNet \cite{shen2019frustum} is another work relying on frustums. Instead of using the whole frustums, it determines the most interested parts and voxelize these parts of the frustums.
Inspired by these methods, we also utilize frustum area to calculate 3D bounding boxes.

\noindent \textbf{Weakly Supervised / Semi-supervised 3D Object Detection:}
Seldom works \cite{tang2019transferable, qin2020weakly, meng2020weakly, zhao2020sess} focus on weakly supervised or semi-supervised 3D object detection.  Leveraging 2D labels and part of 3D labels, Tang \emph{et al.} \cite{tang2019transferable} propose a cross-category semi-supervised method (CC), which transfers 3D information from other classes to the current class while maintaining 2D-3D consistency. With weaker supervision, our method still outperforms CC for a large margin. Also without using any 3D labels, VS3D \cite{qin2020weakly} only adopts a pretrained model as the supervision. However, the performance of their method is much worse than fully-supervised methods. Recently, Meng \emph{et al.} \cite{meng2020weakly} propose a two-stage architecture for weakly supervised 3D object detection, which requires a few precisely labeled object instances. Compared with their method, our FGR does not need any 3D labels. Mentioned in \cite{tang2019transferable, qin2020weakly}, annotating 3D bounding boxes is time-consuming and it's worthy exploring 3D object detection from only 2D labels.

\begin{figure*}[tb]
	\centering
	\includegraphics[width=0.95\linewidth]{pipeline.jpg}
	\caption{The pipeline of FGR. Our framework consists of two parts. For coarse 3D segmentation, we first estimate ground plane and remove it from the whole point clouds. Then we adopt context-aware adaptive region growing algorithm to get segmentation mask. However, there still exists noise in segmented points set. To solve this problem, we propose the anti-noise key vertex localization. Finally we use key vertex and key edges to intersect the frustum to predict 3D bounding boxes.}
	\label{fig:pipeline}
	\vspace{-1mm}
\end{figure*}

\section{Frustum-aware Geometric Reasoning}
Given 2D bounding boxes and LiDAR point clouds, our method aims at 3D vehicle detection without 3D labels. Fig. \ref{fig:pipeline} illustrates the pipeline of frustum-aware geometric reasoning. Our framework can generate pseudo 3D labels, which is then used to train a 3D detector.
There are two components in our approach: coarse 3D segmentation and 3D bounding box estimation. We will introduce these two parts respectively in this section.

\subsection{Coarse 3D Segmentation}

\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}

A straightforward method for point clouds segmentation is applying region growing algorithm \cite{adams1994seeded} to separate out objects from frustum area, which is regarded as our baseline. However, our experimental results demonstrate that the baseline method is not accurate.

\noindent \textbf{Ground Plane Estimation:}
We observe that because most of vehicles lying on the ground, ground points will heavily affect region growing results. To alleviate the influence, we first adopt RANSAC \cite{fischler1981random} to estimate ground plane and remove it.

\noindent \textbf{Context-aware Adaptive Region Growing:}
It is difficult to detect the vehicle which is highly occluded by others. However, if we can first detect other vehicles and remove their points from the whole point clouds, it will be much easier to segment current vehicle. Given a series of 2D bounding boxes $\{B_i\}$, we extract frustums and sort them according to the median depth of points in frustums. The vehicles nearer to camera will be executed with algorithm earlier.
\begin{algorithm*}[tb]
	\setlength{\belowcaptionskip}{-20pt}
	\caption{ The pipeline of context-aware adaptive region growing}
	\label{alg}
	\begin{algorithmic}[1]
		\Require Whole point cloud $P_{all}$,  \ points set in frustum region $F^i$ of vehicle $O^i$, \ distance thresholds $\{\phi^k\}$
		\Ensure Segmented points set $M_{best}^i$
		\For{$k=1$ to $n$:}
		\While{ there exists a point $P^i_j$ in $F_i$ which doesn't  belong to any connected component in $\{C_{jk}^{i}\}$}
		\State Set up an empty points set  $C_{jk}^{i}$, add $P^i_j$ to $C_{jk}^{i}$
		\While{there exists a point $p$ in $C_{jk}^{i}$ which has not been processed}
		\State Search points set $q$ from whole point cloud $P_{all}$ whose Euclidean distance to $p$ is smaller than $\phi^k$
		\State For each point in $q$, \ check if the point has already been processed, \  and if not add it to $C_{jk}^{i}$
		\EndWhile
		\State If $\frac{|C_{jk}^{i}  \cap F^i|}{|C_{jk}^{i}|} < \theta_{seg}$, we treat this connected component as a outlier and remove it
		\EndWhile
		\State From connected component set $\{C_{jk}^i\}$, select $M^i_k$ which has the most points as the segmentation result for  $\phi^k$
		\EndFor
		\State From $\{M^i_k\}$, select $M_{best}^i$ which has the most points as the final segmentation result.

	\end{algorithmic}
\end{algorithm*}

Algorithm \ref{alg} shows the pipeline of context-aware adaptive region growing. For a distance threshold $\phi_{k}$, we treat two points as neighbour points if their Euclidean distance is smaller than $\phi_k$. According to this definition, for each vehicle $O^i$, we randomly select a point $P_{j}^i$ in frustum area and  calculate its connected component $C_{jk}^{i}$. We repeat the previous steps until each point in $F^i$ belongs to a connected component. Finally, the vehicle segmentation mask is selected from  $\{C_{jk}^{i}\}$. Among these connected components, we filter out $C_{jk}^{i}$ if it satisfies following criterion:
\begin{equation}
\frac{|C_{jk}^{i}  \cap F^i|}{|C_{jk}^{i}|} < \theta_{seg}
\end{equation}
where $\theta_{seg}$ is a threshold and $| \cdot |$ means the size of the set. This criterion indicates that if the proportion of a connected component inside frustum is smaller than a threshold, we consider this component belongs to other objects but not current vehicle. After removing these outliers, we choose the connected component $M^i_k$ which has the most points as the segmentation set of vehicle $O^i$.

For each threshold $\phi_k$, we calculate segmentation set $M^i_k$. It's reasonable to set different $\phi_k$ for different objects according their point densities. Thus, we traverse threshold set $\{\phi_k\}$ and select the segmented points set which has the most points as the final segmentation result $M_{best}^i$.

\subsection{3D Bounding Box Estimation}
A baseline method to calculate 3D bounding boxes of segmented points set $M_{best}^i$ is to estimate minimum-area rectangle which encases all points. Although this is a mature algorithm, it is sensitive to noise points and doesn't make use of frustum information. Different from this baseline approach, for the vehicle $O^i$, we first locate the key vertex $V^i$ of $M_{best}^i$ in an anti-noise manner. Then based on this key vertex, we calculate the final 3D bounding box by computing the intersection with frustum boundaries. This stage is conducted on Bird's Eye View (BEV).

\begin{figure}[tb]
	\centering
	\includegraphics[width=0.9\linewidth]{vertex.pdf}
	\caption{The illustration of anti-noise key vertex localization. Adopting a iterative framework, our method can detect noise points and remove them. The algorithm will stop if key vertex's position changes slightly between two iterations.}
	\label{fig:vertex}
	\vspace{-6mm}
\end{figure}

\noindent \textbf{Anti-noise Key Vertex Localization:}
Fig. \ref{fig:vertex} illustrates the procedure of anti-noise key vertex localization. For a bounding rectangle, we can separate out four right triangles according to its four vertexes. We define the right angle vertex of the triangle containing the most points as the key vertex and its corresponding legs as key edges.

\begin{figure}[tb]
	\centering
	\includegraphics[width=0.7\linewidth]{ambiguity.png}
	\caption{Using the area as the objective function will bring ambiguity. For the points in the red triangle, there are two optimal bounding rectangles (yellow rectangle and green rectangle have the same area). However, the yellow rectangle is what we want. }
	\label{fig:ambiguity}
	\vspace{-2mm}
\end{figure}

In many cases, we can only observe points in one right triangle. If we use the area as the objective function to calculate the optimal bounding rectangle, we will get two results, which are shown in Fig. \ref{fig:ambiguity}. To address this problem, we design a new objective function. We assume that we want to estimate the bounding rectangle of points set $Q$, and key edges are $l_1, l_2$ respectively. The following equation describes our new objective equation $f$:

\begin{equation}
\begin{aligned}
f &= \frac{|Q_{0}|}{|Q|} \\
Q_{0} = \{q|\ q \in Q, &\ ||q, l_1|| > \theta_{rect} , \ ||q, l_2|| > \theta_{rect}\}
\end{aligned}
\end{equation}
where $||q, l||$ is the vertical distance between point $q$ and edge $l$, and $\theta_{rect}$ is the threshold. If most points are close to key edges, we think this bounding rectangle is well-estimated. We first enumerate rectangles encasing $Q$ whose orientations are from 0 degree to 90 degrees with 0.5 degree interval. Then we select one which has minimum $f$ as the bounding rectangle of $Q$.

Because accurate segmentation labels are unavailable, our predicted segmentation masks always obtain noise, which will heavily influence the calculation of bounding rectangles. For the noise point, if we delete it from $Q$ and recompute bounding rectangle, the position of the key vertex will vary a lot. In contrast, the key vertex is stable if we remove a few inlier points because there are many points lying on the key edges. In light of this discovery, we develop an iterative framework. For an estimated bounding rectangle, we delete points near to the key edges and recompute a new bounding rectangle. If the key vertex changes slightly, we treat the key vertex of this new bounding rectangle as $V^i$, and if not we will repeat above process until satisfying this condition.

\begin{figure}[tb]
	\centering
	\includegraphics[width=0.8\linewidth]{intersection.pdf}
	\caption{We use the key vertex and key edges to intersect the frustum. From these intersection points, we can get final 3D bounding box. }
	\label{fig:intersection}
	\vspace{-5mm}
\end{figure}

\noindent \textbf{Frustum Intersection:}
Due to the occlusion, the observed points sometimes only cover a part of the whole vehicle. In other words, it's inaccurate to directly use bounding rectangles as the final prediction. Fortunately, we are able to compute accurate results leveraging 2D bounding boxes (frustums). From Fig. \ref{fig:intersection}, we can easily get 8 vertexes of the 3D bounding boxes given the key vertex and key edges of bounding rectangles.

\section{Experiments}
In this section, we conduct extensive experiments to verify the effectiveness of our method. First we will describe the dataset and implementation details. Then we will exhibit main results on KITTI dataset \cite{geiger2012we}. Experiments in \S\ref{sec:ablation} verify the rationality of our method. Finally, we will show some qualitative results.

\subsection{Dataset and Implementation Details}
\noindent \textbf{KITTI Dataset:}
As a frequently used 3D object detection benchmark, the KITTI dataset  \cite{geiger2012we} contains 7,481 training pairs and 7,518 testing pairs of RGB images and point clouds. We follow \cite{chen2017multi} to divide original training samples into train split (3712 samples) and val split (3769 samples). Our algorithm was evaluated on car category samples. For experiments using 3D detectors, we report Average Precision for 3D and  Bird's Eye View (BEV) tasks. According to the 2D bounding box height, occlusion and truncation level, the benchmark is divided into three difficulty levels: Easy, Moderate and Hard. When evaluating the quality of generated labels, we set mean IoU and precision as evaluation metric.  To better evaluate precision, we utilized IoU threshold of 0.3, 0.5 and 0.7 respectively.

\noindent \textbf{Implementation Details:}
In KITTI dataset \cite{geiger2012we}, vehicles' roll/pitch angle can be seemed as zero and we only need to estimate yaw angle (\emph{i.e. } orientation). Therefore, we can leverage Bird's Eye View (BEV) to detect vehicles instead of operating on 3D space. We set threshold $\theta_{seg}$ as 0.8  for context-aware adaptive region growing. $\{\phi^k\}$ was enumerated from 0.1 to 0.7 with 0.1 interval. As for anti-noise key vertex localization,  we choosed one-tenth length of key edges as $\theta_{rect}$. During the localization process, if the change of key vertex's position was smaller than 0.01 meter, we would stop iterative algorithm and get final bounding rectangles.  PointRCNN \cite{shi2019pointrcnn} was adopted as our 3D detector backbone. We finally filtered 3D bounding boxes with extremely wrong sizes.

\begin{table*}[t!]
	\centering
	\caption{Performance comparison with other fully supervised methods on KITTI test set.}
	\resizebox{0.7\textwidth}{!}{
		\begin{tabular}{|c|c||ccc|ccc|}
			\hline
			\multirow{2}{*}{Method}  & \multirow{2}{*}{3D labels}& \multicolumn{3}{c|}{AP\textsubscript{3D}(IoU=0.7)} & \multicolumn{3}{c|}{AP\textsubscript{BEV}(IoU=0.7)}   \\
			 & & \multicolumn{1}{c}{Easy} & \multicolumn{1}{c}{Moderate} & Hard
			 & \multicolumn{1}{c}{Easy} & \multicolumn{1}{c}{Moderate} & Hard     \\   \hline
			MV3D \cite{chen2017multi}& \checkmark &74.97& 63.63 &	54.00 &86.62 & 78.93 &69.80  \\
			AVOD \cite{ku2018joint}& \checkmark & 76.39 &66.47 & 60.23 &89.75& 84.95&	78.32  \\
			F-PointNet \cite{qi2018frustum} & \checkmark&  82.19 &69.79 &60.59 &91.17 &84.67 &74.77  \\
			SECOND \cite{yan2018second}& \checkmark&  83.34 &72.55 &65.82&89.39 &83.77 &78.59  \\
            PointPillars \cite{lang2019pointpillars} & \checkmark&  82.58 &74.31 &68.99&90.07 &86.56 &82.81  \\
			PointRCNN \cite{shi2019pointrcnn}& \checkmark&  86.96 &75.64 &70.70 & 92.13& 87.39& 82.72   \\
			SegVoxelNet \cite{yi2020segvoxelnet}& \checkmark&  86.04 &76.13 &70.76 & 91.62& 86.37& 83.04   \\
			F-ConvNet \cite{wang2019frustum} & \checkmark&87.36 &76.39 &66.69& 91.51 & 85.84 &76.11 \\
			Part-$A^2$ \cite{shi2020points} & \checkmark&87.81 &78.49&	73.51 &	91.70 & 87.79 & 84.61  \\
			SASSD \cite{he2020structure} & \checkmark&88.75 &79.79 &74.16  & 95.03 & 91.03 &85.96  \\ \hline

			FGR  & $\times$& 80.26& 68.47& 61.57& 90.64& 82.67& 75.46\\
			 \hline
	\end{tabular}}

	\vspace{-4mm}
	\label{tab:test}
\end{table*}

\begin{table}[tb]
	\centering
	\caption{Performance comparison with other weakly supervised methods on KITTI val set.}
	\resizebox{0.37\textwidth}{!}{
		\begin{tabular}{|c||c|c|c|}
			\hline
            \multirow{2}{*}{Method} &  \multicolumn{3}{c|}{AP\textsubscript{3D}(Easy)}        \\
			   &  \multicolumn{1}{c}{IoU=0.25} &\multicolumn{1}{c}{IoU=0.5} &\multicolumn{1}{c|}{IoU=0.7 }     \\   \hline
            CC \cite{tang2019transferable} &69.78 &- &- \\
		    VS3D \cite{qin2020weakly}& - & 40.32& - \\
            WS3D \cite{meng2020weakly}& - &- & 84.04
			\\ \hline
			FGR & \bf{97.24} & \bf{97.08} & \bf{86.11}  \\
			\hline
	\end{tabular}}

	\label{tab:val}
\end{table}

\iffalse
\begin{table}[tb]
	\centering
	\caption{Performance comparison with other weakly supervised methods on KITTI val set.}
	\resizebox{0.4\textwidth}{!}{
		\begin{tabular}{|c|c||c|c|c|}
			\hline
			\multirow{2}{*}{IoU} & \multirow{2}{*}{Method} &  \multicolumn{3}{c|}{AP\textsubscript{3D}}        \\
			 &    &  \multicolumn{1}{c}{Easy} &\multicolumn{1}{c}{Moderate} &\multicolumn{1}{c|}{Hard }     \\   \hline
			 \multirow{2}{*}{0.25} &CC \cite{tang2019transferable} &69.78 &58.66 &51.40 \\
			&FGR  & \bf{97.24} & \bf{89.67} & \bf{89.06}  \\ \hline
			\multirow{2}{*}{0.5}&VS3D \cite{qin2020weakly}& 40.32 & 37.36  & 31.09 \\
			&FGR &  \bf{97.08} & \bf{89.55} & \bf{88.76}  \\  \hline
			\multirow{2}{*}{0.7}&WS3D \cite{meng2020weakly}& 84.04 &75.10& \bf{73.29}
			\\
			 &FGR & \bf{86.11} & \bf{74.86} & 67.87  \\
			\hline
	\end{tabular}}

	\label{tab:val}
\end{table}
\fi

\begin{table}[tb]
	\centering
	\caption{Evaluation results with different 3D backbones on KITTI val set}
	\resizebox{0.45\textwidth}{!}{
		\begin{tabular}{|c|c||ccc|}
			\hline
			\multirow{2}{*}{Method} & \multirow{2}{*}{3D labels} &   \multicolumn{3}{c|}{AP\textsubscript{3D}}        \\
			&    &  \multicolumn{1}{c}{Easy} &\multicolumn{1}{c}{Moderate} &\multicolumn{1}{c|}{Hard}     \\   \hline
			PointRCNN \cite{shi2019pointrcnn} & \checkmark& 88.88 &78.63& 77.38
			 \\
			F-ConvNet \cite{wang2019frustum} & \checkmark&  89.02 &78.80 &77.09  \\
			Part-$A^2$ \cite{shi2020points} & \checkmark&  89.47& 79.47 & 78.54    \\ \hline
			Ours (PointRCNN) & $\times$  &86.11 & 74.86 & 67.87  \\
			Ours (F-ConvNet) & $\times$  & 86.40 & 73.87 & 67.03  \\
			Ours (Part-$A^2$) & $\times$  & 86.33 & 73.75 & 67.53  \\
			\hline
	\end{tabular}}
	\vspace{-4mm}
	\label{tab:backbone}
\end{table}
\iffalse
\begin{table*}[t!]
	\centering
	\caption{Performance comparison with other methods on KITTI val set.}
	\resizebox{0.95\textwidth}{!}{
	\begin{tabular}{|c||ccc|ccc|ccc|ccc|}
		\hline
		\multirow{2}{*}{Method} &  \multicolumn{3}{c|}{AP\textsubscript{3D}(IoU=0.5)} & \multicolumn{3}{c|}{AP\textsubscript{BEV}(IoU=0.5)} & \multicolumn{3}{c|}{AP\textsubscript{3D}(IoU=0.7)} & \multicolumn{3}{c|}{AP\textsubscript{BEV}(IoU=0.7)}        \\
		&                      \multicolumn{1}{c}{Easy} & \multicolumn{1}{c}{Moderate} & Hard              & \multicolumn{1}{c}{Easy} & \multicolumn{1}{c}{Moderate} & Hard  & \multicolumn{1}{c}{Easy} & \multicolumn{1}{c}{Moderate} & Hard    & \multicolumn{1}{c}{Easy} & \multicolumn{1}{c}{Moderate} & Hard     \\   \hline
		MonoGRNet\cite{qin2019monogrnet}  &53.63& 44.29& 33.26 &56.49& 47.37& 38.39  & 16.46& 13.61& 10.36 &28.47& 23.98& 19.57 \\
		Stereo R-CNN \cite{li2019stereo}  &84.90 & 75.35&59.07 &84.78 &76.23& 65.75 &  62.78 &47.32& 39.81 &71.50&58.64&49.65 \\
		F-PointNet  \cite{qi2018frustum} & 98.14 & 88.58& 78.55&  98.31& 88.88& 78.89  &  86.49& 81.11& 65.52&  95.28& 85.34&68.65\\
		F-ConvNet  \cite{wang2019frustum} &99.09&90.57& 80.44 & 99.14& 90.58& 80.49  & 89.04& 86.88& 70.29&  98.10&89.90& 71.90\\  \hline
		CubeSLAM  \cite{yang2019cubeslam} & 1.20 & 0.59& 0.08 &1.45  &0.74 &0.19  &0.00 & 0.00& 0.00&0.00 &0.00 &0.00\\
		CC  \cite{tang2019transferable} &51.32& 42.56& 31.22 &57.19& 46.63& 36.78  & 14.89& 12.91& 9.95 &27.76& 23.04& 18.90\\
		Ours  &84.98 &76.12 &66.84 &84.89& 84.22& 67.93 &  64.41& 57.90& 44.77& 74.48& 68.92& 54.71\\ \hline
	\end{tabular}}

	\vspace{-4mm}
	\label{tab:detector}
\end{table*}
\fi

\subsection{Main Results}
Table \ref{tab:test} shows the comparison with other fully supervised methods on KITTI test set. Despite without using any 3D labels, our FGR still achieves comparable performance with some fully supervised methods. We also compare FGR with other weakly-supervised methods on KITTI validation set. However, the supervision of these methods are different with ours (\emph{i.e. } CC\cite{tang2019transferable} uses 2D labels and 3D labels from other classes, VS3D \cite{qin2020weakly} uses a 2D pretrained model and WS3D \cite{meng2020weakly} uses part of 3D labels) and we just report these results for reference.  From Table \ref{tab:val}, we can see that our approach outperforms these weakly supervised methods.  We should notice that despite the supervision of our method is weaker than CC\cite{tang2019transferable} , FGR still outperforms it for a large margin. To better demonstrate the effectiveness of FGR, we also show results using other 3d detector backbones in Table \ref{tab:backbone}. In addition, compared with fully supervised baselines, the performances of our FGR degrade marginally, which indicates that a potential application of FGR is to annotate 3D data given 2D bounding boxes and point clouds. These experimental results confirm that we are able to detect vehicles accurately without the aid of 3D groundtruth.

\subsection{Ablation Studies}  \label{sec:ablation}
We conducted experiments to confirm the effectiveness of each module in our method. To decouple the influence of 3D detectors, we evaluate the quality of generated labels in this subsection.
\begin{table}[tb]
		\vspace{-5mm}
	\centering
	\caption{Ablation studies for coarse 3D segmentation. }
	\resizebox{0.48\textwidth}{!}{
	\begin{tabular}{|c|c||c|c|c|c|}
		\hline
		ground plane & \multirow{2}{*}{adaptive} & \multirow{2}{*}{Mean IoU}  &   \multicolumn{3}{c|}{Precision}        \\
		estimation&  &   &  \multicolumn{1}{c}{IoU=0.3} &\multicolumn{1}{c}{IoU=0.5} &\multicolumn{1}{c|}{IoU=0.7}     \\   \hline
		& & 0.5311 & 74.40 & 61.05 & 41.09\\
		$\checkmark$& & 0.6563 & 84.82 & 74.34 & 58.96 \\
		$\checkmark$&$\checkmark$ &  \bf{0.7845} & \bf{97.90} & \bf{96.70} & \bf{83.28}   \\ \hline
	\end{tabular}}

	\label{tab:ablation_seg}

\end{table}

\begin{figure}[tb]
	\centering
	\includegraphics[width=0.9\linewidth]{seg.pdf}
	\caption{Ablation studies for coarse 3D segmentation. Without ground plane estimation, ground points will be segmented as inlier points. Also, the best threshold for different vehicles varys according to point densities. Yellow bounding boxes indicate groundtruth. Best viewed in BEV.  }
	\label{fig:ablation_seg}
	\vspace{-5mm}
\end{figure}

\noindent \textbf{Coarse 3D Segmentation:}
We did ablation studies to verify the necessity of ground plane estimation and effectiveness of context-aware adaptive region growing. We fixed $\phi$ as 0.4 for the experiment that did not use adaptive thresholds. Table \ref{tab:ablation_seg} and Fig. \ref{fig:ablation_seg} illustrate quantitative and qualitative results respectively. From the second column in the figure, we can see that some ground points will be treated as inliers without ground plane estimation. In this way, the points of the vehicle will connect other regions through ground points, which accounts for a failed segmentation. From the third and fourth columns, we conclude that a fixed threshold $\phi$ is unable to handle all cases. On the one hand, if $\phi$  is too small, we will get incomplete segmentation masks (the first and second rows). On the other hand, if $\phi$  is too large, the segmented set will include many outliers (the third row).  The best threshold $\phi$ for these three cases are 0.7, 0.6 and 0.3 respectively.

\begin{figure*}[tb]
	\centering
	\includegraphics[width=1.0\linewidth]{visual.pdf}
	\caption{Qualitative results. We illustrate RGB images, Bird's Eye View (BEV) and LiDAR point clouds respectively. Predicted bounding boxes are drawn in purple while groundtruth is in yellow.}
	\label{fig:visual}
	\vspace{-3mm}
\end{figure*}
\begin{figure}
	\centering
	\includegraphics[width=0.9\linewidth]{rect.pdf}
	\caption{Ablation studies for 3D bounding box estimation. If we adopt baseline method,  the bounding rectangle will include noise points. Due to the occlusion, it's necessary to implement frustum intersection to better leverage 2D bounding boxes. Yellow bounding boxes indicate groundtruth while purple ones are prediction results. Green points represent key vertexes and cyan lines are frustum boundaries.  Best viewed in BEV.}
	\label{fig:ablation_rect}

\end{figure}
\begin{table}[tb]
	\centering
	\caption{Ablation studies for 3D bounding box estimation. }
	\resizebox{0.48\textwidth}{!}{
		\begin{tabular}{|c|c||c|c|c|c|}
			\hline
			anti-noise key & frustum & \multirow{2}{*}{Mean IoU}  &   \multicolumn{3}{c|}{Precision}        \\
			vertex localization & intersection &   &  \multicolumn{1}{c}{IoU=0.3} &\multicolumn{1}{c}{IoU=0.5} &\multicolumn{1}{c|}{IoU=0.7}     \\   \hline
			& & 0.5418 & 76.42 & 56.25 & 41.32\\
			$\checkmark$& & 0.6255 & 89.37 & 69.31 & 54.50 \\
			$\checkmark$&$\checkmark$ &  \bf{0.7845} & \bf{97.90} & \bf{96.70} & \bf{83.28}  \\ \hline
	\end{tabular}}

	\label{tab:ablation_rect}
	\vspace{-3mm}
\end{table}
\noindent \textbf{3D Bounding Box Estimation:}
In this part, we incrementally applied anti-noise key localization module and frustum intersection module to our framework. Table \ref{tab:ablation_rect} and Fig. \ref{fig:ablation_rect} exhibit experimental results. From the second column in the figure, we find that there still exists noise points after coarse 3D segmentation. If we directly operate on these noise points, we will get loose bounding rectangles. In addition, attributed to the occlusion, bounding rectangles tend to be smaller than groundtruth (the third column in Fig. \ref{fig:ablation_rect}). We will get wrong results if we directly use these bounding boxes. However, the position of the key vertex is correct, which can be leveraged to intersect frustums along with key edges.

\subsection{Qualitative Results}
To better illustrate the superiority of our method, we visualize some generated labels in Fig. \ref{fig:visual}. For simple cases which are non-occluded and have enough points, our method can achieve remarkable accuracy. Moreover, we are surprised to find that our method is also able to handle occlusion cases. These qualitative results demonstrate the effectiveness of FGR and also verify that it's possible to combine semantic information in 2D bounding boxes and 3D structure in point clouds for vehicle detection.

\section{Conclusion}
In this paper, we propose frustum-aware geometric reasoning for 3D vehicle detection without using 3D labels. The proposed framework consists of coarse 3D segmentation and 3D bounding box estimation modules. Our method achieves promising performance compared with fully supervised methods. Experimental results indicate that a potential application of FGR is to annotate 3D labels.

\section*{ACKNOWLEDGMENT}

This work was supported in part by the National Natural Science Foundation of China under Grant U1713214, Grant U1813218, Grant 61822603, in part by Beijing Academy of Artificial Intelligence (BAAI), and in part by a grant from the Institute for Guo Qiang, Tsinghua University.

{\small
\bibliographystyle{IEEEtran}


\section{Introduction}

Knowledge Base Population (KBP) is the process of populating (or building from scratch) a Knowledge Base (KB) with new knowledge elements. A considerable number of Natural Language Processing (NLP) tasks, such as question answering, use knowledge bases. Thus, there is an unfading need for more completed and populated knowledge bases.

FarsBase \cite{asgari2019farsbase} is the first knowledge base in the Persian language. Despite other knowledge bases like DBpedia~\cite{auer2007dbpedia} and BabelNet~\cite{navigli2010babelnet}, which minimally support the Persian language, FarsBase is specially designed for the Persian Language. Similar to other knowledge bases, FarsBase faces some severe challenges such as staying up-to-date and expanding with existing knowledge. Some knowledge bases such as Wikidata~\cite{vrandevcic2014wikidata} rely on the human resources to annotate structured data and to prevent the entrance of erroneous knowledge instances to the knowledge base. Unlike Wikidata, FarsBase does not have such community support, which emphasized the need for a system to extract knowledge automatically and to prevent wrong relation instances from being passed to the knowledge base. In this paper, we introduce FarsBase Knowledge Base Population System (FarsBase-KBP) to address this issue.

Note that, while our KBP system uses different modules employing different methods for relation extraction, the extracted knowledge by them should be checked to detect possible redundancy or conflict. For example, if two modules extract two different birthdays for the same person, it means that one or both of these modules have generated erroneous fact, indicating that there is a conflict that should be detected. Additionally, there is also the problem of mapping predicates, subjects, and objects in the extracted triples, which can be extracted as raw text, to the canonicalized predicates and entities in the knowledge base. We address these issues using canonicalization techniques. It should also be noted that despite English and other high-resource languages which can rely on annotated data and use supervised methods, in Persian, which is considered a low-resource language, due to the lack of required training gold datasets, supervised methods are generally not applicable. To overcome this problem, we employed unsupervised and distantly supervised methods which do not require such data.

Our contributions are as follows:
\begin{enumerate}
    \item Unlike Relation Extraction (RE) systems, in an Information Extraction (IE) system, the relations are not canonicalized, the arguments of the extracted relations are in the form of plain text, and they are not linked to an existing knowledge base. To address this issue, we proposed an entity linking system which links subjects and objects to knowledge base entities, alongside with a canonicalization system which maps relations to knowledge (see section \ref{sec:mapper}). The entity linking system applies an entity linking method for the Persian language, namely ParsEL~\cite{asgaribidhendi2020parsel}, to link the arguments of the relation as the subject and object in a FarsBase triple. For the first time, we introduce a canonicalization system which is designed especially for the Persian language and links relationship types to the pre-defined FarsBase predicate set. In this linking process, we use current triples and mapping data in FarsBase, as well as extracted knowledge from other FarsBase-KBP modules. Both entity linking and canonicalization systems are state-of-the-art in the Persian language.
    \item To the best of our knowledge, there are few studies on relation and knowledge extraction for the Persian language, and above that, there are no studies on canonicalization (relation to knowledge mapping) and knowledge fusion in the Persian language as well. Here we propose four modules for information extraction and two modules for relation extraction; all of them are innovative and state-of-the-art methods for the Persian language. For example, the Dependency Pattern (DP) and Persian Syntactic Information Extractor (PSIE{}) modules introduced and devised for the first time in FarsBase-KBP. TokensRegex is also used in the Persian language for the first time and uses FarsBase hirechial ontology classes instead NER.
    \item For the first time, we introduce and publish a gold dataset by which we evaluate the performance of knowledge extraction in FarsBase-KBP. This dataset contains 22015 sentences, in which the entities and relation types are linked to the FarsBase ontology. This gold dataset can be reused for benchmarking KBP systems in the Persian language.
\end{enumerate}

The rest of this paper is structured as follows. In section \ref{related-work}, a review of the literature of knowledge base population will be presented, then we mention other researches that employ knowledge fusion to make use of different sources for knowledge extraction, and finally, we discuss studies on knowledge extraction and knowledge base population for the Persian language. In section \ref{proposed-approach}, we give a brief background of other researches related to each of the modules of FarsBase-KBP, and then we explain how each component of our system operates and how all these components work together to improve the quality of the extracted knowledge. In the last two sections, we present the results of our experiments and the obtained conclusions.

\section{Related Work}
\label{related-work}
Knowledge Base Population is defined as the process of extending a knowledge base with information extracted from the text. The goal is to update the knowledge base or keeping it current with new information \cite{glass2018dataset}.
In this section, we first review literature in the field of knowledge base population, then we categorize and describe studies in this field. The last subsection presents related studies focused on the Persian language.
\subsection{Automatic Knowledge Base Population}
Automatic knowledge base construction and population have recently received significant attention in academic researches. As the size of the knowledge bases kept growing, the need for automatic construction and population of knowledge bases arose. Existing knowledge bases are usually highly incomplete. For example, only 6.2\% of \textit{persons} from Freebase \cite{bollacker2008freebase} have \textit{place of birth} \cite{min2013distant}, and only 1\% of them have \textit{ethnicity} \cite{west2014knowledge}. Additionally, manual completion of existing knowledge bases is expensive and time-consuming. Therefore, automatic construction of knowledge bases from scratch, populating them with missing information, and adding new knowledge to them has attracted a lot of academic attention \cite{adel2018deep}.

Text Analysis Conference (TAC) is an annual series of open technology evaluations organized by the National Institute of Standards and Technology (NIST). The KBP track of TAC encourages the development of the systems that can extract information from unstructured text in order to populate an existing knowledge base or to construct a cold-start (built from scratch) knowledge base \cite{getman2018laying}. TAC KBP track consists of several tasks such as entity discovery and linking, and slot filling \cite{adel2018deep}. Slot Filling is a version of KBP where specific missing information (slots) are searched through the document collection and filled with desired values \cite{glass2018dataset}.

Knowledge base population task is a follow-up task to relation extraction. KBP systems usually consist of one or more relation extractors or knowledge extractors. Relation extraction and knowledge extraction are well studied, yet growing fields of research. These extractors usually utilize basic NLP tasks such as Named Entity Recognition (NER), dependency or constituency parsing, and Entity Linking (EL) to find triples (subject, object, and predicate). Some systems such as FRED \cite{gangemi2017semantic} propose the integration of a stack of native Semantic Web (SW) machine reading tasks. FRED is a machine reader for the semantic web which extracts knowledge from the raw text in the form of RDF graph representation. This extracted knowledge can be used to populate a knowledge base \cite{consoli2015using}. We will provide more details about relation and knowledge extraction literature in the proposed approach section when FarsBase-KBP extractor modules are described.
\subsection{KBP Categories}
Regarding their method of extraction and source of information, there are four main categories of literature in this field \cite{dong2014knowledge}

\textbf{Built on Structured Data:} Approaches which populate knowledge bases using structured data sources (like Wikipedia infoboxes), such as DBpedia \cite{auer2007dbpedia}, Freebase \cite{bollacker2008freebase} YAGO \cite{suchanek2007yago}, and YAGO2 \cite{hoffart2013yago2}.
Yago facts are automatically extracted from Wikipedia, using a combination of heuristic and rule-based methods. YAGO2 is a more recent extension of YAGO in which entities, facts, and events are anchored in both time and space.

\textbf{Open Information Extraction, Web-scale:} Approaches which apply open Information Extraction (IE) techniques which are applied to the entire Web, ranging from Reverb~\cite{fader2011identifying}, PRISMATIC~\cite{fan2010prismatic} and OLLIE~\cite{schmitz2012open} to MINIE~\cite{gashteovski2017minie} and Graphene~\cite{cetto2018graphene}. Graphene is a recent Open IE approach that presents a two-layered hierarchical representation of syntactically simplified sentences in the form of core facts and accompanying contexts that are semantically linked by rhetorical relations.

\textbf{Making Taxonomies:} Approaches which construct taxonomies (is-a hierarchies), as opposed to general KBs with multiple types of predicates such as Probase \cite{wu2012probase}. Probase is a universal, general-purpose, and probabilistic taxonomy which is automatically constructed from a corpus of 1.6 billion web pages. It consists of an iterative learning algorithm to extract ``is-a'' pairs from web texts, plus a taxonomy construction algorithm to connect these pairs to a hierarchical structure.

\textbf{Fixed Ontology, Web-scale:} Approaches which extract information from the entire web, but use a fixed ontology (schema) such as PROSPERA~\cite{nakashole2011scalable}, DeepDive~\cite{niu2012deepdive}, Knowledge Vault~\cite{dong2014knowledge}, and Never-Ending Language Learner (NELL)~\cite{carlson2010toward}. NELL is the first knowledge base that uses automatic extraction of knowledge with very little human intervention. The original architecture of NELL consists of four knowledge extraction components. The knowledge integrator module handles knowledge fusion in NELL. This module promotes knowledge instances to beliefs if it is extracted by one high-confidence source or by multiple sources. The latest version of NELL \cite{mitchell2018never} keeps the same architecture plus five new extractor modules are added, such as an image classifier. Other more recent knowledge bases employ automatic knowledge extraction methods, such as DeepDive and Knowledge Vault. The main difference between Knowledge Vault and NELL is that Knowledge Vault fuses facts extracted from the text with the prior knowledge derived from the Freebase~\cite{bollacker2008freebase}.

The proposed knowledge base population system, FarsBase-KBP, is classified in the last category. FarsBase-KBP knowledge fusion module utilizes a similar approach as knowledge integrator in NELL, although despite NELL, it links the entities of the extracted triples to FarsBase. FarsBase-KBP and NELL also differ in how human intervenes in the knowledge fusion module. In NELL, the triples are transferred directly to the knowledge base after the fusion stage, while at a limited daily time window, some of the knowledge base triples are examined by experts, and false triples are identified. However, in FarsBase-KBP, any triple extracted by the fusion module must be checked by a human expert and added to the knowledge base if approved.

\subsection{Entity Linking}
In the context of KBP, Entity Linking (EL) is the task of mapping all of the subjects and some of the objects of the triples in a raw text to their corresponding entities in a knowledge base. With the appearance of FarsBase, EL has become a possible task for the Persian language as well. In our previous work, we proposed ParsEL~\cite{asgaribidhendi2020parsel}, which is a language-independent end-to-end entity linking system that utilizes both context-dependent and context-independent features for entity linking. ParsEL is the first entity linking system applied to the Persian language with FarsBase as the external dataset. Our experiments showed that the proposed method outperforms Babelfy~\cite{moro2014entity} as the state-of-the-art of multilingual end-to-end entity linking algorithm.

\subsection{Information Extraction}
Information extraction (IE) is one of the essential tasks in NLP. Its purpose is to extract structured information from raw, unstructured text. An IE system receives the raw text and generates a set of triples or n-ary propositions, usually in the form of (subject, predicate, object) as structured information, in which the predicate is a part of the raw text that represents the relationship between the subject and some objects~\cite{shi2019brief}.

Roy et al. provided a new supervised OIE approach that uses a set of unsupervised OIE systems and a small amount of tagged data. As its input features, this method utilizes the output of several unsupervised OIE systems as well as a diverse set of lexical and syntax information including word embedding, POS embedding, syntactic role embedding and dependency structure~\cite{roy2019supervising}.

\subsection{Relation Extraction}
Relation Extraction (RE) is a specific case of IE, in which entities and semantic (mapped) relations between them are identified in the input text.  In other words, an RE system can predict whether the input text has a relationship for some arguments or not. Besides, the RE system must predict which relational class from a particular ontology predicts the identified relation of the input text. Supervised RE methods require training datasets to learn the model. Generating such annotated datasets for RE is time-consuming and expensive, hence resource-poor languages lack of such datasets. In a review study, Shi et al. Showed that if it is not possible to use supervised methods for relation extraction, distant supervision methods will yield the best results~\cite{shi2019brief}.

Trisedya et al. proposed an end-to-end RE model for KB enrichment based on a neural encoder-decoder model, utilizing distantly supervised training data with co-reference resolution and paraphrase detection~\cite{trisedya2019neural}. Gao and his colleagues have proposed Neural Snowball, a new bootstrapping method for learning new relations by transferring semantic knowledge about existing relations. They designed Relation Siamese Networks (RelSN) to learn the criteria for the similarity of relations based on their tagged data and existing relations~\cite{gao2019neural}. Smirnova and Cudré-Mauroux have reviewed RE methods utilizing distant supervision and summarized the two main challenges in this field, noisy labels automatically collected from the KB and the evaluation and training problems induced by the incompleteness of the KB~\cite{smirnova2018relation}.

\subsection{Canonicalization}
One of the essential systems which contribute to construction and population of KBs are Open Information Extraction (OIE) approaches.  However, one fundamental problem is that the relation phrases in the extracted triples of OIE system are not linked or mapped to the knowledge base ontology predicates, which leads to a large number of ambiguous and redundant triples. Canonicalization is the task of mapping the plain text relation phrases to appropriate predicates in the knowledge base. The quality of canonicalization can directly affect the quality of the KBP system.

Various methods have been proposed for canonicalization. Putri et al. proposed a novel approach based on distant supervision and a Siamese network that compares two sequences of word embeddings, representing an OIE relation and a predefined KB relation~\cite{putri2019aligning}. Vashishth et al. proposed CESI~\cite{vashishth2018cesi}, a system for Canonicalization using Embeddings and Side Information which is a novel approach that performs canonicalization above learned embeddings of Open KBs. Side information earned from AMIE~\cite{galarraga2013amie} and Stanford KBP~\cite{surdeanu2012multi}. Lin and Chen proposed an approach for canonicalization which utilizes the side information of the original data sources, including the entity linking knowledge, the types of the candidate entities detected, as well as the domain knowledge of the source text. They jointly modelled the canonicalization problem of entity and relation phrases and proposed a clustering method and demonstrated the effectiveness of this approach through extensive experiments on two different datasets~\cite{lin2019canonicalization}. Other studies have suggested clustering methods for canonicalization. Galárraga and Heitz presented a machine-learning-based strategy utilizing the AMIE algorithm~\cite{galarraga2013amie} that can canonicalize Open IE triples, by clustering synonymous names and phrases~\cite{galarraga2014canonicalizing}.

\subsection{Literature in Low-Resource Languages}
In this subsection, by low-resource language (resource-poor language), we mean a language without a lot of labelled datasets and corpora dedicated for the training of the supervised methods. If we look at KBP as a whole system, there are different approaches proposed, most of which in the TAC-KBP challenges, which typically utilizes components such as IE, RE, EL, Canonicalization and Fusion components. The TAC-KBP challenges benchmarking are performed over three languages, namely English, Spanish and Chinees, which are not low-resource languages~\cite{getman2018laying}. The suggested state-of-the-art KBP benchmarking methods such as KnowledgeNet~\cite{mesquita2019knowledgenet} and CC-DBP~\cite{glass2018dataset} is also focused on English. To the best of our knowledge, considering KBP as a whole system, no specific research has been found for low-resource languages.

However, given the subsystems of a KBP system such as IE and RE, some research has been done in low-resource languages, but most of them are cross-lingual. A few of these types of research, which are novel, up to date and state-of-the-art, are discussed subsequently. Taghizadeh et al. proposed a system for RE in Arabic by a cross-language learning method, utilizing the training data of other languages such as Universal Dependency (UD)~\cite{straka2015parsing} parsing and the similarity of UD trees in different languages and trains a RE model for Arabic text~\cite{taghizadeh2018cross}. Zakria et al. proposed a method for RE in Arabic exploiting Arabic Wikipedia articles properties. The proposed system extracts sentences that contain principle entity, secondary entity and relation from Wikipedia article, then utilizes WordNet and DBpedia to build the training set. Then a Naive Bayes Classifier is used to train and test the datasets~\cite{zakria2019relation}. AlArfaj have reviewed the different state-of-the-art RE methods for Arabic and showed that the majority of RE approaches utilize a combination of statistical and linguistic-pattern techniques. The review proposes that linguistic-pattern methods provide high precision, but their recall is very low, and patterns are specified in the regular expression form, which is challenging to cope with language variety~\cite{alarfaj2019towards}. Sarhan et al. proposed a semisupervised pattern-based bootstrapping technique for Arabic RE utilizing a dependency parser to omit negative relations as well as features like stemming, semantic expansion using synonyms, and an automatic scoring technique to measure the reliability of the generated patterns and extracted relations~\cite{sarhan2016semi}.

SUN et al. proposed a distantly supervised RE model based on Piecewise Convolutional Neural Network (PCNN) to expand the Tibetan corpus. They also added self-attention mechanism and soft-label method to decrease wrong labels, and use Embeddings from ELMo language model~\cite{peters2018deep} to solve the semantic ambiguity problem~\cite{sun2019improved}. Lkhagvasuren and Rentsendorj presented MongoIE, which is an OIE system for the Mongolian language, utilizing rule-based and classification approaches. Their classification method showed better results than the rule-based approach~cite{lkhagvasuren2020open}. Peng~\cite{peng2017jointly} suggested that learning under low-resource conditions needs special techniques when there is inadequate training data, including distant supervision, multi-domain learning, and multi-task learning. However, no particular approach proposed for low-resource languages.

\subsection{Literature in the Persian Language}
There are few studies on knowledge extraction and knowledge base population in the Persian language. Moradi et al. \cite{moradi2015commonsense} propose a system focused on certain generic relation types, such as is-a relationship, to extract relation instances. Hasti Project proposed by Shamsfard and Barforoush \cite{shamsfard2004learning} is another research which introduced an automatic ontology building approach. It extracts lexical and ontological knowledge from Persian raw text and uses logical, template-driven, and semantic analysis methods.  Momtazi and Moradiannasab \cite{momtazi2019statistical} also proposed a statistical n-gram method which extracts knowledge from unstructured Persian language texts.

\section{Proposed Approach}
\label{proposed-approach}
In this section, we describe the architecture of FarsBase-KBP and propose our approach in which extracted triples from different extractors are integrated and stored into a knowledge graph. Six extractor components are used in total, including four information extractors and two relation extractors. The predicates in extracted triples by information extractors are plain texts, not predefined relations. Consequently, their predicates are not linked to relations in the knowledge base. Thus a canonicalization process is needed to link predicates of extracted relations to the Knowledge Graph (KG) resources.

\subsection{FarsBase-KBP Architecture}
The architecture of our system, FarsBase-KBP works on the top of six extractor modules and fuses their output triples. Figure \ref{fig:Arch} shows a block diagram of the architecture and data flow of FarsBase-KBP. First, a crawler crawls the Web and extracts raw texts. After applying a pre-processing stage, the extracted text will be delivered to the entity linking module. This module links entities of the pre-processed text and passes the processed text to each of the six extractor components.
Two relation extractors produce triples which are mapped to KG entities and delivered to the Candidate-Fact-Triples-Repository (CFTR) for further process. Note that each triple has a corresponding confidence value which is assigned by its extractor module. Four information extractors work independently and produce information triples containing a subject, object, and not-canonicalized predicate. Then, these triples are delivered to the Canonicalization Module (CM). Next, the CM maps the plain-text predicates to KG ontology and produces triples with their corresponding confidences. At this stage, the canonicalized triples are ready to be stored into the candidate-facts-triples-repository.
\begin{figure}
	\includegraphics[width=\linewidth]{Figures/Arch.jpg}
	\caption{The architecture of FarsBase knowledge base population system (FarsBase-KBP)}
	\label{fig:Arch}
\end{figure}
Afterwards, Knowledge Fusion module integrates triples which are stored in the CFTR and extracts triples with minimum confidence threshold predefined in the module. After this stage, triples are passed to human experts to check the correctness of the triples. The correct triples then are added to the FarsBase.

\subsection{Extraction Modules}
\subsubsection{PredPatt Extractor}
To present a multilingual state-of-the-art unsupervised baseline method alongside with our other extraction modules, we utilized PredPatt of Decomp project~\cite{white2016universal} on Universal Dependencies (UD) Project~\cite{nivre2016universal} for the Persian language. UD project provides a standard syntactic annotation which can be used in different languages. This project purposes unified grammatical representation which is valid cross-linguistically and includes, universal set of dependency relations and POS tags. Seraji et al. presented UD for the Persian language~\cite{seraji2016universal}.

While the UD project present syntactic annotation, Decomp project aims to propose a set of protocols for augmentation datasets in UD format with universal decompositional semantics. In more straightforward language, the UD project proposes syntactic annotation while Decomp project proposes semantic annotation in different languages. PredPatt is a software package for processing UD annotated corpora and annotating them with Decomp protocols. If we look at PredPatt as a black box, it takes a UD annotated dataset in the desired language as input, and its output is that dataset augmented with predicate-argument information alongside other syntactic and semantic annotations. Different studies propose that PredPatt can be utilized as an Open IE system~\cite{govindarajan2019decomposing,claro2019multilingual} and it overcomes other state-of-the-art Open IE systems~\cite{zhang2017evaluation}.

Our main idea here is to utilize PredPatt to extract sets of predicate-arguments from input Persian sentences. For every input sentence, this module provides a predicate and two or more arguments. All possible permutation of arguments are then considered as subjects and objects, and the resulting triples are sent to the output as candidate relationships. For example, if three arguments are specified for a predicate in a sentence in the form of (arg1, pred, arg2, arg3), six triples are generated at the output including (arg1, pred, arg2), (arg1, pred, arg3), (arg2, pred, arg1), (arg2, pred, arg3), (arg3, pred, arg1), and (arg3, pred, arg2). It should be noted that these triples are not necessarily correct relationships. In the next section, we will discuss the details of the performance of this module as a state-of-the-art baseline on our gold dataset.

\subsubsection{Dependency Pattern Extractor}
The dependency pattern information extractor is the first of our novel information extractor modules. Extraction with dependency patterns is an innovative method that attempts to extract information triples using ``unique dependency trees''.
The main idea is that sentences with the same structure of dependency trees can contain similar relationships. That means if a particular form of a dependency tree is observed in a sentence and the sentence contains a relationship, other sentences with the same dependency tree structure can probably contain a relationship with the same pattern.  Extraction with dependency pattern is based on the fact that if a sentence contains some triples, other sentences with the same structure (same dependency pattern) contain the same triples too. In such cases, the subject, object, and predicate can be extracted from the words with the same indexes in all sentences. For example, if in a sentence, the first word is the subject, then in all other sentences with the same dependency pattern, the subject can be extracted from the first word as well~\cite{asgari2019farsbase}.

Two dependency parse trees haves the same dependency pattern if similar trees are generated when words are replaced with their corresponding POS tags.  In other words, the sequences of the POS tags in both sentences are precisely the same, and also the dependency type and head of each word in the same position are equal.

To build this module, we utilized expert annotators to extract the desired rules. It should be noted that despite the use of human annotators, this method is not a supervised method. Because experts once extract the rules of this method and then can be used forever, and in fact, the system works as an unsupervised rule-based system which does not need any learning dataset.

In order to produce the rules, we first extracted dependency patterns for all of the sentences in a raw text corpus and then calculated most frequent dependency patterns in the corpus. We then asked human annotators to inspect the sentences of each of those patterns and annotate subject, object and predicate, for all of the relationships which can be mined from the sentences. In this way, we can determine which of the word positions in a given dependency tree and in what order, define the components of a relationship, namely the subject, object, and predicate. In this fashion, we were able to extract a set of relationship patterns in specific dependency tree structures. Currently, 3499 frequent dependency patterns are extracted automatically from Wikipedia texts, and experts annotate 6\% of them.

This module operates based on these extracted rules. For every new sentence the module receives, it first produces the dependency tree of the sentence and then compares this tree with the tree structure of the patterns generated earlier. If a matched dependency pattern is found, the module extracts a relation based on annotations defined for the pattern.

\subsubsection{Persian Syntactic Information Extractor{} (PSIE{})}
In the Persian Syntactic Information Extractor{}, the goal is to extract relation instances from an unlabeled and unannotated text, without any predefined relations as a train set. This method uses grammatical structures of sentences to extract relation instances. In this approach, all the predicates are based on verbs of the sentences, not in the predefined relations set, which is already known. Therefore, the evaluation of the results is not easy so that human experts intervention is required. Like other IE modules, another problem which was mentioned before is that predicates in the relation instances must be canonicalized to FarsBase ontology.

This approach can be classified as an open information extraction approach. For the first time, KnowItAll \cite{etzioni2004web} introduced unsupervised knowledge extraction. The first model introduced in this project was TextRunner \cite{banko2007open}. TextRunner is comprised of three main components, namely Self-supervised learner, single-pass extractor, and redundancy-based assessor.

Wikipedia-based open extractor(WOE) \cite{wu2010open} is another method of unsupervised knowledge extraction, which extracts knowledge from Wikipedia articles and uses dependency path between entities to improve the performance. OLLIE \cite{schmitz2012open} and BONIE \cite{saha2017bootstrapping} are two more recent unsupervised methods used for knowledge extraction.

Our system uses a different unsupervised method for triple extraction, i.e. dependency parsing and constituency tree are combined to extract the triples. Details of this module have been published in \cite{asgari2019farsbase}.

\subsubsection{RePersian - an automated ReVerb approach for the Persian language}
ReVerb \cite{etzioni2011open} is an unsupervised knowledge extraction method that was first introduced and applied in the KnowItAll \cite{etzioni2004web} project. ReVerb aimed to improve the performance of TextRunner by imposing lexical and syntactical constraints on relations. We employ RePesian~\cite{sahebnassagh2020repersian}, which uses a novel approach to produce regular expressions in ReVerb automatically. To find out the most frequent regular expressions, Dadegan Dependency Parsing Treebank~\cite{dadegan2012persian} processed with an automated algorithm. The algorithm found the best regular expressions for the subject, object, and Mosnad in the Persian language. We have already explained the details of this method in the previously published RePersian~\cite{sahebnassagh2020repersian} article.

\subsubsection{Distant Supervision}
One of the main challenges in relation extraction is the time and effort needed to make a manually labelled dataset. Distant supervision module utilizes a knowledge base to address this issue. To be more precise, the idea is that if a sentence contains a pair of entities which are relevant to each other in the knowledge base, this sentence may represent the required semantic relation.
Mintz et al. \cite{mintz2009distant} used DBpedia and a collection of Wikipedia articles to make a distantly supervised dataset, and then extracted thousands of new relation instances from that dataset. There have been many modifications to this method, like using multi-instance learning approach proposed by Manning \cite{surdeanu2012multi}. The single-instance approach assumes that each sentence has one pair of entities and only one semantic relation between them, but multi-instance approach jointly models all the instances of a pair of entities in text and all their relations. Recent studies have employed word embedding and various type of deep neural networks to perform relation extraction from distantly supervised datasets. Using word embedding eliminates the need for manually extracted features from sentences.
Trisedya et al. \cite{distiawan2019neural} have recently proposed another distantly supervised relation extractor for knowledge base enrichment. Their neural end-to-end relation extraction model integrates the extraction and canonicalization tasks. Thus, their model reduces the error propagation between relation extraction and Named Entity Disambiguation. As a result, the existing approaches which are error-prone can be addressed by this model. To obtain high-quality training data, they adapt distant supervision and augment it with co-reference resolution and paraphrase detection.

We use FarsBase, as our knowledge base, and the unstructured text that our crawler module retrieves to create a Distantly Supervised dataset (DS-dataset). First, all the sentences of the raw text are checked by the module. Any sentence that contains a pair of related FarsBase entities is considered as a candidate containing the semantic relation of the same relation of entity pair. Then, a piecewise convolutional neural network (PCNN) with multi-instance learning is employed to extract knowledge from the dataset. Should be noted that our distantly supervised dataset is the first dataset provided for the Persian language. Moreover, the distant supervision component also produces frequent tokens and compound verbs for every predicate, which is used in our canonicalization and integration phases.

\subsubsection{TokensRegex}
TokensRegex extraction module is a rule-base extraction module, which works based on Stanford TokensRegex \cite{chang2014tokensregex}. TokensRegex is a framework for specifying regular expressions over sequences of tokens and their additional side information, such as NER and POS tags, for identifying and acting on patterns in text. After the entity linking on the text, we have used the class of each entity instead of using named entity tags. Human experts have defined 166 TokensRegex rules for 58 frequent predicates on our gold dataset based on words, POS tags, and entity classes. Subject and object are defined in the body of each regular expression, and each expression refers to a predicate in FarsBase. Then, these rules were considered as the rules for relation extraction for this extractor module. Each sentence of the input raw text is examined according to the rules, and if the desired pattern is found, a relationship will be extracted. Figure \ref{fig:TokensRegex} shows an example of TokensRegex that matches a sentence, and Token Regex module extracts a fb:nationality triple.
\begin{figure}
	\frame{\includegraphics[width=\linewidth]{Figures/TokenRegex.png}}
	\caption{A TokensRegex example matching fb:nationality predicate}
	\label{fig:TokensRegex}
\end{figure}

\subsection{Relation to Knowledge Mapper}
\label{sec:mapper}
To extract a standard triple for knowledge graph, we must link two types of data:
\paragraph{Entity Linking (Relation arguments to KG entities)}
In KG, all subjects must be an entity. Our entity linker, ParsEL, finds all entity mentions in the tokens of the sentence and disambiguates and links them to the knowledge graph. Disambiguation process is a challenging task. Entity linker assigns a confidence value to each link. If all tokens of an argument belong to the same entity, the argument will be linked to it. When there are two different entities for an argument with different confidence values, two triples are generated using both entities with lower confidence values (weighted by entity confidences). These triples have the chance to be transferred to the final KG by Knowledge Fusion Module if they are extracted from multiple sources.
\paragraph{Canonicalization (Relation type to KG ontology predicate)}
We use a canonicalization module (CM) component to map relations extracted by four IE components to Knowledge Graph ontology. A baseline method proposed alongside with our FarsBase Canonicalization module.

\textbf{Baseline:} To implement a canonicalization baseline method, we examined the state-of-the-art methods as likely to be applicable to the Persian language. CESI~\cite{vashishth2018cesi} is one of the canonicalization methods introduced in the related works section. CESI needs side information to perform canonicalization, which is earned from AMIE~\cite{galarraga2013amie} and Stanford KBP~\cite{surdeanu2012multi}, latest is not present the Persian language. Therefore, this method could not be implemented for the Persian language. Lin and Chen approach for canonicalization is another state-of-the-art method, which we have considered to implement for the Persian language. Like CESI, this method utilizes the side information of the original data sources, including the entity linking knowledge, the types of the candidate entities detected, as well as the domain knowledge of the source text. The entity linking knowledge and domain knowledge in this approach is not available for the Persian language. Thus, this method could not be implemented for the Persian language as well. To prepare a baseline for canonicalization, we implemented the Galárraga and Heitz method~\cite{galarraga2014canonicalizing} for the Persian language. This method was the only method that could be implemented for the Persian language among the state-of-the-art methods introduced in the related works section.

For this aim, we first gave the DS-dataset generated by the distant supervision module to the PredPatt, Dependency Pattern, RePersian and PSIE{} modules and several tuples were constructed. Then we gave all these generated tuples, which were 3059530 relationships, to the AMIE algorithm, and a number of rules were extracted. Then, we cluster each pair of rules if they have this logical relationship:

(s1, Pred1, o1) => (s1, Pred2, o1) AND (s2, Pred2, o2) => (s2, Pred1, o2)

Subsequently, a mapping table extracted from these clusters in the form of (raw text relations to the standard FarsBase relations), then this mapping table was controlled and filtered by human experts to remove the incorrect mappings, and finally, a mapping table was prepared as a baseline.

It should be noted here that in Galárraga and Heitz method~\cite{galarraga2014canonicalizing}, the input data is extensive, and therefore this method offers outstanding results in English. Nevertheless, there is no such big data in Persian. All extractors have been applied to the DS-dataset, and more than three million tuples extracted, which is far less from corresponding dataset in English. As a result, only about six thousand rules were extracted by the AMIE algorithm. Among these rules, a large number of them, extracted as transitive relations, which are not useful in canonicalization. Also, some extracted rules did not follow the aforementioned logical relationship. Eventually, 22 clusters of rules were obtained. Also, in some of these clusters, there were only rules between FarsBase predicates, and no plain text relationship was available in this type of cluster, and practically no new information was produced for the canonicalization. Some of these clusters only included plain text relationship, without any FarsBase predicate. In the latter case, we mapped these types of clusters to the FarsBase predicates manually. With all these modifications, there are still 15 clusters left to prepare the mapping table. As a result, this algorithm did not provide good results due to the lack of sufficient resources in Persian, despite the fact that it is the only available state-of-the-art method that could be implemented for the Persian language.

It should be noted that we did not use available Persian raw text corpora larger than Persian Wikipedia, such as MirasText~\cite{SABETI18.385}, to create DS-dataset. We created our  DS-dataset with the help of Persian Wikipedia containing more than 125 million words, while The MirasText corpus has more than 1.4 billion words. Despite its much larger size, the MirasText corpus is not suitable for use in this case. MirasText have been extracted from the texts of Persian news websites. To create a distantly supervised dataset in Persian, we need to look for entities of FarsBase, the only available knowledge base for the Persian, in a raw corpus. In contrary to the MirasText, it is expected that in the Persian Wikipedia sentences, there are many more sentences in which the FarsBase entities are presentو for two reasons: (I) Because most of the FarsBase triplets are extracted from Wikipedia infoboxes and (II) most of the entities in the infoboxes are included in the sentences of Wikipedia articles, while this condition exists with much less density in the MirasText.

\textbf{FarsBase Canonicalization module:} This module performs some steps to map a relation to standard FarsBase predicates which are explained below:
\begin{enumerate}
	\item If the extracted relation predicate phrase exists in the mapping set of the Wikipedia to FarsBase, the algorithm will map it to the corresponding KG predicate. The mapping table of the Wikipedia infobox predicates to FarsBase ontology had been introduced in our previous work \cite{asgari2019farsbase}.
	\item If the previous criterion is not satisfied, mapper uses the information extracted in Distant Supervision Component to find the best match of candidate KG predicate. Mapper matches words and compound verbs of the sentence of the predicate with corresponding information from distant supervision component. For each candidate predicate match, the mapper denotes one positive point for the candidate predicate. Finally, the mapper selects the predicate with the highest rank.
\end{enumerate}
\subsection{Knowledge Fusion Module}
Knowledge Fusion Module is a simple ensemble classifier. This module accepts a triple as a fact and sends it to the next component if one of these two conditions is satisfied:
\begin{enumerate}
	\item If a triple (with any confidence) is extracted by more than one (two to five) extractor.
	\item If a triple is extracted by just one extractor while its corresponding confidence is higher than a specified threshold.
\end{enumerate}

\section{Experimental Results}
\label{experiments}
In this section, we introduce FarsBase-KBP Gold Dataset containing 22015 facts which are labelled and verified by human experts as gold data. This corpus has been used in the evaluation of all the components of FarsBase-KBP. Note that, all the modules are unsupervised, and the corpus is created only for the evaluation purposes. In this section, we introduce FarsBase-KBP Gold Dataset and then provide experimental results.

\subsection{FarsBase-KBP Gold Dataset}
We trained FarsBase-KBP and its components over a corpus of 22015 facts which are labelled by human experts as gold data. To build this corpus, we searched for subject and object of each FarsBase triples in Wikipedia articles in the Persian language. More precisely, a sentence is a candidate for a triple's predicate if the sentence contains both subject and object of the considered triple.
Also, we have chosen 406 distinct frequent FarsBase predicates for which the sentences are annotated by human experts. At last, we collected a gold dataset with 22015 sentence and its corresponding subject, object, and predicates.

This corpus has some automatic preprocessing as follows:
\begin{itemize}
	\item Tokens of each sentence.
	\item Part of speech tags (POS).
	\item Named Entity Recognition tags (NER).
	\item Dependency Parsing trees.
	\item Linked entities to the FarsBase.
	\item FarsBase classes for subject and object.
\end{itemize}

Figure \ref{fig:GoldData} shows an instance of each triple in JSON format. In this example, the subject is ``Belarus'', and the object is ``Alexander Lukashenko'' and both are linked to the corresponding entities in FarsBase knowledge graph. The predicate is ``fkgo:leaderName'' which is the standard ontology-predicate in FarsBase and is approved by a human expert; thus this attribute is the gold part of the corpus. Other features like the subject, object, KG classes, tokens, NER tags, and POS tags are also included in the gold data.
\begin{figure}
	\frame{\includegraphics[width=\linewidth]{Figures/GoldData.png}}
	\caption{An instance of gold corpus used to train FarsBase-KBP components}
	\label{fig:GoldData}
\end{figure}

Figure \ref{fig:PredicateFrequency} shows the distribution of predicate instances frequency in the gold data, which is sorted by frequency. Observation shows that there exists a normal distribution among them. In Table  \ref{tab:expcond}, an example of the most frequent predicates in the gold corpus has been written.
\begin{figure}
	\frame{\includegraphics[width=\linewidth]{Figures/PredicateFrequency.jpg}}
	\caption{Distribution of predicate instances based on frequency in the gold data}
	\label{fig:PredicateFrequency}
\end{figure}

\begin{table}[htbp]
\centering
\caption{The most frequent predicates in the gold corpus}
\label{tab:expcond}
\begin{tabular}{lclc}
\hline
Predicate & Instance Count & Predicate & Instance Count\\
\hline
fkgo:location       & 1110  &  fkgo:type         & 457\\
foaf:name           & 915   &  fkgo:order        & 388\\
fkgo:birthPlace     & 694   &  fkgo:field        & 347\\
fkgo:occupation     & 679   &  fkgo:team         & 309\\
fkgo:speciality     & 679   &  fkgo:releaseDate  & 292\\
fkgo:genus          & 555   &  fkgo:language     & 269\\
fkgo:nationality    & 474   &  fkgo:family       & 255\\
fkgo:birthDate      & 471   &  fkgo:notableWork  & 237\\

\end{tabular}
\end{table}

\subsection{Evaluations}

Table \ref{tab:Eval} shows results and evaluation metrics of each FarsBase-KBP extraction module. We fed system with the previously mentioned corpus and used it as a gold dataset to evaluate the Precision, Recall, and F\textsubscript{1} (harmonic mean of precision and recall) of each module. Note that, every sentence in the dataset may have multiple triples, but one triple per sentence has been defined. Also, each module may extract more than one triple for some of the sentences. To calculate the recall, we only consider the gold triples of our dataset.
\begin{table}[htbp]
	\centering
	\caption{Evaluation Statistics}
	\label{tab:stat}

	\begin{tabular}{llllll}
\hline
    Module   Name      & Triples        & Corrects      & Wrongs         & OSO            & Tri./Sen.       \\
\hline
    DependencyPattern  & 418            & 71            & 24             & 323            & 0.019           \\
    DistantSupervision & 17745          & \textbf{4632} & \textbf{13113} & 0              & 0.806           \\
    PredPatt           & \textbf{66384} & 13            & 82             & \textbf{66289} & \textbf{3.0154} \\
    RePersian          & 7865           & 51            & 241            & 7573           & 0.3573          \\
    TokensRegex         & 37351          & 3306          & 917            & 33128          & 1.6966          \\
    PSIE               & 44809          & 94            & 484            & 44231          & 2.0354         \\
\hline
	Fusion			   & 39375          & 3730          & 1280           & 34365          & 1.8119     \\
\hline
	\end{tabular}
\end{table}
\begin{table}[htbp]
	\centering
	\caption{Evaluation Results}
	\label{tab:Eval}

	\begin{tabular}{llll}
\hline
	Module Name       &    Precision   &    Recall     &      F\textsubscript{1}       \\
\hline
    DependencyPattern  & 0.7474          & 0.0032          & 0.0064          \\
    DistantSupervision & 0.261           & \textbf{0.2104} & 0.233           \\
    PredPatt           & 0.1368          & 0.0006          & 0.0012          \\
    RePersian          & 0.1747          & 0.0023          & 0.0046          \\
    TokensRegex.json    & \textbf{0.7829} & 0.1502          & 0.252           \\
    PSIE               & 0.1626          & 0.0043          & 0.0083          \\
\hline
	Fusion			   & 0.7313          & 0.1779          & \textbf{0.2862}     \\
\hline
	\end{tabular}
\end{table}

The first five rows in the table shows the following information for the five relation and knowledge extraction modules:
\begin{itemize}
	\item Triples: The number of triples which are extracted from 22015 sentences.
	\item Corrects: The number of extracted triples which are correct and exists in the gold dataset.
	\item Wrongs: The number of extracted triples that their entities are correct but the relation is incorrect.
	\item OSO: The number of extracted triples not existing in the gold dataset.
	\item Tri. /Sen.: The number of extracted triples per sentence.
\end{itemize}

The last row shows these metrics for Fusion module when the confidence threshold is 0.9. As can be seen, the fusion module outperformed all of the individual modules based on f1 metric, while its precision and recall are comparable with the performance of the best extractors. The experts must approve each triple before adding to the knowledge graph. Therefore, the precision metric has the highest importance during the entire extraction process. Human intervention in a web-scale system, such as our proposed system, as well as other web-sclase machine reading systems, is for verification, not to create a supervised system. In this case, some human users are asked to review and accept or reject the triples which the system has calculated relatively high confidence for them. We can reduce the confidence threshold, and as a result, the recall will reach much higher results. However, this dramatically increases the number of triples that should be examined by human users and changes their function from a verifier to an annotator. Because human verifiers help the system voluntarily, providing such a volume of incorrect triplets for them to verify, will exhaust their motivation to continue doing so. In other words, we want human users to have only the role of verification and provide them with triples of high confidence to verify. Also, if for any reason human users are not available to verify triples, the system must work with the maximum precision to prevent the promotion of wrong triples to the FarsBase, even if it is at the cost of reducing the recall.

Figure \ref{fig:Recall-F1-Threshold} shows the effect of changing the value of the confidence threshold on the Fusion module in contrast to recall and F\textsubscript{1} measures. The chart depicts values of recall and f-measure of fusion module for the different thresholds, and it shows the inverse relationship between the threshold and recall values. The plotted bullets define recall and  F\textsubscript{1} measures for the other modules. Distant supervision meets the recall line before threshold=0.1, but other modules meet the line after 0.9. F\textsubscript{1} is also decreasing from 0.1. Because of the low precision of Distant Supervision module, it meets the line after 0.4 with the F\textsubscript{1} line. Other modules meet the line after 0.9. As we expect, the higher thresholds lead to a lower recall.

\begin{figure}
	\frame{\includegraphics[width=\linewidth]{Figures/Recall-F1-Threshold.jpg}}
	\caption{Effect of changing the confidence threshold on Recall and F\textsubscript{1} measure of the Fusion module}
	\label{fig:Recall-F1-Threshold}
\end{figure}

Figure \ref{fig:threshold_changes} shows the effect of changing the confidence threshold on the precision value of the Fusion module. As we expect, higher thresholds lead to higher precision. The bullets on the chart show the precision of each single extraction module. According to the results, the precision metric of the fusion module outperforms two of the extraction modules when threshold=0, and when threshold=0.1, the fusion module extracts triples with precision higher than all of the modules except DependencyPattern module. Finally, the fusion module overtakes all the extraction modules when threshold=0.8.

\begin{figure}
	\frame{\includegraphics[width=\linewidth]{Figures/Precision-Threshold.jpg}}
	\caption{Effect of changing the confidence threshold on Precision of the Fusion module}
	\label{fig:threshold_changes}
\end{figure}

Comparing \ref{fig:Recall-F1-Threshold} and Figure \ref{fig:threshold_changes}, we can induce that while higher threshold results in higher precision, it has a converse effect on recall and F\textsubscript{1} measures.

Table \ref{tab:Common} shows the number of commonly extracted triples between each pair of modules. The second column shows the number of all triples extracted by the modules, and the third column illustrates the number of corrected triples, which are existed in the gold data. Note that, the common triples are calculated on all triples extracted by modules, not only the triples which are existed in the gold data. The Acquired results describe that Distant Supervision and TokensRegex modules can extract the most common triples (2300). To find out which pair of modules extract similar triples, we should pay attention to the number of triples extracted by the modules, and consider the portion of common triples to the average number of triples extracted by both modules.

\begin{table}[htbp]
	\centering
	\caption{The number of common extracted triples between every pair of modules}
	\label{tab:Common}
	\begin{tabular}{lllllll}
\hline
    Module   & DisSup & TokReg & DepPat & PSIE & RePer & PredPatt \\
\hline
    DisSup   & 0      & 2300   & 104    & 36   & 104   & 20       \\
    TokReg   & 2300   & 0      & 36     & 124  & 334   & 8        \\
    DepPat   & 104    & 36     & 0      & 20   & 36    & 10       \\
    PSIE     & 36     & 124    & 20     & 0    & 442   & 62       \\
    RePer    & 104    & 334    & 36     & 442  & 0     & 496      \\
    PredPatt & 20     & 8      & 10     & 62   & 496   & 0       \\
\hline
    Triples  & 17745  & 37351  & 418    & 44809 & 7865  & 66384    \\
    Corrects & 4632   & 3306   & 71     & 94    & 51    & 13      \\
\hline
	\end{tabular}
\end{table}

\section{Conclusion}
\label{conclusions}
In this paper, we provide a novel gold corpus for training and evaluating knowledge extractors in the Persian language. We also introduce multiple unsupervised components for relation and knowledge extraction in the Persian language. The acquired results show that using a fusion module can increase the precision of knowledge extraction when it works above all individual extractor components.

In future work, we will address the low recall and precision of some of our extractor components. We also will provide some new supervised and unsupervised modules to increase the recall and precision of the system. Also, we plan to add some extractors which will work on other types of resources, such as tabular data. Another improvement is to add a rule learner module to conclude facts and generate new triples. Also, by adding a type checker to the Knowledge Fusion module, the wrong triples based on the domain and range of each predicate can be filtered.
Moreover, one of the problems is when the modules extract many triples at the end of each day so that verifying all the triples will be a tough task. One of the potential solutions is to utilize crowdsourcing to accelerate the triple verification process. The output of PredPatt module is hugely affected by the quality of universal dependency parsing. Based on our observations, the output of current universal dependency parser for the Persian language has many errors, especially for long sentences. We can create another UD corpus by converting Dadegan dependency parser dataset and implement a better UD parser in the future works.

\section*{Acknowledgment}

The authors would like to thank Mr. Mehrdad Nasser and Ms. Raana Saheb-Nassagh at Data Mining Laboratory, Faculty of Computer Science, Iran University of Science and Technology, for implementing their methods, namely Distant Supervision RE and RePersian, for FarsBase. The authors also wish to express their sincere appreciation to Dr. Sayyed Ali Hossayni for his constructive guidance and valuable comments.
\section*{References}