Conventionally, a variety of person search methods, surveillance systems, and image storing/search systems has been developed. PLT 1 disclosed a person search method that searches for persons based on stored video in a surveillance system. When recording video, the surveillance system extracts information, regarding person's faces and clothes, so as to store information in a database. When searching persons, the surveillance system compares face/clothing features of an image given as a query with face/clothing features stored in the database so as to determine whether or not they indicate the same person.
A person search system disclosed in PLT 1 will be described in detail with reference to FIG. 10. FIG. 10 shows the constitution of a person search device, which includes face region detection/face feature extraction parts 1000, 1020, clothing region detection/clothing feature extraction parts 1010, 1030, a clothing feature database 1040, a face feature database 1050, a face similarity calculation part 1060, a clothing similarity calculation part 1070, and a person identity determination part 1080.
The face region detection/face feature extraction part 1000 detects a face region so as to extract face features based on video captured by the surveillance system, thus sending extracted face features to the face feature database 1050. The clothing region detection/clothing feature extraction part 1010 detects a clothing region so as to extract clothing features based on video captured by the surveillance system, thus sending extracted clothing features to the clothing feature database 1040. On the other hand, the face region detection/face feature extraction part 1020 detects a face region so as to extract face features from a query image, thus sending query face features to the clothing similarity calculation part 1070. The face similarity calculation part 1060 compares query face features, received from the face region detection/face feature extraction part 1020, with face features stored in the face feature database 1050, thus calculating and sending a face similarity to the person identity determination part 1080. The clothing similarity calculation part 1070 compares query clothing features, received from the clothing region detection/clothing feature extraction part 1030, with clothing features stored in the clothing feature database 1040, thus calculating and sending a clothing similarity to the person identity determination part 1080. The person identity determination part 1080 determines person identity based on a face similarity calculated by the face similarity calculation part 1060 and a clothing similarity calculated by the clothing similarity calculation part 1070, thus producing a person search result.
Next, the operation of the person search device shown in FIG. 10 will be described. First, video captured by the surveillance system is input into the face region detection/face feature extraction part 1000 and the clothing region detection/clothing feature extraction part 1010. The face region detection/face feature extraction part 1000 detects a face region per each frame of input video so as to extract face features from the detected face region. Face features which are extracted from a face region detected from input video are stored in the face feature database 1050.
The clothing region detection/clothing feature extraction part 1010 detects a clothing region from input video so as to extract its visual features, i.e. clothing features. Extracted clothing features are stored in the clothing feature database 1040.
In case of person searching, a query image is input into the face region detection/face feature extraction part 1020 and the clothing region detection/clothing feature extraction part 1030. The face region detection/face feature extraction part 1020 and the clothing region detection/clothing feature extraction part 1030 function similarly to the face region detection/face feature extraction part 1000 and the clothing region detection/clothing feature extraction part 1010 so as to extract query face features and query clothing features.
The face similarity calculation part 1060 compares query face features with face features stored in the face feature database 1050 so as to calculate a face similarity. On the other hand, the clothing similarity calculation part 1070 compares query clothing features with clothing features stored in the clothing feature database 1040 so as to calculate a clothing similarity. The person identity determination part 1080 integrates a face similarity with a clothing similarity so as to determine person identity, thus producing a person search result.
PLT 2 disclosed an image storing/search system that searches for image data including image features equivalent to color sensation languages representing coloring which humans may subjectively sense. Herein, it is necessary to preset a correlation between color expressions, included in human natural languages, and images in a color space. Additionally, it is necessary to extract pixels from image data stored in a database, calculate similarities with color expressions, and save them in memory. Provided a color expression as a query, this system checks similarities of image data with color expressions so as to search and display image data with a high similarity.
USPTO Backgrounds
USPTO Backgrounds