Research on video anomaly detection with variational auto-encoder based on multi-level memory enhancement

Using autoencoders to reconstruct current video frames or predict future frames is a popular method in the task of video anomalous behavior detection based on weakly supervised learning. Recent studies have shown that introducing a memory module into an autoencoder can capture a limited number of normal patterns and cannot cope with new scenarios in the test set. Therefore, this paper proposes a dual-stream based multi-level memory-enhanced conditional variational autoencoder model (TS-MemCVAE), which uses RGB image and optical flow image dual-stream input, and adds memory modules at the bottle-neck. The memory module contains normal mode features of different sizes. At the same time, with the aid of optical flow information, the model can sensitively identify abnormal behaviors with large reconstruction errors. The model is divided into two parts: a multi-level memory-enhanced auto-encoder and a conditional variational autoencoder. The former is responsible for the reconstruction of the input video, and the latter is used to capture the high correlation between the reconstructed video and the optical flow image. further predictions. The model is validated on two benchmark datasets, UCSD Ped2 and CUHK Avenue, and achieves 95.83% and 84.16% on AUC, respectively, and its excellent performance proves the effectiveness of the model.