Micro-expression recognition algorithm based on separate long-term recurrent convolutional network
-
摘要: 基于面部動態表情序列，針對靜態表情缺少時間信息等問題，將空間特征與時間特征融合，利用神經網絡在圖像分類領域良好的特征，對需要進行細節分析的表情序列進行處理，提出基于分離式長期循環卷積網絡(Separate long-term recurrent convolutional networks, S-LRCN)的微表情識別方法。首先選取微表情數據集提取面部圖像序列，引入遷移學習的方法，通過預訓練的卷積神經網絡模型提取表情幀的空間特征，降低網絡訓練中過擬合的危險，并將視頻序列的提取特征輸入長短期記憶網絡(Long short-team memory, LSTM)處理時域特征。最后建立學習者表情序列小型數據庫，將該方法用于輔助教學評價。Abstract: With the rapid development of machine learning and deep neural network and the popularization of intelligent devices, face recognition technology has rapidly developed. At present, the accuracy of face recognition has exceeded that of the human eyes. Moreover, the software and hardware conditions of large-scale popularization are available, and the application fields are widely distributed. As an important part of face recognition technology, facial expression recognition has been a widely studied subject in the fields of artificial intelligence, security, automation, medical treatment, and driving in recent years. Expression recognition, an active research area in human–computer interaction, involves informatics and psychology and has good research prospect in teaching evaluation. Micro-expression, which has great research significance, is a kind of short-lived facial expression that humans unconsciously make when trying to hide some emotion. Different from the general static facial expression recognition, to realize micro-expression recognition, besides extracting the spatial feature information of facial expression deformation in the image, the temporal-motion information of the continuous image sequence also needs to be considered. In this study, given that static expression features lack temporal information, so that the subtle changes in expression cannot be fully reflected, facial dynamic expression sequences were used to fuse spatial features and temporal features, and neural networks were used to provide good features in the field of image classification. Expression sequences were processed, and a micro-expression recognition method based on separate long-term recurrent convolutional network (S-LRCN) was proposed. First, the micro-expression data set was selected to extract the facial image sequence, and the transfer learning method was introduced to extract the spatial features of the expression frame through the pre-trained convolution neural network model, to reduce the risk of overfitting in the network training, and the extracted features of the video sequence were inputted into long short-term memory (LSTM) to process the temporal-domain features. Finally, a small database of learners’ expression sequences was established, and the method was used to assist teaching evaluation.
-
Key words:
- micro-expression recognition /
- spatial-temporal features /
- LRCN /
- LSTM /
- education evaluation
-
表 1 劃分情況
Table 1. Dataset classification
Classify CASME-Ⅱ Samples Happiness Happiness (32) 32 Surprise Surprise (28) 28 Disgust Disgust (63) 63 Repression Repression (27) 27 Others Others (99) 105 Sadness (4) Fear (2) 
下載: 導出CSV
表 3 不同算法識別準確率
Table 3. Recognition accuracy of different algorithms
Methods Accuracy/% F1-Score/% LBP-TOP 52.6 42.6 STCLQP 58.6 58.0 CNN+LSTM 61.0 58.5 HOOF+LSTM 59.8 56.0 S-LRCN 65.7 60.8
下載: 導出CSV
中文字幕在线观看表 4 不同序列長度實驗效果
Table 4. Experimental results of different sequence lengths
Sequence length Accuracy/% F1-Score/% 6 62.0 56.6 10 65.7 60.8 15 63.1 58.6 30 56.5 49.6
下載: 導出CSV
-
參考文獻
[1] Mehrabian A. Nonverbal Communication. New York: Routledge, 2017 [2] Ekman P. Facial expression and emotion. Am Psychol, 1993, 48(4): 384 doi: 10.1037/0003-066X.48.4.384 [3] Ekman P, Friesen W V. Nonverbal leakage and clues to deception. Psychiatry, 1969, 32(1): 88 doi: 10.1080/00332747.1969.11023575 [4] Yan W J, Wang S J, Liu Y J, et al. For micro-expression recognition: Database and suggestions. Neurocomputing, 2014, 136(136): 82 [5] Wang S Y. CNN-RNN Based Micro-Expression Recognition [Dissertation]. Harbin: Harbin Engineering University, 2018王思宇. 基于CNN-RNN的微表情識別[學位論文]. 哈爾濱: 哈爾濱工程大學, 2018 [6] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60(6): 84 doi: 10.1145/3065386 [7] Donahue J, Hendricks L A, Guadarrama S, et al. Long-term recurrent convolutional networks for visual recognition and description // 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, 2015: 2625 [8] Ekman P, Rosenberg E L. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). 2nd Ed. USA: Oxford University Press, 2005 [9] Gunes H, Pantic M. Automatic, dimensional and continuous emotion recognition. Int J Synthetic Emotions, 2010, 1(1): 68 doi: 10.4018/jse.2010101605 [10] Song Y L, Morency L P, Davis R. Learning a sparse codebook of facial and body microexpressions for emotion recognition // Proceedings of the 15th ACM International Conference on Multimodal Interaction. New York, 2013: 237 [11] Li D, Xie L, Lu T, et al. Capture of microexpressions based on the entropy of oriented optical flow. Chin J Eng, 2017(11): 1727李丹, 解侖, 盧婷, 等. 基于光流方向信息熵統計的微表情捕捉. 工程科學學報, 2017(11):1727 [12] Polikovsky S, Kameda Y, Ohta Y. Facial micro-expression detection in Hi-speed video based on facial action coding system (FACS). IEICE Trans Inform Syst, 2013, E96-D(1): 81 doi: 10.1587/transinf.E96.D.81 [13] Shreve M, Godavarthy S, Goldgof D, et al. Macro- and micro-expression spotting in long videos using spatio-temporal strain // 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG). Santa Barbara, 2011: 51 [14] Pfister T, Li X B, Zhao G Y, et al. Recognising spontaneous facial micro-expressions // 2011 International Conference on Computer Vision. Barcelona, 2011: 1449 [15] Liang J, Yan W J, Wu Q, et al. Recent advances and future trends in micro-expression research. Bull Natl Nat Sci Foundation China, 2013(2): 75梁靜, 顏文靖, 吳奇, 等. 微表情研究的進展與展望. 中國科學基金, 2013(2):75 [16] Wang Y D, See J, Phan R C W, et al. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition // Asian Conference on Computer Vision ? ACCV2014. Switzerland, 2015: 525 [17] Liong S T, See J, Phan R C W, et al. Spontaneous subtle expression detection and recognition based on facial strain. Signal Process Image Commun, 2016, 47: 170 doi: 10.1016/j.image.2016.06.004 [18] Le Ngo A C, See J, Phan R C W. Sparsity in dynamics of spontaneous subtle emotion: analysis & application. IEEE Trans Affective Comput, 2017, 8(3): 396 doi: 10.1109/TAFFC.2016.2523996 [19] Xu F, Zhang J P, Wang J Z. Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affective Comput, 2017, 8(2): 254 doi: 10.1109/TAFFC.2016.2518162 [20] Patel D, Hong X P, Zhao G Y. Selective deep features for micro-expression recognition // 2016 23rd International Conference on Pattern Recognition (ICPR). Cancun, 2016: 2258 [21] Kim D H, Baddar W J, Ro Y M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations // Proceedings of the 24th ACM international conference on Multimedia. Amsterdam, 2016: 382 [22] Khor H Q, See J, Phan R C W, et al. Enriched long-term recurrent convolutional network for facial micro-expression recognition // 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018). Xi’an, 2018: 667 [23] Verburg M, Menkovski V. Micro-expression detection in long videos using optical flow and recurrent neural networks // 2019 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019). Lille, 2019: 1 [24] Itti L, Koch C. Computational modelling of visual attention. Nat Rev Neurosci, 2001, 2(3): 194 doi: 10.1038/35058500 [25] Wang C Y, Peng M, Bi T, et al. Micro-attention for micro-expression recognition [J/OL]. arXiv Preprint (2019-08-27) [2020-04-21]. https://arxiv.org/abs/1811.02360. [26] Parkhi O M, Vedaldi A, Zisserman A. Deep face recognition // Proceedings of the British Machine Vision Conference (BMVC). Swansea, 2015: 45 [27] Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011 doi: 10.1109/TPAMI.2019.2913372 [28] Cao Q, Shen L, Xie W D, et al. VGGFace2: A dataset for recognising faces across pose and age // 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018). Xi’an, 2018: 67 [29] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 770 [30] Cootes T F, Edwards G J, Taylor C J. Active appearance models. IEEE Trans Pattern Anal Mach Intell, 2001, 23(6): 681 doi: 10.1109/34.927467 [31] Peng M. Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition [Dissertation]. Chongqing: Southwest University, 2017彭敏. 基于雙時間尺度卷積神經網絡的微表情識別[學位論文]. 重慶: 西南大學, 2017 [32] Yan W J, Li X B, Wang S J, et al. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE, 2014, 9(1): e86041 doi: 10.1371/journal.pone.0086041 [33] Zhou Z H, Zhao G Y, Pietikinen M. Towards a practical lipreading system // The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011). Providence, RI, 2011: 137 [34] Zhao G Y, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell, 2007, 29(6): 915 doi: 10.1109/TPAMI.2007.1110 [35] Huang X H, Zhao G Y, Hong X P, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing, 2016, 175: 564 doi: 10.1016/j.neucom.2015.10.096 -
點擊查看大圖
圖(11) / 表(4)
計量
- 文章訪問數: 2334
- HTML全文瀏覽量: 976
- PDF下載量: 130
- 被引次數: 0
