-
摘要: 醫生診斷需要結合臨床癥狀、影像檢查等各種數據，基于此，提出了一種可以進行數據融合的醫療輔助診斷方法。將患者的影像信息（如CT圖像）和數值數據（如臨床診斷信息）相結合，利用結合的信息自動預測患者的病情，進而提出了基于深度學習的醫療輔助診斷模型。模型以卷積神經網絡為基礎進行搭建，圖像和數值數據作為輸入，輸出病人的患病情況。該醫療輔助診斷方法能夠利用更加全面的信息，有助于提高自動診斷準確率、降低診斷誤差；另外，僅使用提出的醫療輔助診斷模型就可以一次性處理多種類型的數據，能夠在一定程度上節省診斷時間。在兩個數據集上驗證了所提出方法的有效性，實驗結果表明，該方法是有效的，它可以提高輔助診斷的準確性。Abstract: In the field of medicine, in order to diagnose a patient’s condition more efficiently and conveniently, image classification has been widely leveraged. It is well established that when doctors diagnose a patient’s condition, they not only observe the patient’s image information (such as CT image) but also make final decisions incorporating the patient’s clinical diagnostic information. However, current medical image classification only puts the image into a convolution neural network to obtain the diagnostic result and does not use the clinical diagnosis information. In intelligent auxiliary diagnosis, it is necessary to combine clinical symptoms with other imaging data for comprehensive diagnosis. This paper presented a new method of assistant diagnosis for the medical field. This method combined information from patients’ imaging with numerical data (such as clinical diagnosis information) and used the combined information to automatically predict the patient’s condition. Based on this method, a medical assistant diagnosis model based on deep learning was proposed. The model takes images and numerical data as input and outputs the patient’s condition. Thus, this method is comprehensive and helps improve the accuracy of automatic diagnosis and reduce diagnostic error. Moreover, the proposed model can simultaneously process multiple types of data, thus saving diagnosis time. The effectiveness of the proposed method was verified in two groups of experiments designed in this paper. The first group of experiments shows that if the unrelated data are fused for classification, the proposed method cannot enhance the classification ability of the model, although it is able to predict multiple diseases at one time. The second group of experiments show that the proposed method could significantly improve classification results if the interrelated data are fused.
-
Key words:
- image classification /
- convolution neural network /
- feature fusion /
- medical diagnosis /
- deep learning
-
圖 1 基于提出的方法構建的模型結構
Figure 1. Diagram of the model structure based on the proposed method
圖 2 基本單元和下采樣單元的結構。（a）基本單元的結構；（b）下采樣單元的結構
Figure 2. Structure of the basic unit and down sampling unit：(a) structure of the basic unit; (b) structure of the down sampling unit
圖 3 訓練過程中的預測準確率和損失的變動。（a）準確率的變動；（b）損失的變動
Figure 3. Changes in predictive accuracy and loss during training: (a) changes in accuracy; (b) changes in the loss
圖 4 COVID數據集中兩種類型的樣本。（a）未患COVID?19的樣本；（b）患有COVID?19的樣本
Figure 4. Two types of samples in COVID: (a) samples without COVID?19; (b) samples with COVID?19
圖 5 訓練過程中的預測準確率和損失的變動。（a）準確率的變動；（b）損失的變動
Figure 5. Changes in predictive accuracy and loss during training: (a) changes in accuracy; (b) changes in the loss
表 1 PHD中四種類型的樣本
Table 1. Four types of samples in PHD
Class Image Age Sex CPT RBP/kPa SC/（mg·dL?1） FBS/（mg·dL?1） RER MHR/（times·min?1） EIA ST/mV SP NV Thal PH 
66 0 3 20 226 0 1 114 0 2.6 0 0 2 PNH 
54 1 0 14.7 239 0 1 126 1 2.8 1 1 3 NPH 
65 0 2 20.7 269 0 1 148 0 0.8 2 0 2 NPNH 
70 1 0 17.3 322 0 0 109 0 2.4 1 3 2 
下載: 導出CSV
表 2 在PHD數據集上僅通過圖像數據學習的預測結果
Table 2. Prediction results learned only from image data in PHD dataset
Prediction Label No pneumonia Pneumonia All No pneumonia 79 11 90 Pneumonia 12 80 92 All 91 91 182
下載: 導出CSV
表 3 在PHD數據集上僅通過結構化的數值數據學習的預測結果
Table 3. Prediction results learned only through structured numerical data
Prediction Label No pneumonia Pneumonia All No pneumonia 72 16 88 Pneumonia 11 83 94 All 83 99 182
下載: 導出CSV
表 4 本文方法在PHD數據集上的預測結果
Table 4. Predictive results of proposed method in PHD dataset
Prediction Label NPNH NPH PNH PH All NPNH 33 12 5 2 52 NPH 8 29 0 0 39 PNH 2 2 24 10 38 PH 2 3 9 39 53 All 45 46 38 53 182
下載: 導出CSV
表 5 在PHD數據集上三組實驗的準確率和其他評價指標
Table 5. Accuracy and other evaluation indicators of three groups of experiments in PHD dataset
Model Class TP FP FN Precision Recall F1-score Accuracy Fusion method NHPH 33 19 12 0.635 0.733 0.680 0.687 NPH 29 10 17 0.744 0.630 0.682 PNH 24 14 14 0.632 0.632 0.632 PH 39 14 14 0.736 0.736 0.736 ShuffleNetv2(Only image data) No pneumonia 79 11 12 0.878 0.868 0.873 0.874 Pneumonia 80 12 11 0.870 0.879 0.874 DNN(Only structured data) No heart disease 72 16 11 0.818 0.867 0.842 0.852 Heart disease 83 11 16 0.883 0.838 0.860
下載: 導出CSV
表 6 在COVID數據集上僅通過圖像數據學習的預測結果
Table 6. Prediction results learned only from image data in COVID dataset
Prediction Label NonCOVID COVID All NonCOVID 55 20 75 COVID 14 49 63 All 69 69 138
下載: 導出CSV
表 7 在COVID數據集上僅通過結構化的數值數據學習的預測結果
Table 7. Prediction results learned only by structured numerical data in COVID dataset
Prediction Label NonCOVID COVID All NonCOVID 53 0 53 COVID 16 69 85 All 69 69 138
下載: 導出CSV
表 8 本文方法在COVID數據集上的預測結果
Table 8. Predictive results of proposed method in COVID dataset
Prediction Label NonCOVID COVID All NonCOVID 65 4 69 COVID 4 65 69 All 69 69 138
下載: 導出CSV
表 9 在COVID數據集上三組實驗的準確度和其他評價指標
Table 9. Accuracy and other evaluation indicators of three groups of experiments in COVID dataset
Model Class TP FP FN Precision Recall F1-score Accuracy Fusion method NonCOVID 65 4 4 0.942 0.942 0.942 0.942 COVID 65 4 4 0.942 0.942 0.942 ShuffleNetv2(Only image data) NonCOVID 55 20 14 0.733 0.797 0.764 0.754 COVID 49 14 20 0.778 0.710 0.742 DNN(Only structured data) NonCOVID 53 0 16 1.00 0.768 0.869 0.884 COVID 69 16 0 0.812 1.00 0.896
下載: 導出CSV
表 10 本文方法和僅通過圖像學習對138個樣本進行分類的時間
Table 10. Time required to classify 138 samples using proposed method and using only image data
Model Proposed method Image only Time 3.58 3.56
下載: 導出CSV
中文字幕在线观看表 11 Fusion method、ResNet50、VGG16、ShuffleNetv2和AlexNet的準確度和其他評價指標
Table 11. Accuracy and other evaluation indicators of Fusion method, ResNet50, VGG16, ShuffleNetv2 and AlexNet
Model Class TP FP FN Precision Recall F1-score Accuracy Fusion method NonCOVID 65 4 4 0.942 0.942 0.942 0.942 COVID 65 4 4 0.942 0.942 0.942 ResNet50 NonCOVID 56 15 13 0.789 0.812 0.800 0.797 COVID 54 13 15 0.806 0.783 0.794 VGG16 NonCOVID 54 16 15 0.771 0.783 0.777 0.775 COVID 53 15 16 0.779 0.768 0.774 ShuffleNetv2 NonCOVID 55 20 14 0.733 0.797 0.764 0.754 COVID 49 14 20 0.778 0.710 0.742 AlexNet NonCOVID 50 18 19 0.735 0.725 0.730 0.732 COVID 51 19 18 0.728 0.739 0.734
下載: 導出CSV
-
參考文獻
[1] Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proc IEEE, 1998, 86(11): 2278 doi: 10.1109/5.726791 [2] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60(6): 84 doi: 10.1145/3065386 [3] Deng J, Dong W, Socher R, et al. ImageNet: A large-scale hierarchical image database // 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, 2009: 248 [4] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition [J/OL]. ArXiv Preprint (2014-09-04) [2021-01-12]. https://arxiv.org/abs/1409.1556 [5] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 770 [6] Szegedy C, Liu W, Jia Y Q, et al. Going deeper with convolutions // 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, 2015: 1 [7] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift [J/OL]. ArXiv Preprint (2015-02-11) [2021-01-12]. https://arxiv.org/abs/1502.03167 [8] Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 2818 [9] Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, inception-resnet and the impact of residual connections on learning [J/OL]. ArXiv Preprint (2016-02-24) [2021-01-12]. https://arxiv.org/abs/1602.07261 [10] Iandola F N, Han S, Moskewicz M W, et al. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size [J/OL]. ArXiv Preprint (2016-02-24) [2021-01-12]. https://arxiv.org/abs/1602.07360 [11] Howard A G, Zhu M, Chen B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications [J/OL]. ArXiv Preprint (2016-02-24) [2021-01-12]. https://arxiv.org/abs/1704.04861v1 [12] Zhang X Y, Zhou X Y, Lin M X, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, 2018: 6848 [13] Sandler M, Howard A, Zhu M L, et al. MobileNetV2: inverted residuals and linear bottlenecks// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, 2018: 4510 [14] Ma N N, Zhang X Y, Zheng H T, et al. ShuffleNet V2: practical guidelines for efficient CNN architecture design // 2018 European Conference on Computer Vision (ECCV). Munich, 2018: 122 [15] Howard A, Sandler M, Chen B, et al. Searching for MobileNetV3//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, 2019: 1314 [16] Li L, Xu M, Liu H R, et al. A large-scale database and a CNN model for attention-based glaucoma detection. IEEE Trans Med Imaging, 2020, 39(2): 413 doi: 10.1109/TMI.2019.2927226 [17] Yang H, Kim J Y, Kim H, et al. Guided soft attention network for classification of breast cancer histopathology images. IEEE Trans Med Imaging, 2020, 39(5): 1306 doi: 10.1109/TMI.2019.2948026 [18] Xu X Y, Wang C D, Guo J X, et al. MSCS-DeepLN: Evaluating lung nodule malignancy using multi-scale cost-sensitive neural networks. Med Image Anal, 2020, 65: 101772 doi: 10.1016/j.media.2020.101772 [19] Mobiny A, Lu H Y, Nguyen H V, et al. Automated classification of apoptosis in phase contrast microscopy using capsule network. IEEE Trans Med Imaging, 2020, 39(1): 1 doi: 10.1109/TMI.2019.2918181 [20] Zhou Y, Li G Q, Li H Q. Automatic cataract classification using deep neural network with discrete state transition. IEEE Trans Med Imaging, 2020, 39(2): 436 doi: 10.1109/TMI.2019.2928229 [21] Wang Y, Wang N, Xu M, et al. Deeply-supervised networks with threshold loss for cancer detection in automated breast ultrasound. IEEE Trans Med Imaging, 2020, 39(4): 866 doi: 10.1109/TMI.2019.2936500 [22] Liu T J, Guo Q Q, Lian C F, et al. Automated detection and classification of thyroid nodules in ultrasound images using clinical-knowledge-guided convolutional neural networks. Med Image Anal, 2019, 58: 101555 doi: 10.1016/j.media.2019.101555 [23] Yao C, Zhao J Z, Ma B Y, et al. Fast detection method for cervical cancer abnormal cells based on deep learning. Chin J Eng, https://doi.org/10.13374/j.issn2095-9389.2021.01.12.001姚超, 趙基淮, 馬博淵, 等. 基于深度學習的宮頸癌異常細胞快速檢測方法. 工程科學學報, https://doi.org/10.13374/j.issn2095-9389.2021.01.12.001 [24] Zeng Y W, Liu X K, Xiao N, et al. Automatic diagnosis based on spatial information fusion feature for intracranial aneurysm. IEEE Trans Med Imaging, 2020, 39(5): 1448 doi: 10.1109/TMI.2019.2951439 [25] Wang L T, Zhang L, Zhu M J, et al. Automatic diagnosis for thyroid nodules in ultrasound images by deep neural networks. Med Image Anal, 2020, 61: 101665 doi: 10.1016/j.media.2020.101665 [26] Kumar A, Fulham M, Feng D G, et al. Co-learning feature fusion maps from PET?CT images of lung cancer. IEEE Trans Med Imaging, 2020, 39(1): 204 doi: 10.1109/TMI.2019.2923601 [27] Joyseeree R, Otálora S, Müller H, et al. Fusing learned representations from Riesz filters and deep CNN for lung tissue classification. Med Image Anal, 2019, 56: 172 doi: 10.1016/j.media.2019.06.006 [28] Kingma D, Ba J. Adam: A method for stochastic optimization[J/OL]. ArXiv Preprint (2014-12-22) [2021-01-12]. https://arxiv.org/abs/1412.6980 [29] Bio. Heart Disease UCI [J/OL ]. Kaggle (2018-06-25) [2021-01-12]. https://www.kaggle.com/ronitf/heart-disease-uci [30] Società Italiana di Radiologia Medica e Interventistica. Covid−19 Database[J/OL]. Database Online (2020-03-18) [2021-01-12]. https://www.sirm.org/category/senza-categoria/covid-19 [31] Zhao J, Zhang Y, He X, et al. COVID−CT−Dataset: A CT scan dataset about COVID−19[J/OL]. ArXiv Preprint (2020-03-30) [2021-01-12]. https://github.com/UCSD-AI4H/COVID-CT -
點擊查看大圖
計量
- 文章訪問數: 944
- HTML全文瀏覽量: 518
- PDF下載量: 89
- 被引次數: 0
