Neural-Network Monitoring of Train Driver Vigilance from Video and Speech: a Review of Methods and a Concept of a Multimodal System

Standard train-driver vigilance devices (the vigilance handle, ALSN cab signalling, KLUB-U and the TSKBM electrodermal bracelet) are reviewed and shown to be either reactive, i.e. triggered only after control of the train has been lost, or contact-based and single-channel. Published evidence on contact-less markers of fatigue and drowsiness is systematised for video (blink dynamics, PERCLOS, head pose) and speech (tempo, pauses, fundamental frequency, cepstral coefficients), each with its source and the accuracy reported there. A concept of an on-board system that fuses the video and speech channels with a neural-network model and a hybrid fusion strategy is proposed together with a valida-tion programme: a metric set, on-board platform criteria and test stages.

1. Federal Railroad Administration. Railroad Operating Rules: Program of Operational Tests and In-spections; Railroad Operating Practices: Handling Equipment, Switches and Fixed Derails. Final rule / Federal Railroad Administration. – Federal Register, Vol. 73, No. 30, pp. 8442–8477, 2008. – URL: https://www.govinfo.gov/link/fr/73/8442

2. U.S. Department of Transportation, Office of Inspector General. FRA Lacks Written Procedures and Formal Planning for Oversight of Railroad Hours of Service Compliance and the Passenger Railroad Fatigue Management Requirements / U.S. Department of Transportation, Office of In-spector General. – U.S. DOT Office of Inspector General, 2024. – URL: https://www.oig.dot.gov/libraryitem/46212

3. European Union Agency for Railways. Report on Railway Safety and Interoperability in the EU 2024 / European Union Agency for Railways. – Luxembourg : European Union Agency for Railways; Publications Office of the European Union, 2024. – ISBN 978-92- 9477-458-3. – DOI: 10.2821/64343

4. Fakhri, P. Train driver drowsiness detection using deep learning approach / P. Fakhri, R. Fathi, A. Mirabadi // International Journal of Railway Research. – 2024. – Vol. 11, № 1. – P. 1–9. – DOI: 10.22068/ijrare.344

5. Russian Railways (RZD). Sustainability Report 2024. Section "Traffic Safety" / Russian Railways. – 2025. – URL: https://sr2024.rzd.ru/ru/social-aspect/healthsafety/traffic-safety

6. Ventsevich, L. E. Locomotive-based train safety systems and the analysis of their operational data / L. E. Ventsevich. – Moscow : Marshrut, 2006. – 328 p. – ISBN 5-89035- 339-X

7. Mikhailova, N. A. Ergonomic design and implementation of a wakefulness control device for a railway train driver / N. A. Mikhailova, A. Yu. Nikolaev, O. G. Noskova // National Psychological Journal. – 2021. – Vol. 43, № 3. – P. 87–95. – DOI: 10.11621/npj.2021.0307

8. Dorrian, J. The driver vigilance telemetric control system (DVTCS): Investigating sensitivity to experimentally induced sleep loss and fatigue / J. Dorrian, N. Lamond, K. Kozuchowski [et al.] // Behavior Research Methods. – 2008. – Vol. 40, № 4. – P. 1016– 1025. – DOI: 10.3758/BRM.40.4.1016

9. Afanasyev, I. Taking a Device to Safety. RZD Inspectors Shocked: Manufacturers of RZD Locomotive Equipment Block Engineers' "Cheating Devices" / I. Afanasyev. – Vgudok, "Rails//Ties: Letters to the Editor" section, 2024. – URL: https://vgudok.com/light/nabezopasnost-s-priborom-revizory-rzhd-vshoke-proizvoditeli-tehniki-dlyalokomotivov-rzhd.

10. Maksimov, V. "Now I Know What Satan Looks Like!" – RZD Train Drivers Lose Sleep from Anger. TSKBM Telemechanical Vigilance Monitoring System: A Quest for Earnings or a Necessity? / V. Maksimov. – Vgudok, "Strategy. Safety" section, 2021. – URL: https://vgudok.com/lenta/teper-yaznayu-kak-vyglyadit-satana-mashinisty-rzhdteryayut-son-ot-zlosti-telemehanicheskaya

11. Zhang, X. Design of a fatigue detection system for high-speed trains based on driver vigilance us-ing a wireless wearable EEG / X. Zhang, J. Li, Y. Liu [et al.] // Sensors. – 2017. – Vol. 17, № 3. – P. 486. – DOI: 10.3390/s17030486

12. Wang, L. EEG and ECG-Based MultiSensor Fusion Computing for Real-Time Fatigue Driving Recognition Based on Feedback Mechanism / L. Wang, F. Song, T. H. Zhou [et al.] // Sensors. – 2023. – Vol. 23, № 20. – P. 8386. – DOI: 10.3390/s23208386

13. Sikander, G. Driver fatigue detection systems: A review / G. Sikander, S. Anwar // IEEE Transac-tions on Intelligent Transportation Systems. – 2019. – Vol. 20, № 6. – P. 2339–2352. – DOI: 10.1109/TITS.2018.2868499

14. Kashevnik, A. Driver distraction detection methods: A literature review and framework / A. Ka-shevnik, R. Shchedrin, C. Kaiser [et al.] // IEEE Access. – 2021. – Vol. 9. – P. 60063– 60076. – DOI: 10.1109/ACCESS.2021.3073599

15. Hao, Z. Train driver fatigue detection based on facial multi-information fusion / Z. Hao, Y. Wang, X. Chen [et al.] // Journal of Measurement Science and Instrumentation. – 2023. – Vol. 14, № 4. – P. 493–500. – DOI: 10.62756/jmsi.1674-8042.2023054

16. Lozano-Reyes, G. S. A vision-based drowsiness detection system for railway operators using lightweight convolutional neural networks / G. S. Lozano-Reyes, C. A. Mugruza-Vassallo // Frontiers in Future Transportation. – 2025. – Vol. 6. – P. 1677442. – DOI: 10.3389/ffutr.2025.1677442

17. Nocentini, O. Graph-based online monitoring of train driver states via facial and skeletal features / O. Nocentini, M. Lagomarsino, G. Solak [et al.]. – 2025. – URL: https://arxiv.org/abs/2505.08800

18. Popov, P. A. Application of Artificial Intelligence Technologies in Railway Transport / P. A. Popov // Railway Engineering. – 2024. – No. 1 (65). – P. 22–25. – URL: https://niias.ru/upload/iblock/385/qoq0tq4q4aen6rg08mf884o1yb19z3jo.pdf

19. Ozerov, A. V. Machine Vision in Automatic Train Movement Control Systems / A. V. Ozerov, A. S. Marshova // Advanced Information Technologies (PIT 2022): Proceedings of the International Scientific and Technical Conference (Samara, April 18–21, 2022) / Ed. by S. A. Prokhorov. – Samara : Samara Scientific Center of the Russian Academy of Sciences Publishing House. – 2022. – P. 201–205. – URL: https://repo.ssau.ru/bitstream/Perspektivnyeinformacionnyetehnologii/TEHNIChESKOE-ZRENIE-VSOSTAVE-SISTEMAVTOMATIChESKOGO-UPRAVLENIYaDVIZhENIEM-POEZDOV-101915/1/978-5-93424-880-3_2022_201-205.pdf.

20. Baranov, A. G. Train traffic control system : patent / A. G. Baranov, V. A. Voronin, G. K. Kiselgof [et al.]. – 2014. – URL: https://patents.google.com/patent/RU2508218C1/ru.

21. Gradusov, A. N. Device for monitoring electric train locomotive control and driver vigilance : patent / A. N. Gradusov, V. F. Ivanov, S. V. Kudryashov [et al.]. — 2025. — URL: https://patents.google.com/patent/RU2851854 C1/ru

22. Wierwille, W. W. Research on vehiclebased driver status/performance monitoring: development, validation, and refinement of algorithms for detection of driver drowsiness. Final report / W. W. Wierwille, S. S. Wreggit, C. L. Kirn [et al.]. — National Highway Traffic Safety Administration, 1994. — 247 p.

23. Dinges, D. F. PERCLOS: A valid psychophysiological measure of alertness as assessed by psychomotor vigilance / D. F. Dinges, R. Grace. — Federal Highway Administration, Office of Motor Carrier Research; Standards, 1998. — 4 p.

24. Abe, T. PERCLOS-based technologies for detecting drowsiness: current evidence and future directions / T. Abe // SLEEP Advances. — 2023. — Vol. 4, № 1. — P. zpad006. — DOI: 10.1093/sleepadvances/zpad006

25. Trutschel, U. PERCLOS: An alertness measure of the past / U. Trutschel, B. Sirois, D. Sommer [et al.] // Proceedings of the Sixth International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design. — 2011. — DOI: 10.17077/drivingassessment.1394

26. Bentivoglio, A. R. Analysis of blink rate patterns in normal subjects / A. R. Bentivoglio, S. B. Bressman, E. Cassetta [et al.] // Movement Disorders. — 1997. — Vol. 12, № 6. — P. 1028–1034. — DOI: 10.1002/mds.870120629

27. Tsubota, K. Dry eyes and video display terminals / K. Tsubota, K. Nakamori // New England Journal of Medicine. — 1993. — Vol. 328, № 8. — P. 584. — DOI: 10.1056/NEJM199302253280817

28. Caffier, P. P. Experimental evaluation of eye-blink parameters as a drowsiness measure / P. P. Caffier, U. Erdmann, P. Ullsperger // European Journal of Applied Physiology. — 2003. — Vol. 89, № 3. — P. 319–325. — DOI: 10.1007/s00421-003-0807-5

29. Schleicher, R. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? / R. Schleicher, N. Galley, S. Briest [et al.] // Ergonomics. — 2008. — Vol. 51, № 7. — P. 982–1010. — DOI: 10.1080/00140130701817062

30. Buckley, R. J. Attention lapses and behavioural microsleeps during tracking, psychomotor vigilance, and dual tasks / R. J. Buckley, W. S. Helton, C. R. H. Innes [et al.] // Consciousness and Cognition. — 2016. — Vol. 45. — P. 174–183. — DOI: 10.1016/j.concog.2016.09.002

31. Viola, P. Rapid object detection using a boosted cascade of simple features / P. Viola, M. Jones // Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001). — 2001. — P. I-511-I-518. — DOI: 10.1109/CVPR.2001.99051

32. Zhang, K. Joint face detection and alignment using multitask cascaded convolutional networks / K. Zhang, Z. Zhang, Z. Li [et al.] // IEEE Signal Processing Letters. — 2016. — Vol. 23, № 10. — P. 1499–1503. — DOI: 10.1109/LSP.2016.2603342. — URL: https://arxiv.org/abs/1604.02878

33. Deng, J. RetinaFace: Single-shot multilevel face localisation in the wild / J. Deng, J. Guo, E. Ververas [et al.] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). — 2020. — P. 5202– 5211. — DOI: 10.1109/CVPR42600.2020.00525

34. Sagonas, C. 300 Faces in-the-wild challenge: The first facial landmark localization challenge / C. Sagonas, G. Tzimiropoulos, S. Zafeiriou [et al.] // 2013 IEEE International Conference on Computer Vision Workshops. — 2013. — P. 397–403. — DOI: 10.1109/ICCVW.2013.59

35. Kazemi, V. One millisecond face alignment with an ensemble of regression trees / V. Kazemi, J. Sullivan // 2014 IEEE Conference on Computer Vision and Pattern Recognition. — 2014. — P. 1867–1874. — DOI: 10.1109/CVPR.2014.241

36. Kartynnik, Y. Real-time facial surface geometry from monocular video on mobile GPUs / Y. Kartynnik, A. Ablavatski, I. Grishchenko [et al.]. — 2019. — URL: https://arxiv.org/abs/1907.06724

37. Soukupová, T. Real-time eye blink detection using facial landmarks / T. Soukupová, J. Čech // 21st Computer Vision Winter Workshop (CVWW 2016) / eds. L. Čehovin, R. Mandeljc, V. Štruc. — Rimske Toplice, Slovenia. — 2016. — URL: https://cmp.felk.cvut.cz/ftp/articles/cech/Soukupova-CVWW-2016.pdf

38. Weng, C.-H. Driver drowsiness detection via a hierarchical temporal deep belief network / C.-H. Weng, Y.-H. Lai, S.-H. Lai // Computer Vision – ACCV 2016 Workshops. — Springer. — 2017. — P. 117–133. — DOI: 10.1007/978-3-319-54526-4_9

39. Ghoddoosian, R. A realistic dataset and baseline temporal model for early drowsiness detection / R. Ghoddoosian, M. Galib, V. Athitsos // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). — 2019. — P. 178– 187. — DOI: 10.1109/CVPRW.2019.00027. — URL: https://arxiv.org/abs/1904.0731

40. Vogel, A. P. Acoustic analysis of the effects of sustained wakefulness on speech / A. P. Vogel, J. Fletcher, P. Maruff // The Journal of the Acoustical Society of America. — 2010. — Vol. 128, № 6. — P. 3747–3756. — DOI: 10.1121/1.3506349

41. Greeley, H. P. Fatigue estimation using voice analysis / H. P. Greeley, J. Berg, E. Friets [et al.] // Behavior Research Methods. — 2007. — Vol. 39, № 3. — P. 610–619. — DOI: 10.3758/BF03193033

42. Krajewski, J. Acoustic sleepiness detection: Framework and validation of a speech-adapted pattern recognition approach / J. Krajewski, A. Batliner, M. Golz // Behavior Research Methods. — 2009. — Vol. 41, № 3. — P. 795–804. — DOI: 10.3758/BRM.41.3.795.

43. Schuller, B. The INTERSPEECH 2011 speaker state challenge / B. Schuller, S. Steidl, A. Batliner [et al.] // Interspeech 2011. — 2011. — P. 3201–3204. — DOI: 10.21437/Interspeech.2011-801

44. Kashevnik, A. Multimodal corpus design for audio-visual speech recognition in vehicle cabin / A. Kashevnik, I. Lashkov, A. Axyonov [et al.] // IEEE Access. — 2021. — Vol. 9. — P. 34986–35003. — DOI: 10.1109/ACCESS.2021.3062752

45. Dasgupta, A. A smartphone-based drowsiness detection and warning system for automotive drivers / A. Dasgupta, D. Rahman, A. Routray // IEEE Transactions on Intelligent Transportation Systems. — 2019. — Vol. 20, № 11. — P. 4045–4054. — DOI: 10.1109/TITS.2018.2879609

46. Virk, J. S. A multimodal feature fusion framework for sleep-deprived fatigue detection to prevent accidents / J. S. Virk, M. Singh, M. Singh [et al.] // Sensors. — 2023. — Vol. 23, № 8. — P. 4129. — DOI: 10.3390/s23084129

47. Jo, J. Detecting driver drowsiness using feature-level fusion and user-specific classification / J. Jo, S. J. Lee, K. R. Park [et al.] // Expert Systems with Applications. — 2014. — Vol. 41, № 4. — P. 1139–1152. — DOI: 10.1016/j.eswa.2013.07.108

48. Cao, S. Optimized driver fatigue detection method using multimodal neural networks / S. Cao, P. Feng, W. Kang [et al.] // Scientific Reports. — 2025. — Vol. 15. — P. 12240. — DOI: 10.1038/s41598-025-86709-1