Skip to main content
Log in

Visualizing natural language interaction for conversational in-vehicle information systems to minimize driver distraction

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

In this paper we investigate how natural language interfaces can be integrated with cars in a way such that their influence on driving performance is being minimized. In particular, we focus on how speech-based interaction can be supported through a visualization of the conversation. Our work is motivated by the fact that speech interfaces (like Alexa, Siri, Cortana, etc.) are increasingly finding their way into our everyday life. We expect such interfaces to become commonplace in vehicles in the future. Cars are a challenging environment, since speech interaction here is a secondary task that should not negatively affect the primary task, that is driving. At the outset of our work, we identify the design space for such interfaces. We then compare different visualization concepts in a driving simulator study with 64 participants. Our results yield that (1) text summaries support drivers in recalling information and enhances user experience but can also increase distraction, (2) the use of keywords minimizes cognitive load and influence on driving performance, and (3) the use of icons increases the attractiveness of the interface.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. https://www.apple.com/ios/carplay/, last access: 2017-08-12.

  2. https://www.android.com/auto/, last access: 2017-08-12.

  3. https://itunes.apple.com/de/app/remote-s-for-tesla/id991623777, last access: 2017-08-12.

  4. https://www.theguardian.com/technology/2017/feb/07/amazon-alexa-car-logitech-zerotouch-voice-services-assistant, last access: 2017-08-12.

  5. https://azure.microsoft.com/de-de/services/cognitive-services/speech/,last access: 2017-05-14.

  6. https://www.ibm.com/watson/developercloud/text-to-speech.html, last access: 2017-05-14.

  7. http://www.ergoneers.com/eye-tracking/dikablis-glasses/, last access: 2017-05-13.

References

  1. Asif A, Boll S (2010) Where to turn my car?: comparison of a tactile display and a conventional car navigation system under high load condition. In: Proceedings of the 2nd international conference on automotive user interfaces and interactive vehicular applications, Automotiveui ’10. ACM, New York, pp 64–71. https://doi.org/10.1145/1969773.1969786

  2. Ballagas R, Borchers J, Rohs M, Sheridan JG (2006) The smart phone: a ubiquitous input device. IEEE Pervasive Comput 5(1):70–77. https://doi.org/10.1109/MPRV.2006.18

    Article  Google Scholar 

  3. Beanland V, Fitzharris M, Young KL, Lenné MG (2013) Driver inattention and driver distraction in serious casualty crashes: Data from the australian national crash in-depth study. Accid Anal Prev 54:99–107. https://doi.org/10.1016/j.aap.2012.12.043 http://www.sciencedirect.com/science/article/pii/S000145751300047X

  4. Bellegarda JR (2014) Spoken language understanding for natural interaction: the siri experience. In: Natural interaction with robots, knowbots and smartphones: putting spoken dialog systems into practice. Springer, New York, pp 3–14. https://doi.org/10.1007/978-1-4614-8280-2

  5. Braun M, Pfleging B, Alt F (2018) A survey to understand emotional situations on the road and what they mean for affective automotive uis. Multimodal Technol. Interact 2(4). https://doi.org/10.3390/mti2040075 http://www.mdpi.com/2414-4088/2/4/75

  6. Braun M, Pfleging B, Broy N, Alt F (2017) A design space for conversational in-vehicle information systems. In: Proceedings of the 19th international conference on human–computer interaction with mobile devices and services, MobileHCI ’17. ACM, New York

  7. Braun M, Völkel ST, Wiegand G, Puls T, Steidl D, WeißD, Alt F (2018) The smile is the new like: controlling music with facial expressions to minimize driver distraction. In: Proceedings of the 17th international conference on mobile and ubiquitous multimedia, MUM ’18. ACM, New York. https://doi.org/10.1145/3282894.3289729

  8. Bubb H (2003) Fahrerassistenz—primär ein beitrag zum komfort oder fuer die sicherheit? In: Der Fahrer im 21. Jahrhundert: Anforderungen, Anwendungen, Aspekte für Mensch-Maschine-Systeme; Tagung Braunschweig, 2. und 3. Juni 2003, VDI-Berichte 1768. VDI-Verlag, Düsseldorf, pp 25–44

  9. Caird JK, Willness CR, Steel P, Scialfa C (2008) A meta-analysis of the effects of cell phones on driver performance. Accid Anal Prev 40(4):1282–1293

    Article  Google Scholar 

  10. Commission of the European Communities (2007) Commission recommendation of 22 december 2006 on safe and efficient in-vehicle information and communication systems: Update of the european statement of principles on human machine interface (2007/78/ec). Off J Eur Union 50(L32): 200–241. http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2007.032.01.0200.01.ENG

  11. Cooper JM, Strayer DL (2015) Mental workload of voice interactions with 6 real-world driver interfaces. In: Proceedings of the eighth international driving symposium on human factors in driver assessment, training, and vehicle design, DA ’15, pp 2–8. http://drivingassessment.uiowa.edu/sites/default/files/DA2015/papers/002.pdf

  12. Department of Transportation, National Highway Traffic Safety Administration (2013) Visual-manual nhtsa driver distraction guidelines for in-vehicle electronic device. Federal Regist 78(81), pp 24, 817–24, 890. https://federalregister.gov/a/2013-09883

  13. Ehsani J, Li K, Simons-Morton BG (2015) Teenage drivers portable electronic device use while driving. In: Proceedings of the eighth international driving symposium on human factors in driver assessment, training, and vehicle design, DA ’15, pp 219–225. http://drivingassessment.uiowa.edu/sites/default/files/DA2015/papers/034.pdf

  14. Engström J, Johansson E, Östlund J (2005) Effects of visual and cognitive load in real and simulated motorway driving. Transp Res Part F: Traffic Psychol Behav 8(2):97–120. https://doi.org/10.1016/j.trf.2005.04.012 http://www.sciencedirect.com/science/article/pii/S1369847805000185. The relationship between distraction and driving performance: towards a test regime for in-vehicle information systems In-vehicle information systems

  15. Eyben F, Wöllmer M, Schuller B (2010) Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the 18th ACM international conference on Multimedia. ACM, pp 1459–1462

  16. Foley JP, Young R, Angell L, Domeyer JE (2013) Towards operationalizing driver distraction. In: Proceedings of the seventh international driving symposium on human factors in driver assessment, training, and vehicle design, pp 57–63. http://drivingassessment.uiowa.edu/sites/default/files/DA2013/Papers/010_Foley_0.pdf

  17. Geiger M, Zobl M, Bengler K, Lang M (2001) Intermodal differences in distraction effects while controlling automotive user interfaces. In: Proceedings of HCI international 2001 volume 1: usability evaluation and interface design: cognitive engineering, intelligent agents and virtual reality, HCII ’01. Lawrence Erlbaum, Hillsdale

  18. Green P (2013) Standard definitions for driving measures and statistics: overview and status of recommended practice j2944. In: Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’13. ACM, New York, pp 184–191. https://doi.org/10.1145/2516540.2516542

  19. Green PA (2008) Driver interface/hmi standards to minimize driver distraction/overload. In: Convergence 2008 conference proceedings. Society of Automotive Engineers, SAE International, Warrendale (2008)

  20. Group ADFTW (2006) Statement of principles, criteria and verification procedures on driver interactions with advanced in- vehicle information and communication systems. Technical report, Alliance of Automobile Manufacturers

  21. Hackenberg L, Bongartz S, Härtle C, Leiber P, Baumgarten T, Sison JA (2013) International evaluation of nlu benefits in the domain of in-vehicle speech dialog systems. In: Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’13. ACM, New York, pp 114–120. https://doi.org/10.1145/2516540.2516553

  22. Haeuslschmid R, Pfleging B, Alt F (2016) A design space to support the development of windshield applications for the car. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, pp 5076–5091

  23. Harvey C, Stanton NA, Pickering CA, McDonald M, Zheng P (2011) In-vehicle information systems to meet the needs of drivers. Int J Hum Comp Interact 27(6):505–522. https://doi.org/10.1080/10447318.2011.555296

    Article  Google Scholar 

  24. Hassenzahl M, Koller F, Burmester M (2008) Der user experience (ux) auf der spur: zum einsatz von www.attrakdiff.de. In: Brau H, Diefenbach S, Hassenzahl M, Koller F, Peissner M, Röse K (eds) Tagungsband UP08. Fraunhofer Verlag, Stuttgart, pp 78–82

  25. Heenan A, Herdman CM, Brown MS, Robert N (2014) Effects of conversation on situation awareness and working memory in simulated driving. Hum Factors 56(6):1077–1092. https://doi.org/10.1177/0018720813519265 PMID: 25277018

    Article  Google Scholar 

  26. Hernandez J, McDuff D, Benavides X, Amores J, Maes P, Picard R (2014) Autoemotive: bringing empathy to the driving experience to manage stress. In: Proceedings of the 2014 companion publication on designing interactive systems, DIS Companion ’14. ACM, New York, pp 53–56. https://doi.org/10.1145/2598784.2602780

  27. ITFG on Driver Distraction (2013) Report on user interface requirements for automotive applications. Technical report, International Telecommunication Union

  28. Jakus G, Dicke C, Sodnik J (2015) A user study of auditory, head-up and multi-modal displays in vehicles. Appl Ergon 46, Part A:184–192. https://doi.org/10.1016/j.apergo.2014.08.008 http://www.sciencedirect.com/science/article/pii/S0003687014001471

  29. Kapoor A, Burleson W, Picard RW (2007) Automatic prediction of frustration. Int J Hum Comput Stud 65(8):724–736. https://doi.org/10.1016/j.ijhcs.2007.02.003 http://www.sciencedirect.com/science/article/pii/S1071581907000377

  30. Kennington C, Kousidis S, Baumann T, Buschmeier H, Kopp S, Schlangen D (2014) Better driving and recall when in-car information presentation uses situationally-aware incremental speech output generation. In: Proceedings of the 6th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’14. ACM, New York, pp 7:1–7:7. https://doi.org/10.1145/2667317.2667332

  31. Kern D, Marshall P, Schmidt A (2010) Gazemarks: gaze-based visual placeholders to ease attention switching. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2093–2102. ACM

  32. Kern D, Schmidt A (2009) Design space for driver-based automotive user interfaces. In: Proceedings of the 1st international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’09. ACM, New York, pp 3–10. https://doi.org/10.1145/1620509.1620511

  33. Kim S, Dey AK (2009) Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’09. ACM, New York, pp 133–142. https://doi.org/10.1145/1518701.1518724

  34. Kountouriotis GK, Wilkie RM, Gardner PH, Merat N (2015) Looking and thinking when driving: the impact of gaze and cognitive load on steering. Transp Res Part F: Traffic Psychol Behav 34:108–121. https://doi.org/10.1016/j.trf.2015.07.012 http://www.sciencedirect.com/science/article/pii/S1369847815001163

  35. Laberge-Nadeau C, Maag U, Bellavance F, Lapierre SD, Desjardins D, Messier S, Saidi A (2003) Wireless telephones and the risk of road crashes. Accid Anal Prev 35(5):649–660

    Article  Google Scholar 

  36. Lamble D, Kauranen T, Laakso M, Summala H (1999) Cognitive load and detection thresholds in car following situations: safety implications for using mobile (cellular) telephones while driving. Accid Anal Prev 31(6):617–623

    Article  Google Scholar 

  37. Large DR, Burnett G, Anyasodo B, Skrypchuk L (2016) Assessing cognitive demand during natural language interactions with a digital driving assistant. In: Proceedings of the 8th international conference on automotive user interfaces and interactive vehicular applications, Automotive’UI 16. ACM, New York, pp 67–74. https://doi.org/10.1145/3003715.3005408

  38. Lo VEW, Green PA (2013) Development and evaluation of automotive speech interfaces: Useful information from the human factors and the related literature. Int J Veh Technol 2013:924,170:1–924,170:13. https://doi.org/10.1155/2013/924170

  39. Luger E, Sellen A (2016) “like having a really bad pa”: the gulf between user expectation and experience of conversational agents. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ’16. ACM, New York, pp 5286–5297. https://doi.org/10.1145/2858036.2858288

  40. McEvoy SP, Stevenson MR, McCartt AT, Woodward M, Haworth C, Palamara P, Cercarelli R (2005) Role of mobile phones in motor vehicle crashes resulting in hospital attendance: a case-crossover study. Bmj 331(7514):428

    Article  Google Scholar 

  41. Müller C, Weinberg G (2011) Multimodal input in the car, today and tomorrow. IEEE MultiMed 18(1):98–103. https://doi.org/10.1109/MMUL.2011.14

    Article  Google Scholar 

  42. Nafari M, Weaver C (2013) Augmenting visualization with natural language translation of interaction: a usability study. Comput Graph Forum 32(3pt4):391–400. https://doi.org/10.1111/cgf.12126

    Article  Google Scholar 

  43. Neßelrath R, Feld M (2013) Towards a cognitive load ready multimodal dialogue system for in-vehicle human-machine interaction. In: Adjunct proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI Adjunct ’13, pp 49–52. http://www.auto-ui.org/13/docs/aui_adjunct_proceedings_final.pdf#page=49

  44. Oh A, Fox H, Van Kleek M, Adler A, Gajos K, Morency LP, Darrell T (2002) Evaluating look-to-talk: a gaze-aware interface in a collaborative environment. In: CHI ’02 extended abstracts on human factors in computing systems, CHI EA ’02. ACM, New York, pp 650–651. https://doi.org/10.1145/506443.506528

  45. Paas F, Tuovinen JE, Tabbers H, Van Gerven PWM (2003) Cognitive load measurement as a means to advance cognitive load theory. Educ Psychol 38(1):63–71. https://doi.org/10.1207/S15326985EP3801_8

    Article  Google Scholar 

  46. Pauzié A (2008) A method to assess the driver mental workload: the driving activity load index (dali). Intell Transp Syst IET 2(4):315–322. https://doi.org/10.1049/iet-its:20080023  http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4706518&tag=1

  47. Petzoldt T, Bellem H, Krems JF (2014) The critical tracking task: a potentially useful method to assess driver distraction? Hum Factors 56(4):789–808. https://doi.org/10.1177/0018720813501864

    Article  Google Scholar 

  48. Pieraccini R, Dayanidhi K, Bloom J, Dahan JG, Phillips M, Goodman BR, Prasad KV (2004) Multimodal conversational systems for automobiles. Commun. ACM 47(1):47–49. https://doi.org/10.1145/962081.962104

    Article  Google Scholar 

  49. Ranney TA, Garrott WR, Goodman MJ (2001) Nhtsa driver distraction research: Past, present, and future. In: Proceedings of the 17th international technical conference on the enhanced safety of vehicles. u.s. department of transportation, National Highway Traffic Safety Administration (2001). http://www-nrd.nhtsa.dot.gov/pdf/esv/esv17/proceed/00177.pdf

  50. Redelmeier DA, Tibshirani RJ (1997) Association between cellular-telephone calls and motor vehicle collisions. N Engl J Med 336(7):453–458

    Article  Google Scholar 

  51. Reimer B, Mehler B (2013) The effects of a production level “voice-command” interface on driver behavior: summary findings on reported workload, physiology, visual attention, and driving performance. Technical Report White Paper 2013-18A, MIT AgeLab—Massachusetts Institute of Technology (2013). http://agelab.mit.edu/files/MIT_AgeLab_White_Paper_2013-18A_(Voice_Interfaces).pdf

  52. Rümelin S, Siegl P, Butz A (2013) Could you please... investigating cooperation in the car. In: Adjunct proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI Adjunct ’13, pp 61–64. http://www.auto-ui.org/13/docs/aui_adjunct_proceedings_final.pdf#page=63

  53. SAE International (2013) Operational definitions of driving performance measures and statistics. Technical Report SAE J2944, SAE International. http://standards.sae.org/wip/j2944/. Proposed Draft

  54. Strayer DL, Cooper JM, Turrill J, Coleman JR, Hopman RJ (2015) The smartphone and the driver’s cognitive workload: a comparison of apple, google, and microsoft’s intelligent personal assistants. Technical report, AAA Foundation for Traffic Safety, Washington, DC. https://www.aaafoundation.org/sites/default/files/strayerIIIa_FINALREPORT.pdf

  55. Tsimhoni O, Smith D, Green P (2004) Address entry while driving: speech recognition versus a touch-screen keyboard. Hum Factors 46(4):600–610. https://doi.org/10.1518/hfes.46.4.600.56813 PMID: 15709323

    Article  Google Scholar 

  56. Vollrath M, Maciej J, Niederée U (2008) In-car distraction study—final report. Technical report, Kognitions-und Ingenieurpsychologie, Technische Universität Carolo-Wilhelmina zu Braunschweig, Braunschweig

  57. Ward N (2006) Non-lexical conversational sounds in American english. Pragmat Cognit 14(1):129–182

    Article  Google Scholar 

  58. Weinberg G, Harsham B, Medenica Z (2011) Investigating huds for the presentation of choice lists in car navigation systems. In: Proceedings of the sixth international driving symposium on human factors in driver assessment, training and vehicle design, DA ’11, pp 195–202

  59. White MP, Eiser JR, Harris PR (2004) Risk perceptions of mobile phone use while driving. Risk Anal 24(2):323–334. https://doi.org/10.1111/j.0272-4332.2004.00434.x

    Article  Google Scholar 

  60. Yan B, Weng F, Feng Z, Ratiu F, Raya M, Meng Y, Varges S, Purver M, Lien A, Scheideck T, Raghunathan B, Lin F, Mishra R, Lathrop B, Zhang Z, Bratt H, Peters S (2007) A conversational in-car dialog system. In: Proceedings of human language technologies: the annual conference of the North American chapter of the association for computational linguistics: demonstrations, NAACL-demonstrations ’07. Association for Computational Linguistics, Stroudsburg, pp 23–24. http://dl.acm.org/citation.cfm?id=1614164.1614176

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Michael Braun.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Braun, M., Broy, N., Pfleging, B. et al. Visualizing natural language interaction for conversational in-vehicle information systems to minimize driver distraction. J Multimodal User Interfaces 13, 71–88 (2019). https://doi.org/10.1007/s12193-019-00301-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-019-00301-2

Keywords

Navigation