Skip to main content
SpringerLink
Log in

Interruption Timing Prediction via Prosodic Task Boundary Model for Human-Machine Teaming

  • Conference paper
  • First Online:
Book cover Advances in Information and Communication (FICC 2019)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 70))

Included in the following conference series:

Abstract

Human-machine teaming aims to meld the human cognitive strengths with the unique capabilities of smart machines to create intelligent teams adaptive to rapidly changing circumstances. One major problem within human-machine teaming is a lack of communication skills on the part of the machine such as the inability to know when to communicate information to or interrupt human teammates. To address this issue, an intelligent interruption system that monitors the speech within human-machine teaming interactions and predicts when to interrupt based on where human teammates are within the primary task is proposed. The intelligent interruption system leverages the raw audio within a simulated human-machine teaming interaction, extracts prosodic information, and predicts task boundaries as candidate interruption timings. Various machine learning techniques are evaluated as a prosody-only task boundary model and their task boundary detection performance is compared. The prosody-only task boundary model is implemented in real-time and the system latency and task boundary detection performance is evaluated. The final results indicate that although prosodic information processes with a low latency making it tractable in real-time, the prosody-only task boundary model performance is degraded in robust dialogues of human-machine teaming interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adams, J.: Human machine teaming research (2017)

    Google Scholar 

  2. Stefik, M.: Half-human, half-computer? meet the modern centaur (2017)

    Google Scholar 

  3. Arroyo, E., Selker, T.: Attention and intention goals can mediate disruption in human-computer interaction, pp. 454–470 (2011)

    Chapter  Google Scholar 

  4. Bailey, B.P., Konstan, J.A.: On the need for attention-aware systems: measuring effects of interruption on task performance, error rate, and affective state. Comput. Hum. Behav. 22(4), 685–708 (2006)

    Article  Google Scholar 

  5. Czerwinski, M., Cutrell, E., Horvitz, E.: Instant messaging and interruption: influence of task type on performance. In: OZCHI 2000 Conference Proceedings, vol. 356, pp. 361–367 (2000)

    Google Scholar 

  6. Monk, C.A., Boehm-Davis, D.A., Gregory Trafton, J.: The attentional costs of interrupting task performance at various stage. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 46, pp. 1824–1828. SAGE Publications, Los Angeles, CA (2002)

    Article  Google Scholar 

  7. Czerwinski, M., Cutrell, E., Horvitz, E.: Instant messaging: effects of relevance and timing. People and Computers XIV: Proceedings of HCI 2, 71–76 (2000)

    Google Scholar 

  8. Adamczyk, P.D., Bailey, B.P.: If not now, when?: the effects of interruption at different moments within task execution. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 271–278. ACM (2004)

    Google Scholar 

  9. Zijlstra, F.R.H., Roe, R.A., Leonora, A.B., Krediet, I.: Temporal factors in mental work: effects of interrupted activities. J. Occup. Organ. Psychol. 72(2), 163–185 (1999)

    Article  Google Scholar 

  10. Scott McCrickard, D., Chewar, C.M., Somervell, J.P., Ndiwalana, A.: A model for notification systems evaluation?assessing user goals for multitasking activity. ACM Trans. Comput.-Hum. Interact. (TOCHI), 10, 312–338 (2003)

    Article  Google Scholar 

  11. McFarlane, D.C., Latorella, K.A.: The scope and importance of human interruption in human-computer interaction design. Hum.-Comput. Interact. 17(1), 1–61 (2002)

    Article  Google Scholar 

  12. Dabbish, L., Kraut, R.E., Controlling interruptions: awareness displays and social motivation for coordination. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, pp. 182–191. ACM (2004)

    Google Scholar 

  13. Latorella, K.A.: Investigating interruptions: an example from the flightdeck. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 40, pp. 249–253. SAGE Publications, Los Angeles, CA (1996)

    Article  Google Scholar 

  14. Kreifeldt, J.G., McCarthy, M.E.: Interruption as a test of the user-computer interface (1981)

    Google Scholar 

  15. Rubinstein, J.S., Meyer, D.E., Evans, J.E.: Executive control of cognitive processes in task switching. J. Exp. Psychol.: Hum. Percept. Perform. 27(4), 763 (2001)

    Google Scholar 

  16. Altmann, E.M., Trafton, J.G.: Task interruption: resumption lag and the role of cues. In: Proceedings of the Cognitive Science Society, vol. 26 (2004)

    Google Scholar 

  17. Sasse, A., Johnson, C., et al.: Coordinating the interruption of people in human-computer interaction. Hum.-Comput. Interact. 99, 295 (1999)

    Google Scholar 

  18. Horvitz, E.C.M.C.E., Notification, disruption, and memory: effects of messaging interruptions on memory and performance. In: Human-Computer Interaction: INTERACT’01: IFIP TC. 13 International Conference on Human-Comupter Interaction, 9th–13th July 2001, Tokyo, Japan, p. 263. IOS Press (2001)

    Google Scholar 

  19. McFarlane, D.C.: Interruption of people in human-computer interaction: a general unifying definition of human interruption and taxonomy. Technical Report, Office of Naval Research, Arlington VA (1997)

    Book  Google Scholar 

  20. Bailey, B.P., Iqbal, S.T.: Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management. ACM Trans. Comput.-Hum. Interact. (TOCHI), 14(4), 21 (2008)

    Article  Google Scholar 

  21. Iqbal, S.T., Bailey, B.P.: Investigating the effectiveness of mental workload as a predictor of opportune moments for interruption. In: CHI’05 Extended Abstracts on Human Factors in Computing Systems, pp. 1489–1492. ACM (2005)

    Google Scholar 

  22. Peters, N., Romigh, G., Bradley, G., Raj, B.: When to interrupt: a comparative analysis of interruption timings within collaborative communication tasks. In: Advances in Human Factors and System Interactions, pp. 177–187. Springer (2017)

    Google Scholar 

  23. Horvitz, E., Principles of mixed-initiative user interfaces. In: Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pp. 159–166. ACM (1999)

    Google Scholar 

  24. Horvitz, E., Apacible, J.: Learning and reasoning about interruption. In: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 20–27. ACM (2003)

    Google Scholar 

  25. Horvitz, E., Koch, P., Kadie, C.M., Jacobs, A.: Coordinate: probabilistic forecasting of presence and availability. In: Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence, pp. 224–233. Morgan Kaufmann Publishers Inc. (2002)

    Google Scholar 

  26. Begole, J.B., Matsakis, N.E., Tang, J.C.: Lilsys: sensing unavailability. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, pp. 511–514. ACM (2004)

    Google Scholar 

  27. Fogarty, J., Lai, J., Christensen, J.: Presence versus availability: the design and evaluation of a context-aware communication client. Int. J. Hum.-Comput. Stud. 61(3), 299–317 (2004)

    Article  Google Scholar 

  28. Iqbal, S.T., Bailey, B.P., Leveraging characteristics of task structure to predict the cost of interruption. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 741–750. ACM (2006)

    Google Scholar 

  29. Miyata, Y., Norman, D.A.: Psychological issues in support of multiple activities. In: User Centered System Design: New Perspectives on Human-Computer Interaction, pp. 265–284 (1986)

    Google Scholar 

  30. Bannon, L., Cypher, A., Greenspan, S., Monty, M.L.: Evaluation and analysis of users’ activity organization. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 54–57. ACM (1983)

    Google Scholar 

  31. Miller, S.L.: Window of opportunity: Using the interruption lag to manage disruption in complex tasks. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 46, pp. 245–249. SAGE Publications, Los Angeles, CA (2002)

    Article  Google Scholar 

  32. Peters, N., Romigh, G., Bradley, G., Raj, B., A comparative analysis of human-mediated and system-mediated interruptions for multi-user, multitasking interactions. In: International Conference on Applied Human Factors and Ergonomics, pp. 339–347. Springer (2017)

    Google Scholar 

  33. Mushin, I., Stirling, L., Fletcher, J., Wales, R.: Discourse structure, grounding, and prosody in task-oriented dialogue. Discourse Process. 35(1), 1–31 (2003)

    Article  Google Scholar 

  34. Mixdorff, H., Quantitative analysis of prosody in task-oriented dialogs. In: Speech Prosody 2004, International Conference (2004)

    Google Scholar 

  35. Syrdal, A.K., Kim, Y.-J.: Dialog speech acts and prosody: considerations for TTS. In: Proceedings of Speech Prosody, pp. 661–665 (2008)

    Google Scholar 

  36. Hastie, H.W., Poesio, M., Isard, S.: Automatically predicting dialogue structure using prosodic features. Speech Commun. 36(1), 63–79 (2002)

    Article  MATH  Google Scholar 

  37. Carletta, J., Isard, S., Doherty-Sneddon, G., Isard, A., Kowtko, J.C., Anderson, A.H.: The reliability of a dialogue structure coding scheme. Comput. Linguist. 23(1), 13–31 (1997)

    Google Scholar 

  38. Eyben, F., Weninger, F., Gross, F., Schuller, B.: Recent developments in opensmile, the munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 835–838. ACM (2013)

    Google Scholar 

  39. Banerjee, S.: Simple example code and generic function for random forests. https://www.mathworks.com/matlabcentral/fileexchange/51985-simple-example-code-and-generic-function-for-random-forests (2016)

  40. Ho, T.K.: Random decision forests. In: Proceedings of the Third International Conference on Document Analysis and Recognition, vol. 1, pp. 278–282. IEEE (1995)

    Google Scholar 

  41. Peters, N., Raj, B., Romigh, G.: Topic and prosodic modeling for interruption management in multi-use multitasking communication interactions. In: AI-HRI (2018)

    Google Scholar 

  42. Russell, S., Norvig, P., Intelligence, Artificial: “A modern approach”. In: Artificial Intelligence, vol. 25, p. 27. Prentice-Hall, Englewood Cliffs (1995)

    Google Scholar 

  43. Cortes, C., Vapnik, V.: Support-vector networks. Machine Learn. 20(3), 273–297 (1995)

    Google Scholar 

  44. Swerts, M., Ostendorf, M.: Prosodic and lexical indications of discourse structure in human-machine interactions. Speech Commun. 22(1), 25–41 (1997)

    Article  Google Scholar 

  45. Shriberg, E., Stolcke, A., Hakkani-Tür, D., Tür, G.: Prosody-based automatic segmentation of speech into sentences and topics. Speech Commun. 32(1), 127–154 (2000)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Air Force Research Laboratory, Wright Patterson Air Force Base, Dayton, OH, 45433, USA

    Nia Peters

Authors
  1. Nia Peters
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nia Peters .

Editor information

Editors and Affiliations

  1. Faculty of Science and Engineering, Saga University, Saga, Japan

    Kohei Arai

  2. The Science and Information (SAI) Organization, Bradford, UK

    Rahul Bhatia

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Peters, N. (2020). Interruption Timing Prediction via Prosodic Task Boundary Model for Human-Machine Teaming. In: Arai, K., Bhatia, R. (eds) Advances in Information and Communication. FICC 2019. Lecture Notes in Networks and Systems, vol 70. Springer, Cham. https://doi.org/10.1007/978-3-030-12385-7_37

Download citation

Publish with us

Policies and ethics

Search

Navigation