Abstract
User-intent classification is a sub-task in natural language understanding of human-computer dialogue systems. To reduce the data volume requirement of deep learning for intent classification, this paper proposes a transfer learning method for Chinese user-intent classification task, which is based on the Bidirectional Encoder Representations from Transformers (BERT) pre-trained language model. First, a simulation experiment on 31 Chinese participants was implemented to collect first-handed Chinese human-computer conversation data. Then, the data was augmented through back-translation and randomly split into the training dataset, validation dataset and test dataset. Next, the BERT model was fine-tuned into a Chinese user-intent classifier. As a result, the predicting accuracy of the BERT classifier reaches 99.95%, 98.39% and 99.89% on the training dataset, validation dataset and test dataset. The result suggests that the application of BERT transfer learning has reduced the data volume requirement for Chinese intent classification task to a satiable level.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chen, H., Liu, X., Yin, D., Tang, J.: A survey on dialogue systems: recent advances and new frontiers. SIGKDD Explor. Newsl. 19, 25–35 (2017)
Schuurmans, J., Frasincar, F.: Intent classification for dialogue utterances. IEEE Intell. Syst. 35, 82–88 (2020)
Shridhar, K., et al.: Subword semantic hashing for intent classification on small datasets. arXiv, pp. 1–6 (2018)
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (Almost) from scratch. J. Mach. Learn. Res. 12, 2493–2537 (2011)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016-December, pp. 770–778 (2016)
Vaswani, A., et al.: Attention is all you need. Advance Neural Information Processing Systems 2017-December, pp. 5999–6009 (2017)
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), pp. 328–339 (2018)
Peters, M.E., et al.: Deep contextualized word representations. In: NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, pp. 2227–2237 (2018)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, pp. 4171–4186 (2019)
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv (2019)
Zhang, Y., et al.: Dialogpt : large-scale generative pre-training for conversational response generation. arXiv, pp. 270–278 (2019)
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: a lite BERT for self-supervised learning of language representations. arXiv, pp. 1–17 (2019)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv 21, 1–67 (2019)
Petrock, V.: US Voice Assistant Users 2019 Who, What, When, Where and Why (2019)
Edunov, S., Ott, M., Auli, M., Grangier, D.: Understanding back-translation at scale. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 (2020)
Sun, C., Qiu, X., Xu, Y., Huang, X.: How to fine-tune BERT for text classification? In: Sun, M., Huang, X., Ji, H., Liu, Z., Liu, Y. (eds) Chinese Computational Linguistics. CCL 2019. LNCS, vol. 11856. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32381-3_16
Acknowledgments
We would like to thank all participants in the experiment. This paper is supported by the Key Platforms and Scientific Research Projects Foundation of Guangdong Education Department (Project approval no. 2016WTSCX002).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, H., Peng, H. (2021). Using BERT Model for Intent Classification in Human-Computer Dialogue Systems to Reduce Data Volume Requirement. In: Ayaz, H., Asgher, U., Paletta, L. (eds) Advances in Neuroergonomics and Cognitive Engineering. AHFE 2021. Lecture Notes in Networks and Systems, vol 259. Springer, Cham. https://doi.org/10.1007/978-3-030-80285-1_59
Download citation
DOI: https://doi.org/10.1007/978-3-030-80285-1_59
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-80284-4
Online ISBN: 978-3-030-80285-1
eBook Packages: EngineeringEngineering (R0)