Abstract
Neural network has been proved to be effective in sequence annotation task. Since it does not require task-specific knowledge, the same network structure can be easily applied to a wide range of applications. However, domain sequence tagging tasks still suffer from lack of available data. First, there is fewer available domain annotated data to train the recurrent neural network adequately. Second, the corpus maybe not available for domain-specific word embedding training. In this paper, we explore the problem of transfer learning of domain name entity recognition task. We proposed a modified skip-gram model for training cross-domain word embeddings, and we use source task with a large number of annotations (e.g. NER on CoNLL2003) to improve the performance on target task with fewer available annotations (e.g. NER on biomedical dataset). We evaluate our approach on a range of sequence tagging benchmarks, and the results show that significant improvement can be achieved using our approach.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Ando, R.K., Zhang, T.: A framework for learning predictive structures from multiple tasks and unlabeled data. J. Mach. Learn. Res. 6(Nov), 1817–1853 (2005)
Bollegala, D., Maehara, T., Kawarabayashi, K.i.: Unsupervised cross-domain word representation learning. arXiv preprint arXiv:1505.07184 (2015)
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12(Aug), 2493–2537 (2011)
Finkel, J.R., Manning, C.D.: Hierarchical bayesian domain adaptation. In: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics. pp. 602–610. Association for Computational Linguistics (2009)
Glorot, X., Bordes, A., Bengio, Y.: Domain adaptation for large-scale sentiment classification: A deep learning approach. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11). pp. 513–520 (2011)
Kim, J.D., Ohta, T., Tateisi, Y., Tsujii, J.: Genia corpusa semantically annotated corpus for bio-textmining. Bioinformatics 19(\({\rm suppl\_1}\)), i180–i182 (2003)
Kim, J.D., Ohta, T., Tsuruoka, Y., Tateisi, Y., Collier, N.: Introduction to the bio-entity recognition task at jnlpba. In: Proceedings of the international joint workshop on natural language processing in biomedicine and its applications. pp. 70–75. Association for Computational Linguistics (2004)
Krallinger, M., Leitner, F., Rabal, O., Vazquez, M., Oyarzabal, J., Valencia, A.: Chemdner: the drugs and chemical names extraction challenge. J. Cheminformatics 7(1), S1 (2015)
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360 (2016)
McClosky, D., Charniak, E., Johnson, M.: Automatic domain adaptation for parsing. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. pp. 28–36. Association for Computational Linguistics (2010)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv:1301.3781 (2013)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems. pp. 3111–3119 (2013)
Rau, L.F.: Extracting company names from text. In: Artificial Intelligence Applications, 1991. In: Seventh IEEE Conference on Proceedings, vol. 1, pp. 29–32. IEEE (1991)
Rei, M., Crichton, G.K., Pyysalo, S.: Attending to characters in neural sequence labeling models. arXiv:1611.04361 (2016)
Ritter, A., Clark, S., Etzioni, O., et al.: Named entity recognition in tweets: an experimental study. In: Proceedings of the Conference on Empirical Methods in natural Language Processing. pp. 1524–1534. Association for Computational Linguistics (2011)
Schnabel, T., Schütze, H.: Flors: Fast and simple domain adaptation for part-of-speech tagging. Trans. Assoc. Comput. Linguist. 2, 15–26 (2014)
Sienčnik, S.K.: Adapting word2vec to named entity recognition. In: Proceedings of the 20th nordic conference of computational linguistics, nodalida 2015, may 11-13, 2015, vilnius, lithuania. pp. 239–243. No. 109, Linköping University Electronic Press (2015)
Smith, L., Tanabe, L.K., nee Ando, R.J., Kuo, C.J., Chung, I.F., Hsu, C.N., Lin, Y.S., Klinger, R., Friedrich, C.M., Ganchev, K., et al.: Overview of biocreative ii gene mention recognition. Genome Biol. 9(2), S2 (2008)
Yarowsky, D., Ngai, G., Wicentowski, R.: Inducing multilingual text analysis tools via robust projection across aligned corpora. In: Proceedings of the first international conference on Human language technology research. pp. 1–8. Association for Computational Linguistics (2001)
Zirikly, A., Hagiwara, M.: Cross-lingual transfer of named entity recognizers without parallel corpora. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). vol. 2, pp. 390–396 (2015)
Acknowledgements
We would like to thank Prof. Li from Northeast University, China and Dr. Zheng from IBM Innovation Lab, without whose help, our work could not be finished so smoothly. We also thank all the reviewers for their useful feedback to the earlier draft of this paper and the anonymous reviewers for their constructive comments to revise the paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Cao, M., Zhang, C., Li, D., Zheng, Q., Luo, L. (2020). Transfer Learning for Cross-Domain Sequence Tagging Tasks. In: Arai, K., Bhatia, R. (eds) Advances in Information and Communication. FICC 2019. Lecture Notes in Networks and Systems, vol 70. Springer, Cham. https://doi.org/10.1007/978-3-030-12385-7_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-12385-7_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-12384-0
Online ISBN: 978-3-030-12385-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)