Abstract
Occluded face recognition, which has an attractive application in the visual analysis field, is challenged by the missing cues due to heavy occlusions. Recently, several face inpainting methods based on generative adversarial networks (GANs) fill in the occluded parts by generating images fitting the real image distributions. They can lead to a visually natural result and satisfy human perception. However, these methods fail to capture the identity attributes, thus the inpainted faces may be recognized at a low accuracy by machine. To enable the convergence of human perception and machine perception, this paper proposes an Identity Preserving Generative Adversary Networks (IP-GANs) to jointly inpaint and recognize occluded faces. The IP-GANs consists of an inpainting network for regressing missing facial parts, a global-local discriminative network for guiding the inpainted face to the real conditional distribution, a parsing network for enhancing structure consistence and an identity network for recovering missing identity cues. Especially, the novel identity network suppresses the identity diffusion by constraining the feature consistence from the early subnetwork of a well-trained face recognition network between the inpainted face and its corresponding ground-true. In this way, it regularizes the inpaintor, enforcing the generated faces to preserve identity attributes. Experimental results prove the proposed IP-GANs capable of dealing with varieties of occlusions and producing photorealistic and identity-preserving results, promoting occluded face recognition performance.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of SIGGRAPH, vol. 4, no. 9, pp. 417–424 (2005)
Deng, Y., Dai, Q., Zhang, Z.: Graph Laplace for occluded face completion and recognition. IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc. 20(8), 2329–2338 (2011)
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: International Conference on Neural Information Processing Systems, pp. 2672–2680 (2014)
Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst (October 2007)
Ishikawa, H., Ishikawa, H., Ishikawa, H.: Globally and locally consistent image completion. ACM (2017)
Jia, Y., Shelhamer, E., Donahue, J., et al.: Caffe: convolutional architecture for fast feature embedding (2014). arXiv:1408.5093
Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)
Lu, H., Li, B., Zhu, J., Li, Y., Li, Y., Xu, X., He, L., Li, X., Li, J., Serikawa, S.: Wound intensity correction and segmentation with convolutional neural networks. Concur. Comput. Pract. Exp. 29(6) (2016)
Lu, H., Li, Y., Chen, M., Kim, H., Serikawa, S.: Brain intelligence: go beyond artificial intelligence. Mob. Netw. Appl. 23(2), 368–375 (2017)
Lu, H., Li, Y., Mu, S., Wang, D., Kim, H., Serikawa, S.: Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet Things J 5(4), 2315–2322 (2018)
Oh, H.J., Lee, K.M., Sang, U.L.: Occlusion invariant face recognition using selective local non-negative matrix factorization basis images. Image Vis. Comput. 26(11), 1515–1523 (2008)
Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference (2015)
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536–2544 (2016)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR (2015). arXiv:1511.06434
Serikawa, S., Lu, H.: Underwater image dehazing using joint trilateral filter. Pergamon Press, Inc. (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR (2014). arXiv:1409.1556
Smith, B.M., Zhang, L., Brandt, J., Lin, Z., Yang, J.: Exemplar-based face parsing. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3484–3491 (2013)
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. Computer Science (2013)
Wang, C., Xu, C., Wanga, C., Tao, D.: Perceptual adversarial networks for image-to-image transformation. IEEE Trans. Image Process. 27(8), 4066–4079 (2018)
Wright, J., Ganesh, A., Zhou, Z., Wagner, A., Ma, Y.: Demo: robust face recognition via sparse representation. In: IEEE International Conference on Automatic Face & Gesture Recognition, pp. 1–2 (2009)
Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: International Conference on Neural Information Processing Systems, pp. 341–349 (2012)
Xu, X., He, L., Lu, H., Gao, L., Ji, Y.: Deep adversarial metric learning for cross-modal retrieval. In: World Wide Web-internet & Web Information Systems, pp. 1–16 (2018)
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. CoRR (2018). arXiv:1801.07892
Zhang, S., He, R., Sun, Z., Tan, T.: Demeshnet: blind face inpainting for deep meshface verification. IEEE Trans. Inf. Forensics Secur. 13(3), 637–647 (2017)
Zhang, W., Shan, S., Chen, X., Gao, W.: Local Gabor binary patterns based on Kullback–Leibler divergence for partially occluded face recognition. IEEE Signal Process. Lett. 14(11), 875–878 (2007)
Acknowledgements
This work is supported in part by the National Natural Science Foundation of China (61772513 & 61402463), the National Key Research and Development Plan (2016YFC0801005) and the International Cooperation Project of Institute of Information Engineering, Chinese Academy of Sciences (Y7Z0511101). Shiming Ge is also supported by Youth Innovation Promotion Association, CAS.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Li, C., Ge, S., Hua, Y., Liu, H., Jin, X. (2020). Occluded Face Recognition by Identity-Preserving Inpainting. In: Lu, H. (eds) Cognitive Internet of Things: Frameworks, Tools and Applications. ISAIR 2018. Studies in Computational Intelligence, vol 810. Springer, Cham. https://doi.org/10.1007/978-3-030-04946-1_41
Download citation
DOI: https://doi.org/10.1007/978-3-030-04946-1_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04945-4
Online ISBN: 978-3-030-04946-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)