Abstract
The illumination effect is essential for the realistic results in images which are created by inserting virtual objects into real scene. For outdoor scenes, automatic estimation of sun orientation condition from a single outdoor image is fundamental for inserting 3D models to a single image. Traditional methods for outdoor sun orientation estimation often use handcraft illumination features or cues. These cues heavily rely on the experiences of human and pre-processing progresses using other image understanding technologies such as shadow and sky detection, geometry recovery and intrinsic image decomposition, which limit their performances. We propose an end to end way of outdoor sun orientation estimation via a novel deep convolutional neural network (DCNN), which directly outputs the orientation of the sun from an outdoor image. Our proposed SunOriNet contains a contact layer that directly contacts the intermediate feature maps to the high-level ones and learns hierarchical features automatically from a large-scale image dataset with annotated sun orientations. The experiments reveal that our DCNN can well estimate sun orientation from a single outdoor image. The estimation accuracy of our method outperforms model state-of-the-art DCNN based methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Basri, R., Jacobs, D., Kemelmacher, I.: Photometric stereo with general, unknown lighting. Int. J. Comput. Vis. 72, 239–257 (2007)
Debevec, P.: Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques, New York, pp. 189–198 (1998)
Frahm, J.M., Koeser, K., Grest, D., et al.: Markerless augmented reality with light source estimation for direct illumination. In: Proceedings of the European Conference on Visual Media Production, London, pp. 211–220 (2005)
Lalonde, J.F., Efros, A., Narasimhan, S.: Estimating natural illumination from a single outdoor image. In: Proceedings of the IEEE International Conference on Computer Vision, Kyoto, pp. 183–190 (2009)
Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Estimating the natural illumination conditions from a single outdoor image. Int. J. Comput. Vis. (IJCV). 98(2), 123–145 (June 2012)
Chen, X., Jin, X., Wang, K.: Lighting virtual objects in a single image via coarse scene understanding. Sci. China Inf. Sci. (SCIS) 57(9), 1–14 (2014)
Liu, Y., Gevers, T., Li, X.: Estimation of sunlight direction using 3D object models. IEEE Trans. Image Process. 24(3), 932–942 (March 2015)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 1097–1105 (2012)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: ICCV (2015)
Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Webcam clip art: appearance and illuminant transfer from time-lapse sequences. ACM Trans. Graph. (SIGGRAPH Asia 2009), 28(5) (2009)
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding (2014). arXiv:1408.5093
Zhang, J., Lalonde, J.F.: Learning high dynamic range from outdoor panoramas. In: IEEE International Conference on Computer Vision (ICCV), Venice, pp. 4529–4538 (2017)
Ma, W.-C., Wang, S., Brubaker, M.A., Fidler, S., Urtasun, R.: Find your way by observing the sun and other semantic cues. In: IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017
Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., Lalonde, J.-F.: Deep outdoor illumination estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Lu, H., Li, Y., Mu, S., Wang, D., Kim, H., Serikawa, S.: Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet Things J. (2017). https://doi.org/10.1109/jiot.2017.2737479
Lu, H., Li, Y., Chen, M., Kim, H., Serikawa, S.: Brain intelligence: go beyond artificial intelligence. Mob. Netw. Appl. 1–8 (2017)
Lu, H., Li, B., Zhu, J.: Wound intensity correction and segmentation with convolutional neural networks. Concurr. Comput. Pract. Exp. (2017). https://doi.org/10.1002/cpe.3927
Lu, H., Li, Y., Uemura, T., Kim, H., Serikawa, S.: Low illumination underwater light field images reconstruction using deep convolutional neural networks. Futur. Gener. Comput. Syst. (2018). https://doi.org/10.1016/j.future.2018.01.001
Serikawa, S., Lu, H.: Underwater image dehazing using joint trilateral filter. Comput. Electr. Eng. 40(1), 41–50 (2014)
Acknowledgements
We thank all the reviewers and ACs. This work is partially supported by the National Natural Science Foundation of China (Grant Nos. 61772047, 61772513, 61402021), the Science and Technology Project of the State Archives Administrator (Grant No. 2015-B-10), the open funding project of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (Grant No. BUAA-VR-16KF-09), the Fundamental Research Funds for the Central Universities (Grant No. 3122014C017), the China Postdoctoral Science Foundation (Grant No. 2015M581841), and the Postdoctoral Science Foundation of Jiangsu Province (Grant No. 1501019A).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Jin, X. et al. (2020). Synthesizing Virtual-Real Artworks Using Sun Orientation Estimation. In: Lu, H. (eds) Cognitive Internet of Things: Frameworks, Tools and Applications. ISAIR 2018. Studies in Computational Intelligence, vol 810. Springer, Cham. https://doi.org/10.1007/978-3-030-04946-1_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-04946-1_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04945-4
Online ISBN: 978-3-030-04946-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)