Skip to main content
SpringerLink
Log in

ISRGAN: Improved Super-Resolution Using Generative Adversarial Networks

  • Conference paper
  • First Online:
Advances in Computer Vision (CVC 2019)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 943))

Included in the following conference series:

  • 2734 Accesses

Abstract

In this paper, we propose an approach for single image super-resolution (SISR) using generative adversarial network (GAN). The SISR has been an attractive research topic over the last two decades and it refers to the reconstruction of a high resolution (HR) image from a single low resolution (LR) observation. Recently, SISR using convolutional neural networks (CNNs) obtained remarkable performance in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics. Despite this, these methods suffer with a serious drawback in terms of visualization quality of the SR images; the results look overly-smoothed. This is due to the loss function in those methods has a pixel level difference which increases the values of PSNR and SSIM metrics; however the visualization quality is degraded. The GAN has a capability to generate visually appealable solutions. It can also recover the high-frequency texture details due to the discrimination process involved in GAN. Here, we propose improved single image super-resolution using GAN (ISRGAN) with the concept of densely connected deep convolutional networks for image super-resolution. Our proposed method consists two networks: ISRNet and ISRGAN. The ISRNet is trained using MSE based loss function to achieve higher PSNR and SSIM values and ISRGAN is trained by using a combination of VGG based perceptual loss and adversarial loss in order to improve the perceptual quality of the SR images. This training step forces the SR results more towards the natural image manifold. The efficiency of the proposed method is verified by conducting experiments on the different benchmark testing datasets and it shows that the proposed method of ISRGAN outperforms in terms of perception when compared to the other state-of-the-art GAN based SISR techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/jbhuang0604/SelfExSR.

  2. 2.

    http://cv.snu.ac.kr/research/VDSR/.

  3. 3.

    http://cv.snu.ac.kr/research/DRCN/.

  4. 4.

    http://vllab.ucmerced.edu/wlai24/LapSRN/.

  5. 5.

    http://webdav.tuebingen.mpg.de/pixel/enhancenet/.

  6. 6.

    https://twitter.app.box.com/s/lcue6vlrd01ljkdtdkhmfvk7vtjhetog.

References

  1. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  2. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)

  3. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint arXiv:1609.04802 (2016)

  4. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711. Springer (2016)

    Google Scholar 

  5. Gunturk, B.K., Batur, A.U., Altunbasak, Y., Hayes, M.H., Mersereau, R.M.: Eigenface-domain super-resolution for face recognition. IEEE Trans. Image Process. 12(5), 597–606 (2003)

    Article  Google Scholar 

  6. Goto, T., Fukuoka, T., Nagashima, F., Hirano, S., Sakurai, M.: Super-resolution system for 4K-HDTV. In: 2014 22nd International Conference on Pattern Recognition, ICPR, pp. 4453–4458. IEEE (2014)

    Google Scholar 

  7. Peled, S., Yeshurun, Y.: Superresolution in MRI: application to human white matter fiber tract visualization by diffusion tensor imaging. Magn. Reson. Med. 45(1), 29–35 (2001)

    Article  Google Scholar 

  8. Thornton, M.W., Atkinson, P.M., Holland, D.: Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping. Int. J. Remote Sens. 27(3), 473–491 (2006)

    Article  Google Scholar 

  9. Zhang, L., Zhang, H., Shen, H., Li, P.: A super-resolution reconstruction algorithm for surveillance images. Signal Process. 90(3), 848–859 (2010)

    Article  Google Scholar 

  10. Yang, C.-Y., Ma, C., Yang, M.-H.: Single-image super-resolution: a benchmark. In: European Conference on Computer Vision, pp. 372–386. Springer (2014)

    Google Scholar 

  11. Hayat, K.: Super-resolution via deep learning. arXiv preprint arXiv:1706.09077 (2017)

  12. Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)

    Article  Google Scholar 

  13. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 349–356. IEEE (2009)

    Google Scholar 

  14. Kim, K.I., Kwon, Y.: Single-image super-resolution using sparse regression and natural image prior. IEEE Trans. Pattern Anal. Mach. Intell. 32(6), 1127–1133 (2010)

    Article  Google Scholar 

  15. Timofte, R., De Smet, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: 2013 IEEE International Conference on Computer Vision, ICCV, pp. 1920–1927. IEEE (2013)

    Google Scholar 

  16. Yang, J., Lin, Z., Cohen, S.: Fast image super-resolution based on in-place example regression. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1059–1066. IEEE (2013)

    Google Scholar 

  17. Peleg, T., Elad, M.: A statistical prediction model based on sparse representations for single image super-resolution. IEEE Trans. Image Process. 23(6), 2569–2582 (2014)

    Article  MathSciNet  Google Scholar 

  18. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  19. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

    Google Scholar 

  20. Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1637–1645 (2016)

    Google Scholar 

  21. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: The IEEE Conference on Computer Vision and Pattern Recognition, CVPR, vol. 1, no. 4 (2017)

    Google Scholar 

  22. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  23. Lai, W.-S., Huang, J.-B., Ahuja, N., Yang, M.-H.: Fast and accurate image super-resolution with deep Laplacian pyramid networks. arXiv preprint arXiv:1710.01992 (2017)

  24. Barron, J.T.: A more general robust loss function. arXiv preprint arXiv:1701.03077 (2017)

  25. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)

    Google Scholar 

  26. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN. arXiv preprint arXiv:1701.07875 (2017)

  27. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.: Improved training of Wasserstein GANs. arXiv preprint arXiv:1704.00028 (2017)

  28. Berthelot, D., Schumm, T., Metz, L.: BEGAN: boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717 (2017)

  29. Bruna, J., Sprechmann, P., LeCun, Y.: Super-resolution with deep convolutional sufficient statistics. arXiv preprint arXiv:1511.05666 (2015)

  30. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  31. Sajjadi, M.S., Schölkopf, B., Hirsch, M.: EnhanceNet: single image super-resolution through automated texture synthesis. In: 2017 IEEE International Conference on Computer Vision, ICCV, pp. 4501–4510. IEEE (2017)

    Google Scholar 

  32. Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2261–2269 (2017)

    Google Scholar 

  33. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, vol. 1, no. 2, p. 3 (2017)

    Google Scholar 

  34. Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)

  35. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

    Google Scholar 

  36. Odena, A., Dumoulin, V., Olah, C.: Deconvolution and checkerboard artifacts. Distill 1(10), e3 (2016)

    Article  Google Scholar 

  37. Dosovitskiy, A., Brox, T.: Generating images with perceptual similarity metrics based on deep networks. In: Advances in Neural Information Processing Systems, pp. 658–666 (2016)

    Google Scholar 

  38. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: International Conference on Curves and Surfaces, pp. 711–730. Springer (2010)

    Google Scholar 

  39. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)

    Google Scholar 

  40. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the Eighth IEEE International Conference on Computer Vision, ICCV 2001, vol. 2, pp. 416–423. IEEE (2001)

    Google Scholar 

  41. Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 20th International Conference on Pattern Recognition, ICPR, pp. 2366–2369 (2010)

    Google Scholar 

  42. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  43. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, pp. 265–283 (2016). https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf

  44. Dang-Nguyen, D.-T., Pasquini, C., Conotter, V., Boato, G.: RAISE: a raw images dataset for digital image forensics. In: Proceedings of the 6th ACM Multimedia Systems Conference, pp. 219–224. ACM (2015)

    Google Scholar 

  45. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

Download references

Acknowledgment

We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.

Author information

Authors and Affiliations

  1. Sardar Vallabhbhai National Institute of Technology, Surat, 395007, Gujrat, India

    Vishal Chudasama & Kishor Upla

Authors
  1. Vishal Chudasama
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kishor Upla
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kishor Upla .

Editor information

Editors and Affiliations

  1. Saga University, Saga, Saga, Japan

    Kohei Arai

  2. The Science and Information (SAI) Organization, Bradford, West Yorkshire, UK

    Supriya Kapoor

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chudasama, V., Upla, K. (2020). ISRGAN: Improved Super-Resolution Using Generative Adversarial Networks. In: Arai, K., Kapoor, S. (eds) Advances in Computer Vision. CVC 2019. Advances in Intelligent Systems and Computing, vol 943. Springer, Cham. https://doi.org/10.1007/978-3-030-17795-9_9

Download citation

Publish with us

Policies and ethics

Search

Navigation