Skip to main content
SpringerLink
Log in

Distributed Evolution of Deep Autoencoders

  • Conference paper
  • First Online:
Intelligent Computing

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 283))

  • 2416 Accesses

Abstract

Autoencoders have seen wide success in domains ranging from feature selection to information retrieval. Despite this success, designing an autoencoder for a given task remains a challenging undertaking due to the lack of firm intuition on how the backing neural network architectures of the encoder and decoder impact the overall performance of the autoencoder. In this work we present a distributed system that uses an efficient evolutionary algorithm to design a modular autoencoder. We demonstrate the effectiveness of this system on the tasks of manifold learning and image denoising. The system beats random search by nearly an order of magnitude on both tasks while achieving near linear horizontal scaling as additional worker nodes are added to the system.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/j-haj/brokered-deep-learning.

References

  1. Abdi, H., Williams, L.J.: Principal component analysis. WIREs Comput. Stat. 2(4), 433–459 (2010)

    Article  Google Scholar 

  2. Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15(6), 1373–1396 (2003)

    Article  Google Scholar 

  3. Brock, A., Lim, T., Ritchie, J.M., Weston, N.: SMASH: one-shot model architecture search through hypernetworks. CoRR arXiv:1708.05344 (2017)

  4. Charte, F., Rivera, A.J., Martínez, F., del Jesus, M.J.: Automating autoencoder architecture configuration: an evolutionary approach. In: International Work-Conference on the Interplay Between Natural and Artificial Computation, pp. 339–349. Springer, Cham (2019)

    Google Scholar 

  5. Coates, A., Ng, A.Y., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Gordon, G.J., Dunson, D.B., Dudík, M. (eds.) Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, 11–13 April 2011, volume 15 of JMLR Proceedings, pp. 215–223. JMLR.org (2011)

    Google Scholar 

  6. Google. grpc

    Google Scholar 

  7. Ha, D., Dai, A.M., Le, Q.V.: Hypernetworks. CoRR arXiv:1609.09106 (2016)

  8. Hajewski, J., Oliveira, S.: A scalable system for neural architecture search. In: IEEE Computing and Communication Workshop and Conference, CCWC 2020 (2020)

    Google Scholar 

  9. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778. IEEE Computer Society (2016)

    Google Scholar 

  10. Hindman, B., et al.: Mesos: a platform for fine-grained resource sharing in the data center. In: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, NSDI 2011, pp. 295–308, Berkeley, CA, USA. USENIX Association (2011)

    Google Scholar 

  11. Koutník, J., Cuccu, G., Schmidhuber, J., Gomez, F.: Evolving large-scale neural networks for vision-based reinforcement learning. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, GECCO 2013, pp. 1061–1068. ACM, New York (2013)

    Google Scholar 

  12. Krizhevsky, A., Nair, V., Hinton, G.: CIFAR-10 (Canadian Institute for Advanced Research)

    Google Scholar 

  13. Kyriakides, G., Margaritis, K.G.: Neural architecture search with synchronous advantage actor-critic methods and partial training. In: Proceedings of the 10th Hellenic Conference on Artificial Intelligence, SETN 2018, pp. 34:1–34:7. ACM, New York (2018)

    Google Scholar 

  14. Lander, S., Shang, Y.: EvoAE - a new evolutionary method for training autoencoders for deep learning networks. In: 2015 IEEE 39th Annual Computer Software and Applications Conference, vol. 2, pp. 790–795, July 2015

    Google Scholar 

  15. Lin, T., Zha, H.: Riemannian manifold learning. IEEE Trans. Pattern Anal. Mach. Intell. 30(5), 796–809 (2008)

    Article  Google Scholar 

  16. Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K.: Hierarchical representations for efficient architecture search. CoRR arXiv:1711.00436 (2017)

  17. Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. CoRR arXiv:1806.09055 (2018)

  18. Ma, Y., Fu, Y.: Manifold Learning Theory and Applications. CRC Press, Boca Raton (2011)

    Book  Google Scholar 

  19. Miikkulainen, R., et al.: Evolving deep neural networks. CoRR arXiv:1703.00548 (2017)

  20. Paszke, A., et al.: Automatic differentiation in PyTorch. In: NIPS Autodiff Workshop (2017)

    Google Scholar 

  21. Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4095–4104, Stockholmsmässan, Stockholm Sweden. PMLR, July 2018

    Google Scholar 

  22. Real, E., et al.: Large-scale evolution of image classifiers. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6–11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2902–2911. PMLR (2017)

    Google Scholar 

  23. Rensin, D.K.: Kubernetes - scheduling the future at cloud scale (2015)

    Google Scholar 

  24. Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)

    Article  Google Scholar 

  25. Sciuto, C., Yu, K., Jaggi, M., Musat, C., Salzmann, M.: Evaluating the search phase of neural architecture search. CoRR arXiv:1902.08142 (2019)

  26. Suganuma, M., Ozay, M., Okatani, T.: Exploiting the potential of standard convolutional autoencoders for image restoration by evolutionary search. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4771–4780, Stockholmsmässan, Stockholm, Sweden. PMLR, July 2018

    Google Scholar 

  27. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. CoRR arXiv:1512.00567 (2015)

  28. Talwalkar, A., Kumar, S., Rowley, H.: Large-scale manifold learning. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)

    Google Scholar 

  29. Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)

    Article  Google Scholar 

  30. Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: Efficient object localization using convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, 7–12 June 2015, pp. 648–656. IEEE Computer Society (2015)

    Google Scholar 

  31. Varda, K.: Protocol buffers: Google’s data interchange format. Technical report, Google, June 2008

    Google Scholar 

  32. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, PA.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, ICML 2008, pp. 1096–1103. ACM, New York (2008)

    Google Scholar 

  33. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)

    MathSciNet  MATH  Google Scholar 

  34. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53

    Chapter  Google Scholar 

  35. Zhang, C., Ren, M., Urtasun, R.: Graph hypernetworks for neural architecture search. CoRR arXiv:1810.05749 (2018)

  36. Zoph,B., Le, Q.V.: Neural architecture search with reinforcement learning (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Salesforce, San Francisco, USA

    Jeff Hajewski

  2. Department of Computer Science, University of Iowa, Iowa City, USA

    Jeff Hajewski & Suely Oliveira

  3. Amazon, Washington, D.C., USA

    Xiaoyu Xing

Authors
  1. Jeff Hajewski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Suely Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoyu Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jeff Hajewski .

Editor information

Editors and Affiliations

  1. Faculty of Science and Engineering, Saga University, Saga, Japan

    Kohei Arai

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hajewski, J., Oliveira, S., Xing, X. (2022). Distributed Evolution of Deep Autoencoders. In: Arai, K. (eds) Intelligent Computing. Lecture Notes in Networks and Systems, vol 283. Springer, Cham. https://doi.org/10.1007/978-3-030-80119-9_6

Download citation

Publish with us

Policies and ethics

Search

Navigation