Artificial Intelligence Applied to Brain-Computer Interfacing with Eye-Tracking for Computer-Aided Conceptual Architectural Design in Virtual Reality Using Neurofeedback

  • Claudiu Barsan-PipuEmail author
Conference paper


This paper proposes a new method of integrating the latest technologies joining brain-computer interfaces (BCIs) with eye-tracking (E-T) and applying this combination to conceptual design for architecture using AI-driven neurofeedback (NFB) to help identify the designer’s intent and respond dynamically to it. Using integrated state-of-the-art E-T and BCI solutions for the latest head-mounted display (HMD) devices, this paper aims to provide an insight into the applicability of these solutions and their potential benefits and pitfalls to creating innovative conceptual design instruments. By harnessing artificial intelligence (AI) within a Game Engine (GE) context, the proposed solution tries to create a new procedural design-interaction approach that uses neurofeedback to learn and adapt to its user’s design intent without the need to truly understand the complex decision-making processes taking place inside the designer’s mind. While limited in its scope, this approach raises some interesting topics and questions that are discussed in more detail in the last section of the paper.


Brain-Computer interface (BCI) Eye-tracking (E-T) Virtual reality (VR) Neurofeedback (NFB) Artificial intelligence (AI) Conceptual architectural design 



The author will like to thank Adam Molnar and Brian Selzer from Neurable Inc. for supplying the Neurable EEG Dev Kit that enabled this research. Furthermore, the author expresses his gratitude for the feedback provided by Prof. Neil Leach, as the PhD supervisor for the “Digital Futures” Ph.D. Program, CAUP, Tongji University, Shanghai (CN), where this academic investigation took place.


  1. 1.
    Huang, X., White, M., Burry, M.: Design globally, immerse locally: a synthetic design approach by integrating agent-based modelling with virtual reality. In: CAADRIA 2018, Beijing, pp. 473–482Google Scholar
  2. 2.
    Barsan-Pipu, C.: Neomorph, V.R.: A multi-user virtual reality conceptual design platform for architecture and urbanism using procedural game technologies. In: TMCE 2018, Las Palmas de Gran Canaria, pp. 237–250Google Scholar
  3. 3.
    Kiefer, P., Giannopoulos, L., Raubal, M., Duchowski, A.: Eye tracking for spatial. Spat. Cogn. Comput. 17, 1–19 (2017)CrossRefGoogle Scholar
  4. 4.
    Goldberg, J., Kotval, X.: Computer interface evaluation using eye movements: methods and constructs. Int. J. Ind. Ergon. 24, 631–645 (1999)CrossRefGoogle Scholar
  5. 5.
    Jacob, R.J., Karn, K.S.: Eye tracking in human-computer interaction and usability research: ready to deliver the promises. In: Mind 2003 2(3)Google Scholar
  6. 6.
    Zhang, L.M., Jeng, T.S., Zhang, R.X.: Integration of virtual reality, 3-D Eye-tracking, and protocol analysis for re-designing street space. In: CAADRIA(23), Beijing, vol. 1, pp. 431–440 (2018)Google Scholar
  7. 7.
    Chen, H., Dey, A., Billinghurst, M., Lindeman, R.W.: Exploring pupil dilation in emotional virtual reality environments. In: ICAT-EGVE, Adelaide, pp. 1–8 (2017)Google Scholar
  8. 8.
    Sherstyuk, A., Vincent, D., Treskunov, A.: Toward natural selection in virtual reality. IEEE Comput. Graph. Appl. II(30), pp. 93–96 (2010)Google Scholar
  9. 9.
    Emotiv. EMOTIV (2019).
  10. 10.
    Neurable Inc. Neurable (2019).
  11. 11.
    Jatupaiboon, N., Pan-ngum, S., Israsena, P.: Real-time EEG-based happiness detection system. Sci. World J. (2013)Google Scholar
  12. 12.
    Lin, Y.P., Wang, C.H., Wu, T.L., Jeng, S.K., Chen, J.H.: EEG-based emotion recognition in music listening: a comparison of schemes for multiclass support vector machine, pp. 489–492. Speech Signal Process., Acoustics (2009)Google Scholar
  13. 13.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction Cambridge. The MIT Press, Cambridge (2016)zbMATHGoogle Scholar
  14. 14.
    Schulman, J., Dhariwal, F., Radford, P., Klimov, O.: Proximal policy optimization (2017). arXiv:1707.06347
  15. 15.
    Juliani, A., Berges, V.P., Vckay, E., Gao, Y., Henry, H., Mattar, M., et al. Unity: A General Platform for Intelligent Agents (2018). arXiv:1809.02627
  16. 16.
    Konda, V.R., Tsitsiklis, J.N.: Actor-critic algorithms. Adv. Neural Inf. pp. 1008–1014 (2000)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  1. 1.College of Architecture and Urban PlanningTongji UniversityShanghaiChina

Personalised recommendations