Cyborgian Approach of Eco-interaction Design Based on Machine Intelligence and Embodied Experience

Abstract

The proliferation of digital technology has swelled the amount of time people spent in cyberspace and weakened our sensibility of the physical world. Human beings in this digital era are already cyborgs as the smart devices have become an integral part of our life. Imagining a future where human totally give up mobile phones and embrace nature is neither realistic nor reasonable. What we should aim to explore is the opportunities and capabilities of digital technology in terms of fighting against its own negative effect - cyber addiction, and working as a catalyst that re-embeds human into outdoor world.

Cyborgian systems behave through embedded intelligence in the environment and discrete wearable devices for human. In this way, cyborgian approach enables designers to take advantages of digital technologies to achieve two objectives: one is to improve the quality of environment by enhancing our understanding of non-human creatures; the other is to encourage a proper level of human participation without disturbing eco-balance.

Finally, this paper proposed a cyborgian eco-interaction design model which combines top-down and bottom-up logics and is organized by the Internet of Things, so as to provide a possible solution to the concern that technologies are isolating human and nature.

1 Tracing Cyborgian Theory and Embodied Cognition

1.1 A Hybrid of Part Clock Part Swarm [10]

Since the concept of “cybernetics” was introduced by Norbert Wiener in the 1940s to deal with complex systems of communications and control in machines and animals [24], the word has been imparted the meaning of “cross-species communication”. The term “Cyborgian” or “Cyborg”, a combination of “Cybernetic” and “Organism”, was coined in 1960 by NASA scientists [4]. Different from bio-robot or artificial intelligence, “Cyborg” is a rejection of human-machine dualism by obscuring the rigid boundary and advocating the man-machine symbiosis.

By the end of 1980s, the concept of “Cyborg” has already widespread in science fictions such as Ghost in Shell and Blade Runner. The world of tomorrow predicted in these futurism works, with the highly hybrid cyber-organ relationship as an essential feature, is becoming today’s reality.

Attempts and practices of cyborg individuals has been conducted in various areas, starting at a relatively basic and safe level - as the substitutes of lost or damaged body parts. Then, the continuous breakthrough of technology propelled the development and acceptance of “enhancement prosthetics” - for example, British artist Neil Harbisson has had a cyborg antenna implanted in his head that allows him to extend his perception of colors beyond the human visual spectrum [17]. However, implanted cyborg remains a controversial issue. Opponents concern that this technology would aggravate social polarization and impair social order and ethics.

In a broad sense, implant surgery is not necessary for becoming a cyborg. Everyone holding a smartphone is a cyborg. Because the external apparatus has become an integral part of us, as a cyber-extension of our organic corporeity. Embedded and external devices share the same purpose – to enhance the perception, communication, interconnection and control of everything (Fig. 1).

1.2 The Importance of the Presence and the Bodily Experience

Embodied cognition is a promising theory that has been developing rapidly since the “postcognitivism revolution” [3] in the middle of last century. Embodied cognition challenges traditional theories such as Connectionism and Computationalism, which hold the notion of Disembodiment and Mind–body Dualism [25]. It opens a new chapter of cognitive psychology with emphasis on the indispensability of human body in the process of cognition.

In 1945, phenomenological philosopher Maurice Merleau-Ponty claimed that “the body is our general medium for having a world” [14]. In 1979, James J. Gibson, who fathered the school of ecological psychology, also expressed similar idea that we acquire the information in the environment through our active body [8]. It is widely acknowledged that emotions influence behaviours, while according to the theory of embodied cognition, vice versa (Fig. 2).

“A designer and a cognitive scientist seem like an unlikely pair … both are trying to decode how humans interact with the world” [13]. Relational art/aesthetics is a mode or tendency in fine art practice with embodied cognition as one of its theoretical foundations. It is defined as a set of artistic practices which depart from the concerns of human relations and social context, instead of independent and private spaces [2]. Relational art values the encounter between an audience and an artwork, and the encounters between people.

On the basis of these two theory, here comes a question: since the corporeity is the main source of knowing the world and the others, when our body is augmented by advancing cyber technologies, how would human cognitive abilities and experiences be improved with the help of machine intelligence, and how would this be beneficial for the construction of a user-experience-oriented interactive environment? (Fig. 3).

Fig. 1.

Cyborg human. Author’s own work.

Fig. 2.

Embodied mind. Author unknown.

Fig. 3.

Relational art - geological memory. Image credit: teamLab

2 How Cyborgian Approach Activates Plants?

To encourage human-nature interaction, one way is to make plants more sensible, intelligent and interactive. This chapter is going to research on how cyborgian approach influences the procedures of plant’s behaviour, namely sense (input), think (algorithm and feedback) and actuate (output).

2.1 How They Sense

We underestimated plants’ sensing abilities and their feelings for a long time. Plants “are just very slow animals” [20]. In addition to the five senses that animals have, scientists believe plants have at least 15 other ways to feel the world. For example, they can perceive and calculate gravity, electromagnetic fields, moisture, and chemical substances [9]. Plant’s sensing mechanisms are talented and exquisite enough, therefore, the question is how to make use of their powerful but implicit sensing capability for interactive functions. Focus on two aspects: the input and output of sensing process.

The former means to make them more sensitive to human behavioural inputs which is difficult for plants to understand without a cyborgian medium. It works in the way of exaggerating voice or motion signals or interpreting them into another type of signal that plants are more sensitive to, like electronic signals (Fig. 4).

The later means to externalize and visualize plants’ internal biologic process. With the aid of cyber-devices, plants are augmented as sensors by visualizing their invisible sensing process. In the case of Cyborg Botany by MIT media lab, the electronics is transferred into plants. Internal wires are connected to sampling instrumentations, turning a plant into an inconspicuous sensor to detect motion and more [18]. These cyborg plant sensors are applied into many interesting scenarios, for example when a motion-sensitive rosebush senses that a cat run out of door, it will send an alert to its owner’s computer.

2.2 How They Think

Recent research on plant intelligence has proved that plants are capable of certain levels of intelligent behaviours including communication, learning and memory.

Plants, trees in particular, communicate with each other relying on underground fungi networks [11]. Cyborgian plants, gifted by the cyber characteristics, become nodes in a more efficient communication network compared to biologic ones. These two forms of networks work cooperatively, sharing a large dataset of information, thus enhancing the adaptiveness of plants. Cyborgian plants are resonating and able to make predictions because they are in a network where they can talk to neighbours and understand what is happening at the other end of network. By simulating the process in the backend they can prepare themselves for the upcoming changes.

There’s another project by MIT media lab - a pair of couple plants that can feel and respond to each other even when they’re far apart. When one plant is gently poked, the other wiggles [18]. Cyber network enables them to “say hello” crossing distance. In the Resonating Forest at Jewel Changi Airport by teamLab, when someone passing by a tree, light color changes and a new tone resonates out. These information is transmitted to nearby plants, spreading continuously as if the plants are discussing the presences and locations of people.

As for learning and memory, it is proved by a team from Western Australia University that plants can build up classical conditioning through training and form memories through experiences [7]. The cyborgian approach imparts plants machine intelligence, featuring incredible capability of data processing and memory storage. By mass-analysing human behaviours as training inputs, cyborg plants learn about users’ habits and preferences so as to better indulge them in nature.

2.3 How They Actuate

Plants are always sensing, thinking and responding to our voice and movements but in an extremely subtle way. Since we understand their sensing and thinking mechanisms, now we are able to guide and supervise their actuation by controlling what they sense and how they think.

By applying external stimulus, like changing light intensity and direction, human is able to guide the growth and movement of plants taking advantages of phototaxis or other biologic tropism. But the growth of plants is too inconspicuous to be noticed in a short time, therefore, their responds are usually transformed into other forms and are visualized through external devices which can be seen as their extended cyber-body. For example, MIT Media Lab created a robotic plant called Elowan. When there’s light nearby, electronic signals within the leaves are detected by embedded wires, and the wheels of the robotic planter are triggered to move autonomously toward the light [19]. In the case of Breeze, an ambient robot inhabits the body of a Japanese maple, allowing her to sense and reach out to nearby people [5]. One difference between the two cases is: the bio-corporeity of Elowan itself acts as a sensor while in the project Breeze, there are embedded sensors around (Fig. 5). Anyway, all roads lead to the same purpose: to make human behaviours sensible for plants while make their responds visible for us.

Fig. 4.

Cyborg plants. Image credit: Michael Lucenkiw.

Fig. 5.

Elowan - augmented plants as sensors, Displays and actuators. Image credit: MIT Media Lab.

In summary, cyborgian approach would be instrumental in the construction of a more interactive environment because:

• 1. Rather than augmenting the way plants sense, it augments the way how we understand and benefit from their gifted sensing abilities for interactive functions.

• 2. On the basis of big data flowing in cyber-network, it augments plant intelligence including communication, learning and memory performance, thus improving capability of cross-species interaction.

• 3. It guides the growth and movement of plants by controlling input stimulus and even guides the process of evolution to a certain extent, towards the objective of the most appropriate level of interactivity.

3 How Cyborgian Approach Encourages Human Participation?

To encourage human-nature interaction, the other way is to improve bodily interaction experience. This chapter is trying to explore how would cyborgian system play a valuable role under the frame of experience hierarchy and assessment matrix (Fig. 6).

3.1 Experience Level

Rational Level.

Communication is the most basic and rational need in the interaction with nature. Embedded intelligence makes plants cyborgs while wearable devices arm human as cyborgs, so we are able to communicate in a common language - binary machine language. Take self-caring planting pot linked with smart phones as an example, embedded technology provides plants the ability to respond to environmental changes in order to better survive, as well as enables human to understand their living conditions and feelings better.

Sensational Level.

“Pleasure” and “the sense of alien” are the two dimensions to describe sensational stimulus [23]. Digital technologies expand the concept of reality. They allow us to explore the reality beyond human limits, to see the familiar world from an alien perspective, which can be quite interesting and motivating. For example, the VR project Mashmallow Laser Feast by B. C. Steel provides alien experience of discovering forests in the eyes of different animals.

Emotional Level.

What eco-interaction aims to achieve is not only sensational pleasure, but more importantly, is emotional bonding and deep reflection. For example, in the project Talking tree by EOS magazine, anthropomorphic plants post their living conditions and feelings online. The purpose is to build up an empathetic bonding which would last longer and deeper than just rational or sensational memories, and would inspire reflects on the relationship with nature. This “reflective level of emotions” which Donald described as the supreme level can be achieved through aesthetic immersion as well (Fig. 7). Steel hopes to bridge the gap between science and art. For teamLab, their concept and purpose behind the aesthetic enjoyment is to cherish the balance between technology and nature and to create a global beauty culture.

Fig. 6.

Diagram of interaction assessment. Author’s own work.

Fig. 7.

Mashmallow laser feast. Image credit: Barnaby Churchill steel

3.2 Experience Assessment

“Interaction design is all about user experience” [1]. Experience of all users (not just human) should be the key assessment indicators for a design project. Cyborgian approach would be advantageous in human-nature bonding process. Its application and efficacy are investigated from four aspects: maneuverability, immersibility, joviality, and attractivity.

Maneuverability.

It focuses on the quality and convenience of the operating process, or in other words, the pleasure of behavioural level [16]. Maneuverability includes fluency and freedom. Fluency concerns whether users operate naturally and smoothly with a clear logic. Freedom refers to the range and types of interactive inputs that users can play with. For example, in the interactive musical plants project Akousmaflore, by applying multi-touch interaction technology, every leaf becomes a tiny instrument. While in the project Forest Entrance by TeamLab, users get involved by walking around, as the project adopts somatosensory devices to capture and respond to users’ position changes. The interaction mode in Forest Entrance is more fluent, but with less freedom of control, as there’s only one type of corresponding input and output.

The development of human-machine interface (HMI) is a process with increasing naturalness, fluency and fitness in line with human cognitive and usage habits. Currently, somatosensory device is one of the most cutting edge HMI interface. Its innovation lies in the realization of contactless operation, which completely liberates people’s hands and fingers and enables participants to freely control machines with their whole bodies. This development brings new opportunities for the interaction between man and nature. Interaction projects that respond to human bodily behaviour is more welcomed because it is the presence of human in outdoor environment that really matters.

Fig. 8.

The history of HMI (Human-Machine Interface). Author’s own work

Immersibility.

Immersion is the engagement level of interactions. Immersibility is related to sensational experience and mental state. Immersive quality can be improved by enhancing multi-sensory experiences, balancing the relationship between “challenge” and “skill” based on feedbacks.

Sensationally.

Although 83% of human’s perception of the outside world comes from visual sense, the importance of other senses should not be neglected. For perceiving the world, the whole is greater than the sum of its parts [15]. High immersibility can be achieved not only by ensuring a wide scope of environment with minimalized visual distractions, but also by multi-sense activation including shape, smell, texture, softness and roughness with the help of perceptual augmentation devices.

Emotionally.

According to Csikszentmihalyi, there are eight mental states in terms of the relationship between skill and challenge level: flow, control, relaxation, boredom, apathy, worry, anxiety and arousal (Fig. 10). These relationships can be constantly evaluated and adjusted with the aid of real-time concentration analysis and massive personal feedbacks, so as to reach the “Flow State” or “Peak Experience” [6], namely a status where fulfilment and enjoyment are coming out of high concentration and totally indulgence [12] (Fig. 9).

Fig. 9.

Two ways of emotion sensing. Health data analysis & facial expression recognition. Image credit: Neurodata Lab, Mocквa

Fig. 10.

Csikszentmihalyi mental states model. Author’s own drawing.

Fig. 11.

Emotional design language Orb. Image source: MIT Media Lab.

Joviality.

The assessment of joviality is a composite index of all positive emotions in an interaction process. Electronic devices nowadays are not only able to monitor physical data such like heart rate and running pace, but are also capable of sensing and recognizing mental states (Fig. 8). Smart treadmills would automatically adjust their slope and speed in accordance with users’ physical conditions. Similarly, an intelligent interaction system can also adjust in real time its difficulty level, surprise level, etc., according to users’ emotional conditions in order to ensure constantly positive experience. In this way, emotion is no longer just the output or by-product of an interaction process. It participates in the process of feedback loop and becomes an input parameter that affects final output. There’re already biometric sensors that can recognize micro emotions after deep learning. In the case of Emotional Design Language Orb, various emotions are visualized by different color and form of the orb (Fig. 11).

Attractivity.

It can be interpreted as “seductiveness” and “replayability.” It evaluates the appeal for first-time users as well as the capability of encouraging re-participation without losing fun. Attractivity derives from beauty and creativity which can be enhanced by cyborgian approach. For instance, in the project Bio-responsive Garden, botanicals were fitted with microelectronics to make them physically animated. Creativity emerges from the unusual combination of “plants” + “dance”. Another example is the Forest Entrance mentioned before. The resonating art work is rendered in real time in response to human behaviours. It is neither a pre-recorded nor imagery on loop [22]. Its attractiveness emerges from the non-repeatable organic aesthetics empowered by machine intelligence.

In summary, cyborgian approach would be instrumental as it allows us to better understand the physical world, improve bodily experience and add more joy and attractiveness to the interaction with nature. The analysis above provides anchor points for machine intelligence to intervene, facilitate and encourage human-nature interaction by making use of its sensing and computational capabilities.

4 Design an Interactive Outdoor Environment

As a matter of fact, many of the projects mentioned above are installed indoors. What hinders outdoor interactions? This chapter aims to figure out current constraints of outdoor eco-interaction projects and the opportunities provided by cyborgian technologies.

4.1 Challenges and Opportunities of Outdoor Interaction

One of the biggest technical obstacles of outdoor interactions is the more complex environment compared with indoor ones, which calls for better performance of input and output equipment. To address this problem, personal wearable devices which equip users as cyborgs would be a key tool. With fixed sensors only, it would be difficult to accurately capture motions and track positions of all users. Smart wearable devices greatly increase the accuracy of somatosensory input. Subtle movements and gestures are likely to be recognized, thus improving operational experience.

Besides, personal wearable devices increase the feasibility of multisensory enhancement in outdoor scenarios. As the name suggested, multisensory enhancement technologies aim to create alluring sensational experiences with the aid of various equipment, such as stereo, blower, perfume machine, tactile gloves, etc. It’s like updating a 2D movie into a 3D (visually enhanced only) and even a 4D movie (multisensory enhanced). People need more attractive and comprehensive sensory experience to indulge themselves in a distractive outdoor environment.

Another challenge as well as principle of outdoor eco-interaction is to minimize the impact on voiceless ecological entities. Cyborgian approach would be helpful because: the living condition of plants can be always monitored, adjusted and guaranteed by their cyber apparatus; and our cyber extensions work as the medium for harmless virtual interactions that bring about genuine feelings. For plants, behavioural response in accordance with human inputs may be a negative interference. That’s why in general, designers would apply non-material digital technologies such as light and sound which have no physical impact on the environment, turning nature into living art without harming it. However, due to the requirement of light and sound conditions, the suitable time for outdoor interaction is limited and unpredictable. The application of AR devices eliminates time limits in a way of mixing interactions happen in the cyberspace with reality in physical space in suitable lighting and volume settings.

Additionally, distributed wearable devices enable mass-customized multi-user interactions, which encourage not only physical but also social activities in natural environment. Although there’re hundreds of users involved simultaneously, interactions can be tailor-made according to individually physical and mental state. Even the viewpoint can be set to be unique, for example, experiencing the world from the perspective of fish or birds, which brings alien experiences that can be attractive for potential users.

One weakness of cyborgian approach is that currently wearable devices are not light or user-friendly enough to be totally ignored while using, which may hinder users’ movements and experience. There’s always a gap between the real action (e.g., press the button) and the action executed in virtual world (e.g., pick up something). Similarly, the rationality of virtual viewpoint should be ensured. The sense of inconsistencies and the learning process keep reminding users the existence of a physical interface. Technologies are being optimized to reduce such inconsistencies of viewpoints and actions to minimize the perceptibility of external devices.

In summary, cyborgian approach, supplemented by multi-sensory enhancement technology, would work as a powerful toolkit to the challenges of outdoor interactions with enhanced immersibility, customized experience and minimal impact on the ecology.

4.2 A Cyborgian Eco-interaction Design Model

After all the researches and discussions above, this paper proposes a cyborgian eco-interaction design model depending on a distributed network of cyborgian intelligence.

On the one hand, non-human users are indispensable parts in an eco-interaction project, which means the ecological inputs that represent their status and interaction experience should not be neglected. Environmental data and plant living conditions that are monitored and evaluated by ubiquitous sensors have decisive impact on the system. Because the health and balance of ecosystem is the premise of human interaction.

On the other hand, for human users, their behavioural inputs including motions, gestures and voice commands etc. are recognised by wearable devices and stationary sensors. Besides, their interactive experience are also essential input parameters in an intelligent interactive system. As discussed before, users’ feelings and experiences are no longer by-products of the interaction, but rather become inputs that trigger self-tuning mechanisms based on feedback loops.

The algorithms at the back end of interface are not fixed, instead, they are supposed to be determined by user experience. Users’ experiences are evaluated from the four aspects mentioned before. The system automatically would customise its difficulty and intensity level according to individuals’ feedback in order to help them reach and maintain “flow state”. If the system notices that relying on bottom up self-tuning process only is not efficient enough to enhance user experiences, then human designers are required to get involved in order to modify the interactive design in a top down way. This procedure is similar to what Kevin Kelly called “control of control” [10]. In this way, the system provides customized and optimized experiences which in turn increase the attractivity (Fig. 12).

Fig. 12.

Diagram of cyborgian eco-interaction model. Author’s own work.

In sum, what distinguishes this model from the others is that it concerns more about the user experience at all levels for all ecological entities. The capability of enhancing user experience by applying a synesthetic approach to reality and the capability of mass-customising and self-evolving based on sensible user experience, are both gifted by a cyborgian methodology.

5 Conclusion and Outlook

The argument throughout this paper is to explore new opportunities and capabilities that are imparted by advancing digital technologies when it comes to establishing an empathetic relationship with the natural environment where we inhabit. The cyborgian theory supports the skeleton of the whole paper while the embodied cognition theory lays a solid foundation.

Cyborgian approach is a hybrid of machinery intelligence and biological intelligence as well as a mixture of top down and bottom up design methodologies. It allows designers to take advantages of both methods: the ability to secure efficiency and to manage complexity.

Embedded intelligence turns plants and other ecological entities into cyborgs while wearable devices transform human into cyborgs as well. Input data from human and non-human users are collected from the interface, processed at the backend and the output can be given back to any interface in this network. In this way human and environment are communicating and interacting in “the third space” [21] which is organized by the network of interface and augmented by digital technologies.

It is revealed by this paper that cyborgian approach would be instrumental in the construction of a more successful human-nature interactive system. This can be justified by the facts that cyborgian intelligence is gifted at: facilitating the whole process of cross-species interaction, namely sensing, processing, actuation and feedback; breaking the limitations of outdoor interaction; improving experiences of all participants which determines the quality of interaction.

The cyborgian eco-interaction design model this paper proposed may be far from satisfactory, but meaningfully, it is expected to work as a minnow that is thrown out to catch a whale, to broach the subject for further concerns and investigations.

Opportunities always come along with threats. Advancing technology is facilitating our life while challenging our relationship with nature. Insightful designers should be keenly aware of the pros and cons of technology, trying to come up with a new paradigm to rehabilitate this relationship, which would then be more than a design paradigm, but even become a new lifestyle in the upcoming digital future.