Keywords

1 Introduction

Digitality has become ubiquitous, permeating every realm of our lives [1]. The boundaries of the human self may extend beyond the physical body, and the consciousness of those extended boundaries has been driven by the development of the outside world [2]. It's important for humans to adapt to the more profound sense of being human-technological symbionts rather than the merely superficial sense of combining fresh and wires [3]. Since how humans interact with the world is greatly influenced and shaped by the tools used by them [3], wearables are considered as the second skin of people to inter-act directly and broadly with the built environment [4].

Pneumatic-actuated soft robotics has been a widely studied field recently with diverse applications [5]. Via pneumatic actuation, soft robotics has become increasingly accessible [6], and widely researched for application in biomimicry [7]. Furthermore, with the metaverse concept announced by Facebook, virtuality has been meant to have constant and seamless integration with existing physical reality [8]. Therefore, this research uses pneumatic actuated structures as smart garments to improve the comfort of digital wearables as the second skin of people. By using pneumatic actuated structures and physical sensors, this study will try to explore the digital wearables as the linker of physical body and XR metaverse, to enhance the XR sensory and physical comfort concurrently.

Contrasting with other digital bionic structures merely working with formal similarities. This study used pneumatic wearables as the representation of the latent living process embodied in creatures, like heart beating, blood flowing and breathing, to reconstruct the interaction between humans and their surrounding environments. For example, the pneumatic system combined the physical dynamics of plants and the flexible structures of human muscles. At the same time, the wearer's data and the signals from the surrounding environment can be transferred via wearables, triggering the interaction in both the physical and virtual system. In this way. A new dynamic of extended reality has been built with the wearables as an integrated interface of sensing and externalizing. Due to the bulkiness of the air pump of a traditional industrial pneumatic system, in this study, the pump and power system has been further downsized, so that it can meet the size restriction of the body-scale pneumatic system. Last but not least, with the synchronization of the pneumatic actuated wearables communicating via XR metaverse, the comfort of the physical body and the richness and vividness of the senses in the virtual world are simultaneously satisfied, providing prototypes verifying the feasibility for future application (Fig. 1).

Fig. 1.
figure 1

The embodied interaction with XR metaverse space on pneumatic actuated structures: metaverse space and pneumatic wearable

Full size image

2 Methodology and Prototype

2.1 XR Interaction Design—Physical Interaction of Digital Wearable

Cognitive philosopher Andy Clark raised the idea in 1998 that human beings are best regarded as an extended system, a coupling of biological organisms and external resources [2]. His ideas coincide with the status of the metaverse today: more than 4.6 billion people can access the virtual worlds of the metaverse via smart phones, laptops, desktops, headsets or consoles [8]. However, communication with the virtual environment is based more on visual content, such as virtual reality platforms providing an engaging and immersive environment [9], or 3D virtual worlds for communication via PC and smart phones [10].

As researchers have concluded, the main feature of metaverse is a twofold link between the virtual and physical worlds: (a) behavior in the physical world influences the experience in the virtual one and, (b) behavior in the virtual world influences the experience in the real one [11]. Touching as a channel for a great variety of information has always been regarded as a crucial site for mediating social perceptions [12] and comprehensive environmental perception [13].

This project explores how the wearables interact with extended reality as an integrated interface of sensing physicality and externalizing XR experience. As a means of telecommunications, the combination of electronic components and sensors for data transferring between the human body and digital system has already been explored in the above-mentioned research. Therefore, instead of adding new applications of telecommunications into the field of human-computer interaction, this project is more about how to introduce the richness and vividness of the XR senses and physical wearable experiences.

In this research, digital wearable devices break through the sensory boundary between physical wearers and its digital avatars. The body movement and heartbeat change of the physical wearer are collected via three-axis acceleration sensor, heartbeat sensor and infrared distance sensor. The movement of the physical wearer is synchronized with its digital avatar, as the touch between digital avatars will cause haptic pressure changes of the physical wearer via its digital pneumatic wearable device in further application.

At the same time, pneumatic wearables visualize and externalize the latent life process of the physical wearer. The digital wearables’ inflation and deflation according to the wearer's heartbeat rhythm, trigger changes in the bionic shape of the wearables and thereby change the tactile sensation of the wearer. Wearable devices draw on the techniques of three-dimensional tailoring and the study of muscle composition. Through the tailoring of pneumatics and clothing, the wearer's muscle dynamics during walking can also be creatively represented via wearables in both physical and XR worlds (Fig. 2).

Fig. 2.
figure 2

Interaction flow

Full size image

In the current research of this project, only movement and heartbeat have been introduced into XR interaction and biotic visualization. In future applications, more scenarios of extended reality can be provided by richer collection of physical data such as environmental data of the wearer.

2.2 Pneumatic Actuated Structures and Soft Material

Under the spread of the digital environment and the popularity of digital devices, the fusion of traditional design and interactive technology has been accelerated in various fields [14], such as the application of 3D print in fashion design pioneered by Iris Van Herpan [15].

However, the above mentioned applications of digital technology devices have downplayed the typical characteristics of traditional fields, such as the basic requirements for comfort in the field of clothing [16]. Therefore, in this research, we have adapted the application of pneumatic systems and programmable materials to meet the basic needs of wearables for comfort and convenience.

2.2.1 Programmable Soft Materials

The heavy and expensive fabrication of the rigid body [5] hampers its efficiency and flexibility in body-device fit [17]. Therefore, silicone as a thoroughly explored material for soft robotics [18] has been used in this project as a programmable soft material. By computational design supported by Rhino and Grasshopper, we made a variety of 3D molds for silicone casting and fabricated the silicone airbags precisely according to their computed structures. Furthermore, through the combination of programable fabrication and pneumatic system, the deformation of functional garments can be assembled precisely to fit different modes of human activities. Last but not least, TPU has been fabricated as an alternative soft material for creating heat sealable sheets and laminated airtight layer [19]. Via patten prototyping designed in CAD and fabricated with laser cutting, the TPU airbag has also become programmable in the generating process from material to structure (Figs. 3 and 4).

Fig. 3.
figure 3

Programable Soft Materials: TPU and silicone and the process of their computed structures supported by Rhino and CAD. The Fig. 3 shows the process of TPU computed structures which was supported by Tongji University College of Architecture and Urban Planning and developed in DigitalFUTURES Shanghai 2018 Workshop

Full size image
Fig. 4.
figure 4

Programable Soft Materials: TPU and silicone and the process of their computed structures supported by Rhino and CAD. The Fig. 3 shows the process of TPU computed structures which was supported by Tongji University College of Architecture and Urban Planning and developed in DigitalFUTURES Shanghai 2018 Workshop

Full size image

2.2.2 Pneumatic System

The actuation system consists of three parts: Arduino toolkits for telecommunications between physical and digital space, sensors embedded in wearables, and pneumatic system for controlling the deformation of wearables. This section will introduce the downsized body-scale pneumatic system.

Air pump is controlled by two sets of solenoid valves which have three states (Inflate, Deflate and Hold), to meet the deformation requirement of the wearables via the inflation and deflation of airbags. For safety consideration, the solenoid valve selects the model SMC's S070C-SDG-32 powered by low voltage DC connecting with battery box.

Due to the bulkiness of the air pump of a traditional industrial pneumatic system, this project refits the air port of the portable air pump and connects it to the corresponding set of solenoid valves. Therefore, the entire pneumatic system only needs one portable air pump, two sets miniature electromagnetic valves and an Arduino toolkit connected with sensors. Through the design of digital wearable, the pneumatic system is organized into a pocket-sized box, which satisfies the portability and aesthetics of wearables simultaneously (Figs. 5 and 6).

Fig. 5.
figure 5

Pneumatic system design. The pictures on the left show how the pneumatic system is designed. The two pictures on the right show the testing of pneumatic system

Full size image
Fig. 6.
figure 6

An example of pocket-sized pneumatic system with one portable air pump, one pair of two-sets miniature electromagnetic valves that can control two sets of airbags, and an Arduino toolkit which can connect sensors easily

Full size image

2.2.3 Morphology and Pneumatic Structures

The combination of biological structure and morphology design has been widely used in various industrial fields, such as the mechanical properties of natural fiber cells [20] and soft elastic tissues of the human body [21].

To satisfy the comfort of the physical body and the vividness of tactile senses simultaneously, the morphology design draws on the dynamics of flexible structures and soft tissues, specifically, muscles as active elastic elements, skin as passive elastic elements and flytrap as pneumatic actuated structures.

The main structure of wearables follows the dynamic of muscle movement, while the holistic structure design takes both the static parts and kinematic joints into consideration. Therefore, the pneumatic structure can be compatible with the daily activities of wearer, even assist the muscles’ movement.

The airbag position mapping is generated by Grasshopper from the heat map of body movement. In the original prototype, the pneumatic structures consisted of airbag units fabricated with TPU and teflon after laser cutting. In further iterations, the pneumatic structures are added inflatable muscular structures made of silicone casting. In the composed pneumatic structures, the active parts fitted body movement are composed of muscular silicone structures, while the static parts are added on more biological features, to improve the comfort and richness of wearable devices (Figs. 7 and 8).

Fig. 7.
figure 7

Various prototypes of pneumatic air-bags’ structures inspired by muscles and flytrap

Full size image
Fig. 8.
figure 8

Various prototypes of pneumatic air-bags’ structures inspired by muscles and flytrap

Full size image

2.3 Interaction with XR Metaverse Space

The real-time synchronization between the physical wearer's behavior and one's virtual avatar is realized by Kinect motion capture via processing, physical sensors and their connections with metaverse space. As introduced in the physical interaction chapter, the wearer’s activity data triggers deformation of the wearable device in physical space, and the mirrorly changes of the digital avatar's position and appearance happen concurrently via the linkage of physical data and metaverse space. Furthermore, beyond mere synchronization of the physical wearer's activities, a richer interactive experience has been introduced into metaverse space via HoudiniFX space rendering. Walking and turning of the physical wearer will trigger smoke and light effects in metaverse space. Therefore, the wearer of Digital Wearable can experience extended-sensory interaction between XR metaverse and physical environment with integrated perception of tactile changes and visual effects (Figs. 9, 10, 11 and 12).

Fig. 9.
figure 9

XR design: the connections between the wearer, the digital wearable and the digital avatar

Full size image
Fig. 10.
figure 10

Real-time synchronization between the physical and digital ones by Kinetic motion capture via Processing

Full size image
Fig. 11.
figure 11

Various experiences of metaverse space made by HoudiniFX

Full size image
Fig. 12.
figure 12

An example of The Embodied Interaction with XR Metaverse Space based on Pneumatic Actuated Structures

Full size image

3 Conclusion and Discussion

The vision of this research is to build an integrated-sensory XR interactive system breaking through the sensory boundary between physical environment and metaverse. The current prototype has achieved the three goals mentioned above:

  1. 1.

    The programmable fabrication of pneumatic structures satisfies the comfort of the physical body and the richness of the wearing experiences simultaneously.

  2. 2.

    Morphological generation based on muscle structure research and human body heat map as representation of the latent life process and assistance with daily activities.

  3. 3.

    Digital wearables as the sensory bridge of digital avatars in metaverse space and its physical wearer in physical space.

Future development will focus on improving the delicacy of sensory transmission and the richness of interactive activities between physicality and extended reality. Specifically:

  1. 1.

    Now an interactive system consisting of wearables and metaverse has already been established, with only heartbeat and body movement as physical input from the wearer. In further research, the wearable can embed richer collections of physical data such as environmental data of the wearer to raise the vividness of experience.

  2. 2.

    The current research has already provided a prototype of deformable Digital Wearables for Body-Scale, and the wearable design mainly refers to movement pattern of shoulder and arm. With growing accuracy and delicacy of morphological design, the wearability of devices will break through the boundaries of experiential device and become daily wear in the metaverse era.

  3. 3.

    In this research, wearables have been verified as an effective medium connecting people and metaverse. Through the introduction of real-time space editing and web communication, metaverse social experiences (such as shaking hands and touching) can be comprehensive physicalized and extended to the wearer's embodied perception.