Keywords

1 Introduction

In recent years, the maker movement has arisen a trend of handmade implementation around the world. As makers work by their own hands, practical problems or design errors will not appear until they really trying to combine pieces together in the real world. In order to lower the manufacturing threshold and obtain more rapid prototyping method, makers often manufacture in digital ways. But It requires a professional division of labor in the construction process. Therefore, the prototyping process needs a lot of time to consultation, waiting, and manufacturing. Design-Fabrication-Assembly (DFA)- an integrated prototyping process proposed by [5] which enables users to take advantage of fabrication as part of a collaborative process. But there’s a need for the communication media in DFA frame [5]. Therefore, we via combine mixed reality and the fabrication machine–a device that uses a virtual interface to control a physical mechanism for fabrication and assembly. We use a “seeing-moving-seeing” design thinking model, which allows designers to refine their ideas [10]. Designers can analyze and present these ideas with the use of design media making it easier to explain their thinking patterns A co-existing space representation [7, 8] is used for modeling such interplay transitions from virtual to physical spaces [13]. The possibilities of multiple operation by designing, manufacturing and integrating tools through digital means, such as robotic-arm were expanded [2, 4].

Three aspects of the interactive fabrication process based on “seeing-moving-seeing” model are human behavior, digital stream, and physical entity. Additional three inter-plays comprise of control, communication and computation and are the interface between human behavior, digital sensing, physical entities. By Interacting with the digital stream and the physical entity via combine digital twin (DT) with mixed reality technology, a designer can embed himself/herself into a network of a manufacturing system [11]. We implement a Coexisting Fabrication System (CoFabs) to support an interactive process for designers that allows for real-time modification and manufacturing in the design sequence. The process integrates both real-world and virtual environments based on the “Digital Twin” (DT) concept, which proposes a system where physical entities and virtual information are referenced to each other in a recursive way through a series of physical changes, information analysis, and generative fabrication suggestions. In this way, the workflow is optimized [3]. We also analyze each thread that results and create a system model to simplify the workflow of controlling physical machines and lower its technical threshold.

2 Related Work

Digital manufacturing is changing the way people design, produce, and interact with objects and devices. The diversity of current manufacturing processes includes laser cutting, 3D printing, CNC milling and printed circuit board (PCB) manufacturing and they may produce parts in a variety of forms and materials. Because of the rise of digital fabrication technology, it redefines and integrates industrial manufacturing logic. Designers must be capable of abstract design thinking, describing and keeping design results under control. With the advent of a series of rapid prototyping (RP) technologies, designers are able to see design results more quickly.

2.1 Fabrication Process of Maker

The prototyping process of interactive surfaces is divided into three stages: Design, Fabrication and Assembly, DFA framework (as shown in Fig. 1). Two main issues are: (1) Lack of unified communication media across D-F-A stages. While design is changed, the fabrication will adapt according to the geometry and methods. The assembly methods and sequence should also adapt. Most digital tools are for synchronizing the design and fabrication, not assembly methods. (2) As mentioned above, participants during assembly stage are often unskilled and need to be trained in order to adapt to the changes from fabrication stage. Such condition is often not possible. DFA is a collective prototyping process that will provide a framework to allow many participants to join the process, experiment with the concept and respond to the adaptation of interactive surfaces. Three stages of DFA are: (1) design stage: design installation prototype; (2) fabrication stage: convert design into components and carriers, and find the suitable fabrication methods; (3) assembly stage: assemble fabricated components and carriers for final installation [6]. There are three retrospective-modification process and two communication tools required for completion of DFA process [5].

Fig. 1.
figure 1

DFA process

Full size image

2.2 Towards Co-existing Environment

Mixed reality is the integration of both real and virtual worlds to create new environments and visualizations of physical and digital objects coexisting and interacting in real time. Mixed reality occurs not only in the physical or virtual world, but also in the combination of reality and virtual reality, including augmented reality and enhanced virtual immersive technology. Currently, the range of mixed reality technology and applications has expanded to include entertainment and interactive arts as well as engineering and medical applications.

In 2014, Weichel et al. established a mixed reality environment that lowered the barrier for users to engage in personal fabrication (Fig. 2). Users design objects in an immersive augmented reality environment, interact with virtual objects in a direct gestural manner and can introduce existing physical objects effortlessly into their designs [12].

Fig. 2.
figure 2

MixFab’s user interface

Full size image

2.3 Automation Digital Fabrication Tools

In the course of explicit bricks in 2010, the robot arm was first time combine with hot-wire cutting to fabricate. The architectural potential of the developed system was tested through applying it on the design and fabrication in a 1:1 scale prototypical building structure. The participants were challenged to analyze and test different interlocking systems by making simple prototypes in order to apply the outcome to an overall system. The prototypes were fabricated using a robot in combination with a hot wire cutting machine. Existing constraints set by the fabrication process hat do be considered as well as to check stability and flow of forces (Fig. 3). Gaining stability due to friction and interlocking system were seen as the potential of the fabrication process and had to be exploited as far as possible [1].

Fig. 3.
figure 3

Robot arm hot-wire cutting process (left) and result (right).

Full size image

2.4 Summary

Through the analysis of above literature, the concept of the DFA process was introduced into the digital fabrication process. Under the operating conditions of rapid prototyping and virtual reality blending, it allows designers to join the process, experiment with the concept and respond to the adaptation of interactive surfaces. Generally, the devices in a mixed reality are limited to their own resolution and interactivity in the display of the interface. The HoloLens [9] developed by Microsoft not only provides a wireless head-mounted display, but also gave the ability to assemble AR commands and the opportunity for users to face the appearance of such applications. Therefore, HoloLens was used in this research as a mixed reality interface device for the system.

3 Methodology

Participatory behavioral observations were made and explored in this paper. Hence, makers with cross domains were the main targets and various manufacturing methods capable of rapid prototyping were analyzed. It included metal bending, metal printing, hot wire cutting, incremental sheet forming, CNC engraving and panel striking. In order to allow CoFabs extend rapid prototyping to the other fabrication process. We divide the fabrication process into 4 steps: (1) Interface setting, (2) Component setting, (3) fabrication and (4) finish, for verifying DFA process (seen in Fig. 4). But maker needs technical knowledge in every stage, such as tools assembly and parametric adjustment. Therefore, we analyzed the different tasks for each stage and systematic via computation design in this paper, and control the physical machine by combining the co-existing technology and digital twin to reduce the fabrication technical threshold. We analyzed the work area setting, initial model and tool setting in the interface setting stage. The stage of component setting is combined co-existing technology and fabrication system to control machines and perform the task. Moving path and angle were analyzed by the user’s drag gesture in the fabrication stage. In the final finish stage, we used color to display the assemble position and check for any part that didn’t match.

Fig. 4.
figure 4

CoFabs process

Full size image

4 The Experiment

In this paper, we use HoloLens to develop a co-existing fabrication system (CoFabs). Currently, we focus on research between HoloLens GUI and interactive behavior. The virtual graphic computation system can make users see the combination of digital and physical model environments more efficiently. But it needs detection material to achieve an acceptable simulation. We have to optimize the digital model when we are deforming virtual objects in this co-existing fabrication process.

We used Fab Car to perform a small test on a unit component, utilizing robot arm hot-wire cutting fabrication. Fab Car is a collaborative project aimed towards the implementation of a modular car. It gives designers the ability to modify, customize and adapt the vehicle to their specific needs at any given time and allows designers to iterate on the project in real-time.

First, we set up three different Fab Car templates model at the left of the interface (seen in Fig. 5-a), to allow for fabrication customization. System will compute the tire position and revise template when the user selects one of the templates. and the user can also revise the length and width to decrease space at the right of interface (seen in Fig. 5-b), to adjust Fab Car architecture (Fig. 5-c).

Fig. 5.
figure 5

(a) Fab Car template, (b) template parameter and (c) Fab Car architecture.

Full size image

We provide the user with an internal structure component to select and assemble individual components, such as tire, engine, transmission shaft etc. We divide work area into 4 workplaces: (1) Frame setting, (2) structure setting, (3) Car shell design, (4) Assembly component, to make it easier for the user to select and place component and allow step-by-step fabrication process. we connect component of every workplace to allow for iterations.

First, we reset the three different architectures and tire templates. The template model will import to second workplace by selection, and system will compute the space to place by selected size of the tire (seen in Fig. 6-a). We set up the component of engine, seat, transmission shaft. User can click and drag to pick up the component, and customized Fab Car internal structure by placing the components in the desired position. (seen in Fig. 6-b)

Fig. 6.
figure 6

(a) first workspace: select template, (b) second workspace: pick and place component

Full size image

In the third workspace, we extruded the model of the base upward into a cube to provide user with the initial model of cutting Styrofoam, and computation with Boolean delete on the reserved space in the previous step to limit the cutting scope. Then, through track two hand positions and generate the line from the middle as cutting tool in virtual. Users can generate the cutting path through click and moving two hands to reduce path unsmooth due to muscle fatigue when hand-floating. Each click once to generate a line as forming initial cutting surface and path (Fig. 7-a). In order to allow the user can modify the model and precisely cutting, we set up a few buttons to provide user modify, edit and divide modifiable, modifying and confirm status. In the process, we through the initial interface for distinguishing the use of gestures as follow:

  • UndoCrv: Provide user to undo and regenerate curve.

  • Confirm: Transform multi-curve to the surface after generating curves and proceed to the next curve and surface.

  • UndoSrf: Provide user to undo and rebuild the surface after generating surfaces.

  • Edit: Click to generate the edit point on the surface, and modify the surface via click and drag the point.

  • Clear: Clear all models on screen for initial setting.

Fig. 7.
figure 7

(a): Click gesture to generate cutting surface, (b): Robot arm according moving path to hot-wire cutting.

Full size image

We use the editor to transfer the script and parameter of Fab Car to the MR device when the user finished set cutting surface and starts to perform the hot-wire cut. Through preview every piece of Styrofoam that needs to be cut, this allows the user design car shell component step-by-step. Finally, user can select the component which to be cut, and generate cutting path according to the set cutting surface via the editor to generate RAPID code to control the robot arm perform hot-wire cut (Fig. 7-b).

In the fourth stage, we through MR device simulate the component, assemble sequence and position of the final project in physical environment and via click and drag gesture to preview and assemble the component. It provided the user clear assembly process and position of each component. In addition, we used color to distinguish car shell and internal structure collision position. It allows the user to check and make sure all the component is the match, and modify the component via the display of collision position (Fig. 8-a). Finally, we assembled a Fab Car step-by-step after modifying the components (Fig. 8-b). We observed the user don’t need to wait for the final project to make sure it’s available when they operated easy interface to get stronger custom tools.

Fig. 8.
figure 8

(a): Used red to distinguish collision position, (b): Fab Car finish project

Full size image

5 Conclusions

This paper via “seeing- moving- seeing” design model with robot arm combine to MR technology for digital fabrication. Under the operating conditions of rapid prototyping and virtual reality blending, and using the robot arm which is a three-axis fabrication tool, and integrating mixed reality for remote control to achieve the DFA prototyping process, users can utilize different “seeing methods” to observe the finished product. To reduce complications of robot arm operation, allowing the designer to operate the robotic arm more simply. Thus, the CoFabs process immerses users into an interactive co-existence environment more suitable for user immersion.

Not only did the user witness the model building progress, but also determined the manufacturing detail of fabrication with prompted feedback information. Designers can also modify several specific parameters to change the shape from simulated model to real materials. Through this human-machine collaboration task, we can make design decisions and fabricate designs more simply, intuitively, as well as easily communicating design intent in real-time. In the result of experiment for the development of Cofabs prototype, we learned:

Distributed Fabrication

In the situation which don’t need oral instructions and diagram, user can share fabrication process and working on different projects together. In this process, we’ll face the interface and interactivity problem. It can via the user experience research method to solve the problem. Then, in order to allow the user easier to operate, design and fabricate, we will automatic the fabrication process via digital computation in future work. It makes user control the design development process.

Co-existing Fabrication Accuracy

In the co-existing fabrication, imprecise virtual and physical information delays interactive feedback. We use HoloLens to solve this problem but it still needs fault tolerance.

Information Synchronizes

Virtual and physical information cannot be complete synchronize through the transmission of network signals. In the future, it can via 5G network to increase the wireless transmission speed to solve this problem.