Keywords

1 Background

Today’s architecture, construction, and engineering (ACE) industry have tremendously benefited from the development of computation and automation, urging the exploration of non-standard forms and material systems. Facing such complex design requirements, a fully autonomous CAD-CAM workflow remains a challenging mission insofar as it still demands human cognition and intervention in response to ill-defined construction tasks or material inconsistency. Hence, compared to highly automated design productions, a ‘human-in-the-loop’ system setup may increase workflow flexibility and efficiency [1]. Architectural discourses have entered a post-digital realm where the non-digital aspects started to regain importance and it aims at addressing the humanization of digital technologies [2]. Thus, the goal of this research is to investigate a ‘human-in-the-loop’ workflow that benefits from both the computation and machine capacity and the rich repertory of experience and knowledge from humans. To investigate this question, the research first introduces the idea of HCPS and based on which, it develops a system setup hinging on interactive human-machine collaborations. The research uses an EPS-made Taihu rock, as a demonstrator, to elaborate on the technical aspects. Here, the post-digital design and fabrication processes highlight bi-directional communications among humans, the computational model, and the industrial robot, which are made possible via parametric design and MR interface.

1.1 Idea of HCPS

As humans possess tacit knowledge vital to the cultural, political, and economic dimensions of their daily lives, the combination of which with machines’ data processing capacity and high-level precision may alter the landscape of today’s manufacturing industry [3]. Sowe et al. and Zhou et al. defined an adaptive human-centered project delivery system as HCPS, which values human control and communication peculiarities in the system so that real-time, efficient, and reliable data intelligence can be feasible [4, 5]. The idea of HCPS highlights ‘human-in-the-loop’. It aims at achieving collective efforts by absorbing human’s capacity of processing arbitrary information into digital systems, and by facilitating collaborative context-awareness based on data acquisition and status monitoring techniques [6]. The integrated system dedicates to augmenting humans for the sake of a dynamic interaction with machines, and the enrichment of human sensing and cognitive capabilities [7]. In this case, the idea allows the system to overlap the CAD realm with the CAM realm avoiding a predefined machine-centered ‘workmanship of certainty’ [8], instead, it enables humans to creatively intervene in the fabrication process according to their perception of aesthetics and ad-hoc decision-making.

1.2 Shape Grammar-Based Design and Making

Shape grammars have been identified with computational design and offer a natural basis for a computational theory of making. They use rule-based visual systems to describe and generate designs [9]. In today’s digital practices, shape grammars can be parametrized using parametric design tools. Knight and Stiny referred to the idea of shape grammar-based making as ‘doing and sensing with stuff to make things’ [10]. Here, ‘doing’ and ‘sensing’ are interactively carried out by humans and machines, and based on what they extract from the elements, designers can define algebras for the ‘stuff’ of shapes and use these rules to make ‘things’. For example, according to the analysis of the geometric rationale (sensing) of Gaudi’s Sagrada Familia, today’s digital designers have created parametric rules based on hyperbolic paraboloids and solid Boolean subtractions (stuff), then used these scripts to recreate the sculpture-like Passion Façade rose windows (things) [11]. Thus, shape grammar-based design and making can inspire human-machine collaborations as it ensures both rationality and openness at the same time. Such an idea can be found in architectural experiments such as ‘Interactive robotic plastering’ [8], ‘RobotSculptor’ [12], and ‘RoMA’ [13]. From different scales, these projects have demonstrated the technical feasibility of interactive human-robot collaborations and the possibility of real-time bi-directional communication between humans, cyber, and physical systems.

1.3 Human-Guided Robotic Fabrication

Human-guided robotic fabrication allows humans to interact with robots in real-time and to manipulate physical form during the fabrication process. Mitterberger et al. suggested such a semi-autonomous system setup features either manual handcraft with robotic precision or dynamic fabrication based on human-robot collaboration [12]. On one hand, handcrafting with robotic precision can establish a safer collaboration environment in which it requires devices to provide live feedback on design information and to help correct fabricators’ operations. On the other hand, interactive fabrication allows the insertion of human creativity and aesthetics during the ongoing materialization process. Recent developments in machine vision, kinetic control, and MR have opened up new opportunities for ‘human-in-the-loop’ production, as human-guided robotic fabrication may offer a solution for ill-defined design requirements, complex material systems, and non-linear realization processes.

2 Method and Materials

This research presents a ‘human-in-the-loop’ workflow that tackles the idea of HCPS and roots in the advances of shape grammar design and human-guided robotic fabrication. With parametric design and augmented reality technology, the system setup creates an open-ended design space allowing for humans’ intuitive inputs and in-situ adjustments to the fabrication process. The research findings are highlighted in the realization of an EPS-made Taihu rock.

2.1 System Setup

The system setup, as shown in Fig. 1, consists of two major components: (1) a parametric associative model and (2) an MR-aided robotic fabrication platform. To implement shape grammar-based design, allow in-situ design and fabrication modification, and be used as Robot Operating System (ROS), the central associative model is created using Rhinoceros Grasshopper. The research started with extracting geometric information from natural Taihu rocks using a FARO Focus laser scanner, then the data was interpreted and transformed into the design generation tool via Grasshopper-based plug-in Wasp. The system adopts the MR application Fologram and HMD Microsoft HoloLens to establish bi-directional communications among the human fabricator, the design, and the fabrication platform. The robotic fabrication system includes an ABB IRB120 robot with customized foam-cutting end effectors and the live robotic control is enabled by Grasshopper-based plug-in Taco ABB. Also, to keep track of the ongoing fabrication, the investigators set up a stand-alone Microsoft Kinect for monitoring the as-built condition of the Taihu rock. Figure 1 elaborates on the system setup.

Fig. 1.
figure 1

System setup where A is the cyber system connecting three agents; B is the MR sys-tem; C is the robotic fabrication platform; D is the EPS unit; and E is the stand-alone moni-toring system.

Full size image

2.2 Form Generation

Taihu rocks are common elements in private Chinese gardens. Their aesthetic characteristics can be generalized using ‘slender, wrinkle, clear, leaky’, as ‘slender’ represent the sculpture’s odd shape with clear edges; ‘wrinkle’ represents concave and convex surfaces; ‘clear’ indicates the delicate carving from nature; and ‘leaky’ refers to perforations. Thus, the parametric interpretation of Taihu rocks will focus on geometric generation.

2.2.1 Data Collection and Interpretation

The shape grammar-based design contains 3 steps (Fig. 2). Firstly, a FARO Focus laser scanner was used to acquire the geometric data of natural Taihu rock. It took 6 scans separated with a 60-degree angle between each other to rebuild a holistic virtual model. The retrieved point clouds have been converted to a continuous Mesh surface in Rhinoceros, which was then analyzed for its geometrical characteristics. In Fig. 3, the gradient colors indicate the undulation depth of the rocky surface, and the extracted information was categorized into structure factor and texture factor in order to interpret the Taihu rock’s natural form.

Fig. 2.
figure 2

Design and materialization setup of a shape grammar formed Taihu rock.

Full size image
Fig. 3.
figure 3

3D scan of a natural Taihu rock and the analysis of its geometric characteristics.

Full size image

2.2.2 Shape Grammar-Based Design

The investigators applied Boolean operation to recreate a target form based on the gained knowledge, they used cylinders, spheres, and ellipsoids to carve out the perforations, angularities, and surface undulations from cube aggregations. Wasp, a combinatorial tool in Grasshopper for designing discrete elements, was introduced for shape grammar-based parametric design. The plug-in can process a series of aggregations from the input elements based on the pre-defined topological graph of connections. There were three steps in shape grammar-based design generation. The investigators first randomly placed connecting points on the surfaces of a single cube, defining a basic module for subtraction (‘algebra’ in Fig. 2). Then, the cube was further refined with Boolean operations using the aforementioned primitive geometries. Based on the cube aggregation and their connection points, the algorithm generated several cylinders and ellipsoids for further Boolean operations. Also, a boundary constraint was added to control the overall dimensions. A final design model was achieved through iterative Boolean operations with modules and primitive geometries.

In this case, the computationally generated form was for visual reference. To make grammars for design materialization, the subtraction logic was then transformed into robotic motion paths and corresponding carving tools. Based on Knight and Stiny’s idea, the basic elements in shape grammar, “stuff”, were replaced with customized tools. In terms of ‘doing’, the investigators rationalized the results from Boolean operations. To cope with the robotic end effectors including the how-wire cutter, L-shape, and U-shape carving tools, the perforations were transformed into hyperbolic paraboloids, and the undulation details were realized with 3D trajectories.

2.3 MR-Aided Robotic Fabrication

A parametric associative model was created using Rhinoceros Grasshopper to bridge the virtual and the actual environment. With parameters that define the logic and shape of the Taihu rocks, the investigators were able to convert the design into instructions or trajectories for the later human-machine collaborative implementation based on material properties and site conditions. With the help of Fologram and Taco ABB, a tailor-made ROS based on Grasshopper, the human operators have been able to establish an MR-aided robotic fabrication process that relied on human intuitions that allows for direct human intervention during the foam-cutting process.

2.3.1 Fabrication Setup

In this case, a marker-tracking method was used to overlay the digital model with the physical environment. The investigators placed a 20 * 20 cm ArUco marker on the robot’s fabrication platform and by fine-tuning the model’s position in Fologram GUI, there were able to increase alignment accuracy.

The human-guided robotic fabrication contained rough cutting and fine carving. In the first stage, the fabricators removed large portions from the EPS cubes to shape a basic form of the Taihu rock. And in the fine carving stage, the investigators switched to U-shape and L-shape tools to add details of surface undulations and angularities. MR technology is the medium connecting humans, the cyber, and the physical system. It omits a great number of command inputs via keyboard and mouse, and the technology creates an intimate bond between humans (fabricators) and machines (robots). Such an HCPS setup was built on immersive yet natural human-machine interactions, and thanks to the off-the-peg applications available in Grasshopper, the investigators could realize a human-guided robotic fabrication workflow with a single manufacturing information model.

2.3.2 Human-Guided Robotic Fabrication

With the Grasshopper-based MR platform Fologram, the investigators have customized the holographic projection, including the cutting shape (hyperbolic paraboloid), control points of the central curve of the hyperbolic paraboloid (green spheres), and the robot’s moving trajectory, in the Hololens operators’ line of sight (Fig. 4). With Hololens’ gesture tracking capacity, the investigators can interact directly with the digital model in a freehand manner. By switching on and off model layers, they were able to selectively choose the information needed for the fabrication tasks at hand and to avoid excessive information displayed on the screen.

Fig. 4.
figure 4

Human-guided robotic fabrication with customized tools. Rough cut with hyperbolic paraboloid (left) and fine carving with trajectories (right).

Full size image

During the fabrication process, the investigators first properly position the EPS in the workspace according to holographic instructions. Then, based on the digital model generated from the shape grammar approach, they ‘drag’ the control points of the hyperbolic paraboloid per in-situ conditions. By combining Fologram and TacoABB, such manual modifications of the surface could be directly transformed into the updates of ‘RAPID code’ driving the IRB120 robot. Also, with Hololens, the investigators could have a visual preview of the robot’s moving path, this helped to avoid potential human injuries caused by human-machine collision (Fig. 5). After the first stage of rough cutting with hot-wire cutter and hyperbolic paraboloids, the investigators changed the robotic end effectors into L-shaped and U-shaped tools for the next stage fine carving.

Fig. 5.
figure 5

MR-aided robotic fabrication allows for a safe human-machine collaboration envi-ronment and flexible system setup.

Full size image

The fine carving operations led to the final features of the EPS-made Taihu rock. The initial robotic carving trajectories were inherited from the shape grammar-based design. Compared to the traditional robotic fabrication processes where the physical objects normally remain stationary, the system setup in this experiment allowed the investigators to rotate the foam at will meanwhile accordingly altering the carving trajectories and the approaching angles. The investigators eventually carried out the fine carving operations in a sequence that required minimal machine calibrations.

The human-guided robotic fabrication also permitted the absorption of human creation and design input to the ongoing materialization process. By connecting the image-based scanner Kinect with the parametric model through Grasshopper plug-in Project Owl, the investigators were able to track the geometric complexity of the Taihu rock in real time. Figure 6 illustrates the in-progress scan result of the EPS sculpture from different angles, where the color gradients indicate texture depths. Based on the available carving tools, in this case, it was the L-shape and U-shape stainless metal tubes, the investigators could improvise the actual carving outcome by manipulating robotic paths. For example, in order to create crumple moments in certain surface areas, they have overlaid several motion paths and tilted some of them to mimic natural appearances.

Fig. 6.
figure 6

In-progress scan using Kinect to evaluate geometric complexity.

Full size image

3 Result and Conclusion

In this case, the investigators used a small-scale EPS-made foam sculpture to demonstrate a workflow involving computational form generation and human-guided robotic fabrication. To accomplish this, the investigators developed a parametric associative model providing live holographic projections to human fabricators, tracking while translating human gestures into model inputs, and being used as the robotic operating system. By manipulating hyperbolic paraboloids and Boolean operations, the investigators were able to create complex geometric moments responding to the ‘slender, wrinkle, clear, leaky’ feature of the natural Taihu rocks. At the same time, the steady robotic motions resulted in smooth surfaces and continuous edges revealing a contrast to natural rocks, insofar as the aesthetics of the built outcome (Fig. 7) is the combination of digital cleanness and natural randomness.

Fig. 7.
figure 7

The built outcome of EPS-made Taihu Rock.

Full size image

The ‘human-in-the-loop’ workflow demonstrated in this project can potentially benefit large-scale industrial designs and building practices. In this case, the result was limited by the robotic operating range, the type of end effector, and the material used. Future applications, however, may combine customized fabrication or construction robotics, tasks-oriented tools, and specific materials in order to have real-world meanings.

To conclude, the design-to-build experiment discussed in this paper aims to illustrate a ‘human-in-the-loop’ workflow that effectively combines human tacit knowledge, computation power, and robotic capacity. This design strategy enables humans to intervene creatively in the fabrication process, offering novel design-to-fabrication workflows within human-machine interaction systems. The system setup was designed to inherit the proportion and geometric peculiarities of natural Taihu rocks, a more intuitive and dynamic MR-assisted workflow could assist or replace the more conventional approaches, as well as to augment human craftsmanship and aesthetic sensitivity during an open-ended yet interactive fabrication process. With the emergence of HCPS in architecture, designers no longer adopt digital means for digital sake, instead, they are trending to seek the interplay between high-tech and high-touch experiences, between virtual and actual reality, between digital and analog material systems, wherein the objects they created can address humanization [2]. The experiment on the Taihu rocks can reflect an explorative human-machine collaboration that may be beneficial to real-world building assignments. By taking advantage of human’s peculiar competence in handling ambiguous information and making judgment calls accordingly in most architectural intelligent manufacturing, the idea of ‘human-in-the-loop’ in architectural practice can offer the industry an effective approach for implementing digital twins and setting up a design solution space with appropriate tolerance.