Tangible 3D modeling of coherent and themed structures

Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

1 2 3 4 5 6 7 8 9 10 11 Q2 12 13 14 Q1 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Q3 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

Contents lists available at ScienceDirect

Computers & Graphics journal homepage: www.elsevier.com/locate/cag

Special Issue: SMI 2016

Tangible 3D modeling of coherent and themed structures Jeppe U. Walther n, Andreas Bærentzen, Henrik Aanæs DTU Compute, Richard Petersens Plads, bygning 324, 2800 Kgs. Lyngby, Denmark

art ic l e i nf o

a b s t r a c t

Article history: Received 14 March 2016 Received in revised form 7 May 2016 Accepted 9 May 2016

We present CubeBuilder, a system for interactive, tangible 3D shape modeling. CubeBuilder allows the user to create a digital 3D model by placing physical, non-interlocking cubic blocks. These blocks may be placed in a completely arbitrary fashion and combined with other objects. In effect, this turns the task of 3D modeling into a playful activity that hardly requires any learning on the part of the user. The blocks are registered using a depth camera and entered into the cube graph where each block is a node and adjacent blocks are connected by edges. From the cube graph, we transform the initial cubes into coherent structures by generating smooth connection geometry for some edges of the graph. Based on an analysis of the cube graph, we identify subgraphs that match given graph templates. These subgraph templates map to predefined geometric refinements of the basic shape. This, in turn, allows the user to tangibly build structures of greater details than the blocks provide in and of themselves. We show a number of shapes that have been modeled by users and are indicative of the expressive power of the system. Furthermore, we demonstrate the scalability of the tangible interface which appears to be limited only by the number of blocks available. & 2016 Elsevier Ltd. All rights reserved.

Keywords: Interactive physical model building TUI (Tangible User Interfaces) 3D shape modeling Constructive assembly Augmented reality Depth cameras Object tracking Building blocks

1. Introduction The idea of building edifices or structures with blocks, slabs or bricks is clearly ancient. Building with (small) blocks as a playful activity is more recent, but certainly much older than the digital computer. For instance, the Froebel blocks which inspired Frank Lloyd Wright [1] and had a big influence on kindergarten education, predate computer aided design by more than a hundred years [2]. Thus, we surmise that most people would find blocks familiar and would be able to start building with little instruction. In this work, we present CubeBuilder. This system lets the user build with physical blocks, simultaneously acquiring a digital 3D model of the construction which is transformed into a contiguous, themed structure as the user builds. Our interface is based on a Microsoft Kinect 2 [3,4] depth camera which allows us to detect where a block is placed. The CubeBuilder interface is highly scalable, since the addition of blocks does not increase the computational cost of detecting future blocks. Moreover, the Kinect may be moved during construction which greatly leverages the ability to place many blocks. Finally, we allow the building blocks to be placed freely and do not rely on interlocking building blocks. n

Corresponding author. E-mail addresses: [email protected] (J.U. Walther), [email protected] (A. Bærentzen), [email protected] (H. Aanæs).

Looking beyond the user interaction, we surmise that in many cases, building blocks will be a part of a playful activity where the blocks are placeholders that represent more complicated shapes. Our system allows users to actually create such shapes in two ways. Firstly, the user is not required to place blocks in a completely regular fashion, but we create a graph of block connections: each face of each block may be connected to another adjacent block. For each such connection, we create a smooth connection geometry surface as shown in Fig. 6. Secondly, we encode block configurations as templates for far more detailed structures based on a theme. When the user has placed an ensemble of blocks it is immediately interpreted according to this theme. Assuming a castle theme, the blocks will be embellished with arches, battlements, turrets etc. according to the structure of the brick ensemble. To summarize, our contributions are 1. A tangible user interface to block modeling which has no inherent limitation on the number of blocks that a user can place. 2. The generation of connection geometry which allows for smooth, nonaxis-aligned structures to be created using our system. 3. A template based interpretation of the block structure, transforming a simple collection of blocks into a far more detailed shape.

http://dx.doi.org/10.1016/j.cag.2016.05.004 0097-8493/& 2016 Elsevier Ltd. All rights reserved.

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

We envision that our system could be used in an augmented reality setting with creative and educational games in mind. The 3D models that users create could then coexist with interactive and animated characters.

2. Related work The idea of creating Tangible User Interfaces (TUIs) with construction kits has previously been explored within the constructive assembly domain [5]. Early solutions were based on mechanical building blocks enriched with electronics [6–8] while more recent systems use depth cameras to register building blocks [9,10]. CubeBuilder is distinguished from many of these systems by allowing for motion of the depth camera and by leveraging the Kinect Fusion system [11,12] to maintain a separate volumetric model of the scene. This model makes it possible to completely decouple the representation of the designed block model from the computer vision system and facilitates a very scalable approach to block recognition. Since we can also move the sensor, the number of blocks can be very large. To the best of our knowledge, earlier systems do not allow for arbitrary (non-aligned) placement of blocks. The same is true about the application of a theme based interpretation of the structures with the exception of Anderson et al. [7] who use logical predicates to recognize blocks belonging to, say, a wall or a roof. This is used for embellishment. We posit that our subgraph matching combined with restrictions that are very similar to the predicates in [7] allows for better matching of structures that emerge when several blocks form a pattern. If we consider systems for 3D modeling that are based on purely digital blocks, the game Minecraft [13] is without doubt the best known example. However, Minecraft strictly enforces a regular grid on the structures that one may build which is very different from our system where blocks may be placed freely. The same is true of the system LEGO Digital Designer (LDD) with which LEGO models may be built. Clearly, LDD impose certain structural constraints that are carried over from physical LEGO

bricks. The block based modeling system by Leblanc et al. [14] is closer in spirit to our work: their system is not tangible, but users may place 3D blocks that are also joined to form a coherent mesh. Since the blocks are not physical, their system has few constraints on placement but does not offer the theme based reinterpretation of the block structure that our system does. However, Aliaga et al. [15] presented a system for visualization and interactive design of architectural models that applies architectural details to coarser structures in a procedurally defined facade model. In an abstract sense, our system for detecting subgraphs that represent particular structures can be seen as an example of inverse procedural modeling [16] where the inverse modeling is greatly aided by the simplicity of block structures, and the goal is to replace the recognized structures by detailed models.

3. System overview CubeBuilder allows the user to build virtual 3D models using a set of cube shaped physical building blocks. As the blocks are placed one by one (Fig. 1) to form larger structures, their positions and orientations are registered using input from a Kinect V2. Users are presented with a real-time rendering of the digital model. This rendering does not necessarily correspond one to one with how cubes are placed in real-life. Instead details are added to the geometry in order to reflect what the user might imagine the cubes to represent. For instance, we can apply a castle theme as seen in Fig. 2. CubeBuilder uses a SLAM [17] solution to track movement of the Kinect device in 6DoF. This allows the sensor to be moved which in turn allows for the creation of structures that are larger than what fits inside the Kinect's field of view. Knowing the position and orientation of the Kinect relative to the scene also allows us to create an augmented reality viewport where the themed geometry is rendered on top of a video stream from the RGB camera of the device.

Fig. 1. With our system users can construct rich digital 3D models by combining physical blocks in front of a Kinect. A: The first cube is placed in the building area and registered via our setup. B: To make cubes levitate they can be combined with support objects for example in the form of building plates. C: While building, registered cubes are visualized in our user interface with applied themes. D: Structures larger than what can fit in the Kinects field of view can be achieved by moving the sensor while building.

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

3

Fig. 2. Screenshot from the web-based user interface. Center of the UI is a real-time rendering of processed cube geometry. The model can be panned, zoomed and rotated using either a mouse or fingers on a touch device. At the top is large round touch buttons each button activates one of the registration actions, these actions can also be triggered by voice commands. In the right side there is a panel with options for configuration parameters.

3.1. Setup & user interaction

4. Cube placement

In our setup, the Kinect is mounted on top of a tripod and it is placed in such a way that it monitors a building area. An iPad is mounted on top of the Kinect using a custom bracket. On the iPad a graphical user interface is presented that shows a real-time visualization of registered building blocks and processed geometry. We use cubic blocks with an edge length of 43 mm (and have 104 such blocks, although more could be used). All blocks are painted white with matte paint. There is also a set of thin plates with length equal to three or four cubes, that can be used to support building blocks in lieu of other blocks. The Kinect is connected to a stationary Linux PC (Intel i75820K, Nvidia GTX 980) that processes incoming depth images to perform a registration of placed cubes. The PC also functions as a web server that can simultaneously present the user interface over wireless network to the iPad as well as other device. The user interface (Fig. 2) presents a real-time rendering of the currently registered cubes and the applied castle theme as an overlay to a video stream from the Kinect's RGB camera. At the top of the viewport six large touch buttons are shown. These buttons activate certain actions: the first two switch between Move and Place mode. In move mode the user is allowed to translate and rotate the Kinect sensor, but in order to register new blocks, the mode must be set to “place”. The two next buttons are Add and Remove. These actions are used to notify the system that changes have been made to the scene when the user has either placed or removed a block. Finally, there is an Initiate button that is used to do an initial calibration and an Undo button that can be used to remove the last placed cube. In terms of implementation, there are two main parts to CubeBuilder. First there is the tracker: a multi threaded C þ þ/ CUDA application that performs real-time analysis on incoming depth images in order to register placed building blocks. Second, there is the geometry processor, a web based JavaScript application that processes the registered poses of building blocks into detailed and themed geometry. The JavaScript application is also responsible for creating the real-time visualizations and user interface.

When users place building blocks one by one to form a larger structure, the blocks are simultaneous registered in a digital model. In order to do so, translations and orientations of the blocks are computed using the data from the Kinect. We perform a segmentation of a newly placed cube from the background (including previously placed blocks) to isolate it in the depth images. To obtain its pose, a model cube is fitted to the data points from the segmented part of the image. 4.1. Data acquisition and pre-processing The first step in our pipeline is to acquire data from the Kinect. The sensor delivers three image streams: infrared, color, and depth; each at 30 Hz. Infrared images I and depth images D are captured with the same camera chip since the depth image is computed from the infrared image. These channels all have a resolution of 525  424 pixels. To reduce noise the depth map is filtered by two filters. The first filter simply labels low intensity pixels in I as errors in D. This is because badly lit areas of the image are more likely to contain noise. The second filter is a Temporal Per Pixel Median Filter, that calculates the median for each pixel across the last n ¼ 10 incoming depth frames. Using the camera calibration method in [18] the internal parameters of the Kinect's infrared camera have been estimated. This allows us to convert D into a 3D point cloud, P, relative to the camera. 4.2. Detection and pose estimation In order to get a good segmentation of newly placed blocks, CubeBuilder uses the Kinect Fusion (KF) [11,12,19] algorithm to perform a real-time reconstruction of the working area. KF works by integrating P into a TSDF (truncated signed distance function) volume [20]. Voxels in this volume hold two values: a signed distance to the nearest surface and a weight. When points from the point cloud are integrated into the same voxel over time, the weight is increased and if no points are present for a voxel, its

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

Current pointcloud

Unfiltered ICP outliers

Ray-casted reconstruction

Filtered ICP Outliers

Fig. 3. The ICP outliers map in (c) is a product from Kinect Fusion. Notice that one of the building blocks (a) have not yet been integrated into the reconstruction, hence it will be detected as errors in the ICP alignment step. (b) Noise in the outliers map is filtered using binary morphology and a connected components labeling. (d) When the outlier map is filtered they are used as a mask to segment newly placed blocks.

weight is decreased. To reduce noise, points that fall within the same voxel are averaged. The practical result is that objects will fade into the reconstruction when introduced and fade away when removed. Since KF is a SLAM algorithm the reconstruction works even when the sensor is moved. It can do so by constantly calculating a camera pose which is relative to an initial pose. This is done by finding a transformation for each frame using Projective ICP [21] between the current depth frame and a ray-casting of the reconstruction from the previous sensor pose. When these transformation between frames is combined over time, the result is a sensor pose ½R; t. Having this pose allows KF to integrate incoming point clouds into the TSDF volume from the newly calculated sensor pose. For more details, we refer to [19]. Since Kinect Fusion uses Projective ICP, an ICP outliers map can be constructed (Fig. 3). All pixel values in the outliers map O are set to a non zero value where ICP fails. This will typically be where a foreground object is introduced to. However, the outliers map also contains a high amount of noise. We use binary morphology [22] to remove isolated points and narrow lines from O. After this process, a depth aware connected component analysis is applied [23,24]. This method identifies a number of connected blobs of pixels in the depth map and each pixel is applied with a blob label. Since the system only supports adding or removing a single cube at the time, only the largest blob is allowed to remain in the outliers map. The result is a filtered version of O that can be used as a mask on P to remove most points that do not belong to the newly placed building block. Having filtered previously placed cubes and background areas away from P it is now possibly to start computing a pose for the building block of interest. Our pose estimation is based on ICP [25], where an artificial point cloud (Fig. 4) is fitted to P. However ICP is

infamous for getting stuck in local minima and needs a good preliminary start pose. Each face of a cube can be represented by a plane, and unless a face of the cube points directly towards the sensor, there are at least two sides of the cube visible to the sensor. This means that if these two planes and cube face centroids can be estimated, a cube pose can be calculated since there is a known cube size. To segment the points in P according to what cube face they belong, a set of surface normals N are computed from P [26,27]. Observing that these surface normals originate from a cube, they should be organized in two or three clusters perpendicular to one another. Clustering is performed using the K-Means algorithm [28]. Fig. 5 is applied on N . Planes are now fitted to the two clusters with the largest point count using RANSAC [29]. We always search for three clusters with K-Means, even in the case where only two faces of a cube are visible. In practice this is not a problem since the third cluster will only contain a few points with noisy normals, and only the two clusters with the largest point count are actually used. Because the cube size is known, we can compute a 4  4 transformation matrix C from the plane normals and centroids of the two clusters. As mentioned, C is refined in an ICP step before it is ultimately stored. The cube pose is always flipped such that the Y-axis is up and negative Z points towards the camera. 4.3. User interaction There are two main states that the system can be in. These are called Move and Place mode. When the state is set to Move, KF integration is activated to perform SLAM. While when in Place mode KF integration is deactivated and the sensor position is

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

1 2 3 4 5 6 7 8 9 10 11 12 13 14 Q5 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

5

Fig. 4. Left: The artificial cube is shaped as a cap since points on the lower third of the cube tend to be distorted due to reflections from the table surface. Right: Red points inside the white lines are ICP target points, grey points inside the blue line are source points. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this paper.)

Masked Cube 3D Points Gaussian Image of Normal Directions (Cluster's Labeled

−0.85 −0.90 −0.95 −1.00 Y −1.05 −1.10 −1.15 0.15

1.0 0.5 0.0 Y

0.10 0.05

0.05

0.00

0.10 0.15 X

−0.5 −1.0

−1.0

Z

−0.05 0.20 −0.10 0.25

1.0

−0.5 X

0.5

0.0

0.0 0.5

−0.5

Z

1.0 −1 0

Fig. 5. An example of a masked point cloud (left). Points are segmented into three group using K-Means Clustering in Normal Space (right).

locked. When the Add or Remove action is triggered, integration is activated as soon as a building block has been registered and automatically deactivated again after a short duration. Since our blocks are completely plain, they need to be supported from below. However, in some cases we want this support to be invisible. To attain this goal, we use support objects as illustrated in Fig. 1b and c. Support objects are added to the scene by placing them in the building area as a part of the structure and then temporarily switch the system to Move mode, what will activate KF. At this point, the objects are simply integrated into the volumetric reconstruction and will no longer be present in the ICP outliers map. Hence, even building blocks could be used as support objects without risk of changing the cube graph. However, we often use thin plates as a support objects. A thin plate can rest on some blocks and support others. The process of removing building blocks is almost identical to the one of adding cubes. But instead of using the current point cloud P that will no longer contain a removed cube, a new point cloud P bg is ray-casted from KF's TSDF volume. This new point cloud will contain the removed cube as it has not yet been allowed to fade away from the reconstruction. All that is left is to repeat the process for adding blocks, but instead remove the block from our model. 5. Connection geometry We assume that the user intends the digital model to be more than just a collection of cubes. In fact, we consider the individual

cubes as tangible UI elements that allow the user to define a coherent (not necessarily axis aligned) structure as illustrated in Fig. 6. Our strategy for defining such a structure is to build a graph G of adjacent cubes and to further create geometry between connected cube faces. In our cube graph, we allow each cube to have a single connection for each face. Thus, two connected faces define a connection surface which is simply the surface obtained by lofting the two face outlines along a path given by a cubic Bézier curve. In practice, as illustrated in Fig. 8, we define a path for each vertex pair. The final geometry output at this point is a triangulated surface that encompasses both the individual faces and also the connection surfaces. 5.1. Connecting cubes A given cube face is only allowed to connect to a single other face, and we simply choose the closest face that also fulfills some additional conditions itemized below:

 We only allow connections if the distance is less than a user defined threshold t max  distance .

 Only faces roughly opposite each other should be connected. This condition is illustrated in Fig. 9. In this figure, we see that the angle α between the face normal from cube A (green arrow) and a vector between the faces (blue arrow) can be used to determine if the faces are eligible. We use the threshold t cos α 4 0:34.

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

6

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

Geometry is curved based on the rotation of cubes

Distance threshold is too high for this model t max-distance= 0.2m

Notice how faces perpendicular to each other are not connected.

A more fitting distance threshold t max-distance = 0.04m

Fig. 6. Selection of models that shows how connection geometry is made between cubes in different situations.

Without Y-Lock Y-Average

With Y-Lock Y-Average

Fig. 7. Examples of a model with and without Y-Lock Y-Average. Notice how this method allows cubes to keep their overall rotation while suppressing pose estimation errors. The missing cube in this model is an unwanted side effect of forcing cubes into an Y-grid.

 The two faces should opposite orientation. Specifically, we 

require that the angle between the normals t cos β o  0:34. Cube poses are registered with some errors and often stacked cubes will intersect a little. When this happens cubes will not be connected unless a cheat distance threshold is used t cheat  distance ¼ 0:01 m.

All of the restriction and special cases are used to test all cubes and their faces against all other faces. In this process, a record is kept of connected pairs of faces. Since we allow only a single connection per face, there is a maximum degree of ΔðGÞ ¼ 6 and a minimum degree δðGÞ ¼ 0. Our graph is not necessarily connected and can consist of n connected components sub graphs G0-n. We also construct a second graph Y with the same nodes as G but where only connections to cube faces þ y (up) and  y (down) are registered as edges. The Y graph will typically be less

connected than G, and it has a maximum degree ΔðYÞ ¼ 2. Again, it will consist of a number of connected sub graphs Y0-n. Having a graph representation of cubes and cube connections yields the possibility of addressing pose estimation errors. The idea is that stacked cubes are most often supposed to have the same rotation around the up axis. The Y graph tells us exactly which cubes are stacked – each connected component with more than one node will be a series of stacked cubes. The solution is then to average rotations around the up vector y for cubes in a sub graph Y i A Y 0  n . Besides averaging rotation the cubes are locked in a Y-grid. An example of a model with and without this method (Ylock Y-average) is shown in Fig. 7. 6. Themes by connectivity and structure In CubeBuilder there are several ways to form a model, with a short connection distance models like the house in Fig. 7a can be

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

Arrows in green are face normals. Face normals and the distance between cubes is used place control points p2, p3

7

Two cubes connected by four b´ezier curves (red). Triangulated geometry are created between curves

Fig. 8. Example on how cubic Bézier curves are used to generate connection geometry between cubes. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this paper.)

connections are interpreted as peasant houses, two stacked cubes without any other connections produce a tree, and cubes stacked three cubes high (or more) yield a tower, see (Figs. 13 and 14). The recognized subgraphs do not have to be isolated. For instance, the “gate” pattern that is identified by 5 cubes arranged as in Fig. 1c is allowed to have connections to the two lower cubes while still being identified within a larger structure. 6.1. Searching the cube graph

Fig. 9. Left: Case where the cubes should not connect, because of the wide angle between A's normal (green) and the face to face vector (blue). Middle: Case where the cubes should connect. Right: Vectors from middle case. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this paper.)

built, and with longer connection distances, complex models can be achieved with only a few cubes, see Fig. 6a and b. Ideally, we would like to be able to model objects with features smaller than a single cube. In general, this is clearly impossible, but if we assume that the user is restricted to modeling shapes that belong to a certain class of shapes, more possibilities emerge. Here, it is assumed that the user models shapes that are defined by what we call a theme. In the current system, there is only few themes available, but many other themes could be developed. The Castle theme assumes that users are building a medieval castle, fortress, or adjacent structures, and the system then adds detail like towers, roofs and parapet to the block model. Themes are applied by searching for predefined substructures within the cube graph and then replacing these with theme defined geometry. The pre-modeled objects for the Castle theme can be seen in Fig. 10. In the castle theme, isolated cubes with no

Structure is deduced from the graph representation of connections between cubes. In practice, we define and search for subgraphs with a particular layout that is specified using an adjacency list. It is also possible to restrict the search by imposing specific conditions (restrictions) on some cubes in the subgraph. These restrictions are implemented in terms of cube properties. For instance, we can impose conditions on the degree of a cube, whether specific faces are connected, and the vertical position (height) of the cube in a stack of cubes. The following are our elementary conditions on a given cube:

Above, we have used the convention that the vertical bar indicates alternatives and the contents of square brackets is optional. Based on these elementary conditions, we have defined composite conditions. For instance, a “peak” is a cube of degree one whose singular connection must be to the cube below. Other examples of such composite conditions are “corner”, “peak_corner”

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

8

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

Fig. 10. Castle theme objects. A: Keep. B: Gate. C: Tower top and tower base. D: Mini keep variations. E: House and tree. F: The basic shape for a standard cube.

and “edge_connector” (Fig. 11). Definitions on composite conditions are given here:

precise, a search object is defined by the following pieces of information:

 Number of cubes.  Subgraph: A subgraph defined as a list of connections between two cube, face pairs.

 Restrictions: A list of conditions, either elementary or composite as discussed above.

where ? is a placeholder for x, y, or z. To identify actual structures, we now define search objects that build upon the features mentioned above. To be more

To give a concrete example of a search object, the “Gate” object is shown in the bottom of Fig. 12. A gate subgraph is composed of five cubes. The center cube (0) is connected sideways to two cubes (1 and 3). These, in turn, connect down to 3 and 4. The restrictions are that the topmost cubes must have degree two. Since this is their degree in the subgraph, further connections are not possible. The bottom cubes are restricted only in that they are not allowed to connect downwards: a gate subgraph must rest on the workspace not be supported by other cubes.

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

Corner

Peak

Peak Corner

Edge Connector

9

Fig. 11. Cubes in green are found to be of the type Corner, Peak, Peak Corner or Edge Connector. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this paper.)

Note that finding a subgraph which matches the search object really amounts to solving a subgraph isomorphism problem which is NP-complete in general [30]. However, our specific problem is much simpler than the general case because the edges must connect specific cube faces: thus starting from the first cube defined in the search object, we simply traverse all connections and validate all conditions until a connection is missing or a condition fails. Clearly, as explained so far, the fact that we define subgraphs in terms of connections between specific faces means that our scheme is not rotationally invariant. We address this problem by searching for all four planar rotations of the search objects that we define.

7. Results This section outlines our experimental results, for the evaluation of pose estimation quality of building blocks and our user experience tests. Robustness and quality of our pose estimation have been verified through a number of tests. Cube pose translation and orientation offsets was measured against planes fitted to large planer areas in the point cloud. We found that the mean displacement on X, Y and Z was around 1 mm and around 1.5° for rotation. When building large structures, there are occasional fallout where errors in pose estimation is not acceptable to the user. Users can the activate the Undo function to try again. Modeling in this way, CubeBuilder has proven robust enough to build structures consisting of more than one hundred building blocks (Fig. 18). However, our system is not restricted to this number of blocks; the limit of 104 is simply imposed by the number of physical blocks produced for our user tests.

Since we use no information about previously placed cubes when estimating the pose of the last placed cube, the computational costs for pose estimation is constant. We are able to process depth frames in nearly the same rate as the Kinect delivers them (Fig. 15). Compute time for the geometry processing will rise with the number of building blocks, but only needs to be computed once each time new cubes are added. Compute times for geometry processing are plotted in Fig. 16. Often large models will not fit in the Kinect's field of view, and the sensor will have to be moved during a session. We have experienced some problems with drift in sensor pose estimation. However, it has proven robust enough to achieve models like the top one in Fig. 14 where the sensor was moved several times during the build. We also created a second, simpler theme aimed at facilitating models at a finer scale than supported by the Castle theme. The premise here is that whenever the blocks define a staircase, we would like to replace that with a constant slope. This theme is shown in Fig. 17. Whenever a pair of faces form the tread and riser of adjacent steps, we replace this geometry with a constant slope. 7.1. User tests CubeBuilder was tested on a handful of users, some with previous digital 3D modeling experience and some without. The design of our usability tests was inspired by the micro-usability test guide [31]. The goal of the tests was both to identify problems and to find what works well with our system. After a short introduction to the system, users were simply asked to create a structure using CubeBuilder. Users where allowed several tries until they had created a model of their liking. After the session they where briefly interviewed about what they liked and disliked about the experience. Some visual results from these sessions can

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

10

Fig. 14. Selection of structures with the castle theme created by test users.

Tracker - Performance per frame update 90 KinFu

80

Porint Cloud Processing

70

Total

60

Tracker - Normals

ms

Tracker - Total

50

Tracker - Visualizer

40 30 20 10 0

Fig. 12. This figure features three examples of search objects definitions, from the castle theme. Tower tops, Gates and Small keeps at the end of a wall segment. Search objects are defined by a connection array and a restriction array. The connection array defines the search graph that we search for and the restriction array defines extra restrictions.

0

10

20

30

40

50

60

70

80

Number of placed cubes Fig. 15. Performance charts from building block pose tracker. The total time per frame is around 65 ms. However readers should notice that (Tracker – Visualizer) takes around 30 ms, this visualizer is not the same as the one presented in our web interface it is only for debugging and development. Hence the true time per frame is around 35 ms, very close to the rate of sensor input 30 Hz. The performance data in this figure is from a structure consisting of 85 cubes.

Geometry Processor - Performance at cube placed 900 Connections Cubes Graphs

800 700 600

ms

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

500 400

Fig. 13. Registered cubes before connection geometry is constructed and the castle theme is applied. The resulting model with theming can be seen at the top of Fig. 14.

300 200 100

be seen in Fig. 14. Below is included selected quotes from the user interviews. What users liked about Cube Builder:

 “I found the system to be user friendly and completely different from what I have seen before.”

 “I really liked the idea of themes.”  “It was fun to combine the computer with something real.”  “It was fun to imagine what could be built given the restrictions  

of the physical cubes and how the system worked.” “The system was really simple to get started with.” “The fact that the bricks are being interpreted as you lay them down makes it feel a bit like an AI is assisting the construction.”

0

0

10

20

30

40

50

60

70

80

Number of placed cubes Fig. 16. Performance measurements from the geometry processor. Especially creating cube connections are computational expensive, since all faces are tested against each other. Fortunately cube connections only need to be rebuild when new cubes are added. Subgraph searching and insertion of theme geometry scales linearly.

What users disliked:

 “I found it annoying that I could not place more than one cube at the time, I wanted to just put 10 cubes on the table and then place them.”

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

Physical building blocks

11

Blocks with constant slope theme applied

Fig. 17. Constant slope is a theme that can be used to model more detailed models where we replace tread and riser pairs of a step with a sloping surface.

Fig. 18. Model consisting of all 104 building blocks in our construction set.

 “It is not quite as fast as one might like and It is a little tricky to remove bricks.”

 “I had to get used to the limitations.”  “I had to look at the “Development UI” to figure out when it was OK to place a new cube, that somehow separated me from what I was building.” CubeBuilder is a prototype software and has a few pitfalls that users needs to be aware off. For example they need to physically place the building block before they press “Add” in the UI, and they will have to switch to move mode, before physically moving the sensor. One problem that some users discovered is that building blocks can occlude large portions of the build area if placed near the sensor. Resolving this issue would then require a large sensor movement. Another limitation mentioned by several users is that we presently do not allow users to place or move several blocks simultaneously. However even with these limitations users seemed to quickly adapt and were able to achieve great looking structures.

8. Discussion Thanks especially to the template based system for applying themes and the ability to connect blocks by lofted geometry, CubeBuilder has proven to be an intuitive and effective system for modeling fairly complex structures. We believe that our system might be useful for a number of tasks. The most obvious use might be the modeling of environments intended for augmented reality experiences. As long as the transformed 3D geometry is not too different from the original blocks, it is possible to mix real props with computer graphics. We also imagine that having a tangible interface may prove useful when it comes to many tasks where an environment needs to be

designed: obviously level design and conceivably planning tasks where certain types of entities need to be placed. One big advantage of CubeBuilder is that the blocks can be placed in a complex environment that might also contain other objects which are simply integrated into the reconstruction without becoming part of the block structure. One unique aspect of our design is the ability to build models that are not axis aligned. Normally designers of building block sets will have to decide between interlocking systems with inherent limitations on user freedom and non-interlocking systems that are not able to achieve truly coherent structures. However with CubeBuilder's connection geometry, users are able to get the best from both worlds. During the development and test of the system, we also encountered certain limitations. Theme creation is a manual, creative process where the user needs to define certain patterns that are transformed into specific objects according to the theme. This requires both rules to be specified and objects to be created using CAD software. However, we provide a library of functions that facilitate the detection of features. The algorithm used to detect cube connections contains several parameters, but only t max  distance can be adjusted in the UI. The other parameters have been set so as to make the system robust and to ensure that not too many blocks are considered when detecting connections. Changing these parameters would generally have a negative effect on robustness and efficiency without affecting the function of the system. For the SLAM part of our system to work properly there needs to be strong features present in the depth image. Specifically, Kinect Fusion is dependent on variations in the derived surface normals. One possibly way to improve robustness of the pose estimation for both the sensor and building blocks could be by including color features from the Kinects RGB camera. However we have been reluctant to add color patterns to the building blocks because in the future, color coding might be used to distinguish between different types of blocks. The method used for pose estimation of placed building blocks demands that at least two faces must be visible to the sensor. Clearly, this imposes certain limitations on where blocks can be placed. However, these limitations were easily understood by the users who planned their construction sequence to avoid such situations. From a user experience point of view, the speed of placement could be improved. The block detection takes only a fraction of a second, but requires the user to remove her hands from the Kinect's field of view. Initially, we believed that using a voice command to place blocks would facilitate faster building, but we

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 Q4 62 63 64 65 66

J.U. Walther et al. / Computers & Graphics ∎ (∎∎∎∎) ∎∎∎–∎∎∎

came to realize that physically pressing the button ensures that the user removes her hands from the workspace. In future versions, we hope to make the system automatically detect multiple new blocks with or without extraneous objects such as hands being inside the workspace. A simple detection of cubes from blobs in the outliers map coupled with a segmentation of non static parts of the depth image could be the first step towards such functionality. Further we would like the system to automatically detect when support objects are placed in the scene. The current system provides an inherent tactile feedback given by the physical properties of the building blocks. Going further, we might allow the physical model to be changed by the computer. For instance, this could be achieved with the momentum driven magnetic blocks proposed in the M-Block system [32].

9. Conclusion In this paper, we have presented CubeBuilder – an augmentedreality based, interactive, tangible 3D modeling system that reinterprets building blocks as coherent, themed structures. Our design is not dependent on any system of interlocking building blocks, but instead features plain cube shaped blocks, that can be placed as users see fit. We have detailed a robust and real-time method for pose registration of placed building blocks that allows for the creation of large models both spatially and regarding the number of included building blocks. With our geometry processing system we have shown how digital connection geometry can be created between placed cubes, and in this way form a graph that can be used as the basis for deducing model features and structures from nonaxis aligned models. Using the graph and inferred features we have presented a method of applying themes to enrich the simple cubes with more detailed geometry in an augmented-reality environment. From an application point of view, we are already able to combine the visualization with video from the Kinect, turning the building process into an AR experience. It would be possible to add both more real and more virtual elements to the scene with relative ease, indicating that the system has utility as an AR experience modeling tool. Several other applications come to mind as well. For video games where we imagine our system could serve as an input method as well as for tabletop games that could be enhanced with context aware AR graphics. Finally the geometric freedom is probably insufficient for architectural modeling as such, but as a tool for planning the spatial layout of, say, urban areas, a system such as ours might prove very effective.

Appendix A. Supplementary material Supplementary data associated with this paper can be found in the online version at http://dx.doi.org/10.1016/j.cag.2016.05.004.

References [1] Wright F. Frank Lloyd Wright: an autobiography. Pomegranate 2005 ISBN 9780764932434. URL: 〈https://books.google.dk/books?id ¼ S8zlZcJjNEMC〉.. [2] Provenzo E, Brett A. The complete block book. Syracuse University Press; 1983 ISBN 9780815623007. URL: 〈https://books.google.dk/books?id¼n10BGmBRZ50C〉.. [3] Payne A, Daniel A, Mehta A, Thompson B, Bamji CS, Snow D, et al. 7.6 a 512  424 CMOS 3D time-of-flight image sensor with multi-frequency photodemodulation up to 130 MHz and 2 GS/s ADC. In: 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC). IEEE; 2014. p. 134–5. URL: 〈http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber ¼6757370〉.

[4] Sarbolandi H, Lefloch D, Kolb A. Kinect range sensing: structured-light versus time-of-flight kinect; 2015. URL: http://www.arxiv.org/abs/1505.05459. [5] Ishii H. Tangible bits: beyond pixels. In: Proceedings of the 2nd international conference on Tangible and embedded interaction. ACM; 2008. p. xv–xxv. URL: 〈http://dl.acm.org/citation.cfm?id ¼1347392〉. [6] Aish R, Noakes P. Architecture without numbers – {CAAD} based on a 3D modelling system. Comput-Aided Des 1984;16(6):321–8. http://dx.doi.org/ 10.1016/0010-4485(84)90116-7 URL: 〈http://www.sciencedirect.com/science/ article/pii/0010448584901167〉.. [7] Anderson D, Frankel JL, Marks J, Agarwala A, Beardsley P, Hodgins J, et al. Tangible interaction þ graphical interpretation: a new approach to 3D modeling. In: Proceedings of the 27th annual conference on computer graphics and interactive techniques; 2000. p. 393–402. [8] Kitamura Y, Itoh Y, Kishino F. Real-time 3D interaction with ActiveCube. In: CHI '01 extended abstracts on human factors in computing systems. CHIEA '01. ACM. ISBN 1-58113-340-5; 2001. p. 355–6. http://dx.doi.org/10.1145/634067. 634277. [9] Gupta A, Fox D, Curless B, Cohen M. DuploTrack: a real-time system for authoring and guiding duplo block assembly. In: Proceedings of the 25th annual ACM symposium on user interface software and technology. ACM; 2012. p. 389–402. URL: 〈http://dl.acm.org/citation.cfm?id ¼ 2380167〉. [10] Miller A, White B, Charbonneau E, Kanzler Z, LaViola JJ. Interactive 3D model acquisition and tracking of building block structures. IEEE Trans Vis Comput Graph 2012;18(4):651–9 URL: 〈http://ieeexplore.ieee.org/xpls/abs_all.jsp? arnumber ¼6165147〉.. [11] Baksheev A. Kinfu remake; 2014. URL: 〈https://github.com/Nerei/kinfu_ remake〉. [12] Newcombe R, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison A, et al. KinectFusion: real-time dense surface mapping and tracking; 2011. http://dx. doi.org/10.1109/ISMAR.2011.6092378. [13] Persson M, Bergensten J. Minecraft. [Computer software] Stockholm, Sweden: Mojang AB. Retrieved from: 〈http://minecraft.net〉; 2011. [14] Leblanc L, Houle J, Poulin P. Modeling with blocks. Vis Comput 2011;27(6– 8):555–63. [15] Aliaga DG, Rosen PA, Bekins DR. Style grammars for interactive visualization of architecture. IEEE Trans Vis Comput Graph 2007;13(4):786–97. [16] Bokeloh M, Wand M, Seidel HP. A connection between partial symmetry and inverse procedural modeling. In: ACM Transactions on Graphics (TOG); vol. 29. ACM; 2010, p. 104. [17] Aulinas J, Petillot YR, Salvi J, Lladó X. The slam problem: a survey. In: CCIA. Citeseer; 2008. p. 363–71. [18] Wiedemeyer T. IAI Kinect2. 〈https://github.com/code-iai/iai_kinect2〉; 2014– 2015 [accessed 12.06.15]. [19] Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, et al. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th annual ACM symposium on user interface software and technology. ACM. ISBN 978-1-4503-0716-1; 2011. p. 559–68. URL: 〈http://dl.acm.org/citation.cfm?id ¼2047270〉. [20] Jones MW, Baerentzen JA, Sramek M. 3D distance fields: a survey of techniques and applications. IEEE Trans Vis Comput Graph 2006;12(4):581–99. http: //dx.doi.org/10.1109/TVCG.2006.56. [21] Rusinkiewicz S, Levoy M. Efficient variants of the icp algorithm. In: 2001 Proceedings of third international conference on 3-D digital imaging and modeling; 2001. p. 145–52. http://dx.doi.org/10.1109/IM.2001.924423. [22] Carstensen JM. Analysis, vision and computer graphics. Lyngby, Denmark: DTU; 2002. [23] Kalentev O, Rai A, Kemnitz S. Connected component labeling on a 2D grid using CUDA; 2010. http://dx.doi.org/10.1016/j.jpdc.2010.10.012. [24] Walther JU, Poulsen SV. Human interaction on real-time reconstructed surfaces; 2013. [25] Rusinkiewicz S, Levoy M. Efficient variants of the ICP algorithm. in: 2001 Proceedings of third international conference on 3-D digital imaging and modeling. IEEE; 2001. p. 145–52. URL: 〈http://ieeexplore.ieee.org/xpls/abs_all. jsp?arnumber ¼ 924423〉. [26] Estimating surface normals in a PointCloud; 2016. URL: 〈http://pointclouds. org/documentation/tutorials/normal_estimation.php〉. [27] Rusu R, Cousins S. 3D is here: point cloud library (pcl). In: 2011 IEEE International conference on robotics and automation (ICRA). IEEE; 2011. p. 1–4. URL: 〈http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber ¼5980567〉. [28] Kanungo T, Mount DM, Netanyahu NS, Piatko CD, Silverman R, Wu AY. An efficient k-means clustering algorithm. Anal Implement 2002;24(7):881–92 URL: 〈http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber ¼ 1017616〉.. [29] Fischler MA, Bolles RC. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography 1981;24(6):381–95. URL: 〈http://dl.acm.org/citation.cfm?id ¼358692〉. [30] Cook SA. The complexity of theorem-proving procedures. In: Proceedings of the third annual ACM symposium on theory of computing. ACM; 1971. p. 151–8. [31] Kuniavsky M. Observing the user experience: a practitioners guide to user research (Morgan Kaufmann series in interactive technologies) (The Morgan Kaufmann series in interactive technologies). Morgan Kaufmann Publishers Inc.; 2003. ISBN 1-55860-923-7. [32] Romanishin JW, Gilpin K, Rus D. M-blocks: momentum-driven, magnetic modular robots. In: 2013 IEEE/RSJ international conference on intelligent robots and systems; 2013. p. 4288–95. http://dx.doi.org/10.1109/IROS.2013. 6696971.

Please cite this article as: Walther JU, et al. Tangible 3D modeling of coherent and themed structures. Comput Graph (2016), http://dx. doi.org/10.1016/j.cag.2016.05.004i

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132