3D Computational Sketch Synthesis Framework: Assisting Design Exploration Through Generating Variations of User Input Sketch and Interactive 3D Model Reconstruction

Journal Pre-proof 3D Computational Sketch Synthesis Framework: Assisting Design Exploration Through Generating Variations of User Input Sketch and Interactive 3D Model Reconstruction Seonghoon Ban, Kyung Hoon Hyun

PII: DOI: Reference:

S0010-4485(18)30172-6 https://doi.org/10.1016/j.cad.2019.102789 JCAD 102789

To appear in:

Computer-Aided Design

Received date : 16 April 2018 Revised date : 1 July 2019 Accepted date : 9 November 2019 Please cite this article as: S. Ban and K.H. Hyun, 3D Computational Sketch Synthesis Framework: Assisting Design Exploration Through Generating Variations of User Input Sketch and Interactive 3D Model Reconstruction. Computer-Aided Design (2019), doi: https://doi.org/10.1016/j.cad.2019.102789. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

© 2019 Elsevier Ltd. All rights reserved.

Journal Pre-proof

3D Computational Sketch Synthesis Framework: Assisting Design Exploration Through Generating Variations of User Input Sketch and Interactive 3D Model Reconstruction.

pro of

Seonghoon Bana and Kyung Hoon Hyuna, Hanyang University, Seoul 04763, Republic of Korea

Jo

urn a

lP

re-

a



Corresponding author. Tel.: + 82-2-2220-1189. E-mail address: [email protected] (K. H. Hyun).

1

Journal Pre-proof

Abstract A framework is proposed for facilitating the exploration process during the early design phase through computational sketch synthesis and interactive 3D reconstruction. In that phase, designers concentrate on developing concepts through numerous alternatives. Therefore, they constantly sketch so that they can rapidly

pro of

visualize their ideas. Recently, the design industry has attempted to streamline the design process by

implementing 3D model generation in the early design phase so that ideas may be more thoroughly explored, thus improving concept and final design conformance; however, efficiency issues have arisen. In this study, a 3D computational sketch synthesis framework was developed comprising two major components. First, a robust method was proposed to synthesize design alternatives by interpolating an input sketch with sketches in a database so that unvisited combinations may be explored. Secondly, a novel interactive 3D model reconstruction method was developed to facilitate the shape transition of design elements so that designers can quickly evaluate the potential of a large number of design variations. Finally, an interface for through design refinement was developed so that designs may be embodied by sketching over the 3D model. To test the proposed methodology, expert designers were recruited for a validation experiment with two conditions followed up by in-depth

re-

interviews. In the first condition, the participants were asked to sketch based on a design brief in their current working manner. In the second condition, they were asked to create designs using the proposed framework. It was tested whether there was a difference in the design outcomes. It was demonstrated that the proposed framework resulted in more satisfactory and higher-quality designs and generated design alternatives faster and in greater quantities. All participants agreed that the framework could be useful in the early design phase and

lP

responded that the proposed system provides more design inspiration than traditional design methods. Most importantly, it was demonstrated that the proposed framework could enhance the reevaluation potential of design concepts and assist in making better-informed design decisions. Keywords: Intelligent Design System; Assisted Creativity; Sketch-Based Modeling; Computational Design; Virtual Reality.

urn a

1. Introduction1

Designers spend a considerable amount of time exploring aesthetic aspects of their designs [1]. They constantly visualize design concepts by repetitively generating every possible variation [2]. Sketching is a major component of the concept selection and embodiment phases [3]. Sketching facilitates the development of design solutions, knowledge acquisition, and representation during the early design phase of concept development, in which designers concentrate on generating design ideas [4]. Sketching is the most natural way to design [5] and enables the expeditious production, evaluation, refinement, and replacement of ideas [6–8]. Also, sketches

Jo

1 Abbreviations: CSS: Computational sketch synthesis 3DCSS: 3D computational sketch synthesis FOV: Field of view FP: Front point RP: Rear point SSM: Style synthesis methodology TLX: Task load index

2

Journal Pre-proof

contain design information from concept development [9], which is considered the most crucial phase in design development [10]. In the early design phase, designers start with ambiguous and imprecise sketches to investigate unexplored design spaces. Ambiguous and imprecise sketches are an important component of conceptual design because there may be several alternative interpretations of the same sketches [11]. However,

pro of

the exploration of design ideas involves creation and evaluation of specific configurations including shapes, proportions, and characteristic features [1]. Therefore, the overall sketch can be loosely defined, but the specific design elements can be highly precise. For instance, car designers often start sketching with the details of a specific design element and produce incomplete combinations of design elements for additional inspiration. To facilitate the design exploration process, researchers have concentrated on developing methods that increased sketching efficiency by improving the interactivity and usability of sketching interfaces. If design is regarded as a combination of design elements, then numerous combinations of design elements can be created [2].

Consequently, it is difficult for humans to visit every possible design variation through freehand sketching. Unlike humans, who have limited time to pursuit creative activities owing to fatigue [12], computers can generate large numbers of novel design ideas. Moreover, novel designs can be created by combining and modifying existing designs [13]. Design problems are often approached based on previous examples. Therefore,

re-

it is natural to implement past examples in a computational design tool for more effective workflow [14]. Thus, computers can discover unexplored combinations that may lead to creative design solutions. Moreover, computational tools can facilitate the design process by automating tasks that require heavy calculations, thus reducing the design manufacturing cycle from one week to one day [15]. Hence, computationally synthesizing variations of a designer’s input sketch using sketches in a database may significantly facilitate the early design

lP

phase.

Despite the significance of sketching, there are qualities that are better captured in 3D models, such as dimensionality, proportion, and shape transition from various views. Car designers, for example, constantly and consistently sketch variations to create designs that look appealing form various views. A design sketch can be aesthetically pleasing from front and side views but not from perspective view. Thus, the shape transition of design elements from various views often requires a design decision regarding which curves should be

urn a

maintained or suppressed so that aesthetically pleasing 3D models from various views may be created. Car designers create rough 3D models of a 2D sketch using CAD software to evaluate design concepts from various views; however, creating 3D models requires time and effort. 3D models require accurate dimensions and constrains that represent the design intent for the detailed design phase [16]. Compared with traditional 3D modeling processes, computational methods allow designers to iterate through a large number of design solutions [17]. Researchers in sketch-based modeling have developed state-of-the-art systems that accurately reconstruct 2D sketches into 3D models [18, 19] or assist in accurately creating 3D curves in the air [20]. Sketch-based modeling is an active research field that concentrates on transforming of 2D sketches into finely detailed 3D models [6]. However, facilitating the shape transition of 2D sketches at the design element level to

Jo

3D models that are appealing from various views is an unexplored research area. In summary, the hybrid of computational synthesis of user input sketches and 3D model reconstruction to

aid in the stylistic evaluation of a large exploration space has the potential to assist in the stylistic decision making during the design process, in particular, to assist designers to evaluate the shape transition of 3D models reconstructed from computationally synthesized 2D sketches at the design element level. In this respect, the

3

Journal Pre-proof

3DCSS framework is proposed to facilitate the design exploration process through generating variations of user input sketches and through interactive 3D model reconstruction in the early design phase. The framework consists of two major components: CSS and interactive 3D model reconstruction. CSS allows designers to interpolate an input sketch with previous sketches stored in a database. Designers can choose to utilize either a

pro of

full or a partial list of design elements to generate variations of complete or incomplete design alternatives. Interactive 3D model reconstruction assists designers to interactively explore 3D models by automatically generating a series of 3D model variations obtained by shape transition methods. Unlike previous studies that concentrated on accurately reconstructing 3D models based on 2D images, the present study proposes an interactive system that reconstructs variations of 3D models so that shape transition may be refined. To the best of our knowledge, this study is the first investigation of 3D model generation from sketch variations at the design element level for quick and thorough evaluation of design concepts. Furthermore, we incorporated a virtual reality for 3D model shape transition so that the designs can be evaluated in an immersive environment. To test the proposed framework, professional designers were recruited for a validation experiment followed up by in-depth interviews. The contributions of this study are as follows:

re-

1) A robust framework is proposed to facilitate the design exploration process in the early design phase by computationally synthesizing variations of user input sketches and interactively reconstructing 3D models.

sketches in the database.

lP

2) A robust method is proposed to synthesize design alternatives by interpolating an input sketch with

3) A novel interactive 3D model reconstruction method is developed to facilitate the shape transition of design elements so that the potential of a large number of design variations can be quickly evaluated. 4) We confirmed that the proposed framework enhances the reevaluation potential of design concepts and

urn a

assists in making better informed design decisions.

2. Related Work

2.1. Sketching and computational synthesis The introduction of the Sketchpad by Ivan Sutherland in 1963 revolutionized computer graphics and human–computer interaction [21]. Despite advances in new digital design tools, designers still sketch with pen and pencil in the early design phase. Muller et al. [22] identified that sketching is used both in preparing for

Jo

CAD and during CAD. Their study indicated that 35% of designers always sketch in the concept design phase, and 90% replied that the primary reason for sketching is to create new design solutions. Expert designers can quickly explore design variations through a minimal representation of the design concept and the expression intensity of the design idea, and the number of design alternatives significantly varies depending on expertise [23]. Meanwhile, computational synthesis automatically generates design variations by simply providing values for parameters. A large design space can be explored to search for near-optimal design solutions generated

4

Journal Pre-proof

computationally [24]. In this case, an interface for selecting suitable solutions is as important as creating large sets of design solutions. Zaman et al. [25] developed GEM-NI with effective interface for computational synthesis. GEM-NI is a graph-based system for creating and managing design alternatives with several novel functions. The system allows parallel editing and posthoc merging of design alternatives. Matejka et al. [26]

pro of

proposed a method for exploring large-scale generative design datasets through Dream Lens. Unlike previous studies on computational design that fully utilize algorithmic solvers, Matejka et al. [26] utilizes exiting solvers from a large dataset. A “by example” approach is used to define rankings based on preferences. Dream Lens provides visualization and an interface for finding design solutions by rating and searching presynthesized datasets. However, computationally synthesizing design alternatives from given parameters is not an efficient method for stylistic exploration. For instance, designs from competing brands are a major source of stylistic inspiration for car designers. Car designers consider the appearance of generation and competing products (family look and design trend, respectively) when designing new models [2]. Thus, interpolating with other design populations is an important feature of stylistic exploration.

In connection with the point previously made, the implementation of computational synthesis for freehand sketching can be an ideal method for reducing cognitive workload in drawing production, evaluation, and

re-

reinterpretation of design concepts. Recently, it has been attempted to develop a hybrid of sketching and computational synthesis. Kazi et al. [17] introduced DreamSketch, which utilizes ambiguous and incomplete aspects of sketching and computer-driven generative design to provide a glimpse into a larger space of design exploration. Therefore, designers can be better informed when making decisions. However, DreamSketch considers the position of 2D objects to be a design variation, and stylistic exploration of design elements

lP

involving comparisons with other designs is not supported. Ha and Eck [27] developed an automatic sketching program based on Sketch-RNN that allows users to create novel sketches. Specifically, it allows digital sketching, generates sketch variations, and even completes unfinished designs, thus facilitating the process of artistic creation. However, this methodology has limited reconstruction capability when the object is complicated. ShadowDraw by Lee et al. [28] guides freehand drawing by dynamically updating the underlying image as the user draws. The underlay image is constructed from a large dataset, and it updates the image

urn a

corresponding to the user’s input strokes in real time. Despite the edge detection techniques in [28], the averaged shadow results in ghosting, which is suitable for proportion and spacing guidance but is lacking in reconstructing design details. Detailed expressions of the design elements are required for high design quality. World-renowned designers Charles Eames and Dieter Rams stressed the importance of detail in design [29, 30]. Thus, ShadowDraw can be useful to the sketching novice for learning freehand drawing but not to design experts who are familiar with exploring design details through sketching. Meanwhile, Kwok et al. [31] implemented an interactive evolutionary algorithm for tight-fitting garments to generate design variations automatically. The users rate the design variations to score good and bad designs and synthesize a crossover of novel designs through a topology metamorphosis approach. Topology metamorphosis allows regional crossover

Jo

of the design inputs, but it is not possible to synthesize designs at the design element level, such as arm sleeves and pants. A similar method to ours is that by Arora et al. [32], who proposed SketchSoup; it allows the exploration of design ideas through image blending of design sketches. The design spaces can be better presented, and the result of SketchSoup can be used as an underlay image for drawing new concepts. SketchSoup utilizes image blending to morph a design sketch into another. Design is a combination of design

5

Journal Pre-proof

elements, and morphing a design element can change the overall design. Even car designs undergo facelift and partial changes for a new product appearance. Designers often want to change the design elements while maintaining the other combinations. In the development of a new system for computational synthesis in the early design phase, it is important to

pro of

consider two features: interpolating with other design populations and morphing at the design element level. We incorporated shape interpolation with existing designs from a large database. Car designers consider various requirements for design exploration, but stylistic exploration of brand identity and design trend is essential. Interpolating alternatives with other design populations is a stylistic exploration approach that is more similar to the actual car design process than the parameter-driven approach. Finally, the proposed method in this study is capable of interpolating and tracking the history of synthesized alternatives at the design element level. The morphed design elements are the key components for exploring complete and incomplete design variations by synthesizing designs with or without using the full list of design elements.

2.2. Sketch-based modeling

The objective of sketch-based modeling is to use a natural user interface for sketching, from rough 3D

re-

model generation to a detailed and accurate final design [6, 33]. Sketch-based modeling allows multiple features, such as recognizing a sketch and transforming it into a 3D model [34, 35], providing a quick structural analysis of the 3D model [36], and providing an intuitive interface for the modification of the 3D model [5, 37]. Creating a detailed 3D model has been a major concern in sketch-based modeling [6]. Car designers and managers want to determine and evaluate the principal shape as early as possible [38]. To this end, designers

lP

create a full-scale orthographic sketch of the design using tape drawing. In this respect, Grossman et al. [38] proposed a sketch-based modelling method based on a natural user interface for tape drawing. The digital tape drawing method automatically retrieves the coordinates of the drawing on a large display and generates 3D curves. Similarly, Bae and Kijima [3] developed a method for drawing and modifying real-size 2D sketches using a projector. Thereby, designers can quickly test different design variations on actual scale, thus facilitating decision making. Immersive technology in design industry is primarily implemented in the late design phase,

urn a

when manufacturing and assembling are simulated to support decision making [39]. However, the recent advancement of immersive technology allows designers to formulate, deform, and trim 3D surfaces with or without hand-held devices in the early design phase [40–42]. Arora et al. [20] created SymbiosisSketch, a hybrid system that combines drawing in the air (3D) and drawing on surface (2D). Augmented reality was used for creating 3D model designs to leverage the affordances of 3D (immersive, unconstrained life-sized) and 2D (precise, constrained, ergonomic) interactions. SymbiosisSketch utilizes series of unorganized collection of 3D curves to create surfaces. Consequently, the surfaces function as shape proxies and as a canvas to map strokes down to the 2D tablet. Thus, immersive technology allows the rapid creation, viewing, and modification of 3D objects on their real-world scale. Another advantage of 3D sketching in an immersive environment is the

Jo

recognition of spatiality and spatial thinking [43]. Although Israel et al. [43] identified that 3D sketching has no clear benefit compared with 2D sketching in terms of creativity, aesthetics, overall quality, and abstraction level, 3D sketching users exhibited more interactivity toward their drawings, such as sitting down in a sketched chair, and walking in the virtual bars. The use of immersive technology for 3D experience may assist in defining and evaluating a concept in the early design phase.

6

Journal Pre-proof

In addition to accurately reconstructing 3D models, another important aspect of sketch-based modeling is the shape transition of design elements from various views. A 2D design sketch with aesthetically pleasing front, side, and rear views is not necessarily pleasing from perspective view. Studies on sketch-based modeling focused on generating 3D curves, which forces users to consider curve transition from multiple views. For

pro of

example, ILoveSketch developed by Bae et al. [44] allows designers to draw detailed 3D curves intuitively. The proposed method assists in creating scaffold models for design exploration. In addition to ILoveSketch, True2Sketch, introduced by Xu et al. [45], not only generates 3D curves but also creates a surface though these curves, transforming the 3D sketch into a 3D model. However, users of both ILoveSketch and True2Sketch should be familiar with curve objects to create 3D curves without guided lines. Meanwhile, Grossman et al. [46] proposed a method for guiding users to create an accurate 3D curve by constructing a series of curves drawn on planes projected from multiple views. More importantly, they created an animated curve transition feature that aid users to retain and understand the relationships among views. Orbay and Kara [1] developed a geometric modeling approach for surface generation using a malleable curve network. Unlike traditional 3D CAD tools, where users receive only visual feedback when constructing surfaces, the method in [1] provides semi-automatic surface fairing and improves the curve network with frequent use. Thus, users can test various shape transitions

re-

by surface fairing based on the feature 3D curves. Previous shape transition methods allow the evaluation of the transition from various views by creating 3D curves directly. These methods require the users to be familiar with 3D curve generation, which is considerably different from conventional sketching. Moreover, previous methods allow designers to explore the transition of feature curves in a 3D environment not in terms of design elements. Evaluating the shape transition of the design mass (overall design shape) can be effective through 3D

lP

curves, but design is a combination of design elements. Therefore, car designers spend a considerable amount of time shaping detailed design elements, such as grille, headlight, and windshields, along with stylistically important profile curves. Delanoy et al. [18, 19] developed a state-of-the-art system that reconstructs a 3D model from user input perspective sketches, and the 3D model is refined as sketches are added from different views. This can aid designers to refine overall shape transitions. However, in the early design phase, checking sketch variations at the design element level is essential. It is also essential to create a 3D model of the sketch

urn a

variations to assess how seamless the shape transition is. In the early design phase, it is more important to evaluate sketch quality and potential through 3D, than to create accurate 3D model. Both these state-of-the-art methods assist designers in sketching and exploring 3D designs that are suitable for multi-view but not for exploring large volume of design variations because this requires repeated sketching. Therefore, we propose an interactive 3D model reconstruction method vital for CSS. This method implements interactive 3D model reconstruction, where the designer can determine and evaluate the shape transitions from multiple views at the design element level. Consequently, a system combining all the key characteristics of CSS, sketch-based modelling, and immersive environment can significantly improve the design process. With such a system, it is possible to make better-informed design decisions, which can ultimately lead to novel design

Jo

solutions in the early design phase.

7

Journal Pre-proof

3. 3D Computational Sketch Synthesis Framework The objective of the 3DCSS framework is to facilitate design exploration in the early design phase. Figure 1 shows the conceptual framework of 3DCSS, which consists of four major components: (1) data conversion, (2)

Figure 1

3.1. Data Conversion

lP

re-

pro of

CSS, (3) interactive 3D model reconstruction, and (4) design refinement.

urn a

The 3DCSS framework utilizes freehand sketch data drawn with pen and paper or digital tablet. To minimize the conformance of the actual design process and the proposed framework design process, data conversion uses a natural user interface for sketching as well. The sketch is converted to a digital file by Wacom

Jo

Intuos Paper, which captures the strokes of the Wacom pen as it marks with physical ink (Figure 2).

Figure 2

8

Journal Pre-proof

To digitize the sketch, it suffices to outline the design elements individually. Each time a design element is drawn, a digital toggle is activated and saved by the system as a separate file. The digitized strokes are then converted to vector file format (SVG). The saved SVG is represented as a double-layer stroke because the input

pro of

strokes were stored as outline strokes with line thickness information. We extract only the outer strokes from the center of the sketch so that they can be expressed in one continuous curve. The curve is closed by connecting the start and the end points of the strokes. All curves are drawn in a single stroke. If the distance between the starting and the ending point is large and the curve is not closed, we sketch the design element again. Finally, the digital representation of the design elements is created by transforming the vector data expressed as closed curves into a list of points.

For the car design, we used 29 design elements, ten additional design elements compared with Hyun et al. [47]. As shown in Figure 3, these elements from front (silhouette, window, hood vid, headlight, grille, and bumper), side (silhouette, window, hood vid, body vid, tail light, headlight, tail vid, and door), rear (silhouette, window, trunk, tail light, and bumper), and top (silhouette, bodyline, tail vid, headlight, hood vid, front window,

Figure 3

urn a

lP

re-

door, side window, rear window, tail light) views constitute a car design.

3.2. Computational Sketch Synthesis

Once the sketch has been converted into digital form, each design element undergoes point interpolation. Sketch synthesis is based on SSM developed by Hyun and Lee [2]. SSM allows synthesizing design alternatives based on product appearance similarity through genetic algorithms. Design alternatives are synthesized using the Fourier decomposition of design elements represented as a point list. Each design element is expressed as a series of 1500 points for accurate shape construction and generates design alternatives based on shape morphing

Jo

at the design element level. Unlike previous studies that used a fitness function to synthesize design alternatives, the present study incorporates an interactive genetic algorithm, which, in contrast to fitness-function-based genetic algorithms, uses human input as the fitness value. Such algorithms are widely employed in design synthesis [48, 49]. We used an initial population of 83 mid-sized sedans from 26 brands. Once the user inputs a sketch, the proposed system shows nine variations of the input sketch, which are random crossovers with the 83

9

Journal Pre-proof

car designs. Then, the system presents a scoring slider for each variation, and the users rate the generated design

re-

pro of

alternatives (Figure 4).

Figure 4

The slider is used to indicate whether the design outcome well reflects the design briefs using a seven-point Likert scale, 1 being “strongly disagree” and 7 being “strongly agree.” The probability of the design alternative

lP

to be selected as a parent is proportional to its fitness value (between 1 and 7). Hence, a selection pool is generated based on the ratings. In addition to the initial population of 84 designs (83 existing designs and one user input sketch), we generated a fixed number of 100 design alternatives for each generation but only showed the nine design alternatives with the highest fitness value so that they may fit in the interface. Thus, design alternatives with higher ratings will have a higher selection probability, thereby generating offspring; therefore, designs similar to their predecessor are created. The new design alternatives are then added to the population as

urn a

shown in Figure 4. Unlike conventional IGA, the computational sketch synthesis interface has a “selection button” that keeps the specific population in the next generation. With this feature, designers can keep the baseline of the sketch that they wish to explore further. If the selection button is not activated, no designs will be stored, and design alternatives will be generated based on the fitness value. Figure 5 shows the sketch synthesis process for seven generations of an input sketch. Users can always export any design alternative for further exploration. For example, the user-selected designs of gen_4_5 and gen_5_7 are identical. The user wanted to see more design variations (Figure 5), and in generation 5, the user found a satisfactory design and continued the

Jo

synthesis process until finalizing the exploration with design gen_7_1.

10

re-

pro of

Journal Pre-proof

lP

Figure 5

As shown in Figure 6, the design alternatives are synthesized by interpolating designs elements. The crossover of two designs (gen_6_2 and Lexus ES 2008) is created based on a random weighting value ranging from 0 to 100. If the weighting value is 50%, the crossover is a 50%–50% combination of gen_6_2 and Lexus ES 2008, and if the value is 0%, the crossover is identical to Lexus ES 2008. To generate one crossover design

urn a

alternative, crossing over among 29 design elements should be performed. The interpolating design elements do not necessarily come from a single design. The front grille can be from Hyundai Sonata 2012, whereas the side silhouette is from BMW 5-Series 2008. For example, the headlight of gen_7_1 in Figure 5 is a crossover of gen_6_2 and Lexus ES 2008 with 10% weighting value (Figure 6). However, the side silhouette is from Nissan Altima 2010 with 54% weighting value, and the side windshield from Honda Accord 2007 with 76 % weighting value. The gen_7_1 was synthesized with gen_6_2 and design elements from 27 different car designs. In every crossover process, 1% of the design alternatives are mutated. Mutations of design alternatives randomly replace design elements, thus creating new combinations. The proposed system also stores the rating and crossover genes for every design element. Users can analyze the synthesis process by checking the full lineage of the

Jo

DNA of the design alternatives. Most importantly, CSS is performed in parallel with interactive 3D model reconstruction (Figure 4), as described in detail in the next section. The interactive 3D model reconstruction of the synthesized sketches assists in making a better-informed design evaluation of the potential of the sketch variations.

11

pro of

Journal Pre-proof

re-

Figure 6

3.3. Interactive 3D Model Reconstruction

Through this process, a designer can interactively transform the generated 2D design alternatives into a three-dimensional model through our proposed 3D reconstruction algorithm, as well as explore for detailed shapes in the design element level. The 3D reconstruction algorithm proposed in this research has two major

lP

contributions. First, the proposed algorithm does not only convert the sketch into a 3D model, but also allows the designer to explore various forms by interactively deforming the shape. Basically, our algorithm is similar to a visual hull algorithm that computes the intersection volume by projecting four sketches. However, unlike the conventional visual hull algorithm, which cannot produce a plausible shape with only four sketches and inaccurate freehand sketches, our proposed algorithm successfully transforms the two-dimensional sketch into a three-dimensional model by shifting the viewpoint using the three principles [47]. Second, the proposed

urn a

algorithm allows the reconstruction and modification of each design element in the 3D model. In the visual hull method, the 3D model outcome is in carved voxel form, which is sliced from a single mass. For stylistic evaluation, it is essential to reconstruct 3D models at the design element level for seamless shape transition of the elements. Moreover, it is important to model the design elements as separate 3D objects to make them clear. For this purpose, the car body shape is a carved voxel, and a 3D model of the design elements is created by intersecting the projected volumes of the design element’s orthographic sketches and the body. Various shapes can be constructed by selecting the sketch type and synthesizing sketches from various views to evaluate shape transitions among 3D model variations [48]. River et al. [48] generated sharp and clear shapes by matching elements from various orthographic sketches. However, in car design, the identification and selection of an

Jo

orthographic sketch combination is particularly important because a car design element is represented as a projected profile in the car body rather than as having a separate volume, such as a headlight or grille. Accordingly, we established the following three principles for system configuration:

12

Journal Pre-proof

Principle 1 automatically adjust the height of the viewpoint so that the top and bottom boundaries of the projected volume of the front or rear sketch overlap the boundary of the projected volume of the side sketch, and the left and right boundaries of the projected volume of the front or rear sketch overlap the boundary of the projected volume of the top sketch. For both front- and rear-view sketches, the difference between the width-to-

pro of

height ratios is greater when the view is from the car center to the front, and the difference between the widthto-height ratios decreases when the view is from a slightly higher position. That is, a height adjustment method is used to maintain the ratio of each orthographic sketch but match the silhouette with the other orthographic views.

Principle 2 generates the silhouette of the entire car body with the side and top view; thus, this is closer to an orthographic view from a distance than a close-up perspective. In this process, the extruded volumes of the side and the top view sketches are firstly intersected to form the overall shape of the car, and the perspective projected volume of the front and rear sides are also intersected to a detailed shape.

Principle 3 reconstructs the shape of the car body first, and the design elements are created by the

re-

intersection of the body and the projected volume of the design elements based on the optimized viewpoint derived during the car body reconstruction process.

These principles enable the interactive readjustment of both the overall shape and the design elements to the 3D model for exploring novel design variations. As the details of each design element are critical elements that

lP

should be captured and expressed [26, 27], the proposed method is vital. Figure 7 shows the computationally synthesized sketch variations reconstructed into 3D models and their evaluation for final design selection. First, the designer adjusts the viewpoint positions of the front / rear sketch and intersects the projected volume models. Then, each design element is repeatedly generated and evaluated according to the shape transition

Jo

urn a

method until a satisfactory design outcome is reconstructed.

Figure 7

13

Journal Pre-proof

The interactive 3D model reconstruction according to the above principles is performed as follows: Step 1. FP and RP position calculation according to distance parameter values, i.e., height, field direction, and FOV.

pro of

Step 2. Swept volume creation and intersection calculation at each viewpoint. Step 3. Shape detailing of overall car body and design elements.

At the first step, the parameters for perspective projection of the front sketch and the rear sketch should be calculated based on principles 1 and 2 explained above. The perspective projection of each sketch varies depending on several parameters including viewpoint position, viewing direction, and FOV (field of view). In order to calculate all of the parameters for viewpoint position (3 dimensional vector), viewing direction (three angles known as Euler angles), and FOV, a total of seven parameters are required. However, in the case of using orthographic sketch, three parameters (one position parameter, two rotation parameters) can be ignored. Since the view positions of the front sketch and the rear sketch are not shifted to the left or right side of the car, coordinates on the side-axis can be ignored. Yaw rotation and roll rotation values on the front view can also be

re-

ignored when the sketch is symmetrical. Since the remaining four parameters are mutually influential (for example, the FOV parameter should decrease when the distance between the viewpoint and the 3D model increases), we have defined equations for the relationship between the two. We set the three equations based on the principles defined above, and when the designer adjusts a single parameter value, the other three parameters are automatically recalculated through their relationships. For intuitive operation, we have set the distance

lP

between the viewpoint and the 3D model as an adjustable parameter. Figure 8 shows how to set the coordinates to adjust the equations. As defined in principle 2, the intersection of the extruded side sketch and top sketch is set as the reference of the coordinate axis in the 3D space. The left-right of the side sketch (front-back of the car) is set to the X-axis, and the top-bottom (top-bottom of the car) is set to the Y-axis. In addition, the topbottom of the top sketch (left-right of the car) is set to the Z-axis. According to the first principle, when the height value (Y coordinate) is increased, the sketch is drawn in a direction looking down from above, and thus

urn a

the Z-axis rotation value also increases. In addition, the boundaries of the silhouette projected from the elevated viewpoint should meet the surface of the top-side intersection model reconstructed according to the second principle. We can calculate the Y-axis coordinates and the Z-axis rotation value as the X-axis coordinates 𝐹𝑃⃗ (viewpoint of front sketch) and 𝑅𝑃⃗ (viewpoint of rear sketch), which move based on the second principle. As a result, it is possible to simplify the interactive 3D model reconstruction process by automatically calculating other values (Y-axis position and Z-axis rotation value of view point, and FOV constant) through adjusting only the X-axis position of viewpoints (𝐹𝑃⃗, 𝑅𝑃⃗) as parameters. In the equation, the Z-axis rotation value and the FOV constant are denoted by theta and k, respectively. The FOV constant, k, is calculated as the projected sketch length divided by the front and rear sketch

Jo

lengths at unit projection distance. For example, if a line with a length of 10 pixels in a sketch is projected at a distance of d, the length of the projected line is 10 × k × d. Based on principle 1, we set up the equations so that the top and bottom boundaries of the front sketch and the rear sketch (𝐹𝑇𝑜𝑝, 𝐹𝐵𝑜𝑡𝑡𝑜𝑚, 𝑅𝑇𝑜𝑝, 𝑅𝐵𝑜𝑡𝑡𝑜𝑚) are projected from the 𝐹𝑃⃗ toward the side sketch to meet at the tangent point. For example, when the top boundary of the front sketch (𝐹𝑇𝑜𝑝) is projected to meet the upper tangent point (𝑆𝐹𝑃1⃗) with the side sketch, the projected

14

Journal Pre-proof

distance can be represented as cos 𝜃

𝑆𝐹𝑃1⃗. 𝑥

sin 𝜃 𝑆𝐹𝑃1⃗. 𝑦

𝐹𝑃⃗ . 𝑥

𝐹𝑃⃗ . 𝑦

through a geometric calculation (𝜃 : z-

axis rotation of viewing direction for front sketch projection). When the distance is projected, the length of the be represented by 𝐹𝑇𝑜𝑝  𝑘 

cos 𝜃 𝑆𝐹𝑃1⃗. 𝑥

𝐹𝑃⃗ . 𝑥

sin 𝜃

𝑆𝐹𝑃1⃗. 𝑦

𝐹𝑃⃗ . 𝑦

𝑆𝐹𝑃1⃗. 𝑥

𝐹𝑃⃗ . 𝑥

cos 𝜃

𝑆𝐹𝑃1⃗. 𝑦

𝐹𝑃⃗ . 𝑦 . Equation

length is found without k, it is equal to sin 𝜃

method, and the equations for the lower tangent point

(𝑆𝐹𝑃2⃗)

using the value of k. If this

and the tangent points

1 is set by this

(𝑆𝑅𝑃1⃗, 𝑆𝑅𝑃2⃗)

of the rear

pro of

𝐹𝑇𝑜𝑝 can

sketch projection are set in the same method as in Equations 2,4 and 5. The same approach can be used to set the equation with the tangent point (𝑇𝐹𝑃⃗, 𝑇𝑅𝑃⃗) for the top sketch. One thing to consider is that the distance from the top view appears to be shorter than the actual throw distance as the view angle (theta) increases. To correct this, instead of d in the equations for 𝑇𝐹𝑃⃗ and 𝑇𝑅𝑃⃗, which are shown in Equations 3 and 6. We use these

we use

six equations to approximate the six parameters (𝐹𝑃⃗. 𝑦, 𝑅𝑃⃗. 𝑦, 𝜃

,𝜃 ,𝑘 ,𝑘 )

according to two given parameters

lP

re-

(𝐹𝑃⃗. 𝑥 , 𝑅𝑃⃗. 𝑥) from the designer. To find the value one uses the root finding algorithm in the python scipy library.

urn a

Figure 8

The equation for calculating perspective parameters of the front-view sketch is as follows: 𝐹𝑇𝑜𝑝  𝑘  cos 𝜃

𝑆𝐹𝑃1⃗. 𝑥

𝐹𝐵𝑜𝑡𝑡𝑜𝑚  𝑘  cos 𝜃 



⃗.

⃗.

𝐹𝑃⃗ . 𝑥

𝑆𝐹𝑃2⃗. 𝑥

sin 𝜃

𝐹𝑃⃗ . 𝑥

𝑆𝐹𝑃1⃗. 𝑦

sin 𝜃

sin 𝜃 𝑆𝐹𝑃1⃗. 𝑥

𝐹𝑃⃗ . 𝑦

𝑆𝐹𝑃2⃗. 𝑦

𝐹𝑃⃗ . 𝑦

sin 𝜃

cos 𝜃 𝑆𝐹𝑃1⃗. 𝑦

𝐹𝑃⃗ . 𝑥

𝑆𝐹𝑃2⃗. 𝑥

𝐹𝑃⃗. 𝑦

cos 𝜃 𝑆𝐹𝑃2⃗. 𝑦

𝐹𝑃⃗ . 𝑥

(1)

𝐹𝑃⃗ . 𝑦

𝑇𝐹𝑃⃗. 𝑧

(2) (3)

Jo

The equation for calculating perspective parameters of the rear-view sketch is as follows: 𝑅𝑇𝑜𝑝  𝑘  cos 𝜃 𝑆𝑅𝑃1⃗. 𝑥 𝑅𝐵𝑜𝑡𝑡𝑜𝑚  𝑘   𝑘𝑟 

⃗.

𝑅𝑃⃗ . 𝑥

cos 𝜃 𝑆𝑅𝑃2⃗. 𝑥 ⃗.

𝑇𝑅𝑃⃗. 𝑧

sin 𝜃 𝑆𝑅𝑃1⃗. 𝑦 𝑅𝑃⃗ . 𝑥

sin 𝜃 𝑆𝑅𝑃1⃗. 𝑥

𝑅𝑃⃗ . 𝑦

sin 𝜃 𝑆𝑅𝑃2⃗. 𝑦

𝑅𝑃⃗ . 𝑦

𝑅𝑃⃗ . 𝑥

sin 𝜃 𝑆𝑅𝑃2⃗. 𝑥

cos 𝜃 𝑆𝑅𝑃1⃗. 𝑦 𝑅𝑃⃗ . 𝑥

𝑅𝑃⃗ . 𝑦

cos 𝜃 𝑆𝑅𝑃2⃗. 𝑦

(4) 𝑅𝑃⃗. 𝑦

(5) (6)

15

Journal Pre-proof

In the second step, projected volumes are created based on the viewpoint calculated perspective projection parameters of each sketch, and a 3D model is created by the intersections of the volumes (Figure 9). First, we create a closed 2D mesh based on the sketch, as shown in Figure 9(a), and a 3D projected volume by projecting the 2D mesh of each sketch, as shown in Figure 9(b). The front and rear sketches are created by perspective

pro of

projections based on the calculated perspective parameters of the sketches. The side and top sketches are created by orthographic projections. As shown in Figure 9(c), the created 3D mesh is placed and overlapped according to the viewpoint to perform the intersection operation in the order of side, top, front, and rear. Finally, we repeat

lP

re-

steps (a)–(b) to reconstruct the design elements in 3D and intersect with the car body model (Figure 9(d)).

urn a

Figure 9

As mentioned in Section 3.2, the designer evaluates the obtained 3D models based on the nine generated sketch variations. The computationally synthesized sketch variations, as in Figure 10, are simultaneously reconstructed into a 3D model, and the user can freely rotate the viewpoint with the mouse to assess the quality of the design. The overall shape of each 3D model and the shape transition of the design elements can be transformed by adjusting the front and rear sketch perspective viewpoints (Figure 10). The viewpoints can be adjusted by modifying the FP and RP X-axis coordinate values. When the viewpoint is close, the shape of the overall car body becomes more angular, and it becomes more streamlined when the viewpoint is distant. As the viewpoint approaches the car body, the design elements are also placed in a narrow space. As the viewpoint

Jo

moves further away, the placement of the design elements spreads widely and the shape grows longer. The 3D model of the design elements can be reconstructed in various shapes depending on the orthographic sketch combination. The user chooses an orthographic sketch combination for each design element, thus creating 3D model variations by various shape transitions methods (Figure 10). Figure 10 shows the 3D model variations of the headlight according to three viewpoints and three orthographic sketch combinations. The vertical axis in Figure 8 shows the 3D model reconstructed from three viewpoints at a distance of 2, 4, or 8 times the total

16

Journal Pre-proof

length of the car. The horizontal axis is the design obtained from the front sketch, side sketch, and the intersection of the front and side sketches. By iterating the above process, designers can interactively explore the

Figure 10

urn a

lP

re-

pro of

shape transition of 2D sketches into 3D models.

3.4. Design Evaluation and Refinement

Jo

In this step, the 3D model can be viewed in the virtual environment, where designers can perform a detailed

real-scale evaluation. The virtual reality headset Oculus Rift was used to evaluate the 3D models in an immersive environment (Figure 11(a)). Designers can intuitively view and transform the 3D models using the touch sensors. After the evaluation, the 3D model is used as an underlay image to refine the design by sketching over. According to Arora et al. [20], 3D models are often used for the basic layout of perspective drawing grids from a desired viewpoint before design details exploration. Designs can be embodied by sketching over the 3D

17

Journal Pre-proof

model. The proposed system offers a multi-view reference for tablet drawing. The orientation of the 3D model can be varied for design exploration, and screenshots can be exported (Figure 11 (b)). The screenshots of the

pro of

multi-view models are then used as background reference for further sketching using Wacom 27HD.

re-

Figure 11

As shown Figure 12, users can explore variations of design refinements while sketching over the underlay image. For example, Figure 12 was obtained by three different refinement variations. When the designer wants to check a variation of the refined design, orthographic sketches are drawn by the data conversion process

lP

explained in Section 3.1. Then, computational sketch synthesis and 3D model shape transition may be repeated

Jo

urn a

until a satisfactory result is obtained.

18

Figure 12

urn a

lP

re-

pro of

Journal Pre-proof

4. Implementation and Discussion

Jo

To test the proposed framework, a series of lab setting experiments and in-depth interviews were conducted. Six expert design practitioners with at least three years of professional experience in car design industry were recruited (three in-house car exterior designers in a Korean automobile manufacturer and three freelance car exterior designers). All participants were male. Car design industry is a male-dominant community. Only 1/10 of the designers are female. The experiment comprised four parts: preliminary in-depth interview, experiment condition 1, experiment condition 2, and in-depth interview. A preliminary interview was

19

Journal Pre-proof

conducted to confirm whether the proposed methodology shared characteristics with the design process in the early design phase. Experiment condition 1 tested novel design generation capability using only familiar design tools. Experiment condition 2 evaluated design capability using the proposed framework. Finally, an in-depth interview was conducted to evaluate the performance and limitations of the methodology. The total duration of

pro of

the experiments and the in-depth interviews was 3–4 h.

4.1. Apparatus

We used a series of input devices depending on the 3DCSS process. Wacom Intuos Paper was used to capture the user input sketch in digital format. Then, the Wacom 27HD Cintiq with a resolution of 2560 by 1440 with adjustable stand was used for design refinement. The stand was calibrated depending on the participants. All the devices were connected to a Linux computer with an i7-7700 CPU at 3.60 GHz, a 1080Ti GPU, and 64 GB of DDR4 RAM.

4.2. Preliminary interview

Four weeks prior to the validation experiment, expert interviews were conducted to confirm whether the

re-

proposed methodology could be adopted in the design process. The interviews were individual, and participants discussed and shared their design experience in a casual atmosphere. The participants explained the current design process and discussed what should be considered in the development of an assisting tool for the early design phase. Participants responded that during the early design phase they mostly sketch, which requires 3–8 min to complete. They sketch in quarter or orthogonal view depending on their preferences. Some designers

lP

prefer front and rear quarter views, as they reflect information on surface transitions. Others prefer orthogonal views to capture a “glamour shot,” an aesthetically pleasing view of the design. Toyota, for example, uses parts of the front and rear views together with parts of the side in the design of new product generations. Another reason for the preference for orthogonal views is that they facilitate 3D modeling. Once promising design alternatives have been sketched, they can be visualized in a 3D model so that the designs may be checked from various views. In some cases, designers create rough CAD models but usually ask in-house CAD modelers to

urn a

transform the 2D sketches into 3D models. When they finalize the design alternative, they create a clay model; otherwise, they start from the sketches again. This iterative process is time-consuming because the designer should ideate and visualize alternatives in both 2D and 3D. Despite the assistance from the CAD modeler, the quality of the design differs depending on the modeler’s expertise, as it requires considerable mental effort to derive 3D features from 2D sketches without any dimensional information. Thus, it was confirmed that the proposed framework matches the current design process, and its efficiency and effectiveness should be validated.

4.3. Experiment with Design Expert

Jo

After confirming that the proposed system can be used in the design process, we asked the experiment

subjects to participate in experiment condition 1 and 2 (Figure 13) based on two different design briefs consisting of design objectives and directions.

20

pro of

Journal Pre-proof

Figure 13.

In both briefs, the requirement was to create designs that would be in production by 2020 but with different design constraints. The objective of the design brief A was to create a new Kia Optima design that appeals to the 20s with the keywords “glamorous and luxurious” while continuing the heritage of its predecessors. The

re-

objective of the design brief B was a new Jaguar XF design that appeals to the 40s with the keywords “dynamic and ambitious” while continuing the heritage of its predecessors. For each subject, the design briefs were assigned in random order. The goal of the participants was to create novel designs that fulfill the requirements in the briefs. The subjects were told that the purpose of the study was to understand the design development process followed by car exterior designers in the early design phase. Moreover, the overview of the proposed system was briefly explained. In experiment condition 1, they could use familiar design tools including pens,

lP

pencils, and markers. Once the subjects had fully understood the provided design brief, they were given 15 to 20 min to create design alternatives. Subsequently, they were asked to create one refined design alternative within

Figure 14

urn a

15 min (Figure 14).

After experiment condition 1, the subjects filled out an evaluation survey consisting of questionnaires on

Jo

quality and satisfaction regarding the design outcome on a 7-point Likert scale, 1 being “strongly disagree” and 7 being “strongly agree.” The questionnaires were as follows: “How satisfied were you with the outcome of the task? (satisfaction with the design outcome)” “How satisfied were you with the quality of the design outcome? (quality of the design outcome).” The survey also contained questionnaires using TLX. TLX is a widely used evaluation metric developed by NASA based on the subjective and multidimensional assessment of perceived workload [52]. The questionnaires included the following: “Mental Demand—How mentally demanding was

21

Journal Pre-proof

the task?,” “Physical Demand—How physically demanding was the task?,” “Temporal Demand—How hurried or rushed was the pace of the task?,” “Performance—How successful were you in accomplishing what you were asked to do?,” “Effort—How hard did you have to work to accomplish your level of performance?,” and “Frustration—How insecure, discouraged, irritated, stressed, and annoyed were you?” The questionnaire also

pro of

used the same 7-point Likert scale as before. After a 10 to 30-min break, the subjects were asked to participate in experiment condition 2, where they sketched a design alternative in 5 min using familiar design tools based on the other design brief. Then, they used the proposed methodology to create a design that best meets the requirements of the brief in 10 to 15 min (Figure 15). It took approximately 0.1 s to generate a 2D design variation, and approximately 6 s to reconstruct nine 3D models from the 2D design. Subsequently, the subjects evaluated the design alternatives in a virtual reality environment. Again, they were asked to sketch a refined

Figure 15

urn a

lP

re-

design in quarter view using Wacom 27HD in 15 min; finally, they filled out the evaluation survey.

After the design refinement stage, the subjects filled out the evaluation survey and an additional questionnaire with the same 7-point Likert scale: “How inspirational were you with the design outcome provided by the proposed system? (How inspirational),” “How satisfied are you with the design outcome provided by the system? (How satisfied),” “How useful do you find the proposed system in the early design phase? (How useful),” and “How much more were you inspired when using the proposed systems than traditional design methods? (How inspirational compared to traditional method).” None of the participants

Jo

decided to iterate their refined sketches to the CSS after the design refinement stage. Experiment condition 1 and 2 were completed in approximately 1–1.5 h. The overall user assessment of the proposed system was positive. All participants agreed that the proposed

methodology could be inspiring and useful in design practice. They commented that the craftsmanship of the design outcome generated by the framework should be improved; however, they were satisfied with the novelty and quantity of the sketch variations. The participants said that quantity is as important as quality; therefore, the

22

Journal Pre-proof

computational synthesis reduced workload in the early design phase. Moreover, if a participant did not like a specific design element, the proposed system was able to change only that element. The evaluation of the framework is well reflected in the experimental results (Figure 16). Compared with the traditional framework, the proposed framework improves satisfaction with the design outcome (p < 0.02), quality of the design

pro of

outcome (p = 0.10), TLX-temporal demand (p = 0.10), and TLX-performance (p < 0.04) of the design process. The survey demonstrated that novel design outcomes could be produced with less temporal demand. The TLXmental demand did not exhibit significant difference between the proposed and the traditional framework. This is because the participants exerted comparable mental effort when sketching the design and selecting the design alternatives. They said that there seemed to be no significant difference in the mental demand between

traditional sketching and the proposed system because the level of concentration is equally high for choosing the variation. Furthermore, evaluating and comparing the 3D models with various shape transitions was a mentally demanding process. However, they agreed that CSS reduces physical and temporal demand compared to sketching. Subjects also responded that the proposed framework enhanced design inspiration. The participants were satisfied with the ability of the proposed system to facilitate the design ideation process by generating design variations that they could not conceive. All subjects agreed that the framework could be useful in the

re-

early design phase and responded that they had more design inspiration when using the proposed system than the traditional design methods. One of the participants commented that automatically generating the variation of

Figure 16

urn a

lP

an input sketch is similar to throwing free resources to develop novel design concepts.

4.4. In-depth Interview with Design Experts To understand the designer’s behavior better and validate the experimental results, an in-depth interview

Jo

was conducted after the experiment. The interview was conducted in a casual atmosphere, and the participants provided feedback regarding the performance and applicability of the proposed system. We identified that the computational support in the early design phase can assist three different aspects during the design process: design revaluation, design embodiment, and design decision.

23

Journal Pre-proof

4.4.1. Design Reevaluation First, the participants responded that the proposed system could assist designers in reevaluating sketches through sketch variations, thus minimizing discarded design concepts. That is, CSS and interactive 3D model reconstruction are novel features that allow designers to double-check numerous design variations before

pro of

discarding them. Participants said that despite the rough quality of the design outcome generated by the proposed methodology, 3DCSS generates inspiring design variations of the sketch. Design sketches are often discarded owing to the competitive nature of the car design industry. Sketch variations generated by 3DCSS can assist in revisiting and reevaluating previous sketches and generate novel designs by combining and reusing discarded ideas. Especially, 3D shape transition assisted in evaluating the overall shape transition, which allowed a thorough reevaluation of the design potential. The participants provided positive feedback regarding the automatic generation of design variations and 3D models. They stated that evaluating the generated design alternatives in various views inspired new designs. Previously, designers should mentally conceive 3D

variations of design elements. The designers utilize alias, 3D modelling software, or construct a clay model and a 3D scanner for quickly creating a crude 3D model to embody the design sketches. According to the

re-

participants, the quick 3D model is so crude that the only its creator can recognize it; therefore, it contains limited information, such as proportional and approximate forms. Although the 3D model is imperfect and ambiguous, it significantly assists in reducing mental stress even if the overall shape and various viewpoints are provided. In this respect, the ambiguous sketch need only be recognized in the early design phase because the designer will embody the design while sketching over time. Participant C, for example, liked the interpretation

lP

of the ambiguousness of the 3D models constructed by 3DCSS to reevaluate the potential of the initial concept: “My initial design looked different, but the computational synthesis gave sharp and torn look to several design elements. I liked the combinations, so I develop a new design (Figure 12 (b)). Sometimes, it is more helpful to create novel designs when limited information is provided rather than full information. CSS can be

urn a

applied in a similar context.”

It was also mentioned that the proposed system would be more useful if it used front and rear quarter views instead of orthogonal views because this would better capture the design. However, this is contradictory, as other participants preferred orthographic views, and the choice depends on the design. There are designs that are more aesthetically pleasing and better capture various features in orthographic view. Some argued that the quarter view best captures the surface quality of the sketch, whereas others claimed that there is insufficient information, and thus there is no difference from the orthographic view. In most cases, the curves and surfaces between front-top-side views or rear-top-side views are highly complex and thus difficult to replicate. Hence, the quarter view still carries limited information regarding complex curves. Therefore, everyone (including the

Jo

chief designer) interprets the surfaces and arrangements of design elements differently. For instance, participant B said:

“Sometimes car designers develop design concepts from competing brands. Even if a design is sketched

using an underlay image from competing brands, the clay modeler and the design chief interpret it differently, so that designs are never identical.”

24

Journal Pre-proof

All the participants agreed that 3DCSS created unintended and creative design directions, which was quite beneficial. Moreover, 3DCSS has the potential to prevent design fixations by providing novel variations on the

pro of

input sketches.

4.4.2. Design Embodiment

During the in-depth interview, we identified that in-house designers use rough 3D models of their initial design. This process is repeatedly performed until the design is finalized. Design concepts are embodied through iteration of sketching and 3D modeling. As previously mentioned, even car exterior designers, who are experts in 3D design, cannot immediately sketch accurate drawings of 3D objects. During the experiment condition 2, we observed that the participants thoroughly examined 3D models when rating the design alternatives. They checked 3D models at various viewpoint angles. During the interview, the designers responded that interactive 3D shape transition is a novel feature that facilitates the thorough evaluation of the sketch variations:

re-

“The proposed system allows me to visualize the sketches in 3D in various views. This is important because it is difficult to know exactly how my design will appear when it is reconstructed in 3D. We also create very rough 3D models (much rougher than 3D model generated by the proposed framework), and it takes a lot longer than six seconds. We ask in-house CAD modelers to create a rough model of the sketches or clay models; this

lP

takes a lot longer.”

This was a surprising comment, as the initial assumption was that designers would fully imagine the stereoscopic qualities of their sketches. In current professional car design, the initial concept is first sketched, and the design is refined through rendering to provide a pseudo-3D effect. Therefore, design colleagues, administrators, engineers, cad modelers and clients can better understand and evaluate the potential of the design. Then, the designers would create 3D and clay models to check the discrepancy between the initial

urn a

concept and the 3D design. In this respect, the designers liked interactive 3D model reconstruction, which assisted in refining the relationships among design elements for seamless shape transition. This made clear that even expert car exterior designers can benefit from interactive 3D shape transition when examining design alternatives. The interactive viewer in 3DCSS allowed users to evaluate the 3D model from various viewpoints and export screenshots of multiple views of 3D shape transition for design refinement. Participants agreed that this feature was highly useful. Imagining the 3D aspects of a design is a mentally demanding activity, and the proposed framework can reduce cognitive workload. The participants stressed the difficulties of designing 3D objects with seamless shape transition. With the proposed system, the participants were able to see the variations of 3D shape transition methods at the design element level. Checking 3D models of the sketches from various

Jo

views is quite helpful because a design should be created considering shape transitions from every view. Participant A said:

“With 3DCSS, I moved around the 3D model and embodied the design when I saw a new shape transition.

Sometimes, the shape transition works for a certain view but not for another; therefore, I made a revision. Also, I think it is important to draw sketches with the synthesized 3D model as underlay image. As it is a process of

25

Journal Pre-proof

developing new sketch variations, it seems to be similar to the matching process for my response in the answer sheet. It’s like self-verification.”

4.4.3. Design Decision

pro of

Chief designers do not always make decisions based on the rendered image. They are involved from the very beginning of the design process and discuss with designers to make effective managerial decisions. Designers present initial concept sketches and sketch variations to decision makers. This process is repeated several times before a final decision can be made, which may be revoked, and then the entire procedure is reiterated. For example, it is often desirable to check variations of a specific design element while maintaining all the other parts, or to shift the placement of certain design elements. The design decision process is iterative. It is not carried out in a “Let us stick with the design we developed so far and not come back for modification” fashion. The parts that require revision are constantly added in the course of design development, and thus modifications are made continuously. Another important aspect of design decision in the early design phase is that the trend on the release date of a new design should be predicted. Car manufacturing span is 2–5 years.

re-

Therefore, designers should be able to predict how their design would be received within that timeframe. The proposed framework can visualize design variations instantly, thus allowing the evaluation of a large number of alternatives and facilitating decision making. However, all the participants stated that choosing from the generated design alternatives cannot be a replacement for the ideation ability, because designers should have an excellent aesthetic taste to make the right choices.

lP

Interestingly, the instant generation of the 3D model assists designers to self-check the integrity of the design. The design conformance is an important aspect for design decision. The participants commented that when they sketch, they intentionally change the dimensions and perspectives of the car to make their design more attractive:

“When we (car exterior designers) sketch car designs, we tend to distort the perspective to make it look

urn a

more attractive. We tend to draw bottom elements wider and roof narrower to make it look more dynamic (Figure 17). We are often criticized for the unrealistic perspective by our design chief, but it is a widely known car designers’ behavior. The problem of “cheat” (exaggerated) sketches is that they often cannot be constructed in 3D models owing to unrealistic dimensions and surfaces. Aesthetic qualities that were initially planned often get lost. Therefore, by checking the sketched design instantly, we can better evaluate the potential of our sketches.”

Thus, exaggeration renders sketches more attractive but often results in unmanufacturable designs. However, the participants agreed that the proposed interactive 3D model reconstruction allows designers to self-

Jo

check through visual feedbacks to aid design conformance. The current trend in car design industry is that automobile manufacturers (both European and Asian) are encouraging designers to test sketches using 3D models, as this results in a more efficient iterative design refinement process and reduces design conformance. In-house designers have the capability to create 3D models but spend more time testing design alternatives through sketching. Utilizing a 3D model in the early design phase can minimize the discrepancy between the design concept and the manufacturable design. According to the participants, 3DCSS in the early design phase

26

Journal Pre-proof

cannot shorten the overall design duration but help to make better informed design decisions. As the car design process requires the emotional design aspect of styling design elements, the proposed method of providing

pro of

instant variations of 3D model can assist in improving the quality of the design outcome.

Figure 17

re-

5. Conclusion

In this study, we proposed 3DCSS, which allows effective exploration of a large design space and has the capability of efficiently evaluating 3D shape transitions at the design element level. In the early design phase, evaluating the quality and potential of design alternatives is critical. 3DCSS automatically generates variations of the user input sketch and reconstructs it into a 3D model instantly. 3DCSS utilizes car design sketch datasets with detailed design elements that can be interpolated to synthesize novel design alternatives. We also proposed

lP

an interactive 3D model reconstruction method that is suitable for computational sketch synthesis so that a large number of design alternatives may be efficiently and effectively assessed in 3D. Interactive 3D model reconstruction allows the quick evaluation of the shape transitions of 2D sketches when they are turned into 3D model. The experiment and the in-depth interviews with design experts demonstrated that the proposed system can assist in creating novel design outcomes with less cognitive workload. It was also demonstrated that the

urn a

proposed 3DCSS framework can assist in establishing relationships among design elements with seamless shape transition. It was identified that design variations can be created for each design element, and the design details can be refined for efficient decision making. This can be an essential advantage for improving the usefulness of sketch data by continuously utilizing the discarded sketches that have not been previously considered. However, the experiment participants noted that the framework should be improved in terms of craftsmanship if it is to be applied to the actual design of projects. Interestingly, despite the comments on craftsmanship improvement, the participants were satisfied with the quality of the automatically generated 3D models. They were inclined to draw an exaggerated design with unrealistic perspective because they aimed for more dramatic and dynamic designs. It was assumed that car exterior designers could sketch design alternatives with a perfect understanding

Jo

of stereoscopic qualities; however, there appeared to be a serious discrepancy between the exaggerated and the realistic model when the designs were finalized. Consequently, the participants responded that the 3D models generated by the framework are a guide to refining designs, thus reducing mental workload and working time. However, the experts participated in the experiment in two different conditions (traditional system, new system) and the new system was always used second. The participants were car designers in practice with a minimum professional experience of three years, and the tasks given in experiment condition 1 were similar to those in

27

Journal Pre-proof

their job. Despite this limitation of the possible ordering effect, the participants agreed that the proposed method can be useful in professional projects. Also, the final designs of the experiments were not assessed by external designers because the full potential of a sketch cannot be evaluated without proper presentation. According to the participants, when car designers evaluate the design sketches in the early design phase, they bring panels

pro of

filled with sketches of various levels of detail. Then, they give presentations to colleagues and the design executive officer. Communication is an important aspect in the evaluation of sketch quality in the early design phase. Thus, the designer of the concept can best evaluate the quality of the sketch.

During the course of this study, future work was identified that can improve the proposed system. First, the current version of 3DCSS concentrates on car design. However, the proposed system can be extended to general use by adding automatic design element detection. Moreover, a limitation of the study is that in order for the designers to iterate 3DCSS with the design outcome of the refinement process, the outcome should go through the data conversion process again. Both generalization ability and improved efficiency can be achieved by incorporating design element feature classification through machine learning. Secondly, implementing front and rear quarter views may facilitate determining whether sketches drawn in orthogonal views or quarter views change the quality of the design outcome and the efficiency of the design process. Thirdly, a generative

re-

adversarial network can be implemented in the interactive 3D model reconstruction section of the proposed framework for better capturing convex and concave surfaces in 3D models if it can generate shape details of the design elements. Fourthly, adding a history analysis interface can improve stylistic exploration by allowing designers to analyze the synthesis DNA of the design. With this interface, the design genes of variation processes can be analyzed, and the proportion of the initial design that was modified and developed can be

lP

evaluated for future reference. Finally, using the 3D computational sketch synthesis framework in design education may facilitate understanding how instantly generating of large volume of design concepts through intelligent design systems can influence future of design education. Funding: This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP, Ministry of Science, ICT and Future Planning) [grant number NRF-

Jo

urn a

2017R1C1B5018240].

28

Journal Pre-proof

References

Jo

urn a

lP

re-

pro of

[1] Orbay G, Kara LB. Sketch-based surface design using malleable curve networks. Comput Graph 2012; 36: 916–929. [2] Hyun KH, Lee JH. Balancing homogeneity and heterogeneity in design exploration by synthesizing novel design alternatives based on genetic algorithm and strategic styling decision. Adv Eng Inform 2018; 38: 113– 128. [3] Bae SH, Kijima R. Digital Styling for Designer: In Prospective Automotive Design. Proceedings of Virtual Systems and Multimedia; 2003, p. 546–553. [4] Bilda Z, Gero JS, Purcell T. To sketch or not to sketch? That is the question. Design stud 2006; 27: 587–613. [5] Wang CC, Yuen MM. Sketch based mesh extrusion with remeshing techniques. In: 21st Computers and Information in Engineering Conference, Paper No. DETC2001/CIE-21305, ASME, Pittsburgh, Pennsylvania; 2001. [6] Olsen L, Samavati FF, Sousa MC, Jorge JA. Sketch-based modeling: A survey. Comput Graph 2009; 33: 85–103. [7] Plimmer B, Apperley M. Computer-aided sketching to capture preliminary design. In: Australian Computer Science Communications, Australian Computer Society, Inc.; 2002, p. 9–12. [8] Tian C, Masry M, Lipson H. Physical sketching: Reconstruction and analysis of 3D objects from freehand sketches. Comput Aided Design 2009;41: 147–158. [9] Kraft B, Nagl M. Visual knowledge specification for conceptual design: Definition and tool support. Adv Eng Inf 2007;21:67–83. [10] Tang H, Lee Y, Gero J. Comparing collaborative co-located and distributed design processes in digital and traditional sketching environments: A protocol study using the function–behaviour–structure coding scheme. Des Stud 2011;32:1–29. [11] Gross MD, Do E. Ambiguous intentions: a paper-like interface for creative design. In: Proceedings of the ACM Conference on User Interface Software Technology, ACM; 1996, p.183. [12] Yuan X, Lee JH. A quantitative approach for assessment of creativity in product design. Adv Eng Inform 2014; 28: 528–541. [13] Chen SE, Parent RE. Shape averaging and its applications to industrial design. IEEE Comput Graph Appl 1989; 9: 47–54. [14] Sönmez NO. A review of the use of examples for automating architectural design tasks. Comput Aided Design 2018; 96: 13–30. [15] Wang CC, Zhang Y, Sheung H. From styling design to products fabricated by planar materials. Submitted to IEEE Comput Graph Appl; 2010. [16] Cheon SU, Kim BC, Mun D, Han S. A procedural method to exchange editable 3D data from a free-hand 2D sketch modeling system into 3D mechanical CAD systems. Computer-Aided Des 2012; 44:123–31. [17] Kazi RH, Grossman T, Cheong H, Hashemi A, Fitzmaurice GW. DreamSketch: Early Stage 3D Design Explorations with Sketching and Generative Design. In: UIST; 2017, p. 401–414. [18] Delanoy J, Aubry M, Isola P, Efros AA, Bousseau A. 3D Sketching using Multi-View Deep Volumetric Prediction. Proc ACM Comput Graph Interact Tech 2018; 1: 21. [19] Delanoy J, Coeurjolly D, Lachaud JO, Bousseau A. Combining Voxel and Normal Predictions for MultiView 3D Sketching. Comput Graph 2019. [20] Arora R, Habib Kazi R, Grossman T, Fitzmaurice G, Singh K. SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ACM; 2018, p. 185. [21] Obendorf H. Minimalism: designing simplicity, Springer Science & Business Media; 2009. [22] Müller F, Pache M, Lindemann U. Digital free-hand sketching in 3D-a tool for early design phases, in: DS 31: Proceedings of ICED 03, the 14th International Conference on Engineering Design, Stockholm; 2003. [23] Kavakli M, Gero JS. Sketching as mental imagery processing. Design Stud, 22 (2001) 347-364. [24] Lo CH, Ko YC, Hsiao SW. A study that applies aesthetic theory and genetic algorithms to product form optimization. Adv Eng Inform 2015;29: 662–679. [25] Zaman L, Stuerzlinger W, Neugebauer C, Woodbury R, Elkhaldi M, Shireen N, Terry M. Gem-ni: A system for creating and managing alternatives in generative design. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, ACM; 2015, p. 1201–1210. [26] Matejka J, Glueck M, Bradner E, Hashemi A, Grossman T, Fitzmaurice G. Dream lens: Exploration and visualization of large-scale generative design datasets. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ACM; 2018, p. 369. [27] Ha D, Eck D. A neural representation of sketch drawings, arXiv preprint arXiv:1704.03477; 2017.

29

Journal Pre-proof

Jo

urn a

lP

re-

pro of

[28] Lee YJ, Zitnick CL, Cohen MF. Shadowdraw: real-time user guidance for freehand drawing. In: ACM Transactions on Graphics; 2011, p. 27. [29] Ostroff D. THE DETAILS ARE NOT THE DETAILS . . . Eames Office; 2014. [30] Rosenfield K, Rams D. 10 Principles of “Good Design”. Archdaily; 2012. [31] Kwok TH, ZhangYQ, Wang CC, Liu YJ, Tang K. Styling evolution for tight-fitting garments. IEEE Trans Vis Comput Graph 2015; 22: 1580–1591. [32] Arora R, Darolia I, Namboodiri VP, Singh K, Bousseau A. SketchSoup: Exploratory Ideation Using Design Sketches. In: Computer Graphics Forum, Wiley Online Library; 2017, p. 302–312. [33] Do EYL. Drawing marks, acts, and reacts: Toward a computational sketching interface for architectural design. AI EDAM 2002; 16: 149–171. [34] Cheon SU, Han S. A template-based reconstruction of plane-symmetric 3D models from freehand sketches. Comput Aided Design 2008;40: 975–986. [35] Schweikardt E, Gross MD Digital Clay: deriving digital models from freehand sketches. Automat Constr 2000; 9: 107–115. [36] Masry M, Lipson H. A sketch-based interface for iterative design and analysis of 3d objects. In: ACM SIGGRAPH courses; 2007, p. 31. [37] Nam S, Chai Y. SPACESKETCH: Shape modeling with 3D meshes and control curves in stereoscopic environments. Comput Graph 2012; 36: 526–533. [38] Grossman T, Balakrishnan R, Kurtenbach G, Fitzmaurice G, Khan A, Buxton B. Interaction techniques for 3D modeling on large displays. In: Proceedings of the 2001 symposium on Interactive 3D graphics, ACM; 2001, p. 17–23. [39] Chandrasegaran SK, Ramani K, Sriram RD, Horváth I, Bernard A, Harik RF, Gao W. The evolution, challenges, and future of knowledge representation in product design systems. Comput Aided Design 2013; 45: 204–228. [40] Fuge M, Yumer ME, Orbay G, Kara LB. Conceptual design and modification of freeform surfaces using dual shape representations in augmented reality environments. Comput Aided Design 2012; 44: 1020–1032. [41] Jackson B, Keefe DF. Lift-off: Using reference imagery and freehand sketching to create 3d models in vr. IEEE Trans Vis Comput Graph 2016; 22: 1442–1451. [42] Keefe D, Zeleznik R, Laidlaw D. Drawing on air: Input techniques for controlled 3D line illustration. IEEE Trans Vis Comput Graph 2007; 13: 1067–1081. [43] Israel JH, Wiese E, Mateescu M, Zöllner C, Stark R. Investigating three-dimensional sketching for early conceptual design—Results from expert discussions and user studies. Comput Graph 2009; 33: 462–473. [44] Bae SH, Balakrishnan R, Singh K. ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models. In: Proceedings of the 21st annual ACM symposium on User interface software and technology, ACM; 2008, p. 151–160. [45] Xu B, Chang W, Sheffer A, Bousseau A, McCrae J, Singh K. True2Form: 3D curve networks from 2D sketches via selective regularization. ACM Trans Graph 2014; 33. [46] Grossman T, Balakrishnan R, Kurtenbach G, Fitzmaurice G, Khan A, Buxton B. Creating principal 3D curves with digital tape drawing, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM; 2002, p. 121–128. [47] Hyun KH, Lee JH, Kim M, Cho S. Style synthesis and analysis of car designs for style quantification based on product appearance similarities. Adv Eng Inform 2015; 29: 483–494. [48] Cluzel F, Yannou B, Dihlmann M. Using evolutionary design to interactively sketch car silhouettes and stimulate designer's creativity. Eng Appl Artif Intel 2012; 25: 1413–1424. [49] Kim SH, Cho SB. Application of interactive genetic algorithm to fashion design. Eng Appl Artif Intel 2000; 13: 635–644. [50] Levi Z, Gotsman C. ArtiSketch: A system for articulated sketch modeling. In: Computer Graphics Forum, Wiley Online Library; 2013, p. 235–244. [51] Rivers A, Durand F, Igarashi T. 3D modeling with silhouettes, ACM; 2010. [52] Hart SG. NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of the human factors and ergonomics society annual meeting, Sage Publications Sage CA: Los Angeles, CA; 2006, p. 904–908.

30

Journal Pre-proof

Figure captions Figure 1. Conceptual framework of 3D computational sketch synthesis

Figure 3. Stylistically significant design elements Figure 4. CSS interface and engine algorithm

pro of

Figure 2. Freehand sketch and data conversion process

Figure 5. CSS of seven generations. Each generation has nine sketch variations Figure 6. Crossover process of CSS

Figure 7. Interactive 3D model reconstruction interface and engine algorithm Figure 8. Viewpoint calibration for interactive 3D model reconstruction

Figure 9. 3D model reconstruction process: (a) triangulated sketch, (b) projected volume of sketch, (c) intersecting volumetric sketches, (d) final 3D model reconstructed from design elements.

re-

Figure 10. Interactive 3D model reconstruction interface based on controlling two parameters: (y-axis) distance between center point and viewpoint, (x-axis) combination orthogonal sketch views used. Figure 11. Design evaluation and refinement process: (a) virtual reality evaluation process, (b) design refinement process. Figure 12. Design refinement outcomes

lP

Figure 13. Study diagram of experiment condition 1 and 2

Figure 14. Design outcomes by familiar tools: (1) orthographic sketch, (2) quarter view sketch Figure 15. Design outcome by the proposed system: (1-1) computational sketch synthesis, (1-2) interactive 3D model reconstruction, (2) quarter view sketch over the 3D model.

urn a

Figure 16. Survey responses

Jo

Figure 17. Dimensions and perspectives of exaggerated and honest designs

31

Jo

urn a

lP

re-

pro of

Journal Pre-proof

Journal Pre-proof

1) A robust framework is proposed to facilitate the design exploration process in the early design phase by computationally synthesizing variations of user input sketches and interactively reconstructing 3D models. 2) A robust method is proposed to synthesize design alternatives by interpolating an input sketch with sketches in the database.

pro of

3) A novel interactive 3D model reconstruction method is developed to facilitate the shape transition of design elements so that the potential of a large number of design variations can be quickly evaluated.

Jo

urn a

lP

re-

4) We confirmed that the proposed framework enhances the reevaluation potential of design concepts and assists in making better informed design decisions.

Jo

urn a

lP

re-

pro of

Journal Pre-proof