Shrinking city layouts

Computers & Graphics 86 (2020) 15–26

Contents lists available at ScienceDirect

Computers & Graphics journal homepage: www.elsevier.com/locate/cag

Technical Section

Shrinking city layoutsR Oriol Pueyo a,∗, Albert Sabrià a, Xavier Pueyo a, Gustavo Patow a, Michael Wimmer b a b

University of Girona. ViRVIG. Dept. IMAE Girona Montilivi 17003, Spain TU Wien. Institute of Visual Computing and Human-Centered Technology, Austria

a r t i c l e

i n f o

Article history: Received 12 August 2019 Revised 7 October 2019 Accepted 15 November 2019 Available online 22 November 2019 Keywords: Layout simplification Urban models Resizing Computer graphics

a b s t r a c t One important use of realistic city environments is in the video game industry. When a company works on a game whose action occurs in a real-world environment, a team of designers usually creates a simplified model of the real city. In particular, the resulting city is desired to be smaller in extent to increase playability and fun, avoiding long walks and “boring” neighborhoods. This is manual work, usually started from scratch, where the first step is to take the original city map as input, and from it create the street network of the final city, removing insignificant streets and bringing important places closer together in the process. This first draft of the city street network is like a kind of skeleton with the most important places connected, from which the artist can (and should) start working until the desired result is obtained. In this paper, we propose a solution to automatically generate such a first simplified street network draft. This is achieved by using the well-established seam-carving technique applied to a skeleton of the city layout, built with the important landmarks and streets of the city. The output that our process provides is a street network that reduces the city area as much as the designer wants, preserving landmarks and key streets, while keeping the relative positions between them. For this, we run a shrinking process that reduces the area in an irregular way, prioritizing the removal of areas of less importance. This way, we achieve a smaller city but retain the essence of the real-world one. To further help the designer, we also present an automatic filling algorithm that adds unimportant streets to the shrunken skeleton. © 2019 Elsevier Ltd. All rights reserved.

1. Introduction Many modern computer games take place in existing cities, R for instance Grand Theft Auto IV (RockStar Games ) in Manhattan R (New York), or Assassin’s Creed II (Ubisoft ) in Venice and Florence. For such games, the importance of the architectural elements is on par with character design in creating the player’s visual experience. In particular, the detailed recreation of landmarks and streets is crucial to achieving the right atmosphere. To create a synthetic model of a realistic city, designers start by drafting the street network with the essential important places of interest. In general, they start with manual 2D designs and then move to 3D modeling, texturing, illumination, etc. The majority of these designs greatly resemble the original city, but are considerably smaller. The reason is that the essential places where the action takes place may, in the real world, be too far apart, thus triggering long, boring displacements for the player. For example, in

R ∗

This article was recommended for publication Bedrich Benes. Corresponding author. E-mail address: [email protected] (O. Pueyo).

https://doi.org/10.1016/j.cag.2019.11.004 0097-8493/© 2019 Elsevier Ltd. All rights reserved.

the game Grand Theft Auto IV, the number of blocks from Central Park to the southern limit of the island was reduced from the more than fifty in the real Manhattan, to about twelve in the game. In this paper we propose an automatic technique to help designers with the city layout simplification process described. In the simplified layout, the gamer should not have to walk through dozens of too small blocks with many crossroads (a scenario that would happen in an standard regular reduction); or conversely, along a seemingly infinite street without finding any crossroad after minutes of walking. The method starts by computing a skeleton of the city composed of landmarks defined by the designer and/or taken automatically from a server. Then, the algorithm, based on seam carving, removes parts of the city that do not correspond to the landmarks. Finally, we fill the empty areas to offer the designers a first guess from which they may start their creative and scripting work (see Fig. 1). The main contribution of our work is an algorithm to automatically shrink a city layout while preserving important elements and their relative positions, as well as the original block-size distribution; hence recognisable for designers and players. Even though in the recognisability of a city, 3D elements (building facades, monuments) have a preeminent role, a suitable layout is also essential for the video-game designer to place the 3D

16

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

Fig. 1. The shrinking process on Florence center: (a) The original center with the important places (its skeleton) in red; (b) the colored areas (yellow, blue and green) without important places; (c) the city after empty space has been reduced by 36%; (d) the final shrunk city. Note how unimportant areas are smaller and important places are closer together while respecting their relative positions. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

objects and for the video gamer to complement the knowledge of his/her current position, for examle in map insets. We have applied the algorithm to several cities and have verified that the resulting shrunken cities maintain their real appearance. This is corraborated by a formal evaluation of the shape and block distribution of the shrunken cities as well as by a users evaluation. 2. Previous work In this paper we focus on simplifying street layouts for videogames. The technique we propose is not intended to design city street networks from scratch nor to generate cities with similar features as other cities [1]. Procedural city modeling [2] is particularly relevant in urban design [3–7] as well as in road modeling [8]. Actually, procedural city modeling may be seen as an inverse of simplification, as it tackles the generation of complete street networks from a rough or abstract (simple) description, whereas in this work we are interested in the simplification of a real, complex city network. With GIS and digital urban design as the catalyst, since the end of the 1990s many contributions have been published on the simplification of maps [9], following two main trends: simplification techniques and customized maps. The techniques in the former are mainly intended to improve the legibilty of networks with a global perspective. These algorithms are based on LoD trees and graph centrality measures [10–14] or information theory [15–17]. The ultimate goal of these methods is to produce a city layout that accurately represents the city in a simpler way, all the while maintaining the accuracy of the map structure, i.e., that the information remaining after simplification is accurate. In contrast, our objective is to create a recognizable city by preserving its main features, but for entertainment purposes, where accuracy is not a necessity and thus, not part of the main algorithm. In fact, some deformations of the layout are not a problem, as long as the relative positions of important landmarks are preserved, as well as the overall look of the streets. The second family of simplification techniques are targeted at generating ad hoc routes. This includes solutions for moving from a starting point to a destination point [18], routes from anywhere in a given region to a destination point within the region [19] and routes connecting multiple destination points [19,20]. This type of work is somewhat local in the sense that it focuses on the desired route and may allow a certain degree of deformation in the representation of the simplified city, with the aim of generating guidelines or sketches for people traveling through the cities. Still, the accuracy of the route information must be maintained. Our aim, however, is to produce a global approximation of the city layout with the unique constraint of keeping the relative position of the

important places, trying to preserve the essence of the entire layout, not just a limited area, and accepting a significant degree of deformation. Many resizing algorithms have been proposed in image processing, the seam-carving technique [21–23] being a major achievement for non-uniform content-aware image retargeting. This algorithm has been the seed for a number of subsequent significant contributions that face different problems in computer graphics) [24,25] and image processing [26,27]. 3. Overview The input of our algorithm is an initial street network (e.g., from freely available GIS data) plus the description of the city’s important landmarks and places. As output it produces a shrunken city street network. Our technique attempts to meet the following goals: i) the resulting city should have a surface area smaller than its real-world counterpart and ii) landmarks have to keep their geometries and relative positions within the street network. After a preprocess for cleaning the usual defects of the input data (Section 4), the first step is to create a simplified city layout that contains a subset of the input network with the most important streets preserved, which we term city skeleton (Section 5). Then follows the shrinking process itself, which is a resizing process based on the well-known seam-carving algorithm [21]. To apply seam-carving, the network skeleton has to be rasterized and, after the resizing, the algorithm will recover the vectorial nature of the network (Section 6). Even though we do not aim to produce a complete reduced city, we propose some postprocessing (Subsection 6.3) for aesthetic reasons and a first approach to automatically fill the unimportant areas (Section 7). Then, the resulting city network is presented to the artist for the finishing touches. Fig. 1 shows a visualization of the process. 4. Input To start the shrinking process, the algorithm needs an input consisting of two elements: the city map to shrink and the list of important places to preserve. Our algorithm works with open GIS data (.osm files from OpenStreetMap [28]) for the city street network, which can be obtained from many sources. The important places in the city are landmarks (buildings, parks, streets,...) or any other important area that needs to be fully preserved. These areas are specified by a polygon, like the red one in Fig. 2 (a). An important place is defined by a list of one or more nodes and way elements. Important places are grouped in a file we call.osmi (osm important places), and tagged using an importance

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

17

Fig. 2. Pipeline of the skeleton building process. (a) Important elements. (b) Indirect important streets (red) are added. (c) Connecting cluster streets (red) are added. (d) Connections of undesired fringe paths to the skeleton (red) are added. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

value impPl between 0 and 1. The contents of the.osmi file can be created manually, or automatically by running a process able to find the most important places in the city and finding their.osm element references, just like Grabler et al. [29] do. OSM, as any real-world data, is prone to inaccuracies; hence the process starts with a preliminary cleaning step. We apply different cleaning operations to obtain a robust street network graph: •

•

•

Add intersections: Detects and fixes missing intersections between ways, taking into account possible crossings through bridges. Remove empty ways: Removes ways with a void list of nodes or with only one node. Remove segments outside bounding box: Removes all nodes falling outside a defined bounding box.

5. City skeleton In this section we describe the operations performed by the algorithm to generate the city skeleton, progressing from the original full-city to a skeleton of the most important places and streets, and the best streets with which to connect them. This skeleton is created by adding streets from more to less important. The number of streets the skeleton will have is determined by a shrinking factor (shF) described later. 5.1. Adding indirect important streets In this first step, we add to the list of important places provided by the.osmi file the streets that are closely related. These new streets are those surrounding an important place (that is not a street) or connecting to it, and the streets located inside areas specified as important in the.osmi file. For example, see Fig. 2 (b) for the streets added (red) close to the Piazza del Duomo and other important landmarks and the streets (red) added inside the important area (red polygon in Fig. 2 (a)). 5.2. Selecting streets The second step is to sort all the streets in the full city (including those that are in the current list of important places resulting from Section 5.1 and take the top-ranked ones to form the skeleton. The ordering is based on the importance of the streets given by impStr ∈ [0, 1], described below. The number of streets taken from the top of the list is determined by the shrinking factor shF, provided by the designer, which specifies to what size the city should be shrunk (i.e., the percentage of streets that will be in the skeleton). The most common values are shF ∈ [0.20, 0.60]. In Fig. 3 we can see the results for shF = 0.10 (10%) and shF = 0.20 (20%). If we compare the left image in Fig. 3 with Fig. 2 (b),

Fig. 3. Selected direct and indirect important streets, with shF = 0.10 and shF = 0.20.

we will realise that for shF = 0.10 we do not have all the important places and associated streets in the skeleton. Therefore, we should use a bigger shF to guarantee that all important places and indirect important streets are in the skeleton, as with shF = 0.20 (Fig. 3, right). To establish the importance impStr of the streets, we use a heuristic measure, which can eventually be tuned according to the designer’s criteria. There are two cases to distinguish, either a street belongs to the important element list and will be assigned a value impStr ∈ [0.5, 1], or it is an unimportant one and will receive a value impStr ∈ [0, 0.5]. Therefore, streets associated with important landmarks will be selected before unimportant ones. In the first case, we compute the importance value as

impStr = 0.5 + 0.5 · impP l where impPl is the place importance (given in the OSMI file). In the case of unimportant streets, we calculate their weight by first processing all the streets to obtain the number of intersection nodes for each street (nInt), the length of each street (len), the shortest street length in the city (lenm ), the longest city street length (lenM ), and the maximum number of intersection nodes in a street (nIntM ). The street importance impStr will always result in a value between 0.05 and 0.5, and is computed as

impStr = strT + lenF + intF where strT is an associated value depending on the type of street, with strT ∈ {0.05,0.10,0.15,0.20,0.25,0.30}, lower values denote less significance and higher ones more significance, e.g., strT = 0.30 for motorways. The length factor lenF ∈ [0, 0.1] is a normalized value:

l enF = 0.1 ·

l en − l enm l enM − l enm

and the intersection factor intF ∈ [0, 0.1] is a normalized value

intF = 0.1 ·

nInt nIntM

18

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

5.3. Connecting clusters At this point in the process, it is highly probable that the streets selected so far form disconnected clusters (Fig. 2 (b)). Thus, we must find these clusters and connect them by using a few streets sourced from the list of previously sorted streets that were ranked below those at the top and which are already in the skeleton. First, we detect all the clusters using connected-component analysis on the skeleton. Then, we compute the cluster centroids, and sort them according to their distance to the city boundingbox center. In this way, we avoid the possibility of getting bad situations like starting the cluster connection process with the two most distant clusters. The following step is to select the first (i.e., closest) one, and iteratively connect each cluster in the list, trying to create a single, large aggregate cluster. This is done using a flood-fill like algorithm for which it is usually recommended to start from the centre. As several paths may be found to connect two clusters, we take the one with the highest importance value. To guarantee the new streets are as short as possible, we use the A algorithm with a specifically tailored distance function. To be consistent with the previously presented heuristic and, at the same time, to allow the algorithm to freely chose its path on the original city, we define a new distance function. This function assigns 0 to all transitions between two nodes inside the same cluster and to any segment already in the skeleton. For the other pairs of nodes (A, B), the distance function combines the inverse importance of the street segment impAB and the Euclidean distance dAB . The importance of the street segment (A, B), is defined as

impAB = impStr ·

lenAB , len

where lenAB is the length of the segment (A, B), len is the length of the street the segment belongs to and impStr is the street importance. Then, the distance function is

distAB = α ·

1 + (1 − α ) · dAB . impAB

Fig. 2 (c) shows, highlighted in red, the streets connecting the clusters after running our algorithm. 5.4. Connecting fringe paths Now we have a skeleton with a connected street network, but perhaps with some undesirable (aesthetically unpleasant) deadend streets. To correct this, if desired, we can reconnect those dead-end streets in the skeleton that were not dead-ends in the original city. We first detect such streets by finding the ending nodes with only one connected neighbor. Then, the algorithm tries to find a way through the original city streets, from the ending nodes to other nodes that are in the skeleton, but without using segments already in the skeleton. We restrict the connecting paths to have a maximum of m steps (street segments) between the ending node and the connecting one. If more than one connection is possible, then the closest one (in the A sense) is selected. See the result of this step in Fig. 2 (d) for m = 3. In Fig. 4 we show the real city (left) and the resulting skeleton (right) computed for shF = 0.15, distAB with α = 0.5 and a connecting fringe paths of degree m = 10. 6. City shrinking The actual shrinking process is a directed area-reduction algorithm with the goal of reducing the empty areas considering the content of the city by applying a non-homogeneous resizing. The algorithm is based on the well-known seam-carving technique [21]. As with shrinking, afterwards we will have to generate

Fig. 4. Original city layout (Left) and the resulting completed skeleton (Right).

a vector model of the shrunken city skeleton and be careful not to lose any crucial connectivity information. Before the rasterization, we colour with grey levels the elements, depending on their importance. Black corresponds to the most important ones and white to non important. Then we rasterize the vectorial image (Fig. 5 topleft) and invert the grey scale (Fig. 5 top-right) so that the unimportant pixels are black (0) and the most important are white (1). imp For pixel (i, j), we term this value Ei, j . 6.1. Shrinking pre-process As we might lose information in the rasterization, we generate a regular grid indexing each pixel to the original street network, i.e. each pixel corresponds to a cell in the data structure. This additional data will be used in the seam-carving process to calculate the energy function and to determine the seams to remove. For each cell, we store the following information: Distance field: Normalized shortest distance to any skeleton element. Cell skeleton nodes: List of all skeleton node identifiers whose coordinates fall inside the cell area. Cell full nodes: List of all original city node identifiers whose coordinates fall inside that cell. Original image coordinates: Coordinates (longitude, latitude) of the center of the cell in the original map. Frozen: Flag that signals that a cell cannot be added to a seam path to be removed. This happens when the cell has a skeleton node of an important non-street element or a street intersection node. 6.2. Shrinking using seam-carving The original seam-carving algorithm seamlessly resizes images by finding paths that minimize the energy in the original image (i.e., traversing smoothly varying pixels) and using these paths to remove pixels. Analogously, our algorithm shrinks the city while preserving important elements and removing unimportant areas, which are empty areas in the skeleton. Thus, we have to define an energy function suited to our goal, by using the information of the regular grid introduced in the previous subsection. Energy function. With seam carving, we compute the path of least energy within the city image, from left to right or from top to bottom. The energy function expresses the cost of removing a pixel. In our case, we use an energy function that is a linear comimp bination of how important a pixel in the city image is (Ei, j ) and the distance of the pixel to the skeleton (Ei,dist ). j

Ei, j = αe · Ei,imp + (1 − αe ) · Ei,dist j j where Ei,j is the energy function value of pixel (i, j) and α e is a blending factor to promote importance or distance. The rationale behind using the distance field is to drive the seam-carving

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

19

Fig. 5. Computation pipeline of the energy function. Top-Left: Raster image of the skeleton. Top-Right: Inverse gray scale image. Bottom-Right: Distance field. BottomLeft: Energy function.

algorithm away from important elements and towards empty areas, which have much lower energy values. In Fig. 5 we can see an example of this process and the resulting energy function image (bottom-left) by blending gray scale and distance field images, with a blending factor αe = 0.5. Seam finding. With our energy function, we can find the best seam to remove at each iteration as the one with minimum energy. This process can be done via Dijkstra’s algorithm, graph cuts or dynamic programming. Our implementation uses the latter with one modification: when computing the energy value of a pixel with the frozen flag on, it adds an infinite value to avoid going through that pixel. Dynamic programming stores the result of the sub-calculations to simplify calculating more complex results. For example, if attempting to compute a vertical seam path, top to bottom, for each pixel we compute its energy plus the energy of one of the three possible pixels above it, to obtain the one with minimum accumulated energy. When arriving at the bottom of the image, the optimal seam is the one ending with the lowest accumulated energy. In Fig. 6, top, we see the city skeleton image and, superimposed in red (Fig. 6, middle), the seams to be removed. Note that seams run over the darker pixels of the left image, which are those with smaller energy. In Fig. 6, bottom, we show the resulting shrunken image. Removing seams. To obtain a city with the desired aspect ratio, the user can decide how many seams to remove in each direction. To remove a vertical seam means to delete all the pixels that form the seam path, and shift left all the pixels to the right of the seam. Likewise for the upper pixels, when the removed seam is a horizontal one. When shifting the pixels, we also move the corresponding cells in the regular grid to keep the information of the data structure consistent. Note how all the important elements are preserved and are closer together, all the while maintaining their relative positions (Fig. 6). However, this process might result in some inconsistencies, highlighted in red circles in Fig. 6, bottomleft. These artifacts are corrected using the information of the grid data structure as described in Subsection 6.3 (Fig. 6, bottom-right). 6.3. Shrinking post-processing After running the seam-carving operator on the city skeleton image, we have to translate the result back to vectorial space, thus

Fig. 6. Seam Carving. Remove 100 × 100 seams from the skeleton image. Top: Energy function (Fig. 5). Middle: Seams to remove in red. Bottom: Shrunken skeleton image. The bottom-left image shows some inconsistencies (red circles) produced by the seam-carving operation. These artifacts are corrected in the bottom-right image.. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 7. Left: Shrunken city map with undesired sinuosity of streets in the red circles. Right: Shrunken city after the stretching process. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

obtaining the final shrunken city skeleton. To provide new coordinates to all shrunken city nodes, we process the nodes of the grid data structure. The edges do not need to be processed since all the node connectivity has been stored in the grid for later use. In this process, we decrease the original node coordinates by the number of the pixels it has been moved, times a pixel length. Note that some nodes may be removed during the seam-carving process (but not intersection street nodes), which might result in some street deformation. When a node has been removed, we update the ways sharing it by removing the node from their node lists. One possibility to correct deformations is by tagging some nodes as frozen. However, we can still get streets with undesired extra curvature, as seen in Fig. 7 (left). In these cases, we stretch the streets that were straight in the original city and are deformed in the shrunken one. The resulting city map after the stretching process has a better

20

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

Fig. 8. Left: Shrunken skeleton (black) with all city streets (grey). Right: Streets added/modified by the designer (red). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.) Fig. 10. Districts and bounding box. Left: original city skeleton. Right: shrunken skeleton. In blue one of the districts. The right image shows in magenta the convex bounding box, which is projected onto the original layout (left image magenta polygon). . (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 9. Three streets added consecutively to the empty shrunken district marked in blue in Figure 10. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

appearance with improved straightness, as we see in Fig. 7 (right), although it has lost the orthogonality of certain intersections, which is an important feature for certain cities like Barcelona, which are characterised by a strong orthogonal pattern over a large area of the city. This characteristic can be recovered by the designer with some editing, or by applying a modification process as we discuss in Section 9. 7. Filling unimportant empty areas The shrinking process results in a reduced city skeleton, which the designer will use to generate a complete city layout. This is an interactive and creative task which involves editing the layout to reach the desired final city (Fig. 8, right) by adding new streets to the empty areas, removing undesired streets or editing their shapes. For this, we offer the designer the option to add streets from the original city, as shown in Fig. 8, left (lighter gray streets). To maintain consistency, these streets have been shrunk with the skeleton energy function. To facilitate the filling process, we provide an automatic filling option. The designer may use it as a first step and will be able to edit it to fulfil the creative and script requirements. The automatic filling algorithm tries to add streets to the shrunken empty areas so that the size and distribution of the blocks approaches the block size and distribution of the original city. This criterion is taken because the player wants to feel as if they are in a local environment of the same magnitude as the real one. Otherwise, a simple scaling of the original blocks would be enough. The important elements of the final city are already in the shrunken skeleton. Therefore, the automatic filling process looks for a result which may be a rough approximation of the missing streets. The filling algorithm is applied separately to each empty area of the shrunken skeleton, which we refer to as district. A shrunken district (e.g. blue polygon of Fig. 10 right) is recursively split into two partitions using unimportant streets of the corresponding original district (e.g. blue polygon of Fig. 10 left). Obviously, these streets will also have been shrunk to fit the shrunken district. The algorithm stops when it reaches partitions of the desired block size (when the mean area of the generated partitions is of the order of the mean area of the blocks of the empty area in the original layout) or when no more valid division streets can be found. Each of these final partitions is taken as a block of the shrunken city. In

Fig. 11. Left: Adding a path starting at an inner node. Right: Adding a path starting at a district boundary node.

Fig. 9 we show the result of the three first iterations for the blue empty area in Fig. 10. To split a partition, we use an empirical approach that randomly selects one node from a list with the nodes of the boundary and the inner nodes of the partition. Then, the algorithm builds a path that connects two nodes of the boundary of the partition and includes the selected node (Fig. 11 left). If the selected node belongs to the boundary of the partition (Fig. 11 right), to build the path we have to connect it to an inner node. We assume that the block density and the node density correlate in the original layout. Thus, choosing nodes randomly will create more subdivisions and consequently, more blocks in areas with higher block density in the original layout, which is how the distribution of block sizes of the original layout is approximated. To connect the selected node to the boundary, a path is created by iteratively adding neighboring nodes. In each step, the neighbor that leads to the straightest path locally is chosen, i.e., one that maximizes the angle between incoming and outgoing edge. For an inner starting node (Fig. 11 left), two neighboring nodes with a maximum angle are chosen in the first step. For a starting node on the boundary (Fig. 11 right), a neighbor that is as orthogonal as possible to the two adjacent boundary edges is chosen. To generate the list of candidates for each partition, we have to map the nodes inside the original empty areas onto the shrunken empty areas taking into account the shrinking deformation. To deal with this, we enclose the shrunken skeleton in a convex hull (Fig. 10 (b)) and map it onto the original city (Fig. 10 (a)). Then, we map the nodes inside the original city’s empty areas onto the shrunken empty areas using a cage deformation approach [30]. On the other hand, as the empty areas have to be closed to apply the algorithm, we have to deal with eventual dead-end paths that may

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

21

Fig. 12. Filling the shrunken skeleton obtained with ShF=0.15 and ShF=0.30, and reduction of 36%.

Fig. 14. Comparison between the result of the proposed technique and a real game R ), which highlights the similarity of the result of the (Assassin’s Creed II by Ubisoft automatic technique with the one obtained with a manual procedure..

Fig. 13. Filling the shrunken skeleton obtained with different reductions of 44% and 52%, and fixed shrinking factor ShF=0.15.

appear in the periphery of the shrunken skeleton. This is done by connecting these elements to the convex hull. In Fig. 12 we show an example of the filling algorithm using different shrinking factors, ShF=0.15 and ShF=0.30 and a fixed area reduction of 36% (100x100 seams). Fig. 13 presents an example with fixed shrinking factor (ShF=0.15) and different reductions of the city area: 44% (150x100 seams) and 52% (200x100 seams). 8. Results In this section we first show some examples of our shrinking technique in action. Afterwards, we introduce the tools we have used for the formal evaluation of the accuracy of our algorithm and, finally, we present an evaluation based on both the formal approach and a set of user tests. Further results and discussion are presented in Supplemental Material. Fig. 1 summarizes our shrinking process. Note how the resulting city has a smaller area and maintains the most important elements, but places them in closer positions. Fig. 1 (a) is the full original city with the elements selected as the skeleton highlighted in red. Fig. 1 (d) shows the final city after being shrunk and edited (red lines) by the designer. In Fig. 1 (b) and (c), we can see the unimportant areas (colored) and how they have been shrunk. Important elements come closer together, as illustrated by the shortened black arrows in Fig. 1 (c). 8.1. Skeleton building and shrinking To test our technique, we have executed the algorithm for different city layouts and different configurations of the two main parameters: shrinking factor and number of seams to remove, i.e., desired reduction of the city area. Other parameters described in the paper could be tuned as well. In Fig. 15 we present some examples where we use real-world city layouts with different features. The first example (top images) was chosen because Florence is the sceR nario of a well-known video-game, Assassins Creed II (Ubisoft ). In Fig. 14 we show a sample of original and shrunken city layouts from this game (left) and the original and shrunken skeleton generated by our algorithm (right). We note that the location of the

landmarks in the manually reduced layout of the video-game (bottom left) and the location of the layout landmarks generated by our shrunking algorithm (bottom right) suffer a similar displacement, and the reductions of the empty (unimportant) areas are similar as well. Please note that the resizing process performed by different designers would also give similar results, but not identical ones. In the real game, the designer has performed other actions such as changing the orientation of some landmarks, which would as well be performed by the designer if he/she used our approach. The other examples were chosen to test the algorithm with cities of different topologies. In some cases, they help to see some weak points of our technique. We propose solutions to these weak points in our concluding discussion. We ran our algorithm in a computer with an Intel Core i77700HQ @ 2.80 Ghz CPU and 16Gb RAM memory. The computational cost of the algorithm has three main components: building the skeleton, generating the grid data structure and running seam-carving. For the examples presented in the paper, building the skeleton has a cost of about 10 s for shF ∈ [0.20, 0.60]. Generating the grid data structure from scratch may demand, in the current implementation, around 60 s. Concerning seam-carving, Avidan et al. [21,22] already pointed out the feasability of obtaining real-time performance. In effect, since it was first proposed, seamcarving has been optimized and real-time implementations have been proposed [31] including, in some cases, the use of GPUs [32]. Therefore, we may assert that our technique performs in a bit more than 1 min for the first execution and a few seconds in subsequent executions, given that the generation of the grid data structure is not repeated each time we change the shrinking parameters (shF and seams). If it were necessary to change the resolution of the grid more often, spatial coherence could be used to minimise the effect of recomputing the grid. The current implementation of the other steps is intended to verify the feasibility of our technique and are not optimal. Please see Section 9 for a discussion on possible improvements.

8.2. Evaluation framework In order to validate our algorithm, we have run several experiments and analyzed the behaviour of two parameters, the shape of the shrunken city layout and the size and the distribution of its blocks. The former evaluates the global confidence of the result, while the latter evaluates the local confidence.

22

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

Fig. 15. Examples of the skeleton generation and the shrinking results for different cities and values of shF and reduction.

Shape. We define the shape of a city layout through a set of points representing its important elements (Fig. 16 top). To evaluate the similarity between the shapes of the real and the shrunken cities, we use the shape context technique [33,34], which has shown robust behaviour when comparing shapes in many application fields. The shape-context technique considers the vectors originating from a point p of the shape to all the other points of the shape and expresses the appearance of the entire shape relative to the reference point p (Fig. 16, middle). This appearance relative to p is then described by a histogram of the relative coordinates of the remaining points. In the bottom image of Fig. 16, we show a zoom of the set of points mapped to a log-polar coordinate system for point p. Then the number of points mapped to each bin is stored on an array (Fig. 17) that represents the histogram hp . Thus, to evaluate the similarity of two shapes with respect to p, we have to compute p p the histograms hr and hs for the real and the shrunken city and, by comparing them, we will have a measure of the shapes’ simi-

larity relative to p. Afterwards, we obtain the global similarity of the two layouts by repeating this operation for each point of the shape. p p To compare hr and hs , we compute the error ep given by the χ 2 distance between the two histograms as suggested in [33,34], i.e.,

ep =

1  (hrp (i ) − hsp (i ))2 2 hrp (i ) + hsp (i ) ∀i

(1)

where i denotes the bins of the histogram. p We have observed that for relevant differences between hr and p hs , the distance ep moves in the range [0.4,1.0], while for histograms with the same (or close) distribution of non-empty elements but with different values, ep moves in the range [0.0,0.4]. To obtain a global measure (e p ), we repeat the previous process for all the n points of the shape and we aggregate this information using

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

23

Fig. 18. Top: Subdivision of the original city with its distribution histograms dhr . Bottom: Subdivision of the shrunken city with its distribution histogram dhs .

Fig. 16. Top: Points defining the shape of the city layout. Middle: Vectors that characterise the appearance of the city relative to p. Bottom: Populated area of the diagram zoomed.

the mean error and its standard deviation.

ds =

1  (dhr (i ) − dhs (i ))2 2 dhr (i ) + dhs (i )

(2)

∀i

Block distribution. To evaluate the similarity of the resulting blocks in the shrunken filled city, we compare the mean area of the blocks and their distribution among the city with the ones in the real city. To estimate the distribution, we use a uniform subdivision of the layout and we compute the number of blocks in each cell, generating the two distribution histograms dhr and dhs (see Figure 18). To quantify the similarity of dhr and dhs we use the χ 2 distance between the two histograms. Hence we estimate the distribution similarity ds as In analysing the behaviour of this metric, we have concluded that for values of ds under 0.10 the distribution of blocks is useful and values under 0.05 are really good. Note that in the diagrams of Figure 19 (right), the dissimilarity axis only shows the rank of useful values ds < 0.10, with 1.0 being the maximum dissimilarity value. 8.3. Evaluation In this subsection we present the results for the described metrics for the center of Florence and Barcelona (Fig. 19), and we discuss those of Florence. In order to keep the paper in a resonable size, we do not discuss here the figures for Barcelona. These fig-

ures are similar to the ones of Florence, except for Barcelona’s (green) first dissimilarity value in the top-right graph of Fig. 19. Even though this value is under the threshold value of acceptable dissimilarities, it may catch the reader’s attention. This deviation with respect to the other marks of the graph may be attributed to the fact that it corresponds to a very low value of the shF, i.e., where the process starts with very little input information. In the supplementary material we provide further discussion and results. In order to validate our technique, we perform two sets of experiments: •

•

Fixed number of seams (area reduction) with varying shrinking factors. Fixed shrinking factor with varying numbers of seams.

Taking several shrinking factors in the range [0.15,0.60] and a fixed city area reduction of 36% (100x100 seams), we show the behaviour of e p in Fig. 19, top-left. The results depicted in this figure (blue marks) show that e p is stable and takes very small values e p ∈ [0.028, 0.095]. Hence, we can state that our technique provides good results even when starting with a simple skeleton of the real city (small shF). For the same values of the shrinking factor and the city reduction, Fig. 19, top-left, also shows (red marks) that the filling step provides blocks for the shrunken city with a mean area close to the mean area of the real city, over 80%. As for the distribution of the blocks, we obtain dissimilarity values in the interval [0.006,0.026] (Fig. 19, top-right). Hence, the proposed filling algorithm satisfies very well the constraints demanded. In Fig. 19, bottom-left, we present the results of the experiments performed with a fixed shrinking factor (shF=0.20), changing the area reduction (number of seams) in the range [36.0%, 70.0%] ([100x100,250x250]). The error e p takes values in the inter-

Fig. 17. Histogram corresponding to the polar diagram of Fig. 16, bottom. Each element of the array has the value of the number of points in the corresponding bin of the polar diagram.

24

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

Fig. 19. Evaluation results. Florence, blue. Barcelona, green. Left top image: Plot of the e p (bottom curves) and the block area ratio (top curves), for several values of shF and a fixed city area reduction of 36.0% for Florence and 44% for Barcelona. Left bottom image: Plot of the e p (bottom curves) and the block area ratio (top curves), for several values of the city area reduction and a fixed shF = 0.20. Right top image: Plot of the block distribution dissimilarity, for several values of the shF and a fixed city area reduction of 36.0% for Florence and 44% for Barcelona. Right bottom: Plot of the block distribution dissimilarity, for several values of the city area reduction and a fixed shF = 0.20. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

val [0.04,0.16], which are again in the low band of the acceptable range. Fig. 19, bottom-left, also shows how the ratio of the area of the blocks in the shrunken city slightly decreases for areareduction < 70%, keeping the block area ratio over 80%. Then it falls down, but maintains good block area ratios. The behaviour of the block distribution is shown in Fig. 19, bottom-right, where we see that the similarity is very high with ds ∈ [0.017, 0.05]. Therefore, we may assert that our filling algorithm is also performant for this second set of experiments. From these results, we can conclude that the proposed shrinking technique satisfies the main requirement stated to reach our goal, i.e., the shrunken cities maintain shapes highly similar to those of the original cities. In addition, the distribution of blocks in the shrunken cities is very close to that of the real cities, and the magnitude of the block areas is of the same order in both the original and the shrunken cities. To confirm the reliability of the formal evaluation of the shrinking process, we have performed a user evaluation. We designed an experiment to compare our results with the results obtained manually by 11 users. Each user has performed four exercises; for two cities of different topology and two different reductions for each city: Florence (44%, 125x125 seams and 51%, 125x175 seams) and Girona (44%, 125x125 seams and 58%, 175x175 seams). Hence, we performed a total of 44 experiments. Then we have compared the forty-four manually shrunken layouts with the results of our shrinking algorithm. To do so, we have computed the errors of the manually shrunk layouts with the original one using the framework presented in Section 8.2. In Fig. 20 we show these errors (green points), the error of the result provided by our algotithm (blue) and the threshold (red) under which the error is considered acceptable, as we discussed in Section 8.2. For the sake of clarity, we have cut the vertical axes at 0.6 even though the error value interval is [0.0, 1.0]. We appreciate in this figure that our algorithm

provides a solution very close to the solutions obtained in a manual process that required approximately ten to fifteen minutes for each exercise instead of a few seconds. Please note that this experiment concerns only the shrinking phase, the central part of our paper, not the complete process. 9. Discussion and future work We have proposed an innovative technique to provide videogame designers with a shrunken layout containing the important elements of a real city without losing the city’s essential appearance. The results and evaluation presented in Section 8 demonstrate the performance of our method. We have seen that our technique provides good results, especially for the most useful values of the main parameters shF ∈ [0.20, 0.60] and area-reduction ∈ [30.0%, 70.0%]. We have used an energy function (Section 6.2) using the distance field to drive the seam-carving algorithm away from important elements and towards empty areas, other functions may be implemented for other criteria as well. For example, the orography of the city could be taken into account by considering the terrain slope in the energy function, using the contour lines of the topographic map of the city. In the current implementation, the number of horizontal and vertical seams to be removed is fixed by the user. In the future, this could be determined automatically. We see several ways to explore this. One would be based on using the shape of the skeleton, while another would be to use spatial coherence, e.g., using bundles of seams. It would also be interesting to apply the multisize approach proposed by Avidan and Shamir [21] to offer several skeleton reductions at once, in realtime, to the designer before the filling process. As seen in the Barcelona results, cities with strong street patterns can have some deformation even after the stretching process.

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

25

Fig. 20. Users test errors compared with our algotihm’s error and the error threshold. Florence: Top-left 44%, 125x125 seams. Top-right 51%, 125x175 seams. Girona: Bottomleft 44%, 125x125 seams. Bottom-right 58%, 175x175 seams.

Fig. 21. Top: Original and shrunken city skeletons. The filled red circle shows undesired sinuosities. Bottom-left: Skeleton after a 45◦ rotation. Bottom-right: Shrunken city skeleton rotated back to its original orientation. The empty red circle shows how the undesired sinuosities of the top-right image have been corrected, while the sinuosity has been transferred to another street with less impact (filled red circle.). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

This may be corrected with adapted beautification techniques like the ones proposed in Xu et al. [35] and Galan et al. [36] or, following the suggestion of Avidan and Shamir [21], higher-level cues could be added. Another way to correct undesired “sinuosities” in cities with some specific strong patterns like the one of Barcelona is to rotate the whole skeleton before running seam carving and rotate the shrunken city back to the original orientation. See Fig. 21 where red circles show how the undesired deformation is translated to other streets where it is less significant.

Our automatic filling algorithm does not provide an accurate final solution, but rather an initial suggestion from which the designer may start the next step. This first approximation may have some undesired deformations in the unimportant areas. Even though these problems may sometimes be acceptable, since even well-known commercial games come with a low fidelity in unimportant zones, they could be corrected by adapting existing tools for procedural urban design like those proposed by Chen et al. [3], Aliaga et al. [37] and Nishida et al. [7], based on patterns of the real city. Another alternative that could be analysed to fill the empty unimportant areas would be the use of Generative Adversarial Networks (GAN). GANs have recently been applied to generate urban areas based on the style of other layouts [38,39]. This could provide well-suited synthetic networks, as procedural techniques do, keeping the style of the original city in the empty shrunken areas. Our proposed shrinking technique can be seen as a more general simplification algorithm that can be used in other applications where some loss of unimportant information is acceptable. Some algorithms [18–20] highlight useful streets while removing superfluous ones. This often generates empty areas, which could be reduced using our algorithm. The same goes for general-purpose simplification algorithms [10–12]. On the other hand, the skeleton generated in the first stages of our algorithm can be used to give a starting point for procedural processes; in particular, for procedural techniques that design cities reproducing patterns of real cities, so that its not needed to start from scratch and they can apply their abilities on top of a reduced area but recognizable real-world city skeleton. One avenue of future work is to investigate whether a purely vector-based algorithm can be used for the city-shrinking step (Section 6). Another relevant open problem is to generate a 3D shrunken city built on top of the shrunken layout. To do so, we would have to explore the feasibility of adapting techniques like the ones proposed by Grabler et al. [29] and Qu et al. [24], where important buildings are highlighted and unimportant ones are simplified.

26

O. Pueyo, A. Sabrià and X. Pueyo et al. / Computers & Graphics 86 (2020) 15–26

Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

[17]

[18]

Acknowledgments [19]

work was partially funded by project TIN2017-88515-C2-2-R of the Ministerio de Ciencia, Innovación y Universidades, Spain. We want to thank our colleagues Santi Thió and Xavier Lladó who assited us in the evaluation phase of our work. We want also express our gratitude to the users who have invested a part of their precious time helping us in the user tests to validate the results of our algorithm.

[20] [21] [22] [23] [24]

Supplementary material Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.cag.2019.11.004.

[25]

[26]

References [1] Musialski P, Wonka P, Aliaga D, Wimmer M, van Gool L, Purgathofer W. A survey of urban reconstruction. Computer Graphics Forum 2013;32(6):146–77. [2] Smelik RM, Tutenel T, Bidarra R, Benes B. A survey on procedural modelling for virtual worlds. Comput Graph Forum 2014;33(6):31–50. doi:10.1111/cgf.12276. [3] Chen G, Esch G, Wonka P, Müller P, Zhang E. Interactive procedural street modeling.. ACM Transactions on Graphics (SIGGRAPH) 2008;27(3):103:1–103:10. [4] Lipp M, Scherzer D, Wonka P, Wimmer M. Interactive modeling of city layouts using layers of procedural content. Computer Graphics Forum (EUROGRAPHICS) 2011;30(2):345–54. [5] Aliaga DG, Vanegas CA, Beneš B. Interactive example-based urban layout synthesis. ACM Transactions on Graphics (SIGGRAPH Asia) 2008;27(5):1–10. [6] Peng C-H, Yang Y-L, Wonka P. Computing layouts with deformable templates. ACM Trans Graph 2014;33(4):99:1–99:11. doi:10.1145/2601097.2601164. [7] Nishida G, Garcia-Dorado I, Aliaga DG. Example-driven procedural urban roads. Computer Graphics Forum 2015:1–14. doi:10.1111/cgf.12728. [8] Galin E, Peytavie A, Guérin E, Marechal N. Procedural generation of roads. Computer Graphics Forum 2010;29:429–38. [9] Pueyo O, Pueyo X, Patow G. An overview of generalization techniques for street networks. Graphical Models 2019:101049. doi:10.1016/j.gmod.2019.101049. [10] Edwardes A, Mackaness W. Intelligent generalisation of urban road networks. In: Proceedings of GISRUK 20 0 0 Conference; 20 0 0. p. 81–5. [11] Edwardes A, Mackaness W. Road network simplification in urban areas. Computers, Environment and Urban Systems 2005. [12] Edwardes A, Regnauld N. Preserving the pattern of density in urban network simplification. In: Proceedings of GIScience 20 0 0; 20 0 0. p. 104–5. [13] Chang R, Butkiewicz T, Ziemkiewicz C, Wartell Z, Pollard N, Ribarsky W. Hierarchical simplification of city models to maintain urban legibility. ACM SIGGRAPH 2006 Sketches. ACM; 2006. [14] Chang R, Butkiewicz T, C Z, Wartell Z, Pollard N, Ribarsky W. Legible simplification of textured urban models. IEEE Computer Graphics and Applications 2008;28(3):27–36. [15] Li Z, Huang P. Quantitative measures for spatial information of maps. International Journal of Geographical Information Science 2002;16(7):699–709. [16] Bjørke JT, Myklebust I. Map generalization: Information theoretic approach to feature elimination. In: Proceedings of ScanGIS’2001, 8th Scandinavian

[27]

[28] [29] [30] [31] [32]

[33]

[34]

[35]

[36] [37] [38]

[39]

Research Conference on Geographical Information Science. Also, Proceedings of ICC97, 18th International Cartographic Conference pp.480–486; 2001. p. 203–11. Bjørke JT, Isaksen E. Map generalization of road networks. Case study from Norwegian small scale maps. In: Proceedings XXII International Cartographic Conference; 2005. Agrawala M, Stolte C. Rendering effective route maps: improving usability through generalization. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ACM; 2001. p. 241–9. Kopf J, Agrawala M, Bargeron D, Salesin D, Cohen M. Automatic generation of destination maps. ACM Transactions on Graphics (SIGGRAPH Asia) 2010;29(6):158:1–158:12. Birsak M, Musialski P, Wonka P, Wimmer M. Automatic generation of tourist brochures. Computer Graphics Forum (EUROGRAPHICS) 2014;33(2):449–58. Avidan S, Shamir A. Seam carving for content-aware image resizing. ACM Transactions on Graphics (SIGGRAPH) 2007;26(3):10. Rubinstein M, Shamir A, Avidan S. Improved seam carving for video retargeting. ACM Trans Graph 2008;27(3):16:1–16:9. doi:10.1145/1360612.1360615. Shamir A, Avidan S. Seam carving for media retargeting. Commun ACM 2009;52(1):77–85. doi:10.1145/1435417.1435437. Qu H, Wang H, Cui W, Wu Y, Chan M-Y. Focus+ context route zooming and information overlay in 3D urban environments. IEEE Transactions on Visualization and Computer Graphics 2009;15(6):1547–54. Emilien A, Vimont U, Cani M-P, Poulin P, Beneš B. Worldbrush: interactive example-based synthesis of procedural virtual worlds.. ACM Transactions on Graphics (SIGGRAPH) 2015;34(4):106:1–106:11. Daw-Sen Hwang, Shao-Yi Chien. Content-aware image resizing using perceptual seam carving with human attention model. In: 2008 IEEE International Conference on Multimedia and Expo; 2008. p. 1029–32. doi:10.1109/ICME. 2008.4607613. Domingues D, Alahi A, Vandergheynst P. Stream carving: An adaptive seam carving algorithm. In: 2010 IEEE International Conference on Image Processing; 2010. p. 901–4. doi:10.1109/ICIP.2010.5653984. Foundation O. OpenStreetMap. 2011. URL http://wiki.openstreetmap.org. Grabler F, Agrawala M, Sumner RW, Pauly M. Automatic generation of tourist maps. ACM Transactions on Graphics (SIGGRAPH) 20 08;27(3):10 0:1–10 0:11. Nieto J, Susin A. Cage based deformations: a survey.. Deformation Models Lecture Notes in Computational Vision and Biomechanics 2013;7. Huang HFu T, Rosin PLQi C. Real-time content-aware image resizing. Science in China Series F: Information Sciences 2009. Kim I, Zhai J, Li Y, Chen W. Optimizing seam carving on multi-gpu systems for real-time content-aware image resizing. J Supercomput 2015;71(9):3500– 24. doi:10.1007/s11227-015-1446-4. Belongie S, Malik J. Matching with shape contexts. In: Proceedings of the IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL’00). Washington, DC, USA: IEEE Computer Society; 20 0 0 http://dl.acm.org/citation. cfm?id=791222.791946. ISBN 0-7695-0695-X. 20–. Belongie S, Malik J, Puzicha J. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002;24(4):509–22. Xu P, Fu H, Igarashi T, Tai C-L. Global beautification of layouts with interactive ambiguity resolution. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology. New York, NY, USA: ACM; 2014. p. 243–52. ISBN 978-1-4503-3069-5. doi:10.1145/2642918.2647398. Galán SF, Mengshoel OJ. Neighborhood beautification: graph layout through message passing. Journal of Visual Languages and Computing 2018;44:72–88. Aliaga DG, Beneš B, Vanegas CA, Andrysco N. Interactive reconfiguration of urban layouts. IEEE Computer Graphics and Applications 2008;28(3):38–47. Hartmann S, Weinmann M, Wessel R, Klein R. Streetgan: Towards road network synthesis with generative adversarial networks. In: International Conference on Computer Graphics, Visualization and Computer Vision; 2017. Albert A, Strano E, Kaur J, Gonzlez M. Modeling urbanization patterns with generative adversarial networks. In: IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium; 2018. p. 2095–8. doi:10.1109/ IGARSS.2018.8518032.