- Email: [email protected]
New advances in obtaining three-dimensional models from conical perspectives
Advances in Engineering Software 98 (2016) 38–57
Contents lists available at ScienceDirect
Advances in Engineering Software journal homepage: www.elsevier.com/locate/advengsoft
New advances in obtaining three-dimensional models from conical perspectives J. Conesa∗, F. Mula Graphical Expression Department, Universidad Politécnica de Cartagena, Campus Muralla del Mar, C/ Doctor Fleming s/n, Cartagena 30202, Spain
a r t i c l e
i n f o
Article history: Received 30 December 2015 Revised 8 February 2016 Accepted 29 February 2016
Keywords: Image reconstruction Single image technique Image-based modeling Conical perspective
a b s t r a c t In a previous work we presented algorithms which allow obtaining three-dimensional models from graphs which represent a projection in conical parallel perspectives and conical oblique perspectives of polyhedral models with normalon and quasi-normalon typology. In this paper the new advances that we have achieved in this field are presented, allowing increasing the set of models which can be reconstructed to other typologies different from the normalon and quasi-normalon ones. Moreover, we present a new technique which extends the previous work in order to be implemented to conical perspectives with three vanishing points, and the method proposed for the detection of the type of conical perspective represented by the graph, including the detection and subsequent reconstruction of graphs which represent a flat shape, has been improved. The results obtained on a total of 336 tests, with a success ratio of 100%, make the method a proposal to be considered for obtaining models from conical perspectives automatically. © 2016 Elsevier Ltd. All rights reserved.
1. Introduction The automatic obtaining of three-dimensional models from graphs which represent one of their projections is a field of great interest for the scientific community as evidenced in the continuous investigations carried out from the 1960s with the work done in [1] until now with the recent work published in [2]. From the approach of reconstructing models from conical perspectives, on which this work is focused, we can group them, paying attention to the technique used, in two blocks: labeling methods and perceptual methods. Labeling methods are based on a classification of the vertexes according to the directions of the edges which come together in each corner. The first valid labeling methods are presented in the work carried out in parallel in [3,4] where the vertexes of a cube were classified after considering all the corners from all possible viewpoints of an axonometric perspective. From this work, new methods were developed in order to improve and expand the procedure of labeling [5,6], but all of them were based on heuristic rules and restrictions were presented. In [7] a new labeling scheme of drawings, including nonnecessarily polygonal objects, was developed. However, the pro-
∗ Corresponding author at: Graphical Expression Department, Universidad Politécnica de Cartagena, Campus Muralla del Mar, C/Doctor Fleming s/n, 30202 Cartagena, Spain. Tel.: +34 968 32 64 77; fax: +34 968 32 64 74. E-mail address: [email protected], [email protected] (J. Conesa).
http://dx.doi.org/10.1016/j.advengsoft.2016.02.012 0965-9978/© 2016 Elsevier Ltd. All rights reserved.
cedure allows labeling the same drawing in more than one way, which contrasts with human perception in which a projection has very few ways of dimensional interpretation. The latest contributions in this line are the labeling methods as shown in [8,9] where the geometry of the hidden parts can be established, and where the front model geometry can be defined. A reference in the reconstruction of models from conical perspectives based on labeling models is the work presented in [10] where the quantitative reconstruction of the 3D structure of a scene from a line drawing, by using the geometrical constraints provided by the location of vanishing points, is investigated. In this work the design of an algorithm, which has several advantages with respect to the usual approach based on a reduction to linear programming, is presented. These advantages range from a lower computational complexity to error tolerance and exact reconstruction of the 3D-geometry of the objects. The perceptual methods are characterized by trying to implement, through the sequential language of computers, the way humans perceive. In [11] the first perceptual method is designed based on the proportionality between the projection and the threedimensional model, starting from a labeling method that allows us to obtain a graph of adjacency to define the orientation of each edge regarding the main axes. In their algorithm, the parallel lines of the projection are parallel in the model and the parallel edges to the main axes are drawn proportional to the actual dimensions, thus allowing us to define parallelograms with faces parallel to the planes of projection. Although the method allowed flexibility to inaccuracies in the drawing, the algorithm is characterized by a high
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
degree of user interaction needed to designate the principal axes (intersection of the planes of projection). Furthermore, the method was very limited since rebuilding models should not contain hidden edges given the ambiguities that the representation of these edges causes. The other methods have been based on perceptual optimization methods. It is considered that such methods are the mathematical process that more closely resembles the way humans interpret a projection. The strategy called “inflation”, applied to an axonometric projection of a model, works assigning coordinates “Z” to vertexes of a projection while maintaining its coordinates (x, y) by minimizing an objective function. The function to be optimized has been modified and extended by different authors. In [12] the objective function is formed by a single component, the MSDA (minimum standard deviation of angles) in [13,14] the mentioned objective function was expanded with the DP (deviation from flatness of the faces of a model) and [15–18] propose an objective function composed of a sum of subfunctions representing regularities of the projection, i.e. characteristics of the model that can be deduced from the observation of the projection. However, the reconstruction algorithms based on optimization present common difficulties: given the non-linearity of the objective functions proposed, there is a high number of local minimums, therefore the solution reached does not always represent the desired model (the own projection of the model that constitutes the starting point for the optimization is in many times a local minimum). Focused on these problems, some authors have tried to propose models which allow obtaining a tentative initial model which approaches the desired model as much as possible, so that the optimization process reaches the global minimum [19,20]. In [21] a new method based on an iterative optimization process, in which, once the reconstruction has been performed, the model obtained is projected on a plane and compared with the starting projection, is proposed. If both projections do not coincide in a preset threshold, the restrictions are readjusted and the reconstruction process is repeated. However, the computer cost of the process is not mentioned. A reference of the implementation of methods based on optimization for the reconstruction of conical perspectives can be found in [22], where the problems derived from the use of nonlinear functions in the optimization processes are highlighted. This non-linear function is owing to the fact that the flatness of the faces is usually expressed according to the vertexes which define it. In their work an objective function, defined from a set of linear restrictions, is proposed. The algorithm can process perspectives with or without representation of hidden edges, and uses the algorithms given in [23,24] to determine the faces defined in the model from its projection. Another trend, which has as its final aim the reconstruction, is focused on the division of complex models into simpler parts to later, once they are reconstructed, put them together again in order to form a single solid. However, most of the works presented from this view [25–29] need, as starting information, the faces defined in the model. In [30] a division algorithm, which does not need the previous detection of the faces, is presented, but its results are not always the expected ones. Most of the algorithms and strategies proposed for the reconstruction use as starting information the faces defined in the model. However, in spite of the great number of investigations directed towards the identification of faces from the projection of a model which have extended from the pioneering work given in [17] until more current woks like the one carried out in [31], the problem has not been solved completely. Although the existing methods can find faces in most cases, there is not a
39
mechanism to determine if the result is right, leaving this decision to the user. Therefore, the reconstruction methods that need a previous identification of the faces must be considered as semi-automatic models, and in no case it is possible to talk of automatic reconstruction of models. In the work presented in [11], and later revised in [32], an algorithm that starts with the calculation of the coordinates of a vertex on an orthogonal corner is proposed, and it is proved that, knowing these coordinates, it is possible to determine the coordinates of the rest of the vertexes through an expansion tree that uses the equations of the plane which define the faces of the model. However, the faces defined in the model are used as starting information again, it is assumed that the projection always corresponds with a conical perspective with three vanishing points, and the projections cannot contain hidden edges. Focused on improving the proposals presented by these authors, in [33] it is presented a work which allows the automatic reconstruction of polyhedral models with normalon and quasinormalon typology from parallel and oblique conical perspectives with two vanishing points which do not need the previous detection of the faces and can or cannot contain hidden edges. This paper represents an improvement of the work presented in [33]. On the one hand, the set of models that can be reconstructed is increased and they do not need to be of normalon or quasi-normalon typology any longer. On the other hand, the reconstruction method proposed widens including conical perspectives with three viewpoints. The system, completely automatic, does not require obtaining the faces of the model. 2. Previous work The work presented in [33] is focused on the reconstruction of graphs with normalon and quasi-normalon typology defined as: •
•
Normalon graph: The name is a generalization to the world of polyhedra, related to the normalon polygons concept, such as those with the property in which all angles between two concurrent edges are 90°. In Fig. 1 we can see a graph of this type since all edges converge towards three vanishing points (one improper). Quasi-normalon graph: In these graphs, despite not belonging to the normalon type, it is possible to reach all vertexes, exclusively through edges that are parallel to three directions, and which form 90º to each other. That is, they are graphs that meet the condition where the elimination of all non-parallel edges to these three directions does not imply the loss of all vertexes, also allowing them to remain connected. We name them main directions. In Fig. 1 we can see a graph of this type in which the removal of non-parallel edges in three main directions (1–2, 1– 5, 4–6, 5–7, 7–12, 5–12, 5–8, 1–8, 1–9, 2–9) does not imply the loss of vertexes and they remain connected (there is only one graph).
For the reconstruction of graphs we propose a method based on [32] with the advantage of not using, as starting information, the defined faces in the model. Knowing the three-dimensional coordinates of the vertexes of an edge of the model (edge AD in Fig. 2), it is possible to determine the coordinates of the vertexes connected to it by edges oriented according to the main directions, like the intersection of the perpendicular plane to the edge in one of its vertexes and the straight line that joins the viewpoint with the projection of the vertexes (vertexes B and C in Fig. 2). The method proposed can be expanded following the edges of the graph oriented according to the main directions until the complete reconstruction of the model. The evaluated vertexes are
40
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 1. Typologies of graphs.
Fig. 2. Method for obtaining the three-dimensional model.
added to a model list of vertexes (MLV) which establishes the path to follow in the expansion of the model. In Fig. 3 we show the expansion tree used for the reconstruction of a graph with quasinormalon typology as well as the variations in the MLV. Given the characteristics of the parallel and oblique conical projection of two vanishing points, in which the vertical edges keep this orientation in the projection, the reconstruction process can be started in one of these edges keeping the coordinates (x, y) that
its vertexes have in the projection in the model and assigning coordinate z equal to zero. 3. Reconstruction of conical perspectives with three vanishing points In this paper we show a method to expand the reconstruction process commented above in the case of conical perspectives with
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
41
Fig. 3. Expansion tree for the reconstruction of a graph with quasi-normalon typology.
B aligned according to a straight line that vanishes to VP1. Tracing a straight line on the picture plane through A parallel to VP1 VP 2, in the intersection points of this one with the straight lines traced from A and B to MP1 (points Ao, Bo) we obtain the real magnitude of the segment. Knowing the real magnitude of one of the edges of the model, and given that according to the principles of the conical perspective the vertexes of a model will converge on the straight lines that join the viewpoint with the corresponding projections of the vertexes on the picture plane, assigning coordinates Z = 0 to any of the vertexes, it will be possible to define the orientation of an edge passing through that one. Therefore, the steps to be taken for the reconstruction of graphs which represent conical perspectives with three vanishing points will be the following ones:
Fig. 4. Obtaining the real magnitudes in perspectives with three vanishing points.
three vanishing points. The problem with this is that none of the main directions of the model is parallel to the picture plane since all of them converge in three vanishing points, so not knowing the orientation of any of the edges in the model it is not possible to assign starting coordinates to their vertexes. Based on the principles of the conical perspective with three vanishing points, it can be stated that the families of edges oriented according to the main directions converge to points that constitute the vertexes of the triangle of three vanishing points. The heights traced in that triangle coincide with the projections of the axes of the trihedro and their gathering point (ortho-centre of the triangle of three vanishing points) will be the orthogonal projection of the vertex of the trihedro on the picture plane, coinciding with the main point of the perspective. Dropping one of the faces of the triangle of three vanishing points we can obtain the metric points for each one of the main directions, and we can use the metric points to measure segments which are part of the straight lines that vanish in the corresponding vanishing point. In order to further clarify these concepts, in Fig. 4 we show the way to determine the real magnitude between points A and
(1) Take an edge of the graph oriented according to one of the main directions and assign coordinates to one of its vertexes keeping the values (x,y) of the graph and setting z equal to zero. (2) Determine the coordinates of the other vertex of the edge taken in point 1 bringing on the straight line that joins the viewpoint and the projection of the vertex the real magnitude of the edge obtained from the metric point. (3) Apply the expansion method in the same way as it is applied in the cases of parallel and oblique perspectives with two vanishing points. In order to further clarify the procedure described, we analyze step by step the reconstruction of a hexahedron. Just as it is shown in Fig. 5, the process is started taking an edge of the graph oriented according to the main directions (in this case A E ) assigning coordinates to one of its vertexes (A) in the way (A x, A y, 0). Later, it is determined the real magnitude of that edge. To this end, given that the edge A E vanishes to the point VP1 , we will determine metric point MP1 and trace through A a straight line R on the picture plane parallel to the line that joins VP1 and VP2 . The real magnitude of the AE will be obtained as the distance between the point A and Eo, being this one the intersection point of the straight line R and the straight line MP1 E . Knowing the value of the real magnitude, it will be sufficient to take this distance from A on the straight line that joins the viewpoint V with E . Once the coordinates of the vertexes of the edge A E have been obtained, both vertexes are added to the MLV and the reconstruction process continues through an expansion tree whose progression is determined by the order of the vertexes in the MLV, and it
42
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
straight line that joins the viewpoint V with the projection G’ of the vertex (see Fig. 6d). 4. Analysis of the principal directions and type of conical perspective It becomes clear that the expansion of the reconstruction method through edges oriented according to the main directions needs a prior knowledge of those directions in the projection of the model. This paper means a continuation of the methods developed in [33] where the criteria followed to identify parallel and oblique perspectives with two vanishing points are expounded. That work is focused on the reconstruction of graphs that comes from the vectorization of a photograph, so it is considered that the main point (projection of the viewpoint) is centered in the frame that defines the size of the photograph. Furthermore, as much the size of the photograph as the focal length (distance from the viewpoint to the picture lane) are considered starting data since both of them are intrinsic parameters of the device used to capture the image and can be extracted from the own file of the image. The algorithm implemented to analyze the type of conical perspective starts from the criteria given [33] which, in turn, derives from the work presented in [34], to which we have introduced some modifications that mean the following improvements: •
•
Fig. 5. Start of the reconstruction process of a graph with three vanishing points.
finishes when the last vertex of the MLV is evaluated or when all the vertexes of the graph are in the MLV. The expansion tree follows the progression represented in Fig. 6: (1) Taken the first vertex of the MLV as reference vertex, vertex A’, the vertexes B’ and D’ connected to A’ by edges oriented according to the main directions are analyzed. The coordinates of the vertexes B’ and D’ are obtained as the intersection of the perpendicular plane in A to the edge AE and the straight lines VB’ and VD’ respectively. Finally, the new vertexes are added to the MLV (see Fig. 6a). (2) Taking the following vertex of the MLV, vertex E’, as reference, the coordinates of the vertexes F’ and H’ as the intersection points between the perpendicular plane to the edge AE in the vertex E and the straight lines traced from the viewpoint V to their corresponding projections on the picture plane F’ and H’ are calculated, adding the new vertexes to the MLV. The vertex A’ is not calculated from E’ since it is already in the MLV (see Fig. 6b). (3) Repeating the same process, taking now the vertex B’ as a reference, the coordinates of the vertex C as the intersection of the perpendicular plane in B to the edge BA and the straight line that joins the viewpoint V with C’ are calculated (see Fig. 6c). (4) The next vertex of the MLV, vertex D’, is not considered as a reference vertex given that coordinates of a new vertex are not determined since all the vertexes are already connected to it, vertexes C’ and H’, in the LVC. (5) The last vertex to be considered as a reference is F’, calculating the coordinates of the vertex G as the intersection point of the perpendicular plane in F to the edge FE and the
It is possible to determine the type of perspective represented in a graph as much in models as in flat figures. The detection of conical perspectives with three vanishing points is included.
The algorithm implemented involves exploring all the points to become vanishing points. To this end, we have proposed a round through the list of edges of the graph taking them two by two and calculating their intersection points according to the following considerations: • •
If the edges converge in a vertex, the point is rejected. If among the edges there are vertical or horizontal ones, the calculation of the intersection points is disregarded and the existence of this kind of edges is recorded.
Due to some inaccuracies because of truncation of decimals in the analytic resolution of the problem, an error tolerance has been established whose value has been set in an experimental way at 3% of the maximum dimension of the graph so that: •
•
If the distance between two intersection points of two edges is less than the error tolerance, those points are grouped as an only vanishing point whose coordinates are fixed to the first intersection point obtained. If the distance from an intersection point of two edges to the horizontal traced by the main point is less than the error tolerance, it is assumed that the intersection point is on the horizontal.
Once the intersections between all the edges of the graph have been analyzed, the vanishing points, the type of conical perspective represented and the main directions are determined according to the following considerations: •
If all the edges of the graph are horizontal and vertical, it is assumed that the graph represents a parallel conical perspective in which the main directions are given by families of horizontal and vertical edges, and a third one that is not necessary to determine. This particular case allows solving graphs which represent parallel perspectives of flat figures on parallel planes to the picture plane.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
43
Fig. 6. Reconstruction of a graph with three vanishing points.
•
If the main point coincides with a vanishing point and there are horizontal or vertical edges in the graph, it is assumed that the graph represents a parallel perspective and the main directions are given by the horizontal, the vertical and the family of edges which vanish to the main point. The way in which this point has been formulated allows solving graphs that represent parallel perspectives of flat shapes on planes perpendicular to the picture plane and parallel or perpendicular to the ground plane like the ones which are shown in Fig. 7.
•
If there are, at least, two vanishing points aligned according to the horizontal traced by the main point, those points are taken two by two in a way that they take opposite positions regarding
the main point, and it is checked whether the focal length obtained by geometrical procedure coincides with the focal length defined in the file of the image. If that distance coincides for at least a pair of vanishing points, it is assumed that the graph represents an oblique perspective with two vanishing points, and the family of edges of the graph which converge in those vanishing points and the vertical are considered as main directions. In Fig. 8 we show a graph that represents a model in oblique perspective with two vanishing points in which, once the position of the main point (P) is set in the center of the framework, the vanishing points VP1 , VP2 and VP3 aligned with the horizontal traced in P are obtained. Among them the pairs of vanishing points VP1 –VP2 and VP1 VP3 to be possible points where the main
44
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 7. Flat figures in parallel perspective contained in planes perpendicular to the picture plane.
Fig. 9. Flat figure contained in a plane parallel to the ground plane.
This particular case allows solving graphs that represent flat shapes in perspective with three vanishing points like the one shown in Fig. 11.
•
Fig. 8. Procedure to determine oblique conical perspectives with two vanishing points.
directions vanish are analyzed, given that both pairs are placed in opposite sides with regard to P. Finally, the edges that converge to VP1 and VP2 are selected as main directions, since their geometric mean regarding the point P coincides with the focal length. The way in which this point has been formulated allows solving graphs that represent oblique perspectives with two vanishing points of flat shapes on planes parallel to the ground plane like the one shown in Fig. 9. •
If there is only a vanishing point on the horizontal traced by the main point and the rest of the edges of the graph are vertical, it is assumed that the graph represents an oblique perspective with two vanishing points in which the main directions are given by the families of edges that converge to the vanishing point, the vertical and a third one which is not necessary to determine. This particular case allows solving graphs that represent, in oblique perspectives with two vanishing points, flat figures on planes perpendicular to the picture plane and the ground plane like the ones represented in Fig. 10.
•
If there are only two vanishing points non-aligned according to the horizontal traced by the center of the framework, it is assumed that the graph represents a perspective with three vanishing points in which the main directions are given by the families of edges that converge to the two vanishing points and a third one which is not necessary to determine.
If there are at least three vanishing points in the graph, the points are taken three by three and it is checked if the focal length obtained through geometrical procedure coincides with the focal length defined in the file of the image, and whether the ortho-center of the triangle of three vanishing points coincides with the center of the framework. If those conditions are fulfilled for at least three vanishing points, it is assumed that the graph represents a perspective with vanishing points, being the family of edges of the graph that converge in those vanishing points main directions.
5. Expansion of the reconstruction method through edges non-oriented according to the main directions The limitations of the expansion method through edges oriented according to the main directions cause that very simple models, like the one shown in Fig. 12, cannot be reconstructed as the removal of the edges non-parallel to the main directions, although it does not mean the loss of information about the number of vertexes, would divide the starting graph into two graphs connected to each other. However, in the conical projection system, in the same way as the inappropriate point of a set of parallel straight lines has its own representation focusing on a vanishing point, the inappropriate straight line of a set of parallel planes also has it, which is named vanishing line. On the whole, in order to determine an orientation it is enough to know two directions of it since two coplanar straight lines determine the corresponding plane. This leads to affirm that when the vanishing points of the straight lines contained in any of the planes of the set are known, their vanishing line is obtained when joining them. That is, the vanishing line is obtained joining two vanishing points of straight lines contained in any of those planes. Using the concept defined in conical perspective as vanishing line, we have defined a new method which allows the expansion of the reconstruction method through vertexes joined in the graph by edges non-parallel to the main directions.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
45
Fig. 10. Flat figures on planes perpendicular to the picture plane and the ground plane.
Fig. 11. Flat shapes with three vanishing points.
Fig. 12. Graph that cannot be reconstructed by expansion through main directions.
5.1. Reconstruction of graphs that represent models in conical oblique perspective with two and three vanishing points by expansion through edges non-parallel to the main directions In Fig. 13 we show a model in axonometric perspective and the corresponding conical oblique projection with two vanishing points obtained from the viewpoint V which we will use to illustrate the method proposed. Following the expansion method through edges oriented according to the main directions given in [33], it is possible to reconstruct the vertexes 1, 2, 4, 5, 6, 7, 9 and 10 of the model, remaining unresolved the convergence with the edge 3–8. According to what has been stated in the paragraphs above, the face of the model defined by the vertexes 1’, 2’, 3’, 4’ and 5’ can remain oriented according to the edges 1’–5’ and 1’–2’ and, therefore, the joining of the vanishing points of those edges define the vanishing line where any edge converge on this face.
The vanishing points of two edges that determine the orientation of the face are given by VP, intersection point of the edges 1’–5’ and 6’–10’ and an inappropriate point oriented according to the vertical. Therefore, the vanishing line defined for that face will be formed by the vertical traced by VP. Consequently, any pair of edges of the graph which are cut on the vanishing line, and at least one of them goes through a vertex of the edges that define the orientation of the face, are capable of representing an edge on the face and another one parallel to it. This is the case, in this example, of the edges 2’–3’ and 7’–8’ that being cut in T represent the projections of the edge 2–3 which is part of the face defined by 1–2 and 1–5 and the edge 7–8 which is an edge parallel to that one in the model. The angle that the vanishing line in T (joining of VT) forms with the ground plane will give us the angle that the edge 2–3 forms with the horizontal and, moreover, it will coincide with the angle formed by the straight line that joins the metric point of VP with the point T. Once that angle is determined, a straight line parallel to VT can be traced through the already known vertex 2, which allows us to place the vertex 3 in its intersection point with the straight line traced from V to 3’. Solving in such a way for the edge 7’–8’ we can obtain the coordinates of the vertex 8 and, as a result, the model is reconstructed. Therefore, once all the possible coordinates of the vertexes of the model have been obtained, using the expansion method through the edges oriented according to the main directions, it is possible to implement a new algorithm in order to continue with the expansion of the reconstruction through other edges as follows: (1) For each vertex Vi of the graph with already known coordinates in the model, the pair of edges A1 and A2, that with vertex in Vi have end vertexes VA1 and VA2 with coordinates
46
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 13. Expansion of the reconstruction through edges non-parallel to the main directions in oblique perspectives with two vanishing points.
also known in the model, are analyzed (those vertexes define the orientation of a face). In the example in Fig. 13 a vertex that fulfills these conditions could be vertex 1, given that the coordinates of the vertexes of the edges 1–2 and 1– 6 are known by the expansion of the method through edges oriented according to main directions. (2) For each end vertex VA1 and VA2 the edges that derive from them and are not parallel to the main directions are analyzed and it is checked their convergence with any other in a vanishing point aligned with the vanishing line defined by edges A1 and A2 . In the example in Fig. 13, from vertex 2’ the edge 2´–3’starts and converge with the edge 7’–8’ in the point T which is placed on the vanishing line defined by the edges 1’–2’ and 1’–6’. (3) For each one of these edges the coordinates of the vertex as the intersection of straight lines commented above are calculated. In the example in Fig. 13 the coordinates of the vertex 3 are calculated as the intersection of the straight line traced by 2, parallel to VT and the straight line V–3’. The method stated expands perspectives with three vanishing points considering, in this case, that the vanishing lines are defined by the edges that define the triangle of three vanishing points as there is not an inappropriate vanishing point of the edges parallel to the main directions. 5.2. Reconstruction of graphs that represent models in parallel conical perspective by expansion through edges non-parallel to the main directions In the case of parallel conical perspectives, given the existence of two inappropriate vanishing points corresponding to the edges represented in the graph as horizontal and vertical, the edges in a model parallel to one of the orthogonal planes defined by the main directions converge in a different way, so it is necessary to distinguish the following cases: 5.2.1. Edges oblique parallel to the picture plane This is the most obvious case. In the example in Fig. 14 the pairs of edges parallel to each other 1–5, 6–10 and 4–5, 9–10 are
parallel to the picture plane and are projected according to the straight lines 1’–5,’ 6’–10’ and 4’–5’, 9’–10’ parallel to each other on the picture plane vanishing to an inappropriate point. The angle that the projections form with the horizontal traced in the picture plane coincides with the angle which those edges form in the three-dimensional model with the ground plane. Therefore, once we have obtained the coordinates of the vertexes 1, 2, 3, 4, 6, 7, 8 and 9 by the expansion method through edges oriented according to main directions, we can obtain the coordinates of the vertexes 5 and 10 as intersection of the straight lines traced by 1 and 6 parallel to the picture plane forming an angle a with the ground plane, and the straight line that joins the projections 5’ and 10’ with the viewpoint V. The algorithm implemented, once the coordinates of the vertexes have been obtained by expansion through edges oriented according to the main directions, carries out the following verifications:
(1) For each vertex Vi of the graph with known coordinates in the model, the pairs of edges A1 and A2 horizontal and vertical of the graph, that with vertex in Vi have their end vertexes VA1 and VA2 with coordinates also known in the model, are analyzed. In Fig. 14 a vertex that fulfills these conditions could be the edge 2, since the coordinates of the vertexes of the edges 2–3 and 2–1 are known by expansion of the reconstruction through edges oriented according to main directions. (2) For each vertex VA1 and VA2 it is analyzed if the edges nonparallel to the main directions which converge in those vertexes in the graph are parallel to another oblique edge whose starting or final vertex is aligned with the vertex VA1 or VA2 respectively and the main point. In Fig. 14, in the vertex 1’ the edge 1´– 5’ parallel to 6’–10’ converges, and the vertexes 1’ and 6’ are aligned with the main point. (3) For each one of these edges the coordinates of the remaining vertex as the intersection of the straight lines commented above are calculated. In Fig. 14 the coordinates of the vertex 5 are calculated as the intersection of the straight line traced by 1 parallel to the picture plane forming an angle a with the ground plane and the straight line V–5’.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
47
Fig. 14. Expansion of the reconstruction through edges parallel to the picture plane in parallel conical perspectives.
5.2.2. Oblique edges contained in planes perpendicular to the picture plane and the ground plane In Fig. 15 a model is represented in which the edges 1–5, 6–10 and 4–5, 9–10 are parallel to each other and on planes perpendicular to the picture plane and the ground plane. The above-mentioned projections of the pairs of edges 1’–5’, 6’– 10’ and 4’–5’, 9’10’ converge to the vanishing points T1 and T2 respectively which are on the vertical traced by the main point. The straight lines traced from the viewpoint to the vanishing points form with the horizontal angles a and β which coincide with the angle that the edges of the model form with the ground plane. Therefore, knowing the coordinates of the vertexes 1, 2, 3, 4, 6, 7, 8 and 9 by expansion through edges oriented according to the main directions, we can obtain the coordinates of the vertexes 5 and 10 as intersection of the straight lines that traced by 4 and 9 on planes perpendicular to the picture plane and the ground plane form an angle a with the ground plane and the straight line that joins the projections 5’ and 10’ with the viewpoint. The algorithm implemented, once all the possible coordinates of the vertexes of the model have been obtained by expansion through edges oriented according to the main directions, carries out the following verifications: (1) For each vertex Vi of the graph with known coordinates in the model, the pairs of edges of the graph A1 and A2 , a vertical and another one going through the main point, that containing Vi have end vertexes VA1 and VA2 with coordinates also known in the model, are analyzed. In Fig. 15 a vertex that fulfills these conditions could be the vertex 3, since the coordinates of the vertexes of the edges defined by 3–2 and 3–4 are known by the expansion of the method through edges oriented according to main directions. (2) For each end vertex VA1 and VA2 of these edges, it is analyzed if the edges non-parallel to the main directions that converge in those vertexes of the graph converge with any other edge in a vanishing point aligned according to the vertical traced by the main point. In Fig. 15, in the vertex 4’ the edge 4´–5’, that converges with 9’–10’ in T1 (point aligned with the vertical traced by the main point) converges. (3) For each one of these edges the coordinates of its vertex as the above-mentioned intersection of the straight lines are
calculated. In Fig. 15 the coordinates of the vertex 5 are calculated as the intersection of the straight line traced by 4 on a plane perpendicular to the picture plane and the ground plane forming an angle a with the ground plane and the straight line V–5’. 5.2.3. Edges oblique parallel to the ground plane In Fig. 16 we represent a model with pairs of edges 2–3, 8–9 and 4–5, 10–11 parallel to the ground plane. Their projections 2’– 3’, 8’–9’ and 4’–5’, 10’–11’ converge to the vanishing points T1 and T2 aligned with the horizontal traced by the main point. The straight lines traced from the viewpoint to the vanishing points and the main point define angles a and β which coincide with the angles that the edges of the model form with the picture plane. Therefore, once we have obtained the coordinates of the vertexes 3, 4, 9 and 10 by the expansion method through edges oriented according to main directions, we can obtain the coordinates of the vertexes 2, 5, 8 and 11 as the intersection of the straight lines traced by 3, 4, 9 and 10 contained in planes parallel to the ground plane and that form an angle a or β with the picture plane, and the straight line that joins the projections 3’, 4’ 9’ and 10’ with the viewpoint. The algorithm implemented, once all the possible coordinates of the vertexes of the model have been obtained by expansion through edges oriented according to the main directions, carries out the following verifications: (1) For each edge Ai vertical defined by vertexes VA1 and VA2 with known coordinates in the model, it is analyzed the existence of edges that going through VA1 and VA2 converge in a vanishing point aligned according to the horizontal traced by the main point. In Fig. 16 the edge 3’–9’ fulfills these conditions since the coordinates of the vertexes 3 and 9 are known by the expansion of the method through edges oriented according to main directions, and from vertexes 3 and 9 the pairs of edges 2’–3’ and 8’–9’ start, which converge in the point T1 aligned according to the horizontal traced by the main point. (2) For each one of these edges the coordinates of its vertex as the above-mentioned intersection of the straight lines
48
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 15. Expansion of the reconstruction through edges perpendicular to the picture plane and the ground plane in parallel conical perspectives.
Fig. 16. Expansion of the reconstruction through edges parallel to the ground plane in parallel conical perspectives.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
49
Fig. 17. Vectorization of a photograph.
Fig. 18. A detailed operation example.
are calculated. In Fig. 16 the coordinates of the vertex 2 are calculated as the intersection of the straight line traced by 3 parallel to the ground plane forming an angle a with the picture plane and the straight line V–2’.
the scope of this paper, has been investigated for different purposes by several authors [35–37]. The graph, resulting from the vectorization, constitutes the starting information for our algorithms which are executed in the following way (see Fig. 18):
6. A detailed operation example For further clarification of the proposed algorithms, in this section we show, step by step, the reconstruction process of a model from a perspective with three vanishing points. As a preliminary step for the reconstruction of an image we need its vectorization; that is, to transform the information given in the form of bitmap into a vector image defined by vertexes and edges. This process, which is outlined in Fig. 17, which is outside
Step 1: The intersections of the edges which form the graph are analyzed according to the conditions given in Section 4, and the type of perspective is determined. In the present case, given that there are three vanishing points which define a triangle whose ortho-center coincides with the main point and, moreover, the geometrically obtained focal length coincides with the known focal length of the image, the perspective is defined as the type of three vanishing points being
50
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 19. Method used to obtain graphs of models.
the main directions those which vanish to points VP1 , VP2 and VP3 . Step 2: The reconstruction process starts in the edge 1–2 oriented according to one of the main directions determining the three-dimensional coordinates of its vertexes as indicated in Section 3. Subsequently, and in accordance with the same section, the reconstruction is expanded through the edges parallel to the main directions obtaining the coordinates of the vertexes 3 and 4. Step 3: At this stage, the method cannot continue expanding according to edges parallel to the main directions, so other possible ways are examined. In our case, the edges 1–5 and 2–6 converge on a point aligned with the vanishing points VP1 and VP2 and, moreover, both edges go through vertexes with known three-dimensional coordinates so, according to Section 5.1, the coordinates of the vertexes 5 and 6 are calculated. Step 4: Knowing the coordinates of the edge 5–6, the reconstruction method can continue expanding according to edges parallel to the main directions obtaining the coordinates of the vertexes 7, 8, 9, 10 and 11 ending, in this way, the reconstruction process. 7. Experimental work and results Given that a vectorization algorithm has not been implemented in this work, the capture of images has been carried out from software that allows the positioning of cameras and the exportation of perspectives to dxf format. In the different examples tested the framework that defines the size of the picture has been added, and the focal length has been modified using angle lens of 24, 35 and standard of 50 mm, being the width/height coefficient of 4:3. In Fig. 19 the graph obtained following this process is shown. In it, the positioning of the camera in the software used, the image captured by the camera and the graph obtained once the image is exported to dxf format can be seen. The methods proposed have been implemented in Visual C# under the platform of Visual Studio (WinForm) using the libraries CsGl (OpenGl for Visual C#).
In Figs. 20 and 21 we show the 80 examples analyzed in order to validate the algorithm proposed for the reconstruction of conical perspectives with three vanishing points. Those examples correspond to projections of the models presented in [33] in the case of oblique and parallel perspectives with two vanishing points. Moreover, the 160 examples presented in that publication have been analyzed again to evaluate the criteria proposed for the analysis of the main directions and the type of conical perspective in point 4 in this document. In Figs. 22–24 we show the 96 examples in order to validate the algorithms for the reconstruction of flat shapes and the expansion method through edges non-parallel to the main directions. Those examples correspond to parallel and oblique conical perspectives with two and three vanishing points in 32 new models proposed. In the settings of the application shown in Fig. 25, two windows have been defined. In one of them (left window), you may observe the starting graph where it is possible to see the detected vanishing points, the main directions, the graph typology and the type of perspective detected. In the other one (on the right), we show the model obtained from the reconstruction of the graph. This window has a tool that allows us to rebuild the graph step by step. The success rate in the graphs tested is of 100%. Moreover, although in a wide majority the graphs have been represented with hidden edges, tests with graphs without hidden edges have been carried out obtaining their correct representation, although, obviously, the final model does not constitute a solid but a model without depth. 8. Conclusions In this work we show the latest advances achieved in obtaining polyhedral models from conical perspectives. We present a set of criteria that allow analyzing in an automatic way if a certain graph corresponds to a parallel, oblique conical or with three vanishing points projection. A new method for the reconstruction of conical perspectives with three vanishing points is proposed and we have defined a methodology that allows the expansion of the calculation of the coordinates of a model from a projection, not only through edges
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 20. Graphs in conical perspective with three vanishing points (I).
51
52
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 21. Graphs in conical perspective with vanishing points (II).
oriented according to directions perpendicular to each other, but it also allows doing it from other edges which do not fulfill these conditions. Moreover, we include projections of flat shapes as a possible input to the algorithm. None of the methods implemented requires the defined faces of the model as starting information, which makes these strategies a completely automatic system that does not need to interact with the user.
To sum up, the necessary methodologies for the completely automatic reconstruction of three-dimensional models from parallel and oblique conical perspectives and with three vanishing points have been defined for a wide family of polyhedral models. The methods have been implemented in Visual C# under the platform of Visual Studio (WinForm) using the libraries CsGl (OpenGl for Visual C#) carrying out a total of 336 tests obtaining
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 22. New graphs in parallel conical perspective.
53
54
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 23. New graphs in oblique conical perspective with two vanishing points.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 24. New graphs in conical perspective with three vanishing points.
55
56
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 25. Work environment.
a success rate of 100%. The results achieved reveal the interest of the method proposed in the field of obtaining models from conical perspectives.
References [1] Robert LG. Machine perception of three-dimensional solids. Optical and electro-optical information processing, Cambridge, Massachusetts and London, England: The MIT Press; 1965. [Chapter 9]. [2] Zou CQ, Chen SF, Fu HB, Liu JZ. Progressive 3D reconstruction of planar-faced manifold objects with DRF-based line drawing decomposition. IEEE Trans Vis Comput Graph 2015;21(2):252–63. doi:10.1109/TVCG.2014.2354039. [3] Huffman DA. Impossible objects as nonsense sentences. Mach Intell 1971;6:295–323. [4] Clowes MB. On seeing things. Artif Intell 1971;2:79–116. [5] Thorpe C, Shafer S. Correspondence in line drawings of multiple views of objects. In: Proceedings of the eighth international joint conference on artificial intelligence, vol. 2; 1983. [6] Wang W, Grinstein G. A survey of 3D solid reconstruction from 2D projection line drawings. Comput Graph Forum 1993;12(2):137–58. [7] Malik J. Interpreting line drawing of curved objects. Int J Comput Vis 1987;1:73–103. [8] Varley P, Martin R. A system for constructing boundary representation solid models from a two-dimensional sketch. Frontal geometry and sketch categorisation. In: Proceedings of the 1st Korea_UK joint workshop on geometric modeling and computer graphics; 20 0 0. [9] Varley P, Martin R. TA system for constructing boundary representation solid models from a two-dimensional sketch. Topology of hidden parts. In: Proceedings of the 1st Korea_UK joint workshop on geometric modeling and computer graphics; 20 0 0. [10] Parodi P, Piccioli G. 3D shape reconstruction by using vanishing points. IEEE Trans Pattern Anal Mach Intell 1996;18(2):211–17. [11] Lamb D, Bandopadhay A. Interpreting a 3D object from a rough 2D line drawing. In: Proceedings of Visualization’90; 1990. p. 59–66. [12] Marill T. Emulating the human interpretation of line-drawings as three-dimensional objects. Int J Comput Vis 1991;6(2):147–61. [13] Baird LC, Wang PS. 3D object recognition using gradient descent and the universal 3D ray grammar. In: Proceedings of SPIE conference on intelligent robots and computer vision X: algorithms and techniques, vol. 1607, Boston, MA; 1991. [14] Leclerc Y, Fischler M. An optimization – based approach to the interpretation of single line drawings as 3D wire frames. Int J Comput Vis 1992;9(2):113–36. [15] Lipson H, Shpitalni M. A new interface of conceptual design based on object reconstruction from a single freehand sketch. Ann CIRP 1995;44(1):133–6.
[16] Lipson H, Shpitalni M. Optimization-based reconstruction of a 3D object from a single freehand line drawing. Comput Aided Des 1996;28(8):651–63. [17] Sphitalni M, Lipson H. Identification of faces in a 2D line drawing projection of a wireframe object. IEEE Trans Pattern Anal Mach Intell 1996;18(10):10 0 0–12. [18] Sphitalni M, Lipson H. Classification of sketch strokes and corner detection using conic sections and adaptive clustering. J Mech Des 1997;119(1):131–5. [19] Company Calleja P, Contero González M, Conesa Pastor J, Piquer Vicent A. An optimisation-based reconstruction engine for 3D modelling by scketching. Comput Graph 2004;28(6) pp. 995–979. [20] Lee YT, Fang F. A new hybrid method for 3D object recovery from 2D drawings and its validation against the cubic corner method and the optimisation-based method. Computer-Aided Des 2012;44(11):1090–102. [21] Chu J, Gao M, Zhang G, Feng R. Line-Based Optimization for 3D object reconstruction from single line drawings. In: Proceedings of international conference on information technology for manufacturing systems; 2010. p. 910–15. [22] Liu JZ, Cao LL, Li ZG, Tang XO. Plane-based optimization for 3D object reconstruction from single line drawings. IEEE Trans Pattern Anal Mach Intell 2008;30(2):315–27. [23] Liu JZ, Lee YT, Cham W. Identifying faces in a 2D line drawing representing a manifold object. IEEE Trans Pattern Anal Mach Intell 2002;24(12):1579–93. [24] Liu JZ, Tang XO. Evolutionary search for faces from line drawings. IEEE Trans Pattern Anal Mach Intell 2005;27(6):861–72. [25] Chen Y, Liu J, Tang X. A divide-and-conquer approach to 3D object reconstruction from line drawings. In: Proceedings of the 11th IEEE international conference on computer vision; 2007. p. 1807–14. [26] Liu J, Chen Y, Tang X. Decomposition of complex line drawings with hidden lines for 3D planar-faced manifold object reconstruction. IEEE Trans Pattern Anal Mach Intell 2011;33(1):3–15. [27] Xue T, Liu J, Tang X. Object cut: complex 3D object reconstruction through line drawing separation. In: Proceedings of 23rd IEEE conference on computer vision and pattern recognition (CVPR); 2010. p. 1149–56. [28] Yang L, Liu J, Tang X. Complex 3D general object reconstruction from line drawings. In: Proceedings of IEEE international conference on computer vision (ICCV); 2013. p. 1433–40. [29] Zou CQ, Chen SF, Fu HB, Liu JZ. Progressive 3D reconstruction of planar-faced manifold objects with DRF-based line drawing decomposition. IEEE Trans Vis Comput Graph 2015;21(2):252–63. doi:10.1109/TVCG.2014.2354039. [30] Fang F, Lee YT. Efficient decomposition of line drawings of connected manifolds without face identification. Comput Aided Des 2014;51:18–30. [31] Leong MC, Lee YT, Fang FA. Search-and-validate method for face identification from single line drawings. IEEE Trans Pattern Anal Mach Intell 2013;35(11):2576–91. [32] Fang F, Lee YT. 3D reconstruction of polyhedral objects from single perspective projections using cubic corner. 3D Res 2012;3(2):1–8 No. 59. [33] Conesa J, Mula F. Obtaining three-dimensional models from conical perspectives. Adv Eng Softw 2015;90:83–97. doi:10.1016/j.advengsoft.2015.07.002.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57 [34] Rother C. A new approach to vanishing point detection in architectural environments. Image Vis Comput 2002;20(9–10):647–55. [35] Colombo C, Del Bimbo A, Pernici F. Metric 3D reconstruction and texture acquisition of surfaces of revolution from a single uncalibrated view. IEEE Trans Pattern Mach Intell 2005;27(1):99–114.
57
[36] Suman Kumar M, Hussein MY, Hicham Badri. Reconstructing an image from its edge representation. Digit Signal Process 2013;23:1867–76. [37] Vouzounaras G, Daras P, Strintzis MG. Automatic generation of 3D outdoor and indoor building scenes from a single image. Multimed Tools Appl 2014;70(1):361–78.
Contents lists available at ScienceDirect
Advances in Engineering Software journal homepage: www.elsevier.com/locate/advengsoft
New advances in obtaining three-dimensional models from conical perspectives J. Conesa∗, F. Mula Graphical Expression Department, Universidad Politécnica de Cartagena, Campus Muralla del Mar, C/ Doctor Fleming s/n, Cartagena 30202, Spain
a r t i c l e
i n f o
Article history: Received 30 December 2015 Revised 8 February 2016 Accepted 29 February 2016
Keywords: Image reconstruction Single image technique Image-based modeling Conical perspective
a b s t r a c t In a previous work we presented algorithms which allow obtaining three-dimensional models from graphs which represent a projection in conical parallel perspectives and conical oblique perspectives of polyhedral models with normalon and quasi-normalon typology. In this paper the new advances that we have achieved in this field are presented, allowing increasing the set of models which can be reconstructed to other typologies different from the normalon and quasi-normalon ones. Moreover, we present a new technique which extends the previous work in order to be implemented to conical perspectives with three vanishing points, and the method proposed for the detection of the type of conical perspective represented by the graph, including the detection and subsequent reconstruction of graphs which represent a flat shape, has been improved. The results obtained on a total of 336 tests, with a success ratio of 100%, make the method a proposal to be considered for obtaining models from conical perspectives automatically. © 2016 Elsevier Ltd. All rights reserved.
1. Introduction The automatic obtaining of three-dimensional models from graphs which represent one of their projections is a field of great interest for the scientific community as evidenced in the continuous investigations carried out from the 1960s with the work done in [1] until now with the recent work published in [2]. From the approach of reconstructing models from conical perspectives, on which this work is focused, we can group them, paying attention to the technique used, in two blocks: labeling methods and perceptual methods. Labeling methods are based on a classification of the vertexes according to the directions of the edges which come together in each corner. The first valid labeling methods are presented in the work carried out in parallel in [3,4] where the vertexes of a cube were classified after considering all the corners from all possible viewpoints of an axonometric perspective. From this work, new methods were developed in order to improve and expand the procedure of labeling [5,6], but all of them were based on heuristic rules and restrictions were presented. In [7] a new labeling scheme of drawings, including nonnecessarily polygonal objects, was developed. However, the pro-
∗ Corresponding author at: Graphical Expression Department, Universidad Politécnica de Cartagena, Campus Muralla del Mar, C/Doctor Fleming s/n, 30202 Cartagena, Spain. Tel.: +34 968 32 64 77; fax: +34 968 32 64 74. E-mail address: [email protected], [email protected] (J. Conesa).
http://dx.doi.org/10.1016/j.advengsoft.2016.02.012 0965-9978/© 2016 Elsevier Ltd. All rights reserved.
cedure allows labeling the same drawing in more than one way, which contrasts with human perception in which a projection has very few ways of dimensional interpretation. The latest contributions in this line are the labeling methods as shown in [8,9] where the geometry of the hidden parts can be established, and where the front model geometry can be defined. A reference in the reconstruction of models from conical perspectives based on labeling models is the work presented in [10] where the quantitative reconstruction of the 3D structure of a scene from a line drawing, by using the geometrical constraints provided by the location of vanishing points, is investigated. In this work the design of an algorithm, which has several advantages with respect to the usual approach based on a reduction to linear programming, is presented. These advantages range from a lower computational complexity to error tolerance and exact reconstruction of the 3D-geometry of the objects. The perceptual methods are characterized by trying to implement, through the sequential language of computers, the way humans perceive. In [11] the first perceptual method is designed based on the proportionality between the projection and the threedimensional model, starting from a labeling method that allows us to obtain a graph of adjacency to define the orientation of each edge regarding the main axes. In their algorithm, the parallel lines of the projection are parallel in the model and the parallel edges to the main axes are drawn proportional to the actual dimensions, thus allowing us to define parallelograms with faces parallel to the planes of projection. Although the method allowed flexibility to inaccuracies in the drawing, the algorithm is characterized by a high
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
degree of user interaction needed to designate the principal axes (intersection of the planes of projection). Furthermore, the method was very limited since rebuilding models should not contain hidden edges given the ambiguities that the representation of these edges causes. The other methods have been based on perceptual optimization methods. It is considered that such methods are the mathematical process that more closely resembles the way humans interpret a projection. The strategy called “inflation”, applied to an axonometric projection of a model, works assigning coordinates “Z” to vertexes of a projection while maintaining its coordinates (x, y) by minimizing an objective function. The function to be optimized has been modified and extended by different authors. In [12] the objective function is formed by a single component, the MSDA (minimum standard deviation of angles) in [13,14] the mentioned objective function was expanded with the DP (deviation from flatness of the faces of a model) and [15–18] propose an objective function composed of a sum of subfunctions representing regularities of the projection, i.e. characteristics of the model that can be deduced from the observation of the projection. However, the reconstruction algorithms based on optimization present common difficulties: given the non-linearity of the objective functions proposed, there is a high number of local minimums, therefore the solution reached does not always represent the desired model (the own projection of the model that constitutes the starting point for the optimization is in many times a local minimum). Focused on these problems, some authors have tried to propose models which allow obtaining a tentative initial model which approaches the desired model as much as possible, so that the optimization process reaches the global minimum [19,20]. In [21] a new method based on an iterative optimization process, in which, once the reconstruction has been performed, the model obtained is projected on a plane and compared with the starting projection, is proposed. If both projections do not coincide in a preset threshold, the restrictions are readjusted and the reconstruction process is repeated. However, the computer cost of the process is not mentioned. A reference of the implementation of methods based on optimization for the reconstruction of conical perspectives can be found in [22], where the problems derived from the use of nonlinear functions in the optimization processes are highlighted. This non-linear function is owing to the fact that the flatness of the faces is usually expressed according to the vertexes which define it. In their work an objective function, defined from a set of linear restrictions, is proposed. The algorithm can process perspectives with or without representation of hidden edges, and uses the algorithms given in [23,24] to determine the faces defined in the model from its projection. Another trend, which has as its final aim the reconstruction, is focused on the division of complex models into simpler parts to later, once they are reconstructed, put them together again in order to form a single solid. However, most of the works presented from this view [25–29] need, as starting information, the faces defined in the model. In [30] a division algorithm, which does not need the previous detection of the faces, is presented, but its results are not always the expected ones. Most of the algorithms and strategies proposed for the reconstruction use as starting information the faces defined in the model. However, in spite of the great number of investigations directed towards the identification of faces from the projection of a model which have extended from the pioneering work given in [17] until more current woks like the one carried out in [31], the problem has not been solved completely. Although the existing methods can find faces in most cases, there is not a
39
mechanism to determine if the result is right, leaving this decision to the user. Therefore, the reconstruction methods that need a previous identification of the faces must be considered as semi-automatic models, and in no case it is possible to talk of automatic reconstruction of models. In the work presented in [11], and later revised in [32], an algorithm that starts with the calculation of the coordinates of a vertex on an orthogonal corner is proposed, and it is proved that, knowing these coordinates, it is possible to determine the coordinates of the rest of the vertexes through an expansion tree that uses the equations of the plane which define the faces of the model. However, the faces defined in the model are used as starting information again, it is assumed that the projection always corresponds with a conical perspective with three vanishing points, and the projections cannot contain hidden edges. Focused on improving the proposals presented by these authors, in [33] it is presented a work which allows the automatic reconstruction of polyhedral models with normalon and quasinormalon typology from parallel and oblique conical perspectives with two vanishing points which do not need the previous detection of the faces and can or cannot contain hidden edges. This paper represents an improvement of the work presented in [33]. On the one hand, the set of models that can be reconstructed is increased and they do not need to be of normalon or quasi-normalon typology any longer. On the other hand, the reconstruction method proposed widens including conical perspectives with three viewpoints. The system, completely automatic, does not require obtaining the faces of the model. 2. Previous work The work presented in [33] is focused on the reconstruction of graphs with normalon and quasi-normalon typology defined as: •
•
Normalon graph: The name is a generalization to the world of polyhedra, related to the normalon polygons concept, such as those with the property in which all angles between two concurrent edges are 90°. In Fig. 1 we can see a graph of this type since all edges converge towards three vanishing points (one improper). Quasi-normalon graph: In these graphs, despite not belonging to the normalon type, it is possible to reach all vertexes, exclusively through edges that are parallel to three directions, and which form 90º to each other. That is, they are graphs that meet the condition where the elimination of all non-parallel edges to these three directions does not imply the loss of all vertexes, also allowing them to remain connected. We name them main directions. In Fig. 1 we can see a graph of this type in which the removal of non-parallel edges in three main directions (1–2, 1– 5, 4–6, 5–7, 7–12, 5–12, 5–8, 1–8, 1–9, 2–9) does not imply the loss of vertexes and they remain connected (there is only one graph).
For the reconstruction of graphs we propose a method based on [32] with the advantage of not using, as starting information, the defined faces in the model. Knowing the three-dimensional coordinates of the vertexes of an edge of the model (edge AD in Fig. 2), it is possible to determine the coordinates of the vertexes connected to it by edges oriented according to the main directions, like the intersection of the perpendicular plane to the edge in one of its vertexes and the straight line that joins the viewpoint with the projection of the vertexes (vertexes B and C in Fig. 2). The method proposed can be expanded following the edges of the graph oriented according to the main directions until the complete reconstruction of the model. The evaluated vertexes are
40
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 1. Typologies of graphs.
Fig. 2. Method for obtaining the three-dimensional model.
added to a model list of vertexes (MLV) which establishes the path to follow in the expansion of the model. In Fig. 3 we show the expansion tree used for the reconstruction of a graph with quasinormalon typology as well as the variations in the MLV. Given the characteristics of the parallel and oblique conical projection of two vanishing points, in which the vertical edges keep this orientation in the projection, the reconstruction process can be started in one of these edges keeping the coordinates (x, y) that
its vertexes have in the projection in the model and assigning coordinate z equal to zero. 3. Reconstruction of conical perspectives with three vanishing points In this paper we show a method to expand the reconstruction process commented above in the case of conical perspectives with
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
41
Fig. 3. Expansion tree for the reconstruction of a graph with quasi-normalon typology.
B aligned according to a straight line that vanishes to VP1. Tracing a straight line on the picture plane through A parallel to VP1 VP 2, in the intersection points of this one with the straight lines traced from A and B to MP1 (points Ao, Bo) we obtain the real magnitude of the segment. Knowing the real magnitude of one of the edges of the model, and given that according to the principles of the conical perspective the vertexes of a model will converge on the straight lines that join the viewpoint with the corresponding projections of the vertexes on the picture plane, assigning coordinates Z = 0 to any of the vertexes, it will be possible to define the orientation of an edge passing through that one. Therefore, the steps to be taken for the reconstruction of graphs which represent conical perspectives with three vanishing points will be the following ones:
Fig. 4. Obtaining the real magnitudes in perspectives with three vanishing points.
three vanishing points. The problem with this is that none of the main directions of the model is parallel to the picture plane since all of them converge in three vanishing points, so not knowing the orientation of any of the edges in the model it is not possible to assign starting coordinates to their vertexes. Based on the principles of the conical perspective with three vanishing points, it can be stated that the families of edges oriented according to the main directions converge to points that constitute the vertexes of the triangle of three vanishing points. The heights traced in that triangle coincide with the projections of the axes of the trihedro and their gathering point (ortho-centre of the triangle of three vanishing points) will be the orthogonal projection of the vertex of the trihedro on the picture plane, coinciding with the main point of the perspective. Dropping one of the faces of the triangle of three vanishing points we can obtain the metric points for each one of the main directions, and we can use the metric points to measure segments which are part of the straight lines that vanish in the corresponding vanishing point. In order to further clarify these concepts, in Fig. 4 we show the way to determine the real magnitude between points A and
(1) Take an edge of the graph oriented according to one of the main directions and assign coordinates to one of its vertexes keeping the values (x,y) of the graph and setting z equal to zero. (2) Determine the coordinates of the other vertex of the edge taken in point 1 bringing on the straight line that joins the viewpoint and the projection of the vertex the real magnitude of the edge obtained from the metric point. (3) Apply the expansion method in the same way as it is applied in the cases of parallel and oblique perspectives with two vanishing points. In order to further clarify the procedure described, we analyze step by step the reconstruction of a hexahedron. Just as it is shown in Fig. 5, the process is started taking an edge of the graph oriented according to the main directions (in this case A E ) assigning coordinates to one of its vertexes (A) in the way (A x, A y, 0). Later, it is determined the real magnitude of that edge. To this end, given that the edge A E vanishes to the point VP1 , we will determine metric point MP1 and trace through A a straight line R on the picture plane parallel to the line that joins VP1 and VP2 . The real magnitude of the AE will be obtained as the distance between the point A and Eo, being this one the intersection point of the straight line R and the straight line MP1 E . Knowing the value of the real magnitude, it will be sufficient to take this distance from A on the straight line that joins the viewpoint V with E . Once the coordinates of the vertexes of the edge A E have been obtained, both vertexes are added to the MLV and the reconstruction process continues through an expansion tree whose progression is determined by the order of the vertexes in the MLV, and it
42
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
straight line that joins the viewpoint V with the projection G’ of the vertex (see Fig. 6d). 4. Analysis of the principal directions and type of conical perspective It becomes clear that the expansion of the reconstruction method through edges oriented according to the main directions needs a prior knowledge of those directions in the projection of the model. This paper means a continuation of the methods developed in [33] where the criteria followed to identify parallel and oblique perspectives with two vanishing points are expounded. That work is focused on the reconstruction of graphs that comes from the vectorization of a photograph, so it is considered that the main point (projection of the viewpoint) is centered in the frame that defines the size of the photograph. Furthermore, as much the size of the photograph as the focal length (distance from the viewpoint to the picture lane) are considered starting data since both of them are intrinsic parameters of the device used to capture the image and can be extracted from the own file of the image. The algorithm implemented to analyze the type of conical perspective starts from the criteria given [33] which, in turn, derives from the work presented in [34], to which we have introduced some modifications that mean the following improvements: •
•
Fig. 5. Start of the reconstruction process of a graph with three vanishing points.
finishes when the last vertex of the MLV is evaluated or when all the vertexes of the graph are in the MLV. The expansion tree follows the progression represented in Fig. 6: (1) Taken the first vertex of the MLV as reference vertex, vertex A’, the vertexes B’ and D’ connected to A’ by edges oriented according to the main directions are analyzed. The coordinates of the vertexes B’ and D’ are obtained as the intersection of the perpendicular plane in A to the edge AE and the straight lines VB’ and VD’ respectively. Finally, the new vertexes are added to the MLV (see Fig. 6a). (2) Taking the following vertex of the MLV, vertex E’, as reference, the coordinates of the vertexes F’ and H’ as the intersection points between the perpendicular plane to the edge AE in the vertex E and the straight lines traced from the viewpoint V to their corresponding projections on the picture plane F’ and H’ are calculated, adding the new vertexes to the MLV. The vertex A’ is not calculated from E’ since it is already in the MLV (see Fig. 6b). (3) Repeating the same process, taking now the vertex B’ as a reference, the coordinates of the vertex C as the intersection of the perpendicular plane in B to the edge BA and the straight line that joins the viewpoint V with C’ are calculated (see Fig. 6c). (4) The next vertex of the MLV, vertex D’, is not considered as a reference vertex given that coordinates of a new vertex are not determined since all the vertexes are already connected to it, vertexes C’ and H’, in the LVC. (5) The last vertex to be considered as a reference is F’, calculating the coordinates of the vertex G as the intersection point of the perpendicular plane in F to the edge FE and the
It is possible to determine the type of perspective represented in a graph as much in models as in flat figures. The detection of conical perspectives with three vanishing points is included.
The algorithm implemented involves exploring all the points to become vanishing points. To this end, we have proposed a round through the list of edges of the graph taking them two by two and calculating their intersection points according to the following considerations: • •
If the edges converge in a vertex, the point is rejected. If among the edges there are vertical or horizontal ones, the calculation of the intersection points is disregarded and the existence of this kind of edges is recorded.
Due to some inaccuracies because of truncation of decimals in the analytic resolution of the problem, an error tolerance has been established whose value has been set in an experimental way at 3% of the maximum dimension of the graph so that: •
•
If the distance between two intersection points of two edges is less than the error tolerance, those points are grouped as an only vanishing point whose coordinates are fixed to the first intersection point obtained. If the distance from an intersection point of two edges to the horizontal traced by the main point is less than the error tolerance, it is assumed that the intersection point is on the horizontal.
Once the intersections between all the edges of the graph have been analyzed, the vanishing points, the type of conical perspective represented and the main directions are determined according to the following considerations: •
If all the edges of the graph are horizontal and vertical, it is assumed that the graph represents a parallel conical perspective in which the main directions are given by families of horizontal and vertical edges, and a third one that is not necessary to determine. This particular case allows solving graphs which represent parallel perspectives of flat figures on parallel planes to the picture plane.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
43
Fig. 6. Reconstruction of a graph with three vanishing points.
•
If the main point coincides with a vanishing point and there are horizontal or vertical edges in the graph, it is assumed that the graph represents a parallel perspective and the main directions are given by the horizontal, the vertical and the family of edges which vanish to the main point. The way in which this point has been formulated allows solving graphs that represent parallel perspectives of flat shapes on planes perpendicular to the picture plane and parallel or perpendicular to the ground plane like the ones which are shown in Fig. 7.
•
If there are, at least, two vanishing points aligned according to the horizontal traced by the main point, those points are taken two by two in a way that they take opposite positions regarding
the main point, and it is checked whether the focal length obtained by geometrical procedure coincides with the focal length defined in the file of the image. If that distance coincides for at least a pair of vanishing points, it is assumed that the graph represents an oblique perspective with two vanishing points, and the family of edges of the graph which converge in those vanishing points and the vertical are considered as main directions. In Fig. 8 we show a graph that represents a model in oblique perspective with two vanishing points in which, once the position of the main point (P) is set in the center of the framework, the vanishing points VP1 , VP2 and VP3 aligned with the horizontal traced in P are obtained. Among them the pairs of vanishing points VP1 –VP2 and VP1 VP3 to be possible points where the main
44
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 7. Flat figures in parallel perspective contained in planes perpendicular to the picture plane.
Fig. 9. Flat figure contained in a plane parallel to the ground plane.
This particular case allows solving graphs that represent flat shapes in perspective with three vanishing points like the one shown in Fig. 11.
•
Fig. 8. Procedure to determine oblique conical perspectives with two vanishing points.
directions vanish are analyzed, given that both pairs are placed in opposite sides with regard to P. Finally, the edges that converge to VP1 and VP2 are selected as main directions, since their geometric mean regarding the point P coincides with the focal length. The way in which this point has been formulated allows solving graphs that represent oblique perspectives with two vanishing points of flat shapes on planes parallel to the ground plane like the one shown in Fig. 9. •
If there is only a vanishing point on the horizontal traced by the main point and the rest of the edges of the graph are vertical, it is assumed that the graph represents an oblique perspective with two vanishing points in which the main directions are given by the families of edges that converge to the vanishing point, the vertical and a third one which is not necessary to determine. This particular case allows solving graphs that represent, in oblique perspectives with two vanishing points, flat figures on planes perpendicular to the picture plane and the ground plane like the ones represented in Fig. 10.
•
If there are only two vanishing points non-aligned according to the horizontal traced by the center of the framework, it is assumed that the graph represents a perspective with three vanishing points in which the main directions are given by the families of edges that converge to the two vanishing points and a third one which is not necessary to determine.
If there are at least three vanishing points in the graph, the points are taken three by three and it is checked if the focal length obtained through geometrical procedure coincides with the focal length defined in the file of the image, and whether the ortho-center of the triangle of three vanishing points coincides with the center of the framework. If those conditions are fulfilled for at least three vanishing points, it is assumed that the graph represents a perspective with vanishing points, being the family of edges of the graph that converge in those vanishing points main directions.
5. Expansion of the reconstruction method through edges non-oriented according to the main directions The limitations of the expansion method through edges oriented according to the main directions cause that very simple models, like the one shown in Fig. 12, cannot be reconstructed as the removal of the edges non-parallel to the main directions, although it does not mean the loss of information about the number of vertexes, would divide the starting graph into two graphs connected to each other. However, in the conical projection system, in the same way as the inappropriate point of a set of parallel straight lines has its own representation focusing on a vanishing point, the inappropriate straight line of a set of parallel planes also has it, which is named vanishing line. On the whole, in order to determine an orientation it is enough to know two directions of it since two coplanar straight lines determine the corresponding plane. This leads to affirm that when the vanishing points of the straight lines contained in any of the planes of the set are known, their vanishing line is obtained when joining them. That is, the vanishing line is obtained joining two vanishing points of straight lines contained in any of those planes. Using the concept defined in conical perspective as vanishing line, we have defined a new method which allows the expansion of the reconstruction method through vertexes joined in the graph by edges non-parallel to the main directions.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
45
Fig. 10. Flat figures on planes perpendicular to the picture plane and the ground plane.
Fig. 11. Flat shapes with three vanishing points.
Fig. 12. Graph that cannot be reconstructed by expansion through main directions.
5.1. Reconstruction of graphs that represent models in conical oblique perspective with two and three vanishing points by expansion through edges non-parallel to the main directions In Fig. 13 we show a model in axonometric perspective and the corresponding conical oblique projection with two vanishing points obtained from the viewpoint V which we will use to illustrate the method proposed. Following the expansion method through edges oriented according to the main directions given in [33], it is possible to reconstruct the vertexes 1, 2, 4, 5, 6, 7, 9 and 10 of the model, remaining unresolved the convergence with the edge 3–8. According to what has been stated in the paragraphs above, the face of the model defined by the vertexes 1’, 2’, 3’, 4’ and 5’ can remain oriented according to the edges 1’–5’ and 1’–2’ and, therefore, the joining of the vanishing points of those edges define the vanishing line where any edge converge on this face.
The vanishing points of two edges that determine the orientation of the face are given by VP, intersection point of the edges 1’–5’ and 6’–10’ and an inappropriate point oriented according to the vertical. Therefore, the vanishing line defined for that face will be formed by the vertical traced by VP. Consequently, any pair of edges of the graph which are cut on the vanishing line, and at least one of them goes through a vertex of the edges that define the orientation of the face, are capable of representing an edge on the face and another one parallel to it. This is the case, in this example, of the edges 2’–3’ and 7’–8’ that being cut in T represent the projections of the edge 2–3 which is part of the face defined by 1–2 and 1–5 and the edge 7–8 which is an edge parallel to that one in the model. The angle that the vanishing line in T (joining of VT) forms with the ground plane will give us the angle that the edge 2–3 forms with the horizontal and, moreover, it will coincide with the angle formed by the straight line that joins the metric point of VP with the point T. Once that angle is determined, a straight line parallel to VT can be traced through the already known vertex 2, which allows us to place the vertex 3 in its intersection point with the straight line traced from V to 3’. Solving in such a way for the edge 7’–8’ we can obtain the coordinates of the vertex 8 and, as a result, the model is reconstructed. Therefore, once all the possible coordinates of the vertexes of the model have been obtained, using the expansion method through the edges oriented according to the main directions, it is possible to implement a new algorithm in order to continue with the expansion of the reconstruction through other edges as follows: (1) For each vertex Vi of the graph with already known coordinates in the model, the pair of edges A1 and A2, that with vertex in Vi have end vertexes VA1 and VA2 with coordinates
46
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 13. Expansion of the reconstruction through edges non-parallel to the main directions in oblique perspectives with two vanishing points.
also known in the model, are analyzed (those vertexes define the orientation of a face). In the example in Fig. 13 a vertex that fulfills these conditions could be vertex 1, given that the coordinates of the vertexes of the edges 1–2 and 1– 6 are known by the expansion of the method through edges oriented according to main directions. (2) For each end vertex VA1 and VA2 the edges that derive from them and are not parallel to the main directions are analyzed and it is checked their convergence with any other in a vanishing point aligned with the vanishing line defined by edges A1 and A2 . In the example in Fig. 13, from vertex 2’ the edge 2´–3’starts and converge with the edge 7’–8’ in the point T which is placed on the vanishing line defined by the edges 1’–2’ and 1’–6’. (3) For each one of these edges the coordinates of the vertex as the intersection of straight lines commented above are calculated. In the example in Fig. 13 the coordinates of the vertex 3 are calculated as the intersection of the straight line traced by 2, parallel to VT and the straight line V–3’. The method stated expands perspectives with three vanishing points considering, in this case, that the vanishing lines are defined by the edges that define the triangle of three vanishing points as there is not an inappropriate vanishing point of the edges parallel to the main directions. 5.2. Reconstruction of graphs that represent models in parallel conical perspective by expansion through edges non-parallel to the main directions In the case of parallel conical perspectives, given the existence of two inappropriate vanishing points corresponding to the edges represented in the graph as horizontal and vertical, the edges in a model parallel to one of the orthogonal planes defined by the main directions converge in a different way, so it is necessary to distinguish the following cases: 5.2.1. Edges oblique parallel to the picture plane This is the most obvious case. In the example in Fig. 14 the pairs of edges parallel to each other 1–5, 6–10 and 4–5, 9–10 are
parallel to the picture plane and are projected according to the straight lines 1’–5,’ 6’–10’ and 4’–5’, 9’–10’ parallel to each other on the picture plane vanishing to an inappropriate point. The angle that the projections form with the horizontal traced in the picture plane coincides with the angle which those edges form in the three-dimensional model with the ground plane. Therefore, once we have obtained the coordinates of the vertexes 1, 2, 3, 4, 6, 7, 8 and 9 by the expansion method through edges oriented according to main directions, we can obtain the coordinates of the vertexes 5 and 10 as intersection of the straight lines traced by 1 and 6 parallel to the picture plane forming an angle a with the ground plane, and the straight line that joins the projections 5’ and 10’ with the viewpoint V. The algorithm implemented, once the coordinates of the vertexes have been obtained by expansion through edges oriented according to the main directions, carries out the following verifications:
(1) For each vertex Vi of the graph with known coordinates in the model, the pairs of edges A1 and A2 horizontal and vertical of the graph, that with vertex in Vi have their end vertexes VA1 and VA2 with coordinates also known in the model, are analyzed. In Fig. 14 a vertex that fulfills these conditions could be the edge 2, since the coordinates of the vertexes of the edges 2–3 and 2–1 are known by expansion of the reconstruction through edges oriented according to main directions. (2) For each vertex VA1 and VA2 it is analyzed if the edges nonparallel to the main directions which converge in those vertexes in the graph are parallel to another oblique edge whose starting or final vertex is aligned with the vertex VA1 or VA2 respectively and the main point. In Fig. 14, in the vertex 1’ the edge 1´– 5’ parallel to 6’–10’ converges, and the vertexes 1’ and 6’ are aligned with the main point. (3) For each one of these edges the coordinates of the remaining vertex as the intersection of the straight lines commented above are calculated. In Fig. 14 the coordinates of the vertex 5 are calculated as the intersection of the straight line traced by 1 parallel to the picture plane forming an angle a with the ground plane and the straight line V–5’.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
47
Fig. 14. Expansion of the reconstruction through edges parallel to the picture plane in parallel conical perspectives.
5.2.2. Oblique edges contained in planes perpendicular to the picture plane and the ground plane In Fig. 15 a model is represented in which the edges 1–5, 6–10 and 4–5, 9–10 are parallel to each other and on planes perpendicular to the picture plane and the ground plane. The above-mentioned projections of the pairs of edges 1’–5’, 6’– 10’ and 4’–5’, 9’10’ converge to the vanishing points T1 and T2 respectively which are on the vertical traced by the main point. The straight lines traced from the viewpoint to the vanishing points form with the horizontal angles a and β which coincide with the angle that the edges of the model form with the ground plane. Therefore, knowing the coordinates of the vertexes 1, 2, 3, 4, 6, 7, 8 and 9 by expansion through edges oriented according to the main directions, we can obtain the coordinates of the vertexes 5 and 10 as intersection of the straight lines that traced by 4 and 9 on planes perpendicular to the picture plane and the ground plane form an angle a with the ground plane and the straight line that joins the projections 5’ and 10’ with the viewpoint. The algorithm implemented, once all the possible coordinates of the vertexes of the model have been obtained by expansion through edges oriented according to the main directions, carries out the following verifications: (1) For each vertex Vi of the graph with known coordinates in the model, the pairs of edges of the graph A1 and A2 , a vertical and another one going through the main point, that containing Vi have end vertexes VA1 and VA2 with coordinates also known in the model, are analyzed. In Fig. 15 a vertex that fulfills these conditions could be the vertex 3, since the coordinates of the vertexes of the edges defined by 3–2 and 3–4 are known by the expansion of the method through edges oriented according to main directions. (2) For each end vertex VA1 and VA2 of these edges, it is analyzed if the edges non-parallel to the main directions that converge in those vertexes of the graph converge with any other edge in a vanishing point aligned according to the vertical traced by the main point. In Fig. 15, in the vertex 4’ the edge 4´–5’, that converges with 9’–10’ in T1 (point aligned with the vertical traced by the main point) converges. (3) For each one of these edges the coordinates of its vertex as the above-mentioned intersection of the straight lines are
calculated. In Fig. 15 the coordinates of the vertex 5 are calculated as the intersection of the straight line traced by 4 on a plane perpendicular to the picture plane and the ground plane forming an angle a with the ground plane and the straight line V–5’. 5.2.3. Edges oblique parallel to the ground plane In Fig. 16 we represent a model with pairs of edges 2–3, 8–9 and 4–5, 10–11 parallel to the ground plane. Their projections 2’– 3’, 8’–9’ and 4’–5’, 10’–11’ converge to the vanishing points T1 and T2 aligned with the horizontal traced by the main point. The straight lines traced from the viewpoint to the vanishing points and the main point define angles a and β which coincide with the angles that the edges of the model form with the picture plane. Therefore, once we have obtained the coordinates of the vertexes 3, 4, 9 and 10 by the expansion method through edges oriented according to main directions, we can obtain the coordinates of the vertexes 2, 5, 8 and 11 as the intersection of the straight lines traced by 3, 4, 9 and 10 contained in planes parallel to the ground plane and that form an angle a or β with the picture plane, and the straight line that joins the projections 3’, 4’ 9’ and 10’ with the viewpoint. The algorithm implemented, once all the possible coordinates of the vertexes of the model have been obtained by expansion through edges oriented according to the main directions, carries out the following verifications: (1) For each edge Ai vertical defined by vertexes VA1 and VA2 with known coordinates in the model, it is analyzed the existence of edges that going through VA1 and VA2 converge in a vanishing point aligned according to the horizontal traced by the main point. In Fig. 16 the edge 3’–9’ fulfills these conditions since the coordinates of the vertexes 3 and 9 are known by the expansion of the method through edges oriented according to main directions, and from vertexes 3 and 9 the pairs of edges 2’–3’ and 8’–9’ start, which converge in the point T1 aligned according to the horizontal traced by the main point. (2) For each one of these edges the coordinates of its vertex as the above-mentioned intersection of the straight lines
48
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 15. Expansion of the reconstruction through edges perpendicular to the picture plane and the ground plane in parallel conical perspectives.
Fig. 16. Expansion of the reconstruction through edges parallel to the ground plane in parallel conical perspectives.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
49
Fig. 17. Vectorization of a photograph.
Fig. 18. A detailed operation example.
are calculated. In Fig. 16 the coordinates of the vertex 2 are calculated as the intersection of the straight line traced by 3 parallel to the ground plane forming an angle a with the picture plane and the straight line V–2’.
the scope of this paper, has been investigated for different purposes by several authors [35–37]. The graph, resulting from the vectorization, constitutes the starting information for our algorithms which are executed in the following way (see Fig. 18):
6. A detailed operation example For further clarification of the proposed algorithms, in this section we show, step by step, the reconstruction process of a model from a perspective with three vanishing points. As a preliminary step for the reconstruction of an image we need its vectorization; that is, to transform the information given in the form of bitmap into a vector image defined by vertexes and edges. This process, which is outlined in Fig. 17, which is outside
Step 1: The intersections of the edges which form the graph are analyzed according to the conditions given in Section 4, and the type of perspective is determined. In the present case, given that there are three vanishing points which define a triangle whose ortho-center coincides with the main point and, moreover, the geometrically obtained focal length coincides with the known focal length of the image, the perspective is defined as the type of three vanishing points being
50
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 19. Method used to obtain graphs of models.
the main directions those which vanish to points VP1 , VP2 and VP3 . Step 2: The reconstruction process starts in the edge 1–2 oriented according to one of the main directions determining the three-dimensional coordinates of its vertexes as indicated in Section 3. Subsequently, and in accordance with the same section, the reconstruction is expanded through the edges parallel to the main directions obtaining the coordinates of the vertexes 3 and 4. Step 3: At this stage, the method cannot continue expanding according to edges parallel to the main directions, so other possible ways are examined. In our case, the edges 1–5 and 2–6 converge on a point aligned with the vanishing points VP1 and VP2 and, moreover, both edges go through vertexes with known three-dimensional coordinates so, according to Section 5.1, the coordinates of the vertexes 5 and 6 are calculated. Step 4: Knowing the coordinates of the edge 5–6, the reconstruction method can continue expanding according to edges parallel to the main directions obtaining the coordinates of the vertexes 7, 8, 9, 10 and 11 ending, in this way, the reconstruction process. 7. Experimental work and results Given that a vectorization algorithm has not been implemented in this work, the capture of images has been carried out from software that allows the positioning of cameras and the exportation of perspectives to dxf format. In the different examples tested the framework that defines the size of the picture has been added, and the focal length has been modified using angle lens of 24, 35 and standard of 50 mm, being the width/height coefficient of 4:3. In Fig. 19 the graph obtained following this process is shown. In it, the positioning of the camera in the software used, the image captured by the camera and the graph obtained once the image is exported to dxf format can be seen. The methods proposed have been implemented in Visual C# under the platform of Visual Studio (WinForm) using the libraries CsGl (OpenGl for Visual C#).
In Figs. 20 and 21 we show the 80 examples analyzed in order to validate the algorithm proposed for the reconstruction of conical perspectives with three vanishing points. Those examples correspond to projections of the models presented in [33] in the case of oblique and parallel perspectives with two vanishing points. Moreover, the 160 examples presented in that publication have been analyzed again to evaluate the criteria proposed for the analysis of the main directions and the type of conical perspective in point 4 in this document. In Figs. 22–24 we show the 96 examples in order to validate the algorithms for the reconstruction of flat shapes and the expansion method through edges non-parallel to the main directions. Those examples correspond to parallel and oblique conical perspectives with two and three vanishing points in 32 new models proposed. In the settings of the application shown in Fig. 25, two windows have been defined. In one of them (left window), you may observe the starting graph where it is possible to see the detected vanishing points, the main directions, the graph typology and the type of perspective detected. In the other one (on the right), we show the model obtained from the reconstruction of the graph. This window has a tool that allows us to rebuild the graph step by step. The success rate in the graphs tested is of 100%. Moreover, although in a wide majority the graphs have been represented with hidden edges, tests with graphs without hidden edges have been carried out obtaining their correct representation, although, obviously, the final model does not constitute a solid but a model without depth. 8. Conclusions In this work we show the latest advances achieved in obtaining polyhedral models from conical perspectives. We present a set of criteria that allow analyzing in an automatic way if a certain graph corresponds to a parallel, oblique conical or with three vanishing points projection. A new method for the reconstruction of conical perspectives with three vanishing points is proposed and we have defined a methodology that allows the expansion of the calculation of the coordinates of a model from a projection, not only through edges
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 20. Graphs in conical perspective with three vanishing points (I).
51
52
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 21. Graphs in conical perspective with vanishing points (II).
oriented according to directions perpendicular to each other, but it also allows doing it from other edges which do not fulfill these conditions. Moreover, we include projections of flat shapes as a possible input to the algorithm. None of the methods implemented requires the defined faces of the model as starting information, which makes these strategies a completely automatic system that does not need to interact with the user.
To sum up, the necessary methodologies for the completely automatic reconstruction of three-dimensional models from parallel and oblique conical perspectives and with three vanishing points have been defined for a wide family of polyhedral models. The methods have been implemented in Visual C# under the platform of Visual Studio (WinForm) using the libraries CsGl (OpenGl for Visual C#) carrying out a total of 336 tests obtaining
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 22. New graphs in parallel conical perspective.
53
54
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 23. New graphs in oblique conical perspective with two vanishing points.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 24. New graphs in conical perspective with three vanishing points.
55
56
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57
Fig. 25. Work environment.
a success rate of 100%. The results achieved reveal the interest of the method proposed in the field of obtaining models from conical perspectives.
References [1] Robert LG. Machine perception of three-dimensional solids. Optical and electro-optical information processing, Cambridge, Massachusetts and London, England: The MIT Press; 1965. [Chapter 9]. [2] Zou CQ, Chen SF, Fu HB, Liu JZ. Progressive 3D reconstruction of planar-faced manifold objects with DRF-based line drawing decomposition. IEEE Trans Vis Comput Graph 2015;21(2):252–63. doi:10.1109/TVCG.2014.2354039. [3] Huffman DA. Impossible objects as nonsense sentences. Mach Intell 1971;6:295–323. [4] Clowes MB. On seeing things. Artif Intell 1971;2:79–116. [5] Thorpe C, Shafer S. Correspondence in line drawings of multiple views of objects. In: Proceedings of the eighth international joint conference on artificial intelligence, vol. 2; 1983. [6] Wang W, Grinstein G. A survey of 3D solid reconstruction from 2D projection line drawings. Comput Graph Forum 1993;12(2):137–58. [7] Malik J. Interpreting line drawing of curved objects. Int J Comput Vis 1987;1:73–103. [8] Varley P, Martin R. A system for constructing boundary representation solid models from a two-dimensional sketch. Frontal geometry and sketch categorisation. In: Proceedings of the 1st Korea_UK joint workshop on geometric modeling and computer graphics; 20 0 0. [9] Varley P, Martin R. TA system for constructing boundary representation solid models from a two-dimensional sketch. Topology of hidden parts. In: Proceedings of the 1st Korea_UK joint workshop on geometric modeling and computer graphics; 20 0 0. [10] Parodi P, Piccioli G. 3D shape reconstruction by using vanishing points. IEEE Trans Pattern Anal Mach Intell 1996;18(2):211–17. [11] Lamb D, Bandopadhay A. Interpreting a 3D object from a rough 2D line drawing. In: Proceedings of Visualization’90; 1990. p. 59–66. [12] Marill T. Emulating the human interpretation of line-drawings as three-dimensional objects. Int J Comput Vis 1991;6(2):147–61. [13] Baird LC, Wang PS. 3D object recognition using gradient descent and the universal 3D ray grammar. In: Proceedings of SPIE conference on intelligent robots and computer vision X: algorithms and techniques, vol. 1607, Boston, MA; 1991. [14] Leclerc Y, Fischler M. An optimization – based approach to the interpretation of single line drawings as 3D wire frames. Int J Comput Vis 1992;9(2):113–36. [15] Lipson H, Shpitalni M. A new interface of conceptual design based on object reconstruction from a single freehand sketch. Ann CIRP 1995;44(1):133–6.
[16] Lipson H, Shpitalni M. Optimization-based reconstruction of a 3D object from a single freehand line drawing. Comput Aided Des 1996;28(8):651–63. [17] Sphitalni M, Lipson H. Identification of faces in a 2D line drawing projection of a wireframe object. IEEE Trans Pattern Anal Mach Intell 1996;18(10):10 0 0–12. [18] Sphitalni M, Lipson H. Classification of sketch strokes and corner detection using conic sections and adaptive clustering. J Mech Des 1997;119(1):131–5. [19] Company Calleja P, Contero González M, Conesa Pastor J, Piquer Vicent A. An optimisation-based reconstruction engine for 3D modelling by scketching. Comput Graph 2004;28(6) pp. 995–979. [20] Lee YT, Fang F. A new hybrid method for 3D object recovery from 2D drawings and its validation against the cubic corner method and the optimisation-based method. Computer-Aided Des 2012;44(11):1090–102. [21] Chu J, Gao M, Zhang G, Feng R. Line-Based Optimization for 3D object reconstruction from single line drawings. In: Proceedings of international conference on information technology for manufacturing systems; 2010. p. 910–15. [22] Liu JZ, Cao LL, Li ZG, Tang XO. Plane-based optimization for 3D object reconstruction from single line drawings. IEEE Trans Pattern Anal Mach Intell 2008;30(2):315–27. [23] Liu JZ, Lee YT, Cham W. Identifying faces in a 2D line drawing representing a manifold object. IEEE Trans Pattern Anal Mach Intell 2002;24(12):1579–93. [24] Liu JZ, Tang XO. Evolutionary search for faces from line drawings. IEEE Trans Pattern Anal Mach Intell 2005;27(6):861–72. [25] Chen Y, Liu J, Tang X. A divide-and-conquer approach to 3D object reconstruction from line drawings. In: Proceedings of the 11th IEEE international conference on computer vision; 2007. p. 1807–14. [26] Liu J, Chen Y, Tang X. Decomposition of complex line drawings with hidden lines for 3D planar-faced manifold object reconstruction. IEEE Trans Pattern Anal Mach Intell 2011;33(1):3–15. [27] Xue T, Liu J, Tang X. Object cut: complex 3D object reconstruction through line drawing separation. In: Proceedings of 23rd IEEE conference on computer vision and pattern recognition (CVPR); 2010. p. 1149–56. [28] Yang L, Liu J, Tang X. Complex 3D general object reconstruction from line drawings. In: Proceedings of IEEE international conference on computer vision (ICCV); 2013. p. 1433–40. [29] Zou CQ, Chen SF, Fu HB, Liu JZ. Progressive 3D reconstruction of planar-faced manifold objects with DRF-based line drawing decomposition. IEEE Trans Vis Comput Graph 2015;21(2):252–63. doi:10.1109/TVCG.2014.2354039. [30] Fang F, Lee YT. Efficient decomposition of line drawings of connected manifolds without face identification. Comput Aided Des 2014;51:18–30. [31] Leong MC, Lee YT, Fang FA. Search-and-validate method for face identification from single line drawings. IEEE Trans Pattern Anal Mach Intell 2013;35(11):2576–91. [32] Fang F, Lee YT. 3D reconstruction of polyhedral objects from single perspective projections using cubic corner. 3D Res 2012;3(2):1–8 No. 59. [33] Conesa J, Mula F. Obtaining three-dimensional models from conical perspectives. Adv Eng Softw 2015;90:83–97. doi:10.1016/j.advengsoft.2015.07.002.
J. Conesa, F. Mula / Advances in Engineering Software 98 (2016) 38–57 [34] Rother C. A new approach to vanishing point detection in architectural environments. Image Vis Comput 2002;20(9–10):647–55. [35] Colombo C, Del Bimbo A, Pernici F. Metric 3D reconstruction and texture acquisition of surfaces of revolution from a single uncalibrated view. IEEE Trans Pattern Mach Intell 2005;27(1):99–114.
57
[36] Suman Kumar M, Hussein MY, Hicham Badri. Reconstructing an image from its edge representation. Digit Signal Process 2013;23:1867–76. [37] Vouzounaras G, Daras P, Strintzis MG. Automatic generation of 3D outdoor and indoor building scenes from a single image. Multimed Tools Appl 2014;70(1):361–78.