- Email: [email protected]
Ontology Evaluation: An Example of Delay Analysis
Available online at www.sciencedirect.com
ScienceDirect Procedia Engineering 85 (2014) 61 – 68
&UHDWLYH&RQVWUXFWLRQ&RQIHUHQFH&&
2QWRORJ\HYDOXDWLRQ $QH[DPSOHRIGHOD\DQDO\VLV *R]GH%LOJLQD ,UHP'LNPHQE07DODW%LUJRQXOE a
Research Assistant, Department of Civil Engineering, Middle East Technical University, Ankara, 06800, Turkey b Professor, Department of Civil Engineering, Middle East Technical University, Ankara, 06800, Turkey
$EVWUDFW 2QWRORJLHV KDYH D SODFH LQ NQRZOHGJH VKDULQJ ZLWK WKHLU DELOLW\ WR FDSWXUH UHDO ZRUOG LQIRUPDWLRQ LQ D PDFKLQH UHDGDEOH HQYLURQPHQW7KHLUDSSOLFDWLRQYDULHVIURPDQ\DSSURDFKVWUXFWXUHGRQRQWRORJLFDOFRPPLWPHQWWRPHWDOHYHORQWRORJLHVVHUYLQJ IRU YDULHW\ RI SXUSRVHV 7KLV YDULHW\ FRQWLQXHV DOVR LQ PHWKRGRORJLHV IRU FRQVWUXFWLQJ WKH RQWRORJLHV 7KHUH DUH ZLGH UDQJH RI RQWRORJLHVGLIIHULQJLQVL]HDQGFRPSOH[LW\WKDWDUHFUHDWHGZLWKGLIIHUHQWGHVLJQFULWHULD7KHPDLQSRLQWZLWKWKHRQWRORJLHVLV WKHLUVWUXFWXUHDQGDELOLW\WRVHUYHIRUWKHLQWHQGHGSXUSRVH7KLVLVRQO\JUDQWHGZLWKWKHFRQWLQXRXVHYDOXDWLRQRIRQWRORJLHVLQ WKH FRQVWUXFWLRQ SURFHVV DQG DOVR EHIRUH UHOHDVH RU UHXVH RI WKH RQWRORJ\ 2QWRORJ\ HYDOXDWLRQ PHWKRGV FOXVWHU DURXQG WZR FRQFHSWV VXFK DV YHULILFDWLRQ PHWKRGV WKDW HQVXUH WKH VWUXFWXUH RI WKH RQWRORJ\ DQG YDOLGDWLRQ PHWKRGV WKDW H[DPLQH WKHLU DSSOLFDELOLW\LQUHDOZRUOG,QOLJKWRIWKHVHGLIIHUHQWDSSURDFKHVDVTXDQWLWDWLYHDQGTXDOLWDWLYHPHWKRGVDUHGHSLFWHGLQOLWHUDWXUH IURPVXEMHFWLYHHYDOXDWLRQVE\H[SHUWVWRWRROVDOORIZKLFKLQYHVWLJDWHGLIIHUHQWFKDUDFWHULVWLFVRIRQWRORJLHV 2EMHFWLYHVRIWKLVVWXG\DUHWRXQGHUOLQHWKHLPSRUWDQFHRIRQWRORJ\HYDOXDWLRQDQGWRSUHVHQWDQHPSOR\HGYDOLGDWLRQPHWKRG XVHGGXULQJRQWRORJ\GHYHORSPHQWIRUGHOD\DQDO\VLV,QWKLVFRQWH[WILUVW OLWHUDWXUHUHYLHZRQRQWRORJ\HYDOXDWLRQLVSUHVHQWHG 6HFRQGDGHOD\DQDO\VLVRQWRORJ\LVLQWURGXFHGZLWKLWVEDVLFVWROD\WKHIRXQGDWLRQVRIWKHVWXG\&DVHVWDNHQIURPWKH7XUNLVK FRQVWUXFWLRQ LQGXVWU\ DUH H[HPSOLILHG WR H[SODLQ WKH XWLOL]HG RQWRORJ\ YDOLGDWLRQ SURFHVV &RPSDULVRQ RI WKH FRQFHSWV LQ WKH FRQVWUXFWHG RQWRORJ\ ZLWK WKH H[SHUW UHSRUWV ZULWWHQ IRU HDFK FDVH LV XVHG WR HYDOXDWH WKH RQWRORJ\ )XUWKHUPRUH DOWHUQDWLYH YDOLGDWLRQWHFKQLTXHVDUHGLVFXVVHGDVZHOODVSRVVLEOHDWWHPSWVWRNHHSLWUHVSRQVLYHDQGXSWRGDWH
© 2014 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license 7KH$XWKRUV3XEOLVKHGE\(OVHYLHU/WG (http://creativecommons.org/licenses/by-nc-nd/3.0/). 3HHUUHYLHZXQGHUUHVSRQVLELOLW\RIWKH'LDPRQG&RQJUHVV.IW Peer-review under responsibility of the Diamond Congress Kft. Keywords:FRQVWUXFWLRQVHFWRU GHOD\DQDO\VLV RQWRORJ\ RQWRORJ\HYDOXDWLRQ RQWRORJ\YDOLGDWLRQ
&RUUHVSRQGLQJDXWKRU7HO ID[ E-mail address:JELOJLQ#PHWXHGXWU
1877-7058 © 2014 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/3.0/). Peer-review under responsibility of the Diamond Congress Kft. doi:10.1016/j.proeng.2014.10.529
62
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
,QWURGXFWLRQ 2QWRORJLHVRFFXS\DSURPLQHQWSODFHLQNQRZOHGJHVKDULQJ7KHRQWRORJ\FRQFHSWLVIOH[LEOHDQGLWVDGDSWDELOLW\ OHDGV YDULHW\ DQGVRSUHYHQWV D FRPPRQ RQWRORJ\GHYHORSPHQW RUHYDOXDWLRQ PHWKRG WR EH DFFHSWHG 7KHLUXVDJH DUHDLVZLGHDQGWKH\DUHGHYHORSHGWRUHVSRQGPDQ\QHHGLQWKHDUHD7KHVRXQGQHVVRIDQRQWRORJ\EHWZHHQDOOWKH FUHDWHGRQHVLVPDQLIHVWHGRQO\E\LWVHYDOXDWLRQ$FFRUGLQJO\HYDOXDWLRQLVKLJKO\UHFRPPHQGHGGXULQJZKROHOLIH F\FOH RI DQ RQWRORJ\ ,Q WKLV FRQWH[W PDQ\ HYDOXDWLRQ PHWKRGV DUH DYDLODEOH WKDW DUH UHVSRQVLYH IRU GLIIHUHQW HYDOXDWLRQDLPV $QRQWRORJ\RIGHOD\DQDO\VLVLQFRQVWUXFWLRQVHFWRUXQGHUOLHVWKLVVWXG\7KLVLQHYLWDEOHSUREOHPRIFRQVWUXFWLRQ VHFWRULVWULHGWREHKDQGOHGLQWKLVZD\WRLQFUHDVHNQRZOHGJHLQWKHLVVXHDQGWRIRUPDEDVHIRUDQ\DWWHPSWWKDW ZRXOG EH KHOG LQ PDFKLQHV HQYLURQPHQW 3RVVLEOH XVH RI WKH RQWRORJ\ E\ FRQVWUXFWLRQ FRPSDQLHV LQ SURMHFW ULVN DQGRUFODLPPDQDJHPHQW DQGDVDGDWDEDVHIRUWKHFRPSDQLHVLVH[SHFWHGE\WKLVRQWRORJ\7RPHHWWKLVREMHFW YDOLGDWLRQ E\ UHDO H[SHUW UHSRUWV RQ GHOD\ DQDO\VLV LV VHOHFWHG WR HQVXUH WKH VRXQGQHVV RI WKH OLWHUDWXUH EDVHG RQWRORJ\ ,QWKLVSDSHUOLWHUDWXUHUHYLHZRQRQWRORJ\HYDOXDWLRQPHWKRGVLVKDQGOHGDQGWKHIROORZHGYDOLGDWLRQPHWKRGLV SUHVHQWHGWRVKRZWKHYDULHW\DQGLPSRUWDQFHRIRQWRORJ\HYDOXDWLRQPHWKRGV 5HVHDUFKEDFNJURXQG ,Q WKLV VHFWLRQ ILUVWO\ RQWRORJ\ LV GHILQHG DQG IROORZLQJ WKDW OLWHUDWXUH UHYLHZ RQ LWV HYDOXDWLRQ LV SUHVHQWHG WR UHYHDOLWVH[WHQW 2.1. Ontology 2QWRORJLHVKDYHEHHQZLGHO\XVHGLQNQRZOHGJHHQJLQHHULQJVLQFH VWRPHHWWKHQHHGRIXQLILHGIRUPVIRU LQIRUPDWLRQ VKDULQJ WR SURYLGH FRPPRQ XQGHUVWDQGLQJ DQG UHXVH >@ :LWK LWV ZLGHO\ NQRZQ GHILQLWLRQ DQ RQWRORJ\LV³DQH[SOLFLWVSHFLILFDWLRQRIDFRQFHSWXDOL]DWLRQ´>@&RUFKRHWDO>@SUHVHQWDXQLILHGGHILQLWLRQDV ³RQWRORJLHV DLP WR FDSWXUH FRQVHQVXDO NQRZOHGJH LQ D JHQHULF DQG IRUPDO ZD\ DQG WKDW WKH\ PD\ EH UHXVHG DQG VKDUHGDFURVVDSSOLFDWLRQV VRIWZDUH DQGE\JURXSVRISHRSOH´$GGLWLRQDOO\*DQJHPLHWDO>@VWUXFWXUDOO\GHILQH DQRQWRORJ\DV³DJUDSKZKRVHQRGHVDQGDUFVUHSUHVHQWFRQFHSWXDOL]DWLRQV´%DVLFDOO\LWLVWKHJUDSKRIPHWDGDWD UDQJLQJIURPWKHVDXULWD[RQRPLHVDQGD[LRPDWL]HGWKHRULHV7KHWD[RQRPLF VXEFODVVRILVD UHODWLRQVKLSVRIDQ RQWRORJ\FRQVWLWXWHWKHGHSWKDQGEUHDGWKRIDQRQWRORJ\ZKHUHDVQRQWD[RQRPLFUHODWLRQVFRQVWLWXWHWKHGHQVLW\RI WKH RQWRORJ\ ,W LV WKH GH[WHULW\ RI LWV GHYHORSHU WR NHHS WKH FRQFHSWXDOL]DWLRQ LQ D VHPDQWLFDOO\ IRUPDOL]HG IRUP QDPHO\LQWKHJUDSK>@2QWRORJLHVDUHEDVLFDOO\WD[RQRPLFWUHHRIFRQFHSWXDOL]DWLRQVIURPJHQHUDOREMHFWVDWWKH WRSOHYHOWRWKHVSHFLILFRQHVDWWKHERWWRPOHYHO>@:KHQWD[RQRPLHVDUHLPSURYHGZLWKPRUHLQIRUPDWLRQWKURXJK UHODWLRQV DQG UXOHV IRU PRGHOLQJ D GRPDLQ WKH\ EHFRPH RQWRORJ\ >@ $FFRUGLQJO\ RQH LV IUHH WR FRQVWUXFW DQ\ OLJKWZHLJKW JHQHUDOO\ LQFOXGLQJ FRQFHSWV FRQFHSW WD[RQRPLHV UHODWLRQVKLSV EHWZHHQ FRQFHSWV DQG SURSHUWLHV RI FRQFHSWV RUKHDY\ZHLJKW DGGLWLRQDOO\ LQFOXGLQJ D[LRPV DQG FRQVWUDLQWV RQWRORJ\ WKDW ZRXOGEH VWUXFWXUHG WR EH UHVSRQVLYHWRDQ\QHHG>@7KHUHDUHYDULRXVRQWRORJ\FRQVWUXFWLRQPHWKRGVUDQJLQJIURPFRQVWUXFWLRQIURPVFUDWFK WRPHUJLQJH[LVWLQJRQWRORJLHVDQGJHQHUDWLQJRQWRORJLHVZLWKDXWRPDWLFFRQVWUXFWLRQSULQFLSOHV RQWRORJ\OHDUQLQJ >@ ,Q UHVSRQVH WR WKLV WKHUH DUH ZLGH UDQJH RI HYDOXDWLRQ PHWULFV DYDLODEOH SUHVHQWHG WR HYDOXDWH WKH LQWHQGHG VWUXFWXUH DQG IXQFWLRQDOLW\ RI WKHRQWRORJ\> @ 0RUHRYHUUHVHDUFKHUV WKDWSURSRVHGHYHORSPHQW VWUDWHJLHV DOVR PHQWLRQ VRPH GHVLJQ DQG HYDOXDWLRQ FULWHULD IRU RQWRORJLHV OLNH WKH RQHV SUHVHQWHG E\ )R[ DQG *UXQLQJHU >@ *RPH]3HUH]>@ DQG*UXEHU>@ 2.2. Ontology evaluation :LWKZLGHVSUHDGXVDJHRIRQWRORJLHVLQNQRZOHGJHUHSUHVHQWDWLRQGHYHORSPHQWDQGUHXVHRIRQWRORJLHVEULQJ XS TXHVWLRQ RI HYDOXDWLRQ RI RQWRORJLHV 7KHUH DUH YDULRXV HYDOXDWLRQ PHWKRGV WKDW VHDUFK IRU GLIIHUHQW PHDVXUHV
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
ZLWKTXDOLWDWLYHRUTXDQWLWDWLYHPHWULFVIRUGLIIHUHQWRQWRORJLHVKRZHYHUQRJOREDODSSURDFKKDVEHHQDGRSWHG>@ $VLWLVSUHYLRXVO\VWDWHG RQWRORJLHVYDU\LQWKHLUGRPDLQVL]HSXUSRVHODQJXDJHDQGPD\EHKDQGFDUYHGPD\EH FRQVWUXFWHG E\ VFKRODUV RU GRPDLQ H[SHUWV RU PD\ EH REWDLQHG E\ DQ DXWRPDWLF RU VHPLDXWRPDWLF SURFHVV 7KLV VLWXDWLRQOHDGVHYDOXDWLRQRIRQWRORJLHVWREHFRPSOLFDWHGDQGSUHYHQWVUHOHDVHRIDVWDQGDUGHYDOXDWLRQSURFHGXUH >@ $GGLWLRQDOO\ HYDOXDWLRQ LV UHTXLUHG GXULQJ ZKROH OLIHF\FOH RI DQ RQWRORJ\ QDPHO\ GXULQJ SUHPRGHOOLQJ GXULQJ PRGHOOLQJ EHIRUH UHOHDVH DQG DIWHU UHOHDVH >@ 0DQ\ VWXGLHV KHOG E\ GLIIHUHQW UHVHDUFKHUV LQ WKH RQWRORJ\HYDOXDWLRQDUHDDGGUHVVWKHSUREOHPDQGSRLQWXSLPSRUWDQFHRIWKHLVVXH7KHUHDUHPHWKRGVWKDWDUHHLWKHU PDQXDORUDXWRPDWLFHLWKHUGHDOLQJZLWKWD[RQRP\RUHYDOXDWLQJWKHFRQWHQWDQGHLWKHUGHSHQGHQWWRVRPHVSHFLILF WRROODQJXDJH RU LQGHSHQGHQW >@ 7KLV GRXEOH YDULHW\ RI RQWRORJ\ GHYHORSPHQW DQG HYDOXDWLRQ LQFUHDVHV WKH FRPSOH[LW\ LQ WKH DUHD DQG NHHSV WKH HYDOXDWLRQ SUREOHP XQVROYHG >@ $FFRUGLQJO\ WR REWDLQ D UHOLDEOH RQWRORJ\FRPSOHPHQWDU\XVHRIGLIIHUHQWHYDOXDWLRQPHWKRGVLVDSSUHFLDWHG>@ 2.2.1. Classification of ontology evaluation methods 7KHUHDUHYDULRXVFODVVLILFDWLRQDWWHPSWVRQHYDOXDWLRQPHWKRGVDQGWRROVWKHGHWDLOVRIZKLFKDUHJLYHQEHORZ 7KHFRPPRQSRLQWEHWZHHQWKHPLV LQJHQHUDOWHUPVWKH\DUHHLWKHUverification methodsWKDWHQVXUHWKHRQWRORJ\LV FRQVWUXFWHGFRUUHFWO\RUvalidation methodsWKDWHQVXUHWKHRQWRORJ\UHSUHVHQWVWKHUHDOZRUOG> @ +DUWPDQQHWDO>@FODVVLI\RQWRORJ\HYDOXDWLRQIRUWKHPHWKRGVDV OntoMetric±$QDO\WLF+LHUDUFK\3URFHVV EDVHGPHWKRGWKDWSUHVHQWVVHWRIPHDVXUHVWRVHOHFWWKHEHVWRQWRORJ\IRUDSDUWLFXODUQHHG Natural Language Application Metrics ± PDLQO\ DVVRFLDWLQJ PHQWLRQV LQ WKH WH[W ZLWK WKH FRQFHSWV LQ WKDW RQWRORJ\ SUHFLVLRQ DQG UHFDOO PHWULFV FRVWEDVHG HYDOXDWLRQ PHWULF RQWRORJ\ ILW ± WKH WHQQLV PHDVXUH DQG OH[LFDO FRPSDULVRQ OHYHO PHDVXUH OntoClean ± HYDOXDWLRQ RI IRUPDO VWUXFWXUH RI WKH WD[RQRP\ E\ WKH RQWRORJLFDO QRWLRQV RI ULJLGLW\ XQLW\LGHQWLW\DQGGHSHQGHQFH EvaLexon±PDLQO\FUHDWLQJRQWRORJLHVIURPWH[WFRPSDULVRQRIWKHYRFDEXODU\ ZLWKWH[WFDOFXODWLRQRIFRYHUDJHDFFXUDF\SUHFLVLRQUHFDOO$GGLWLRQDOO\WKH\SUHVHQWWKHDSSOLFDWLRQVDQGWRROVDV ODEval±WRROWKDWGHWHFWVSRVVLEOHSUREOHPVLQFRQFHSWWD[RQRPLHV OntoManager±WRROWKDWHYDOXDWHVWKH WUXWKIXOQHVVRIDQRQWRORJ\ZLWKUHVSHFWWRWKHGRPDLQ *DQJHPL HW DO >@ LGHQWLI\ WKUHH FDWHJRULHV DV structural measures ± HYDOXDWLRQ RI IRUPDO VWUXFWXUH RI WKH RQWRORJ\ WKH JUDSK functional measures ± HYDOXDWLRQ RI WKH UHODWLRQ EHWZHHQ WKH RQWRORJ\ DQG WKH LQWHQGHG PHDQLQJ KRZ LW VHUYHV IRU WKH SXUSRVH LWV FRJQLWLYH VHPDQWLFV PDLQO\ SUHFLVLRQUHFDOOEDVHG PHDVXUHV DQG usability-profiling measures ± HYDOXDWLRQ RI WKH RQWRORJ\ SURILOH FRPPXQLFDWLRQ FRQWH[W RI WKH RQWRORJ\ LWV SUDJPDWLFV Structural measures DUH GHSWK EUHDGWK WDQJOHGQHVV IDQRXWQHVV VSHFLILF GLIIHUHQFH GHQVLW\ PRGXODULW\ORJLFDODGHTXDF\PHWDORJLFDODGHTXDF\GHJUHHGLVWULEXWLRQFunctional measuresDUHSUHFLVLRQUHFDOO DQG DFFXUDF\ ZKLFK DUH PHDVXUHG DJDLQVW H[SHUWV¶ MXGJPHQW RU D GDWD VHW DVVXPHG DV D TXDOLILHG H[SUHVVLRQ RI H[SHUWV¶ 7KH\ JLYH IXUWKHU GHWDLO IRU IXQFWLRQDO PHDVXUHV DV MXGJPHQW E\ DJUHHPHQW DVVHVVPHQW XVHU VDWLVIDFWLRQDVVHVVPHQW WDVNDVVHVVPHQW±ZKDWLVVXSSRUWHGE\WKHRQWRORJ\" VHUYLFHVSHFLILFDWLRQEDVHGJROG VWDQGDUGVROXWLRQEDVHGH[SOLFLWWDVNEDVHG WRSLFDVVHVVPHQW±ERXQGDULHVRIWKHNQRZOHGJHGRPDLQDGGUHVVHG E\ WKH RQWRORJ\ PRGXODULW\ DVVHVVPHQW ± EXLOGLQJ EORFNV IRU WKH GHVLJQ RI WKH RQWRORJ\ )LQDOO\ usabilityprofiling measuresDUHKDQGOHGDVUHFRJQLWLRQHIILFLHQF\DQGLQWHUIDFLQJ )HUQDQGH]%UHLVHWDO>@VWDWHWKDWRQWRORJ\HYDOXDWLRQLVVXHKDVEHHQKDQGOHGIURPGLIIHUHQWSHUVSHFWLYHVVRLW LV SRVVLEOH WR JURXS WKHP DV HYDOXDWLRQ ZLWK WKH DLP RI ranking correctness RU quality *HQHUDOO\ UDQNLQJ DQG FRUUHFWQHVVDUHHYDOXDWHGIRUWKHstructural property ZKHUHDVXVDELOLW\UHOLDELOLW\DQGIXQFWLRQDOLW\LVFRQVLGHUHGLQ WKH VHDUFK RI RQWRORJ\ quality 7KH\ XQGHUOLQH WKDW RQWRORJ\ GHYHORSPHQW SURFHGXUHV EHDU WUDFHV RI 6RIWZDUH (QJLQHHULQJDQGWKH\XQGHUWDNHWKHHYDOXDWLRQLVVXHIURPWKDWSHUVSHFWLYH ,62 VWDQGDUGIRUVRIWZDUHTXDOLW\ DVZHOO6RIWZDUHHYDOXDWLRQFRQVLVWVRIinternalexternalDQGin-use TXDOLW\PHDVXUHVInternal measuresFRQVWLWXWH WKH PHDVXUHV WKDW DUH UHODWHG ZLWK WKH VRIWZDUH LWVHOI DQG JHQHUDOO\ KHOG GXULQJ HDUO\ GHVLJQ SURFHVV ZKHUHDV external measures DUH WKH RQHV HYDOXDWHG GXULQJ WHVWLQJ RU RSHUDWLQJ SURFHVV LQ ODWHU GHVLJQ RU EHIRUH UHOHDVH SURFHVVDQGWKH\DUHJHQHUDOO\UHODWHGZLWKWKHEHKDYLRURIWKHVRIWZDUH,QOLJKWRIWKHVHWKH\SURSRVHDQRQWRORJ\ HYDOXDWLRQ IUDPHZRUN WKDW LV VWUXFWXUHG IURP VHYHQ TXDOLW\ GLPHQVLRQV DV structural functionality reliability usabilityefficiencymaintainabilityDQGquality in use
63
64
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
1HW]HUHWDO>@JURXSWKHRQWRORJ\HYDOXDWLRQSXUSRVHDVassessing the qualityE\LWVGHYHORSHUVRUranking the ontologies WR FKRRVH IRU VRPH SDUWLFXODU XVH $GGLWLRQDOO\ WKH\ JURXS WKH HYDOXDWLRQ PHWKRGV DV intrinsic DQG extrinsicPHWKRGVIntrinsic methodsIRFXVHYDOXDWLRQRIWKHVWUXFWXUDOSURSHUWLHV ZKHUHDVextrinsic methodsUHTXLUH VRPHH[WHUQDOLQIRUPDWLRQVXFKDVDFRUSXVWKDWUHSUHVHQWVWKHGRPDLQNQRZOHGJH GDWDGULYHQHYDOXDWLRQ H[SHUW RSLQLRQRUDSDUWLFXODUWDVN =RXDTDQG1NDPERX>@PHQWLRQstructural evaluationsDQGRWKHUDSSURDFKHVDVgold standards±FRPSDULQJ WKHRQWRORJ\ZLWKD³JROGVWDQGDUG´application-based±XVLQJWKHRQWRORJ\ZLWKDQDSSOLFDWLRQDQGHYDOXDWLQJWKH UHVXOWVdata-driven±FRPSDULQJWKHRQWRORJ\ZLWKDVRXUFHRIGDWDIURPWKHGRPDLQWREHFRYHUHGDQGassessment by domain experts 3DWULFN DQG )ORULDQ >@ FODVVLI\ RQWRORJ\ HYDOXDWLRQ SURFHVV DFFRUGLQJ WR LWV approach ± JROGHQ VWDQGDUG DSSOLFDWLRQEDVHG GDWDGULYHQ DVVHVVPHQW E\ KXPDQV complexity ± PHWKRGV HYDOXDWLQJ OH[LFDOYRFDEXODU\GDWD OD\HUKLHUDUFK\WD[RQRP\FRQWH[WDSSOLFDWLRQOHYHOV\QWD[VWUXFWXUHDUFKLWHFWXUHGHVLJQ DQGevaluation method± HYROXWLRQEDVHGORJLFDOUXOHEDVHGDQGPHWULFIHDWXUHEDVHG7KH\DOVRPHQWLRQHYDOXDWLRQWRROVDQGPHWKRGVDV 2QWR0DQDJHU 2QWR0HWULF 2QWR&OHDQ (YD/H[RQ :(%&25(DQG 62QWR(YDO %DFKLU %RXLDGMUD DQG %HQVOLPDQH >@ GLYLGH DYDLODEOH PHWKRGV LQWR IRXU JURXSV VWDUWLQJ ZLWK what should be evaluatedDV³ZKROHRQWRORJ\HYDOXDWLRQ´RU³SDUWLDORQWRORJ\HYDOXDWLRQ´7KH\IXUWKHUSUHVHQWWKHJURXSVRIwhy it should be evaluated ± ³YDOLGLW\ RI RQWRORJLFDO VWUXFWXUH ´ RU ³TXDOLW\ IRU XVH ´ when it should be evaluated ± ³EHIRUH WKH RQWRORJ\ EXLOGLQJ SURFHVV´ ³GXULQJ WKH RQWRORJ\ EXLOGLQJ SURFHVV´ ³GXULQJ WKH RQWRORJ\ HYROXWLRQ SURFHVV´³EHIRUHUHXVLQJWKHRQWRORJ\´ DQGbased on what it should be evaluated±³FRUSXVEDVHGHYDOXDWLRQ WR HVWLPDWH DFFXUDF\ DQG FRYHUDJH RI WKH RQWRORJ\ ´ ³JROGVWDQGDUGEDVHG HYDOXDWLRQ WR FRPSDUH RQWRORJ\ ZLWK D JROGVWDQGDUG ´ ³WDVNEDVHG HYDOXDWLRQ WR VHDUFK KRZ XVH RI RQWRORJ\ HIIHFWV WKH DSSOLFDWLRQ ´ ³H[SHUWEDVHG HYDOXDWLRQ HYDOXDWLRQE\KXPDQH[SHUWV ´³FULWHULDEDVHGHYDOXDWLRQ WRPHDVXUHILWRIRQWRORJ\WRVRPHFULWHULD ´ %RORWQLNRYDHWDO>@FODVVLI\WKHPHWKRGVRIRQWRORJ\HYDOXDWLRQDFFRUGLQJWRthe goal of evaluationthe stage of evaluationthe degree of automationthe object of the analysisDQGthe tools applied for analysis$QDO\VLVWRROV DUH FODVVLILHG DV ³GDWDGULYHQ WRROV´ ³H[SHUW HYDOXDWLRQV´ ³LQYHVWLJDWLRQ RI SURILOHV RI XVH´ ³FRPSDULVRQ ZLWK µJROGHQVWDQGDUG¶´DQG³LQYHVWLJDWLRQRIWRSRORJLHVRIWKHRQWRORJ\JUDSK´7KHDXWRPDWLRQGHJUHHRIWKHVHWRROV DUHKDQGOHGWKURXJKWKHLU³DXWRPDWLF´³VHPLDXWRPDWLF´DQG³PDQXDO´LQGLFDWLRQ ZKLOHWKHDSSOLFDWLRQVWDJHRIWKH DQDO\VLVLVJURXSHGDV³GHYHORSPHQWDQGSURWRW\SLQJ´³WHVWLQJEHIRUHUHOHDVH´DQG³XVH´2EMHFWRIWKHHYDOXDWLRQ FDQEH³VWUXFWXUH´³GLFWLRQDU\´DQG³HIILFLHQF\RISUDFWLFDODSSOLFDWLRQ´ DQGWKHUHFDQEHGLIIHUHQWHYDOXDWLRQJRDOV DVGHSLFWHGLQWKHHYDOXDWLRQVFKHPDSURYLGHGLQWKHVWXG\$VDQDGGLWLRQWRWKLVFODVVLILFDWLRQWKH\XQGHUOLQHWKH LPSRUWDQFHRITXDOLW\RIWKHGHVLJQRIWKHRQWRORJLHVDQGPHQWLRQWZRDVSHFWVWREHHYDOXDWHGDV³ FRUUHFWQHVV DQG GHSWK RI UHIOHFWLRQ RI WKH VXEMHFW GRPDLQ DQG HUJRQRPLF DVSHFW RI WKH RQWRORJ\ UHSUHVHQWDWLRQ IURP WKH SRLQW RI YLHZ RI TXDOLW\ DQG KXPDQ VSHHG RI SHUFHSWLRQ´ )XUWKHU WKH\ SUHVHQW GHWDLOHG TXDQWLILDEOH PHWULFV WR HYDOXDWH WKH FRJQLWLYH HUJRQRPLFV RI WKH RQWRORJLHV DQG VWDWH WKH LPSRUWDQFH RI FRJQLWLYH HUJRQRPLFLW\ ZKHQ RQWRORJ\LVDLPHGIRUOHDUQLQJRUSDVVLQJRQNQRZOHGJH 2.2.2. Data-driven evaluation and NLP %UHZVWHUHWDO>@VWDWHWKDWRQWRORJLHVDUHVLPSO\VHWRIFRQFHSWVDQGUHODWLRQVWKDWDUHHLWKHUH[SOLFLWO\GHILQHG RUIROORZLQJIURPVHWRID[LRPVDQGDFFRUGLQJO\WKH\DUHDEVWUDFWLRQVIURPDVHWRIQDWXUDOODQJXDJHWH[WVWKDWDUH GHVFULELQJWKHGRPDLQ$FFRUGLQJO\ WKH\SURSRVHUHYHUVLQJWKHSURFHGXUHIRUHYDOXDWLRQDVILQGLQJFRQFHSWVLQWKH WH[WVWKDWDUHUHODWHGZLWKWKHRQHVLQWKHRQWRORJ\DQGQDPHWKLVSURFHVVDV³FRUSXV´RU³GDWDGULYHQ´HYDOXDWLRQ $GGLWLRQDOO\ *DQJHPL HW DO >@ PHQWLRQ WKDW IXQFWLRQDO PHDVXUHPHQW LV EDVHG RQ ³PDWFKLQJ´ EHWZHHQ WZR VWUXFWXUHVEDVLFDOO\WKHVHDUFKRIFRUUHVSRQGHQFHEHWZHHQWKHRQWRORJ\DQGVRPHWKLQJHOVH7KH\DGGLWLRQDOO\UHIHU 1DWXUDO /DQJXDJH 3URFHVVLQJ 1/3 EDVHG PHWKRGV PHWKRGV EDVHGRQ PDWFKLQJ EHWZHHQ D FRUSXV RI GRFXPHQWV DQG WKH RQWRORJ\ DV JHQHUDOO\ XVHG PHWKRGV IRU IXQFWLRQDO DVVHVVPHQW 7KH\ UHIHU WR 'DHOHPDQV DQG 5HLQEHUJHU >@WKDW1/3EDVHGPHWKRGVSURYLGHWRVKRZWKHRQWRORJLHVDUH³FRQVHQVXDOFRQFHSWXDOL]DWLRQV´UDWKHUWKDQMXVWLWV GHYHORSHUV¶ LGHDV 7KURXJK D FRUSXVEDVHG DQDO\VLV HPSLULFDO HVWLPDWLRQ RI WKH accuracy DQG coverage RI WKH RQWRORJ\FDQEHPDGH>@,QSDUDOOHOZLWKWKHVH+DUWPDQQHWDO>@SRLQWRXWWZRPHWULFVDV³RQHWKDWVKRZVKRZ ZHOO D µVDPSOH RQWRORJ\¶ EXLOW DV WKH UHVXOW RI D FHUWDLQ PHWKRGRORJ\ KXPDQ PRGHOOLQJ RU DXWRPDWHG PLQLQJ
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
µUHSUHVHQWV¶ D FRPPRQO\ DFFHSWHG UHIHUHQFH FRUSXV DQG RQH WKDW GHWDLOV KRZ WKH RQWRORJ\ DFWXDOO\ RUGHUHG µFRUUHVSRQGV¶ZLWKWKHLQSXWWH[W V RIWKHFXVWRPHU¶VDSSOLFDWLRQGRPDLQ´ (YDOXDWLRQRIGHOD\DQDO\VLVRQWRORJ\ ,QWKLVVHFWLRQWKHGHOD\DQDO\VLVRQWRORJ\LVLQWURGXFHGDQGWKHHVWDEOLVKHGHYDOXDWLRQSURFHGXUHLVSUHVHQWHG 3.1. Delay analysis ontology $V LW LV SUHYLRXVO\ VWDWHG D GHOD\ DQDO\VLV RQWRORJ\ PD\ DFW DV D YRFDEXODU\ RI WKH GHOD\ DQDO\VLV GRPDLQ DQG SURYLGHWKHHDV\XQGHUVWDQGLQJRIWKHGHEDWH$GGLWLRQDOO\ZLWKLWVDELOLW\RIHDV\DGDSWDWLRQWRWKHPDFKLQHVZRUOG RQWRORJ\PD\EHXVHGLQDVRIWZDUHSURJUDPRUIRUPWKHEDVLVRIDFRPSDQ\GDWDEDVHRUGHFLVLRQVXSSRUWV\VWHP 0HWKRQWRORJ\ SURFHGXUH >@ LV IROORZHG VWHS E\ VWHS IRU WKH HVWDEOLVKPHQW RI RQWRORJLFDO VWUXFWXUH DQG WKH WRRO 3URWHJH LV XVHG IRU LWV LPSOHPHQWDWLRQ )RU WKH GHVLJQ RI WKH RQWRORJ\ SULQFLSOH RI 1R\ DQG 0F*XLQQHVV >@ DV nouns LQ VHQWHQFHV WKDW GHVFULEH GRPDLQ FRQVWLWXWH objects/concepts LQ DQ RQWRORJ\ ZKHUHDV verbs LQ VHQWHQFHV FRUUHVSRQGWRrelationsRIWKHFRQFHSWVLQWKHRQWRORJ\FRQVWLWXWHVWKHIRXQGDWLRQRIWKHFRQFHSWXDOEDFNJURXQGRI WKH RQWRORJ\ 7KH PDLQ FRQFHSWV LQ WKH LVVXH DUH H[WUDFWHG IURP OLWHUDWXUH UHYLHZ RQ GHOD\ DQDO\VLV DV ³causes of delay´³types of delay´³responsibility of delay´³claim for impact of delay´³analysis of delay´DQG³prevention of delay´ DFFRUGLQJO\WKHGHOD\DQDO\VLVRQWRORJ\LVGHILQHGLQIHZVHQWHQFHVWKDWDUHDEOHWREHUHDGWKURXJKZD\V IRUPHG E\ SDVVDJHV IURP WKH VXUURXQGLQJ FODVVHV WR WKH FRUH FODVV ³GHOD\´ 7KH IROORZLQJ 8QLILHG 0RGHOOLQJ /DQJXDJH 80/ GLDJUDPFRQVWLWXWHVWKHJUDSKRIWKHJHQHUDWHGRQWRORJ\ )LJ 7KHUHODWLRQVDWFRQFHSWOHYHODUH GHSLFWHG LQ EODFN DQG EOXH ZKLOH WKH UHODWLRQV DW LQVWDQFH OHYHO DUH GHSLFWHG LQ UHG )RU WKH IXOO WD[RQRP\ RI FRQFHSWVEHKLQGWKHJUDSKRIWKHRQWRORJ\WKHGHWDLOVRIWKHRQWRORJ\FDQEHLQYHVWLJDWHGLQWKHVWXG\RI%LOJLQ>@
)LJ&RPSOHWHPRGHORIGHOD\DQDO\VLVRQWRORJ\>@
65
66
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
3.2. Evaluation procedure $V LW LV SUHVHQWHG WKHUH DUH PDQ\ PHWKRGV DYDLODEOH IRU DOPRVW DQ\ NLQG RI HYDOXDWLRQ DQG QHZ RQHV FDQ EH DGGHG DFFRUGLQJ WR WKH VSHFLILF IHDWXUHV RI WKH FUHDWHG RQWRORJ\ :LWKLQ WKH YDULHW\ RI HYDOXDWLRQ PHWKRGV LW LV LPSRUWDQWWRXVHFRPSOHPHQWDU\PHWKRGVWKDWZLOOUHVSRQGWRERWKHYDOXDWLRQDVSHFWVDVverificationDQGvalidation 0XOWLSOH XVHRI PHWKRGV OHDGV D FRQVLVWHQWDQGXVDEOHRQWRORJ\>@)RU H[DPSOH (O'LUDE\ DQG:DQJ>@XVH FRPSHWHQF\TXHVWLRQVH[SHUWHYDOXDWLRQVDQGWHVWVRQVDPSOHGRFXPHQWVWRHYDOXDWHWKHLURQWRORJ\LQWHUPVRIHDVH RI QDYLJDWLRQ DEVWUDFWLRQ FRQVHQVXV UHSUHVHQWDWLRQ DQG FRQVLVWHQF\ (YDOXDWLRQ FDQ EH GRQH IRU WKH VHDUFK RI YDULRXV PHDVXUHV KRZHYHU ZKHQ LWV VSHFLILF XVH LV WKRXJKW FULWHULD UHOHYDQW IRU WKDW XVH VKRXOG EH WDNHQ LQWR FRQVLGHUDWLRQ>@*DQJHPLHWDO>@UHIHUWR1R\>@DVLWLVVWDWHGWKDWNQRZLQJWKHRQWRORJ\LVDFFXUDWHWRVRPH IRUPDOFULWHULDQDPHO\LWVVWUXFWXUDOTXDOLW\RQO\LPSOLHVWKDWWKHRQWRORJ\LVXVDEOHEXWLWGRHVQRWJLYHDQ\LGHD DERXWLWVVXLWDELOLW\IRUWKHFDVHLQKDQG,QOLJKWRIWKHVHUHJDUGLQJWKHHYDOXDWLRQRIGHOD\DQDO\VLVRQWRORJ\ ILUVWRI DOO VLQFH LW LV FRQVWUXFWHG ZLWK D PHWKRGRORJ\ 0HWKRQWRORJ\ WKDW LQGLFDWHV HYDOXDWLRQ VWHS DV D FRQWLQXRXV VWHS GXULQJWKHFRQVWUXFWLRQRIWKHRQWRORJ\YHULILFDWLRQRIWKHRQWRORJ\LVFRQVLGHUHGIURPWKHEHJLQQLQJWRWKHHQGRI WKH RQWRORJ\ FRQVWUXFWLRQ SURFHVV &RPSHWHQF\ TXHVWLRQV DUH XVHG LQ WKH VSHFLILFDWLRQ VWHS RI WKH PHWKRGRORJ\ ZKLFKLVDPHWKRGSURSRVHGE\*UXQLQJHUDQG)R[>@DVWKHTXHULHVKHOSLQJWKHFRQVWUXFWLRQDQGDOVRHYDOXDWLRQ RIWKHRQWRORJ\7KHIROORZLQJcompetency questionsDUHXVHGWRVWUXFWXUHWKHRQWRORJ\ ³:K\WKLVRQWRORJ\LVEHLQJ EXLOW"´³:KRDUHLWVXVHUV"´³:KDWLVGHOD\"´³:KDWLVGHOD\DQDO\VLV"´³:KDWGRHVKDSSHQLQFDVHRIDGHOD\"´ ³:KDWDUHWKHFDXVHVRIGHOD\"´³:KRDUHUHVSRQVLEOHIURPGHOD\"´³:KDWVKRXOGEHGRQHLQFDVHRIDGHOD\"´ ³:KDW VKRXOG EH GRQH IRU WKH SUHYHQWLRQ RI GHOD\"´ $GGLWLRQDOO\ YDULRXV GHVLJQ FULWHULD SUHVHQWHG E\ GLIIHUHQW UHVHDUFKHUV>@KDYHDFRQWURORYHUWKHFRQVWUXFWLRQRIWKHRQWRORJ\DQGDUHWDNHQLQWRFRQVLGHUDWLRQDVIDUDV SRVVLEOH2EMHFWLYLW\LVWULHGWREHSURYLGHGWRSUHYHQWDPELJXLW\DQGLQFRQVLVWHQWGHILQLWLRQVDUHQRWLQFOXGHGLQWKH RQWRORJ\ $OVR WKH FUHDWHG RQWRORJ\ LV HDV\ WR XSGDWH GXH WR LWV FRPPRQ UHSUHVHQWDWLRQ V\VWHP DQG KDV RQO\ WKH FRPPRQ GHILQLWLRQV RI WKH VXEMHFW VR GRHV QRW OLPLW LWV XVHUV ZKHQ LWV XVDJH LV LQ TXHVWLRQ 0RUHRYHU IRU WKH LPSOHPHQWDWLRQRIWKHRQWRORJ\DUHOLDEOHWRRO 3URWHJH LVXVHGWKDWZRXOGIRUPDQDQWLFLSDWHGRQWRORJ\VNHOHWRQ $FFRUGLQJO\WKHYHULILFDWLRQRIWKHRQWRORJ\LVVHDUFKHGDVDQLQLWLDOVWHSRIHYDOXDWLRQDQGZKDWLVFUXFLDODWWKLV SRLQWLVWKHLQYHVWLJDWLRQRIPDWFKLQJRIWKHFRQVWUXFWHGRQWRORJ\WRUHDOZRUOG%HVLGHV/RYUHQFLFDQG&XEULOR>@ VWDWHWKDWVPDOORQWRORJLHVFDQEHFUHDWHGZLWKRQO\IHZPLVWDNHVKRZHYHULQFRPSOHWHQHVVSUREOHPDOZD\VH[LVWV $FFRUGLQJO\ IRU D OLJKWZHLJKW DQG WKLV W\SH RIRQWRORJ\ VHDUFKIRU FRPSOHWHQHVV LV DOVR LPSRUWDQW > @ %HWZHHQ DYDLODEOH RSWLRQV IRU YDOLGDWLRQ H[SHUW LQWHUYLHZV FDVH VWXGLHV FRPSDUDWLYH DQDO\VLV RI LQGXVWU\ GRFXPHQWV DQG FRPSHWHQF\ TXHVWLRQV WKH YDOLGDWLRQ RI WKH RQWRORJ\ WKURXJK FDVH VWXGLHV LV VHOHFWHG >@ 5HDO expert reports IRU FODLP DQDO\VLV DUH VHOHFWHG VLQFH WKH\ DUH WKH SULPDU\ GRFXPHQWV IRU GHOD\ DQDO\VLV 7KH WHFKQLTXH KDQGOHGLQWKLVVWXG\LVSDUDOOHOZLWK1/3EDVHGPHWKRGVDQGLWVHDUFKHVIRUWKHcompleteness DQGsuitability for the useRIWKHRQWRORJ\$FFRUGLQJWRWKHFODVVLILFDWLRQSUHVHQWHGE\%RORWQLNRYDHWDO>@ WKHPHWKRGIROORZHGFDQEH GHHPHG DV D ³PDQXDO´ DQG ³GDWDGULYHQ´ PHWKRG DSSOLHG IRU ³WHVWLQJ EHIRUH UHOHDVH´ ZLWK WKH JRDO RI ³FRPSOHWHQHVV DQG SUHFLVLRQ RI WKH GLFWLRQDU\ RI WKH VXEMHFW GRPDLQ´ DQG ZLWK WKH DQDO\VLV REMHFW DV WKH ³GLFWLRQDU\´ 'HSLFWLRQRIHYDOXDWLRQGHWDLOV 7KUHHFDVHVWXGLHVIURPFODLPDQDO\VLVDUHVHOHFWHGEHWZHHQILYHDYDLODEOHGXHWRWKHLUGHWDLOKDQGOHGLQ3KUDVHV LQ WKH UHSRUWV DUH VHSDUDWHG LQWR FRQFHSWV XQGHUOLQHG DQG WKH\ DUH PDWFKHG ZLWK WKH UHODWHG FRQFHSWV LQ WKH RQWRORJ\DVLQWKHH[DPSOHSURYLGHGLQIROORZLQJWDEOH 7DEOH 3KUDVHVRIWKHUHSRUWDUHWRWDOO\NHSWLQTXRWDWLRQ PDUNVZKHUHDVRQO\FRQFHSWVLQWKHRQWRORJ\DUHSUHVHQWHGLQTXRWDWLRQPDUNV7RVSHFLI\WKHSODFHRIWKHFRQFHSW LQ WKH RQWRORJ\ XSSHU FRQFHSWV GHILQHG DUH DOVR LQFOXGHG )RU WKH LQFOXVLRQ RI LQIRUPDWLRQ RI UHODWLRQV LQ WKH RQWRORJ\ WKH ZRUG ³XQGHU´ LV XVHG WR UHIHU WD[RQRPLF UHODWLRQV EHWZHHQ FRQFHSWV ZKHUHDV WKH GHPRQVWUDWLRQ ³ DVVRFLDWLRQBQDPH´LVXVHGIRUWKHQRQWD[RQRPLFUHODWLRQV6R³SKUDVHVLQRQWRORJ\´DUHLQGLFDWHGLQSDWKIRUPV VXFKDV ³FRQFHSW´XQGHU³FRQFHSW´DVVRFLDWLRQ³FRQFHSW´7KHIROORZLQJWDEOH 7DEOH H[HPSOLILHVWKHSURFHVV KHOGGXULQJHYDOXDWLRQ
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
67
7DEOH([DPSOHIRUFRPSDULVRQRIFRQFHSWV 3KUDVHLQWKH5HSRUW 2EWDLQLQJEXLOGLQJ SHUPLWV &RQWUDFWRU LQIRUPVWKH(QJLQHHU DERXWWKHGHOD\GXHWR REWDLQLQJEXLOGLQJ SHUPLWVLVQRWLQWKHLU FRQWUDFWRU V UHVSRQVLELOLW\DQG FODLPVWKHUHTXLUHG H[WHQVLRQRIWLPH (QJLQHHUDSSURYHVWKH FRQWUDFWRU VFODLPDQG ZLWKWKHFRQVHQWRI RZQHUGD\VRI H[WHQVLRQLVDZDUGHG WRFRQWUDFWRUDQGWKH QHZFRQWUDFW FRPSOHWLRQGDWHLV GHWHUPLQHG
3KUDVHLQWKH2QWRORJ\ 1RWLFHIRU&ODLPIRU,PSDFWRI'HOD\ %XLOGLQJSHUPLWVDSSURYDOSURFHVVXQGHU5XOHVDQG5HJXODWLRQV5HODWHG&DXVHVXQGHU([WHUQDO&DXVHVXQGHU &DXVHVRI'HOD\ 2ZQHU RUKLVDJHQWV UHVSRQVLEOHXQGHU5HVSRQVLELOLW\RI'HOD\ 1HLWKHUFRQWUDFWXDOSDUW\UHVSRQVLEOHXQGHU5HVSRQVLELOLW\RI'HOD\ &ODLPIRU,PSDFWRI'HOD\ ([WHQVLRQRIWLPHFODLPXQGHU7LPH5HODWHG&ODLPVXQGHU.LQGVRI&ODLPIRU,PSDFWRI'HOD\ 6HWWOHPHQWLVD5HVXOWRI&ODLPIRU,PSDFWRI'HOD\ ([WHQVLRQRIWLPHXQGHU$ZDUGWKURXJK6HWWOHPHQWLVD5HVXOWRI&ODLPIRU,PSDFWRI'HOD\ $ZDUGWKURXJK6HWWOHPHQWLVD5HVXOWRI&ODLPIRU,PSDFWRI'HOD\ &XUUHQWFRPSOHWLRQGDWHXQGHU'HWDLOVRI&ODLPXQGHU3DUWVRI&ODLPIRU,PSDFWRI'HOD\ 3UDFWLFDOFRPSOHWLRQ6XEVWDQWLDOFRPSOHWLRQDQGLQLWLDOFHUWLILFDWHXQGHU&RQWUDFW&ODXVHVXQGHU0DLQ&RQWUDFW 'RFXPHQWVXQGHU&RQWUDFW'RFXPHQWVXQGHU'DWDXVHGBIRU$QDO\VLVRI'HOD\ 8SGDWHDQGUHDQDO\]HWKHQHWZRUN VSHFLI\WKHSURMHFWLVDKHDGRUEHKLQGWKHVFKHGXOHDWGHOD\GDWHXQGHU$SSO\ $QDO\VLV7HFKQLTXHXQGHU(YHQW$QDO\VLVXQGHU0HWKRGRORJ\6WHSVIRU$QDO\VLVRI'HOD\ $GMXVWHG8SGDWHGVFKHGXOH VFKHGXOHGHSLFWLQJLPSDFWVE\FKDQJHVRQDVSODQQHGVFKHGXOHXQGHU0DMRU6FKHGXOHV XQGHU3RVWFRQWUDFW'RFXPHQWVXQGHU'DWDXVHGBIRU$QDO\VLVRI'HOD\
'LVFXVVLRQDQGFRQFOXVLRQ 7KHFRQFHSWVZLWKLQWKHUHSRUWVDQGWKHRQWRORJ\DUHPDWFKHGRQO\VHYHUDODGGLWLRQVDUHPDGHDVDUHVXOWRIWKLV HYDOXDWLRQ7KHUHODWLRQVDUHQRWLQYHVWLJDWHGVHSDUDWHO\VLQFHWKH\FRPSOHPHQWWKHH[LVWHQFHRIWKHVWDWHGFRQFHSWV :KHQ WKH PDWFKLQJ EHWZHHQ FRQFHSWV LQ WKH RQWRORJ\ DQG UHSRUWV LV FRQVLGHUHG LW LV VHHQ WKDW WKH RQWRORJ\ KDV HQRXJK FRQFHSWV WR UHSUHVHQW UHDO ZRUOG NQRZOHGJH IRU WKHVH FDVHV DQG LW KDV WKH SRWHQWLDO WR HDVH FODLP PDQDJHPHQWSURFHVVOLNHWKHRQHVLQWKHFDVHVWXGLHV$GGLWLRQDOO\FKDQJHVFDQHDVLO\EHPDGHDVLQWKHSURFHVVRI DGGLWLRQ RI FRQFHSWV WR PHHW VRPH UHTXLUHPHQW IRU LWV VSHFLILF XVH 7KLV HYDOXDWLRQ SURFHVV FRQVWLWXWHV D ILQDO HYDOXDWLRQRIWKHFRQVWUXFWHGRQWRORJ\EHIRUHLWVUHOHDVH0RUHH[SHUWUHSRUWVFDQEHKDQGOHGDQGH[SHUWRSLQLRQVRQ RQWRORJ\ FDQ EH WDNHQ IRU IXUWKHU HYDOXDWLRQ +RZHYHU HYDOXDWLRQ RI WKH RQWRORJ\ LV OLPLWHG WR D OHYHO ZLWK WKH PHQWLRQHGPHWKRGV$FWXDOHYDOXDWLRQRIWKHRQWRORJ\FDQEHGRQHE\DFWLYHXVDJHRIWKHRQWRORJ\LQDFRPSDQ\ GDWDEDVHDQGE\WKHSRVVLEOHXSGDWHVWKDWZRXOGEHUHTXLUHGGXULQJLWVXVDJH 2QWRORJLHV DQG WKHLU HYDOXDWLRQ PHWKRGV LV LQWURGXFHG LQ WKLV VWXG\ DQG H[HPSOLILHG ZLWK D GHOD\ DQDO\VLV RQWRORJ\DQGLWVHYDOXDWLRQSURFHVV,WLVEHOLHYHGWKDWRQWRORJ\DQGLWVHYDOXDWLRQDUHDOZD\VZHOFRPHGDVORQJDV WKH\DUHVXSSRUWHGZLWKIRUPDOSURFHVVHVDQGRQWRORJLHVFDQEHDFWXDOO\YDOLGDWHGDVORQJDVWKH\DUHDFWLYHO\XVHG 5HIHUHQFHV >@2&RUFKR0)HUQDQGH]/RSH]$*RPH]3HUH]0HWKRGRORJLHVWRROVDQGODQJXDJHVIRUEXLOGLQJRQWRORJLHV:KHUHLVWKHLUPHHWLQJ SRLQW"'DWD .QRZOHGJH(QJLQHHULQJ ± >@1)1R\'/0F*XLQQHVV2QWRORJ\GHYHORSPHQWDJXLGHWRFUHDWLQJ\RXUILUVWRQWRORJ\.6/7HFKQLFDO5HSRUW6WDQIRUG8QLYHUVLW\ 6WDQIRUG&$86$ >@75*UXEHU$WUDQVODWLRQDSSURDFKWRSRUWDEOHRQWRORJ\VSHFLILFDWLRQV.QRZOHGJH$FTXLVLWLRQ ± >@75*UXEHU7RZDUGSULQFLSOHVIRUWKHGHVLJQRIRQWRORJLHVXVHGIRUNQRZOHGJHVKDULQJ,QWHUQDWLRQDO-RXUQDO+XPDQ&RPSXWHU6WXGLHV ± >@$*DQJHPL&&DWHQDFFL0&LDUDPLWD-/HKPDQQ0RGHOOLQJRQWRORJ\HYDOXDWLRQDQGYDOLGDWLRQ3URFHHGLQJVRIWKHUG(XURSHDQ 6HPDQWLF:HE&RQIHUHQFH (6:& %XGYD0RQWHQHJUR >@%&KDQGUDVHNDUDQ-5-RVHSKVRQ95%HQMDPLQV:KDWDUHRQWRORJLHVDQGZK\GRZHQHHGWKHP",(((,QWHOOLJHQW6\VWHPV ± >@51HFKHV5()LNHV7)LQLQ75*UXEHU76HQDWRU:56ZDUWRXW(QDEOLQJWHFKQRORJ\IRUNQRZOHGJHVKDULQJ$,0DJD]LQH ± >@($OIRQVHFD60DQDQGKDU3URSRVDOIRUHYDOXDWLQJRQWRORJ\UHILQHPHQWPHWKRGV3URFHHGLQJVRIWKH/DQJXDJH5HVRXUFHVDQG(YDOXDWLRQ &RQIHUHQFH /5(& /DV3DOPDV&DQDU\,VODQGV6SDLQ
68
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68 > @+1HI]L0)DUDK,5)DUDK%6RODLPDQ$FULWLFDODQDO\VLVRIOLIHF\FOHVDQGPHWKRGVIRURQWRORJ\FRQVWUXFWLRQDQGHYDOXDWLRQ 3URFHHGLQJVRIVW,QWHUQDWLRQDO&RQIHUHQFHRQ$GYDQFHG7HFKQRORJLHVIRU6LJQDODQG,PDJH3URFHVVLQJ $76,3 6RXVVH7XQLVLD >@,6HUUD5*LUDUGL31RYDLV(YDOXDWLQJWHFKQLTXHVIRUOHDUQLQJQRQWD[RQRPLFUHODWLRQVKLSVRIRQWRORJLHVIURPWH[W([SHUW6\VWHPVZLWK $SSOLFDWLRQV ± >@06)R[0*UXQLQJHU2QRQWRORJLHVDQGHQWHUSULVHPRGHOOLQJ3URFHHGLQJVRIWKH,QWHUQDWLRQDO&RQIHUHQFHRQ(QWHUSULVH,QWHJUDWLRQ 0RGHOOLQJ7HFKQRORJ\ ,&(,07 7XULQ,WDO\ >@$*RPH]3HUH]7RZDUGVDIUDPHZRUNWRYHULI\NQRZOHGJHVKDULQJWHFKQRORJ\([SHUW6\VWHPVZLWK$SSOLFDWLRQV ± >@<1HW]HU'*DED\0$GOHU<*ROGEHUJ0(OKDGDG2QWRORJ\HYDOXDWLRQWKURXJKWH[WFODVVLILFDWLRQ$GYDQFHVLQ:HEDQG1HWZRUN 7HFKQRORJLHVDQG,QIRUPDWLRQ0DQDJHPHQW/HFWXUH1RWHVLQ&RPSXWHU6FLHQFH ± >@$%DFKLU%RXLDGMUD60%HQVOLPDQH)2(YDO )XOORQWRORJ\HYDOXDWLRQ0RGHODQGSHUVSHFWLYHV3URFHHGLQJVRIWKHWK,QWHUQDWLRQDO &RQIHUHQFHRQ1DWXUDO/DQJXDJH3URFHVVLQJDQG.QRZOHGJH(QJLQHHULQJ7RNXVKLPD-DSDQ >@(6%RORWQLNRYD7$*DYULORYD9$*RURYR\7RDPHWKRGRIHYDOXDWLQJRQWRORJLHV-RXUQDORI&RPSXWHUDQG6\VWHPV6FLHQFHV ,QWHUQDWLRQDO ± >@-+DUWPDQQ36S\QV$*LERLQ'0D\QDUG5&XHO0&6XiUH])LJXHURD<6XUH0HWKRGVIRURQWRORJ\HYDOXDWLRQ.QRZOHGJH:HE 'HOLYHUDEOH' >@6/RYUHQFLF0&XEULOR2QWRORJ\HYDOXDWLRQ±&RPSULVLQJYHULILFDWLRQDQGYDOLGDWLRQ3URFHHGLQJVRI&HQWUDO(XURSHDQ&RQIHUHQFHRQ ,QIRUPDWLRQDQG,QWHOOLJHQW6\VWHPV &(&,,6 9DUD]GLQ&URDWLD >@-7)HUQDQGH]%UHLV0($UDQJXUHQ56WHYHQV4XDOLW\HYDOXDWLRQIUDPHZRUNIRUELRRQWRORJLHV3URFHHGLQJVRI,QWHUQDWLRQDO &RQIHUHQFHRQ%LRPHGLFDO2QWRORJ\ ,&%2 %XIIDOR86$ > @$*RPH]3HUH]0)HUQDQGH]/RSH]2&RUFKR2QWRORJLFDO(QJLQHHULQJ6SULQJHU9HUODJ/RQGRQ/LPLWHG86$ >@$=RXDT51NDPERX(YDOXDWLQJWKHJHQHUDWLRQRIGRPDLQRQWRORJLHVLQWKHNQRZOHGJHSX]]OHSURMHFW.QRZOHGJHDQG'DWD(QJLQHHULQJ ± >@)3DWULFN+)ORULDQ2QWRORJ\HYDOXDWLRQ6HPLQHULQ$SSOLHG2QWRORJ\(QJLQHHULQJ:68QLYHUVLW\RI,QQVEUXFN$XVWULD >@&%UHZVWHU+$ODQL6'DVPDKDSDWUD<:LONV'DWDGULYHQRQWRORJ\HYDOXDWLRQ3URFHHGLQJVRI,QWHUQDWLRQDO&RQIHUHQFHRQ/DQJXDJH 5HVRXUFHVDQG(YDOXDWLRQ /5(& /LVERQ3RUWXJDO >@:'DHOHPDQV0/5HLQEHUJHU6KDOORZWH[WXQGHUVWDQGLQJIRURQWRORJ\FRQWHQWHYDOXDWLRQ,(((,QWHOOLJHQW6\VWHPV ± >@$*RPH]3HUH]0)HUQDQGH]/RSH]$GH9LFHQWH7RZDUGVDPHWKRGWRFRQFHSWXDOL]HGRPDLQRQWRORJLHV3URFHHGLQJVRI(&$, :RUNVKRSRQ2QWRORJLFDO(QJLQHHULQJ%XGDSHVW >@*%LOJLQ$QRQWRORJ\EDVHGDSSURDFKIRUGHOD\DQDO\VLV0DVWHUWKHVLV0LGGOH(DVW7HFKQLFDO8QLYHUVLW\7XUNH\ >@7((O'LUDE\%:DQJ(6RFLHW\3RUWDO ,QWHJUDWLQJXUEDQKLJKZD\FRQVWUXFWLRQSURMHFWVLQWRWKHNQRZOHGJHFLW\-RXUQDORI&RQVWUXFWLRQ (QJLQHHULQJDQG0DQDJHPHQW ± >@11R\(YDOXDWLRQE\RQWRORJ\FRQVXPHUV,(((,QWHOOLJHQW6\VWHPV ± >@0*UXQLQJHU06)R[0HWKRGRORJ\IRUWKHGHVLJQDQGHYDOXDWLRQRIRQWRORJLHV3URFHHGLQJVRI:RUNVKRSRQ%DVLF2QWRORJLFDO,VVXHVLQ .QRZOHGJH6KDULQJ0RQWUHDO&DQDGD > @'/DYELF0.ULVSHU)DFLOLWDWLQJRQWRORJ\GHYHORSPHQWZLWKFRQWLQXRXVHYDOXDWLRQ,QIRUPDWLFD ± >@7$(O'LUDE\-=KDQJ$VHPDQWLFIUDPHZRUNWRVXSSRUWFRUSRUDWHPHPRU\PDQDJHPHQWLQEXLOGLQJFRQVWUXFWLRQ$XWRPDWLRQLQ &RQVWUXFWLRQ ± >@+37VHUQJ6</
ScienceDirect Procedia Engineering 85 (2014) 61 – 68
&UHDWLYH&RQVWUXFWLRQ&RQIHUHQFH&&
2QWRORJ\HYDOXDWLRQ $QH[DPSOHRIGHOD\DQDO\VLV *R]GH%LOJLQD ,UHP'LNPHQE07DODW%LUJRQXOE a
Research Assistant, Department of Civil Engineering, Middle East Technical University, Ankara, 06800, Turkey b Professor, Department of Civil Engineering, Middle East Technical University, Ankara, 06800, Turkey
$EVWUDFW 2QWRORJLHV KDYH D SODFH LQ NQRZOHGJH VKDULQJ ZLWK WKHLU DELOLW\ WR FDSWXUH UHDO ZRUOG LQIRUPDWLRQ LQ D PDFKLQH UHDGDEOH HQYLURQPHQW7KHLUDSSOLFDWLRQYDULHVIURPDQ\DSSURDFKVWUXFWXUHGRQRQWRORJLFDOFRPPLWPHQWWRPHWDOHYHORQWRORJLHVVHUYLQJ IRU YDULHW\ RI SXUSRVHV 7KLV YDULHW\ FRQWLQXHV DOVR LQ PHWKRGRORJLHV IRU FRQVWUXFWLQJ WKH RQWRORJLHV 7KHUH DUH ZLGH UDQJH RI RQWRORJLHVGLIIHULQJLQVL]HDQGFRPSOH[LW\WKDWDUHFUHDWHGZLWKGLIIHUHQWGHVLJQFULWHULD7KHPDLQSRLQWZLWKWKHRQWRORJLHVLV WKHLUVWUXFWXUHDQGDELOLW\WRVHUYHIRUWKHLQWHQGHGSXUSRVH7KLVLVRQO\JUDQWHGZLWKWKHFRQWLQXRXVHYDOXDWLRQRIRQWRORJLHVLQ WKH FRQVWUXFWLRQ SURFHVV DQG DOVR EHIRUH UHOHDVH RU UHXVH RI WKH RQWRORJ\ 2QWRORJ\ HYDOXDWLRQ PHWKRGV FOXVWHU DURXQG WZR FRQFHSWV VXFK DV YHULILFDWLRQ PHWKRGV WKDW HQVXUH WKH VWUXFWXUH RI WKH RQWRORJ\ DQG YDOLGDWLRQ PHWKRGV WKDW H[DPLQH WKHLU DSSOLFDELOLW\LQUHDOZRUOG,QOLJKWRIWKHVHGLIIHUHQWDSSURDFKHVDVTXDQWLWDWLYHDQGTXDOLWDWLYHPHWKRGVDUHGHSLFWHGLQOLWHUDWXUH IURPVXEMHFWLYHHYDOXDWLRQVE\H[SHUWVWRWRROVDOORIZKLFKLQYHVWLJDWHGLIIHUHQWFKDUDFWHULVWLFVRIRQWRORJLHV 2EMHFWLYHVRIWKLVVWXG\DUHWRXQGHUOLQHWKHLPSRUWDQFHRIRQWRORJ\HYDOXDWLRQDQGWRSUHVHQWDQHPSOR\HGYDOLGDWLRQPHWKRG XVHGGXULQJRQWRORJ\GHYHORSPHQWIRUGHOD\DQDO\VLV,QWKLVFRQWH[WILUVW OLWHUDWXUHUHYLHZRQRQWRORJ\HYDOXDWLRQLVSUHVHQWHG 6HFRQGDGHOD\DQDO\VLVRQWRORJ\LVLQWURGXFHGZLWKLWVEDVLFVWROD\WKHIRXQGDWLRQVRIWKHVWXG\&DVHVWDNHQIURPWKH7XUNLVK FRQVWUXFWLRQ LQGXVWU\ DUH H[HPSOLILHG WR H[SODLQ WKH XWLOL]HG RQWRORJ\ YDOLGDWLRQ SURFHVV &RPSDULVRQ RI WKH FRQFHSWV LQ WKH FRQVWUXFWHG RQWRORJ\ ZLWK WKH H[SHUW UHSRUWV ZULWWHQ IRU HDFK FDVH LV XVHG WR HYDOXDWH WKH RQWRORJ\ )XUWKHUPRUH DOWHUQDWLYH YDOLGDWLRQWHFKQLTXHVDUHGLVFXVVHGDVZHOODVSRVVLEOHDWWHPSWVWRNHHSLWUHVSRQVLYHDQGXSWRGDWH
© 2014 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license 7KH$XWKRUV3XEOLVKHGE\(OVHYLHU/WG (http://creativecommons.org/licenses/by-nc-nd/3.0/). 3HHUUHYLHZXQGHUUHVSRQVLELOLW\RIWKH'LDPRQG&RQJUHVV.IW Peer-review under responsibility of the Diamond Congress Kft. Keywords:FRQVWUXFWLRQVHFWRU GHOD\DQDO\VLV RQWRORJ\ RQWRORJ\HYDOXDWLRQ RQWRORJ\YDOLGDWLRQ
&RUUHVSRQGLQJDXWKRU7HO ID[ E-mail address:JELOJLQ#PHWXHGXWU
1877-7058 © 2014 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/3.0/). Peer-review under responsibility of the Diamond Congress Kft. doi:10.1016/j.proeng.2014.10.529
62
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
,QWURGXFWLRQ 2QWRORJLHVRFFXS\DSURPLQHQWSODFHLQNQRZOHGJHVKDULQJ7KHRQWRORJ\FRQFHSWLVIOH[LEOHDQGLWVDGDSWDELOLW\ OHDGV YDULHW\ DQGVRSUHYHQWV D FRPPRQ RQWRORJ\GHYHORSPHQW RUHYDOXDWLRQ PHWKRG WR EH DFFHSWHG 7KHLUXVDJH DUHDLVZLGHDQGWKH\DUHGHYHORSHGWRUHVSRQGPDQ\QHHGLQWKHDUHD7KHVRXQGQHVVRIDQRQWRORJ\EHWZHHQDOOWKH FUHDWHGRQHVLVPDQLIHVWHGRQO\E\LWVHYDOXDWLRQ$FFRUGLQJO\HYDOXDWLRQLVKLJKO\UHFRPPHQGHGGXULQJZKROHOLIH F\FOH RI DQ RQWRORJ\ ,Q WKLV FRQWH[W PDQ\ HYDOXDWLRQ PHWKRGV DUH DYDLODEOH WKDW DUH UHVSRQVLYH IRU GLIIHUHQW HYDOXDWLRQDLPV $QRQWRORJ\RIGHOD\DQDO\VLVLQFRQVWUXFWLRQVHFWRUXQGHUOLHVWKLVVWXG\7KLVLQHYLWDEOHSUREOHPRIFRQVWUXFWLRQ VHFWRULVWULHGWREHKDQGOHGLQWKLVZD\WRLQFUHDVHNQRZOHGJHLQWKHLVVXHDQGWRIRUPDEDVHIRUDQ\DWWHPSWWKDW ZRXOG EH KHOG LQ PDFKLQHV HQYLURQPHQW 3RVVLEOH XVH RI WKH RQWRORJ\ E\ FRQVWUXFWLRQ FRPSDQLHV LQ SURMHFW ULVN DQGRUFODLPPDQDJHPHQW DQGDVDGDWDEDVHIRUWKHFRPSDQLHVLVH[SHFWHGE\WKLVRQWRORJ\7RPHHWWKLVREMHFW YDOLGDWLRQ E\ UHDO H[SHUW UHSRUWV RQ GHOD\ DQDO\VLV LV VHOHFWHG WR HQVXUH WKH VRXQGQHVV RI WKH OLWHUDWXUH EDVHG RQWRORJ\ ,QWKLVSDSHUOLWHUDWXUHUHYLHZRQRQWRORJ\HYDOXDWLRQPHWKRGVLVKDQGOHGDQGWKHIROORZHGYDOLGDWLRQPHWKRGLV SUHVHQWHGWRVKRZWKHYDULHW\DQGLPSRUWDQFHRIRQWRORJ\HYDOXDWLRQPHWKRGV 5HVHDUFKEDFNJURXQG ,Q WKLV VHFWLRQ ILUVWO\ RQWRORJ\ LV GHILQHG DQG IROORZLQJ WKDW OLWHUDWXUH UHYLHZ RQ LWV HYDOXDWLRQ LV SUHVHQWHG WR UHYHDOLWVH[WHQW 2.1. Ontology 2QWRORJLHVKDYHEHHQZLGHO\XVHGLQNQRZOHGJHHQJLQHHULQJVLQFH VWRPHHWWKHQHHGRIXQLILHGIRUPVIRU LQIRUPDWLRQ VKDULQJ WR SURYLGH FRPPRQ XQGHUVWDQGLQJ DQG UHXVH >@ :LWK LWV ZLGHO\ NQRZQ GHILQLWLRQ DQ RQWRORJ\LV³DQH[SOLFLWVSHFLILFDWLRQRIDFRQFHSWXDOL]DWLRQ´>@&RUFKRHWDO>@SUHVHQWDXQLILHGGHILQLWLRQDV ³RQWRORJLHV DLP WR FDSWXUH FRQVHQVXDO NQRZOHGJH LQ D JHQHULF DQG IRUPDO ZD\ DQG WKDW WKH\ PD\ EH UHXVHG DQG VKDUHGDFURVVDSSOLFDWLRQV VRIWZDUH DQGE\JURXSVRISHRSOH´$GGLWLRQDOO\*DQJHPLHWDO>@VWUXFWXUDOO\GHILQH DQRQWRORJ\DV³DJUDSKZKRVHQRGHVDQGDUFVUHSUHVHQWFRQFHSWXDOL]DWLRQV´%DVLFDOO\LWLVWKHJUDSKRIPHWDGDWD UDQJLQJIURPWKHVDXULWD[RQRPLHVDQGD[LRPDWL]HGWKHRULHV7KHWD[RQRPLF VXEFODVVRILVD UHODWLRQVKLSVRIDQ RQWRORJ\FRQVWLWXWHWKHGHSWKDQGEUHDGWKRIDQRQWRORJ\ZKHUHDVQRQWD[RQRPLFUHODWLRQVFRQVWLWXWHWKHGHQVLW\RI WKH RQWRORJ\ ,W LV WKH GH[WHULW\ RI LWV GHYHORSHU WR NHHS WKH FRQFHSWXDOL]DWLRQ LQ D VHPDQWLFDOO\ IRUPDOL]HG IRUP QDPHO\LQWKHJUDSK>@2QWRORJLHVDUHEDVLFDOO\WD[RQRPLFWUHHRIFRQFHSWXDOL]DWLRQVIURPJHQHUDOREMHFWVDWWKH WRSOHYHOWRWKHVSHFLILFRQHVDWWKHERWWRPOHYHO>@:KHQWD[RQRPLHVDUHLPSURYHGZLWKPRUHLQIRUPDWLRQWKURXJK UHODWLRQV DQG UXOHV IRU PRGHOLQJ D GRPDLQ WKH\ EHFRPH RQWRORJ\ >@ $FFRUGLQJO\ RQH LV IUHH WR FRQVWUXFW DQ\ OLJKWZHLJKW JHQHUDOO\ LQFOXGLQJ FRQFHSWV FRQFHSW WD[RQRPLHV UHODWLRQVKLSV EHWZHHQ FRQFHSWV DQG SURSHUWLHV RI FRQFHSWV RUKHDY\ZHLJKW DGGLWLRQDOO\ LQFOXGLQJ D[LRPV DQG FRQVWUDLQWV RQWRORJ\ WKDW ZRXOGEH VWUXFWXUHG WR EH UHVSRQVLYHWRDQ\QHHG>@7KHUHDUHYDULRXVRQWRORJ\FRQVWUXFWLRQPHWKRGVUDQJLQJIURPFRQVWUXFWLRQIURPVFUDWFK WRPHUJLQJH[LVWLQJRQWRORJLHVDQGJHQHUDWLQJRQWRORJLHVZLWKDXWRPDWLFFRQVWUXFWLRQSULQFLSOHV RQWRORJ\OHDUQLQJ >@ ,Q UHVSRQVH WR WKLV WKHUH DUH ZLGH UDQJH RI HYDOXDWLRQ PHWULFV DYDLODEOH SUHVHQWHG WR HYDOXDWH WKH LQWHQGHG VWUXFWXUH DQG IXQFWLRQDOLW\ RI WKHRQWRORJ\> @ 0RUHRYHUUHVHDUFKHUV WKDWSURSRVHGHYHORSPHQW VWUDWHJLHV DOVR PHQWLRQ VRPH GHVLJQ DQG HYDOXDWLRQ FULWHULD IRU RQWRORJLHV OLNH WKH RQHV SUHVHQWHG E\ )R[ DQG *UXQLQJHU >@ *RPH]3HUH]>@ DQG*UXEHU>@ 2.2. Ontology evaluation :LWKZLGHVSUHDGXVDJHRIRQWRORJLHVLQNQRZOHGJHUHSUHVHQWDWLRQGHYHORSPHQWDQGUHXVHRIRQWRORJLHVEULQJ XS TXHVWLRQ RI HYDOXDWLRQ RI RQWRORJLHV 7KHUH DUH YDULRXV HYDOXDWLRQ PHWKRGV WKDW VHDUFK IRU GLIIHUHQW PHDVXUHV
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
ZLWKTXDOLWDWLYHRUTXDQWLWDWLYHPHWULFVIRUGLIIHUHQWRQWRORJLHVKRZHYHUQRJOREDODSSURDFKKDVEHHQDGRSWHG>@ $VLWLVSUHYLRXVO\VWDWHG RQWRORJLHVYDU\LQWKHLUGRPDLQVL]HSXUSRVHODQJXDJHDQGPD\EHKDQGFDUYHGPD\EH FRQVWUXFWHG E\ VFKRODUV RU GRPDLQ H[SHUWV RU PD\ EH REWDLQHG E\ DQ DXWRPDWLF RU VHPLDXWRPDWLF SURFHVV 7KLV VLWXDWLRQOHDGVHYDOXDWLRQRIRQWRORJLHVWREHFRPSOLFDWHGDQGSUHYHQWVUHOHDVHRIDVWDQGDUGHYDOXDWLRQSURFHGXUH >@ $GGLWLRQDOO\ HYDOXDWLRQ LV UHTXLUHG GXULQJ ZKROH OLIHF\FOH RI DQ RQWRORJ\ QDPHO\ GXULQJ SUHPRGHOOLQJ GXULQJ PRGHOOLQJ EHIRUH UHOHDVH DQG DIWHU UHOHDVH >@ 0DQ\ VWXGLHV KHOG E\ GLIIHUHQW UHVHDUFKHUV LQ WKH RQWRORJ\HYDOXDWLRQDUHDDGGUHVVWKHSUREOHPDQGSRLQWXSLPSRUWDQFHRIWKHLVVXH7KHUHDUHPHWKRGVWKDWDUHHLWKHU PDQXDORUDXWRPDWLFHLWKHUGHDOLQJZLWKWD[RQRP\RUHYDOXDWLQJWKHFRQWHQWDQGHLWKHUGHSHQGHQWWRVRPHVSHFLILF WRROODQJXDJH RU LQGHSHQGHQW >@ 7KLV GRXEOH YDULHW\ RI RQWRORJ\ GHYHORSPHQW DQG HYDOXDWLRQ LQFUHDVHV WKH FRPSOH[LW\ LQ WKH DUHD DQG NHHSV WKH HYDOXDWLRQ SUREOHP XQVROYHG >@ $FFRUGLQJO\ WR REWDLQ D UHOLDEOH RQWRORJ\FRPSOHPHQWDU\XVHRIGLIIHUHQWHYDOXDWLRQPHWKRGVLVDSSUHFLDWHG>@ 2.2.1. Classification of ontology evaluation methods 7KHUHDUHYDULRXVFODVVLILFDWLRQDWWHPSWVRQHYDOXDWLRQPHWKRGVDQGWRROVWKHGHWDLOVRIZKLFKDUHJLYHQEHORZ 7KHFRPPRQSRLQWEHWZHHQWKHPLV LQJHQHUDOWHUPVWKH\DUHHLWKHUverification methodsWKDWHQVXUHWKHRQWRORJ\LV FRQVWUXFWHGFRUUHFWO\RUvalidation methodsWKDWHQVXUHWKHRQWRORJ\UHSUHVHQWVWKHUHDOZRUOG> @ +DUWPDQQHWDO>@FODVVLI\RQWRORJ\HYDOXDWLRQIRUWKHPHWKRGVDV OntoMetric±$QDO\WLF+LHUDUFK\3URFHVV EDVHGPHWKRGWKDWSUHVHQWVVHWRIPHDVXUHVWRVHOHFWWKHEHVWRQWRORJ\IRUDSDUWLFXODUQHHG Natural Language Application Metrics ± PDLQO\ DVVRFLDWLQJ PHQWLRQV LQ WKH WH[W ZLWK WKH FRQFHSWV LQ WKDW RQWRORJ\ SUHFLVLRQ DQG UHFDOO PHWULFV FRVWEDVHG HYDOXDWLRQ PHWULF RQWRORJ\ ILW ± WKH WHQQLV PHDVXUH DQG OH[LFDO FRPSDULVRQ OHYHO PHDVXUH OntoClean ± HYDOXDWLRQ RI IRUPDO VWUXFWXUH RI WKH WD[RQRP\ E\ WKH RQWRORJLFDO QRWLRQV RI ULJLGLW\ XQLW\LGHQWLW\DQGGHSHQGHQFH EvaLexon±PDLQO\FUHDWLQJRQWRORJLHVIURPWH[WFRPSDULVRQRIWKHYRFDEXODU\ ZLWKWH[WFDOFXODWLRQRIFRYHUDJHDFFXUDF\SUHFLVLRQUHFDOO$GGLWLRQDOO\WKH\SUHVHQWWKHDSSOLFDWLRQVDQGWRROVDV ODEval±WRROWKDWGHWHFWVSRVVLEOHSUREOHPVLQFRQFHSWWD[RQRPLHV OntoManager±WRROWKDWHYDOXDWHVWKH WUXWKIXOQHVVRIDQRQWRORJ\ZLWKUHVSHFWWRWKHGRPDLQ *DQJHPL HW DO >@ LGHQWLI\ WKUHH FDWHJRULHV DV structural measures ± HYDOXDWLRQ RI IRUPDO VWUXFWXUH RI WKH RQWRORJ\ WKH JUDSK functional measures ± HYDOXDWLRQ RI WKH UHODWLRQ EHWZHHQ WKH RQWRORJ\ DQG WKH LQWHQGHG PHDQLQJ KRZ LW VHUYHV IRU WKH SXUSRVH LWV FRJQLWLYH VHPDQWLFV PDLQO\ SUHFLVLRQUHFDOOEDVHG PHDVXUHV DQG usability-profiling measures ± HYDOXDWLRQ RI WKH RQWRORJ\ SURILOH FRPPXQLFDWLRQ FRQWH[W RI WKH RQWRORJ\ LWV SUDJPDWLFV Structural measures DUH GHSWK EUHDGWK WDQJOHGQHVV IDQRXWQHVV VSHFLILF GLIIHUHQFH GHQVLW\ PRGXODULW\ORJLFDODGHTXDF\PHWDORJLFDODGHTXDF\GHJUHHGLVWULEXWLRQFunctional measuresDUHSUHFLVLRQUHFDOO DQG DFFXUDF\ ZKLFK DUH PHDVXUHG DJDLQVW H[SHUWV¶ MXGJPHQW RU D GDWD VHW DVVXPHG DV D TXDOLILHG H[SUHVVLRQ RI H[SHUWV¶ 7KH\ JLYH IXUWKHU GHWDLO IRU IXQFWLRQDO PHDVXUHV DV MXGJPHQW E\ DJUHHPHQW DVVHVVPHQW XVHU VDWLVIDFWLRQDVVHVVPHQW WDVNDVVHVVPHQW±ZKDWLVVXSSRUWHGE\WKHRQWRORJ\" VHUYLFHVSHFLILFDWLRQEDVHGJROG VWDQGDUGVROXWLRQEDVHGH[SOLFLWWDVNEDVHG WRSLFDVVHVVPHQW±ERXQGDULHVRIWKHNQRZOHGJHGRPDLQDGGUHVVHG E\ WKH RQWRORJ\ PRGXODULW\ DVVHVVPHQW ± EXLOGLQJ EORFNV IRU WKH GHVLJQ RI WKH RQWRORJ\ )LQDOO\ usabilityprofiling measuresDUHKDQGOHGDVUHFRJQLWLRQHIILFLHQF\DQGLQWHUIDFLQJ )HUQDQGH]%UHLVHWDO>@VWDWHWKDWRQWRORJ\HYDOXDWLRQLVVXHKDVEHHQKDQGOHGIURPGLIIHUHQWSHUVSHFWLYHVVRLW LV SRVVLEOH WR JURXS WKHP DV HYDOXDWLRQ ZLWK WKH DLP RI ranking correctness RU quality *HQHUDOO\ UDQNLQJ DQG FRUUHFWQHVVDUHHYDOXDWHGIRUWKHstructural property ZKHUHDVXVDELOLW\UHOLDELOLW\DQGIXQFWLRQDOLW\LVFRQVLGHUHGLQ WKH VHDUFK RI RQWRORJ\ quality 7KH\ XQGHUOLQH WKDW RQWRORJ\ GHYHORSPHQW SURFHGXUHV EHDU WUDFHV RI 6RIWZDUH (QJLQHHULQJDQGWKH\XQGHUWDNHWKHHYDOXDWLRQLVVXHIURPWKDWSHUVSHFWLYH ,62 VWDQGDUGIRUVRIWZDUHTXDOLW\ DVZHOO6RIWZDUHHYDOXDWLRQFRQVLVWVRIinternalexternalDQGin-use TXDOLW\PHDVXUHVInternal measuresFRQVWLWXWH WKH PHDVXUHV WKDW DUH UHODWHG ZLWK WKH VRIWZDUH LWVHOI DQG JHQHUDOO\ KHOG GXULQJ HDUO\ GHVLJQ SURFHVV ZKHUHDV external measures DUH WKH RQHV HYDOXDWHG GXULQJ WHVWLQJ RU RSHUDWLQJ SURFHVV LQ ODWHU GHVLJQ RU EHIRUH UHOHDVH SURFHVVDQGWKH\DUHJHQHUDOO\UHODWHGZLWKWKHEHKDYLRURIWKHVRIWZDUH,QOLJKWRIWKHVHWKH\SURSRVHDQRQWRORJ\ HYDOXDWLRQ IUDPHZRUN WKDW LV VWUXFWXUHG IURP VHYHQ TXDOLW\ GLPHQVLRQV DV structural functionality reliability usabilityefficiencymaintainabilityDQGquality in use
63
64
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
1HW]HUHWDO>@JURXSWKHRQWRORJ\HYDOXDWLRQSXUSRVHDVassessing the qualityE\LWVGHYHORSHUVRUranking the ontologies WR FKRRVH IRU VRPH SDUWLFXODU XVH $GGLWLRQDOO\ WKH\ JURXS WKH HYDOXDWLRQ PHWKRGV DV intrinsic DQG extrinsicPHWKRGVIntrinsic methodsIRFXVHYDOXDWLRQRIWKHVWUXFWXUDOSURSHUWLHV ZKHUHDVextrinsic methodsUHTXLUH VRPHH[WHUQDOLQIRUPDWLRQVXFKDVDFRUSXVWKDWUHSUHVHQWVWKHGRPDLQNQRZOHGJH GDWDGULYHQHYDOXDWLRQ H[SHUW RSLQLRQRUDSDUWLFXODUWDVN =RXDTDQG1NDPERX>@PHQWLRQstructural evaluationsDQGRWKHUDSSURDFKHVDVgold standards±FRPSDULQJ WKHRQWRORJ\ZLWKD³JROGVWDQGDUG´application-based±XVLQJWKHRQWRORJ\ZLWKDQDSSOLFDWLRQDQGHYDOXDWLQJWKH UHVXOWVdata-driven±FRPSDULQJWKHRQWRORJ\ZLWKDVRXUFHRIGDWDIURPWKHGRPDLQWREHFRYHUHGDQGassessment by domain experts 3DWULFN DQG )ORULDQ >@ FODVVLI\ RQWRORJ\ HYDOXDWLRQ SURFHVV DFFRUGLQJ WR LWV approach ± JROGHQ VWDQGDUG DSSOLFDWLRQEDVHG GDWDGULYHQ DVVHVVPHQW E\ KXPDQV complexity ± PHWKRGV HYDOXDWLQJ OH[LFDOYRFDEXODU\GDWD OD\HUKLHUDUFK\WD[RQRP\FRQWH[WDSSOLFDWLRQOHYHOV\QWD[VWUXFWXUHDUFKLWHFWXUHGHVLJQ DQGevaluation method± HYROXWLRQEDVHGORJLFDOUXOHEDVHGDQGPHWULFIHDWXUHEDVHG7KH\DOVRPHQWLRQHYDOXDWLRQWRROVDQGPHWKRGVDV 2QWR0DQDJHU 2QWR0HWULF 2QWR&OHDQ (YD/H[RQ :(%&25(DQG 62QWR(YDO %DFKLU %RXLDGMUD DQG %HQVOLPDQH >@ GLYLGH DYDLODEOH PHWKRGV LQWR IRXU JURXSV VWDUWLQJ ZLWK what should be evaluatedDV³ZKROHRQWRORJ\HYDOXDWLRQ´RU³SDUWLDORQWRORJ\HYDOXDWLRQ´7KH\IXUWKHUSUHVHQWWKHJURXSVRIwhy it should be evaluated ± ³YDOLGLW\ RI RQWRORJLFDO VWUXFWXUH ´ RU ³TXDOLW\ IRU XVH ´ when it should be evaluated ± ³EHIRUH WKH RQWRORJ\ EXLOGLQJ SURFHVV´ ³GXULQJ WKH RQWRORJ\ EXLOGLQJ SURFHVV´ ³GXULQJ WKH RQWRORJ\ HYROXWLRQ SURFHVV´³EHIRUHUHXVLQJWKHRQWRORJ\´ DQGbased on what it should be evaluated±³FRUSXVEDVHGHYDOXDWLRQ WR HVWLPDWH DFFXUDF\ DQG FRYHUDJH RI WKH RQWRORJ\ ´ ³JROGVWDQGDUGEDVHG HYDOXDWLRQ WR FRPSDUH RQWRORJ\ ZLWK D JROGVWDQGDUG ´ ³WDVNEDVHG HYDOXDWLRQ WR VHDUFK KRZ XVH RI RQWRORJ\ HIIHFWV WKH DSSOLFDWLRQ ´ ³H[SHUWEDVHG HYDOXDWLRQ HYDOXDWLRQE\KXPDQH[SHUWV ´³FULWHULDEDVHGHYDOXDWLRQ WRPHDVXUHILWRIRQWRORJ\WRVRPHFULWHULD ´ %RORWQLNRYDHWDO>@FODVVLI\WKHPHWKRGVRIRQWRORJ\HYDOXDWLRQDFFRUGLQJWRthe goal of evaluationthe stage of evaluationthe degree of automationthe object of the analysisDQGthe tools applied for analysis$QDO\VLVWRROV DUH FODVVLILHG DV ³GDWDGULYHQ WRROV´ ³H[SHUW HYDOXDWLRQV´ ³LQYHVWLJDWLRQ RI SURILOHV RI XVH´ ³FRPSDULVRQ ZLWK µJROGHQVWDQGDUG¶´DQG³LQYHVWLJDWLRQRIWRSRORJLHVRIWKHRQWRORJ\JUDSK´7KHDXWRPDWLRQGHJUHHRIWKHVHWRROV DUHKDQGOHGWKURXJKWKHLU³DXWRPDWLF´³VHPLDXWRPDWLF´DQG³PDQXDO´LQGLFDWLRQ ZKLOHWKHDSSOLFDWLRQVWDJHRIWKH DQDO\VLVLVJURXSHGDV³GHYHORSPHQWDQGSURWRW\SLQJ´³WHVWLQJEHIRUHUHOHDVH´DQG³XVH´2EMHFWRIWKHHYDOXDWLRQ FDQEH³VWUXFWXUH´³GLFWLRQDU\´DQG³HIILFLHQF\RISUDFWLFDODSSOLFDWLRQ´ DQGWKHUHFDQEHGLIIHUHQWHYDOXDWLRQJRDOV DVGHSLFWHGLQWKHHYDOXDWLRQVFKHPDSURYLGHGLQWKHVWXG\$VDQDGGLWLRQWRWKLVFODVVLILFDWLRQWKH\XQGHUOLQHWKH LPSRUWDQFHRITXDOLW\RIWKHGHVLJQRIWKHRQWRORJLHVDQGPHQWLRQWZRDVSHFWVWREHHYDOXDWHGDV³ FRUUHFWQHVV DQG GHSWK RI UHIOHFWLRQ RI WKH VXEMHFW GRPDLQ DQG HUJRQRPLF DVSHFW RI WKH RQWRORJ\ UHSUHVHQWDWLRQ IURP WKH SRLQW RI YLHZ RI TXDOLW\ DQG KXPDQ VSHHG RI SHUFHSWLRQ´ )XUWKHU WKH\ SUHVHQW GHWDLOHG TXDQWLILDEOH PHWULFV WR HYDOXDWH WKH FRJQLWLYH HUJRQRPLFV RI WKH RQWRORJLHV DQG VWDWH WKH LPSRUWDQFH RI FRJQLWLYH HUJRQRPLFLW\ ZKHQ RQWRORJ\LVDLPHGIRUOHDUQLQJRUSDVVLQJRQNQRZOHGJH 2.2.2. Data-driven evaluation and NLP %UHZVWHUHWDO>@VWDWHWKDWRQWRORJLHVDUHVLPSO\VHWRIFRQFHSWVDQGUHODWLRQVWKDWDUHHLWKHUH[SOLFLWO\GHILQHG RUIROORZLQJIURPVHWRID[LRPVDQGDFFRUGLQJO\WKH\DUHDEVWUDFWLRQVIURPDVHWRIQDWXUDOODQJXDJHWH[WVWKDWDUH GHVFULELQJWKHGRPDLQ$FFRUGLQJO\ WKH\SURSRVHUHYHUVLQJWKHSURFHGXUHIRUHYDOXDWLRQDVILQGLQJFRQFHSWVLQWKH WH[WVWKDWDUHUHODWHGZLWKWKHRQHVLQWKHRQWRORJ\DQGQDPHWKLVSURFHVVDV³FRUSXV´RU³GDWDGULYHQ´HYDOXDWLRQ $GGLWLRQDOO\ *DQJHPL HW DO >@ PHQWLRQ WKDW IXQFWLRQDO PHDVXUHPHQW LV EDVHG RQ ³PDWFKLQJ´ EHWZHHQ WZR VWUXFWXUHVEDVLFDOO\WKHVHDUFKRIFRUUHVSRQGHQFHEHWZHHQWKHRQWRORJ\DQGVRPHWKLQJHOVH7KH\DGGLWLRQDOO\UHIHU 1DWXUDO /DQJXDJH 3URFHVVLQJ 1/3 EDVHG PHWKRGV PHWKRGV EDVHGRQ PDWFKLQJ EHWZHHQ D FRUSXV RI GRFXPHQWV DQG WKH RQWRORJ\ DV JHQHUDOO\ XVHG PHWKRGV IRU IXQFWLRQDO DVVHVVPHQW 7KH\ UHIHU WR 'DHOHPDQV DQG 5HLQEHUJHU >@WKDW1/3EDVHGPHWKRGVSURYLGHWRVKRZWKHRQWRORJLHVDUH³FRQVHQVXDOFRQFHSWXDOL]DWLRQV´UDWKHUWKDQMXVWLWV GHYHORSHUV¶ LGHDV 7KURXJK D FRUSXVEDVHG DQDO\VLV HPSLULFDO HVWLPDWLRQ RI WKH accuracy DQG coverage RI WKH RQWRORJ\FDQEHPDGH>@,QSDUDOOHOZLWKWKHVH+DUWPDQQHWDO>@SRLQWRXWWZRPHWULFVDV³RQHWKDWVKRZVKRZ ZHOO D µVDPSOH RQWRORJ\¶ EXLOW DV WKH UHVXOW RI D FHUWDLQ PHWKRGRORJ\ KXPDQ PRGHOOLQJ RU DXWRPDWHG PLQLQJ
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
µUHSUHVHQWV¶ D FRPPRQO\ DFFHSWHG UHIHUHQFH FRUSXV DQG RQH WKDW GHWDLOV KRZ WKH RQWRORJ\ DFWXDOO\ RUGHUHG µFRUUHVSRQGV¶ZLWKWKHLQSXWWH[W V RIWKHFXVWRPHU¶VDSSOLFDWLRQGRPDLQ´ (YDOXDWLRQRIGHOD\DQDO\VLVRQWRORJ\ ,QWKLVVHFWLRQWKHGHOD\DQDO\VLVRQWRORJ\LVLQWURGXFHGDQGWKHHVWDEOLVKHGHYDOXDWLRQSURFHGXUHLVSUHVHQWHG 3.1. Delay analysis ontology $V LW LV SUHYLRXVO\ VWDWHG D GHOD\ DQDO\VLV RQWRORJ\ PD\ DFW DV D YRFDEXODU\ RI WKH GHOD\ DQDO\VLV GRPDLQ DQG SURYLGHWKHHDV\XQGHUVWDQGLQJRIWKHGHEDWH$GGLWLRQDOO\ZLWKLWVDELOLW\RIHDV\DGDSWDWLRQWRWKHPDFKLQHVZRUOG RQWRORJ\PD\EHXVHGLQDVRIWZDUHSURJUDPRUIRUPWKHEDVLVRIDFRPSDQ\GDWDEDVHRUGHFLVLRQVXSSRUWV\VWHP 0HWKRQWRORJ\ SURFHGXUH >@ LV IROORZHG VWHS E\ VWHS IRU WKH HVWDEOLVKPHQW RI RQWRORJLFDO VWUXFWXUH DQG WKH WRRO 3URWHJH LV XVHG IRU LWV LPSOHPHQWDWLRQ )RU WKH GHVLJQ RI WKH RQWRORJ\ SULQFLSOH RI 1R\ DQG 0F*XLQQHVV >@ DV nouns LQ VHQWHQFHV WKDW GHVFULEH GRPDLQ FRQVWLWXWH objects/concepts LQ DQ RQWRORJ\ ZKHUHDV verbs LQ VHQWHQFHV FRUUHVSRQGWRrelationsRIWKHFRQFHSWVLQWKHRQWRORJ\FRQVWLWXWHVWKHIRXQGDWLRQRIWKHFRQFHSWXDOEDFNJURXQGRI WKH RQWRORJ\ 7KH PDLQ FRQFHSWV LQ WKH LVVXH DUH H[WUDFWHG IURP OLWHUDWXUH UHYLHZ RQ GHOD\ DQDO\VLV DV ³causes of delay´³types of delay´³responsibility of delay´³claim for impact of delay´³analysis of delay´DQG³prevention of delay´ DFFRUGLQJO\WKHGHOD\DQDO\VLVRQWRORJ\LVGHILQHGLQIHZVHQWHQFHVWKDWDUHDEOHWREHUHDGWKURXJKZD\V IRUPHG E\ SDVVDJHV IURP WKH VXUURXQGLQJ FODVVHV WR WKH FRUH FODVV ³GHOD\´ 7KH IROORZLQJ 8QLILHG 0RGHOOLQJ /DQJXDJH 80/ GLDJUDPFRQVWLWXWHVWKHJUDSKRIWKHJHQHUDWHGRQWRORJ\ )LJ 7KHUHODWLRQVDWFRQFHSWOHYHODUH GHSLFWHG LQ EODFN DQG EOXH ZKLOH WKH UHODWLRQV DW LQVWDQFH OHYHO DUH GHSLFWHG LQ UHG )RU WKH IXOO WD[RQRP\ RI FRQFHSWVEHKLQGWKHJUDSKRIWKHRQWRORJ\WKHGHWDLOVRIWKHRQWRORJ\FDQEHLQYHVWLJDWHGLQWKHVWXG\RI%LOJLQ>@
)LJ&RPSOHWHPRGHORIGHOD\DQDO\VLVRQWRORJ\>@
65
66
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
3.2. Evaluation procedure $V LW LV SUHVHQWHG WKHUH DUH PDQ\ PHWKRGV DYDLODEOH IRU DOPRVW DQ\ NLQG RI HYDOXDWLRQ DQG QHZ RQHV FDQ EH DGGHG DFFRUGLQJ WR WKH VSHFLILF IHDWXUHV RI WKH FUHDWHG RQWRORJ\ :LWKLQ WKH YDULHW\ RI HYDOXDWLRQ PHWKRGV LW LV LPSRUWDQWWRXVHFRPSOHPHQWDU\PHWKRGVWKDWZLOOUHVSRQGWRERWKHYDOXDWLRQDVSHFWVDVverificationDQGvalidation 0XOWLSOH XVHRI PHWKRGV OHDGV D FRQVLVWHQWDQGXVDEOHRQWRORJ\>@)RU H[DPSOH (O'LUDE\ DQG:DQJ>@XVH FRPSHWHQF\TXHVWLRQVH[SHUWHYDOXDWLRQVDQGWHVWVRQVDPSOHGRFXPHQWVWRHYDOXDWHWKHLURQWRORJ\LQWHUPVRIHDVH RI QDYLJDWLRQ DEVWUDFWLRQ FRQVHQVXV UHSUHVHQWDWLRQ DQG FRQVLVWHQF\ (YDOXDWLRQ FDQ EH GRQH IRU WKH VHDUFK RI YDULRXV PHDVXUHV KRZHYHU ZKHQ LWV VSHFLILF XVH LV WKRXJKW FULWHULD UHOHYDQW IRU WKDW XVH VKRXOG EH WDNHQ LQWR FRQVLGHUDWLRQ>@*DQJHPLHWDO>@UHIHUWR1R\>@DVLWLVVWDWHGWKDWNQRZLQJWKHRQWRORJ\LVDFFXUDWHWRVRPH IRUPDOFULWHULDQDPHO\LWVVWUXFWXUDOTXDOLW\RQO\LPSOLHVWKDWWKHRQWRORJ\LVXVDEOHEXWLWGRHVQRWJLYHDQ\LGHD DERXWLWVVXLWDELOLW\IRUWKHFDVHLQKDQG,QOLJKWRIWKHVHUHJDUGLQJWKHHYDOXDWLRQRIGHOD\DQDO\VLVRQWRORJ\ ILUVWRI DOO VLQFH LW LV FRQVWUXFWHG ZLWK D PHWKRGRORJ\ 0HWKRQWRORJ\ WKDW LQGLFDWHV HYDOXDWLRQ VWHS DV D FRQWLQXRXV VWHS GXULQJWKHFRQVWUXFWLRQRIWKHRQWRORJ\YHULILFDWLRQRIWKHRQWRORJ\LVFRQVLGHUHGIURPWKHEHJLQQLQJWRWKHHQGRI WKH RQWRORJ\ FRQVWUXFWLRQ SURFHVV &RPSHWHQF\ TXHVWLRQV DUH XVHG LQ WKH VSHFLILFDWLRQ VWHS RI WKH PHWKRGRORJ\ ZKLFKLVDPHWKRGSURSRVHGE\*UXQLQJHUDQG)R[>@DVWKHTXHULHVKHOSLQJWKHFRQVWUXFWLRQDQGDOVRHYDOXDWLRQ RIWKHRQWRORJ\7KHIROORZLQJcompetency questionsDUHXVHGWRVWUXFWXUHWKHRQWRORJ\ ³:K\WKLVRQWRORJ\LVEHLQJ EXLOW"´³:KRDUHLWVXVHUV"´³:KDWLVGHOD\"´³:KDWLVGHOD\DQDO\VLV"´³:KDWGRHVKDSSHQLQFDVHRIDGHOD\"´ ³:KDWDUHWKHFDXVHVRIGHOD\"´³:KRDUHUHVSRQVLEOHIURPGHOD\"´³:KDWVKRXOGEHGRQHLQFDVHRIDGHOD\"´ ³:KDW VKRXOG EH GRQH IRU WKH SUHYHQWLRQ RI GHOD\"´ $GGLWLRQDOO\ YDULRXV GHVLJQ FULWHULD SUHVHQWHG E\ GLIIHUHQW UHVHDUFKHUV>@KDYHDFRQWURORYHUWKHFRQVWUXFWLRQRIWKHRQWRORJ\DQGDUHWDNHQLQWRFRQVLGHUDWLRQDVIDUDV SRVVLEOH2EMHFWLYLW\LVWULHGWREHSURYLGHGWRSUHYHQWDPELJXLW\DQGLQFRQVLVWHQWGHILQLWLRQVDUHQRWLQFOXGHGLQWKH RQWRORJ\ $OVR WKH FUHDWHG RQWRORJ\ LV HDV\ WR XSGDWH GXH WR LWV FRPPRQ UHSUHVHQWDWLRQ V\VWHP DQG KDV RQO\ WKH FRPPRQ GHILQLWLRQV RI WKH VXEMHFW VR GRHV QRW OLPLW LWV XVHUV ZKHQ LWV XVDJH LV LQ TXHVWLRQ 0RUHRYHU IRU WKH LPSOHPHQWDWLRQRIWKHRQWRORJ\DUHOLDEOHWRRO 3URWHJH LVXVHGWKDWZRXOGIRUPDQDQWLFLSDWHGRQWRORJ\VNHOHWRQ $FFRUGLQJO\WKHYHULILFDWLRQRIWKHRQWRORJ\LVVHDUFKHGDVDQLQLWLDOVWHSRIHYDOXDWLRQDQGZKDWLVFUXFLDODWWKLV SRLQWLVWKHLQYHVWLJDWLRQRIPDWFKLQJRIWKHFRQVWUXFWHGRQWRORJ\WRUHDOZRUOG%HVLGHV/RYUHQFLFDQG&XEULOR>@ VWDWHWKDWVPDOORQWRORJLHVFDQEHFUHDWHGZLWKRQO\IHZPLVWDNHVKRZHYHULQFRPSOHWHQHVVSUREOHPDOZD\VH[LVWV $FFRUGLQJO\ IRU D OLJKWZHLJKW DQG WKLV W\SH RIRQWRORJ\ VHDUFKIRU FRPSOHWHQHVV LV DOVR LPSRUWDQW > @ %HWZHHQ DYDLODEOH RSWLRQV IRU YDOLGDWLRQ H[SHUW LQWHUYLHZV FDVH VWXGLHV FRPSDUDWLYH DQDO\VLV RI LQGXVWU\ GRFXPHQWV DQG FRPSHWHQF\ TXHVWLRQV WKH YDOLGDWLRQ RI WKH RQWRORJ\ WKURXJK FDVH VWXGLHV LV VHOHFWHG >@ 5HDO expert reports IRU FODLP DQDO\VLV DUH VHOHFWHG VLQFH WKH\ DUH WKH SULPDU\ GRFXPHQWV IRU GHOD\ DQDO\VLV 7KH WHFKQLTXH KDQGOHGLQWKLVVWXG\LVSDUDOOHOZLWK1/3EDVHGPHWKRGVDQGLWVHDUFKHVIRUWKHcompleteness DQGsuitability for the useRIWKHRQWRORJ\$FFRUGLQJWRWKHFODVVLILFDWLRQSUHVHQWHGE\%RORWQLNRYDHWDO>@ WKHPHWKRGIROORZHGFDQEH GHHPHG DV D ³PDQXDO´ DQG ³GDWDGULYHQ´ PHWKRG DSSOLHG IRU ³WHVWLQJ EHIRUH UHOHDVH´ ZLWK WKH JRDO RI ³FRPSOHWHQHVV DQG SUHFLVLRQ RI WKH GLFWLRQDU\ RI WKH VXEMHFW GRPDLQ´ DQG ZLWK WKH DQDO\VLV REMHFW DV WKH ³GLFWLRQDU\´ 'HSLFWLRQRIHYDOXDWLRQGHWDLOV 7KUHHFDVHVWXGLHVIURPFODLPDQDO\VLVDUHVHOHFWHGEHWZHHQILYHDYDLODEOHGXHWRWKHLUGHWDLOKDQGOHGLQ3KUDVHV LQ WKH UHSRUWV DUH VHSDUDWHG LQWR FRQFHSWV XQGHUOLQHG DQG WKH\ DUH PDWFKHG ZLWK WKH UHODWHG FRQFHSWV LQ WKH RQWRORJ\DVLQWKHH[DPSOHSURYLGHGLQIROORZLQJWDEOH 7DEOH 3KUDVHVRIWKHUHSRUWDUHWRWDOO\NHSWLQTXRWDWLRQ PDUNVZKHUHDVRQO\FRQFHSWVLQWKHRQWRORJ\DUHSUHVHQWHGLQTXRWDWLRQPDUNV7RVSHFLI\WKHSODFHRIWKHFRQFHSW LQ WKH RQWRORJ\ XSSHU FRQFHSWV GHILQHG DUH DOVR LQFOXGHG )RU WKH LQFOXVLRQ RI LQIRUPDWLRQ RI UHODWLRQV LQ WKH RQWRORJ\ WKH ZRUG ³XQGHU´ LV XVHG WR UHIHU WD[RQRPLF UHODWLRQV EHWZHHQ FRQFHSWV ZKHUHDV WKH GHPRQVWUDWLRQ ³ DVVRFLDWLRQBQDPH´LVXVHGIRUWKHQRQWD[RQRPLFUHODWLRQV6R³SKUDVHVLQRQWRORJ\´DUHLQGLFDWHGLQSDWKIRUPV VXFKDV ³FRQFHSW´XQGHU³FRQFHSW´DVVRFLDWLRQ³FRQFHSW´7KHIROORZLQJWDEOH 7DEOH H[HPSOLILHVWKHSURFHVV KHOGGXULQJHYDOXDWLRQ
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68
67
7DEOH([DPSOHIRUFRPSDULVRQRIFRQFHSWV 3KUDVHLQWKH5HSRUW 2EWDLQLQJEXLOGLQJ SHUPLWV &RQWUDFWRU LQIRUPVWKH(QJLQHHU DERXWWKHGHOD\GXHWR REWDLQLQJEXLOGLQJ SHUPLWVLVQRWLQWKHLU FRQWUDFWRU V UHVSRQVLELOLW\DQG FODLPVWKHUHTXLUHG H[WHQVLRQRIWLPH (QJLQHHUDSSURYHVWKH FRQWUDFWRU VFODLPDQG ZLWKWKHFRQVHQWRI RZQHUGD\VRI H[WHQVLRQLVDZDUGHG WRFRQWUDFWRUDQGWKH QHZFRQWUDFW FRPSOHWLRQGDWHLV GHWHUPLQHG
3KUDVHLQWKH2QWRORJ\ 1RWLFHIRU&ODLPIRU,PSDFWRI'HOD\ %XLOGLQJSHUPLWVDSSURYDOSURFHVVXQGHU5XOHVDQG5HJXODWLRQV5HODWHG&DXVHVXQGHU([WHUQDO&DXVHVXQGHU &DXVHVRI'HOD\ 2ZQHU RUKLVDJHQWV UHVSRQVLEOHXQGHU5HVSRQVLELOLW\RI'HOD\ 1HLWKHUFRQWUDFWXDOSDUW\UHVSRQVLEOHXQGHU5HVSRQVLELOLW\RI'HOD\ &ODLPIRU,PSDFWRI'HOD\ ([WHQVLRQRIWLPHFODLPXQGHU7LPH5HODWHG&ODLPVXQGHU.LQGVRI&ODLPIRU,PSDFWRI'HOD\ 6HWWOHPHQWLVD5HVXOWRI&ODLPIRU,PSDFWRI'HOD\ ([WHQVLRQRIWLPHXQGHU$ZDUGWKURXJK6HWWOHPHQWLVD5HVXOWRI&ODLPIRU,PSDFWRI'HOD\ $ZDUGWKURXJK6HWWOHPHQWLVD5HVXOWRI&ODLPIRU,PSDFWRI'HOD\ &XUUHQWFRPSOHWLRQGDWHXQGHU'HWDLOVRI&ODLPXQGHU3DUWVRI&ODLPIRU,PSDFWRI'HOD\ 3UDFWLFDOFRPSOHWLRQ6XEVWDQWLDOFRPSOHWLRQDQGLQLWLDOFHUWLILFDWHXQGHU&RQWUDFW&ODXVHVXQGHU0DLQ&RQWUDFW 'RFXPHQWVXQGHU&RQWUDFW'RFXPHQWVXQGHU'DWDXVHGBIRU$QDO\VLVRI'HOD\ 8SGDWHDQGUHDQDO\]HWKHQHWZRUN VSHFLI\WKHSURMHFWLVDKHDGRUEHKLQGWKHVFKHGXOHDWGHOD\GDWHXQGHU$SSO\ $QDO\VLV7HFKQLTXHXQGHU(YHQW$QDO\VLVXQGHU0HWKRGRORJ\6WHSVIRU$QDO\VLVRI'HOD\ $GMXVWHG8SGDWHGVFKHGXOH VFKHGXOHGHSLFWLQJLPSDFWVE\FKDQJHVRQDVSODQQHGVFKHGXOHXQGHU0DMRU6FKHGXOHV XQGHU3RVWFRQWUDFW'RFXPHQWVXQGHU'DWDXVHGBIRU$QDO\VLVRI'HOD\
'LVFXVVLRQDQGFRQFOXVLRQ 7KHFRQFHSWVZLWKLQWKHUHSRUWVDQGWKHRQWRORJ\DUHPDWFKHGRQO\VHYHUDODGGLWLRQVDUHPDGHDVDUHVXOWRIWKLV HYDOXDWLRQ7KHUHODWLRQVDUHQRWLQYHVWLJDWHGVHSDUDWHO\VLQFHWKH\FRPSOHPHQWWKHH[LVWHQFHRIWKHVWDWHGFRQFHSWV :KHQ WKH PDWFKLQJ EHWZHHQ FRQFHSWV LQ WKH RQWRORJ\ DQG UHSRUWV LV FRQVLGHUHG LW LV VHHQ WKDW WKH RQWRORJ\ KDV HQRXJK FRQFHSWV WR UHSUHVHQW UHDO ZRUOG NQRZOHGJH IRU WKHVH FDVHV DQG LW KDV WKH SRWHQWLDO WR HDVH FODLP PDQDJHPHQWSURFHVVOLNHWKHRQHVLQWKHFDVHVWXGLHV$GGLWLRQDOO\FKDQJHVFDQHDVLO\EHPDGHDVLQWKHSURFHVVRI DGGLWLRQ RI FRQFHSWV WR PHHW VRPH UHTXLUHPHQW IRU LWV VSHFLILF XVH 7KLV HYDOXDWLRQ SURFHVV FRQVWLWXWHV D ILQDO HYDOXDWLRQRIWKHFRQVWUXFWHGRQWRORJ\EHIRUHLWVUHOHDVH0RUHH[SHUWUHSRUWVFDQEHKDQGOHGDQGH[SHUWRSLQLRQVRQ RQWRORJ\ FDQ EH WDNHQ IRU IXUWKHU HYDOXDWLRQ +RZHYHU HYDOXDWLRQ RI WKH RQWRORJ\ LV OLPLWHG WR D OHYHO ZLWK WKH PHQWLRQHGPHWKRGV$FWXDOHYDOXDWLRQRIWKHRQWRORJ\FDQEHGRQHE\DFWLYHXVDJHRIWKHRQWRORJ\LQDFRPSDQ\ GDWDEDVHDQGE\WKHSRVVLEOHXSGDWHVWKDWZRXOGEHUHTXLUHGGXULQJLWVXVDJH 2QWRORJLHV DQG WKHLU HYDOXDWLRQ PHWKRGV LV LQWURGXFHG LQ WKLV VWXG\ DQG H[HPSOLILHG ZLWK D GHOD\ DQDO\VLV RQWRORJ\DQGLWVHYDOXDWLRQSURFHVV,WLVEHOLHYHGWKDWRQWRORJ\DQGLWVHYDOXDWLRQDUHDOZD\VZHOFRPHGDVORQJDV WKH\DUHVXSSRUWHGZLWKIRUPDOSURFHVVHVDQGRQWRORJLHVFDQEHDFWXDOO\YDOLGDWHGDVORQJDVWKH\DUHDFWLYHO\XVHG 5HIHUHQFHV >@2&RUFKR0)HUQDQGH]/RSH]$*RPH]3HUH]0HWKRGRORJLHVWRROVDQGODQJXDJHVIRUEXLOGLQJRQWRORJLHV:KHUHLVWKHLUPHHWLQJ SRLQW"'DWD .QRZOHGJH(QJLQHHULQJ ± >@1)1R\'/0F*XLQQHVV2QWRORJ\GHYHORSPHQWDJXLGHWRFUHDWLQJ\RXUILUVWRQWRORJ\.6/7HFKQLFDO5HSRUW6WDQIRUG8QLYHUVLW\ 6WDQIRUG&$86$ >@75*UXEHU$WUDQVODWLRQDSSURDFKWRSRUWDEOHRQWRORJ\VSHFLILFDWLRQV.QRZOHGJH$FTXLVLWLRQ ± >@75*UXEHU7RZDUGSULQFLSOHVIRUWKHGHVLJQRIRQWRORJLHVXVHGIRUNQRZOHGJHVKDULQJ,QWHUQDWLRQDO-RXUQDO+XPDQ&RPSXWHU6WXGLHV ± >@$*DQJHPL&&DWHQDFFL0&LDUDPLWD-/HKPDQQ0RGHOOLQJRQWRORJ\HYDOXDWLRQDQGYDOLGDWLRQ3URFHHGLQJVRIWKHUG(XURSHDQ 6HPDQWLF:HE&RQIHUHQFH (6:& %XGYD0RQWHQHJUR >@%&KDQGUDVHNDUDQ-5-RVHSKVRQ95%HQMDPLQV:KDWDUHRQWRORJLHVDQGZK\GRZHQHHGWKHP",(((,QWHOOLJHQW6\VWHPV ± >@51HFKHV5()LNHV7)LQLQ75*UXEHU76HQDWRU:56ZDUWRXW(QDEOLQJWHFKQRORJ\IRUNQRZOHGJHVKDULQJ$,0DJD]LQH ± >@($OIRQVHFD60DQDQGKDU3URSRVDOIRUHYDOXDWLQJRQWRORJ\UHILQHPHQWPHWKRGV3URFHHGLQJVRIWKH/DQJXDJH5HVRXUFHVDQG(YDOXDWLRQ &RQIHUHQFH /5(& /DV3DOPDV&DQDU\,VODQGV6SDLQ
68
Gozde Bilgin et al. / Procedia Engineering 85 (2014) 61 – 68 > @+1HI]L0)DUDK,5)DUDK%6RODLPDQ$FULWLFDODQDO\VLVRIOLIHF\FOHVDQGPHWKRGVIRURQWRORJ\FRQVWUXFWLRQDQGHYDOXDWLRQ 3URFHHGLQJVRIVW,QWHUQDWLRQDO&RQIHUHQFHRQ$GYDQFHG7HFKQRORJLHVIRU6LJQDODQG,PDJH3URFHVVLQJ $76,3 6RXVVH7XQLVLD >@,6HUUD5*LUDUGL31RYDLV(YDOXDWLQJWHFKQLTXHVIRUOHDUQLQJQRQWD[RQRPLFUHODWLRQVKLSVRIRQWRORJLHVIURPWH[W([SHUW6\VWHPVZLWK $SSOLFDWLRQV ± >@06)R[0*UXQLQJHU2QRQWRORJLHVDQGHQWHUSULVHPRGHOOLQJ3URFHHGLQJVRIWKH,QWHUQDWLRQDO&RQIHUHQFHRQ(QWHUSULVH,QWHJUDWLRQ 0RGHOOLQJ7HFKQRORJ\ ,&(,07 7XULQ,WDO\ >@$*RPH]3HUH]7RZDUGVDIUDPHZRUNWRYHULI\NQRZOHGJHVKDULQJWHFKQRORJ\([SHUW6\VWHPVZLWK$SSOLFDWLRQV ± >@<1HW]HU'*DED\0$GOHU<*ROGEHUJ0(OKDGDG2QWRORJ\HYDOXDWLRQWKURXJKWH[WFODVVLILFDWLRQ$GYDQFHVLQ:HEDQG1HWZRUN 7HFKQRORJLHVDQG,QIRUPDWLRQ0DQDJHPHQW/HFWXUH1RWHVLQ&RPSXWHU6FLHQFH ± >@$%DFKLU%RXLDGMUD60%HQVOLPDQH)2(YDO )XOORQWRORJ\HYDOXDWLRQ0RGHODQGSHUVSHFWLYHV3URFHHGLQJVRIWKHWK,QWHUQDWLRQDO &RQIHUHQFHRQ1DWXUDO/DQJXDJH3URFHVVLQJDQG.QRZOHGJH(QJLQHHULQJ7RNXVKLPD-DSDQ >@(6%RORWQLNRYD7$*DYULORYD9$*RURYR\7RDPHWKRGRIHYDOXDWLQJRQWRORJLHV-RXUQDORI&RPSXWHUDQG6\VWHPV6FLHQFHV ,QWHUQDWLRQDO ± >@-+DUWPDQQ36S\QV$*LERLQ'0D\QDUG5&XHO0&6XiUH])LJXHURD<6XUH0HWKRGVIRURQWRORJ\HYDOXDWLRQ.QRZOHGJH:HE 'HOLYHUDEOH' >@6/RYUHQFLF0&XEULOR2QWRORJ\HYDOXDWLRQ±&RPSULVLQJYHULILFDWLRQDQGYDOLGDWLRQ3URFHHGLQJVRI&HQWUDO(XURSHDQ&RQIHUHQFHRQ ,QIRUPDWLRQDQG,QWHOOLJHQW6\VWHPV &(&,,6 9DUD]GLQ&URDWLD >@-7)HUQDQGH]%UHLV0($UDQJXUHQ56WHYHQV4XDOLW\HYDOXDWLRQIUDPHZRUNIRUELRRQWRORJLHV3URFHHGLQJVRI,QWHUQDWLRQDO &RQIHUHQFHRQ%LRPHGLFDO2QWRORJ\ ,&%2 %XIIDOR86$ > @$*RPH]3HUH]0)HUQDQGH]/RSH]2&RUFKR2QWRORJLFDO(QJLQHHULQJ6SULQJHU9HUODJ/RQGRQ/LPLWHG86$ >@$=RXDT51NDPERX(YDOXDWLQJWKHJHQHUDWLRQRIGRPDLQRQWRORJLHVLQWKHNQRZOHGJHSX]]OHSURMHFW.QRZOHGJHDQG'DWD(QJLQHHULQJ ± >@)3DWULFN+)ORULDQ2QWRORJ\HYDOXDWLRQ6HPLQHULQ$SSOLHG2QWRORJ\(QJLQHHULQJ:68QLYHUVLW\RI,QQVEUXFN$XVWULD >@&%UHZVWHU+$ODQL6'DVPDKDSDWUD<:LONV'DWDGULYHQRQWRORJ\HYDOXDWLRQ3URFHHGLQJVRI,QWHUQDWLRQDO&RQIHUHQFHRQ/DQJXDJH 5HVRXUFHVDQG(YDOXDWLRQ /5(& /LVERQ3RUWXJDO >@:'DHOHPDQV0/5HLQEHUJHU6KDOORZWH[WXQGHUVWDQGLQJIRURQWRORJ\FRQWHQWHYDOXDWLRQ,(((,QWHOOLJHQW6\VWHPV ± >@$*RPH]3HUH]0)HUQDQGH]/RSH]$GH9LFHQWH7RZDUGVDPHWKRGWRFRQFHSWXDOL]HGRPDLQRQWRORJLHV3URFHHGLQJVRI(&$, :RUNVKRSRQ2QWRORJLFDO(QJLQHHULQJ%XGDSHVW >@*%LOJLQ$QRQWRORJ\EDVHGDSSURDFKIRUGHOD\DQDO\VLV0DVWHUWKHVLV0LGGOH(DVW7HFKQLFDO8QLYHUVLW\7XUNH\ >@7((O'LUDE\%:DQJ(6RFLHW\3RUWDO ,QWHJUDWLQJXUEDQKLJKZD\FRQVWUXFWLRQSURMHFWVLQWRWKHNQRZOHGJHFLW\-RXUQDORI&RQVWUXFWLRQ (QJLQHHULQJDQG0DQDJHPHQW ± >@11R\(YDOXDWLRQE\RQWRORJ\FRQVXPHUV,(((,QWHOOLJHQW6\VWHPV ± >@0*UXQLQJHU06)R[0HWKRGRORJ\IRUWKHGHVLJQDQGHYDOXDWLRQRIRQWRORJLHV3URFHHGLQJVRI:RUNVKRSRQ%DVLF2QWRORJLFDO,VVXHVLQ .QRZOHGJH6KDULQJ0RQWUHDO&DQDGD > @'/DYELF0.ULVSHU)DFLOLWDWLQJRQWRORJ\GHYHORSPHQWZLWKFRQWLQXRXVHYDOXDWLRQ,QIRUPDWLFD ± >@7$(O'LUDE\-=KDQJ$VHPDQWLFIUDPHZRUNWRVXSSRUWFRUSRUDWHPHPRU\PDQDJHPHQWLQEXLOGLQJFRQVWUXFWLRQ$XWRPDWLRQLQ &RQVWUXFWLRQ ± >@+37VHUQJ6</