- Email: [email protected]
Intelligent computer assisted blog writing system
Expert Systems with Applications 39 (2012) 4496–4504
Contents lists available at SciVerse ScienceDirect
Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa
Intelligent computer assisted blog writing system Chien-Liang Liu ⇑, Chia-Hoang Lee, Bo-Yuan Ding Department of Computer Science, 1001 University Road, Hsinchu 300, Taiwan, ROC
a r t i c l e
i n f o
Keywords: Computer assisted writing Natural language processing Blog
a b s t r a c t In this paper, we designed and implemented an intelligent computer assisted blog writing system. The system includes concept expansion module, text generation module and content replacement module. The concept expansion is employed to expand the concept provided by the user and these expanded concepts can provide the system more semantic information. The text generation model is based on keyword generation model and text segment generation model of which the important keywords are extracted from system corpus to serve as the backbone of the template. Meanwhile, the text segments between keywords will make up the content of the template and candidate text segments are retrieved from the corpus based on statistical analysis. The content replacement module employs Google to retrieve content from the Web and ranks the content based on term and POS tagging similarities. The prototype system has shown that it could work well on blog writing application domain and the concept of this research could be extended to other domains easily. Ó 2011 Elsevier Ltd. All rights reserved.
1. Introduction With the popularity of the World Wide Web, the Web has become a new knowledge source. Recently, the concept of ‘‘Web 2.0’’ facilitates communication, information sharing, interoperability, and collaboration on the Web. End users now are not only information consumers, but also information producers. The popularity of blog inspires people to share their feelings or opinions about some events on their own blogs. According to Technorati statistics, over 112.8 million blogs have been recorded and over 250 million pieces of tagged social media exist. Over 1.6 million blog entries are updated every day. In addition to the popularity of the Web, the evolution of tools to facilitate the production and maintenance of Web articles made the publishing process become less technical. Meanwhile, it seems such a potentially rich set of on-line writing activities can provide users opportunities to learn essay writing, because blog writing incorporates familiar writing skills such as organization, paraphrases, and the development of thinking. In recent years, essays have become a major part of a formal education and students are encouraged to have the ability to write in many exams. Although writing is very important, it is still a difficult job for many users to write an article from scratch. Besides, writing is important not only in schools, but also in our daily life. For example, when people communicate with their clients using emails, the ability of writing will facilitate them to express their ideas clearly. According to our observation, many students regard
⇑ Corresponding author. Tel.: +886 3 5131503; fax: +886 3 5734935. E-mail address: [email protected] (C.-L. Liu). 0957-4174/$ - see front matter Ó 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2011.09.139
essay writing as a difficult job, but they are willing to write articles on their own blogs. One of the reasons is that the article’s structure and content are not as formal as the essays and the topic can be anything. Even so, people still can obtain writing experiences from blog writing, since they can learn by doing. Over the last few decades, much research has been done on spelling and grammar checkers and these checkers have been integrated into many word processor applications. In practice, these tools could correct users’ writing errors, but they could not assist the users in writing articles from scratch. For example, if a user would like to compose an article, one major challenge is how to organize the content and how to use appropriate text segments to express his/her ideas rather than the spelling and grammar errors. In this paper, the proposed blog writing system assists users to compose blog articles from a single concept. Basically, a user always starts with a subject or title in his/her mind when he/she is writing a blog article or an essay. For example, if a user wants to write a blog article about travel, he/she will start to collect ideas about travel and use text segments related to travel events. The subject or title can be regarded as the main concept of the article which is supposed to lead the users to finish this article. In addition, people tend to use some text segments or terms that have been appeared in other articles. A common learning approach is learning by doing which could enable users to imitate the structures and patterns in good writings to compose their own articles. Currently, digital content on the Web grows exponentially, and many people are willing to share their blogs on the Web. The Web can be regarded as a big database and many example texts can be obtained from the Web. Based on the above observation, we designed and implemented an intelligent assisted blog writing
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
system. The writing system assists users to compose a blog article from a single concept and regards the Web as system corpus to provide example texts. We collected about 30,000 blog articles from the Internet. Document classification process is first performed to filter out inappropriate articles and the remaining data is used as the basic data corpus. When the user starts to use the system, he/she is asked to provide a concept for the system to generate appropriate content. The concept expansion module derives related concepts based on the concept given by the user and assigns related concepts with weighting information. The system can employ these related concepts with weighting information to rank the content templates each of which contains keywords and text segments between keywords. To generate keywords of the content template, we employ keyword extraction and expansion modules proposed by Liu, Lee, Yu, and Chen (2011) to retrieve representative keywords as well as to expand these keywords. To generate text segments in the content template, we employ statistical approach to obtain the text segments between keywords from the corpus. In addition to content templates, users are allowed to edit or replace content to meet their requirements. The content replacement module employs Google to retrieve candidate content. Since Google may return hundreds of thousands results, we propose a content ranking mechanism, which includes term and PartOf-Speech (POS) tagging similarities, to obtain appropriate content. The prototype system has shown that it could work well on blog application domain and the concept of this research could be extended to another domain easily. The main contributions of this paper include: 1. Design and implement an intelligent computer assisted blog writing system. 2. Propose a replaceable content retrieval mechanism to obtain example texts from the Web. 3. Propose a content ranking mechanism to provide a ranking score for replaceable content. The rest of the paper is structured as follows. In Section 2 a survey of related researches is introduced. In Section 3, the system architecture and design is presented. In Section 4, the experiments and evaluation result are introduced. In Section 5, the conclusion is presented.
2. Related work In essence, the ability to write plays an important role in language learning. Not only can it improve users’ writing skills, but also it helps them develop the ability of communication. Although machine translation is widely used for this purpose, how to find an efficient way in which human collaborates with computers remains an open issue. Recently, various on-line writing assistance tools have been developed to assist users to compose articles. Liu, Zhou, Gao, Xun, and Huang (2000) proposed a computer-aided system that helps Chinese users not only on spelling checking and grammar checking but also on writing. Leacock, Gamon, and Brockett (2009) designed and implemented a prototype web-based writing-assistance tool, the Microsoft Research ESL Assistant, for English language learners. ESL Assistant focuses on types of errors that are typically made by non-native writers of American English and it can actually help a person who produce the text to improve their English language writing skills in course of more realistic everyday writing tasks. Practically, there are many approaches to assist users to compose an article. The availability of enhanced word processors, spelling checkers and grammar checkers could offer assistance to users
4497
in the process of writing. Genthial and Courtin (1992) proposed an architecture of a computer assisted writing system which includes morphological parsing and generation, lexical correction techniques, syntactic parser and document editing and exporting. Kukich (1992) focused on non-words error correction, isolated-word error correction and context-dependent word correction to correct words in text. Bustamante and Leon (1996) presented a grammar and style checker for Spanish and Greek native writers. Paggio (2000) developed a spelling and grammar corrector for Danish and addressed in particular the issue of how a form of shallow parsing is combined with error detection and correction for the treatment of context-dependent spelling errors. Although spelling and grammar checkers could help users correct spelling and grammar errors, these tools could not assist users to organize and to compose an article from scratch. In the past few years, there has been an exponential growth in the number of Internet hosts worldwide and a large amount of data exists in the Internet. The Web can be regarded as a big database and many corpus-based applications started to employ the web to do statistical analysis and hypothesis testing, checking occurrences or validating linguistic rules on a specific universe. Liu et al. (2011) designed and implemented a computer assisted writing system to assist users to compose love letters from scratch. Huang, Cheng, and Huang (2009) designed a system which collected data by searching from Google Blog search1 based on the essays the user has read and written before. Then, the system shows the related documents to user as references. People need to glance at all the paragraphs from beginning to end, since the system presents these primal paragraphs directly. Uchimoto, Isahara, and Sekine (2002) proposed to generate sentences from ‘‘keywords’’ or ‘‘headwords’’. This model considers not only n-gram information, but also dependency information between words. There are various computer-aided writing tools for different topics and fields. Even though these tools offer diverse forms, they have the same purposes, giving user some writing hints immediately for improving the depth and breadth of articles. Predictive text entry is yet another solution, where users can simply select the predictions instead of entering all the characters (Komatsu, Takabayashi, & Masui, 2005). Meanwhile, Komatsu et al. (2005) performed a user study to examine user’s behavior in Japanese writing and the result showed that people prefer selecting to writing in general. The above user study also provides us a direction to design the user interface. 3. System architecture Fig. 1 shows the system flow, which includes data preprocess, concept expansion, keyword extraction and expansion, template article generation and phrase substitution. These modules will be described in the following sections. 3.1. Data preprocess We collected about 30,000 articles from the blog portal of Yahoo! Taiwan, one of the biggest blog communities in Taiwan. Unlike English, Chinese language could not make use of spaces as boundary to separate words in a sentence, so word segmentation for Chinese text is required. We employ ‘‘Chinese Knowledge Information Processing’’ service2 to perform Chinese term segmentation and POS tagging. When the above process is completed, the content is segmented into individual terms with POS tag information. 1
Google Blog Search: http://www.blogsearch.google.com/blogsearch. Chinese Knowledge Information Processing in Academia Sinica: www.ckipsvr.iis.sinica.edu.tw/. 2
http://
4498
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
Fig. 1. System flow.
Obviously, not all the blog articles obtained from the Internet can be used in the system. In this paper, we employ sentiment classification technique to perform the filtering task. In recent years, the problem of ‘‘sentiment classification’’ has seen increasing attention (Esuli & Sebastiani, 2005; Pang, Lee, & Vaithyanathan, 2002; Turney, 2002). Essentially, the task of determining whether a text is positive or negative is similar to the traditional binary classification problem. Given a text, the classifier tries to classify the review into positive category or negative category. However, opinions in natural language are usually expressed in subtle and complex ways. Thus, the challenges may not be addressed by simple text categorization approaches such as n-gram or keyword identification approaches (Mullen & Collier, 2004). Even so, when determining whether a review is positive or negative, standard machine learning outperforms human-proposed baselines (Pang et al., 2002). Ni, Xue, Ling, Yu, and Yang (2007) collected 5000 Chinese blog articles from MSN Space and they classified these articles into informative category and affective category using support vector machines (SVM) approach. Their experimental results achieved about 92% on information retrieval performance measures including precision, recall and F-value. In this paper, the application domain is affective blog articles, and their experimental results will be adopted in the system to facilitate the classification. In general, the blog articles can be roughly divided into two genres, informative and affective (Ni et al., 2007). Informative articles are about some researches, technologies or different kinds of informative news, while affective articles are about personal emo-
tions or feelings. In general, when people are writing informative articles, it is necessary for them to possess background knowledge about domains. For example, if a user wants to write an informative blog article, for instance chemistry, it is reasonable for him/ her to have background knowledge on chemistry. Nowadays, it is still infeasible to apply natural language understanding technique to generate a comprehensive article without external resources about domain knowledge. In this paper, we focus on the issues of the generation of affective articles. Ni et al. (2007) employed SVM to classify blog articles into informative and affective categories. Their experiments yielded 20 most frequent affective terms and 20 most frequent informative terms as shown in Fig. 2. These frequent affective and informative terms are employed to filter out the content with too much informative content. We employ term frequency to obtain the informative and affective scores of an article. In affective score computation, the total frequencies of 20 affective terms can be obtained from the corpus, denoted as Freqsum. Meanwhile, the frequency of each affective term Ti, denoted as Freqi, can be obtained as well. A normalization process is employed to obtain each term’s affective score, which is Freqi/Freqsum. Similarly, the same approach can be applied to informative terms to obtain their informative scores and Fig. 2 shows their scores. Therefore, the affective score and informative score of an article can be obtained from the sum of affective term’s affective score and informative term’s informative score. For example, if an affective term Ti appears in the article K, then we add its affective score to K’s affective score. Each affective
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
4499
Fig. 2. Representative informative and affective terms.
term can contribute an affective score and the sum of these scores represents K’s affective score, denoted as Ak. Similarly, K’s informative score can be obtained using the same approach, and assumed it is Ik. Finally, K’s score can be obtained using the following equation:
Scorek ¼ Ak Ik
ð1Þ
When the above process is completed, each article in the corpus has a score. Only the articles whose scores exceed the average score will be kept and these articles will become content material corpus. 3.2. Concept expansion In our system design, the user is asked to provide a concept or topic for the blog. This concept will then provide the system a direction to select appropriate content templates. The concept expansion module will use HowNet to generate such semantic information as synonyms of the concept. For English language, many researchers employed WordNet (Miller, 1995) to extract synonyms of terms based on synsets to infer whether two terms possessing the same meaning. Meanwhile, the machine can also use WordNet’s tree structure to determine the similarity distance between two terms. In this paper, the application domain is Chinese, so HowNet (Dong & Dong, 2003) is used for semantic information extraction. HowNet is not only a dictionary or thesaurus, but also an on-line common-sense knowledge base unveiling inter-conceptual relations and inter-attribute relations of concepts. Basically, HowNet structure is similar to ontology structure. In HowNet, Concepts are described by Knowledge Description Language (KDL). The basic element of KDL is called sememe which is the basic unit to describe a concept. In HowNet, a concept is defined by a DEF expression with nesting grammar. For example, Fig. 3 shows that the ‘‘doctor’’ concept in HowNet is described by human, own, status, education, HighRank, and most sememes, where human is the main sememe and the other sememes can be regarded as modifiers of main sememe. In general, if two terms T1 and T2 share the same DEF, T1 and T2 are synonyms. However, this approach can only identify the terms with the same meanings. One of the drawbacks is that not all the concepts have synonyms. For the concepts without synonyms, it is not easy for the system to capture the semantic information of the concept. Dai, Liu, Xia, and Wu (2008) regarded the main sememe as being the most important and the other modifying sememes are
Fig. 3. Concept graph of doctor in HowNet.
not distinguished regardless of their position. In this paper, we propose a layer-based weighting mechanism to determine the similarity between two concepts. If we regard each sememe as one layer, we found that the number of layers for most concepts in HowNet (about 77%) is less than three. The main sememe is located at the first position and it is the most important one. Similarly, the second and third layers can be obtained from the positions of sememes. The weighting mechanism is based on the layers shared by two concepts. For any concept C1, if another concept C2 has the first three layers in common with C1, C2 will be assigned three points in C1’s concept expansion. If they only have the first two layers in common, C2 will be assigned two points in C1’s concept expansion. Finally, if only the first layer is shared by C1 and C2, C2 will be assigned one point in C1’s concept expansion. When the above process is completed, the system can expand a single concept into several related concepts with weighting information. For example, Fig. 4 shows a layer-based concept expansion example. In Fig. 4, the first two concepts both represent ‘‘doctor’’, and they share the same sememes, so the second concept is given three points in the first concept’s expansion result. In Fig. 4, the third concept means court physician, and it only shares the sememe in the first layer with doctor concept, so it is given one point in the doctor concept’s expansion result. The concept expansion proposed above can expand a single concept into several related concepts and each extended concept can have a score to represent the degree of similarity with the original concept. Hence, the system can obtain more semantic information from the concept expansion result.
4500
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
Fig. 4. Layer-based concept expansion example.
3.3. Keyword extraction and expansion In this paper, text generation model is keyword based and the user determines a template from the keyword lists each of which represents an article. Practically, it is infeasible to list many keywords for users to determine the content. However, it will be difficult for the system to generate content if the number of keywords is few. Hence, the keyword list plays two important roles in the system design. First, the keywords should be able to provide the insight of the final content. Second, the keywords in the keyword list should provide sufficient information for the system to generate a complete article. Hence, we employ five representative keywords to present the key concepts of the content template to the user and employ hidden keywords to provide content generation backbone to the system. Both representative and hidden keywords are extracted from the corpus. The keyword extraction and expansion flow are shown in Fig. 5, which includes keyword extraction, modified strict phrase likelihood ratio (SPLR) scoring process, and keyword expansion. Practically, nouns and verbs can provide more semantic information than other POS tags, so the system regards nouns and verbs as candidate keywords in keyword extraction step. Basically, an
article contains many nouns and verbs, so it is infeasible to list all the keywords to the users. Meanwhile, the keywords presented to the users should provide insight of the generated content templates to users, so users can determine the final content template from these keywords. In other words, these keywords should not only capture the main concept of the generated article, but also differentiate two different articles. Hence, we employ a modified strict phrase likelihood ratio (SPLR) computational model to obtain the keywords which can highlight the content. Originally, strict phrase likelihood ratio (SPLR) (Chang & Lee, 2003) approach is used to extract Chinese unknown words. In practice, these unknown words could be used to highlight the articles due to their rareness. The system can employ the computation model as shown in Eq. (2) to obtain SPLR score of each candidate keyword. The higher ones will be selected as the core keywords.
SPLR ¼
tf ðKW i Þ ; Maxðtf ðKW i LÞ; tf ðKW i RÞÞ
where KW i len > 1
ð2Þ
In Eq. (2), tf(KWi), tf(KWi_L) and tf(KWi_R) represent term frequency of the keyword, term frequency of left hand side of the keyword and term frequency of right hand side of the keyword respectively. Meanwhile, the basic meaningful unit in Chinese is
Term Extraction
Term List
Modified SPLR Scoring
Expanded Keyword List
Keyword Expansion
Core Keyword List
Corpus
Fig. 5. Keyword extraction and expansion flow.
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
4501
list, the system can compare the keywords in the list with these related concepts and their weighting information to obtain a similarity score. Then, all the keyword lists can be ranked according to the similarity scores, and the top ones can be presented to the user. 3.4. Template article generation
Fig. 6. SPLR scoring example.
In an article, keywords are like the backbone of the article and the text segments between keywords make up the content of the text. When the expanded keywords are extracted from the above process, appropriate text segments between two keywords should be determined as well. Eq. (3) shows how to select candidate text segments between two keywords. jPre KW \ Wordði; jÞ \ Next KWj > 0 CandidateðuÞ C u 2 Unitði; jÞj Wordði; jÞ len 6 Threshold
ð3Þ Fig. 7. Keyword expansion model.
Fig. 8. Keyword expansion model with overlapped keywords.
a phrase rather than a single word, so only the terms with word length larger than one will be taken into account. As shown in Fig. 6, the SPLR score of the keyword ‘‘the president of student association’’ could be obtained from the frequency information. In modified SPLR score computation model, a normalization step is required to normalize the scores so that the final scores will range from 0 to 1. The keywords with the highest five scores will become core keywords. If the number of keywords is less than five, candidate keywords are selected as the core keywords. The benefit of core keywords is that users could have an overview about the final content from the core keywords. As described above, nouns and verbs are selected as candidate keyword list and modified SPLR computational model are used to choose core keywords of the system. In practice, it is not enough to generate an article using five core keywords, since five keywords contain inadequate information to generate an article. Therefore, we propose a keyword expansion process to expand the core keywords and Fig. 7 shows the keyword expansion model. In Fig. 7, W1, W2, . . . , W10 represent the candidate keywords, while W3 and W8 represent core keywords. The expansion model will expand the core keywords to include other keywords as text generation keywords. Practically, if all the candidate keywords are included into the text generation keyword set, the variation of final text will be limited. On the other hand, if the number of keywords is few, it is not easy to find appropriate text segments to fill in the gap between the keywords. In the expansion model, two previous keywords and two next keywords of the core keywords are selected as the generation keywords. Thus, as shown in Fig. 7, W1, W2, W4 and W5 will be expanded from W3. Meanwhile, W6, W7, W9 and W10 will be expanded from W8. If expanded keywords overlap, intersection approach will be adopted and only one overlapped keyword will be selected. Hence, as shown in Fig. 8, keyword W5 is a overlapped keyword and the final generation keywords will be W1, W2, W3, W4, W5, W6, W7, W8 and W9. Based on the core keyword generation model and keyword expansion model, the users could determine the generating text from core keywords and the system could generate text from expanded keywords. As mentioned above, the concept provided by the user will be expanded to related concepts with weighting information. For each keyword
where i is the index of the article, j is the position index of article i, Pre_KW represents the previous keyword, Next_KW represents the next keyword, Unit(i, j) represents the text segments that appear at the jth position of the ith article, Word(i, j) represents the text segments that have appeared between Pre_KW and Next_KW in the corpus and Word(i, j)_len represents the length of Word(i, j). In Eq. (3), the text segments that appear between two keywords and their length are less than threshold value will be selected as candidate text segments. Based on the above process, candidate text segment list could be obtained and these candidate text segments could be mixed with the keywords to generate different texts. Similarly, the content templates can be ranked based on expanding concepts with weighting information. The main goal of the system is to construct an intelligent computer assisted blog writing system to provide a draft version of text for the users. In addition, the users could learn how to improve their writing skills by learning from examples. Fig. 9 shows the text generated by the system and the rough English translation of the content is presented in Appendices. Practically, the content generated by the system provides a reference template for the users and the users could modify the content to meet their requirements. 3.5. Phrase ranking and substitution When the above process is completed, the system presents the user a template article based on the user concept. Obviously, the generated article only provides a template, and the user still need to modify the article to meet his/her requirement. Hence, the system allows the user to edit the generated article. In addition to editing functionality, the system employs Google to provide replaceable content when the user is editing the article. We propose a replaceable content retrieval mechanism to obtain example texts from the Web. Currently, Google provides a snippet, which is a short text, under each search result, and these snippets usually contain query strings. Users can determine whether the result links are useful or not from the snippets. Therefore, if the system sends appropriate query strings to Google, the snippets returned from Google can provide candidate example texts. For example, suppose that a template article contains two keywords KW1 and KW2 and the content between these two keywords is S12. When the users want to change S12, the system will employ KW1 and KW2 with wildcard symbols to compose a query string to obtain snippets. Then, the system analyzes the snippets to obtain replaceable content. In other words, the system makes use of the Google search engine’s outstanding search capability to make the Web become system corpus.
4502
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
shown in the following sections. Meanwhile, ten people were invited to experience the system and several evaluation criteria are listed in the questionnaire to evaluate the system. 4.1. Experimental results
Fig. 9. Template article generated based on keywords.
Most search engines provide wildcard symbols and people can use wildcards to represent any character or range of characters. Google treats the wildcard ‘‘⁄’’, an asterisk character, as a placeholder for a word or more than one word. For example, a query string ‘‘Internet ⁄ my life’’ tells Google to find pages containing a phrase that starts with ‘‘Internet’’ followed by one or more words, followed by ‘‘my life’’. Phrases obtained from Google search result and fit the constraint include: ‘‘Internet in my life,’’ ‘‘Internet changed my life,’’, ‘‘Internet is my life’’, etc. Since Google has indexed a huge amount of pages, the system can obtain several candidate text segments which conform to the constraints. In addition to replaceable content retrieval, we propose a content ranking mechanism to provide a ranking score for replaceable content. We employ the original content as a base to rank the content obtained from Google search. The ranking mechanism includes two ranking scores. The first one is based on the sharing terms between two text segments and another one is based on the similarity of POS tagging between two text segments. In information theory and computer science, the Levenshtein distance between two strings of characters is the number of operations required to transform one of them into the other. For example, if the source string is ‘‘GUMBO’’ and the target string is ‘‘GAMBOL’’, the distance will be two. The transformation from ‘‘GUMBO’’ to ‘‘GAMBOL’’ should consist of a substitution operation (‘‘U’’ for ‘‘A’’) and an addition operation for ‘‘L’’. We apply Levenshtein distance to the distance measurement between two kinds of text segments, which are term sequence and POS tagging sequence. Practically, the term similarity is similar to semantic distance; while the POS tagging similarity is similar to structure distance. The sum of these two scores is used to rank the content obtained from Google. 4. Experiment and discussion In this paper, we designed and implemented an intelligent computer assisted blog writing system. The experimental results are
As described above, the concept expansion module is employed to expand the single concept provided by the user. One of the advantages of this approach is that the system can capture more semantic information and the system can provide more accurate content templates. For example, if the user provides a single concept, which is ‘‘train’’, Fig. 10 shows the keyword lists and content templates, where the system does not employ concept expansion. The content templates in Fig. 10 are generated from the keyword lists with red color. The left content template describes a story where the user and his/her friend went for a bicycle ride and they locked their bicycles in the train station. The right content template describes a story where the user went to eat delicious food when typhoon hit Taiwan. The results show that the relationship between the concept ‘‘train’’ and the content templates is tiny. Meanwhile, Fig. 11 shows the results when the system employs concept expansion mechanism. The left content template describes a scenario where the user went for a bicycle ride and he/she saw many tourists taking pictures when the train went across the road. The right content template describes a scenario where the user enjoyed the outside scene of the train on his/her way home. The results show that the content templates contain more information about train. In content replacement, we employ Levenshtein distance measurement to obtain term and POS tagging similarities between two text segments. Fig. 12 shows the original text segment (the text segment with red color is the source content) and the replaceable content lists obtained from Google. The left list in the Figure is the content list obtained from Google with Google’s ranking order; while the right one is the list using the content ranking approach proposed in this paper. The first two text segments in the left are used to represent category information. Apparently, these two text segments are inappropriate in an article, and their rankings drop a lot when the system employs the content ranking mechanism. Moreover, the rankings of fifth and the seventh text segments are advanced to higher position. If we take into account the context of the original text segment, these two text segments are better than the other ones. 4.2. User evaluation and discussion In user evaluation, user satisfaction evaluation is used. Liu et al. (2011) and Huang et al. (2009) employed similar measurement to evaluate their systems. The system evaluation includes readability, relevance, concept and interface evaluations. Readability item is used to measure the readability of the content generated by the system, relevance item is used to measure the relevance of replaceable content provided by the system, concept item is used to measure whether the content templates generated are related to the concept provided by the users and interface item is used to measure system user interface. Ten people, including 9 males and 1 female and their ages range is from 24 to 40, were invited to experience the system. These people are all interested in blog writing and they are asked to give a score for each item. The score ranges from 1 to 5 and the highest score is 5. Table 1 shows the result which includes average score and the proportion which is higher than 3. Each user was asked to use the system to generate 20 blog articles and then started to use the templates to compose articles. Then he/she evaluated the readability, relevance, interface, and concept items of the articles. The average score of the experiment is higher than 3. Meanwhile, most people thought that the
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
Fig. 10. Keyword list and content templates without concept expansion.
Fig. 11. Keyword list and content templates using concept expansion.
Fig. 12. Comparison of content ranking.
4503
4504
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
Table 1 Evaluation result.
Average Proportion (P3)
Readability
Relevance
Interface
Concept
3.445 86.5%
4.07 92.5%
4.571 99.0%
3.83 89.0%
system could assist them to come up with new ideas from example text segments. As for the system improvement, most people think that the fluency of the content template is not good enough, so content fluency will be our future work. 5. Conclusion An intelligent computer assisted blog writing system, which includes concept expansion, text generation and replaceable content suggestion, is presented in this paper. In this paper, we propose a layer-based concept expansion approach, which is based on HowNet, to extend a single concept into several related concepts with weighting information. Then, the related concepts with weighting information can facilitate the system to rank and determine content templates each of which consists of a keyword list and the text segments between keywords. To generate keywords in the content template, we propose keyword extraction and expansion module to retrieve representative keywords as well as to expand these keywords. To generate text segments in the content template, we employ statistical approach to obtain the text segments between keywords from the corpus. The benefit of this approach is that the template could provide a framework and reduce the physical effort spent on writing so that people can pay attention to organization and content. Moreover, the system allows the users to edit the content template to meet their requirements. When they are editing the content, the system can provide replaceable contents for users’ references. We employ Google to retrieve contents from the Web and we provide a ranking mechanism, which is based on the similarities of terms and POS tagging, to rank the contents obtained from Google. The experiments show that our system can assist users to compose a blog article from a single concept effectively. Acknowledgment This work was supported in part by the National Science Council under the Grants NSC-99-2221-E-009-150 and NSC-099-2811E-009-041. Appendix A Rough translation of the love letter in Fig. 9. Today I visited Ju’s husband’s house with friends. Ju’s husband family took care of us sincerely. When we got there, Ju prepared
fruit and sea food for us. The sea food was very fresh and we finished the sea food immediately. Therefore, Ju had to prepare for another plate of sea food and this photo was about that plate of sea food. Because it was so delicious, we asked Ju to cook next time. After we finished the meal, we went to talk over coffee and enjoyed the view of sea. References Bustamante, F. R., & Leon, F. S. (1996). Gramcheck: A grammar and style checker. In Proceedings of the international conference on computational linguistics (COLING96) (pp. 175–181). Chang, T.-H., & Lee, C.-H. (2003). Automatic chinese unknown word extraction using small-corpus-based method. In Proceedings of international conference on natural language processing and knowledge engineering (pp. 459– 464). Dai, L., Liu, B., Xia, Y., & Wu, S. (2008). Measuring semantic similarity between words using hownet. In ICCSIT’08: Proceedings of the 2008 international conference on computer science and information technology (pp. 601–605). Washington, DC, USA: IEEE Computer Society. Dong, Z., & Dong, Q. (2003). Hownet – a hybrid language and knowledge resource. In Proceedings of the International Conference on Natural Language Processing and Knowledge Engineering, 2003 (pp. 820–824). Esuli, A., & Sebastiani, F. (2005). Determining the semantic orientation of terms through gloss classification. In Proceedings of the 14th ACM international conference on information and knowledge management (pp. 617–624). Genthial, D., & Courtin, J. (1992). From detection/correction to computer aided writing. In Proceedings of the 14th conference on computational linguistics. Association for computational linguistics (pp. 1013–1018). Huang, T.-C., Cheng, S.-C., & Huang, Y.-M. (2009). A blog article recommendation generating mechanism using an sbacpso algorithm. Expert Systems with Applications, 36(7), 10388–10396. Komatsu, H., Takabayashi, S., & Masui, T. (2005). Corpus-based predictive text input. In Proceedings of the 2005 international conference on active media technology, 2005 (AMT 2005) (pp. 75–80). Kukich, K. (1992). Technique for automatically correcting words in text. ACM Computing Surveys, 24, 377–439. Leacock, C., Gamon, M., & Brockett, C. (2009). User input and interactions on microsoft research esl assistant. In EdAppsNLP’09: Proceedings of the fourth workshop on innovative use of NLP for building educational applications. Association for computational linguistics, Morristown, NJ, USA (pp. 73–81). Liu, T., Zhou, M., Gao, J., Xun, E., & Huang, C. (2000). Pens: A machine-aided english writing system for chinese users. In ACL’00: Proceedings of the 38th annual meeting on association for computational linguistics. Association for computational linguistics, Morristown, NJ, USA (pp. 529–536). Liu, C.-L., Lee, C.-H., Yu, S.-H., & Chen, C.-W. (2011). Computer assisted writing system. Expert Systems with Applications, 38(1), 804–811. Miller, G. A. (1995). Wordnet: A lexical database for english. Communications of the ACM, 38(11), 39–41. Mullen, T., & Collier, N. (2004). Sentiment analysis using support vector machines with diverse information sources. In Proceedings of EMNLP-2004 (pp. 412–418). Ni, X., Xue, G.-R., Ling, X., Yu, Y., & Yang, Q. (2007). Exploring in the weblog space by detecting informative and affective articles. In WWW’07: Proceedings of the 16th international conference on world wide web (pp. 281–290). ACM. Paggio, P. (2000). Spelling and grammar correction for danish in scarrie. In Proceedings of the sixth conference on applied natural language processing (pp. 255–261). Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up?: Sentiment classification using machine learning techniques. In EMNLP’02: Proceedings of the ACL-02 conference on empirical methods in natural language processing (pp. 79–86). Turney, P. D. (2002). Thumbs up or thumbs down?: Semantic orientation applied to unsupervised classification of reviews. In ACL’02: Proceedings of the 40th annual meeting on association for computational linguistics (pp. 417–424). Uchimoto, K., Isahara, H., & Sekine, S. (2002). Text generation from keywords. In Proceedings of the 19th international conference on computational linguistics (pp. 1–7).
Contents lists available at SciVerse ScienceDirect
Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa
Intelligent computer assisted blog writing system Chien-Liang Liu ⇑, Chia-Hoang Lee, Bo-Yuan Ding Department of Computer Science, 1001 University Road, Hsinchu 300, Taiwan, ROC
a r t i c l e
i n f o
Keywords: Computer assisted writing Natural language processing Blog
a b s t r a c t In this paper, we designed and implemented an intelligent computer assisted blog writing system. The system includes concept expansion module, text generation module and content replacement module. The concept expansion is employed to expand the concept provided by the user and these expanded concepts can provide the system more semantic information. The text generation model is based on keyword generation model and text segment generation model of which the important keywords are extracted from system corpus to serve as the backbone of the template. Meanwhile, the text segments between keywords will make up the content of the template and candidate text segments are retrieved from the corpus based on statistical analysis. The content replacement module employs Google to retrieve content from the Web and ranks the content based on term and POS tagging similarities. The prototype system has shown that it could work well on blog writing application domain and the concept of this research could be extended to other domains easily. Ó 2011 Elsevier Ltd. All rights reserved.
1. Introduction With the popularity of the World Wide Web, the Web has become a new knowledge source. Recently, the concept of ‘‘Web 2.0’’ facilitates communication, information sharing, interoperability, and collaboration on the Web. End users now are not only information consumers, but also information producers. The popularity of blog inspires people to share their feelings or opinions about some events on their own blogs. According to Technorati statistics, over 112.8 million blogs have been recorded and over 250 million pieces of tagged social media exist. Over 1.6 million blog entries are updated every day. In addition to the popularity of the Web, the evolution of tools to facilitate the production and maintenance of Web articles made the publishing process become less technical. Meanwhile, it seems such a potentially rich set of on-line writing activities can provide users opportunities to learn essay writing, because blog writing incorporates familiar writing skills such as organization, paraphrases, and the development of thinking. In recent years, essays have become a major part of a formal education and students are encouraged to have the ability to write in many exams. Although writing is very important, it is still a difficult job for many users to write an article from scratch. Besides, writing is important not only in schools, but also in our daily life. For example, when people communicate with their clients using emails, the ability of writing will facilitate them to express their ideas clearly. According to our observation, many students regard
⇑ Corresponding author. Tel.: +886 3 5131503; fax: +886 3 5734935. E-mail address: [email protected] (C.-L. Liu). 0957-4174/$ - see front matter Ó 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2011.09.139
essay writing as a difficult job, but they are willing to write articles on their own blogs. One of the reasons is that the article’s structure and content are not as formal as the essays and the topic can be anything. Even so, people still can obtain writing experiences from blog writing, since they can learn by doing. Over the last few decades, much research has been done on spelling and grammar checkers and these checkers have been integrated into many word processor applications. In practice, these tools could correct users’ writing errors, but they could not assist the users in writing articles from scratch. For example, if a user would like to compose an article, one major challenge is how to organize the content and how to use appropriate text segments to express his/her ideas rather than the spelling and grammar errors. In this paper, the proposed blog writing system assists users to compose blog articles from a single concept. Basically, a user always starts with a subject or title in his/her mind when he/she is writing a blog article or an essay. For example, if a user wants to write a blog article about travel, he/she will start to collect ideas about travel and use text segments related to travel events. The subject or title can be regarded as the main concept of the article which is supposed to lead the users to finish this article. In addition, people tend to use some text segments or terms that have been appeared in other articles. A common learning approach is learning by doing which could enable users to imitate the structures and patterns in good writings to compose their own articles. Currently, digital content on the Web grows exponentially, and many people are willing to share their blogs on the Web. The Web can be regarded as a big database and many example texts can be obtained from the Web. Based on the above observation, we designed and implemented an intelligent assisted blog writing
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
system. The writing system assists users to compose a blog article from a single concept and regards the Web as system corpus to provide example texts. We collected about 30,000 blog articles from the Internet. Document classification process is first performed to filter out inappropriate articles and the remaining data is used as the basic data corpus. When the user starts to use the system, he/she is asked to provide a concept for the system to generate appropriate content. The concept expansion module derives related concepts based on the concept given by the user and assigns related concepts with weighting information. The system can employ these related concepts with weighting information to rank the content templates each of which contains keywords and text segments between keywords. To generate keywords of the content template, we employ keyword extraction and expansion modules proposed by Liu, Lee, Yu, and Chen (2011) to retrieve representative keywords as well as to expand these keywords. To generate text segments in the content template, we employ statistical approach to obtain the text segments between keywords from the corpus. In addition to content templates, users are allowed to edit or replace content to meet their requirements. The content replacement module employs Google to retrieve candidate content. Since Google may return hundreds of thousands results, we propose a content ranking mechanism, which includes term and PartOf-Speech (POS) tagging similarities, to obtain appropriate content. The prototype system has shown that it could work well on blog application domain and the concept of this research could be extended to another domain easily. The main contributions of this paper include: 1. Design and implement an intelligent computer assisted blog writing system. 2. Propose a replaceable content retrieval mechanism to obtain example texts from the Web. 3. Propose a content ranking mechanism to provide a ranking score for replaceable content. The rest of the paper is structured as follows. In Section 2 a survey of related researches is introduced. In Section 3, the system architecture and design is presented. In Section 4, the experiments and evaluation result are introduced. In Section 5, the conclusion is presented.
2. Related work In essence, the ability to write plays an important role in language learning. Not only can it improve users’ writing skills, but also it helps them develop the ability of communication. Although machine translation is widely used for this purpose, how to find an efficient way in which human collaborates with computers remains an open issue. Recently, various on-line writing assistance tools have been developed to assist users to compose articles. Liu, Zhou, Gao, Xun, and Huang (2000) proposed a computer-aided system that helps Chinese users not only on spelling checking and grammar checking but also on writing. Leacock, Gamon, and Brockett (2009) designed and implemented a prototype web-based writing-assistance tool, the Microsoft Research ESL Assistant, for English language learners. ESL Assistant focuses on types of errors that are typically made by non-native writers of American English and it can actually help a person who produce the text to improve their English language writing skills in course of more realistic everyday writing tasks. Practically, there are many approaches to assist users to compose an article. The availability of enhanced word processors, spelling checkers and grammar checkers could offer assistance to users
4497
in the process of writing. Genthial and Courtin (1992) proposed an architecture of a computer assisted writing system which includes morphological parsing and generation, lexical correction techniques, syntactic parser and document editing and exporting. Kukich (1992) focused on non-words error correction, isolated-word error correction and context-dependent word correction to correct words in text. Bustamante and Leon (1996) presented a grammar and style checker for Spanish and Greek native writers. Paggio (2000) developed a spelling and grammar corrector for Danish and addressed in particular the issue of how a form of shallow parsing is combined with error detection and correction for the treatment of context-dependent spelling errors. Although spelling and grammar checkers could help users correct spelling and grammar errors, these tools could not assist users to organize and to compose an article from scratch. In the past few years, there has been an exponential growth in the number of Internet hosts worldwide and a large amount of data exists in the Internet. The Web can be regarded as a big database and many corpus-based applications started to employ the web to do statistical analysis and hypothesis testing, checking occurrences or validating linguistic rules on a specific universe. Liu et al. (2011) designed and implemented a computer assisted writing system to assist users to compose love letters from scratch. Huang, Cheng, and Huang (2009) designed a system which collected data by searching from Google Blog search1 based on the essays the user has read and written before. Then, the system shows the related documents to user as references. People need to glance at all the paragraphs from beginning to end, since the system presents these primal paragraphs directly. Uchimoto, Isahara, and Sekine (2002) proposed to generate sentences from ‘‘keywords’’ or ‘‘headwords’’. This model considers not only n-gram information, but also dependency information between words. There are various computer-aided writing tools for different topics and fields. Even though these tools offer diverse forms, they have the same purposes, giving user some writing hints immediately for improving the depth and breadth of articles. Predictive text entry is yet another solution, where users can simply select the predictions instead of entering all the characters (Komatsu, Takabayashi, & Masui, 2005). Meanwhile, Komatsu et al. (2005) performed a user study to examine user’s behavior in Japanese writing and the result showed that people prefer selecting to writing in general. The above user study also provides us a direction to design the user interface. 3. System architecture Fig. 1 shows the system flow, which includes data preprocess, concept expansion, keyword extraction and expansion, template article generation and phrase substitution. These modules will be described in the following sections. 3.1. Data preprocess We collected about 30,000 articles from the blog portal of Yahoo! Taiwan, one of the biggest blog communities in Taiwan. Unlike English, Chinese language could not make use of spaces as boundary to separate words in a sentence, so word segmentation for Chinese text is required. We employ ‘‘Chinese Knowledge Information Processing’’ service2 to perform Chinese term segmentation and POS tagging. When the above process is completed, the content is segmented into individual terms with POS tag information. 1
Google Blog Search: http://www.blogsearch.google.com/blogsearch. Chinese Knowledge Information Processing in Academia Sinica: www.ckipsvr.iis.sinica.edu.tw/. 2
http://
4498
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
Fig. 1. System flow.
Obviously, not all the blog articles obtained from the Internet can be used in the system. In this paper, we employ sentiment classification technique to perform the filtering task. In recent years, the problem of ‘‘sentiment classification’’ has seen increasing attention (Esuli & Sebastiani, 2005; Pang, Lee, & Vaithyanathan, 2002; Turney, 2002). Essentially, the task of determining whether a text is positive or negative is similar to the traditional binary classification problem. Given a text, the classifier tries to classify the review into positive category or negative category. However, opinions in natural language are usually expressed in subtle and complex ways. Thus, the challenges may not be addressed by simple text categorization approaches such as n-gram or keyword identification approaches (Mullen & Collier, 2004). Even so, when determining whether a review is positive or negative, standard machine learning outperforms human-proposed baselines (Pang et al., 2002). Ni, Xue, Ling, Yu, and Yang (2007) collected 5000 Chinese blog articles from MSN Space and they classified these articles into informative category and affective category using support vector machines (SVM) approach. Their experimental results achieved about 92% on information retrieval performance measures including precision, recall and F-value. In this paper, the application domain is affective blog articles, and their experimental results will be adopted in the system to facilitate the classification. In general, the blog articles can be roughly divided into two genres, informative and affective (Ni et al., 2007). Informative articles are about some researches, technologies or different kinds of informative news, while affective articles are about personal emo-
tions or feelings. In general, when people are writing informative articles, it is necessary for them to possess background knowledge about domains. For example, if a user wants to write an informative blog article, for instance chemistry, it is reasonable for him/ her to have background knowledge on chemistry. Nowadays, it is still infeasible to apply natural language understanding technique to generate a comprehensive article without external resources about domain knowledge. In this paper, we focus on the issues of the generation of affective articles. Ni et al. (2007) employed SVM to classify blog articles into informative and affective categories. Their experiments yielded 20 most frequent affective terms and 20 most frequent informative terms as shown in Fig. 2. These frequent affective and informative terms are employed to filter out the content with too much informative content. We employ term frequency to obtain the informative and affective scores of an article. In affective score computation, the total frequencies of 20 affective terms can be obtained from the corpus, denoted as Freqsum. Meanwhile, the frequency of each affective term Ti, denoted as Freqi, can be obtained as well. A normalization process is employed to obtain each term’s affective score, which is Freqi/Freqsum. Similarly, the same approach can be applied to informative terms to obtain their informative scores and Fig. 2 shows their scores. Therefore, the affective score and informative score of an article can be obtained from the sum of affective term’s affective score and informative term’s informative score. For example, if an affective term Ti appears in the article K, then we add its affective score to K’s affective score. Each affective
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
4499
Fig. 2. Representative informative and affective terms.
term can contribute an affective score and the sum of these scores represents K’s affective score, denoted as Ak. Similarly, K’s informative score can be obtained using the same approach, and assumed it is Ik. Finally, K’s score can be obtained using the following equation:
Scorek ¼ Ak Ik
ð1Þ
When the above process is completed, each article in the corpus has a score. Only the articles whose scores exceed the average score will be kept and these articles will become content material corpus. 3.2. Concept expansion In our system design, the user is asked to provide a concept or topic for the blog. This concept will then provide the system a direction to select appropriate content templates. The concept expansion module will use HowNet to generate such semantic information as synonyms of the concept. For English language, many researchers employed WordNet (Miller, 1995) to extract synonyms of terms based on synsets to infer whether two terms possessing the same meaning. Meanwhile, the machine can also use WordNet’s tree structure to determine the similarity distance between two terms. In this paper, the application domain is Chinese, so HowNet (Dong & Dong, 2003) is used for semantic information extraction. HowNet is not only a dictionary or thesaurus, but also an on-line common-sense knowledge base unveiling inter-conceptual relations and inter-attribute relations of concepts. Basically, HowNet structure is similar to ontology structure. In HowNet, Concepts are described by Knowledge Description Language (KDL). The basic element of KDL is called sememe which is the basic unit to describe a concept. In HowNet, a concept is defined by a DEF expression with nesting grammar. For example, Fig. 3 shows that the ‘‘doctor’’ concept in HowNet is described by human, own, status, education, HighRank, and most sememes, where human is the main sememe and the other sememes can be regarded as modifiers of main sememe. In general, if two terms T1 and T2 share the same DEF, T1 and T2 are synonyms. However, this approach can only identify the terms with the same meanings. One of the drawbacks is that not all the concepts have synonyms. For the concepts without synonyms, it is not easy for the system to capture the semantic information of the concept. Dai, Liu, Xia, and Wu (2008) regarded the main sememe as being the most important and the other modifying sememes are
Fig. 3. Concept graph of doctor in HowNet.
not distinguished regardless of their position. In this paper, we propose a layer-based weighting mechanism to determine the similarity between two concepts. If we regard each sememe as one layer, we found that the number of layers for most concepts in HowNet (about 77%) is less than three. The main sememe is located at the first position and it is the most important one. Similarly, the second and third layers can be obtained from the positions of sememes. The weighting mechanism is based on the layers shared by two concepts. For any concept C1, if another concept C2 has the first three layers in common with C1, C2 will be assigned three points in C1’s concept expansion. If they only have the first two layers in common, C2 will be assigned two points in C1’s concept expansion. Finally, if only the first layer is shared by C1 and C2, C2 will be assigned one point in C1’s concept expansion. When the above process is completed, the system can expand a single concept into several related concepts with weighting information. For example, Fig. 4 shows a layer-based concept expansion example. In Fig. 4, the first two concepts both represent ‘‘doctor’’, and they share the same sememes, so the second concept is given three points in the first concept’s expansion result. In Fig. 4, the third concept means court physician, and it only shares the sememe in the first layer with doctor concept, so it is given one point in the doctor concept’s expansion result. The concept expansion proposed above can expand a single concept into several related concepts and each extended concept can have a score to represent the degree of similarity with the original concept. Hence, the system can obtain more semantic information from the concept expansion result.
4500
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
Fig. 4. Layer-based concept expansion example.
3.3. Keyword extraction and expansion In this paper, text generation model is keyword based and the user determines a template from the keyword lists each of which represents an article. Practically, it is infeasible to list many keywords for users to determine the content. However, it will be difficult for the system to generate content if the number of keywords is few. Hence, the keyword list plays two important roles in the system design. First, the keywords should be able to provide the insight of the final content. Second, the keywords in the keyword list should provide sufficient information for the system to generate a complete article. Hence, we employ five representative keywords to present the key concepts of the content template to the user and employ hidden keywords to provide content generation backbone to the system. Both representative and hidden keywords are extracted from the corpus. The keyword extraction and expansion flow are shown in Fig. 5, which includes keyword extraction, modified strict phrase likelihood ratio (SPLR) scoring process, and keyword expansion. Practically, nouns and verbs can provide more semantic information than other POS tags, so the system regards nouns and verbs as candidate keywords in keyword extraction step. Basically, an
article contains many nouns and verbs, so it is infeasible to list all the keywords to the users. Meanwhile, the keywords presented to the users should provide insight of the generated content templates to users, so users can determine the final content template from these keywords. In other words, these keywords should not only capture the main concept of the generated article, but also differentiate two different articles. Hence, we employ a modified strict phrase likelihood ratio (SPLR) computational model to obtain the keywords which can highlight the content. Originally, strict phrase likelihood ratio (SPLR) (Chang & Lee, 2003) approach is used to extract Chinese unknown words. In practice, these unknown words could be used to highlight the articles due to their rareness. The system can employ the computation model as shown in Eq. (2) to obtain SPLR score of each candidate keyword. The higher ones will be selected as the core keywords.
SPLR ¼
tf ðKW i Þ ; Maxðtf ðKW i LÞ; tf ðKW i RÞÞ
where KW i len > 1
ð2Þ
In Eq. (2), tf(KWi), tf(KWi_L) and tf(KWi_R) represent term frequency of the keyword, term frequency of left hand side of the keyword and term frequency of right hand side of the keyword respectively. Meanwhile, the basic meaningful unit in Chinese is
Term Extraction
Term List
Modified SPLR Scoring
Expanded Keyword List
Keyword Expansion
Core Keyword List
Corpus
Fig. 5. Keyword extraction and expansion flow.
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
4501
list, the system can compare the keywords in the list with these related concepts and their weighting information to obtain a similarity score. Then, all the keyword lists can be ranked according to the similarity scores, and the top ones can be presented to the user. 3.4. Template article generation
Fig. 6. SPLR scoring example.
In an article, keywords are like the backbone of the article and the text segments between keywords make up the content of the text. When the expanded keywords are extracted from the above process, appropriate text segments between two keywords should be determined as well. Eq. (3) shows how to select candidate text segments between two keywords. jPre KW \ Wordði; jÞ \ Next KWj > 0 CandidateðuÞ C u 2 Unitði; jÞj Wordði; jÞ len 6 Threshold
ð3Þ Fig. 7. Keyword expansion model.
Fig. 8. Keyword expansion model with overlapped keywords.
a phrase rather than a single word, so only the terms with word length larger than one will be taken into account. As shown in Fig. 6, the SPLR score of the keyword ‘‘the president of student association’’ could be obtained from the frequency information. In modified SPLR score computation model, a normalization step is required to normalize the scores so that the final scores will range from 0 to 1. The keywords with the highest five scores will become core keywords. If the number of keywords is less than five, candidate keywords are selected as the core keywords. The benefit of core keywords is that users could have an overview about the final content from the core keywords. As described above, nouns and verbs are selected as candidate keyword list and modified SPLR computational model are used to choose core keywords of the system. In practice, it is not enough to generate an article using five core keywords, since five keywords contain inadequate information to generate an article. Therefore, we propose a keyword expansion process to expand the core keywords and Fig. 7 shows the keyword expansion model. In Fig. 7, W1, W2, . . . , W10 represent the candidate keywords, while W3 and W8 represent core keywords. The expansion model will expand the core keywords to include other keywords as text generation keywords. Practically, if all the candidate keywords are included into the text generation keyword set, the variation of final text will be limited. On the other hand, if the number of keywords is few, it is not easy to find appropriate text segments to fill in the gap between the keywords. In the expansion model, two previous keywords and two next keywords of the core keywords are selected as the generation keywords. Thus, as shown in Fig. 7, W1, W2, W4 and W5 will be expanded from W3. Meanwhile, W6, W7, W9 and W10 will be expanded from W8. If expanded keywords overlap, intersection approach will be adopted and only one overlapped keyword will be selected. Hence, as shown in Fig. 8, keyword W5 is a overlapped keyword and the final generation keywords will be W1, W2, W3, W4, W5, W6, W7, W8 and W9. Based on the core keyword generation model and keyword expansion model, the users could determine the generating text from core keywords and the system could generate text from expanded keywords. As mentioned above, the concept provided by the user will be expanded to related concepts with weighting information. For each keyword
where i is the index of the article, j is the position index of article i, Pre_KW represents the previous keyword, Next_KW represents the next keyword, Unit(i, j) represents the text segments that appear at the jth position of the ith article, Word(i, j) represents the text segments that have appeared between Pre_KW and Next_KW in the corpus and Word(i, j)_len represents the length of Word(i, j). In Eq. (3), the text segments that appear between two keywords and their length are less than threshold value will be selected as candidate text segments. Based on the above process, candidate text segment list could be obtained and these candidate text segments could be mixed with the keywords to generate different texts. Similarly, the content templates can be ranked based on expanding concepts with weighting information. The main goal of the system is to construct an intelligent computer assisted blog writing system to provide a draft version of text for the users. In addition, the users could learn how to improve their writing skills by learning from examples. Fig. 9 shows the text generated by the system and the rough English translation of the content is presented in Appendices. Practically, the content generated by the system provides a reference template for the users and the users could modify the content to meet their requirements. 3.5. Phrase ranking and substitution When the above process is completed, the system presents the user a template article based on the user concept. Obviously, the generated article only provides a template, and the user still need to modify the article to meet his/her requirement. Hence, the system allows the user to edit the generated article. In addition to editing functionality, the system employs Google to provide replaceable content when the user is editing the article. We propose a replaceable content retrieval mechanism to obtain example texts from the Web. Currently, Google provides a snippet, which is a short text, under each search result, and these snippets usually contain query strings. Users can determine whether the result links are useful or not from the snippets. Therefore, if the system sends appropriate query strings to Google, the snippets returned from Google can provide candidate example texts. For example, suppose that a template article contains two keywords KW1 and KW2 and the content between these two keywords is S12. When the users want to change S12, the system will employ KW1 and KW2 with wildcard symbols to compose a query string to obtain snippets. Then, the system analyzes the snippets to obtain replaceable content. In other words, the system makes use of the Google search engine’s outstanding search capability to make the Web become system corpus.
4502
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
shown in the following sections. Meanwhile, ten people were invited to experience the system and several evaluation criteria are listed in the questionnaire to evaluate the system. 4.1. Experimental results
Fig. 9. Template article generated based on keywords.
Most search engines provide wildcard symbols and people can use wildcards to represent any character or range of characters. Google treats the wildcard ‘‘⁄’’, an asterisk character, as a placeholder for a word or more than one word. For example, a query string ‘‘Internet ⁄ my life’’ tells Google to find pages containing a phrase that starts with ‘‘Internet’’ followed by one or more words, followed by ‘‘my life’’. Phrases obtained from Google search result and fit the constraint include: ‘‘Internet in my life,’’ ‘‘Internet changed my life,’’, ‘‘Internet is my life’’, etc. Since Google has indexed a huge amount of pages, the system can obtain several candidate text segments which conform to the constraints. In addition to replaceable content retrieval, we propose a content ranking mechanism to provide a ranking score for replaceable content. We employ the original content as a base to rank the content obtained from Google search. The ranking mechanism includes two ranking scores. The first one is based on the sharing terms between two text segments and another one is based on the similarity of POS tagging between two text segments. In information theory and computer science, the Levenshtein distance between two strings of characters is the number of operations required to transform one of them into the other. For example, if the source string is ‘‘GUMBO’’ and the target string is ‘‘GAMBOL’’, the distance will be two. The transformation from ‘‘GUMBO’’ to ‘‘GAMBOL’’ should consist of a substitution operation (‘‘U’’ for ‘‘A’’) and an addition operation for ‘‘L’’. We apply Levenshtein distance to the distance measurement between two kinds of text segments, which are term sequence and POS tagging sequence. Practically, the term similarity is similar to semantic distance; while the POS tagging similarity is similar to structure distance. The sum of these two scores is used to rank the content obtained from Google. 4. Experiment and discussion In this paper, we designed and implemented an intelligent computer assisted blog writing system. The experimental results are
As described above, the concept expansion module is employed to expand the single concept provided by the user. One of the advantages of this approach is that the system can capture more semantic information and the system can provide more accurate content templates. For example, if the user provides a single concept, which is ‘‘train’’, Fig. 10 shows the keyword lists and content templates, where the system does not employ concept expansion. The content templates in Fig. 10 are generated from the keyword lists with red color. The left content template describes a story where the user and his/her friend went for a bicycle ride and they locked their bicycles in the train station. The right content template describes a story where the user went to eat delicious food when typhoon hit Taiwan. The results show that the relationship between the concept ‘‘train’’ and the content templates is tiny. Meanwhile, Fig. 11 shows the results when the system employs concept expansion mechanism. The left content template describes a scenario where the user went for a bicycle ride and he/she saw many tourists taking pictures when the train went across the road. The right content template describes a scenario where the user enjoyed the outside scene of the train on his/her way home. The results show that the content templates contain more information about train. In content replacement, we employ Levenshtein distance measurement to obtain term and POS tagging similarities between two text segments. Fig. 12 shows the original text segment (the text segment with red color is the source content) and the replaceable content lists obtained from Google. The left list in the Figure is the content list obtained from Google with Google’s ranking order; while the right one is the list using the content ranking approach proposed in this paper. The first two text segments in the left are used to represent category information. Apparently, these two text segments are inappropriate in an article, and their rankings drop a lot when the system employs the content ranking mechanism. Moreover, the rankings of fifth and the seventh text segments are advanced to higher position. If we take into account the context of the original text segment, these two text segments are better than the other ones. 4.2. User evaluation and discussion In user evaluation, user satisfaction evaluation is used. Liu et al. (2011) and Huang et al. (2009) employed similar measurement to evaluate their systems. The system evaluation includes readability, relevance, concept and interface evaluations. Readability item is used to measure the readability of the content generated by the system, relevance item is used to measure the relevance of replaceable content provided by the system, concept item is used to measure whether the content templates generated are related to the concept provided by the users and interface item is used to measure system user interface. Ten people, including 9 males and 1 female and their ages range is from 24 to 40, were invited to experience the system. These people are all interested in blog writing and they are asked to give a score for each item. The score ranges from 1 to 5 and the highest score is 5. Table 1 shows the result which includes average score and the proportion which is higher than 3. Each user was asked to use the system to generate 20 blog articles and then started to use the templates to compose articles. Then he/she evaluated the readability, relevance, interface, and concept items of the articles. The average score of the experiment is higher than 3. Meanwhile, most people thought that the
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
Fig. 10. Keyword list and content templates without concept expansion.
Fig. 11. Keyword list and content templates using concept expansion.
Fig. 12. Comparison of content ranking.
4503
4504
C.-L. Liu et al. / Expert Systems with Applications 39 (2012) 4496–4504
Table 1 Evaluation result.
Average Proportion (P3)
Readability
Relevance
Interface
Concept
3.445 86.5%
4.07 92.5%
4.571 99.0%
3.83 89.0%
system could assist them to come up with new ideas from example text segments. As for the system improvement, most people think that the fluency of the content template is not good enough, so content fluency will be our future work. 5. Conclusion An intelligent computer assisted blog writing system, which includes concept expansion, text generation and replaceable content suggestion, is presented in this paper. In this paper, we propose a layer-based concept expansion approach, which is based on HowNet, to extend a single concept into several related concepts with weighting information. Then, the related concepts with weighting information can facilitate the system to rank and determine content templates each of which consists of a keyword list and the text segments between keywords. To generate keywords in the content template, we propose keyword extraction and expansion module to retrieve representative keywords as well as to expand these keywords. To generate text segments in the content template, we employ statistical approach to obtain the text segments between keywords from the corpus. The benefit of this approach is that the template could provide a framework and reduce the physical effort spent on writing so that people can pay attention to organization and content. Moreover, the system allows the users to edit the content template to meet their requirements. When they are editing the content, the system can provide replaceable contents for users’ references. We employ Google to retrieve contents from the Web and we provide a ranking mechanism, which is based on the similarities of terms and POS tagging, to rank the contents obtained from Google. The experiments show that our system can assist users to compose a blog article from a single concept effectively. Acknowledgment This work was supported in part by the National Science Council under the Grants NSC-99-2221-E-009-150 and NSC-099-2811E-009-041. Appendix A Rough translation of the love letter in Fig. 9. Today I visited Ju’s husband’s house with friends. Ju’s husband family took care of us sincerely. When we got there, Ju prepared
fruit and sea food for us. The sea food was very fresh and we finished the sea food immediately. Therefore, Ju had to prepare for another plate of sea food and this photo was about that plate of sea food. Because it was so delicious, we asked Ju to cook next time. After we finished the meal, we went to talk over coffee and enjoyed the view of sea. References Bustamante, F. R., & Leon, F. S. (1996). Gramcheck: A grammar and style checker. In Proceedings of the international conference on computational linguistics (COLING96) (pp. 175–181). Chang, T.-H., & Lee, C.-H. (2003). Automatic chinese unknown word extraction using small-corpus-based method. In Proceedings of international conference on natural language processing and knowledge engineering (pp. 459– 464). Dai, L., Liu, B., Xia, Y., & Wu, S. (2008). Measuring semantic similarity between words using hownet. In ICCSIT’08: Proceedings of the 2008 international conference on computer science and information technology (pp. 601–605). Washington, DC, USA: IEEE Computer Society. Dong, Z., & Dong, Q. (2003). Hownet – a hybrid language and knowledge resource. In Proceedings of the International Conference on Natural Language Processing and Knowledge Engineering, 2003 (pp. 820–824). Esuli, A., & Sebastiani, F. (2005). Determining the semantic orientation of terms through gloss classification. In Proceedings of the 14th ACM international conference on information and knowledge management (pp. 617–624). Genthial, D., & Courtin, J. (1992). From detection/correction to computer aided writing. In Proceedings of the 14th conference on computational linguistics. Association for computational linguistics (pp. 1013–1018). Huang, T.-C., Cheng, S.-C., & Huang, Y.-M. (2009). A blog article recommendation generating mechanism using an sbacpso algorithm. Expert Systems with Applications, 36(7), 10388–10396. Komatsu, H., Takabayashi, S., & Masui, T. (2005). Corpus-based predictive text input. In Proceedings of the 2005 international conference on active media technology, 2005 (AMT 2005) (pp. 75–80). Kukich, K. (1992). Technique for automatically correcting words in text. ACM Computing Surveys, 24, 377–439. Leacock, C., Gamon, M., & Brockett, C. (2009). User input and interactions on microsoft research esl assistant. In EdAppsNLP’09: Proceedings of the fourth workshop on innovative use of NLP for building educational applications. Association for computational linguistics, Morristown, NJ, USA (pp. 73–81). Liu, T., Zhou, M., Gao, J., Xun, E., & Huang, C. (2000). Pens: A machine-aided english writing system for chinese users. In ACL’00: Proceedings of the 38th annual meeting on association for computational linguistics. Association for computational linguistics, Morristown, NJ, USA (pp. 529–536). Liu, C.-L., Lee, C.-H., Yu, S.-H., & Chen, C.-W. (2011). Computer assisted writing system. Expert Systems with Applications, 38(1), 804–811. Miller, G. A. (1995). Wordnet: A lexical database for english. Communications of the ACM, 38(11), 39–41. Mullen, T., & Collier, N. (2004). Sentiment analysis using support vector machines with diverse information sources. In Proceedings of EMNLP-2004 (pp. 412–418). Ni, X., Xue, G.-R., Ling, X., Yu, Y., & Yang, Q. (2007). Exploring in the weblog space by detecting informative and affective articles. In WWW’07: Proceedings of the 16th international conference on world wide web (pp. 281–290). ACM. Paggio, P. (2000). Spelling and grammar correction for danish in scarrie. In Proceedings of the sixth conference on applied natural language processing (pp. 255–261). Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up?: Sentiment classification using machine learning techniques. In EMNLP’02: Proceedings of the ACL-02 conference on empirical methods in natural language processing (pp. 79–86). Turney, P. D. (2002). Thumbs up or thumbs down?: Semantic orientation applied to unsupervised classification of reviews. In ACL’02: Proceedings of the 40th annual meeting on association for computational linguistics (pp. 417–424). Uchimoto, K., Isahara, H., & Sekine, S. (2002). Text generation from keywords. In Proceedings of the 19th international conference on computational linguistics (pp. 1–7).