{"word_count": 1216, "character_count": 12319}
BILBO : Automatic reference labeling(C) Copyright 2013 by Young-Min Kim and Jade Tavernier.written by Young-Min Kim, modified by Jade Tavernier.BILBO is an open source software for automatic annotation of bibliographic reference.It provides the segmentation and tagging of input string. It is principally based onConditional Random Fields (CRFs), machine learning technique to segment and labelsequence data. As external softwares, Wapiti is used for CRF learning and inferenceand SVMlight is used for sequence classification. BILBO is licensed under GPL version 2.==============================================External tool installation==============================================------------------------Wapiti------------------------http://wapiti.limsi.fr/------------------------SVM light------------------------http://svmlight.joachims.org/------------------------lxml for BeautifulSoup------------------------http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser------------------------SOAPpy------------------------$ cd dependencies/SOAP$ sudo python setup.py install------------------------fpconst------------------------$ cd dependencies/fpconst$ sudo python setup.py install==============================================Execute==============================================command:$ cd bilbo$ python src/bilbo/Main.py (help full ver.)$ python src/bilbo/Main.py -h (help simple ver.)&#160;==============================================Structure of directory==============================================------------------------src directory------------------------BILBO source code------------------------dependencies directory------------------------External tools used by BILBO.For example :  Wapiti, SVMlight------------------------doc directory------------------------Document files to explain the architecture, operation and conception of BILBO------------------------KB (Knowledge base) directory------------------------config directory :Configuration file for BILBOdata directory :Training corpusvalidation directory :xml validation files------------------------model directory------------------------Model created and used by BILBO------------------------result directory------------------------Result files&#160;***********************For more information***********************==============================================Description==============================================&#160;========================Configuration files========================------------------------externalList directory------------------------External proper noun lists used to improve proper noun tagging. Any proper noun listcan be used, for example, common name list in the world or world city list.&#160;------------------------balise.txt------------------------This file contains the tag replacement information. Several tags in original trainingdata are too detailed, so they are replaced with simple tags.e.g. 'meeting' tag is replaced with 'booktitle'FILE FORMAT(tag name to be replaced) (space) (replacing tag name)...------------------------lexique.txt------------------------This file contains lexical features to be added to specific words. It is used by Ruleclass. There are two types of lexical features : [including] and [matching][including] means check if the corresponding word is INCLUDED in the input token[matching] means check if the corresponding word is exactly matching the input tokene.g. (January), string should de verified as [including] featureFILE FORMAT[including]# (rule name) (space) (feature) (space) (feature) ….wordword...# (rule name) (space) (feature) (space) (feature) ….wordword...[matching]# (rule name) (space) (feature) (space) (feature) ….wordword...# (rule name) (space) (feature) (space) (feature) ….wordword...EXAMPLE :[including]# editor nonimpcap posseditorededsed.eds.-&#62; if a word 'ed' is a input token, 'nonimpact' and 'posseditor' features are added.------------------------features.txt------------------------This file contains basic information about feature names to be assigned and some rulesabout tag names to be excluded or included. It is used by Extract class.# features : valid features during data extraction# nonLabels : invalid tags as label during data extraction. This is to eliminate somemeaningless tags in CRF learning. (*)importance 0:totally invalid tag, 1:can beattached to token when it does not have any tags attached.# bookindicator : indicator words for book, journal, or etc. that includes the referredarticle.Remark:Other rules like 'bookindicator' can be added using same format with 'bookindicator'.FILE FORMAT# featuresfeaturefeature...# nonLabelstag &#60;0 or 1&#62;tag &#60;0 or 1&#62;...# bookindicatorindans...------------------------others.txt------------------------This file contains other configurations about TEI extraction (username of crossref site)and TEI output format. As Bilbo label reference with its own tags, a change rule fromown tags to TEI is necessary. It is used in the functions of identifier.py called fromFile::buildReferences. Lines starting with # are ignored.[crossref] is for crossref user name to call crossref API for TEI extraction[tei] is for replacement of tags for the rewrite of result in TEI format(old tag) = (new tag)if new tag is NONE, it means that the corresponding old tag will be ignored in TEI formatFILE FORMAT[crossref]usrname = (crossref user email)[tei]nonbibl = NONEnolabel = NONEbookindicator = NONEw = NONE#surname =#forename =#publisher =#abbr#datetitle_m = title level="m"title_a = title level="a"title_j = title level="j"title_t = title level="a"title_u = title level="u"title_s = title level="s"biblscope_pp = biblScope unit="pp"biblscope_i = biblScope unit="issue"biblscope_v = biblScope unit="vol"biblscope_pa = biblScope unit="part"biblscope_c = biblScope unit="chap"biblscope = NONE…&#160;------------------------pattern_ref------------------------Wapiti pattern configuration for feature extraction&#160;========================Class========================------------------------Bilbo------------------------A machine Bilbo that trains a CRF (and a SVM) model and automatically labels new references.Created in Main.pyDescription in doc/documentation/Bilbo.html------------------------CRF------------------------CRF object is created in a Bilbo objectCRF model learning and testDescription in doc/documentation/format/CRF.html------------------------SVM------------------------SVM object is created in a Bilbo objectSVM model learning and testDescription in doc/documentation/format/SVM.html------------------------Corpus------------------------A corpus containing a set of training (or test) references.Description in doc/documentation/reference/Corpus.html------------------------File------------------------A file class containing all references in a fileDescription in doc/documentation/reference/File.html------------------------Extract------------------------A class to extract training and test data according to a set of predefined criteriaBase class of Extract_crf and Extract_svmDescription in doc/documentation/format/Extract.html------------------------Extract_crf------------------------A class to extract training and test data for CRFSub class of ExtractDescription in doc/documentation/format/Extract_crf.html------------------------Extract_svm------------------------A class to extract training and test data for SVMSub class of ExtractDescription in doc/documentation/format/Extract_svm.html------------------------Clean------------------------A class that tokenizes xml input data. Navigates the xml tree and extracts tokens, features and labels.It concerns the first step of tokenization such that words are separated by whitespace but not by punctuationmarks. A clean object is created in a File object ("extract" method).Description in doc/documentation/format/Clean.html------------------------CleanCorpus1------------------------A class that tokenizes xml input data for corpus 1 (references).Sub class of CleanDescription in doc/documentation/format/CleanCorpus1.html------------------------CleanCorpus2------------------------A class that tokenizes xml input data for corpus 2 (notes).Sub class of CleanDescription in doc/documentation/format/CleanCorpus2.html------------------------Rule------------------------A class that reorganizes tokens according to the predefined rules. ("lexique.txt" is loaded)Especially the punctuation marks are separated and new Word objects are created.Features about initial expression, capitalized token etc. are verified and attached.Description in doc/documentation/format/Rule.html------------------------ListReferences------------------------A class containing a list of reference objects and corpus type information.Description in doc/documentation/format/ListReferences.html------------------------Reference------------------------A class corresponding to a reference. It contains, word objects.Reference object is first created in CleanCorpus1 and CleanCorpus2.Description in doc/documentation/format/Reference.html------------------------Word------------------------A class corresponding to a word in a reference. It contains word name, features, tags etc.Word object is first created in CleanCorpus1 and CleanCorpus2.Description in doc/documentation/format/Word.html------------------------Feature and Balise------------------------These classes contain feature name or tag name.=========------------------------Name------------------------This class verifies if a given word corresponds to a surname or a forename in a name list in the "externalList" directory.Attributes :Methods :####searchName : this method checks if the entered word is found in the name list and if yes, it adds a feature : SURNAMELIST or FORENAMELIST------------------------Place------------------------This class verifies if a given word corresponds to a place in a list in the "externalList" directory.Attributes :Methods :####searchPlace : this method checks if the entered word is found in the place list and if yes, it adds a feature : PLACELIST------------------------ProperList------------------------This class verifies if a given word corresponds to a proper noun in a list in the "externalList" directory.Attributes :Methods :####searchPlace : this method checks if the entered word is found in the place (for the moment) list and if yes, it adds a feature : PLACELIST (for the moment)&#160;========================OTHERS========================------------------------Main.py------------------------Main creates Bilbo object to train or to label reference------------------------utils.py------------------------utils.py provides APIs to process simple string for labeling.It also contains default option setting function, which is called in main.There are two functions that we can execute :- simpleLabeling : for labeling with a simple CRF model (having less number of labels)- detailLabeling : for labeling with a detailed CRF model (having labels for TEI)You can test simply this function at the directory src.e.g. Labeling a reference$cd src$python&#62;&#62;&#62; from bilbo.utils import *&#62;&#62;&#62; detailLabeling("Y.-M. KIM et al., An Extension of PLSA for Document Clustering, In Proceedings of ACM 17th Conference on Information and Knowledge Management, 2008.")ResultFirst author : KIM Start of title : An Extension of PLSA for Document Clustering'&#60;listBibl&#62;\n&#60;bibl&#62; &#60;author&#62;&#60;forename&#62;Y.-M.&#60;/forename&#62; &#60;surname&#62;KIM&#60;/surname&#62;&#60;/author&#62; et al., &#60;title level="a"&#62;An Extension of PLSA for Document Clustering&#60;/title&#62;, &#60;meeting&#62;In Proceedings of ACM 17th Conference on Information and Knowledge Management&#60;/meeting&#62;, &#60;date&#62;2008&#60;/date&#62;. &#60;/bibl&#62;\n&#60;/listBibl&#62;\n'The result is also saved in a file tmp.xml in the current directory.We can also test it in Main.py (an example in comments in Main.py)To label a note, we just add "note" as second parameter.e.g.&#62;&#62;&#62; detailLabeling("For more information see, Y.-M. KIM et al., An Extension of PLSA for Document Clustering, In Proceedings of ACM 17th Conference on Information and Knowledge Management, 2008.", "note")&#160;========================Bilbo Class diagram========================&#160;&#160;&#160;