9/19/17 11:00 am to 3:30 pm - Added files from Dropbox to Github. - Cleaned mismatched tags from CordialCorpus.xml - Created process_corpus.py to raw_output.txt from CordialCorpus_Cleaned.xml - Downloaded FreeLing to local computer - Input raw text into FreeLing to get freeling_output.txt - Created process_freeling.py to count frequencies of parts of speech and output to separate text files - Wrote comments and development log 9/26/17 4:30 pm to 6:30 pm - Create process_freeling_breakdown.py to break down frequencies by subcategories based on FreeLing tags - Renamed process_freeling.py to process_freeling_frequency.py 9/27/17 5:00 pm to 6:00 pm - Training 9/28/17 2:00 pm to 4:30 pm - Training 9/29/17 7:00 pm to 10:00 pm - Created parse_breakdown.py to take output of process_freeling_breakdown.py and parse the tags associated with each part of speech - This resulted in comma-separated files (*.csv files) that can be easily input into Excel or other database programs. - Used RapidMiner (data mining software--free for educational use) to create bar chart of verb tense frequencies (verb_breakdown.png) 10/3/17 1:30 pm to 3:30 pm - Created "Research Narrative.docx" to document the thought process behind my choices for future use - Created Excel file Analysis.xlsx for breakdown of verbs by mood, tense, person, and number using RapidMiner. 10/4/17 10:30 am to 11:30 am - Training 10/4/17 8:30 pm to 11:30 pm - Created process_corpus_with_metadata.py to handle dividing the corpus into texts and keeping the header information 10/5/17 10:30 am to 11:30 am - Added to Analysis.xlsx analyses of the verbs by category and by key words, as they appear in the headers. - Archived files that didn't have metadata attached. They are superceded by the metadata-attached data files. 10/10/17 2:30 pm to 5:30 pm - Attempted tagging all words in corpus *in situ* with process_corpus_with_metadata_pos_tagged.py - The main issue appears to be that FreeLing output splits contractions and attached pronouns into separate words, making reforming the corpus tricky 10/13/17 9:00 am to 4:00 pm - Further developed process_corpus_with_metadata_pos_tagged.py to deal with edge cases caused by idiosyncrasies in FreeLing output and corpus itself. - Documented code thus far. 10/15/17 3:00 pm to 7:00 pm - Created a version of the previous script that abandoned the annotations that caused all the edge cases, in order to move forward with getting a reada1ble corpus with words tagged in place. Future work can be to find a way to include those annotations. The tagged corpus is in CorpusCordial_Tagged.xml. - Created process_tagged_corpus_with_metadata_xml_to_html.py in order to turn the xml into a readable html document that can be used with javascript to highlight tagged words on command. - Created process_tagged_corpus_with_metadata_html_format.py to make the html converted from xml into a nicer format, including adding titles for each text and printing the metadata in list format. - These scripts result in a basic webpage with, thus far, some highlighting of verbs and nouns as desired. The webpage is stored as "CorpusCordial_Tagged_HTML_Formatted.html" in the folder corpus_html and can be viewed with any web browser. 10/18/17 6:00 pm to 9:30 pm - Added functionality to CorpusCordial_Tagged_HTML_Formatted.html to enable selective highlighting based on all of the options returned by the FreeLing analyzer using Javascript. The script.js file is in the js folder in the corpus_html folder. 10/19/17 11:30 am to 12:00 pm - Improved look/ease-of-use of web page using Bootstrap (a free CSS styling plugin from Twitter). 10/20/17 7:00 pm to 9:00 pm - Attempted to improve functionality of filtering on web page using a tree data structure, but results were unsatisfactory (inefficient w.r.t time), so discarded changes. 10/24/17 5:30 pm to 7:30 pm - Added frequency lists for filtering on webpage and documented code. Pushed new website version. 10/25/17 8:00 am to 9:00 am - Added wildcard functionality to filtering 10/31/17 12:30 pm to 1:00 pm - Added to Research Narrative 10/31/17 2:00 pm to 4:30 pm - Created page to breakdown frequencies by situation (availably via link from normal page) 11/10/17 7:00 pm to 11:00 pm - Found CORLEC spanish language oral corpus (788000+ words) from http://www.lllf.uam.es/ING/Corlec.html - Created script to concatenate multiple files into one large raw file for processing. - Created build_CORLEC_corpus.py to take corpus from raw state to html for the webpage 11/11/17 7:00 am to 9:30 am - Developed build_CORLEC_corpus.py and reconfigured javascript and html page (including C-Or-DiAL page) to accommodate multiple corpuses along with sorting by situation (in CORLEC's case, the closest analog to situation was the category of the text, but this can be adjusted) - Changed filenames to distiguish scripts and documents used for different corpuses - Moved C-Or-DiAL-specific items into its own folder and created new folder for CORLEC items - Changed name of "corpus_html" to "webpage" 11/11/17 1:30 pm to 2:30 pm - Searched for Mexican/Latin American oral corpuses (none found) - Added loading icon to webpage to alert user to delay on loading large CORLEC file 11/14/17 10:00 am to 11:00 am - Added more sorting/grouping categories to both corpuses 11/14/17 4:30 pm to 6:30 pm - Added basic Excel export functionality for frequency lists 12/1/17 5:30 pm tp 6:30 pm - Researched Subcorpus oral del CREA and Corpus del Espanol to see if they can be added to webpage. - Found webtools existing for these corpuses at http://web.frl.es/CREA/view/inicioExterno.view and https://www.corpusdelespanol.org/web-dial/, respectively. 12/05/17 6:00 pm to 9:00 pm - Downloaded PRESEEA corpus at http://preseea.linguas.net/Corpus.aspx by downloading each individual conversation and converting to usable text files - Created build_PRESEEA_corpus.py to turn into database-ready format 12/14/17 8:00 am - 4:00 pm - Worked with build_PRESEEA_corpus.py and IndexedDB-wrapper Dexie.js to create an in-browser database (since this corpus and the Corpus del Espaņol corpus are large, a more robust solution than previously created is needed to make the searching performant) 12/15/17 7:00 am - 12:00 pm - Created IndexedDB/Dexie NoSQL database of PRESEEA corpus using JSON data store that is ready for querying. 12/15/17 12:30 pm - 4:30 pm - Started setting up webpage for PRESEEA and linking with database via basic queries. 12/16/17 12:00 pm - 7:00 pm - Combined highlighting features of previous corpuses to PRESEEA (which required a different approach due to large size/database usage. - Added substrings as option instead of full text, since conversations are so long and queries could return thousands of pages. 12/18/17 8:00 am - 3:30 pm - Removed substrings (creating them was still too slow) and settled on buttons that link to the full text of each conversation, highlighted, as a compromise. - Added grouping ability to this corpus. - Set up frequency analysis and prepared for other corpuses to use same database to improve performance. 12/19/17 8:00 am - 3:30 pm - Fixed bug with limiting frequency list - Added fourth corpus (Corpus del Espanol sample, with approx 2 million words) - Combined all four current corpuses into one database and one webpage with some minor GUI tweaks.