Tag: technical

Basic conceptnet2.1 query

#!/usr/local/bin/python

import sys, os
reload(sys)

os.environ['CONCEPTNET'] = 'libs/conceptnet2.1'
sys.path.append('libs/')
sys.setdefaultencoding("latin1")

import ConceptNetDB
processor = ConceptNetDB.ConceptNetDB(None,'ConceptNet.ini')

phrase = sys.argv[1]
print  phrase

text = processor.nltools.guess_mood(phrase)
print text

$ python _test-mood.py
PC_524727138 <- for/IN good/JJ man/NN PC_902158629 <- of/IN party/NN PC_725835205 <- for/IN all/DT good/JJ men/NNS PC_502591577 <- of/IN the/DT party/NN [('disgusted', 0.4862797314730406), ('angry', 0.44568548377190498), ('sad', 0.25683556430435933), ('fearful', 0.2219426823049894), ('happy', 0.11441016214900629), ('surprised', 0.0)]

Software combination

Current verison

Linux waikato 2.6.32-5-686 #1 SMP Wed Aug 25 14:28:12 UTC 2010 i686 GNU/Linux
Python 2.6.6 (r266:84292, Aug 27 2010, 20:59:12)
feedparser
pytz
paramiko
pyserial
pytwitter
PyRSS2Gen
pysqlite2
AsciiDammit
MontyLingua
xmlrpclib

First version

Python 2.3.5 (#1, Aug 19 2006, 21:31:42)

feedburner 4.1

MontyLingua 2.1

upgraded php & sqlite with macports
$ php -v
PHP 5.2.4 (cli) (built: Oct 13 2007 11:43:55)
$ sqlite3 -version
3.5.1
$ uname -a
Darwin whale.local 8.10.1 Darwin Kernel Version 8.10.1: Wed May 23 16:33:00 PDT 2007; root:xnu-792.22.5~1/RELEASE_I386 i386 i386

Current source feeds.

cursor.execute(‘INSERT INTO “sources” VALUES(2, “ABC News: Breaking Stories”, “http://abc.net.au/news/syndicate/breakingrss.xml”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(3, “ABC News: World”, “http://abc.net.au/news/syndicate/worldrss.xml”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(4, “ABC News: New South Wales”, “http://www.abc.net.au/xmlcontent/indexes/nsw/NSW_rss_index.xml”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(5, “ABC News: Top Stories”, “http://abc.net.au/news/syndicate/topstoriesrss.xml”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(6, “Crikey RSS”, “http://www.crikey.com.au/rss.xml”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(7, ” the Mail online | World news”, “http://feeds.feedburner.com/dailymail/worldnews”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(8, “NYT > Middle East”, “http://www.nytimes.com/services/xml/rss/nyt/MiddleEast.xml”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(9, “NYT > International”, “http://www.nytimes.com/services/xml/rss/nyt/International.xml”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(10, “New Zealand Herald – World”, “http://syndication.apn.co.nz/rss/nzhrsscid_000000002.xml”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(11, “New Zealand Herald – National”, “http://syndication.apn.co.nz/rss/nzhrsscid_000000001.xml”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(12, “The Sydney Morning Herald World Headlines”, “http://feeds.smh.com.au/rssheadlines/world.xml”, 1)’)
cursor.execute(‘INSERT INTO “sources” VALUES(13, “The Sydney Morning Herald National Headlines”, “http://feeds.smh.com.au/rssheadlines/national.xml”, 1)’)

Current database structure

cursor.execute(‘CREATE TABLE articles (id INTEGER PRIMARY KEY, title VARCHAR(50) UNIQUE, content VARCHAR(1000), date DATE, source_id INTEGER, phrase_id VARCHAR(100))’)
cursor.execute(‘CREATE TABLE sources (id INTEGER PRIMARY KEY, title VARCHAR(50), url VARCHAR(100), weight INTEGER)’)
cursor.execute(‘CREATE TABLE phrases (id INTEGER PRIMARY KEY, phrase TEXT UNIQUE, pos VARCHAR(5), rating varchar(100), count INTEGER, words varchar(50))’)
cursor.execute(‘CREATE TABLE score (date DATE PRIMARY KEY, score INTEGER)’)
cursor.execute(‘CREATE TABLE words (id INTEGER PRIMARY KEY, wordtagged TEXT UNIQUE, word TEXT, pos VARCHAR(5), count INTEGER, q1 TEXT, q2 TEXT, q3 TEXT, q4 TEXT, q5 TEXT, q6 TEXT, q7 TEXT, q8 TEXT)’)

Data structure

Table of words
Word/Phrase
Index ( smallint )
Usage/definition
Examples

Table of results
Word/Phrase index ( smallint )
Number of results ( mediumint )
Aggregate score per affect index ( DOUBLE(9,2) UNSIGNED ) up to 1 billion to 2 decimal places

Table of affects
Affect ( varchar )
Affect id ( tinyint )

Table of voters
IP address who have voted
array of Word/Phrase index voted for ( smallint )