lingtools.tokenizer_stanford module

class lingtools.tokenizer_stanford.StanfordTokenizer(language=None, encoding=None, stanford_url=None)

Bases: object

Utility class to tokenize, tag and lemmatize documents using a running Stanford CoreNLP server.

Parameters:

• language – Choose between ‘english’, ‘dutch’, ‘russian’. Default: ‘english’
• encoding – Incoming text encoding. Default: utf-8
• stanford_url – If using stanford tagger, address of the server. Default: http://localhost:9000

get_deep_structure_paragraph(raw_paragraph, lowercase, remove_punctuation)

Converts a raw paragraph into a list of sentences, which contain a list of tagged tokens.

Parameters:

• raw_paragraph – Raw paragraph (unicode)
• lowercase – boolean. If true, words will be converted to lowercase. Default: False
• remove_punctuation – boolean. If true, punctuation will be removed. Default: False