lingtools.tokenizer_nltk module

class lingtools.tokenizer_nltk.NLTKTokenizer(language, encoding, simple_tokenizer)

Bases: object

Utility class to tokenize and tag documents. By default uses NLTK, but it is possible to use also a running Stanford POS Tagger server.

Parameters:

• language – Choose between ‘english’, ‘dutch’, ‘russian’. Default: ‘english’
• encoding – Incoming text encoding. Default: utf-8
• simple_tokenizer – Use a simple white space tokenizer

get_deep_structure_paragraph(input_document, lowercase, remove_punctuation)

Converts a raw paragraph into a list of sentences with a list of tagged tokens

Parameters:

• input_document – Raw document (unicode)
• lowercase – boolean. If true, words will be converted to lowercase. Default: False
• remove_punctuation – boolean. If true, punctuation will be removed. Default: False