Command line usage¶
Table of contents
Lingtools helper script¶
If all you need is to extract features from text, you can use the lingtools_script.py script in the bin directory.
After installing the lingtools package (see Installation), go to the bin directory:
cd ~/src/lingtools/bin
Check the script help for options:
python lingtools_script.py -h
Each functionality of the script has its own help page. For example:
python lingtools_script.py features -h
Generate a configuration file¶
For more control regarding how the text is processed, you can inform extra settings to the script. These settings are the same as described in the API (see, for example, lingtools.tokenizer.LocalTokenizer).
You can generate a sample configuration file with the default options:
python lingtools_script.py generate --output config.yml
The sample configuration file can be edited as needed and informed to the script, for example:
python lingtools_script.py features INPUT_PATH --config=config.yml
Extract features¶
You can extract features from text using the command “features”.
By default, the script takes a simple text file in which each line is a document and outputs the results to the console:
python lingtools_script.py features INPUT_PATH
You can also treat the input file as one single document:
python lingtools_script.py features INPUT_PATH --type txt_single
You can also use CSV files as input, with any separator character (for example a semi-colon) and skipping one line to ignore the header:
python lingtools_script.py features INPUT_PATH --type csv --sepin=";" --skip=1
If you want, the script can write the results to an output file. You can choose which separator character will be used:
python lingtools_script.py features INPUT_PATH --output="output.csv" --sepout="tab"
Logging messages can be directed to a file:
python lingtools_script.py features INPUT_PATH --log="log.txt"
And the debug flag can turn on extensive debugging messages (a log file path must also be provided):
python lingtools_script.py features INPUT_PATH --log="log.txt" --debug
For more configuration options, use the configuration file generated by the command generate (see Generate a configuration file).
Dynamic themes¶
You can calculate cosine distances between a list of theme words (split by commas) and one or more documents:
python lingtools_script.py themes INPUT_PATH "theme, words, write"
Other options are similar as described in Extract features. For a list of all options, consult the command help:
python lingtools_script.py themes -h
Cosine similarity matrix¶
With this command, you can calculate a matrix of cosine distances between the lines of a file. Each line is considered a document, and each document can contain one word or multiple words:
python lingtools_script.py cosines INPUT_PATH
Other options are similar as described in Extract features. For a list of all options, consult the command help:
python lingtools_script.py cosines -h
Nearest neighbors¶
With this command, you can get the nearest neighbors from a piece of text (one or more words):
python lingtools_script.py neighbors TEXT
Note that, unlike other commands, this command takes the raw text and not a file path. If you want to pass the contents of a file, you can pipe in the file:
python lingtools_script.py neighbors < INPUT_PATH
Other options are similar as described in Extract features. For a list of all options, consult the command help:
python lingtools_script.py neighbors -h
Distinctive words¶
With this command, given two sets of texts, you can see the words that most distinguish the two:
python lingtools_script.py words INPUT_PATH_1 INPUT_PATH_2
Other options are similar as described in Extract features, but with different options for file 1 and file 2. For a list of all options, consult the command help:
python lingtools_script.py words -h