KEE CLI Tool

KEE CLI Tool#

Using the kee command line (cli) tool, you can set up and deploy Known Entity Extraction to a Squirro Project. The tool is included in the Squirro Toolbox, which is available in the Downloads space.

For a comprehensive tutorial about setting up KEE using the command line tool, head over to the KEE CLI Tool Tutorial.

Basic Usage#

The KEE Tool is invoked from a command line window. For example the following command runs the test suite of a KEE project:

squirro_kee -v test --stats

This example has three components:

  • -v is a common argument that turns on verbose logging.

  • test is the sub-command to run.

  • --stats is an argument to the test sub-command.

Note

What is a KEE Project?

A Known Entity Extraction project is a single folder on your computer that has all the required data, configuration and code to create a KEE lookup database.

KEE Project Example#

Every KEE Project will include at least the following files:

  • A CSV file of known entities to look for within the unstructured data

  • A JSON configuration file

  • A db/ folder that includes the lookup database once it is compiled

Many KEE projects will also include additional content such as:

  • A pipelet file (a python script which performs the known entity extraction)

  • A fixtures/ folder that includes example Squirro items used to test the KEE configuration

Configuration#

The basis of a KEE project is the config.json file. This configuration file must be added to each KEE project before you execute the kee command.

See the KEE Config Reference for documentation on how to create this file or the KEE Tutorial for a hands-on example.

Common Arguments#

The kee utility accepts the optional arguments listed below. You must specify any of these arguments before the sub-command and its options.

-h:

Show thr help message and exit.

--version:

Output the tool version and exit.

--verbose, -v:

Increase log verbosity.

  • Not specified: the tool outputs all warnings and errors.

  • Specified once or more: informational messages are also output.

  • Specified twice or more (-vv): debugging messages are shown.

  • Specified three times or more (-vvv): more information is included in all messages.

--log-file:

Path to a log file on disk, where the log output is to be stored. If this is not specified, the log messages are shown on the console.

Sub-Commands#

You always need to specify a sub-command. The following sub-commands exist:

  • compile

  • upload

  • rerun

  • test

  • get_fixture

Compile#

squirro_kee -v compile

Compile the lookup database from the input data. (This applies to which ever KEE project is present within the current working directory of the terminal)

This command does not accept any additional arguments. The only usage of this sub-command is shown below:

Upload#

squirro_kee -v upload

Upload the KEE project to a Squirro server. This creates a pipelet on the server which can then be added to any Squirro project.

You must provide the required information in the "squirro" section of the config.json file (see here) for this to work.

You can specify the following arguments for the upload command:

--no-compile:

Don’t compile the lookup database. By default the compile sub-command is automatically executed when uploading

Rerun#

squirro_kee -v rerun

Re-apply the KEE tagging to the Squirro project. This is used to apply new configuration changes to old items. You must provide the required information in the "squirro" section of the config.json file (see here) for this to work.

You can specify the following arguments for the rerun command:

--query:

The Squirro query for which the KEE extractions should be rerun. Every item that matches this query will be processed.

You can omit the query if the version and version_keyword parameters of the kee section are defined. In this case, the KEE will run on all items that have not yet been tagged with the current value of version in the version_keyword. If the versioning is not set up, you have to submit a query.

--no-compile:

Don’t compile the lookup database. By default the compile sub-command is automatically executed when uploading

Test#

squirro_kee -v test

Run the test suite of the current KEE project. By default the test cases are located in the fixtures directory. The KEE Testing documentation explains how those test cases can be created.

You can specify the following arguments for the test command:

[fixtures…]:

A list of fixture files that should be run. If this is not specified, all the fixtures are tested. The following example runs the test on just two fixture files:

squirro_kee test fixtures/acme.json fixtures/other_corp.json
--no-compile:

Don’t compile the lookup database. By default the compile sub-command is automatically executed when running the tests. That can be slow for large databases, in which case you may want to disable the compilation using this flag.

--snapshot:

Creates a new snapshot from the current test results. Snapshots are stored on disk (in the snapshots folder by default) and are used to compare the KEE result quality over time.

--snapshot-message, -m:

Add a comment to the snapshot. This implies --snapshot as well, so the following command is a short version of creating a snapshot and adding a comment:

squirro_kee test -m "Tuned ngrams"
--diff:

Compare the snapshot to the previous snapshot. This outputs how much better or worse the match quality has become.

--stats:

Outputs a summary for all the missed keywords. This provides a quick overview on what kind of entities are not yet detected as they should.

--trace STRING:

Turns on detailed logging whenever the given candidate is being processed. For example if the lookup database contains an entry called “Acme Inc” then invoking kee as follows will result in a verbose log file every time that entry is looked at:

squirro_kee test --trace "Acme Inc"

If this option is present, then the --verbose flag does not have any effect.

Get fixture#

squirro_kee -v get_fixture 'item-id1' 'item-id2'

Download one or more items from the configured Squirro project and store them in the fixtures folder. You must provide the required information in the "squirro" section of the config.json file (see here) for this to work.

Consult the KEE Testing documentation for information on the fixtures.

You can specify the following arguments for the get_fixture command:

[item_ids...]:

List of item identifiers to download (required).

Contents