Data Loader Plugin Configuration

Data loader plugins usually require configuration to provide different ways to access the source.

This may include credentials (but see data-loader-customer-connector for a way to avoid this) or specific data selection properties.

Such configuration is handled by implementing the getArguments() method on the data loader plugin. This method returns a list of dictionaries which describe each argument.

name

The name with which the argument is referenced, using self.args.<name>. For manual passing in of arguments (in the command line or Known Entity Extraction) this is also the name with which the option can be passed in. For command line arguments, underscores are replaced with hyphens.

help

The help string. If display_label is not provided, help takes its place as a main label. Otherwise, it serves as a secondary help text in UI.

display_label

The short label for this argument shown in the UI.

Set explicitly to None to hide this argument from the UI, in which case this is an option that can only be used from the command line.

required

True if this argument is mandatory. For such arguments, the Project Creator won’t be able to proceed with creation of the load job in the UI if this is not provided. Also in the command line version, an error will be thrown if such arguments are omitted.

type

Type to try to convert the value of the argument into.

Basic types are: str, int, float, bool

Special:

• "password" - which will be used as a string in the load script, but will be interpreted as a password field in the UI (hidden characters).

• "file" - allows passing in files. This is passed into the plugin as a filename, which can be passed to the open() builtin. In user interface the user will see a file selector to upload a file.

• "code" - multi-line text, e.g., SQL statements. A code-editor-like box will show when creating the loader from UI.

default

The default value of the argument.

action

Action to perform when this flag is given.

"store" is the default and simply allows the usage of the value of the argument within the script.

"store_true" will turn the value of the argument to True if provided, False if not provided. This is shown as a checkbox in the user interface.

"store_false" will turn the value of the argument to False if provided, True otherwise. This is shown as a checkbox in the user interface.

nargs

Can be set to "*" if multiple values can be provided for this value. "+" can be used if more than one value must be provided.

This is used if you want the user to provide a list of arguments, such as list of IDs, URLs etc.

The resulting value is made available as a list to the data loader plugin.

Example:

def getArguments(self):
    return [
        {
            "name": "feeds",
            "nargs": "+"
        }
    ]

advanced

If this is set to True the setting is moved to the “Advanced Options” section which is collapsed by default.