Question Answering#

Introduction#

Question Answering (QA) is a general purpose task that consists of answering a question by extracting the answer as a span from a contextual text that is supplied together with the question, as in the following example:

Question: "What color is it?"
Context: "The tomato is red."

Answer: "red"

If the answer cannot be found as part of the context, the question is deemed unanswerable and the answer is an empty string, as in the example:

Question: "What is his nickname?"
Context: "The tomato is red."

Answer: "" (unanswerable)

However, QA is not a closed-book Question Answering model, i.e. it always requires a context that is supplied along with the question that possibly contains the answer for the question which is extractable from the text as a span of words.

Furthermore, QA is not a generative Question Answering model, i.e. it can only answer the question verbatim from the context.

Frequently Asked#

Is it necessary to fine-tune a QA model for a specific use-case?#

In general, this is not required. If the questions and context texts are formulated in natural (English) language, the model should be able to predict the answer from the context or say “unanswerable” if the context does not contain an extractable answer.

Only if the target distribution is very different from the training distribution might fine-tuning of a new model be necessary, e.g. if the questions and context are formulated in a different language or if the English terms used are radically different from what might be expected from a natural English speaker.

How many samples are needed for fine-tuning a QA model?#

It is not possible to give a definitive answer to this question, as it depends on many factors that are nonlinearly related. However, SQuAD2.0, for example, the de facto standard data set for the training and evaluation of QA models, can serve as a guide. It contains a total of about 150’000 pairs of question and corresponding context. Of these 150’000 samples, about 50’000 questions are unanswerable by the supplied context and requires the model to generate no answer predictions. The dataset is split into a training set, a validation set, and a test set containing 129’941, 6’078, and 5’915 sample pairs, respectively. Fine-tuning a model to achieve comparable performance to models trained on the SQuAD2.0 task will likely require constructing a dataset with similar properties.

The QA feature can be configured via the Squirro Project Configuration.

Default Configuration#

Project Configuration topic.search.qa-configuration

{
    "min_score": 0.1,
    "use_title_as_fallback": true,
    "num_answers": 3,
    "abstract_size": 5000,
    "lookup_answer_in_top_items": 3,
    "lookup_answer_in_top_abstracts": 3,
    "order_answers_by": ["es_score", "qa_score"],
    "perform_on_languages": ["en"],
    "consider_query_type": false,
    "min_query_length": 3
}

Configuration Reference#

min_score

Specifies a minimum score returned by the model to consider the answer to be relevant. Threshold can be disabled by setting min_score to 0.

Type: number (float)

use_title_as_fallback

Specifies whether to search for answer in the title if item abstract is not provided.

Type: boolean

num_answers

Specifies the number of returned answers.

Type: number (int)

abstract_size

Specifies an item abstract size that is used to search for answer.

Type: number (int)

lookup_answer_in_top_items

Specifies a number of items that are used to search for answer.

Type: number (int)

lookup_answer_in_top_abstracts

Specifies a number of abstracts in the item that are used to search for answer.

Type: number (int)

order_answers_by

Specifies what scores are used to order answers. If more than one value is given, the ordering is performed on multiple scores. Possible values are: es_score, qa_score.

Type: list (of strings)

perform_on_languages

Specifies the list of languages in which the question should be asked to search for an answer. To find an answer regardless of the language of the question, set this option to null. By specifying an empty array, search for answers is not performed for any language (QA is disabled).

Type: list (of strings)

consider_query_type

Specifies whether the type of user query is taken into account. Only natural language questions are considered if this option is set to true. Otherwise the query classification is bypassed and the query is used for question answering, regardless of the predicted question type.

Type: boolean

min_query_length

Specifies the minimum number of words in a query to be considered for question answering. This rule is bypassed if the query ends with a question mark symbol (?).

Type: number (int)