# Standard Types ## base Free functions **`yield_batches(docs, batch_size, yield_batches=True)`** Yield batches of docs from an iterable of docs Parameters - `iterable` (`docs`) – Document iterable - `batch_size` ([`int`](https://docs.python.org/3.11/library/functions.html#int)) – Number of documents per batch - `yield_batches` ([`bool`](https://docs.python.org/3.11/library/functions.html#bool)) – do not apply batches to the dataset. Returns Generator of in-memory batches of documents Return type generator([list](https://docs.python.org/3.11/library/stdtypes.html#list)([Document](../base.md#squirro.lib.nlp.document.Document))) **`encode(x, encoding='utf-8')`** Encode strings in a (possibly) nested object Parameters `x` ([`object`](https://docs.python.org/3.11/library/functions.html#object)) – Object to encode Returns Encoded object Return type [object](https://docs.python.org/3.11/library/functions.html#object) **`flatten(d, parent_key='', sep='.')`** Flattens a dict source: [https://stackoverflow.com/questions/6027558/flatten-nested-python-dictionaries-compressing-keys](https://stackoverflow.com/questions/6027558/flatten-nested-python-dictionaries-compressing-keys) Parameters `d` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)) – Dict to flatten Returns Flattened dict Return type [dict](https://docs.python.org/3.11/library/stdtypes.html#dict) **`unflatten(d, sep='.')`** Unflattens a dict Parameters `d` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)) – Dict to unflatten Returns Unflattened dict Return type [dict](https://docs.python.org/3.11/library/stdtypes.html#dict) **`flatten_list(list_to_flat)`** Flattens a list see: [https://stackoverflow.com/questions/2158395/flatten-an-irregular-list-of-lists](https://stackoverflow.com/questions/2158395/flatten-an-irregular-list-of-lists) Parameters `l` ([`list`](https://docs.python.org/3.11/library/stdtypes.html#list)) – List to flatten Returns Flattened list Return type [list](https://docs.python.org/3.11/library/stdtypes.html#list) ## checkpoint Checkpoint class **`class CheckpointIter(checkpoint)`** Iterator for checkpoint batches **`batch`** List of objects in memory Type [list](https://docs.python.org/3.11/library/stdtypes.html#list)([object](https://docs.python.org/3.11/library/functions.html#object)) **`checkpoint`** Checkpoint object Type [Checkpoint](#squirro.lib.nlp.utils.checkpoint.Checkpoint) **`index`** current index of checkpoint file Type [int](https://docs.python.org/3.11/library/functions.html#int) **`class Checkpoint(batch_size=1, prefix='checkpoint', randomize=False)`** Forms batches of docs and streams them to/from disk **`batch_size`** size of checkpoint batches Type [int](https://docs.python.org/3.11/library/functions.html#int), 1 **`n_files`** number of files in checkpoint Type [int](https://docs.python.org/3.11/library/functions.html#int) **`n_docs`** number of Documents in checkpoint Type [int](https://docs.python.org/3.11/library/functions.html#int) **`prefix`** filename prefix Type [str](https://docs.python.org/3.11/library/stdtypes.html#str), ‘checkpoint’ **`randomize`** whether or not to randomize Documents Type [bool](https://docs.python.org/3.11/library/functions.html#bool), False **`create(docs)`** Setup checkpoint batching Parameters `docs` (`iterable``(`[`Document`](../base.md#squirro.lib.nlp.document.Document)`)`) – iterable of Documents Returns self Return type [Checkpoint](#squirro.lib.nlp.utils.checkpoint.Checkpoint) **`destroy()`** Destroys a checkpoint by removing its files and resetting counters ## classification_metrics **`generate_metric(labels_true, labels_pred)`** ## config Configuration schema validation **`get_schema(step)`** Take in a libNLP Step object and return a schema Parameters `step` ([`Step`](../steps/standard.md#squirro.lib.nlp.steps.base.Step)) – libNLP Step Returns Schema dictionary Return type [dict](https://docs.python.org/3.11/library/stdtypes.html#dict) **`validate(config, schema)`** Validate a config against a schema Parameters - `config` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)) – Configuration dictionary - `schema` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)) – Schema dictionary Returns Configuration dictionary Return type [dict](https://docs.python.org/3.11/library/stdtypes.html#dict) ## email_parsing **`parse_email(email_string, discard_greeting=None)`** Given a non-html email string, parse and clean to extract body. Parsing rules: 1. extract body using regex matching, if not found then use python email parser. 2. given a list as discard_greeting (eg. [“Best regards”, “Warm Regards,”]), discard the body after first appearance of footer string. Parameters - `email_string` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)) – a non-html email string - `discard_greeting` ([`Optional`](https://docs.python.org/3.11/library/typing.html#typing.Optional)[[`Iterable`](https://docs.python.org/3.11/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3.11/library/stdtypes.html#str)]]) – list of footers eg. [“Best regards”, “Warm Regards,”] Return type [`str`](https://docs.python.org/3.11/library/stdtypes.html#str) **`discard_greeting_from_text(body, greetings)`** Remove message content after the salutation at the end of an email. This is mostly done for Salesforce call notes, where the typical email format is not retained. Return type [`str`](https://docs.python.org/3.11/library/stdtypes.html#str) **`class EmailMessage(text)`** Bases: [`object`](https://docs.python.org/3.11/library/functions.html#object) An email message represents a parsed email body. This class was copied from Zapier’s email-reply-parser, which is licensed under the MIT license. [zapier/email-reply-parser](https://github.com/zapier/email-reply-parser/tree/master) **`SIG_REGEX = re.compile('(--|__|-\\w)|(^Sent from my (\\w+\\s*){1,3})')`** **`QUOTE_HDR_REGEX = re.compile('On.*wrote:$')`** **`QUOTED_REGEX = re.compile('(>+)')`** **`HEADER_REGEX = re.compile('^\\*?(From|Sent|To|Subject):\\*? .+')`** **`MULTI_QUOTE_HDR_REGEX = re.compile('(?!On.*On\\s.+?wrote:)(On\\s(.+?)wrote:)', re.MULTILINE|re.DOTALL)`** **`MULTI_QUOTE_HDR_REGEX_MULTILINE = re.compile('(?!On.*On\\s.+?wrote:)(On\\s(.+?)wrote:)', re.DOTALL)`** **`read()`** Creates new fragment for each line and labels as a signature, quote, or hidden. Returns EmailMessage instance Return type [`EmailMessage`](#squirro.lib.nlp.utils.email_parsing.EmailMessage) **`property reply: str`** Captures reply message within email **`quote_header(line)`** Determines whether line is part of a quoted area line - a row of the email message Returns True or False Return type [`bool`](https://docs.python.org/3.11/library/functions.html#bool) **`class Fragment(quoted, first_line, headers=False)`** Bases: [`object`](https://docs.python.org/3.11/library/functions.html#object) A Fragment is a part of an Email Message, labeling each part. **`finish()`** Creates block of content with lines belonging to fragment. Return type [`None`](https://docs.python.org/3.11/library/constants.html#None) **`property content: str`** ## pdf_extract This class aims to convert PDF documents into readable text flow. This is mainly done as input for the catalyst detection algorithms, where we rely on having access to full and properly delimited paragraphs and sentences. The PDF tool we provide out of the box (Tika based) does not handle this case well, as it will randomly insert hard breaks (
for newlines), is terrible at handling tables, can’t copy with inset boxes, and many other issues discovered over time. This solution improves on this. The LineConverter class works with PDFMiner and receives from it the full parsed details of all the content. of the PDF file. Then the MagicPdfAnalyser class takes over. The basic process is to initially detect the layout. This layout detection gives us a list of pages, and for each page it gives us blocks. For this PDFMiner has decent detection of blocks and gives them to us as LTTextBox. We take those as input, but then do a number of corrections: > - If blocks are close to each-other - especially vertically - we merge them. > - If we discover that two blocks form columns, then we merge them. > - If a physical line is drawn between two lines, we will never merge then. > Lines thus always serve to delimit blocks. Next we take those blocks and extract the individual paragraphs from them. **`is_number(string, pos=0, endpos=9223372036854775807)`** Matches zero or more characters at the beginning of the string. **`class BBox(x0, y0, x1, y1)`** Bases: [`tuple`](https://docs.python.org/3.11/library/stdtypes.html#tuple) **`x0`** Alias for field number 0 **`x1`** Alias for field number 2 **`y0`** Alias for field number 1 **`y1`** Alias for field number 3 **`class CharPage(char, page)`** Bases: [`object`](https://docs.python.org/3.11/library/functions.html#object) **`char: LTText`** **`page: dict[str, Any]`** **`class SentenceBox(sentence, bboxes)`** Bases: [`tuple`](https://docs.python.org/3.11/library/stdtypes.html#tuple) **`bboxes`** Alias for field number 1 **`sentence`** Alias for field number 0 **`class LineConverter(rsrcmgr, codec='utf-8', pageno=1, laparams=None)`** Bases: `PDFConverter` A PDFConverter that tries to split the text into lines and columns. Initial PDFMiner stream processing builds up the page structure using the render_* and _render() methods. **`get_pages()`** Return type [`list`](https://docs.python.org/3.11/library/stdtypes.html#list)[[`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)[[`str`](https://docs.python.org/3.11/library/stdtypes.html#str), [`Any`](https://docs.python.org/3.11/library/typing.html#typing.Any)]] **`handle_undefined_char(font, cid)`** This happens when fonts don’t provide the data to be output. The default replaced it with ‘(cid:{cid})’ which won’t look good. We’ll do the same as Tika does and replace it with a space. **`write_text(text)`** **`receive_layout(ltpage)`** **`render_string_vertical(textstate, seq, ncs, graphicstate)`** **`render_string(textstate, seq, ncs, graphicstate)`** **`render_char(matrix, font, fontsize, scaling, rise, cid, ncs, graphicstate)`** **`parse_pdf(fname=None, fp=None)`** Returns the parsed PDF file. This will simply return a page/block/text structure. The MagicPdfAnalyser class will want this as input. **`class MagicPdfAnalyser(pages, get_sentence_spans=, cleaning=None)`** Bases: [`object`](https://docs.python.org/3.11/library/functions.html#object) Takes in detected layout boxes to extract information about text content paragraphs. **`property paragraphs`** **`property sentences: list[tuple[str, dict[int, list[BBox]]]]`** **`get_paragraphs()`** Return type [`Generator`](https://docs.python.org/3.11/library/collections.abc.html#collections.abc.Generator)[[`str`](https://docs.python.org/3.11/library/stdtypes.html#str), [`None`](https://docs.python.org/3.11/library/constants.html#None), [`None`](https://docs.python.org/3.11/library/constants.html#None)] **`debug_boxes(pdf_fname, out_basename)`** Debugging helper which writes information about the boxes extracted to external PNG files. For this we first use pdftoppm to generate the raw PNG files. Then we edit them adding the debugging boxes on top. **`get_sentences_json()`** Get sentences information as JSON-serializable data **`is_sentence_continuation(prev_line, line)`** Return type [`bool`](https://docs.python.org/3.11/library/functions.html#bool) **`has_shared_fonts(line1, line2)`** Checks whether any of the fonts in the two lines are the same. Return type [`bool`](https://docs.python.org/3.11/library/functions.html#bool) **`main()`** ## sentence_splitting **`class TextSpan(text, start, end)`** Bases: [`tuple`](https://docs.python.org/3.11/library/stdtypes.html#tuple) **`end`** Alias for field number 2 **`start`** Alias for field number 1 **`text`** Alias for field number 0 **`sentence_spans(text, language='en')`** This function splits the text into sentences based on the nltk sentence splitter. In addition to sentence text its start and end indexes are returned. Parameters - `text` – text to split into sentences - `language` – input text language Return type [`Iterator`](https://docs.python.org/3.11/library/collections.abc.html#collections.abc.Iterator)[[`TextSpan`](#squirro.lib.nlp.utils.sentence_splitting.TextSpan)] Returns sentence spans **`sentence_splitting(text, rules=None, cleaning=None, language='en')`** This function splits the text into sentences based on the nltk sentence splitter. There is an option to define additional sentence splitting or cleaning rules to match specific document styles Parameters - `text` – text to split into sentences - `rules` – list of additional splitting rules - `cleaning` – dict of additional cleaning rules - `language` – input text language Returns split & cleaned sentences ## squirro_utils Squirro functions **`is_squirro_type(value)`** Returns True if value is an accepted Squirro type, otherwise False. “Accepted Squirro type” is any JSON serializable value. Return type [`bool`](https://docs.python.org/3.11/library/functions.html#bool) **`nlp_doc_to_squirro_item(doc, fields=None, ensure_serializable=True, wrap_into_list=True)`** Converts NLP lib Documents to Squirro items Parameters - `doc` ([`Document`](../base.md#squirro.lib.nlp.document.Document)) – Document - `fields` ([`Optional`](https://docs.python.org/3.11/library/typing.html#typing.Optional)[[`list`](https://docs.python.org/3.11/library/stdtypes.html#list)[[`str`](https://docs.python.org/3.11/library/stdtypes.html#str)]]) – List of flattened fields to include in the Squirro item - `ensure_serializable` ([`bool`](https://docs.python.org/3.11/library/functions.html#bool)) – ensure that the resulting Squirro item is serializable. By default, this is true; it can be turned off for performance improvement in cases where the Squirro item is not meant to be transmitted elsewhere (e.g., intermediate workflow when optimizing workflow execution) - `wrap_into_list` ([`bool`](https://docs.python.org/3.11/library/functions.html#bool)) – Ensure values are wrapped into a list Returns List of dicts in Squirro item format Return type [dict](https://docs.python.org/3.11/library/stdtypes.html#dict) **`squirro_item_to_nlp_doc(item, fields=None)`** Converts Squirro item to NLP lib Document Parameters - `item` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)) – List of dicts in Squirro item format - `fields` ([`Optional`](https://docs.python.org/3.11/library/typing.html#typing.Optional)[[`list`](https://docs.python.org/3.11/library/stdtypes.html#list)[[`str`](https://docs.python.org/3.11/library/stdtypes.html#str)]]) – List of flattened fields to include in NLP Document Returns Document Return type [Document](../base.md#squirro.lib.nlp.document.Document) **`get_squirro_facet_type(facet)`** Determines the facet type of a keyword from a Squirro item Parameters `facet` ([`list`](https://docs.python.org/3.11/library/stdtypes.html#list)`(`[`object`](https://docs.python.org/3.11/library/functions.html#object)`)`) – Facet whose Squirro facet type is to be determined Returns Squirro facet type Return type [str](https://docs.python.org/3.11/library/stdtypes.html#str) **`get_squirro_client(cluster, token, client_id=None, client_secret=None, genai_api_url=None)`** Create and authenticate a Squirro client Parameters - `cluster` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)) – Squirro cluster URI - `token` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)) – Squirro API refresh token - `client_id` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)`, ``None`) – Squirro client ID - `client_secret` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)`, ``None`) – Squirro client secret - `genai_api_url` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)`, ``None`) – GenAI service API URL, if using GenAI resources Returns Squirro client Return type [SquirroClient](../../api/squirro_client/squirroclient-class.md#squirro_client.SquirroClient) ## stopwords ## tqdm **`class TqdmLogger(logger, level=None)`** Bases: [`StringIO`](https://docs.python.org/3.11/library/io.html#io.StringIO) Output stream for TQDM which will output to logger module instead of the StdOut. original author: @ddofborg original source: [tqdm/tqdm#313](https://github.com/tqdm/tqdm/issues/313) **`buf = ''`** **`logger = None`** **`level = None`** **`write(buf)`** Write string to file. Returns the number of characters written, which is always equal to the length of the string. **`flush()`** Flush write buffers, if applicable. This is not implemented for read-only and non-blocking streams.