# PdfSentencesTokenizer **`class PdfSentencesTokenizer(config)`** Bases: [`BatchedStep`](../technical/libnlp/steps/standard.md#squirro.lib.nlp.steps.batched_step.BatchedStep) PDF Sentences [`Tokenizer`](squirro.lib.nlp.steps.tokenizers.Tokenizer.md#squirro.lib.nlp.steps.tokenizers.Tokenizer) splits the PDF files by sentences and keeps track of the positional information of the extracted sentences. **Note For performance reasons, this step has been deprecated in favor of the “content-conversion” extraction step in the data ingestion pipeline.** If “content-conversion” is already in the pipeline, this step will do nothing. This step only works inside the Squirro Platform due to internal dependencies. **Input** - as input it requires a field with the name files of type [`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict) containing the key-value pairs: {“mime_type”:”application/pdf”,”content_url”:”PATH”} where PATH is an existing path to a pdf-file. **Output** - the output field is filled with data of type [`list`](https://docs.python.org/3.11/library/stdtypes.html#list) [ [`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict) ]. Each [`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict) contains following structure: {“text”:”EXTRACTED_SENTENCE”,”page_to_rects”:{PAGE_NUM:[“x”:X-POS,”y”:Y-POS,”height”:HEIGHT,”width”:WIDTH]}} Parameters - `type` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)) – pdf_sentences - `output_field` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)`, ``"texts"`) – output field for PDF sentences data. - `default_language` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)`, ``"en"`) – Default language if language_field is not present. - `language_field` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)`, ``"language"`) – Document field that gives the language. - `cleaning` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)`, ``{}`) – dict of additional cleaning rules. example: {‘U.N.’:’UN’} **Example** ```json { "step": "tokenizer", "type": "pdf_sentences", "cleaning": { "\t": " ", "\n": "", " ": " ", "approx.": "approx", "etc.": "etc", "i.e.": "ie" } } ``` Methods SummaryMethods Documentation **`get_pdf_files(fields)`** Return type [`Iterator`](https://docs.python.org/3.11/library/collections.abc.html#collections.abc.Iterator)[[`tuple`](https://docs.python.org/3.11/library/stdtypes.html#tuple)[[`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict), [`str`](https://docs.python.org/3.11/library/stdtypes.html#str)]] **`process_doc(doc)`** Process a document Parameters `doc` ([`Document`](../technical/libnlp/base.md#squirro.lib.nlp.document.Document)) – Document Returns Processed document Return type [Document](../technical/libnlp/base.md#squirro.lib.nlp.document.Document)