# Incremental Loading The Squirro data loader handles incremental loading for data loader plugins. The built-in support works if the data source has support for a sorted attribute that will change every time the data is updated. A common attribute for this is a last modified timestamp. To handle incremental loading, the [`connect()`](datasource.md#squirro.dataloader.data_source.DataSource.connect) method takes two arguments: `inc_column` and `max_inc_value`. They contain the name of the column on which incremental loading is done (specified on the command line with `--incremental-column`) and the maximum value of that column that was received in the previous load. Well-behaved data loader plugins should implement this property, so that only results are returned where `inc_column >= max_inc_value`. The data loader then takes care of all the required book-keeping. Data loader plugins that do not support incremental loading, should raise an error when this option is specified: ``` def connect(self, inc_column=None, max_inc_value=None): if inc_column: raise ValueError("Incremental loading not supported.") ``` With most sources, the incremental column is only sensible and supported on one specific column or property. In that case, it is recommended to implement [`getIncrementalColumns()`](datasource.md#squirro.dataloader.data_source.DataSource.getIncrementalColumns) and enforce that in [`connect()`](datasource.md#squirro.dataloader.data_source.DataSource.connect) as well: ``` def connect(self, inc_column=None, max_inc_value=None): if inc_column and inc_column != "updated_at": raise ValueError("Incremental loading is only supported on the updated_at column.") def getIncrementalColumns(self): return ["updated_at"] ```