Load by Partition

The Load by partition lets you synchronize data in smaller batches (partitions) instead of loading an entire table at once. Partitions are defined by a date-based field, a start date, and a day range. This approach can improve performance, reduce the risk of timeouts, and allow interrupted loads to resume from the point of failure instead of starting over.

This feature is available for Synchronization and Consolidation extractions. It is in Beta for all connector types except for Sage Intacct and Custom API.

When to load by partition

• Improving performance: Reduce the processing time of large table extractions by splitting them into smaller chunks.
• Increasing reliability: Minimise the impact of an error by resuming from the failed partition instead of restarting the whole load.
• Managing large historical datasets: Gradually load extensive date-based data without locking the entire source or destination.

Default vs. partition load

The two load types differ in how data is processed and in the outcome if a failure occurs:

Load type	Description
Default	Loads the entire table in a single operation. If an error occurs before completion, part of the table may be loaded. The entire load must be restarted from the beginning. Cannot resume from the point of failure.
By partition	Loads the table in sequential date-based partitions, starting from the specified start date and using the defined day range. If an error occurs during a partition, earlier partitions remain loaded. Only the failed partition needs to be reloaded. The process resumes from the failed partition on the next run.

Configure a load by partition

1. From the welcome screen or left navigation pane, select Extractions.
2. Select an extraction from the list.
3. Select the link under the Tables column.
4. Select the table you want to configure.
5. Click the Partition Settings icon in the upper‑right corner.
6. Select Use partition settings to enable partition configuration.
7. From the Field dropdown list, choose the date, datetime, or datetimeoffset field in the source table that will determine how to split the data into partitions.
8. Select the Start Date from the date picker (date is in UTC).
9. In Day Range, enter the number of days each partition should cover (1–30).
   For example, entering 7 creates partitions of 7 days each.
10. Click Confirm. A checkmark appears next to the table under the Partition column when partitioning is enabled.

important

To validate the partition settings, run Validate and Build with the option Only add missing tables/fields/indexes enabled.

View load status

1. In the Tables page of an extraction, find a table with a checkmark in the Partition column and select it.
2. Click the Table status icon in the upper‑right corner.
3. Review the status message to see if the table was fully loaded, partially loaded, or never loaded.
4. (Optional) Enable Override default loading behavior to to choose a different way to resume or reload data:
   • Keep all loaded partitions and resume loading from – Enter a date to resume loading. The date must be on or after the partition start date and not in the future.
   • Force reload all data from the initial partition start date – Reload all partitions from the start date, even if already loaded. Available only when the load failed or was aborted.
5. Close the dialog when done.

Status	Description
Never loaded	The table has never been loaded. Running Truncate and Load or Incremental Load loads all data, in partitions if enabled.
Done successfully	All data was loaded successfully.
Error	An error occurred. With partitioning disabled, no data is loaded. With it enabled, some partitions may be loaded and the process resumes on the next run.
Aborted	The run was stopped by a user. Data may be partially loaded and remaining data loads when resumed.