Chunking

Documentation Index

Fetch the complete documentation index at: https://support.detrics.io/llms.txt

Use this file to discover all available pages before exploring further.

When a sync would be too large to fetch in a single API call — many accounts, long date ranges, or detailed entity levels like individual ads — Detrics automatically splits it into smaller pieces called chunks. Each chunk fetches a subset of the data, persists it to BigQuery as soon as it finishes, and then the next chunk starts. This is purely an execution detail. Your BigQuery tables end up with exactly the same data either way; chunking just changes how Detrics gets there.

​

Why Chunking Exists

Marketing platforms impose limits on how much data a single API request can return — and even when they don’t, very large requests are slow and fragile. Without chunking, a single timeout or transient error would lose the entire sync’s progress. Chunking solves three problems at once:

• Timeouts — Each chunk runs in its own request, well under the platform’s time limits
• Memory — Detrics never holds an entire historical backfill in memory at once
• Recovery — If a sync is interrupted, completed chunks are preserved; only the remaining chunks need to be retried on the next run

​

How Chunks Are Split

Chunking happens along two axes:

• Time range — Long date ranges are split into segments (typically by calendar month)
• Account batches — Many accounts are processed in smaller groups

The exact chunk size depends on the platform and the entity level of the table. Denser data needs smaller chunks: If a table’s data fits comfortably in a single request — for instance, a short date range with few accounts — no chunking is performed, and the run executes as a single fetch. This is normal and expected for small or recently-created tables.

​

Chunk Order

Chunks are executed in recent-first order: the most recent time windows are fetched before the older ones. This means fresh data lands in BigQuery first, so even if a long initial backfill is paused or interrupted, you still have usable recent data to query.

​

What You See in the UI

In the Run History of a transfer, each table row shows X/Y chunks when chunking was used (e.g., 4/4 chunks means all 4 chunks completed successfully, 12/13 chunks means one chunk failed or is still retrying). If you don’t see a chunk count, the table was fetched in a single request — no chunking was needed for that run.

​

Chunking and Resyncs

When you trigger a manual resync or when an adaptive retry runs, Detrics replays only the chunks that didn’t succeed in the original run. Successful chunks are not refetched, which keeps resyncs fast and minimizes API quota usage. See Resync and Recovery for details.