A single bulk export that takes twelve hours is a scheduling problem. Four exports of three hours each is a scheduling opportunity. Chunking a large export into smaller jobs is the pattern that turns unmanageable exports into a rhythm the operations team can plan around, and every large-scale FHIR client eventually adopts some version of it.
Naming the chunking strategy is the discipline. For related walkthroughs, FHIR background reading collects the surrounding material.
Why Chunk at All
Large exports carry risk that scales with duration. A twelve-hour export that fails at hour eleven wastes eleven hours. Four three-hour exports fail one at a time; the recovery window is smaller and the total loss on failure is bounded.
Chunking also unlocks parallelism. Four exports running in sequence take twelve hours; four exports running in parallel take three, provided the server's concurrency budget allows it. A pass through the site's export duration estimator can bracket both patterns and let the client pick.
Date-Range Chunking
The simplest chunking strategy is date-range. Split the export by month, quarter, or year, and run each range as its own job. The _since and _until parameters (where supported) or since alone bound each chunk.
Date-range chunking works well when the archive is evenly distributed across time. It works less well when historical volume differs dramatically from recent volume. For the driver framing, the four things that dominate $export duration covers when the strategy fits.
Group-Based Chunking
Group-based chunking splits the patient population into groups and runs one export per group. The pattern requires the server to support Group/[id]/$export. Some servers do; some do not.
Group-based chunking produces even-sized chunks and matches the way many clinical use cases model cohorts. It requires more upfront work to define the groups.
Type-Based Chunking
Type-based chunking splits by resource type. One job exports Patients; another exports Observations; another exports Encounters. The _type parameter defines each chunk.
Type-based chunking is easy to implement and produces unbalanced jobs. Observations usually dominate the archive; the Observation chunk is much larger than the others. Type-based chunking works best when combined with another dimension.
Composite Chunking
Deployments that run bulk exports at scale usually combine strategies. Type per year, group per month, and similar composites produce many small chunks that fit inside a scheduled window each.
The trade is complexity. More chunks means more scheduling metadata, more manifest tracking, and more delivery destinations. Sizing plans should be honest about the operational cost.
The Delivery Interaction
Chunked exports produce more files than a single monolithic export. Each chunk generates its own manifest and its own set of delivery URLs. The client-side pipeline has to reassemble the chunks into a coherent view.
For the delivery-side framing, delivery method and its impact on end-to-end ETA covers the shape. Streaming delivery handles chunking naturally; file-based delivery needs a client-side reassembly step.
Scheduling the Chunk Cadence
Chunk cadence is the last operational decision. Running all chunks in parallel maximizes throughput and stresses the server the most. Running them serially minimizes stress and takes the longest. Most deployments end up somewhere in between with a small parallelism factor.
For the wider ETA framing, predicting how long a bulk data $export will actually take covers the composition. Chunking is the pattern that turns hopeful schedules into ones the operations team can execute.
Sources
- HL7 FHIR Bulk Data Access IG - HL7 FHIR Bulk Data Access IG, canonical spec covering group and type filters used for chunking