Bulk export duration variance across FHIR deployments is enormous. The same one-million-Patient export can take twenty minutes on one server and eight hours on another. Four factors account for almost all of the variance, and knowing the four is the difference between a hopeful schedule and a defensible one.
Naming the four is the discipline. For related walkthroughs, the FHIR knowledge collection collects the surrounding material.
One: Resource Count in the Scope
Resource count is the biggest driver by a wide margin. A Patient export with a small patient group is fast; the same export against a whole-population group is many hours. Per-resource serialization time is roughly constant per engine; the count multiplies through directly.
Estimating the count before starting the export is essential. Group definitions, type filters, and since parameters all shape the count. A pass through the site's export duration estimator uses count as its primary input.
Two: Resource Complexity
Not every resource type serializes at the same rate. Observations with complex CodeableConcept expansions serialize slower than simple Patient resources. Bundle transactions inside the archive can serialize an order of magnitude slower than the entries they hold.
Resource complexity is the multiplier that most naive ETAs miss. Estimate it per resource type in the export scope, not as a single average across the archive. For the full prediction framing, predicting how long a bulk data $export will actually take covers the composition.
Three: Chunk Size and Overhead
Bulk exports produce one or more NDJSON files per resource type. The number of files depends on the server's chunk size configuration. Smaller chunks produce more files with more per-file overhead; larger chunks produce fewer files with more memory pressure.
The optimal chunk size is deployment-specific. Servers that let clients hint at the chunk size give the client control; servers that pick automatically leave the client with whatever the engine picked.
Four: Concurrent Traffic Contention
Bulk exports share the same persistence tier as interactive traffic. When both are active at once, the export contends with the interactive workload for I/O and pool slots. The contention slows the export and hurts interactive latency.
The two failure modes usually appear together. Scheduling the export around production traffic windows is the mitigation, and it is where most deployments learn the interaction the expensive way. For the specific scheduling framing, planning $export around production traffic windows covers the pattern.
Combining the Four
A defensible ETA multiplies through the four:
- ETA ≈ (count × per_resource_complexity_ms × contention_factor) / parallel_workers + chunk_overhead
Every term is bounded. Count comes from the scope. Complexity comes from engine benchmarks. Contention factor comes from the scheduled window. Parallel workers come from the server's concurrency budget. Chunk overhead comes from the manifest.
The Diagnostic Approach
When an export runs slower than expected, the diagnostic starts with the four factors. Higher-than-estimated resource count is the most common cause. Complexity underestimation is next. Contention with concurrent workloads is the third. Chunk-size mismatch is fourth.
Naming the cause per incident is what compounds over time. Deployments that log the four factors per export build a real benchmark of their own workload.
The Chunking Consequence
Large exports that would take many hours as a single job can be chunked into smaller jobs that complete in parallel. For the specific chunking strategy, chunking a large $export into manageable jobs covers the shape.
Every bulk-data operation that runs on schedule owns the four factors. Everything else is guessing.
Sources
- HL7 FHIR Bulk Data Access IG - HL7 FHIR Bulk Data Access IG, canonical reference for export scope drivers