Most FHIR server performance claims fall apart the moment someone asks how the test was set up. Different hardware, different datasets, different concurrency, different warm-up rules. The new public FHIR server benchmark from Health Samurai is interesting less because of the headline numbers and more because the harness is open. Aidbox, HAPI FHIR, Medplum, and the Microsoft FHIR Server all run through the same scenario on the same machine, and the full setup is published at the open benchmark repo. The author is Marat Surmashev, VP of Engineering at Health Samurai. For the FHIR implementation hub, the surrounding FHIR coverage lives in the broader collection.
The value of an open harness is that the bias becomes auditable rather than hidden.
Hardware Pinning
A fair test runs every server on the same machine with the same per-container allocation. The benchmark uses a single bare-metal host with 64 CPU cores and 500 GB RAM. Each FHIR server gets 8 vCPU and 24 GB RAM, with one exception: Medplum follows its documented deployment model and runs 8 replicas of 1 vCPU and 3 GB each. The replica shape is part of how Medplum is meant to be deployed, so the harness honors that rather than forcing a uniform single-container layout.
Hardware pinning matters because shared infrastructure noise is the most common source of unfair benchmark numbers. A test that runs server A on Monday and server B on Friday is not the same test even if the machine is nominally identical.
Data Parity
Every server loads the same Synthea dataset of 1,000 patients, roughly 2 million FHIR resources. Same generation seed, same resource graph, same value distributions. That removes the most common excuse in any cross-server discussion, where one side argues that the dataset favored a particular implementation.
The trade is the obvious one: a 1,000-patient dataset fits comfortably in 500 GB of RAM. The next post in the series is expected to test at scale. The current snapshot is a clean baseline, not a stress test.
Database Parity Where Possible
Aidbox, HAPI FHIR, and Medplum all run on PostgreSQL 18 in this harness. The Microsoft FHIR Server runs on SQL Server 2022 Developer Edition, because it does not support PostgreSQL. The harness picks the production-shaped database for each server rather than forcing a single backend that would penalize the Microsoft implementation unfairly.
Same Load Generator, Same Workload
Grafana k6 generates the load against every server. The CRUD scenario uses 300 concurrent threads against nine resource types. Bundle import uses 20 concurrent threads. Search runs across all six FHIR search parameter families: string, date, reference, token, quantity, and composite. The k6 scripts and the scenario configs are all in the repo, which means the harness is rerunnable on any team's own hardware.
Daily Rerun and Open Source
The full setup is open source under a permissive license, runs daily on a CI schedule, and publishes the snapshot to a live dashboard. That is the part that makes the harness honest. Anyone can fork the repo, change a knob, and post their own numbers. The Medplum CTO has already forked it, which is what you want to see when an open benchmark lands. For surrounding FHIR-server reading, the best open-source FHIR servers for startups in 2026 and the HAPI vs Microsoft FHIR Service comparison cover the wider field.