A quantum algorithm is not fairly evaluated by comparing one execution time or one accuracy score with a classical program. This guide provides a repeatable benchmarking workflow for Qiskit, PennyLane and cloud-hardware experiments, covering problem formulation, baseline selection, quality, runtime, circuit resources, sampling cost, noise, scalability and reproducibility.
Overview
Quantum algorithm benchmarking means measuring whether a quantum approach is useful for a defined problem under clearly stated conditions. The comparison may involve a simulator, a real quantum processor, a classical optimiser or a complete hybrid quantum-classical workflow. Each choice affects the result.
The central question is not simply, “Is the quantum circuit faster?” A fair benchmark asks:
- Does each method solve the same problem?
- Are the input data, constraints and accuracy targets equivalent?
- Are preprocessing, post-processing, compilation and optimisation included?
- How does performance change as the problem grows?
- What resources are required to obtain a reliable answer?
- Can another developer reproduce the experiment?
This matters because a quantum circuit may produce a good result on a small test case while requiring substantial classical processing, repeated circuit executions or error mitigation. Conversely, a classical baseline that has not been tuned, or that solves a different formulation, can make a quantum method appear stronger than it is.
Use the benchmark as an engineering artefact rather than a one-off result. Store the problem definition, software environment, circuit settings, hardware target, random seeds, measurement counts and evaluation scripts alongside the headline metrics. For broader project decisions, pair the technical results with a quantum computing business case and pilot template.
How to estimate
A useful estimate separates solution quality from the resources required to obtain it. Follow the workflow below before running large experiments.
1. Fix the problem formulation
Define the objective function, decision variables, constraints, input representation and acceptable output. For an optimisation experiment, record whether the algorithm returns a feasible solution, the objective value, or both. For a classification experiment, specify the train, validation and test splits before comparing models.
Do not change the formulation to suit one implementation. If the quantum method uses a binary quadratic model while the classical method receives a simplified version, the results are not directly comparable. Encoding choices also affect qubit requirements and circuit complexity; document them using the same level of detail as the algorithm itself.
2. Choose a credible classical baseline
Use at least one baseline that a practitioner would reasonably select for the problem. Depending on the task, this might be an exact solver for small instances, a well-established heuristic, a linear or tree-based model, a tensor-network simulator, or a domain-specific optimisation method.
Report the baseline configuration rather than naming only the library. Include solver version, stopping criteria, hardware, thread count, tolerances, initialisation and time limit. If the quantum method is hybrid, include the classical optimiser and its settings as part of the quantum workflow.
3. Define the metrics before running the experiment
At minimum, compare quality and resource use separately. A compact metric set is:
| Category | Metrics to record | Why it matters |
|---|---|---|
| Solution quality | Objective value, error, approximation ratio, feasibility, accuracy or F1 score | Shows whether the method solves the task adequately |
| Wall-clock performance | Preprocessing, compilation, queue, execution, optimisation and post-processing time | Prevents a narrow runtime comparison |
| Quantum resources | Logical qubits, physical qubits, circuit depth, two-qubit gate count and shots | Describes the cost and hardware demand of the circuit |
| Reliability | Variance across runs, confidence intervals, failure rate and constraint violations | Separates a repeatable result from a lucky sample |
| Scalability | Performance across instance size, feature count or problem density | Tests whether the result survives beyond a toy example |
4. Use repeated runs and a fixed budget
Quantum measurements are sampled, and many algorithms also include random initialisation or stochastic optimisation. Run each configuration more than once where practical, retaining the individual results rather than reporting only the best run.
Choose a comparison budget in advance. It could be a wall-clock limit, a fixed number of function evaluations, a fixed shot budget, a fixed number of optimiser iterations or a defined cloud execution allowance. If one method is allowed to continue until it finds an acceptable result while another has a strict limit, state that asymmetry clearly.
5. Compare ideal, simulated-noisy and hardware settings
Separate three questions:
- Algorithmic behaviour: Does the circuit work under ideal simulation?
- Noise sensitivity: How does performance change under a representative noise model or noisy simulator?
- Operational behaviour: What happens on the selected hardware, including compilation, queueing, shots and calibration conditions?
An ideal simulator result is useful for debugging and algorithm analysis, but it is not a hardware performance result. A cloud-hardware run should include the backend, date or run window, circuit compilation settings, shots, mitigation options and any failed or discarded jobs. Platform details can change, so review the relevant quantum cloud platform comparison before designing a production-oriented test.
Inputs and assumptions
A benchmark is only as useful as its assumptions. Create a configuration file or experiment record containing the following inputs.
- Problem inputs: dataset identifier, instance generator, number of variables, feature dimensions, density, constraint structure and train-test split.
- Encoding inputs: basis, angle or amplitude encoding; normalisation method; ancilla requirements; and data-loading time. See the guide to quantum data encoding methods for the trade-offs.
- Algorithm inputs: ansatz or circuit structure, depth, optimiser, learning rate, stopping rule, initial parameters and number of repetitions.
- Execution inputs: simulator or hardware target, shots, transpilation or compilation options, error mitigation, batch size and queue time policy.
- Classical inputs: software versions, processor type, memory, thread count, parallelism, solver tolerances and time limits.
- Reproducibility inputs: source revision, dependency lockfile, random seeds, environment details and raw output location.
Define what “runtime” means. A narrow circuit execution time may be appropriate for a low-level systems study, but an end-to-end application benchmark should normally account for data preparation, circuit construction, compilation, submission, measurement, classical optimisation and result processing. Present both where possible.
For a hybrid algorithm, estimate total execution work with a simple model:
Total evaluation cost = number of circuit evaluations × shots per evaluation × cost per shot
This is not a universal pricing formula. It is a planning model that makes hidden sampling work visible. If the platform charges or limits usage differently, replace “cost per shot” with the relevant resource measure and report the platform assumptions separately.
Normalise quality only when the normalisation is meaningful. For a minimisation problem, one possible approximation ratio is:
Approximation ratio = quantum result quality ÷ reference result quality
The exact formula depends on whether values can be negative, whether the reference is optimal and whether constraints are satisfied. State the definition instead of assuming that a ratio has the same interpretation across all problems.
Worked examples
Example 1: QAOA for a small optimisation problem
Suppose a QAOA experiment tests graph instances with increasing numbers of vertices. The quantum workflow uses the same instance generator for every run, records the best feasible objective value, and repeats each circuit configuration with a fixed shot budget. The classical comparison uses an exact solver for small instances and a documented heuristic when the exact solver reaches its time limit.
The report should include:
- Number of vertices, edges and graph-generation seed
- QAOA layer count, logical qubits and compiled circuit depth
- Shots, optimiser iterations and total circuit evaluations
- Best, median and spread of objective values across runs
- Feasibility rate and time spent in compilation, execution and optimisation
- Classical solver settings and whether the reference solution is proven optimal
A fair conclusion might be that the quantum workflow reached comparable solution quality under a specified small-instance budget, while its end-to-end runtime or sampling demand remained higher. That is more informative than claiming a general advantage from one graph size.
Example 2: A PennyLane hybrid model
For a variational classifier, use identical training and test data, preprocessing and evaluation metrics for the quantum and classical models. Match the experimental budget by recording training time, number of parameter updates, model evaluations and hardware or simulator execution work.
Do not compare a quantum model’s test accuracy with a classical model’s training accuracy. Report the mean and variation across fixed seeds, and keep a final test set untouched until model selection is complete. A PennyLane hybrid model tutorial can provide implementation context, but the benchmark should still document the exact circuit and data pipeline used.
Reusable benchmark template
Use the following record for each experiment:
Experiment ID:
Problem and instance source:
Instance size and seed:
Quantum SDK and version:
Classical software and version:
Quantum method and circuit depth:
Logical and physical qubits:
Shots and optimiser evaluations:
Hardware or simulator configuration:
Classical baseline and settings:
Quality metric and reference value:
End-to-end runtime:
Quantum execution work:
Repeated-run summary:
Noise or mitigation settings:
Source revision and environment:
Limitations and next test:
When to recalculate
Revisit a quantum algorithm benchmark whenever an input that affects cost, quality or reproducibility changes. A practical review cycle is at least every six months for an active project, with earlier updates when:
- the SDK, compiler, simulator or device interface changes;
- the hardware target, calibration profile, queue policy or shot configuration changes;
- cloud pricing, quotas or billing units change;
- a classical baseline receives a material algorithm or implementation update;
- the dataset, instance generator or encoding method changes;
- the circuit, optimiser, error mitigation or stopping rule changes;
- the project moves from simulation to hardware or from a prototype to a service-level workflow.
Before recalculating, freeze the previous report so that improvements are not silently mixed with changes in methodology. Then rerun a small validation set, confirm that the metrics still answer the business or research question, and expand to the full benchmark only if the environment is behaving as expected.
For your next experiment, start with three instance sizes, one credible classical baseline and a fixed execution budget. Record quality, end-to-end time, circuit resources and variation across runs. Publish the raw configuration with the summary table, state what was excluded, and label ideal simulation, noisy simulation and hardware results separately. This workflow turns a quantum-versus-classical comparison into evidence that can be checked, repeated and updated as the underlying tools and platforms change.
For enterprise evaluations, combine the technical benchmark with vendor and delivery risk checks using the quantum technology vendor due diligence checklist. For a first implementation workflow, the Qiskit tutorial offers a suitable starting point for building, simulating and running circuits.