Quantum Experiment Reproducibility Checklist: Environments, Seeds, Shots and Backend Metadata
reproducibilityquantum experimentsQiskitquantum SDKsresearch workflowversion control

Quantum Experiment Reproducibility Checklist: Environments, Seeds, Shots and Backend Metadata

SSmart Qubit Labs Editorial Team
2026-11-14
7 min read

Use this practical checklist to record quantum SDK environments, seeds, circuits, shots, backend metadata and outputs for reproducible experiments.

A reproducible quantum experiment is more than a saved circuit. This checklist shows what to record about your environment, random seeds, circuit construction, transpilation, shots, noise assumptions, backend metadata, post-processing and outputs so another developer can rerun the work and understand any differences.

Overview

Quantum software changes quickly, and quantum hardware is not perfectly static. A result that looks straightforward to reproduce can depend on the exact SDK release, compiler options, backend selection, calibration state, random-number generator, measurement interpretation and even the order in which jobs were submitted. Reproducibility therefore means preserving both the experiment and the conditions under which it ran.

Use this page as a pre-publication audit, a project-start template or a handover checklist. The aim is not to freeze every tool forever. It is to make the original run identifiable, to separate deterministic code from changing infrastructure, and to make later reruns informative even when an identical result is no longer possible.

Minimum reproducibility record

  • Code: repository URL, commit identifier, branch or release tag, and the exact entry point used to run the experiment.
  • Environment: operating system, Python version, quantum SDK versions, provider or plugin versions and dependency lockfile.
  • Experiment: input data, circuit-construction parameters, ansatz or algorithm settings, initial values and stopping criteria.
  • Execution: simulator or hardware backend, transpilation options, optimisation level, shots, queue or job identifier and submission timestamp.
  • Randomness: every seed that affects circuit generation, parameter initialisation, sampling, shuffling, noise or post-processing.
  • Results: raw counts or samples, expectation values, intermediate logs, metadata, analysis scripts and generated figures.

For SDK-specific differences, compare the workflow assumptions in a Qiskit, Cirq and PennyLane compatibility guide before moving an experiment between frameworks.

Checklist by scenario

1. Local simulator or development run

Start by recording the local environment and the command used to launch the experiment. A practical project record should include:

  • Python and operating system versions.
  • SDK, simulator, numerical library and provider versions.
  • A lockfile or pinned requirements file, such as a deliberately maintained requirements or project configuration file.
  • The repository commit and a clean working-tree status.
  • Input files, configuration files and their checksums where inputs are too large to store directly.
  • Number of qubits, circuit depth, gate set, parameter values and measurement basis.
  • Shot count, simulator method and any simulator-specific options.
  • Random seeds for circuit creation, parameter initialisation and sampling.

Do not rely on a notebook cell showing a seed unless the seed is set before every relevant random operation. Record whether the SDK, NumPy, Python's random module or another library owns each source of randomness. A single global seed may not control all of them.

2. Cloud simulator or quantum hardware run

Cloud execution adds provider and backend state to the reproducibility record. Save the following before deleting jobs or cleaning temporary files:

  • Provider, account or project context, without exposing credentials or secrets.
  • Backend name, backend type and the provider's job or execution identifier.
  • Submission time, completion time if available, and the region or workspace when it affects routing.
  • Original circuit, transpiled circuit or submitted instruction representation.
  • Transpiler version, basis gates, coupling or connectivity assumptions, layout, routing method and optimisation settings.
  • Requested shots, seeds supplied to the transpiler or simulator, and any resilience, error-mitigation or measurement-mitigation options.
  • Backend configuration and calibration information returned with the job, including the timestamp or validity period.
  • Raw counts, samples, expectation values and the complete response metadata.

Hardware results should be described as time-dependent observations rather than permanent properties of a backend. If calibration data cannot be exported, preserve the provider response, job identifier and the date of execution, then state that a later rerun may encounter different device conditions.

3. Noise-model or error-mitigation study

Noise experiments need an explicit separation between measured noise, a constructed noise model and a noiseless baseline. Record:

  • Whether noise was sampled from hardware data, supplied by a simulator, or defined manually.
  • Error channels, probabilities, correlations, readout assumptions and any truncation or approximation.
  • The source and timestamp of calibration values.
  • How noise was applied to gates, measurements, idle periods or state preparation.
  • Mitigation method, correction matrix or model, training circuits and regularisation settings.
  • Unmitigated output alongside the corrected output.

Keep the noise-model generation code in the repository. A prose description alone is rarely sufficient to rebuild the same model.

4. Variational, optimisation or quantum machine learning run

For VQE, QAOA, hybrid optimisation and quantum machine learning, the classical loop is part of the quantum experiment. Record the dataset version, feature scaling, encoding method, train-validation split, parameter initialisation, optimiser, learning rate, batch size, number of iterations, stopping rule and callback behaviour.

Save parameters at every iteration when storage permits, not only the final score. Also record whether each objective evaluation used a fixed shot budget, a changing shot budget or an adaptive rule. For data-encoding assumptions, the practical comparison of angle, amplitude and basis embedding methods can help make the input pipeline explicit.

What to double-check

Versioning is complete, not merely present

A repository containing a requirements file is not automatically reproducible. Check that the file resolves to the versions used for the recorded run and that transitive dependencies are covered by a lockfile or an exported environment. Test the installation in a clean environment rather than assuming the developer's existing environment is representative.

Pin versions for the experiment you are reporting, but also document the supported range if the project is intended to remain reusable. When an SDK release changes circuit objects, transpilation defaults or result schemas, create a migration note instead of silently replacing the old environment.

The circuit being analysed is the circuit that ran

Many workflows construct one circuit and submit another after decomposition, mapping, optimisation or measurement insertion. Save both the logical circuit and the submitted representation. Record qubit layout, classical-bit mapping and any circuit transformations that affect how counts are interpreted.

Raw data is preserved before analysis

Store raw counts or samples separately from cleaned tables and plots. Document bit order, endianness, discarded outcomes, normalisation, post-selection rules, missing-value handling and any conversion from counts to probabilities or expectation values. A result that cannot be traced back to raw observations is difficult to audit.

Seeds are treated as evidence, not a guarantee

Seeds can make parts of a software workflow repeatable, but they do not make hardware deterministic. Confirm which components consume randomness, whether parallel execution changes ordering, and whether the backend honours a supplied seed. For hardware, report repeated runs or uncertainty where appropriate rather than presenting one seeded run as definitive.

Credentials and provenance are separated

Never commit access tokens, private keys or account secrets with experiment metadata. Replace them with a provider name, project label or redacted configuration summary. Provenance should explain where a job ran without granting access to the account that submitted it.

Common mistakes

  • Saving only the notebook: notebooks often hide execution order, unrecorded state and package versions. Export the notebook with outputs, but keep a script or reproducible command as the authoritative entry point.
  • Recording only the backend name: a backend label does not capture calibration, queue context, transpilation or provider configuration. Preserve the job metadata and submitted circuit.
  • Reporting only a final metric: retain raw results, intermediate objective values and the classical baseline used for comparison.
  • Changing shot counts between runs without saying so: shot count affects sampling uncertainty and can make apparently conflicting results incomparable.
  • Overwriting old environments: upgrade dependencies in a new branch or environment, then rerun the old workflow before comparing it with the upgraded one.
  • Confusing rerun with reproduction: a rerun tests whether the workflow still executes; a reproduction also checks whether the result and interpretation remain consistent under documented conditions.
  • Ignoring classical baselines: a quantum result needs the data, implementation and settings of the classical comparison to be reproducible too. See this framework for benchmarking quantum algorithms against classical methods.

When to revisit

Review the checklist at the start of every new experiment and before sharing results externally. Revisit an existing project when the SDK, provider plugin, simulator, Python version, operating system, transpiler, backend, noise model or analysis library changes. Also review it before seasonal planning cycles, publication deadlines, internal handovers and any migration between quantum cloud platforms.

For a practical update process, create a small reproducibility report for each release:

  1. Run the documented command in a clean environment.
  2. Compare the generated circuit, metadata schema and raw output with the previous recorded run.
  3. Explain any numerical difference, including changes in shots, calibration, randomness or software behaviour.
  4. Archive the old lockfile, job identifiers and outputs rather than replacing them.
  5. Update the README with the new tested versions and a date for the next review.

Copy-and-use project template

experiment_id:
repository_commit:
run_command:
python_version:
sdk_versions:
lockfile:
input_data_version_or_hash:
algorithm_and_parameters:
random_seeds:
backend_or_simulator:
transpilation_settings:
noise_model:
shots:
job_id:
execution_timestamp:
calibration_metadata:
raw_output_path:
post_processing_commit:
result_summary:
known_differences_or_limitations:

Complete this record before interpreting the result, then store it beside the raw outputs and code commit. That small habit turns a one-off quantum computing experiment into a research workflow another person can inspect, rerun and improve.

Related Topics

#reproducibility#quantum experiments#Qiskit#quantum SDKs#research workflow#version control
S

Smart Qubit Labs Editorial Team

Quantum Computing Editors

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.