Scientist Pattern

Contributors🕹️Radu Cioienaru, Paul Atkins

When to use: Implement a control path and a candidate path side by side in Python, compare results, and always return the current production result.

Use this skill when a user asks to implement the Scientist pattern, run a control implementation beside a candidate implementation, compare their results, or migrate data-fetching/business logic safely while always returning the current production result.

Core behavior

The Scientist pattern implementation must:

1. Execute the control path first.
2. Execute the candidate path second.
3. Compare the two results with an experiment-specific comparator.
4. Always return the control result.
5. Log candidate failures and comparison failures without breaking the production path.

First inspect the repository

Before editing code:

1. Find the file and method the user named.
2. Identify the current control call and the requested candidate call.
3. Search for an existing reusable scientist implementation inside the repository that will be the consumer of the control and candidate calls:
4. a scientist directory
5. common.py
6. orchestrator.py
7. an existing run_experiment function
8. Inspect how logging is configured in the current project, including logger names, handler setup, and default log levels.
9. If reusable scientist files already exist, reuse them and follow their local conventions.
10. If they do not exist, add a new reusable scientist package under the most suitable shared/util location, such as:
11. backend_logic/utils/scientist
12. src/utils/scientist
13. app/common/scientist
14. another existing utilities/common package.

Use assets/python/scientist/common.py, assets/python/scientist/orchestrator.py, and assets/python/scientist/__init__.py for a fresh setup.

Keep reusable and scenario-specific code separate

Reusable code belongs in the shared scientist package:

• common.py
• orchestrator.py
• package exports
• generic formatting/logging helpers

Scenario-specific code should normally live next to the business logic being experimented on, or in that feature's module/package. Do not place a business-specific comparator or wrapper inside the reusable scientist core unless the repository already has a clear existing convention for scientist/comparisons or scientist/experiments.

Prefer creating separate files for comparators and comparator-specific helpers instead of adding comparator logic to existing business logic files. For example, add a dedicated file such as compare_<subject>.py, <subject>_comparator.py, or a feature-level comparators.py when there are multiple related experiments. Keep comparator normalization, value-comparison rules, and mismatch formatting in that comparator file unless there is already a shared helper designed for that exact purpose.

Implementation workflow

1. Add or reuse the reusable scientist core.
2. Create a scenario-specific comparator in its own file using the templates in assets/python/comparators.
3. Create a wrapper method using assets/python/experiment_wrapper.py.jinja.
4. Replace the original call site so it calls the wrapper instead of directly calling the control method.
5. Check whether the scientist core, comparator, wrapper, or tests introduce imports that are not already available in the project.
6. Add tests for:
7. run_experiment returning the control result
8. candidate exceptions being logged and swallowed
9. comparison exceptions being logged and swallowed
10. comparator match and mismatch cases
11. wrapper wiring of control, candidate, comparator, and label

Dependency handling

If the implementation or tests require packages that are not already declared by the repository, update the project's dependency manifest in the same change. Use the dependency file that the repository already uses, such as pyproject.toml, requirements.txt, Pipfile, poetry.lock, uv.lock, or a project-specific constraints file.

Only add real external dependencies. Do not add Python standard-library modules such as logging, json, dataclasses, or typing to dependency manifests.

When dependency metadata is changed:

1. Update the corresponding lock file when the repository uses one.
2. Run the existing dependency-management command for that ecosystem when practical.
3. Tell the user explicitly that dependencies were changed and that they need to reinstall or sync dependencies before running the updated code, for example with pip install -r requirements.txt, pip install -e ., poetry install, or uv sync, depending on the repository.

Comparator selection

If the user does not specify comparison behavior, assume:

• both sources must have the same number of items
• item order matters
• each control item must match the candidate item at the same index
• all item properties should match

Use this guide:

User asks for	Template
Same number of items, same order, each item matches fully	assets/python/comparators/ordered_all_properties.py.jinja
Same number of items, same order, only selected properties matter	assets/python/comparators/ordered_selected_properties.py.jinja
Same number of items, different order, match by stable key/id	assets/python/comparators/unordered_keyed_selected_properties.py.jinja
Same number of items, order irrelevant, no stable key	assets/python/comparators/unordered_multiset_selected_properties.py.jinja

Prefer a keyed unordered comparator when there is a stable unique identifier. Use the multiset comparator only when order is irrelevant and no stable unique key exists.

For fields that need special equality logic, such as decimal tolerances, boolean/string normalization, case-insensitive text, or date normalization, extend the selected-property templates with VALUE_COMPARERS rather than weakening the whole comparison.

Performance guardrails

Comparators should bound mismatch discovery and logging so badly divergent datasets do not create excessive work or logs:

• If 10 or more items are already missing, stop looking for more missing items.
• If 20 or more items are already mismatched during item comparison, stop looking for more mismatches.
• Log that the cap was reached so the user knows the output is a sample, not a complete mismatch report.

Comparator console output

When creating comparators, print to the console the items that are present in the control result but missing from the candidate result. Unless the user explicitly requests a different output format, print these missing control items as beautified JSON that is easy for a human to read, such as with indentation and stable key ordering where practical.

If the user asks for another format, respect that requested format instead of beautified JSON. Continue to apply the performance guardrails above so very large missing-item sets are sampled rather than printed in full.

Logging conventions

Use the dedicated scientist logger. Include the experiment label in every log message:

• [label] EXPERIMENT STARTING
• [label] CONTROL START
• [label] CONTROL FINISHED
• [label] CANDIDATE START
• [label] CANDIDATE FINISHED
• [label] COMPARISON STARTED
• [label] COMPARISON FINISHED
• [label] MATCH: ...
• [label] CONCLUSION: ...

On mismatches, log enough detail to diagnose the first meaningful differences. Use format_item_json for readable structured output, and ensure missing control items are visible in the console using the comparator console-output guidance above, but respect the performance guardrails above.

Match the repository's existing logging setup so scientist messages are visible in the user's console terminal. After inspecting project logging, configure the scientist logger at the appropriate level used for informational console output. When the project does not already configure this logger, explicitly set it to INFO, for example:

logging.getLogger("scientist").setLevel(logging.INFO)

Do not add duplicate handlers if the application already configures root or package log handlers. Follow the project's existing logger initialization location and style so scientist logs flow through the same console logging path as the rest of the application.

Label guidance

If the user gives a label, use it exactly. If no label is provided, create a stable label using the call site and experiment subject:

<module_or_file>.<function>:<experiment_subject>

Example:

orders_logic.list_orders:order_line_totals

Wrapper guidance

The wrapper should make the control and candidate callables explicit and return run_experiment(...). Keep wrapper code close to the call site unless the repository already has a dedicated experiment-wrapper convention.

Example shape:

def get_data_with_experiment() -> list[dict[str, Any]]:
    return run_experiment(
        control_fn=lambda: get_data_current(),
        candidate_fn=lambda: get_data_candidate(),
        compare_fn=compare_get_data,
        label="module.function:get_data",
    )

Validation

Run the repository's existing tests, lint, and type checks that are relevant to the changed files. Do not add new tooling just for scientist experiments. If no tests exist, add focused tests using the repository's existing test framework.

---

References

• Python scientist package
• Comparator templates
• Experiment wrapper template

---

Troubleshooting

Symptom	Likely cause	Fix
Candidate exception breaks the request	Candidate ran outside run_experiment	Call run_experiment; it must swallow candidate and comparison failures and return the control result
No scientist logs in the console	Logger level or handlers not wired	Set logging.getLogger("scientist") to INFO only when the app does not already configure it
Mismatch report is huge	Divergent datasets logged in full	Keep the caps: stop after 10 missing items and 20 mismatched items, and say the cap was reached
New import fails at runtime	Package missing from the repo manifest	Add it to the repo's existing dependency file and tell the user to reinstall