sift_client.util.test_results ¶

Test Results Utilities.

This module provides utilities for working with test results.

Context Managers¶

ReportContext - Context manager for a new TestReport.
NewStep - Context manager to create a new step in a test report.

Example¶

client = SiftClient(api_key=api_key, grpc_url=grpc_url, rest_url=rest_url)
with ReportContext(client, name="Example Report") as rc:
    with rc.new_step(name="Setup") as step:
        controller_setup(step)
    with rc.new_step(name="Example Step", description=desc) as parent_step:
        cmd_interface.cmd("ec1", "rtv.cmd", 75.0)
        sleep(0.01)

        with parent_step.substep(name="Substep 1", description="Measure position") as substep:
            ec = "ec1"
            pos_channel = "rtv.pos"
            pos = tlm.read(ec, pos_channel)
            result = substep.measure(pos, name=f"{ec}.{pos_channel}", bounds=(min=74.9, max=75.1))
            return result # This is optional for other uses, but the step and its parents will be updated correctly i.e. failed if the measurement fails.

Manually Updating Underlyling Report¶

You can also manually update the underlying report or steps by accessing the context manager's attributes.

with ReportContext(client, name="Example Report") as rc:
    with rc.new_step(name="Example Step") as step:
        if !conditions:
            step.update({"status": TestStatus.SKIPPED})
        else:
            step.measure(name="Example Measurement", value=test_value, bounds={"min": -1, "max": 10})
    rc.report.update({"run_id": run_id})

For a larger class or script, consider creating the context in a setup method and passing it to the test functions.

def main(self):
    self.sift_client = SiftClient(api_key=api_key, grpc_url=grpc_url, rest_url=rest_url)
    with ReportContext(self.sift_client, name="Test Class", description="Test Class") as rc:
        setup(rc)
        test_one(rc)
        test_two(rc)
        teardown(rc)
    cleanup()

Pytest Plugin¶

The pytest plugin lives at sift_client.pytest_plugin. Opt in from your conftest.py:

# conftest.py
pytest_plugins = ["sift_client.pytest_plugin"]

By default, every test in the session produces a Sift report: one TestReport per session, one step per test function (step), and one parent step per test file (module_substep). The plugin also registers a default sift_client fixture that reads SIFT_API_KEY, SIFT_GRPC_URI, and SIFT_REST_URI from the environment. Override it by defining your own sift_client fixture in your conftest.

Note: FedRAMP users: results are buffered to a temp file and uploaded by a subprocess at session end (no API calls during the run). Disable the buffer entirely with --sift-log-file=false for inline uploads.

Controlling which tests produce reports¶

The autouse fixtures fire for every test by default. To narrow that:

Set sift_autouse = false in pyproject.toml to flip the project default off, then opt tests back in below.
@pytest.mark.sift_include forces reporting on for a test, class, or module. @pytest.mark.sift_exclude forces it off. Closest marker wins. sift_exclude beats sift_include when both apply.
pytestmark at the class or module level inherits to every test in scope.
For a whole directory, apply the marker in bulk from that directory's conftest.py:

# tests/integration/conftest.py
from pathlib import Path

import pytest

_HERE = Path(__file__).parent


def pytest_collection_modifyitems(config, items):
    for item in items:
        try:
            item.path.relative_to(_HERE)
        except ValueError:
            continue
        item.add_marker(pytest.mark.sift_include)

Configuration¶

CLI options registered by the plugin:

--sift-offline: Run without contacting Sift. All create/update calls are written to the JSONL log file for later replay via import-test-result-log. No session-start ping is attempted.
--sift-disabled: Skip Sift entirely. Autouse fixtures yield stub objects; step.measure(...) still returns real pass/fail booleans by evaluating bounds locally, but nothing is sent to Sift and no log file is written. Also honored via the SIFT_DISABLED env var. Supersedes every other flag.
--sift-log-file: Path to write the JSONL log file. true (default) auto-creates a temp file. false or none disables logging. Any other value is treated as a file path.
--no-sift-git-metadata: Exclude git metadata (repo, branch, commit) from the test report. Included by default.

Each option has a matching ini key for per-project configuration under [tool.pytest.ini_options] in pyproject.toml (or [pytest] in pytest.ini). CLI flags override ini values. The sift_autouse ini key (bool, default true) sets the project-wide default for the gate described above. The default sift_client fixture reads sift_grpc_uri and sift_rest_uri as fallbacks when the corresponding env vars are unset (env vars win when both are set). SIFT_API_KEY is env-only. Load it from a .env file via the pytest-dotenv plugin or inject it via your CI secret manager.

[tool.pytest.ini_options]
sift_autouse = false
sift_offline = true
sift_git_metadata = false
sift_grpc_uri = "your-org.sift.example:443"
sift_rest_uri = "https://your-org.sift.example"

To disable the plugin for a single run: pytest -p no:sift_client.pytest_plugin.

MODULE	DESCRIPTION
`bounds`
`context_manager`

CLASS	DESCRIPTION
`NewStep`	Context manager to create a new step in a test report. See usage example in init.py.
`ReportContext`	Context manager for a new TestReport. See usage example in init.py.

NewStep ¶

NewStep(
    report_context: ReportContext,
    name: str,
    description: str | None = None,
    assertion_as_fail_not_error: bool = True,
    metadata: dict[str, str | float | bool] | None = None,
)

Bases: AbstractContextManager

Context manager to create a new step in a test report. See usage example in init.py.

Initialize a new step context.

PARAMETER	DESCRIPTION
`report_context`	The report context to create the step in. TYPE: `ReportContext`
`name`	The name of the step. TYPE: `str`
`description`	The description of the step. TYPE: `str \| None` DEFAULT: `None`
`assertion_as_fail_not_error`	Mark steps with assertion errors as failed instead of error+traceback (some users want assertions to work as simple failures especially when using pytest). TYPE: `bool` DEFAULT: `True`
`metadata`	[Optional] Structured key/value metadata to attach to the step. TYPE: `dict[str, str \| float \| bool] \| None` DEFAULT: `None`

METHOD	DESCRIPTION
`measure`	Measure a value and return the result.
`measure_all`	Ensure that all values in a list are within bounds and return the result. Records measurements for all values outside the bounds.
`measure_avg`	Calculate the average of a list of values, measure the average against given bounds, and return the result.
`report_outcome`	Report an outcome from some action or measurement. Creates a substep that is pass/fail with the optional reason as the description.
`substep`	Alias to return a new step context manager from the current step. The ReportContext will manage nesting of steps.
`update_step_from_result`	Update the step based on its substeps and if there was an exception while executing the step.

ATTRIBUTE	DESCRIPTION
`assertion_as_fail_not_error`	TYPE: `bool`
`client`	TYPE: `SiftClient`
`current_step`	TYPE: `TestStep \| None`
`report_context`	TYPE: `ReportContext`

assertion_as_fail_not_error `class-attribute` `instance-attribute` ¶

assertion_as_fail_not_error: bool = (
    assertion_as_fail_not_error
)

client `instance-attribute` ¶

client: SiftClient = client

current_step `class-attribute` `instance-attribute` ¶

current_step: TestStep | None = create_step(
    name, description, metadata=metadata
)

report_context `instance-attribute` ¶

report_context: ReportContext = report_context

measure ¶

measure(
    *,
    name: str,
    value: float | str | bool | int,
    bounds: dict[str, float]
    | NumericBounds
    | str
    | None = None,
    timestamp: datetime | None = None,
    unit: str | None = None,
    description: str | None = None,
    metadata: dict[str, str | float | bool] | None = None,
    channel_names: list[str] | list[Channel] | None = None,
) -> bool

Measure a value and return the result.

PARAMETER	DESCRIPTION
`name`	The name of the measurement. TYPE: `str`
`value`	The value of the measurement. TYPE: `float \| str \| bool \| int`
`bounds`	[Optional] The bounds to compare the value to. TYPE: `dict[str, float] \| NumericBounds \| str \| None` DEFAULT: `None`
`timestamp`	[Optional] The timestamp of the measurement. Defaults to the current time. TYPE: `datetime \| None` DEFAULT: `None`
`unit`	[Optional] The unit of the measurement. TYPE: `str \| None` DEFAULT: `None`
`description`	[Optional] Notes about the measurement. Server caps at 2000 characters; longer strings are truncated with a warning. TYPE: `str \| None` DEFAULT: `None`
`metadata`	[Optional] Structured key/value metadata to attach to the measurement. For metadata shared across measurements, prefer the `metadata` attribute of the enclosing `TestStep` or `TestReport`. TYPE: `dict[str, str \| float \| bool] \| None` DEFAULT: `None`
`channel_names`	[Optional] Sift channel names or `Channel` instances this measurement is associated with. Enables cross-plotting in Explore using the report's associated Run. TYPE: `list[str] \| list[Channel] \| None` DEFAULT: `None`

returns: The result of the measurement.

measure_all ¶

measure_all(
    *,
    name: str,
    values: list[float | int] | NDArray[float64] | Series,
    bounds: dict[str, float] | NumericBounds,
    timestamp: datetime | None = None,
    unit: str | None = None,
    description: str | None = None,
    metadata: dict[str, str | float | bool] | None = None,
    channel_names: list[str] | list[Channel] | None = None,
) -> bool

Ensure that all values in a list are within bounds and return the result. Records measurements for all values outside the bounds.

Note: Measurements will only be recorded for values outside the bounds. To record measurements for all values, just call measure for each value.

PARAMETER	DESCRIPTION
`name`	The name of the measurement. TYPE: `str`
`values`	The list of values to measure the average of. TYPE: `list[float \| int] \| NDArray[float64] \| Series`
`bounds`	The bounds to compare the value to. TYPE: `dict[str, float] \| NumericBounds`
`timestamp`	[Optional] The timestamp of the measurement. Defaults to the current time. TYPE: `datetime \| None` DEFAULT: `None`
`unit`	[Optional] The unit of the measurement. TYPE: `str \| None` DEFAULT: `None`
`description`	[Optional] Notes attached to each out-of-bounds measurement. Server caps at 2000 characters; longer strings are truncated with a warning. TYPE: `str \| None` DEFAULT: `None`
`metadata`	[Optional] Structured key/value metadata for each out-of-bounds measurement. TYPE: `dict[str, str \| float \| bool] \| None` DEFAULT: `None`
`channel_names`	[Optional] Sift channel names or `Channel` instances to associate with each out-of-bounds measurement. TYPE: `list[str] \| list[Channel] \| None` DEFAULT: `None`

returns: The true if all values are within the bounds, false otherwise.

measure_avg ¶

measure_avg(
    *,
    name: str,
    values: list[float | int] | NDArray[float64] | Series,
    bounds: dict[str, float] | NumericBounds,
    timestamp: datetime | None = None,
    unit: str | None = None,
    description: str | None = None,
    metadata: dict[str, str | float | bool] | None = None,
    channel_names: list[str] | list[Channel] | None = None,
) -> bool

Calculate the average of a list of values, measure the average against given bounds, and return the result.

PARAMETER	DESCRIPTION
`name`	The name of the measurement. TYPE: `str`
`values`	The list of values to measure the average of. TYPE: `list[float \| int] \| NDArray[float64] \| Series`
`bounds`	The bounds to compare the value to. TYPE: `dict[str, float] \| NumericBounds`
`timestamp`	[Optional] The timestamp of the measurement. Defaults to the current time. TYPE: `datetime \| None` DEFAULT: `None`
`unit`	[Optional] The unit of the measurement. TYPE: `str \| None` DEFAULT: `None`
`description`	[Optional] Notes about the measurement. Server caps at 2000 characters; longer strings are truncated with a warning. TYPE: `str \| None` DEFAULT: `None`
`metadata`	[Optional] Structured key/value metadata to attach to the measurement. TYPE: `dict[str, str \| float \| bool] \| None` DEFAULT: `None`
`channel_names`	[Optional] Sift channel names or `Channel` instances this measurement is associated with. TYPE: `list[str] \| list[Channel] \| None` DEFAULT: `None`

returns: The true if the average of the values is within the bounds, false otherwise.

report_outcome ¶

report_outcome(
    name: str, result: bool, reason: str | None = None
) -> bool

Report an outcome from some action or measurement. Creates a substep that is pass/fail with the optional reason as the description.

PARAMETER	DESCRIPTION
`name`	The name of the substep. TYPE: `str`
`result`	True if the action or measurement passed, False otherwise. TYPE: `bool`
`reason`	[Optional] The context to include in the description of the substep. TYPE: `str \| None` DEFAULT: `None`

returns: The given result so the function can be used in line.

substep ¶

substep(
    name: str,
    description: str | None = None,
    metadata: dict[str, str | float | bool] | None = None,
) -> NewStep

Alias to return a new step context manager from the current step. The ReportContext will manage nesting of steps.

update_step_from_result ¶

update_step_from_result(
    exc: type[Exception] | None,
    exc_value: Exception | None,
    tb: TracebackException | None,
) -> bool

Update the step based on its substeps and if there was an exception while executing the step.

PARAMETER	DESCRIPTION
`exc`	The class of Exception that was raised. TYPE: `type[Exception] \| None`
`exc_value`	The exception value. TYPE: `Exception \| None`
`tb`	The traceback object. TYPE: `TracebackException \| None`

returns: The false if step failed or errored, true otherwise.

ReportContext ¶

ReportContext(
    client: SiftClient,
    name: str,
    test_system_name: str | None = None,
    system_operator: str | None = None,
    test_case: str | None = None,
    log_file: str | Path | bool | None = None,
    include_git_metadata: bool = False,
    offline: bool = False,
)

Bases: AbstractContextManager

Context manager for a new TestReport. See usage example in init.py.

Initialize a new report context.

PARAMETER	DESCRIPTION
`client`	The Sift client to use to create the report. TYPE: `SiftClient`
`name`	The name of the report. TYPE: `str`
`test_system_name`	The name of the test system. Will default to the hostname if not provided. TYPE: `str \| None` DEFAULT: `None`
`system_operator`	The operator of the test system. Will default to the current user if not provided. TYPE: `str \| None` DEFAULT: `None`
`test_case`	The name of the test case. Will default to the basename of the file containing the test if not provided. TYPE: `str \| None` DEFAULT: `None`
`log_file`	If True, create a temp log file. If a path, use that path. All create/update operations will be logged to this file. TYPE: `str \| Path \| bool \| None` DEFAULT: `None`
`include_git_metadata`	If True, include git metadata in the report. TYPE: `bool` DEFAULT: `False`
`offline`	If True, do not spawn the import-test-result-log replay subprocess. A log file is required and will be created as a temp file if not provided. TYPE: `bool` DEFAULT: `False`

METHOD	DESCRIPTION
`create_step`	Create a new step in the report context.
`exit_step`	Exit a step and update the report context.
`get_next_step_path`	Get the next step path for the current depth.
`new_step`	Alias to return a new step context manager from this report context. Use create_step for actually creating a TestStep in the current context.
`record_step_outcome`	Report a failure to the report context.
`resolve_and_propagate_step_result`	Resolve the result of a step and propagate the result to the parent step if it failed.

ATTRIBUTE	DESCRIPTION
`any_failures`	TYPE: `bool`
`client`	TYPE: `SiftClient`
`log_file`	TYPE: `Path \| None`
`offline`
`open_step_results`	TYPE: `dict[str, bool]`
`report`	TYPE: `TestReport`
`step_is_open`	TYPE: `bool`
`step_number_at_depth`	TYPE: `dict[int, int]`
`step_stack`	TYPE: `list[TestStep]`

any_failures `instance-attribute` ¶

any_failures: bool = False

client `instance-attribute` ¶

client: SiftClient = client

log_file `instance-attribute` ¶

log_file: Path | None

offline `instance-attribute` ¶

offline = offline

open_step_results `instance-attribute` ¶

open_step_results: dict[str, bool] = {}

report `instance-attribute` ¶

report: TestReport = create(create, log_file=log_file)

step_is_open `instance-attribute` ¶

step_is_open: bool = False

step_number_at_depth `instance-attribute` ¶

step_number_at_depth: dict[int, int] = {}

step_stack `instance-attribute` ¶

step_stack: list[TestStep] = []

create_step ¶

create_step(
    name: str,
    description: str | None = None,
    metadata: dict[str, str | float | bool] | None = None,
) -> TestStep

Create a new step in the report context.

PARAMETER	DESCRIPTION
`name`	The name of the step. TYPE: `str`
`description`	The description of the step. TYPE: `str \| None` DEFAULT: `None`
`metadata`	[Optional] Structured key/value metadata to attach to the step. For metadata shared across every step in a report, prefer the `metadata` attribute of the enclosing `TestReport`. TYPE: `dict[str, str \| float \| bool] \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`TestStep`	The created step.

exit_step ¶

exit_step(step: TestStep)

Exit a step and update the report context.

get_next_step_path ¶

get_next_step_path() -> str

Get the next step path for the current depth.

new_step ¶

new_step(
    name: str,
    description: str | None = None,
    assertion_as_fail_not_error: bool = True,
    metadata: dict[str, str | float | bool] | None = None,
) -> NewStep

Alias to return a new step context manager from this report context. Use create_step for actually creating a TestStep in the current context.

record_step_outcome ¶

record_step_outcome(outcome: bool, step: TestStep)

Report a failure to the report context.

resolve_and_propagate_step_result ¶

resolve_and_propagate_step_result(
    step: TestStep, error_info: ErrorInfo | None = None
) -> bool

Resolve the result of a step and propagate the result to the parent step if it failed.

sift_client.util.test_results ¶

Context Managers¶

Example¶

Manually Updating Underlyling Report¶

Pytest Plugin¶

Controlling which tests produce reports¶

Configuration¶

NewStep ¶

assertion_as_fail_not_error class-attribute instance-attribute ¶

client instance-attribute ¶

current_step class-attribute instance-attribute ¶

report_context instance-attribute ¶

measure ¶

measure_all ¶

measure_avg ¶

report_outcome ¶

substep ¶

update_step_from_result ¶

ReportContext ¶

any_failures instance-attribute ¶

client instance-attribute ¶

log_file instance-attribute ¶

offline instance-attribute ¶

open_step_results instance-attribute ¶

report instance-attribute ¶

step_is_open instance-attribute ¶

step_number_at_depth instance-attribute ¶

step_stack instance-attribute ¶

create_step ¶

exit_step ¶

get_next_step_path ¶

new_step ¶

record_step_outcome ¶

resolve_and_propagate_step_result ¶

assertion_as_fail_not_error `class-attribute` `instance-attribute` ¶

client `instance-attribute` ¶

current_step `class-attribute` `instance-attribute` ¶

report_context `instance-attribute` ¶

any_failures `instance-attribute` ¶

client `instance-attribute` ¶

log_file `instance-attribute` ¶

offline `instance-attribute` ¶

open_step_results `instance-attribute` ¶

report `instance-attribute` ¶

step_is_open `instance-attribute` ¶

step_number_at_depth `instance-attribute` ¶

step_stack `instance-attribute` ¶