sift_client.util.test_results ¶

Test Results Utilities.

This module provides utilities for working with test results.

Context Managers¶

ReportContext - Context manager for a new TestReport.
NewStep - Context manager to create a new step in a test report.

Example¶

client = SiftClient(api_key=api_key, grpc_url=grpc_url, rest_url=rest_url)
with ReportContext(client, name="Example Report") as rc:
    with rc.new_step(name="Setup") as step:
        controller_setup(step)
    with rc.new_step(name="Example Step", description=desc) as parent_step:
        cmd_interface.cmd("ec1", "rtv.cmd", 75.0)
        sleep(0.01)

        with parent_step.substep(name="Substep 1", description="Measure position") as substep:
            ec = "ec1"
            pos_channel = "rtv.pos"
            pos = tlm.read(ec, pos_channel)
            result = substep.measure(pos, name=f"{ec}.{pos_channel}", bounds=(min=74.9, max=75.1))
            return result # This is optional for other uses, but the step and its parents will be updated correctly i.e. failed if the measurement fails.

Manually Updating Underlyling Report¶

You can also manually update the underlying report or steps by accessing the context manager's attributes.

with ReportContext(client, name="Example Report") as rc:
    with rc.new_step(name="Example Step") as step:
        if !conditions:
            step.update({"status": TestStatus.SKIPPED})
        else:
            step.measure(name="Example Measurement", value=test_value, bounds={"min": -1, "max": 10})
    rc.report.update({"run_id": run_id})

For a larger class or script, consider creating the context in a setup method and passing it to the test functions.

def main(self):
    self.sift_client = SiftClient(api_key=api_key, grpc_url=grpc_url, rest_url=rest_url)
    with ReportContext(self.sift_client, name="Test Class", description="Test Class") as rc:
        setup(rc)
        test_one(rc)
        test_two(rc)
        teardown(rc)
    cleanup()

Pytest Fixtures¶

The report context and steps can also be accessed in pytest by importing the report_context and step fixtures.

How to use:¶

These fixtures are set to autouse and will automatically create a report and steps for each test function.
If you want each module(file) to be marked as a step w/ each test as a substep, import the module_substep fixture as well.
The report_context fixture requires a fixture sift_client returning an SiftClient instance to be passed in.

Example at top of your test file or in your conftest.py file:¶

import pytest

@pytest.fixture(scope="session")
def sift_client() -> SiftClient:
    grpc_url = os.getenv("SIFT_GRPC_URI", "localhost:50051")
    rest_url = os.getenv("SIFT_REST_URI", "localhost:8080")
    api_key = os.getenv("SIFT_API_KEY", "")

    client = SiftClient(api_key=api_key, grpc_url=grpc_url, rest_url=rest_url)

    return client

from sift_client.util.test_results import pytest_runtest_makereport, report_context, step, module_substep

Then in your test file:¶

# Because step was already imported and set autouse=True, this test will automatically get a step created for it.
def test_no_includes():
    assert condition, "Example failure"

# Passing the fixtures to the test function allows you to take measurements or create substeps.
def test_example(report_context, step):
    # This will add a measurement to the current step for this function
    step.measure(name="Example Measurement", value=test_string_value, bounds="expected_string_value")

    with report_context.new_step(name="Example Step") as substep:
        example_measurement = tlm.read(channel_name)
        substep.measure(name="Substep Measurement", value=example_measurement, bounds=(min=74.9, max=75.1))

MODULE	DESCRIPTION
`bounds`
`context_manager`
`pytest_util`

CLASS	DESCRIPTION
`NewStep`	Context manager to create a new step in a test report. See usage example in init.py.
`ReportContext`	Context manager for a new TestReport. See usage example in init.py.

FUNCTION	DESCRIPTION
`client_has_connection`	Check if the SiftClient has a connection to the Sift server.
`module_substep`	Create a step per module.
`module_substep_check_connection`	Create a step per module. Doesn't run if the client has no connection to the Sift server.
`pytest_runtest_makereport`	You should import this hook to capture any AssertionErrors that occur during the test. If not included, any assert failures in a test will not automatically fail the step.
`report_context`	Create a report context for the session.
`report_context_check_connection`	Create a report context for the session. Doesn't run if the client has no connection to the Sift server.
`step`	Create an outer step for the function.
`step_check_connection`	Create an outer step for the function. Doesn't run if the client has no connection to the Sift server.

NewStep ¶

NewStep(
    report_context: ReportContext,
    name: str,
    description: str | None = None,
)

Bases: AbstractContextManager

Context manager to create a new step in a test report. See usage example in init.py.

Initialize a new step context.

PARAMETER	DESCRIPTION
`report_context`	The report context to create the step in. TYPE: `ReportContext`
`name`	The name of the step. TYPE: `str`
`description`	The description of the step. TYPE: `str \| None` DEFAULT: `None`

METHOD	DESCRIPTION
`__enter__`	Enter the context manager to create a new step.
`__exit__`
`measure`	Measure a value and return the result.
`substep`	Alias to return a new step context manager from the current step. The ReportContext will manage nesting of steps.
`update_step_from_result`	Update the step based on its substeps and if there was an exception while executing the step.

ATTRIBUTE	DESCRIPTION
`client`	TYPE: `SiftClient`
`current_step`	TYPE: `TestStep \| None`
`report_context`	TYPE: `ReportContext`

client `instance-attribute` ¶

client: SiftClient = client

current_step `class-attribute` `instance-attribute` ¶

current_step: TestStep | None = create_step(
    name, description
)

report_context `instance-attribute` ¶

report_context: ReportContext = report_context

enter ¶

__enter__()

Enter the context manager to create a new step.

returns: The current step.

exit ¶

__exit__(exc, exc_value, tb)

measure ¶

measure(
    *,
    name: str,
    value: float | str | bool,
    bounds: dict[str, float]
    | NumericBounds
    | str
    | None = None,
    timestamp: datetime | None = None,
    unit: str | None = None,
) -> bool

Measure a value and return the result.

PARAMETER	DESCRIPTION
`name`	The name of the measurement. TYPE: `str`
`value`	The value of the measurement. TYPE: `float \| str \| bool`
`bounds`	[Optional] The bounds to compare the value to. TYPE: `dict[str, float] \| NumericBounds \| str \| None` DEFAULT: `None`
`timestamp`	[Optional] The timestamp of the measurement. Defaults to the current time. TYPE: `datetime \| None` DEFAULT: `None`
`unit`	[Optional] The unit of the measurement. TYPE: `str \| None` DEFAULT: `None`

returns: The result of the measurement.

substep ¶

substep(
    name: str, description: str | None = None
) -> NewStep

Alias to return a new step context manager from the current step. The ReportContext will manage nesting of steps.

update_step_from_result ¶

update_step_from_result(
    exc: type[Exception] | None,
    exc_value: Exception | None,
    tb: TracebackException | None,
)

Update the step based on its substeps and if there was an exception while executing the step.

PARAMETER	DESCRIPTION
`exc`	The class of Exception that was raised. TYPE: `type[Exception] \| None`
`exc_value`	The exception value. TYPE: `Exception \| None`
`tb`	The traceback object. TYPE: `TracebackException \| None`

ReportContext ¶

ReportContext(
    client: SiftClient,
    name: str,
    test_system_name: str | None = None,
    system_operator: str | None = None,
    test_case: str | None = None,
)

Bases: AbstractContextManager

Context manager for a new TestReport. See usage example in init.py.

Initialize a new report context.

PARAMETER	DESCRIPTION
`client`	The Sift client to use to create the report. TYPE: `SiftClient`
`name`	The name of the report. TYPE: `str`
`test_system_name`	The name of the test system. Will default to the hostname if not provided. TYPE: `str \| None` DEFAULT: `None`
`system_operator`	The operator of the test system. Will default to the current user if not provided. TYPE: `str \| None` DEFAULT: `None`
`test_case`	The name of the test case. Will default to the basename of the file containing the test if not provided. TYPE: `str \| None` DEFAULT: `None`

METHOD	DESCRIPTION
`__enter__`
`__exit__`
`create_step`	Create a new step in the report context.
`exit_step`	Exit a step and update the report context.
`get_next_step_path`	Get the next step path for the current depth.
`new_step`	Alias to return a new step context manager from this report context. Use create_step for actually creating a TestStep in the current context.
`report_measurement`	Report a failure to the report context.
`resolve_and_propagate_step_result`	Resolve the result of a step and propagate the result to the parent step if it failed.

ATTRIBUTE	DESCRIPTION
`any_failures`	TYPE: `bool`
`open_step_results`	TYPE: `dict[str, bool]`
`report`	TYPE: `TestReport`
`step_is_open`	TYPE: `bool`
`step_number_at_depth`	TYPE: `dict[int, int]`
`step_stack`	TYPE: `list[TestStep]`

any_failures `instance-attribute` ¶

any_failures: bool = False

open_step_results `instance-attribute` ¶

open_step_results: dict[str, bool] = {}

report `instance-attribute` ¶

report: TestReport = create(create)

step_is_open `instance-attribute` ¶

step_is_open: bool = False

step_number_at_depth `instance-attribute` ¶

step_number_at_depth: dict[int, int] = {}

step_stack `instance-attribute` ¶

step_stack: list[TestStep] = []

enter ¶

__enter__()

exit ¶

__exit__(exc_type, exc_value, traceback)

create_step ¶

create_step(
    name: str, description: str | None = None
) -> TestStep

Create a new step in the report context.

PARAMETER	DESCRIPTION
`name`	The name of the step. TYPE: `str`
`description`	The description of the step. TYPE: `str \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`TestStep`	The created step.

exit_step ¶

exit_step(step: TestStep)

Exit a step and update the report context.

get_next_step_path ¶

get_next_step_path() -> str

Get the next step path for the current depth.

new_step ¶

new_step(
    name: str, description: str | None = None
) -> NewStep

Alias to return a new step context manager from this report context. Use create_step for actually creating a TestStep in the current context.

report_measurement ¶

report_measurement(
    measurement: TestMeasurement, step: TestStep
)

Report a failure to the report context.

resolve_and_propagate_step_result ¶

resolve_and_propagate_step_result(
    step: TestStep, error_info: ErrorInfo | None = None
) -> bool

Resolve the result of a step and propagate the result to the parent step if it failed.

client_has_connection ¶

client_has_connection(sift_client)

Check if the SiftClient has a connection to the Sift server.

Can be used to skip tests that require a connection to the Sift server.

module_substep ¶

module_substep(
    report_context: ReportContext, request: FixtureRequest
) -> Generator[NewStep | None, None, None]

Create a step per module.

module_substep_check_connection ¶

module_substep_check_connection(
    report_context: ReportContext,
    client_has_connection: bool,
    request: FixtureRequest,
) -> Generator[NewStep | None, None, None]

Create a step per module. Doesn't run if the client has no connection to the Sift server.

pytest_runtest_makereport ¶

pytest_runtest_makereport(item: Item, call: CallInfo[Any])

You should import this hook to capture any AssertionErrors that occur during the test. If not included, any assert failures in a test will not automatically fail the step.

report_context ¶

report_context(
    sift_client: SiftClient, request: FixtureRequest
) -> Generator[ReportContext | None, None, None]

Create a report context for the session.

report_context_check_connection ¶

report_context_check_connection(
    sift_client: SiftClient,
    client_has_connection: bool,
    request: FixtureRequest,
) -> Generator[ReportContext | None, None, None]

Create a report context for the session. Doesn't run if the client has no connection to the Sift server.

step ¶

step(
    report_context: ReportContext, request: FixtureRequest
) -> Generator[NewStep | None, None, None]

Create an outer step for the function.

step_check_connection ¶

step_check_connection(
    report_context: ReportContext,
    client_has_connection: bool,
    request: FixtureRequest,
) -> Generator[NewStep | None, None, None]

Create an outer step for the function. Doesn't run if the client has no connection to the Sift server.

sift_client.util.test_results ¶

Context Managers¶

Example¶

Manually Updating Underlyling Report¶

Pytest Fixtures¶

How to use:¶

Example at top of your test file or in your conftest.py file:¶

Then in your test file:¶

NewStep ¶

client instance-attribute ¶

current_step class-attribute instance-attribute ¶

report_context instance-attribute ¶

__enter__ ¶

__exit__ ¶

measure ¶

substep ¶

update_step_from_result ¶

ReportContext ¶

any_failures instance-attribute ¶

open_step_results instance-attribute ¶

report instance-attribute ¶

step_is_open instance-attribute ¶

step_number_at_depth instance-attribute ¶

step_stack instance-attribute ¶

__enter__ ¶

__exit__ ¶

create_step ¶

exit_step ¶

get_next_step_path ¶

new_step ¶

report_measurement ¶

resolve_and_propagate_step_result ¶

client_has_connection ¶

module_substep ¶

module_substep_check_connection ¶

pytest_runtest_makereport ¶

report_context ¶

report_context_check_connection ¶

step ¶

step_check_connection ¶

client `instance-attribute` ¶

current_step `class-attribute` `instance-attribute` ¶

report_context `instance-attribute` ¶

enter ¶

exit ¶

any_failures `instance-attribute` ¶

open_step_results `instance-attribute` ¶

report `instance-attribute` ¶

step_is_open `instance-attribute` ¶

step_number_at_depth `instance-attribute` ¶

step_stack `instance-attribute` ¶

enter ¶

exit ¶