Skip to content

Change Log

All notable changes to this project will be documented in this file.

This project adheres to Semantic Versioning.

[v0.23.1] - October 8, 2026

  • Test-results steps updated out of order now have their statuses corrected before the report is finalized, with a SiftReportLifecycleWarning. (#821)

[v0.23.0] - October 1, 2026

  • Fix issue where a test-results log containing an update that named no fields could never finish importing. Re-run import-test-result-log to finish a log written by an earlier version. (#809)
  • A key that is not a field of a create or update model now warns with SiftIgnoredInputWarning, and suggests a near match. (#809)
  • The pytest plugin can archive a report as soon as it creates it, so the run stays out of the default Test Results views. Set archive_on_create under [tool.sift.pytest.report], or the matching CLI flag, ini key, or environment variable. Off by default. See Archiving a run at creation. (#808)
  • Data reads cap concurrent channel requests at 500, so a query that spans many channels no longer trips backend rate limits. (#788)

[v0.22.1] - September 29, 2026

  • Fix streaming ingestion processes crashing with SIGSEGV or SIGABRT at interpreter exit. The sift-stream-bindings tokio runtime was never shut down, so a runtime thread that finished work while Python was finalizing re-entered the interpreter and killed the process after the program had completed. The bindings now stop the runtime from an atexit hook and expose shutdown(timeout=5.0) for callers that exit with os._exit(). If runtime threads outlive the timeout, shutdown() issues a RuntimeWarning. Requires sift-stream-bindings 0.5.2. (#804)

[v0.22.0] - September 25, 2026

  • Added client.user_defined_functions for reusable expressions.
  • Added client.annotations for data reviews, phases, and review history.
  • Added client.campaigns for grouping report results across runs.
  • Added client.webhooks, and RuleAction.webhook to call one when a rule is violated.
  • RuleCreate.action and RuleUpdate.action accept a Webhook directly.
  • RuleCreate.asset_ids and RuleUpdate.asset_ids now accept Asset objects as well as IDs.
  • WebhookTest.text decodes the response body, and Webhook.created_by/modified_by resolve users.

What's New

MCAP imports

client.data_import now supports MCAP (.mcap) files with ROS 2 (ros2msg/cdr) topics.

job = client.data_import.import_from_path("recording.mcap", asset=my_asset)

Importing without a config ingests every supported channel. Topics that cannot be decoded fail the import unless McapParseErrorPolicy.IGNORE_ERROR is set.

detect_config reads the file's channels locally without decoding messages, returning an McapImportConfig whose data holds one entry per field. Edit it to select, rename, or retype channels before importing.

config = client.data_import.detect_config("recording.mcap")
config.complex_types_import_mode = McapComplexTypesImportMode.STRING

Variable-cardinality fields (dynamic and bounded arrays) are typed BYTES. As with Parquet, complex_types_import_mode on the config decides what each becomes: Arrow IPC bytes, a JSON string under <name>.json, both (the default), or neither.

Reading a file locally needs the new mcap extra (pip install sift-stack-py[mcap]), so both detect_config and importing without a config require it.

Credentials from sift-cli profiles

SiftClient now reads the same sift.toml profiles that sift-cli --profile uses. An environment that you configure once for the CLI works from Python with no arguments.

client = SiftClient()                      # the CLI's default profile
client = SiftClient(profile="staging")     # a named profile
client = SiftClient.from_profile("staging")

Precedence, highest first:

  1. Arguments.
  2. A profile that you name in code.
  3. SIFT_API_KEY, SIFT_GRPC_URI, SIFT_REST_URI, and SIFT_APP_URL.
  4. The profile that SIFT_PROFILE names.
  5. The config file's default profile.

client.credential_sources reports the layer that supplied each value. See Credentials and profiles.

The pytest plugin gains --sift-profile, the sift_profile ini key, and SIFT_PROFILE. In the plugin, the existing settings surfaces outrank the profile. The profile supplies only the values that they leave unset, so a key that CI injects stays in effect.

An http:// URL now connects without TLS instead of failing, because transport security follows the scheme of the gRPC URL. An https:// URL and a bare host name behave as before.

In the config file, the canonical spelling of the API key is api_key, which matches the rest of the Sift API. The older apikey spelling is still valid, so you do not need to migrate. If a profile holds both keys, the client uses api_key.

[v0.21.0] - September 4, 2026

What's New

List and get data imports

New in client.data_import: list_ and get (plus find), and a run.data_imports property.

# Imports for a run, e.g. to check on a batch
imports = client.data_import.list_(runs=[run])
imports = run.data_imports  # same query, keyed off the run

# One import by ID
data_import = client.data_import.get("data-import-id")

Each DataImport carries the import's status, error_message, and warning_messages. job.get_data_import() returns it for the Job that import_from_path hands back.

Import enum channels from any file format

Data columns on all import configs now accept enum_types, a mapping of enum name to key.

CsvDataColumn(
    column=2,
    name="state",
    data_type=ChannelDataType.ENUM,
    enum_types={"IDLE": 0, "ARMED": 1},
)

Setting enum_types on a non-enum data type raises a validation error. Multi-channel Parquet single-channel-per-row imports don't support enums, since channel configs there are derived per row.

Bugfixes

  • Fix ingesting ChannelDataType.BYTES channels. Registering a flow with a bytes channel raised ValueError: Unknown data type: bytes, and sending a value on one raised ValueError: Invalid data type: bytes. Both paths now map to the bytes type in sift-stream-bindings. Values may be bytes, bytearray, memoryview, or an iterable of ints. Requires sift-stream-bindings 0.5.1.
  • Fix channels.get_data printing a UserWarning: Discarding nonzero nanoseconds in conversion once per channel on each run-scoped fetch. The channel data cache stored segment bounds as datetime, which truncated the nanosecond timestamps the data already carries; they are now pd.Timestamp. start_time and end_time on get_data and get_data_as_arrow also accept a pd.Timestamp now, and its nanoseconds survive to the wire, where the service filters on them.

[v0.20.0] - August 25, 2026

What's New

Interrupted test-result uploads resume

import-test-result-log <log> now finishes an upload that was cut short instead of creating a second report. Every upload records what reached the server in a tracking sidecar beside the log (<log>.jsonl.tracking), so a re-run reuses the report the earlier attempt created and sends only the entries that are missing. A log that is already fully uploaded is a no-op.

# Resume, or start fresh if there is nothing to resume
import-test-result-log ./offline-runs/a1b2c3/a1b2c3.jsonl

# Abandon the partial upload and create a new report
import-test-result-log --new-report ./offline-runs/a1b2c3/a1b2c3.jsonl

--new-report moves the existing sidecar to <log>.jsonl.tracking.bak rather than overwriting it, so the abandoned report's ID stays recoverable. Resuming requires that report to still exist; if it was deleted, or the sidecar came from another environment, the command fails and names --new-report as the way forward.

client.test_results.import_log_file(...) takes the matching new_report argument. import-test-result-log no longer accepts --incremental: resuming is the default now, and the plugin's background worker moved to its own entry point. (#745)

New generated protos

  • Added the public sift.automation.v1 package and refreshed the generated protobufs across common, data, data imports, external sync, families, filters, ingestion configs, metadata, notifications, panel configurations, remote files, reports, rule evaluation, runs, and saved searches. (#753)

Bugfixes

  • Fix assets.archive raising AttributeError by reading the correct response field, archived_run_ids. (#747)
  • Fix streaming ingestion failing to start when an ingestion config's flow schema exceeded the streaming client's 4 MiB decode limit. sift-stream-bindings 0.5.0 decodes up to 50 MiB. (#751)
  • Fix the pytest plugin's upload worker failing to spawn when the venv's bin/ was not on PATH, such as under sudo or python -m pytest against a non-activated venv. It now runs through the same interpreter as pytest. (#748)
  • Fix client.runs.list(is_stopped=False) returning only stopped runs. False now applies no filter; is_stopped=True is unchanged. (#728)

Full Changelog

[v0.19.1] - July 27, 2026

What's New

page_size on get_data

client.channels.get_data(...) and get_data_as_arrow(...) now take a page_size, the number of points fetched per request. It defaults to 10,000; the server caps it at 1,000,000 and coerces higher values down. Raise it to cut round-trips on large windows.

data = client.channels.get_data(
    channels=channels,
    run=run,
    page_size=1_000_000,
)

New generated protos

  • Added the public sift.canvas.v1 package. (#683)
  • Refreshed the generated protobufs, adding sift.families.v1 and updating common, data imports, exports, reports, rules, rule evaluation, and saved searches. (#689)

Bugfixes

  • Fix get_data failing to build its request: start_time and end_time were passed as datetime objects where protobuf timestamps were required. (#688)
  • Upload a teardown failure on the final test of a pytest session. The session-scoped report_context fixture tore down before pytest reported the outcome, so a run whose last test failed in teardown was recorded as fully passing. Late failures now also correct already-closed ancestor steps from passed to failed. (#691)
  • Record combined @pytest.mark.parametrize("a,b,c", ...) axes as one step instead of one per argument name, and honor ids=. Step names and tree depth change, so reports from before and after this release will not group together. (#691)

Full Changelog

[v0.19.0] - July 10, 2026

What's New

Faster get_data pagination

Up to a ~80x speedup for some get_data calls. get_data now also displays a progress bar while paging through results.

Shared on-disk cache (opt-out, on by default)

client.channels.get_data(...) now caches the channel windows it returns to disk by default. Subsequent calls covering the same channel/time range — including from a fresh process — read straight out of the cache instead of going to the wire. This also bounds memory: nothing is held in process after the call returns.

The cache lives on the SiftClient as a single shared store: every cache-aware resource writes to one global byte budget at one path, with one LRU policy. The default location is <tempfile.gettempdir()>/sift-data-cache, capped at 4 GiB with LRU eviction. If the default path can't be opened (read-only filesystem, restricted container, etc.), the client logs a warning and continues with caching disabled — get_data still works, it just always goes to the wire.

ignore_cache=True on client.channels.get_data(...) now skips writing into the cache as well as reading from it. Previously a "non-caching" workload still appended to the shared cache on every call.

Configuration lives on the new client.cache namespace — knobs are global because the store is shared:

# Opt out — no data persisted to disk; every get_data call goes to the wire.
client.cache.disable()

# Reconfigure the location or byte cap.
client.cache.enable(path="/data/sift-cache", max_bytes=2 * 1024 ** 3)

# Remove a stale or corrupted cache directory.
client.cache.clear()                   # default tmp path
client.cache.clear("/data/sift-cache") # custom path

enable is also the way to turn the cache back on after a prior disable call.

The cache is powered by diskcache (pure-Python, SQLite-backed) with LRU eviction.

Report templates

Added a full report template API, available as a nested resource of reports at client.reports.templates (and asynchronously via client.async_.reports.templates). Report templates can now be created, fetched, listed, updated, archived, and unarchived directly from sift_client:

from sift_client.sift_types import ReportTemplateCreate

template = client.reports.templates.create(
    ReportTemplateCreate(
        name="Motor Checkout",
        client_key="motor-checkout",
        rule_ids=rules,  # Rule objects or IDs; or rule_client_keys=[...]
        tags=["motor"],
    )
)

# Run the template against a run
job = template.create_report(run=run)

Templates can be fetched by ID or client key, and rules can be attached by rule ID or rule client key (rule_ids also accepts Rule objects, and tags accepts Tag objects or names). Updating tags, rule_ids, or rule_client_keys replaces the full list on the template.

Breaking change: client.reports.create_from_template now takes report_template (a ReportTemplate or ID string) and run (a Run or ID string) instead of report_template_id and run_id, matching the other create_from_* methods.

ULog imports

Added PX4 ULog (.ulg) as a supported data import format. ULog files are self-describing, so client.data_import.import_from_path("flight.ulg", asset=asset) imports every channel in the file with no column mapping. Detection runs client-side and requires the ulog extra (pip install sift-stack-py[ulog]).

Call detect_config to inspect and patch the import before uploading, the same as the other formats:

config = client.data_import.detect_config("flight.ulg")

# Import only the accelerometer channels
config.data = [d for d in config.data if d.message_name == "sensor_accel"]

# Anchor the timeline at an explicit log start time; this takes precedence
# over the log's GPS fix and is required for logs without one
config.relative_start_time = datetime(2026, 1, 1, tzinfo=timezone.utc)

# Import firmware version and a parameter as run metadata
config.info_keys = ["ver_sw"]
config.param_keys = ["BAT1_CAPACITY"]

client.data_import.import_from_path("flight.ulg", asset=asset, config=config)

Resource and principal attributes (ABAC)

Added a public API for attribute-based access control (ABAC) attributes under client.access_control. Resource attributes describe the Sift objects an access decision applies to, such as assets, channels, and runs. Principal attributes describe the users or groups an access decision applies to. Async APIs are available under client.async_.access_control.

An attribute key is the entry point. Create or fetch a key, define its enum values, then assign a value to a set of resources. Resource assignments accept supported resource objects, or a typed ResourceAttributeEntity when you only have an ID:

from sift_client.sift_types import ResourceAttributeEntity, ResourceAttributeValueType

key = client.access_control.resource_attributes.keys.get_or_create(
    "licenses",
    ResourceAttributeValueType.SET_OF_ENUM,
)
licenses = key.get_or_create_enum_values(["LICENSE_A", "LICENSE_B"])
key.assign_to([ResourceAttributeEntity.for_channel("channel-id")], value=licenses)

Principal attributes accept User objects, typed PrincipalRef references, or user email addresses (resolved to user IDs automatically). Use PrincipalRef.user_group(...) for user groups:

from sift_client.sift_types import PrincipalAttributeValueType

key = client.access_control.principal_attributes.keys.get_or_create(
    "licenses",
    PrincipalAttributeValueType.SET_OF_ENUM,
)
licenses = key.get_or_create_enum_values(["LICENSE_A"])
key.assign_to(["user@example.com"], value=licenses)

Keys, enum values, and assignments are managed through the nested keys, enum_values, and assignments APIs on each side (for example client.access_control.resource_attributes.assignments.list_()), each supporting the relevant create, get, list, update, and archive operations. For SET_OF_ENUM keys, an assignment replaces the full value set on each target, and each enum value in the set is returned as its own assignment.

Users

Added a users API at client.users (async: client.async_.users). A user's name is their login name, typically their email address:

user = client.users.find(name="alice@example.com")
users = client.users.list_(name_contains="@example.com")
ids = client.users.resolve_ids(["alice@example.com", "bob@example.com"])

Bugfixes

  • Fix get_data pagination returning incorrect results in some windows. (#678)
  • Strip docstrings captured by the pytest plugin so they no longer leak into report step names and descriptions. (#658)

Full Changelog

[v0.18.0] - June 19, 2026

This release is a major overhaul of the pytest plugin. One import enables it, with a hierarchical report tree, offline and disabled run modes, pass/fail and abort handling that rolls up correctly, a terminal report summary, project-level configuration, and an audit log for diagnosing field failures.

What's New

Single-import plugin

The plugin is now sift_client.pytest_plugin, wired in with one line:

pytest_plugins = ["sift_client.pytest_plugin"]

This replaces the previous setup that required conftest.py boilerplate and a from sift_client.util.test_results import * activation. A default sift_client fixture reads SIFT_* env vars with sift_grpc_uri / sift_rest_uri ini fallback, so per-project config lives in pyproject.toml. sift_include / sift_exclude markers and the sift_autouse ini key scope which tests produce Sift reports, and autouse fixtures are gated per test so mixed unit and integration suites share one pytest invocation without unit tests paying a backend round-trip.

Hierarchical report tree

Python packages, test modules, test classes (including nested), and @pytest.mark.parametrize axes each open a nested parent step automatically, so the report mirrors the test layout. Parents are keyed by identity rather than held on a shared stack, so the tree is order-independent: it works with pytest-randomly, pytest-ordering, and fixture-scope reordering. Each parent closes as soon as the last test in its subtree finishes, so containers resolve progressively in incrementally uploaded reports instead of all at session end.

Parametrize placement is scope-aware: function-scoped params nest under the test, while a class, module, or session-scoped parametrized fixture sits at that level, matching where pytest re-runs the work. Four ini keys gate the layers individually: sift_package_step, sift_module_step, sift_class_step, and sift_parametrize_nesting (all default true).

Offline and disabled modes

Two named connection modes replace the previous check-connection toggle:

  • --sift-offline skips the connectivity ping and routes create/update calls to a JSONL log for later replay via import-test-result-log, tolerating missing SIFT_* env vars.
  • --sift-disabled (also SIFT_DISABLED=1) contacts nothing and writes no log, but step.measure(...) still evaluates bounds and returns a real pass/fail result. No credentials required.

is_simulated is exposed on TestReport, TestStep, TestMeasurement, and ReportContext so callers can branch on provenance. In online mode a failed ping now aborts the session once via pytest.exit, before any test body runs and with the full error untruncated.

Pass/fail reporting

Assertion failures record the concise assertion message on error_info (the step stays FAILED with no traceback) so the detail surfaces in the report UI. step.pytest_fail_if_step_failed() fails the pytest test when the step or any descendant failed, closing the gap where a failed substep or report_outcome() marked a step FAILED in the report but left pytest green. The failure message names each out-of-bounds measurement by name, value, and bounds, plus any failed substep.

Status now rolls up correctly on a session abort. Parent steps (class, module, package) resolve from their descendants after pytest's fixture teardown, so a Ctrl-C or KeyboardInterrupt records the report and any open parents as ABORTED instead of letting them close PASSED before the interrupted leaf resolved. sift_client.pytest_plugin.abort(reason) triggers the same path from test code; pytest.exit() and ordinary test failures roll up as FAILED, and an open substep interrupted by pytest.exit() resolves ABORTED.

Terminal summary

A session-start header shows the SDK version and active mode (e.g. Sift: sift-stack-py 0.18.0 — online mode). An end-of-run Sift report panel reports status, step and measurement counts (color-coded), test case, system, log file, and a clickable report link (online) or upload command (offline). New --sift-open-report / sift_open_report opens the report in a browser at session end on interactive terminals. sift_app_url / SIFT_APP_URL sets the web-app origin for on-prem and custom deployments; SiftClient.app_url exposes the resolved origin to non-plugin consumers.

Configurable report content

Report fields are configurable from [tool.sift.pytest.report] in pyproject.toml (name, test_case, test_system_name, system_operator, serial_number, part_number, and a metadata table) or from SIFT_REPORT_* env vars for per-run values, with env winning over TOML. name and test_case accept placeholders including {target}, {command}, {timestamp}, {count}, {git_repo}, {git_branch}, and {git_commit}. When unset, name defaults to {target} {timestamp} and test_case to {target}, where target is derived from the collected tests so it stays stable across flag order and filters. The full invocation is always recorded under the pytest_command metadata key.

Unknown SIFT_* env vars and [tool.sift.pytest.*] keys produce a warning with a closest-match suggestion, so typos surface immediately.

Output directory and audit logging

--sift-output-dir / sift_output_dir sets the directory for a run's artifacts; each run nests a random subfolder and defaults to a temp dir. Two outputs land there and are on by default: the JSONL test-result log (replayable via import-test-result-log) and a DEBUG audit trace that lets field failures be diagnosed from one file without a rerun. Disable either with --no-sift-log-file / --no-sift-audit-log; warnings are still echoed to stdout. The audit trace records config typos, connection state, the resolved-settings snapshot with provenance, status transitions, step open/close, the replay lifecycle, and step.outcome / step.propagate / step.resolve events that show why each step reached its status (own measurement or outcome failure, inherited child failure, or abort). Values for sensitive field names such as token, api_key, and secret are redacted as [REDACTED]. The terminal summary gains an audit-log section.

Bugfixes

  • Fix incremental upload of test-result logs failing on any resume tick after the first, which left the report stuck in progress. Resume ticks now read the report ID from the tracking map instead of requiring an in-memory report, and the recovery hint points at import-test-result-log --incremental <log> so a manual retry resumes from the cursor rather than creating a duplicate report. (#611)

Full Changelog

[v0.17.1] - June 10, 2026

Bugfixes

  • Bound the test-results offline log lock wait to 60 seconds so a held lock no longer hangs synchronous API calls; a stuck lock now raises TimeoutError. Raises the minimum filelock version to 3.15. (#623)
  • Skip progress bars when no console is attached (PyInstaller --noconsole, pythonw.exe, detached processes), so uploads, downloads, and job polling no longer crash. (#622)

[v0.17.0] - June 3, 2026

What's New

Update, Archive, and Unarchive Channels

Channels can now be edited from the Python client. Use client.channels.update(channel, ChannelUpdate(...)) to change a channel's description, unit, metadata, or archived state, or call update(), archive(), and unarchive() directly on a Channel. Units are read and written by name, and Channel now exposes metadata and is_archived.

channel = client.channels.update(channel, ChannelUpdate(unit="volts", metadata={"calibrated": True}))
channel.archive()

HDF5 Schema Detection

Data import now detects HDF5 files under three layouts, selected through the data_type argument with the new DataTypeKey.HDF5_ONE_D, DataTypeKey.HDF5_TWO_D, and DataTypeKey.HDF5_COMPOUND keys. detect_config and import_from_path also accept a new time_format argument; when omitted, HDF5 imports use TimeFormat.ABSOLUTE_UNIX_NANOSECONDS. HDF5 support remains an optional extra (pip install sift-stack-py[hdf5]).

Configurable Request Timeouts

gRPC and REST calls now accept a configurable default request timeout. Set GrpcConfig(request_timeout=...) for unary gRPC calls and RestConfig(request_timeout=...) for REST, both defaulting to 60 seconds. A per-call timeout overrides the default, None disables it, and long-running operations such as wait_until_complete and wait_and_download honor their own timeout_secs.

Bugfixes

  • Bound synchronous API calls so they fail fast instead of hanging the calling thread indefinitely when a request stalls or the client has already been closed. (#613)
  • Release the underlying gRPC channels on close() and close_sync() and make teardown idempotent, eliminating a shutdown warning, stall, and double-close hang at process exit. (#595)

Full Changelog

[v0.16.2] - May 21, 2026

Bugfixes

  • Resolve nested calculated channel references when fetching calculated channels so expressions that depend on other calculated channels evaluate correctly. (#580)
  • Infer the parquet time column for integer timestamp columns in data imports so files with integer epoch time columns no longer fail detection. (#582)

[v0.16.1] - May 19, 2026

Bugfixes

  • Replace the POSIX-only fcntl.flock call in the test-results log with a cross-platform filelock.FileLock so importing sift_client no longer fails on Windows. (#574)

[v0.16.0] - May 14, 2026

What's New

Description, Metadata, and Associated Channels on Test Measurements

TestMeasurementCreate and TestMeasurementUpdate now accept description, metadata, and channel_names to attach contextual information and link measurements to the channels they were derived from. channel_names accepts either channel name strings or Channel objects. Descriptions longer than 2000 characters are truncated with a warning to match the server-side limit.

Metadata on Test Steps

TestStepCreate and TestStepUpdate now accept a metadata dict so test steps can carry structured key/value context the same way reports and measurements do. The ReportContext.new_step / create_step and NewStep.substep helpers in the test-results context manager forward a new metadata kwarg through to the created step.

Bugfixes

  • Pass the configured log file through to test result update calls so update events are written to the .jsonl log alongside create events. (#560)

Full Changelog

[v0.15.0] - May 7, 2026

What's New

v0.15.0 updates the streaming ingestion client to match the new sift-stream-bindings 0.3.0 API. The RecoveryStrategyConfig class and recovery_strategy parameter have been replaced with an explicit StreamingMode enum and discrete per-mode configuration kwargs. The non-blocking send method has been renamed for consistency with the Rust library, and a new try_send method is available for single-flow non-blocking sends. This release contains breaking changes to the ingestion API — see below for details and a migration prompt.

Breaking Changes

1. RecoveryStrategyConfig Removed — Replaced by StreamingMode + Per-Mode Kwargs

RecoveryStrategyConfig and the recovery_strategy parameter on IngestionConfigStreamingClient and IngestionAPIAsync have been removed. Transport mode is now selected via a StreamingMode enum, with separate retry_policy and disk_backup_policy kwargs for per-mode configuration.

The default mode is StreamingMode.LIVE_WITH_BACKUPS, which matches the previous default behavior of RecoveryStrategyConfig.retry_with_backups().

Removed from the public API: - RecoveryStrategyConfig class (and its retry_only() / retry_with_backups() factory methods) - recovery_strategy parameter on IngestionConfigStreamingClient and IngestionAPIAsync

Added: - StreamingMode enum — LIVE_ONLY, LIVE_WITH_BACKUPS, FILE_BACKUP - streaming_mode parameter (default: StreamingMode.LIVE_WITH_BACKUPS) - retry_policy parameter — applies to LIVE_WITH_BACKUPS mode - disk_backup_policy parameter — applies to LIVE_WITH_BACKUPS and FILE_BACKUP modes - checkpoint_interval_seconds parameter — applies to LIVE_WITH_BACKUPS mode

Before: 

from sift_client.resources.ingestion import (
    IngestionConfigStreamingClient,
    RecoveryStrategyConfig,
)

# Default: live streaming with backups
client = await IngestionConfigStreamingClient.create(
    ingestion_config=my_config,
    recovery_strategy=RecoveryStrategyConfig.retry_with_backups(),
)

# Retry only (no disk backups)
client = await IngestionConfigStreamingClient.create(
    ingestion_config=my_config,
    recovery_strategy=RecoveryStrategyConfig.retry_only(),
)

After: 

from sift_client.resources.ingestion import (
    IngestionConfigStreamingClient,
    StreamingMode,
)

# Default: live streaming with backups (no change needed if you were using the default)
client = await IngestionConfigStreamingClient.create(
    ingestion_config=my_config,
    streaming_mode=StreamingMode.LIVE_WITH_BACKUPS,
)

# Live only — no disk backups, lowest overhead
client = await IngestionConfigStreamingClient.create(
    ingestion_config=my_config,
    streaming_mode=StreamingMode.LIVE_ONLY,
)

# File backup only
client = await IngestionConfigStreamingClient.create(
    ingestion_config=my_config,
    streaming_mode=StreamingMode.FILE_BACKUP,
)

To pass a custom retry or disk backup policy: 

from sift_stream_bindings import DiskBackupPolicyPy, RetryPolicyPy

client = await IngestionConfigStreamingClient.create(
    ingestion_config=my_config,
    streaming_mode=StreamingMode.LIVE_WITH_BACKUPS,
    retry_policy=RetryPolicyPy.default(),
    disk_backup_policy=DiskBackupPolicyPy.default(),
    checkpoint_interval_seconds=30,
)

2. send_requests_nonblocking Renamed to try_send_requests

IngestionConfigStreamingClient.send_requests_nonblocking has been renamed to try_send_requests to align with the Rust sift-stream naming convention where try_ methods return immediately without awaiting channel capacity.

Before: 

client.send_requests_nonblocking(requests)

After: 

client.try_send_requests(requests)

New Features

try_send — Non-Blocking Single-Flow Send

A new try_send(flow) method is available on IngestionConfigStreamingClient for non-blocking single-flow sends. It accepts either a Flow or a raw FlowPy object.

client.try_send(flow)

Use try_send in real-time loops where blocking on channel capacity is unacceptable. For most use cases, the async send(flow) method (which applies backpressure) is preferred.

sift-stream-bindings 0.3.0

The sift-stream-bindings dependency has been bumped to 0.3.0, which reflects the sift-stream 0.9.0 breaking API changes (stepped builder, send rename, removed types).

AI-Assisted Migration Prompt (v0.14.x → v0.15.0)

Copy and paste the following prompt to an AI coding agent to automate the upgrade:

You are upgrading a Python project from sift_client v0.14.x to v0.15.0. The streaming ingestion
API has breaking changes. Apply ALL of the following changes precisely. Do not make any other
modifications.

---

## 1. Update `pyproject.toml`

Find every occurrence of `sift-stream-bindings==0.2.2` and replace it with
`sift-stream-bindings==0.3.0`. This may appear under multiple dependency groups (e.g. `all`,
`dev-all`, `sift-stream`, `sift-stream-bindings`).

---

## 2. Remove all imports of `RecoveryStrategyConfig`

Delete any line that imports `RecoveryStrategyConfig`, for example:

    from sift_client.resources.ingestion import RecoveryStrategyConfig
    from sift_client.resources.ingestion import IngestionConfigStreamingClient, RecoveryStrategyConfig

Remove only `RecoveryStrategyConfig` from those imports; keep any other names on the same line.

---

## 3. Add `StreamingMode` to imports where needed

Wherever `IngestionConfigStreamingClient` or `IngestionAPIAsync` is imported and a streaming
mode needs to be specified, add `StreamingMode` to the import:

    from sift_client.resources.ingestion import IngestionConfigStreamingClient, StreamingMode

---

## 4. Replace all `recovery_strategy` call sites

Search for every call to `IngestionConfigStreamingClient.create(...)` and
`IngestionAPIAsync.create(...)` that contains a `recovery_strategy` keyword argument.

### Case A — `RecoveryStrategyConfig.retry_with_backups()` (or no recovery_strategy at all)

This was (and remains) the default. Replace the kwarg:

    # BEFORE
    recovery_strategy=RecoveryStrategyConfig.retry_with_backups()

    # AFTER
    streaming_mode=StreamingMode.LIVE_WITH_BACKUPS

If the call had no `recovery_strategy` argument, no change is needed — `LIVE_WITH_BACKUPS`
is the default.

If the old call passed explicit `retry_policy` or `disk_backup_policy` arguments inside
`RecoveryStrategyConfig.retry_with_backups(...)`, move them to top-level kwargs:

    # BEFORE
    recovery_strategy=RecoveryStrategyConfig.retry_with_backups(
        retry_policy=my_retry_policy,
        disk_backup_policy=my_disk_policy,
    )

    # AFTER
    streaming_mode=StreamingMode.LIVE_WITH_BACKUPS,
    retry_policy=my_retry_policy,
    disk_backup_policy=my_disk_policy,

### Case B — `RecoveryStrategyConfig.retry_only()`

Replace with `streaming_mode=StreamingMode.LIVE_ONLY`. If a `retry_policy` was passed,
keep it as a top-level kwarg (it is ignored for `LIVE_ONLY` in this version, but preserving
it avoids a TypeError):

    # BEFORE
    recovery_strategy=RecoveryStrategyConfig.retry_only(retry_policy=my_policy)

    # AFTER
    streaming_mode=StreamingMode.LIVE_ONLY

### Case C — Raw `RecoveryStrategyPy` object passed directly

If any call passes a raw `RecoveryStrategyPy` instance as `recovery_strategy`, determine
which mode it was configured for and replace accordingly:
- `RecoveryStrategyPy.retry_only(...)` → `streaming_mode=StreamingMode.LIVE_ONLY`
- `RecoveryStrategyPy.retry_with_backups(...)` → `streaming_mode=StreamingMode.LIVE_WITH_BACKUPS`
  with `retry_policy` and `disk_backup_policy` promoted to top-level kwargs.

---

## 5. Rename `send_requests_nonblocking` → `try_send_requests`

Find every call to `.send_requests_nonblocking(...)` on any ingestion client instance and
rename it to `.try_send_requests(...)`. The signature is unchanged.

    # BEFORE
    client.send_requests_nonblocking(requests)

    # AFTER
    client.try_send_requests(requests)

---

## 6. Verify

After applying the above changes:
1. Run `grep -r "RecoveryStrategyConfig" .` — expect zero results.
2. Run `grep -r "send_requests_nonblocking" .` — expect zero results.
3. Run `grep -r "recovery_strategy" .` — expect zero results.
4. Run your test suite to confirm no remaining references.

[v0.14.1] - April 30, 2026

Bugfixes

  • Lazy-import h5py and nptdms so the base install doesn't require them. (#547)
  • Expose page_size on resource list methods so callers can shrink pages when responses hit gRPC message-size limits. (#548)

[v0.14.0] - April 28, 2026

What's New

Data Import API in SiftClient

The sift_client module now exposes a data import API supporting CSV, Parquet, TDMS, and HDF5. With this addition, all features previously available only in sift_py are now available in sift_client, which remains the recommended interface for new development. sift_py (deprecated since v0.10.0) continues to work and ship in this release.

Migrating from sift_py: the per-format upload services (CsvUploadService, ParquetUploadService, Hdf5UploadService, TdmsUploadService) collapse into a single client.data_import.import_from_path method. sift_py only auto-detected for CSV via simple_upload; other formats required more setup. sift_client unifies all four with auto-detection built into import_from_path itself: the config argument is optional, so the common call takes just a file path and target asset. Call client.data_import.detect_config(...) first if you want to inspect or patch the config before importing. import_from_path returns a Job you can optionally wait on. 

# sift_py (deprecated)
from sift_py.data_import.csv import CsvUploadService
from sift_py.rest import SiftRestConfig

rest_conf: SiftRestConfig = {"uri": sift_uri, "apikey": apikey}
csv_service = CsvUploadService(rest_conf)
import_service = csv_service.simple_upload(asset_name="my_asset", path="data.csv")
import_service.wait_until_complete()

# sift_client
from sift_client import SiftClient

client = SiftClient(api_key=apikey, grpc_url=grpc_url, rest_url=rest_url)
job = client.data_import.import_from_path("data.csv", asset="my_asset")
job.wait_until_complete()

Format-by-format support: - CSV: auto-detected from .csv. Supports an optional JSON metadata row (row 1 or row 2) for specifying channel names, units, data types, and the time column format. - Parquet: requires an explicit data_type (PARQUET_FLATDATASET or PARQUET_SINGLE_CHANNEL_PER_ROW) since .parquet alone doesn't disambiguate the layout. Detection only reads the file footer, so it stays fast on large files. - HDF5: new in this release. Auto-detected from .h5 / .hdf5. Detection works out-of-the-box for files with a compound-dataset layout (first field = time) or a shared root-level time dataset; other layouts may need a hand-built config. - TDMS: new in this release. Auto-detected from .tdms and recognizes the common timing conventions (group-level xchannel, first-channel TimeStamp, or per-channel waveform properties). TdmsImportConfig controls handling of untimed channels (fallback_method), complex values (complex_component), scaled vs. raw data, and waveform start-time overrides.

Parquet as an Export Output Format

client.data_export.export(...) now accepts ExportOutputFormat.PARQUET alongside the existing CSV and Sun/WinPlot options. Unlike the sift_py DataService + DataFrame.to_parquet() pattern (async-only, buffers everything in memory, name-strings only), the new export API runs as a server-side job, works sync or async, accepts Asset/Channel objects or IDs, and scales to large exports.

# sift_py (deprecated): no dedicated export API, so query in-memory and write yourself
import pandas as pd
from sift_py.data.query import ChannelQuery, DataQuery
from sift_py.data.service import DataService
from sift_py.grpc.transport import use_sift_async_channel

async with use_sift_async_channel({"uri": sift_uri, "apikey": apikey}) as channel:
    result = await DataService(channel).execute(DataQuery(
        asset_name="my_asset",
        start_time=start,
        end_time=stop,
        channels=[ChannelQuery(channel_name="my_channel")],
    ))
    pd.DataFrame(result.all_channels()[0].columns()).to_parquet("out.parquet")

# sift_client
from sift_client import SiftClient
from sift_client.sift_types.export import ExportOutputFormat

client = SiftClient(api_key=apikey, grpc_url=grpc_url, rest_url=rest_url)
job = client.data_export.export(
    assets=["my_asset"],          # accepts Asset objects or IDs
    channels=["my_channel"],      # accepts Channel objects or IDs
    start_time=start,
    stop_time=stop,
    output_format=ExportOutputFormat.PARQUET,
)
files = job.wait_and_download(output_dir="./exports")

wait_and_download polls the job to completion, downloads the resulting archive, and returns the list of downloaded file paths. Both arguments shown are optional: if output_dir is omitted, a temporary directory is created and used. By default the downloaded zip is extracted (and the archive deleted) so you get the Parquet files directly; pass extract=False to keep the zip instead, which is useful if you want to hand the whole bundle off to another system without unpacking it client-side.

Test Result Logging

Test result create and update events can now be optionally written to a .jsonl log file during a test run, then replayed against the Sift API later via the new import-test-result-log CLI command (installed with the package).

Progress Indicators

Adds progress indicators for job polling and file downloads for better visibility during long-running operations.

Stop Bundling google.api, protoc_gen_openapiv2, and buf.validate

Previously the package shipped its own bundled copies of the generated Python bindings for google.api, protoc_gen_openapiv2, and buf.validate. With this release: - google.api and protoc_gen_openapiv2 are now pulled in via the googleapis-common-protos and protoc-gen-openapiv2 runtime dependencies, so pip installs the upstream-maintained versions. - buf.validate has been removed entirely. The protovalidate field annotations were stripped from the affected .proto files, so the generated _pb2.py files no longer reference buf.validate.

Upgrading from a pre-v0.14.0 install

Upgrading in place can leave behind __init__.py files and the buf/ directory from the previously-bundled copies. When that happens, the leftover google/__init__.py and google/api/__init__.py turn google and google.api into regular packages and collapse the PEP 420 namespace, hiding the upstream googleapis-common-protos contribution. The symptom is an ImportError such as:

ImportError: cannot import name 'annotations_pb2' from 'google.api' (unknown location)

A fresh virtualenv resolves it cleanly:

python -m venv .venv && source .venv/bin/activate
pip install sift-stack-py

To repair an existing environment in place:

SITE=$(python -c "import sysconfig; print(sysconfig.get_paths()['purelib'])")
rm -f  "$SITE/google/__init__.py" "$SITE/google/api/__init__.py"
rm -rf "$SITE/buf"
pip install --force-reinstall --no-deps \
  googleapis-common-protos protoc-gen-openapiv2 sift-stack-py

Verify with:

python -c "from google.api import annotations_pb2; print(annotations_pb2.__file__)"

Bugfixes

  • Add py.typed to the generated proto directory so type checkers pick up protobuf types correctly.

Full Changelog

[v0.13.0] - March 24, 2026

What's New

SiftClient Export API

The sift_client module now exposes a public export interface and wrapper, making it easier to integrate with downstream tools and workflows.

Module Migration to sift_client

The grpc, rest, and time modules have been migrated from sift_py to sift_client. All imports from sift_py for these modules have been removed. This continues the transition toward sift_client as the primary interface ahead of the v1.0.0 release.

Arbitrary File Attachment Uploads

File attachment uploads now support arbitrary file types (e.g., Parquet, pcapng) by falling back to application/octet-stream for unknown MIME types.

Improved SiftChannelConfig URL Handling

SiftChannelConfig now handles URLs more robustly, fixing gRPC connection errors caused by missing URL schemes.

Streaming Ingestion Performance

The streaming ingestion module has been refactored to use sift_stream_bindings types for improved performance.

Full Changelog

[v0.12.0] - February 26, 2026

What's New

Channel Archiving/Unarchiving Support in SiftClient

SiftClient has been updated with the ability to archive and unarchive channels, allowing users to cleanup old channel definitions that are no longer desired.

Full Changelog

[v0.11.0] - February 24, 2026

What's New

Better Rule and Report Support in SiftClient

SiftClient has been updated to provide better support for RuleVersions and better ergonomics for Reports - Additional methods have been added to the SiftClient Rules resource to allow easy access to RuleVersions - Reports can now be generated from RuleVersions - The Jobs and Reports resources now have a wait_until_complete sync/async method, which provides a convenient way to wait until a job or report is complete, with configurable polling and timeout settings. - When creating reports, SiftClient now returns a Job object, to signify that the report is still in progress. Waiting on the job with wait_until_complete will return the completed report once finished. The report_id can also still be accessed directly from the job object if needed.

Full Changelog

[v0.10.1] - February 11, 2026

What's New

[v0.10.0] - January 30, 2026

DEPRECATION NOTICE

The sift_py module is deprecated as of v0.10.0 and will be removed in v1.0.0. Please use sift_client for all new development. Several minor releases will follow before the major release to add features and give users time to migrate.

What's New

[v0.9.6] - December 22, 2025

[v0.9.5] - December 12, 2025

[v0.9.4] - December 10, 2025

[v0.9.3] - December 4, 2025

[v0.9.2] - December 3, 2025

[v0.9.1] - August 18, 2025

[v0.9.0] - September 15, 2025

What's New

Parquet Upload Service

Adds support for uploading Parquet files to Sift for ingestion through the addition of the ParquetUploadService. See examples/data_import/parquet for an example of how to upload Parquet files.

TDMS File Metadata Support

Adds a flag to automatically import TDMS file metdadata into Sift Runs.

Lazy Flow Creation

Adds the ability to create flows lazily which reduces initialization time of the IngestService.

Full Changelog

[v0.8.4] - August 18, 2025

[v0.8.3] - August 11, 2025

[v0.8.2] - August 1, 2025

[v0.8.1] - July 31, 2025

[v0.8.0] - July 29, 2025

What's New

HDF5 Upload Service

Adds support for uploading HDF5 files to Sift for ingestion through the addition of the Hdf5UploadService and Hdf5Config. See examples/data_import/hdf5 for an example of how to upload HDF5 files.

Full Changelog

[v0.7.0] - June 24, 2025

What's New

AssetService and Metadata Support

Support for attaching metadata to a run or asset has been introduced through the addition of the AssetService and updates to the attach_run method of the IngestionService. See examples/assets for an example of updating metadata for an asset.

Full Changelog

[v0.6.2] - June 9, 2025

[v0.6.1] - June 2, 2025

[v0.6.0] - May 23, 2025

What's New

Rule Evaluation Service

The RuleEvaluationService has been introduced to provide functionality for evaluating rules and reports against runs or assets. This includes support for dry-run evaluations and previews, as well as evaluating rules defined in YAML configurations.

External Rules

The RuleService has been updated to support External Rules. External Rules are ideal for automated workflows such as CI/CD pipelines, where external Rules are managed and evaluated programmatically.

Read more about External Rules here here.

Full Changelog

[v0.5.3] - May 20, 2025

[v0.5.2] - April 23, 2025

[v0.5.1] - April 18, 2025

[v0.5.0] - March 31, 2025

What's New

Rosbag2 Uploads

A new RosbagsUploadService has been added to sift_py.data_import.rosbags, which provides the functionality for uploading rosbags, including video frames from a rosbag. See examples/data_import/rosbags for examples on how to use the new service.

Full Changelog

[v0.4.2] - March 4, 2025

[v0.4.1] - February 27, 2025

[v0.4.0] - February 21, 2025

What's New

Calculated Channels Service

A new CalculatedChannelService, in sift_py.calculated_channels.service, provides functionality for creating, updating, searching, and listing Calculated Channels. It also provides a YAML interface for defining Calculated Channels using a YAML file. Examples can be found in examples/calculated_channels.

Deprecation Notices

Channel Component field is being deprecated and will be removed in version 1.0.0. Please see https://docs.siftstack.com/docs/glossary#component

for more information.

Full Changelog

[v0.3.3] - January 24, 2025

[v0.3.2] - January 17, 2025

[v0.3.1] - January 8, 2025

[v0.3.0] - January 7, 2025

What's New

Report Template Service

There is now a ReportTemplateService that is available from sift_py.report_templates.service. This service allows for creating, updating, and retrieving report templates with a client key, where client keys are unique user defined strings associated with a given report template. Report templates may either by defined via a python config (using the ReportTemplateConfig class) or a YAML file, which follows the same schema. Please see examples/report_templates for some examples of how to manage report templates via python and YAML using the ReportTemplateService. - Adds ReportTemplateService and example usage

Rule Service

There is now a RuleService that is available from sift_py.rule.service. This service allows for creating, updating, and retrieving rules. Rules created through this service should now include a rule client key, a unique user defined string associated with a given rule. As before, rules may be defined in python (via the RuleConfigClass) or via YAML (following the RuleYamlSpec class). Please see examples/ingestion_with_yaml_config for an example of how to create rules from YAML using the RuleService. - Updates most examples to use RuleService to create rules

Chapter 10 Support

There is now a base class, BaseCh10File, and an upload service, Ch10UploadService that allows for easily parsing and uploading IRIG Chapter 10 files. Please see examples/data_import/ch10 for an example of how to use these classes. - Allow uploading compressed chapter 10 files

Other Improvements

Bugfixes

Deprecation Notices

The overwrite_rules option of IngestionService is going to be deprecated in the next release, and will emit a warning if set.

Full Changelog

[v0.2.2] - October 9, 2024

Summary of changes: - Loosen various dependency requirements

[v0.2.1] - September 26, 2024

Summary of changes: - add missing double channel computation - update protos - update protos - csv example ingestion - update example - Add multiple CSV ingestion example - Add metadata arg to use_sift_channel

[v0.2.0] - September 3, 2024

Summary of changes: - Add pydantic models for the sift grafana plugin queries - Lowered minimum version requirements for various dependencies and dependency cleanup - Added py.typed to modules that utilize type annotations - Added services derived from protobuf for the following APIs * Remote files * Rule versions * Saved searches - Added helpful validations when processing Sift API URLs - Added support for gRPC keep-alive - Added service to upload and download file attachments - Added timer-based flushing for buffered ingestion and custom error-handling - Added functionality to generate a channel value from a channel config

The following section will cover some of the more notable features in depth. Please refer to the documentation for even more detail.

Table of Contents

Keepalive

Long-lived connections are generally are at risk of being closed due to idle timeouts if they are idle for a particular duration. In other words, if there is no data being exchanged on the connection for a duration specified by a load balancer's idle timeout, the connection will be closed. In order to ensure that this does not happen you can opt into enabling the HTTP/2 PING-based keepalive mechanism either on your own gRPC channel or the one that Sift provides. To configure keep-alive using the Sift-provided gRPC channel:

from sift_py.grpc.transport import use_sift_channel, SiftChannelConfig

sift_channel_config = SiftChannelConfig(
    uri=uri,
    apikey=apikey,
    enable_keepalive=True,
)

with use_sift_channel(sift_channel_config) as channel:
    ...

This uses default values set in the sift_py.grpc.keepalive module. If you'd like to configure your own keepalive parameters you could also do the following:

from sift_py.grpc.transport import use_sift_channel, SiftChannelConfig
from sift_py.grpc.keepalive import KeepaliveConfig

sift_channel_config = SiftChannelConfig(
    uri=uri,
    apikey=apikey,
    enable_keepalive=KeepaliveConfig(
        keepalive_time_ms=keepalive_time_ms,
        keepalive_timeout_ms=keepalive_timeout_ms,
        keepalive_permit_without_calls=keepalive_permit_without_calls,
        max_pings_without_data=max_pings_without_data,
    ),
)

with use_sift_channel(sift_channel_config) as channel:
    ...

File attachments

There is now a sift_py.file_attachment module that contains utilities to programmatically upload and download file attachments. Files currently can be attached to various entities such as runs, annotations, and annotation logs. Various video and image formats are currently supported. Below is an example demonstrating how to upload a file attachment and programmatically downloading it.

from sift_py.grpc.transport import SiftChannelConfig, use_sift_channel
from sift_py.file_attachment.service import FileAttachmentService
from sift_py.file_attachment.entity import Entity, EntityType
from sift_py.file_attachment.metadata import VideoMetadata
from sift_py.rest import SiftRestConfig

from sift.remote_files.v1.remote_files_pb2 import GetRemoteFileRequest
from sift.remote_files.v1.remote_files_pb2_grpc import RemoteFileServiceStub

# Get API credentials setup
...

with use_sift_channel(sift_channel_config) as channel:
    file_attachment_service = FileAttachmentService(channel, rest_config)

    run = entity=Entity(
        entity_id=run_id, # some arbitrary run ID that refers to an existing run
        entity_type=EntityType.RUN,
    )

    # uploading the file attachment and attaching it to a run of `run_id`
    remote_file = file_attachment_service.upload_attachment(
        path="path/to/foo.mp4",
        entity=run,
        # Metatadata.. optional but recommended for optimal viewing in the application
        metadata=VideoMetadata(height=2160, width=3840, duration_seconds=5.5),
        description="thrusters getting too hot" ,
    )

    # retrieving all of the file attachments for our run
    all_file_attachments = file_attachment_service.retrieve_attachments(run)

    # downloading our file_attachment and saving it to our current working dir
    file_attachment_service.download_attachment(remote_file)

    # downloading our file_attachment and saving it somewhere else with a different name
    file_attachment_service.download_attachment(remote_file, "somewhere/else/foo.mp4")

    # deleting out file attachment from Sift
    file_attachment_service.delete_file_attachments(remote_file_1, remote_file_2, remote_file_etc)

Timer based flushing for buffered ingestion

Previously the buffered ingestion API flushes a buffer whenever any one of the following conditions are met: - The caller manually calls flush - The buffer gets filled - The with-block associated with the buffered ingestion service as a context-manager goes out of scope. - An exception is raised inside of the aforementioned with-block.

Note that the last two points only apply if the buffered ingestion service is used as a context manager like so:

with ingestion_service.buffered_ingestion() as buffered_ingestion:
    ...

Now there is support to periodically flush the buffer regardless of whether or not the buffer is filled. To configure a timer to flush periodically:

with ingestion_service.buffered_ingestion(flush_interval_sec=3.2) as buffered_ingestion:
    ...

This will configure the buffered ingestion service to flush its buffer every 3.2 seconds. If the buffer happens to be filled before the timer elapses, then the timer will reset.

Custom error handling for buffered ingestion

Previously when using the buffered ingestion service as a context manager, any exception that gets raised within the with-block will result in the service attempting to flush one more time. The caller can now customize error-handling behavior by passing in a function handler that takes in three arguments: the error that's raised, the buffer containing the remaining requests that weren't ingested, and a function that when called will attempt to flush the buffer.

# Custom code to run when error
def on_error_calback(err, buffer, flush):
    # Maybe try to save contents of buffer to disk
    ...
    # Try once more to flush the buffer
    flush()

with ingestion_service.buffered_ingestion(on_error=on_error_calback) as buffered_ingestion:
    ...

[v0.1.1] - July 17, 2024

Summary of changes: - Extend support for Python 3.8

[v0.1.0] - July 12, 2024

Summary of changes: - Promote to v0.1.0 from release candidate state. - Add module to download telemetry

In addition to the changes above, documentation is now also available online: - Online documentation - Section on how to download telemetry

[v0.1.0-rc.3] - July 3, 2024

Summary of changes: - Automated gRPC retries if an unexpected gRPC status code is returned and if a connection unexpectedly terminates

The following are some gRPC error codes that can happen due to external factors that Sift doesn't directly control: - UNKNOWN - UNAVAILABLE - ABORTED - DEADLINE_EXCEEDED

They are the source of common disruptions, particularly during ingestion, and so this mechanism will automatically retry failed RPCs over an existing connection or will establish a new one if necessary.

[v0.1.0-rc.2] - July 1, 2024

Summary of changes: - Introduced automated ingestion request buffering to improve performance - Added support for multi-config ingestion and creating new flows at run-time - Added methods that combine request creation and ingestion into a single-step. - Updates to documentation.

For in-depth documentation please see the documentation section of the README for instructions on how to build the documentation locally.

Table of Contents

Combining Request Generation and Ingestion into a Single Step

Previously to ingest a single request you needed to do the following:

request = ingestion_service.try_create_ingestion_request(
    flow_name="logs",
    timestamp=timestamp,
    channel_values=[
        {
            "channel_name": "log",
            "value": string_value("some log"),
        },
    ],
)
ingestion_service.ingest(request)

Now you can combine both steps using either of the following APIs: - try_ingest_flows - ingest_flows

ingestion_service.try_ingest_flows({
    "flow_name": "log",
    "timestamp": timestamp,
    "channel_values": [
        {
            "channel_name": "log",
            "value": string_value("some string")
        },
    ],
})

You can also send multiple flows:

# Send data for both logs and readings flows
ingestion_service.try_ingest_flows(
    {
        "flow_name": "readings",
        "timestamp": datetime.now(timezone.utc),
        "channel_values": [
            {
                "channel_name": "velocity",
                "component": "mainmotor",
                "value": double_value(10),
            },
            {
                "channel_name": "voltage",
                "value": int32_value(5),
            },
            {
                "channel_name": "vehicle_state",
                "value": enum_value(2),
            },
        ],
    },
    {
        "flow_name": "logs",
        "timestamp": datetime.now(timezone.utc),
        "channel_values": [
            {
                "channel_name": "logs",
                "value": string_value("INFO: some message")
            },
        ],
    },
)

Request Buffering

Requests are now automatically buffered using the buffered ingestion API. Using this may significantly improve performance as it allows serialization and ingestion to occur in batches:

# Defaults to a buffer size of `sift_py.ingestion.buffer.DEFAULT_BUFFER_SIZE` requests.
with ingestion_service.buffered_ingestion() as buffered_ingestion:
    buffered_ingestion.try_ingest_flows(*lots_of_flows)
    buffered_ingestion.try_ingest_flows(*lots_more_flows)

# Custom buffer size of 750 requests
with ingestion_service.buffered_ingestion(750) as buffered_ingestion:
    buffered_ingestion.try_ingest_flows(*lots_of_flows)
    buffered_ingestion.try_ingest_flows(*lots_more_flows)

Once the with-block ends the remaining requests will be automatically flushed and ingested, but flushing may also be done manually:

with ingestion_service.buffered_ingestion() as buffered_ingestion:
    buffered_ingestion.try_ingest_flows(*lots_of_flows)
    buffered_ingestion.flush()

Contrast this with regular ingestion:

ingestion_service.try_ingest_flows(*lots_of_flows)
ingestion_service.try_ingest_flows(*lots_more_flows)

Creating New Flows on the Fly

If there is a flow you need to create on the fly which wasn't declared in your initial telemetry config, you may use either of the following APIs: - try_create_flow - create_flow

new_flow_config = FlowConfig(
    name="my_new_flow", channels=[ChannelConfig("new_channel", ChannelDataType.DOUBLE)]
)
ingestion_service.try_create_flow(new_flow_config)

Multi-config Ingestion

There is now an ergonomic utility class, IngestionServicesManager, that allows users to manage telemetry for multiple configs:

manager = IngestionServicesManager.from_telementry_configs(grpc_channel, {
    "config_a": config_a,
    "config_b": config_b,
})

with manager.ingestion_service("config_a") as config_a:
    config_a.try_ingest_flow(...)

with manager.ingestion_service("config_b") as config_b:
    config_b.try_ingest_flow(...)