Visual monitoring pipelines¶

The visual-language review defines the shared node, port, connection and state grammar, its current V1 execution profile, and the boundaries an extension must preserve.

The shared Pydantic definition is the public and persisted execution contract. The control plane compiles topology, feature order, lineage, parameter defaults and compatibility without importing numerical packages. Browser wire types are generated by backend/scripts/generate_pipeline_types.py; the component catalog also provides bounded field descriptors. An unsupported parameter schema fails catalog generation instead of falling back to a JSON editor.

Each design has an optimistic-concurrency draft and immutable published revisions. A draft save commits its definition, layout, name and description under one expected version. The editor acknowledges the submitted snapshot while retaining edits made during the request as unsaved work. Publishing freezes that submitted snapshot; later local edits remain in the draft session. A run retains its executable definition, resolved input bindings, initial layout, service identity and predecessor independently of the live service record. Layout updates have their own version and do not change numerical compatibility. Runs also retain the resolved reading, entity and source labels, collection policy, catalogue revision and collection fingerprint. These presentation records do not change execution fingerprints. Historical legacy launches retain their original configuration; a reviewed compatibility normalization is used only when rendering their graph or preparing an explicit release restart.

Each component carries its implementation identity and contract version. Every monitor runs a bound graph (launch version 2). The setup forms submit their configuration, and TACTIC converts it with legacy_definition() into the equivalent graph and activates it like a direct apply. Inputs follow the sorted sensor-key order of the form, and a min-max step in front of a unit-interval detector clips to its range. Monitors saved by the setup forms before they ran as pipelines (launch version 1) are converted the same way at the next release: each restarts as a pipeline and learns again, and one whose readings left the catalogue is stopped instead, keeping its history. Internal image pins and typed MQTT launch overrides are accepted only by trusted activation/release paths, never public monitor creation. Published revisions retain the catalog hash as provenance. Activation recompiles the saved component contracts and requires the saved executable fingerprint to match. Unrelated catalog descriptions or components do not invalidate a revision.

Creation workbench¶

?create=monitoring opens one setup page: the processing graph under a header with the service name, an input summary that opens the Input step (entity and reading bindings) and a learning shortcut that opens Service settings. A new service opens at the Input step. Its main actions are Load template, Save template and Apply monitoring. Users do not save drafts or publish revisions to start a service. Service settings (learning counts and the start-up timeline) and step parameters stay in the side panel; only advanced input-slot editing is collapsed. useGraphMetadata checks every edit after 800 ms, and the status chip offers a retry when the check fails. A design has no strategy to switch: each detector's settings choose its model, and with it how it learns (see The editor without a lane). The start-up estimate (startup.ts) takes the slowest bound reading's typical interval, because RADAR aligns observations across all readings. Entity selection includes configured entities even when there is no recent traffic; collection and reading ownership still gate Apply.

Standard setups show a Readings to monitor checklist directly below the entity: uncheck unwanted readings to monitor any subset, such as AC1 and AC3. Select all available and Clear support larger entities; a service accepts up to 100 readings. Selection changes the executable slots and bindings together and retains input IDs across removal, reselection and undo. Saved/custom templates keep explicit named input mappings, with Add reading and Remove on the setup page. Removing an input referenced by a processing component still requires fixing that component before validation and Apply can succeed.

Load template reads the latest saved definition and layout, independently of any historical published revisions. Loading never changes the source template. Template names and descriptions are separate from the service name. A template can be saved without choosing readings or starting monitoring. Subsequent saves use its expected version; a concurrent edit preserves the local work and offers Save as new template. Applied service snapshots do not appear in this library. Migration 013 adds monitoring_pipelines.is_template, preserving all existing designs as templates.

Apply monitoring validates the graph and sends a typed POST /v1/pipelines/apply request with its definition, layout, bindings, service name and idempotency key. One server transaction creates the immutable execution snapshot, resolves bindings and reserves the existing durable activation. The normal preflight, source claims, capacity checks, worker heartbeat and replacement controller are reused. Retrying the same request returns the same operation. The browser retains uncertain requests across navigation and reload; fields stay fixed until the start finishes or cancellation is acknowledged. An explicit rejection permits corrections; a failed operation offers Return to setup.

Edit service opens its actual running definition, layout and bindings directly. Apply changes clearly explains interruption and fresh learning, then uses the same preflight and durable replacement path. Loading a template or editing settings does not affect the existing service until Apply. History remains immutable. Older library, revision and in-flight activation links remain supported.

Unsaved graphs infer feature metadata with POST /v1/pipelines/validate, using the same typed, bounded compiler report, manage_monitors authorization and gateway compilation rate budget as saved templates. This endpoint creates no rows. Creation and templates share the same graph controller and account-scoped setup session for bounded history, component insertion, arrangement and validation. Template saves acknowledge only their submitted snapshot and preserve later edits. Leaving for Sources keeps a save alive without navigating the detached screen. Returning joins that operation and restores the setup. A remembered login restored after reload retains the same member's drafts; logout and account changes clear them, and late responses cannot repopulate cleared state.

The shared canvas uses a 16-unit snapping grid, fine blueprint lines and stronger cross markers in both themes. Visible lines become coarser at distant zoom levels while remaining aligned with the underlying snap positions. The magnet control switches drop, drag and keyboard positioning between snapping and free movement. This is a presentation preference: toggling it never rewrites saved positions or execution semantics. Existing layouts move only through explicit user actions.

Creation, saved designs and running graphs share a canvas-first workbench. Add component opens a searchable drag-and-drop menu; service and component settings share one collapsible inspector. Both panels start closed. Mobile overlays the inspector instead of narrowing the graph. Cards use larger titles, stage colors and typed ports; descriptions appear in component settings. Opening and arranging retain readable text size, while Fit entire graph explicitly requests an overview.

Detectors are the one step kind a design may repeat outside the add-step menu. addDetector() copies the selected detector, model and settings, as detector_N, connects it to the same input and appends it to the calibration step, up to PIPELINE_LIMITS.detectors, which the type generator exports from MAX_DETECTORS. occupiedInput() and connectNodes() treat the calibration step like a merge (takesSeveral), so another detector's connection is added instead of replacing one, and detectorIds() orders detectors by node ID as RADAR does. Combine detectors has its own control (FUSION_CHOICES) on the calibration step once there are two detectors; changing the calibration method keeps it, and the calibration card leads with Strongest of 2 detectors. featuresView() takes the detector to show, and detectorsAfter() lets a row's menu say when a Prepare step on a shared path also reaches other detectors. A sampled observation of combined detectors carries detectors, each detector's score and surprise (ObservedDetector); each detector's own preview outputs its score. The live inspector lists them for the policy, and the chart tooltip lists an evidence row's detector_results.

A block is a named chain of transform steps that the library keeps (monitoring_pipeline_blocks, migration 031_pipeline_blocks.sql) and the editor copies into a design. No design, revision or run refers to a block, so the compiler, RADAR and activation know nothing of blocks, and deleting one changes nothing that runs. PipelineBlockCreate bounds a block to MAX_BLOCK_STEPS (16) steps and MAX_BLOCK_BYTES (16 KiB) of settings, and the library holds MAX_BLOCKS (100), so GET /pipeline-blocks returns every block and the menus need no paging. require_block_steps() in models/pipeline_blocks.py accepts the components the catalog offers at the saved contract version with kind transform; each takes features and gives features, so any of them can follow any other. It does not check settings: a block has no inputs to check them against, and the design's validation checks the copied steps where they land. PipelineBlocks (TACTIC services/pipelines/blocks.py) writes under begin_library_write(), the guards the template library uses, so blocks are read-only while authoring is off. Names are unique (uq_pipeline_block_name). A block is saved and deleted, never updated.

In the editor, blocks.ts holds the operations. blockChain() gives the steps a card's Save as block can hold and blockSteps() their settings and titles without ids or positions. placeBlock() adds a block as a chain of its own, insertBlock() on a connection and addBlockAfter() after a card: on the card's one connection out when the block fits there, otherwise as a branch. Each returns a BlockOutcome, the changed document with the id of the first new step or the reason the block cannot be added, and blockChoices() pairs every block with its outcome, so the menus list all blocks and say why one does not fit. Steps are added with addComponent(..., exact = true), which keeps the saved settings, feature names included, where a new step's names are made unique. On a connection, a learned step that insertBlocker() refuses is exchanged through counterpartBefore() for the counterpart that learns as the detectors after the connection do, the exchange changeDetector() makes. useBlocks() reads the library once per editor that can change its design; a failed read leaves the menus without blocks. PIPELINE_LIMITS.blockSteps comes from MAX_BLOCK_STEPS through the type generator.

Monitoring history¶

The service Evidence tab presents one timeline for its recorded inputs and joint detector result. Readings retain independent scales and units; the raw anomaly score (or logarithmic static evidence) appears once below them. A dashed line shows the threshold, red input lines mark service alarms, and score dots mark alarms from the selected detector. An alarm a Mute alarms step held back is an open dot, and the key names open dots only when the chart shows one. The view works with multivariate models such as online isolation forest and does not infer per-sensor attribution or a baseline.

Up to six input tracks may be displayed simultaneously. Changing that selection only changes the view: results still describe the service's complete input set. Shared hover uses each result's original input timestamps to retrieve its exact recorded readings. Missing input history is labelled unavailable, never replaced with a nearby observation or a synthetic zero. Zoom and keyboard navigation apply to every track and remain fixed during inspection as new data arrives.

Telemetry polls per visible reading, while evidence polls once per input entity with a service filter applied before database pagination. This optional service_id filter is propagated through the gateway chart-evidence endpoint; existing unfiltered consumers and cursor ordering retain their contract. History uses the signed-in workspace's bounded cache (2,000 retained points per displayed input and 2,000 result rows). Ordinary reading charts remain independent of a monitor; when monitoring is present their alarm markers and results share the same service scope.

Numerical ownership¶

Aberrant 1.3 supplies the standalone aberrant.compose.GraphPipeline numerical API. OFF-KEY compiles into that API and owns bindings, lifecycle, calibration, alarm policy, persistence and presentation. Aberrant has no dependency on these application services or wire formats. Existing linear Aberrant pipelines retain their existing behavior.

Adaptive graphs learn each shared node once in topological order. Only ready outputs reach successors, including on the observation that first establishes readiness. Detector warm-up counts successful terminal updates. Calibration starts at the next scoring opportunity after both that target and actual readiness are satisfied. Calibration and monitoring score before learning. Retained-value readiness can wait for suitable data without retaining an unbounded input buffer. Generated constructor schemas also encode retained-window capacity constraints from the pinned runtime's parameter-sensitive capabilities. For example, geometric averages require two retained values, or three with absoluteValues=true, while harmonic averages permit a one-value window. Catalog generation verifies these rules without introducing numerical imports into the control plane.

Static graphs fit each preprocessing component on training observations only. They freeze before transforming calibration data, fitting conformal calibration, or monitoring. Fitted preprocessing, detector and policy publish together after background training. Static centered batch PCA and adaptive uncentered incremental PCA have different implementation identities. Scalers and random projections also have strategy-specific identities. Changing preprocessing coordinates online does not re-express geometry already retained by an adaptive detector.

Raw branches, selection and explicit renaming retain lineage. Ordered merges join branches from the same aligned observation. Name collisions fail validation; projection features are scoped by their node ID. Selection uses stable slot IDs, while resolved reading topics remain the evidence identity. Expected units must match exactly; the application performs no implicit conversion. Incomplete drafts still expose every knowable feature schema. An invalid node blocks inference for its dependent paths while independent branches and valid upstream outputs remain available to the inspector. Executable compilation still requires the entire graph to validate.

Scalers and projections take an optional scope (pipeline_scope.py): apply_to names the features they use, unset for all, and result is replace or add. Elementwise steps add <feature>_<suffix>; a replacing projection drops the features it uses and appends <node>.component_i. Untouched features keep their order and added ones follow in apply_to order. Scope settings at their defaults, and a suffix when replacing, are left out of the normalized parameters, so designs saved before scopes existed keep their fingerprints. The compiler reports an unknown apply_to name, a repeated output name or more components than used features with their parameter path. Static graphs fit a scoped step on its columns only. Adaptive graphs wrap the transformer in ScopedStep, which forwards readiness and learning transactions; unscoped steps keep their plain runtime shape, so existing checkpoints load unchanged. Tests pin that a scoped step computes exactly what select, the step and an ordered merge compute, under both strategies. Robust scaling uses scikit-learn's RobustScaler on training data and Aberrant's RollingRobustScaler online; both use linear quartiles and scale by 1 when a feature's interquartile range is zero.

Expected value steps (static.expected_value and adaptive.expected_value, EXPECTATIONS in pipeline_scope.py) are always scoped. target names the feature to expect and apply_to the features it is expected from, unset for every other feature. step_scope lists the target first and gives the residual's name, name or <target>_residual, as the scope's added feature. A replacing step drops the target and the features it is expected from and appends the residual, whose lineage is the slots of all of them. The compiler reports an unknown target, a target among apply_to, a step that only the target reaches and a name that repeats a feature that stays, each with its parameter path, and refuses a static step whose training window holds no more observations than the fit has coefficients. ExpectedValue (pipelines/expected.py) solves centred least squares with NumPy on the training window; inputs that are constant or repeat each other get the minimum-norm weights, so a fit never fails. OnlineExpectedValue wraps Aberrant's RLSRegressor, with one coefficient per input and a constant. It is ready after one observation per coefficient and one more, and its learning transaction restores a deep copy of the regressor when a later step rejects the observation. Scoring reads the fit before the observation is learned; in a learning pass Aberrant transforms after the update, as for every online step, so the detector learns the residual of the updated fit. forgetting defaults to 1, because the covariance of a fit that forgets grows without bound while its inputs stop varying. STATE_ESTIMATORS counts a weight and a centre per used feature for the static step and eight covariance matrices for the online one.

Prepare steps (pipeline_prepare.py) are deterministic kernels that run once per observation, in arrival order, before anything learns. stage_split() takes every Prepare node and its ancestors, except the source, as the stage. Its frontier is the stage nodes, and the source, whose features a later node uses, in compiled order. RADAR's PrepareStage runs the stage on the bound observation and passes the frontier's features, keyed <node>.<feature>, to the learned stage: static graphs fit and score those columns, and adaptive graphs read them through Prepared input nodes. With several detectors, each detector's graph gets the Prepared nodes its own steps read across the cut (adaptive_graph), not every frontier node among its ancestors: a detector behind a Prepare step reads that step alone, also when another detector reads the source. Designs without Prepare steps keep their runtime shape, so existing checkpoints load unchanged.

Each kernel advances an immutable state per feature and returns a value, or none while its window fills. The stage commits an observation only when every step succeeds. A step that cannot use a value raises PrepareRejected, an ObservationRejected: every state stays as it was, the observation becomes a pipeline_observation_rejected coverage gap, and the run continues. Rejections are counted per node by reason (non_finite, out_of_domain, out_of_range, out_of_order). A kernel with lead L outputs from its (L+1)-th value, so the stage is ready at one plus the largest sum of leads along a chain. Until then the detectors return preparing results, the run stage reads N/M observations prepared and no learned node runs. The static strategy keeps preparing while it trains in the background, so windows never restart. Checkpoints carry the stage state (prepare, version 1), and a restore refuses state that does not match the design. A valid check reports preparation_observations, the observations before the first prepared one, which the composer shows as Preparing before learning.

The compiler marks Prepare outputs as derived (raw=False) and propagates bounds through them. It rejects a Prepare step downstream of a learned one (stage) and limits all windows to 50,000 values across their features (prepare_budget), counting Kernel.held for each feature a step uses: one window, or two for a Rolling relation. Means use math.fsum, standard deviations divide math.sumprod of the deviations by n − 1, and order statistics keep a sorted copy updated by bisection; an even window's median averages the middle pair. Despike compares against 1.4826 × MAD and leaves a flat window unchanged. Change and Rolling statistic default to result: add, with suffixes named after their settings, such as diff1 and mean60; scope settings at a step's own defaults are left out of the normalized parameters. A rolling quantile's suffix holds the quantile as a percentage, q90_60 for 0.9 over 60 values and q99p9_60 for 0.999, rounded to six decimals of a percent; defaultSuffix in scope.ts derives the same name.

Rolling statistics also support sum, linearly interpolated quantiles, IQR, unscaled MAD and least-squares slope per observation. zscore, robust_zscore and rank place the latest value in the window, and rms, skewness and kurtosis (excess, population moments) describe it. The three moment statistics divide the deviations by the largest one first, so no power overflows or vanishes, and are clamped to the limits no window of n values exceeds: (n - 1) / sqrt(n) for a z-score, (n - 2) / sqrt(n - 1) for skewness and n - 5 + 1 / (n - 1) for excess kurtosis. These limits are the features' known ranges, and rank has [0, 1]. A flat window gives 0, and 0.5 for rank. Deviations from the mean, also for std, come from _deviations(): a window on one side of 0 is measured from its smallest value before the mean is taken, because the mean of large values that differ by a few floats rounds to one of them. std and rms take their root in _root_mean_square() with the largest magnitude standing for 1, and rms never falls below the smallest magnitude in its window, which is the lower end of its known range. robust_zscore falls back from 1.4826 x MAD to sqrt(pi / 2) x the mean absolute deviation where the MAD is 0. The quantile setting is normalized away for every other statistic, preserving existing executable fingerprints. MAD and Despike find the median distance by binary search in the sorted window, since the values nearest the median lie side by side there, so neither sorts deviations. Slope multiplies the window with positions centred on it, computed once per step and counted in admission estimates. prepare.accumulate keeps a running sum or extreme in constant-size immutable state per selected feature. A sum keeps the rounding error of its additions in a second number (Neumaier) and emits total plus error, so a long total does not drift. It emits immediately, shares Prepare's commit and rollback behavior, and resumes from the same checkpoint state. Its default is to add <feature>_total, <feature>_running_min, <feature>_running_max or <feature>_count. State is (seen, value, carry); _carried() reads the (seen, value) of checkpoints from before the carry as a carry of 0.

Three per-feature steps also read the row they run on. A kernel cannot parse a condition, because pipeline_formula.py imports pipeline_prepare.py, so lane_context(component, params, names) in pipeline_combine.py returns a function from the arriving row to the extra arguments of Kernel.advance, or none for a step that reads only its own feature. PrepareStage evaluates it once per step and observation and passes the result to every feature's kernel; a PrepareFailure in it skips the observation like any other.

• prepare.accumulate takes reset_when, a condition (LANE_CONDITIONS). Where it holds, the kernel starts from initial() before it takes the value, so that observation is the first of the new accumulation. The setting is left out of the normalized parameters when empty, so earlier designs keep their fingerprints. count emits the observations since.
• prepare.hold (Hold) takes update_when and keeps (seen, value): it takes the value where the condition holds, and always on its first observation. It has no lead and adds <feature>_held.
• prepare.relation (Relation) receives the value of against and keeps both windows. Deviations from each window's mean are divided by the largest one before math.sumprod, so products neither overflow nor vanish, and the scales return in the result: correlation (clamped to [-1, 1], its known range), covariance (n - 1), slope of the feature on against and residual, the latest value's distance from that line. A window that holds one value is flat by comparison of its values, not of its mean, since fsum([0.1] * 3) / 3 != 0.1; a flat window on either side gives 0, and for residual the latest deviation from the feature's own mean. Suffixes are corr60, cov60, beta60 and resid60. step_scope requires against to arrive, selects every other feature by default and refuses against inside apply_to; the feature's lineage includes the slots of against.

Conditions are checked like a Gate's: names and kinds as formula issues on the setting, and values warns where a condition is always or never true.

Two steps read the time an observation was measured at (reads_time()). MessageProcessor passes the observation time of the aligned vector, the latest of its input_timestamps, through DetectorHandle.process() and process_data_point(..., timestamp) to PrepareStage.run(bound, diagnostics, timestamp); Dojo's Replay passes the recorded time, and a caller without one gets the worker's clock. The stage keeps times[step], when each such step last used an observation, and hands the step a Clock(now, elapsed) (pipeline_time.py). times commits and rolls back with the kernel states, so an observation that is skipped, or that a Gate before the step holds back, leaves it alone and the next elapsed spans it. An observation not later than the step's previous one is rejected as out_of_order: alignment already refuses regressing readings, so this only meets a checkpoint that is ahead of the stream. elapsed is therefore above 0, and the compiler gives dt the minimum POSITIVE, the smallest positive float, so a / dt draws no division warning. rangeText() reads a bound that close to 0 as above 0.

• prepare.time (TimeFeatures, a row step) appends the features its settings switch on, in the order of TIME_FEATURES: dt; hour in [0, 24] and weekday in [1, 7]; hour_sin, hour_cos, weekday_sin and weekday_cos in [-1, 1]. The calendar follows timezone through zoneinfo, validated when the design is checked; UTC needs no zone database. With elapsed the step has a lead of 1 (step_lead()), since the first observation has no dt. A name that already arrives is a scope issue on the setting that would add it, and a step that adds nothing is refused. timezone is left out of the normalized parameters while no calendar feature reads it. The features carry no slots, and the inspector lists them as From no reading.
• prepare.smooth with method: time_constant (Decay) moves its level towards each value by 1 - exp(-elapsed / time_constant), computed with expm1, and starts at the first value. State is (seen, level), the lead is 0 and the suffix names the seconds: smooth60s, smooth2p5s. Normalization drops window_size for this method and time_constant for the others, so earlier designs keep their fingerprints.

Prepare checkpoints stay at version 1 and gain times, which must name exactly the time-reading steps that have used an observation; a checkpoint from before has none and loads as long as the design has no such step that already ran. PipelineTime.tsx is the editor of prepare.time: one checkbox per setting in the classes of Applies to, the last one locked, and a Time zone field with the browser's zone names as suggestions, shown once the hour or the weekday is added.

prepare.constant, prepare.condition and prepare.set are stateless row operations on the existing numeric feature stream. Constant and Condition append a named feature and reject collisions; a condition emits 0.0 or 1.0 without withholding the observation. Set replaces exactly its named column in place or appends it, preserving all other columns. All expression references contribute to lineage; constants have none. A Set whose expression reads the column it replaces keeps that column's previous and extends its steps; otherwise the column starts at the Set. A name Set adds must match the pattern of computed names (scope issue on name). These operations retain the existing source, feature DAG, detector, policy and evidence contracts.

prepare.calculate (Calculation) is a Set per line. formulas holds up to 32 (MAX_CALCULATION_LINES) lines of name = formula; blank lines and lines that start with # are skipped. assignments() parses each line once per text (lru_cache) and checks names top to bottom against the features that arrive plus the names written above, so a line cannot read a later one. The row step copies the row and writes each feature in order; a failure names the feature it was computing. Columns follow Set: an existing name is replaced in place and continues its trail when the line reads it, a new one is appended. Issues carry parameter_path: ["formulas"], a message that starts with Line N: and a position and fix counted in the whole text, so the editor applies a fix to the right line. calculateLine() in formula.ts gives the editor the names written above the caret and whether the caret is still in the name, where only features are offered.

Aggregate, Formula, Gate and Mute alarms (pipeline_combine.py) work on the whole row, keep no state and have output from the first observation. A recursive-descent parser in pipeline_formula.py reads formulas and conditions, never eval. Its tree serves the compiler, which checks names and kinds and infers ranges by interval arithmetic, and RADAR, which compiles it into closures once per run. An expression holds at most 512 characters, 32 nested levels and 100 levels of operations. Problems raise FormulaError with a 1-based character position and a difflib suggestion, reported as formula issues on the setting. Evaluation raises PrepareFailure for a division by 0, a value outside a function's domain or a result that is not finite, so the observation is skipped like any other Prepare rejection; and and or short-circuit, so a guard such as x > 0 and log(x) > 1 holds. The compiler's values check derives, from the ranges of their inputs, the values a Math step, a percent Change, a Formula or a Gate will skip: a warning when some are skipped and an error when every observation would be, or when a condition is never true. It also warns about a Condition whose flag never changes and a Set that reads no feature, and constant_feature warns where a feature whose minimum equals its maximum reaches a detector or a learned step that uses it. Seeded fuzz tests check that any text parses or fails with FormulaError, that evaluation fails only with PrepareFailure, that every value lies in the inferred range, also behind guards of the shape the analysis narrows on, and that a value outside a function's domain is never skipped without a warning.

Further functions take numbers only: sign, floor, ceil, round (a half away from 0), mod (floored, with the divisor's sign; a divisor that can be 0 warns like a division), sin, cos, atan2, hypot, mean, sum and pi(). A bare pi is a feature name, and an unknown one offers pi(). Each has a range: sin and cos [-1, 1], mod between 0 and its divisor, atan2 by the half-planes its point can lie in. Left of the origin a y of 0 gives pi or -pi by its sign, and -0.0 counts as 0 in a range, so only a y that stays off 0 narrows there.

The numeric if(condition, when_true, when_false) function evaluates only its selected branch. Its first argument must be a condition and both remaining arguments must be numeric. Compiler analysis visits reachable branches, narrows simple guarded feature ranges, and keeps conditional domain failures as warnings when another branch can succeed. All referenced feature names are still checked, including names in a currently unreachable branch. The same short-circuit rules apply to conditions used by Gate and Condition.

A Formula with apply_to is a formula for each feature (FormulaForEach): it evaluates the expression once per chosen feature with x bound to that feature's value, always against the incoming row, and either adds <feature>_<suffix> (suffix formula by default) or replaces each value. each_scope checks that the features arrive, that no incoming feature is itself named x, and that the added names are free; name with apply_to is a parameter error. The compiler analyses the expression per feature, so ranges, lineage and values issues are per added feature, and it warns when x is never read. A plain Formula that reads x without such a feature arriving is a formula error at x. Formulas without apply_to normalize as before, so their fingerprints are unchanged.

A Gate whose condition is false withholds the observation: the steps before it commit their states, the steps after it do not run, and PrepareStage.run returns Withheld("held_back", gate). Both detectors return a held_back result, which is neither learned, scored nor persisted and is no coverage gap. The activity tracker counts it as progress, so a Gate that stays closed does not read as stalled inference. A rejection anywhere in the pass wins over a closed Gate. Stage readiness follows each step's own update count, so observations held back before a window do not fill it: a step needs the observations its predecessors still need, then its own warm-up. The stage state counts held_back; checkpoints without the count load it as 0.

A Mute alarms step (prepare.mute, Mute in pipeline_combine.py) passes the row on unchanged and reads its condition. The compiler checks the condition like a Gate's with the verdicts turned round: always true is an error, because no alarm would ever be raised, and never true a warning. PrepareStage.run sets muting when a step's condition holds and the observation reaches the learned stage; an observation that is withheld has no alarm to mute. The stage state counts such observations as muted, revert takes the count back, and checkpoints without it load it as 0. A condition that cannot be read rejects the observation as any step's failure does.

The detection rule sees a muted observation as any other, so calibration, adapting thresholds and consecutive counts go on. Each service asks _raised(fired) (PipelineExecutionMixin) where it decides an alarm: the adaptive and mixed services on the score threshold and the boundary service on its judgement. While the stage is muting, a fired alarm is not raised: the result is no anomaly, the context carries alarm_muted, and the evidence row keeps alarm false and muted true (migration 030_evidence_muted.sql). Those rules judge every observation anew, so an alarm that still stands when the mute ends is raised then. A martingale tracker fires once, where its statistic crosses the threshold, so MartingaleTrackerController.update(p_value, muted=True) holds a muted crossing in _held_alarm: it is raised at the first observation that is not muted while the statistic is still above the threshold, and dropped when the statistic falls back first. The tracker's result carries alarm_muted at the crossing, and pickles from before the field load it as false. A Dojo replay that compares calibration methods records no method alarm for a muted observation. tests/pipelines/test_mute_alarms.py holds each service to these rules.

A failure in the learned stage costs one observation too, when no learned state changed. Aberrant's GraphPipeline.score_one only reads committed state, and learn_one rolls every learner back when a step fails before the detector learns; both raise GraphExecutionError. The adaptive detector turns that into StepRejected(node, "error", message), the ObservationRejected that PrepareRejected also extends. A GraphStateUncertainError, raised when a learner without a transaction, such as the detector, may have changed, still fails the run. In the static strategy BatchFeatureGraph raises NonFiniteFeatures when a fitted step returns values that are not finite: fatal while fitting, and a non_finite skip while scoring one observation, which only reads fitted state. Before raising, _skip() calls PrepareStage.revert(), which restores the states the observation's Prepare commit replaced, so a skipped observation changes no state anywhere. The step's diagnostics count it under rejections.

Each FeatureDescription in a validation report says where it comes from. previous names the feature it continues at the step before, as node.name, and steps lists the steps that changed the feature or its name. Readings have no previous, and neither have the features an Aggregate, a Formula or a projection makes; their steps start with that node. Neither field changes a fingerprint, and the report's budget pass drops them along with slots. The composer's Features view follows previous back from the detector's input to draw one row per feature. Its + places a step with rowPlacement(): a Prepare or Combine step on the wire into the first learned node on the feature's path, or into the detector when there is none, and a learned step on the detector's input wire, each scoped with apply_to to that feature.

A valid report's ready_after maps each step of the Prepare stage to the observation, counted from 1, that gives its first output: one plus the leads along its longest chain. The inspector shows it as Ready after N observations, and as at least N after a Gate, because held-back observations do not count; the start-up timeline says at least for the same reason. A formula issue can carry a fix: the characters start to end and the text that replaces them, such as the closest known feature for an unknown one, written as a formula writes it. The composer offers it as Use …. The formula editor suggests the features that arrive at the step, the functions and, in a condition, and, or and not.

Validation issues carry a severity. Errors block compilation, publishing and Apply; warnings use the same shape, appear on the step and in the status chip, and do not. The inspector keeps the last checked features while an edit is checked, so feature choices do not jump between checks.

A design may combine up to four detectors (MAX_DETECTORS), each with one input. The policy takes one connection from each, and its fusion setting, Combine detectors in the composer, says how their scores make one: max (Strongest, the default) or mean (Average). PipelineValidation.detector_ids orders the detectors by node ID, detector_id is the first of them and detector_inputs maps each to the node it reads; a policy with several inputs has no features of its own. fusion is left out of the normalized parameters at max, and with a single detector, so existing designs keep their fingerprints. The compiler reports a design without a detector, one with more than four and a detector the policy does not read. derive_monitoring_config describes the first detector.

Schema 2: learning modes and calibration¶

Schema 2 (pipeline_versions.py) describes a design without a lane. Each component says how it learns (PipelineComponent.learning: none, batch, online or pretrained), and a calibration step sits between the detectors and the alarm policy. Detectors give scores; the calibration step turns them into p_values for the policy. calibrate.fixed_split is what both lanes did: the learned steps freeze and the calibration window gives conformal ranks. It owns fusion; the policy keeps its own parameters, including the quantile. Every other calibration step keeps adapting (see below). A forecaster step gives forecasts, which only a calibration step reads.

A forecaster step is a detector that scores one feature by how far it misses the forecast made for it. FORECASTER_IDS in models/adaptive.py lists them; each is ABERRANT's residual_detector around one forecaster (seasonal_naive, holt_winters). A step's settings are the forecaster's own plus lead, how many updates ahead it forecasts, and score. aberrant_model() nests them into the residual detector's configuration, binds the step's single feature as the target and sets the forecaster's horizon to the lead. Warm-up is two seasons plus lead - 1 updates and must fit the training window. The composer lists them under Forecaster (stepFamily), together with pretrained forecasters.

A forecaster step's port is forecasts: the score, and with it the forecast the observation was scored against. Any calibration step reads it as it reads scores (_check_ports, takes() in the editor), beside other detectors or alone. A scored result then says what each forecaster step expected, in context["adaptive_stream"]["forecasts"] (forecast_band.py): one entry per step with detector_id, target, lead, expected (the forecast's point at that lead) and slot, the reading the target is when no step changed its values, or null. lower and upper are the calibrated interval, the target values whose score stays at or below the boundary the observation met (aberrant.forecast.scores.interval), so a value outside it is above the boundary. They are null without a boundary in the step's own score units: detectors combined into ranks, or a boundary that is still infinite. A run reads the forecast after the step scored the observation and before it learns it (_expect), because a pretrained step fetches a forecast its service missed while it scores. The key is absent before a forecast exists and outside the operational phase.

Evidence stores the entries as forecasts (migration 026; _normalize_forecasts in result_records.py keeps an interval only whole), and the catalogue API returns them with each evidence row. In the charts, buildDetectorInputSeries puts each forecast on the line of the feature it was made for, which it finds by detector_id and target among the row's detector_inputs. The feature's pane then shows the forecast dashed in the feature's colour and the interval as a band: two stacked lines, the lower bound and the width above it, so only the space between the bounds is filled. A Dojo replay does not store forecasts yet.

A pretrained forecaster is a forecast. component with learning pretrained. Its weights live in the forecast service (off-key-forecast), never in a RADAR worker, so it is not an ABERRANT component and not in adaptive_catalog.json: PRETRAINED_FORECASTERS in models/adaptive.py describes it beside the generated models, keyed by its component, and detector_component() gives the component of either kind. The first is forecast.timestral, with a checkpoint (TIMESTRAL_CHECKPOINTS, the ones the service offers), a context_length (how many recent values the model reads), a minimum_context (how many values must have arrived before the first forecast), and lead, score and min_scale as for the other forecasters. It reads one feature and no step that learns. It scores and then takes in every observation, as a detector that learns continuously does, so it runs wherever one runs: alone or beside detectors of either kind, under the fixed split with the calibrated score threshold or under a calibration step that keeps adapting. The conformal alarm policy does not run on it.

In RADAR, off_key_mqtt_radar/forecasting.py gives the service ABERRANT's forecaster interface. ServiceForecaster keeps the recent values and asks the service for quantile forecasts; forecast_from_quantiles() turns them into ABERRANT's Forecast, with the median as the expected value and the interquartile range as the spread. ServiceResidualDetector is ABERRANT's residual detector around it, and resolve_service_detector() answers what RADAR asks of a resolved ABERRANT detector: configuration, capabilities, fingerprint and build(). A worker finds the service through FORECAST_API_URL and FORECAST_API_TOKEN. TACTIC puts the address (TACTIC_FORECAST_API_URL) into the environment of every RADAR worker and preflight job, outside the configuration fingerprint; the token reaches a production worker as a Swarm secret. Preflight builds the step without calling the service, but a deployment that names no service fails it there, on the step, with the issue forecast_service. Forecast service describes the deployment, the pinned weights and their sizes.

The HTTP client is shared: ForecastClient in off_key_core/utils/forecast_client.py. TACTIC uses it for the composer, in services/pipelines/forecast_models.py: GET /v1/forecast-models answers the service's models with their download state, and whether the deployment has a service at all (not_deployed) or one that does not answer (unreachable); POST /v1/forecast-models/downloads starts a download and needs manage_monitors. In the frontend, useForecastModels loads that list for the workbench and asks again every three seconds while a checkpoint downloads. Where the deployment has no service, withForecastService marks the forecast. components unavailable, which addBlocker returns, so no menu adds the step; a service that only does not answer leaves it on offer. PipelineForecastModels lists the checkpoints in the step's settings.

The first value of a run asks the service to have the checkpoint ready, which starts a download if the weights are not there yet. Until a forecast exists for an observation, that observation is rejected (ObservationRejected): lost coverage, not a normal score. A run does not learn a rejected observation, so the adapter keeps the value itself and the forecaster's series has no gap. The weights never change, so a forecast the service missed is made later from the same values, and scores after an outage are the ones an undisturbed run gives. A service that hangs is left alone for 15 seconds after a slow failure. A service that does not offer the checkpoint ends the run (FatalExecutionError), because no later value could be scored.

Schema 1 designs are read as schema 2: upgrade() moves the policy's inputs onto a calibrate.fixed_split step and its fusion onto that step, and drops the stored strategy. A design holds 65 nodes and 129 edges, the calibration step and its edge into the policy included; a schema 1 design is held to its own 64 and 128 before the step is added. The serializer writes downgrade(), the schema 1 form with the strategy its learned steps share, whenever a design has one: a single fixed split in front of the policy and learned steps that all learn one way. Stored designs and fingerprints therefore do not change. The editor reads and writes schema 2: every response that carries a design for the editor (EditorDefinition) serializes editor_form(), the upgraded design without a strategy, and the catalog serves schema 2 components and templates, with the calibration step, its ports and each calibration method's family, coverage and reference. Validation reports list the calibration step like any other.

The compiler checks schema 2 designs by learning mode instead of by lane (_validate_learning):

• learning: a step reads a learned step that learns differently. A batch detector reads batch or stateless steps, an online one online or stateless steps, and a pretrained or rule detector stateless steps only.
• calibration: the martingale policy needs exact p-values, so no step in front of it may keep learning after calibration (no online step). A policy also has to fit its calibration step: the martingale and the score threshold need the fixed split, the exceedance rule a step that adapts.
• calibration_window (warning): the calibration window is too short for the alarm rate of a step that adapts.

The rules leave one policy for a design. A calibration step that adapts alarms by the exceedance rule. Under the fixed split, detectors that are all fitted once alarm with the martingale, and a design with a detector that takes in every observation (one that learns continuously, a pretrained forecaster or a rule detector) alarms on the score threshold; the compiler reports the threshold behind detectors that are all fitted once as calibration.

A rule detector is the component rule.score (RULE_DETECTOR). It learns nothing, so its learning is none, and it scores every observation as it arrives, so streams() in pipeline_versions.py is true for it as for an online or pretrained detector: the compiler checks its input like theirs, derived_strategy() counts it towards the adaptive lane, and RADAR runs it among the streaming detectors. Like a pretrained forecaster it is RADAR's own and not in adaptive_catalog.json: RULE_DETECTORS in models/adaptive.py describes it beside the generated models, with no settings and exactly one feature, bound as key. resolved_detector_config() resolves it through rules.resolve_rule_detector() to a FeatureScore, an Aberrant model whose score_one returns that feature's value and whose learn_one does nothing, so it sits at the end of an Aberrant graph like any other detector. In the compiler _learned() lists it as rule: a step that learns can't feed it (learning), and the martingale policy does not run on it (calibration). A run whose detectors all follow a rule (_learns_nothing()) starts in calibrating instead of warmup and reports a warm-up requirement of 0; beside a detector that learns, the run warms up as that detector needs. TACTIC's available_models(), which /v1/detectors serves to the setup form, leaves it out: a rule needs a step before it, and the rollback rehearsal expects settings for every model that field discovery lists.

No design stores a lane. PipelineDefinition.lane is derived: the strategy all a design's learned steps share, or none. Routing and the schema 1 written form use it; the compiler does not.

derive_monitoring_config follows the policy: the martingale gives static baseline settings, the score threshold adaptive stream settings named after the first detector that learns continuously. RADAR picks its service the same way (detector_services.detector_service_class): a threshold run with a detector fitted once runs MixedDetectionService. It is the adaptive service with a BatchBranches beside its Aberrant graphs. While the online detectors warm up, the others buffer the latest training window; during calibration the online detectors score and learn while the others buffer the calibration window. Then the run is fitting: the detectors fitted once and the steps they read (BatchFeatureGraph(plan, detector_ids)) fit in the background on those windows, with split conformal calibration as in a static run, and the online detectors keep learning without evidence. When the fit publishes, each detector's calibration scores form one column of the RankFusion, and the run alarms as a combined adaptive run does. A Dojo replay waits for the fit, so it scores every observation after calibration. Checkpoints keep the buffers or the fitted detectors (batch) beside the adaptive state and refit when they were taken while fitting.

Calibration steps that adapt¶

calibrate.<method> runs a method of aberrant.calibration: rolling split CP, ACI, DtACI, AgACI, the quantile tracker, conformal PID, NexCP and eight more. calibration_catalog.json describes them (name, reference, family, coverage, parameters). RADAR generates it from the installed aberrant (python -m off_key_mqtt_radar.catalog --write, checked with --check), so the control plane imports no numerics; pipeline_calibration.py builds one parameter model per method from it. A step takes alpha, the target alarm rate, the method's own parameters and fusion. The method's window is the design's calibration window. The editor offers them as the Calibration method of the calibration step.

There are three stages, and each claims only what it can:

1. Detectors give scores.
2. The calibration step issues a boundary for each observation before it sees the score, then learns the score. The calibration window seeds it. What a method holds is the long-run share of scores above the boundary, at alpha. That share counts every score above the boundary, anomalies included, so it is not a false alarm rate, and a method that adapts also gets used to a lasting anomaly. Methods of the Bayesian family hold no such rate (coverage: none).
3. policy.exceedance, the detection rule, alarms once consecutive scores in a row are above the boundary. With more than one the alarm rate falls below alpha, but not to alpha to that power, because exceedances come together.

The martingale and the score threshold stay with the fixed split: only its p-values are exact, and only it freezes a threshold.

A boundary that is a quantile of the window (split, aci, dtaci, agaci) needs ceil(1 / alpha) - 1 calibration observations, or it sits at the largest score seen. Several detectors need that for any method, and ceil(k / alpha) - 1 for k detectors under max, because each reaches the top of its window about once in n scores. The compiler warns below that (minimum_window).

Comparing calibration methods in the Dojo¶

A Dojo run can try the other calibration methods beside its own (compare_calibration). TACTIC stores the plan with the run (calibration_comparison in dojo_runs.config, from models/dojo_runs.py): the run's own method, the methods to try in order, and the alpha and consecutive they share. The replay installs a CalibrationComparison (calibration_comparison.py) on the detector service. The service hands it the calibration window where its own step starts, and then the detector scores of every operational observation after the detectors took it in, so each method sees exactly what a run with that calibration step would. Several detectors meet as in a run: against their latest scores for a method that adapts (RollingRanks), against their calibration scores for the fixed split (RankFusion). tests/radar/test_calibration_comparison.py holds each compared method to the alarms of a replay whose design has that step.

A monitor never has a comparison; only Replay sets one. Each result carries method_alarms, one bit per method in the plan's order, stored in dojo_run_results.method_alarms (migration 027) and null before the run is operational. A method that raises leaves the comparison and the worker writes the reason to calibration_comparison.failed when the run ends; such a method keeps the counts of what it found and gets no share or rate, which would be over results it never judged. TACTIC scores each method on read like a run of its own (DojoMetrics.calibration), so a moved label re-scores the methods too. All 15 adapting methods together cost about 2 ms per observation, mostly the Bayesian ones, which is why the comparison is asked for per run.

The editor without a lane¶

The composer has no strategy switch and no strategy.ts. learning.ts holds what replaced it:

• A card says how its step learns (learningLabel: Learns continuously, Fitted once, Pretrained, and No training for a detector that learns nothing), and the add-step menu lists every step with that tag. learningBlocker() refuses a step in front of a detector that learns differently, with the compiler's reason, before the step is added.
• Detector model lists the models by how they learn. Choosing one that learns differently (changeDetector) moves the learned steps only that detector reads to their counterparts (COUNTERPARTS) and fits the policy.
• The calibration step has its own card between the detectors and the detection rule. Calibration method lists the methods by family (conformal, Bayesian) and says what the chosen one holds (coverage), with its reference. changeCalibration() keeps fusion, takes the new method's defaults and fits the policy.
• Detection rule is the policy. fittingPolicy() names the one policy the design allows; the picker marks the others (does not fit this design). After a change that moved steps or the policy, a status line says what changed (changeNotice) and Undo reverses it in one step.
• Service settings name their first window after what the detectors do with it: Training observations, Detector learning updates before calibration, or both for a design that mixes them. startupPhases() takes the same view. A design whose detectors all follow a rule has no such window (learningWindow() is none): the setting is left out and the estimate starts with calibration.

normalizeDefinition() drops a strategy and stamps schema_version: 2 on what the editor sends. A layout saved before the calibration step had a card gets one at the policy's place, with the policy and everything right of it moved one column over (placeCalibration). RADAR reports calibration under the policy, so the run view shows the policy's state and diagnostics on the calibration card too.

BoundaryDetectionService runs every design with such a step: detectors fitted once, detectors that learn every observation, or both. It is the mixed service with a ScoreBoundary (boundary.py) where that one freezes a threshold. Warm-up, the calibration window and the background fit are the same. The calibration window's scores then seed the boundary, and each scored observation is judged against the boundary issued for it and then learned by the detectors, the boundary and the rule's count, in that order; an observation a detector fails on moves none of them. Results are adaptive_stream results with the boundary as threshold, so evidence rows, charts and the Dojo read them as a threshold run's. derive_monitoring_config names the first detector that takes in every observation; when every detector is fitted once it gives static baseline settings for the windows and the seed, while the results stay adaptive_stream. Checkpoints keep the boundary and are refused when the design's calibration step or rule changed.

One detector's raw score is the method's score. Several detectors meet as ranks, as under the fixed split, but against each detector's latest n scores (fusion.RollingRanks, n the calibration window) so the ranks follow the detectors. A share alone stops at 1, where a score just above the window and one far above it would tie and a boundary that has reached 1 could never be exceeded. A score above its whole window therefore adds how far above it is, in units of the window's range. The surprise reported per detector stays a share.

fusion.RankFusion combines detectors by rank, so detectors on different scales need no tuning. A detector's surprise for a score s is the share of its n calibration scores strictly below it, u = #{i : c_i < s} / n. Strongest combines the counts by their maximum and Average by their sum; both stay integers, so ties are exact, and value() maps a combined count back to [0, 1]. Counting strictly below keeps one detector's rule: with one detector, or the same detector twice, combined > threshold_count(q) holds exactly when the score exceeds the method="higher" quantile q of the calibration scores.

The static strategy makes the combined count a conformal p-value. p_value() ranks the new observation among the calibration observations: on each detector a calibration observation counts the calibration scores below it, plus one when the new score is lower, and the new observation counts the calibration scores below its score. Without a change these n + 1 combined counts are exchangeable, so the new observation's rank among them, with ties broken by a seeded generator, is uniform on 1 to n + 1. The p-value, rank / (n + 1), takes the same values with the same law as a single detector's, so the martingale trackers and their thresholds work unchanged. A test draws 10,000 calibrations, with continuous and with tied scores, and checks that law with a Kolmogorov–Smirnov test. FusedStaticPipeline keeps a ConformalDetector per detector, fitted on that detector's training and calibration features; BatchFeatureGraph fits each shared step once and returns every detector's input (fit_transform_all). The tie-breaking generator is saved with the checkpoint.

The adaptive strategy builds one Aberrant GraphPipeline per detector from the nodes it reads (ExecutionPlan.reads), so each detector learns its own copy of a shared step; the copies see the same observations and stay equal. Every detector scores an observation before any of them learns it. A failed score skips the observation for all of them, and so does a learning failure of the first detector, before anything has learned. A later detector that fails cleanly misses that one observation and counts an error rejection; GraphStateUncertainError still stops the run. Warm-up ends when every graph is ready and the slowest has made the configured updates. Calibration keeps each observation's scores; when it is full, RankFusion freezes them, the threshold count is the policy's quantile (method="higher") of the calibration observations' combined counts, and an observation alarms when its combined count exceeds it. The anomaly score is the fused surprise. Diagnostics report a shared step once, under the first detector that reads it (ExecutionPlan.owned), and the model nodes per detector.

A combined run's checkpoint holds pipelines, detector_configs and detector_fingerprints in detector order in place of pipeline, and a restore refuses any difference. Admission counts the rank fusion, five n × K arrays of doubles plus the lists an adaptive run collects, and each adaptive copy of a learned step's state. Preflight builds every detector and reports a failure on its node. Results carry each detector's detector_id, score and surprise in their context, and processed_features holds each detector's features under its ID. Evidence stores them as detector_results (migration 015), while p_value or anomaly_score holds the fused value. Runs with one detector write an empty list and keep their results, checkpoints and diagnostics. detector_inputs (migration 017) keeps the features each detector scored, after every step before it, so charts can plot derived features: one entry with a null detector_id for a single detector, or one per detector ID for a combined policy, at most 64 features each.

Activation and recovery¶

Activation persists reservations before remote work. Claims cover the union of predecessor and successor inputs and reserve one successor capacity slot. Every form start, graph activation, collection mutation and release consults this authority under the collection configuration lock.

The controller runs credential-free constructor preflight in the exact target RADAR image digest before retiring a predecessor. The report binds the executable, catalog, collection and bindings to that digest. It reports known numerical allocation costs; it does not prove data-dependent fitting or predict every model allocation. Preflight has separate workload identity, bounded concurrency and timeout, and cleanup ownership. A fresh nonfailed heartbeat confirms activation; warm-up remains a run stage. Activation supervision uses separate bounded capacity for preflight, remote transitions and database checks, each with independent sessions and durable leases. Orphan cleanup runs separately. Configure constructor capacity with TACTIC_PIPELINE_MAX_CONCURRENT_PREFLIGHTS (default 2); remote transitions and checks each use TACTIC_PIPELINE_MAX_CONCURRENT_TRANSITIONS (default 8). Oldest updated operations are considered first. The database records the first healthy heartbeat receipt, preserving timely startup evidence across later heartbeats and delayed controller checks. A first heartbeat received after the deadline cannot establish successful startup.

Remote uncertainty retains reservations until shutdown is confirmed. Cancellation before retirement preserves the predecessor. Cancellation after retirement creates no implicit rollback. Stopping a predecessor also cancels its pending replacement. Reconciliation resumes durable operations after controller restart. The durable sequence is requested, preflighting, validated, retiring, starting, awaiting_heartbeat, then succeeded; cancellation/failure uses cancelling until cleanup is confirmed. A controller token and expiring lease fence every resumed phase and error transition after remote I/O. Stale replies cannot move a newer activation back to an earlier phase.

Terminal lifecycle transactions lock checkpoint, run and service records in that order, including service deletion's checkpoint cascade. Heartbeat acceptance and fatal-run recording serialize their run/service updates so a stale healthy heartbeat cannot overwrite a committed fatal state. Collection stops, deletion and reconciliation share complete_stopped_runs beside the lock helper. It retains failed/cancelled outcomes, error details and the first terminal timestamp across repeated cleanup.

Deployment reserves maintenance before quiescing writers. It refuses a pending activation, and active maintenance prevents creation and handoff in both HTTP and background paths. Cleanup remains permitted. Release snapshots preserve graph definitions, bindings and run lineage; a software upgrade creates a fresh run when numerical compatibility requires it.

An uncertain mutation crosses resilience and message processing as a fatal error. The worker stops graph execution, blocks checkpoint export including shutdown, preserves the last signed snapshot and records failed-run coverage. That failed generation cannot be admitted again. A graph revision rollback starts a fresh run; same-run task recovery restores only compatible signed state.

Bounded inspection¶

Diagnostics observe the actual serial scoring and learning passes. They perform no additional inference, MQTT subscription or Docker inspection. At most one observation envelope is sampled per second and retained, with eight input/output values per node. Unexecuted nodes carry a reason and never reuse previous values. The serialized snapshot is capped at 128 KiB, with status and counters taking priority over previews. Static fitting appears as a batch summary. Adaptive nodes also report committed update counts, observed/required history in the component's own units, and their immediate blocking prerequisites. A zero required count means no quantified minimum was supplied; the ready flag remains authoritative. Committed counts include restored history, while collector call counts can restart after recovery. Readiness reasons and errors retain at most 128 and 256 UTF-8 bytes respectively, with the same ceilings applied after JSON escaping.

The ownership-fenced heartbeat publishes diagnostics every five seconds during healthy operation. One member-scoped workspace request returns service summaries and optional selected-run diagnostics. The browser pauses hidden work and displays snapshot age separately from evidence persistence freshness.

V1 admits at most 64 executable nodes, 128 connections and 100 features at any intermediate boundary. Bound executable configuration has a 96 KiB serialized budget to stay below the worker environment transport limit. Known allocation admission and existing worker CPU/memory hard limits apply independently. Adaptive PCA admission includes each initialization row's NumPy array header, the old and new pointer lists retained during a learning transaction, the dense initialization matrix and reduced SVD outputs. These costs use the target worker's Python/NumPy object sizes; per-projection peaks are summed conservatively. The estimate also covers explicit online update matrices. Detector state, native linear algebra workspaces and other process allocations remain additional costs, so passing admission does not promise that every workload fits the hard limit. External validation reports, including read-only run metadata, have a 512 KiB serialized budget. Large reports omit provenance first, then whole node tables, with an explicit metadata_budget notice and features_complete / provenance_complete flags. A selected_node_id on draft validation or the run query preserves complete ordered feature names for that node and its immediate parents; feature choices are never truncated. Unknown selections return an explicit issue and no stale metadata. Invalid oversized definition echoes can be omitted separately, and individual errors are bounded to 1 KiB of UTF-8 text. The compiler rejects normalized definitions above 96 KiB before publication; bindings must still fit the complete execution budget at activation.

Library lists return summaries with server-side search, archive filtering and pagination (limit, offset; default 25, maximum 100). Design details include at most 25 revision summaries. Revision pages use before revision numbers; individual immutable definitions load on selection. Run history pages contain summaries and a next_monitor_id continuation, resolved in one depth-bounded recursive query per page. History never repeats executable definitions or binding snapshots. A selected run includes its design ID so opening its draft needs no library scan. /pipeline-metadata and immutable revision /metadata requests return only the selected node's and its parents' compiler tables, without run history or another definition echo. These member-readable inspections remain available with authoring disabled.

The gateway assigns compilation and automatic metadata inspection a separate member-scoped budget of 120 requests per minute. They cannot consume the 30-per-minute mutation budget or the 120-per-minute workspace polling budget.

Performance evidence¶

backend/scripts/benchmark_pipeline_runtime.py compares real numerical graph execution with observation disabled and enabled, without MQTT, PostgreSQL or health-publication I/O. Both runners receive identical seeded observations and must produce exactly identical scores and alarms. Short, alternating batches of ten observations balance execution order and slow host/VM speed changes within each trial. Whole-run blocks initially showed large speed drift inside identical forest kernels; their timings are not used to infer observation overhead.

The 2026-10-16 paired probe used an AMD Ryzen 5 7530U, WSL2 Linux x86-64, Python 3.14.8 and Aberrant 1.3.0. Docker limited the process to one CPU (cpu.max=100000 100000) and 1 GiB; twelve logical CPUs were visible. OPENBLAS_NUM_THREADS, OMP_NUM_THREADS, MKL_NUM_THREADS and NUMEXPR_NUM_THREADS were all 1. Each of three trials used 512 warm-up observations, 32 calibration score/learn calls and 100 timed observations. The collector used its normal one-second sampling interval, including committed-update and readiness metadata. The report records hardware, thread controls, cgroup bounds, parameters and raw trial timings.

The representative graph has eight inputs and nine nodes: six selected features pass through online standard scaling and three-component uncentered PCA, two features bypass processing, then both branches merge into a seeded 100-tree online isolation forest (window_size=512, max_leaf_samples=16, n_jobs=1). Its frozen quantile policy and output are included. The cheap comparison uses one input, standard scaling and a moving-average detector with a 128-value window.

These are medians of aggregate trial timings. All three paired representative trials measured lower CPU time with observation: no positive cost was measurable at this probe's precision. Those negative differences reflect residual host variation, with no claimed numerical speed improvement. Individual trials without observation ranged from 42.763 to 182.347 ms/observation, so this provides scoped acceptance evidence rather than a precise overhead estimate. The <=10% target applies to this documented representative workload. Cheap detectors can have much larger relative overhead; their measured absolute CPU increase was 0.110 ms/observation. Absolute throughput varies with host load, and these measurements are not an end-to-end throughput guarantee.

Two separate maximum-size fixtures ran 4,000 scored/learned observations each after warm-up. One has 64 nodes, 128 edges and 100 source features with shared branches and reconvergence; the other carries all 100 features through a 64-node, 63-edge chain. Every observation was captured for this retention probe, a stronger allocation load than normal sampling. Both retain stable node keys, eight preview values per side and one observation envelope.

Retained sizes count collector-owned reachable Python objects once, excluding the numerical models and allocator overhead. Snapshots stay below 131,072 bytes. The memory capture, sampled during the final fixture, recorded a cumulative whole-container cgroup memory.peak of 147,513,344 bytes (140.68 MiB), including all workloads up to that point; this is not a per-fixture measurement. Separate maximum-node tests combine 100-character IDs, 128 dependency links, multibyte/control-character readiness reasons and errors, bounding both UTF-8 and JSON-escaped text. The fixture tests also verify one learning update per eligible node and exact observation-on/off score equivalence. Timing-only reruns use --timing-only; a full run also repeats the sustained bounds probes.

The prepare workload despikes and smooths eight inputs, adds a change and a 60-value rolling standard deviation for four of them, and scales the 16 features into a moving Mahalanobis distance (window_size=128). Its Prepare stage needs 80 observations before its first output. A 2026-11-01 timing-only run with the default settings used a four-CPU cloud container (Intel Xeon at 2.10 GHz, Linux x86-64, Python 3.13.18, Aberrant 1.4.0, one thread per numerical library, no cgroup CPU limit).

The detector dominates; four Prepare steps over eight inputs cost about 55 µs per observation. The Prepare evidence keeps the raw trials of that run.

The runtime evidence preserves the raw trials, component parameters, environment and retention results. Separate browser evidence records editor rendering checks.

Rollout¶

The production rehearsal restores the current release's backup, runs an older pipeline-compatible build, then returns to the current build. Each cutover checks the actual application and worker image digests, fresh graph scoring, bindings, layout, retained run history and single-worker ownership with authoring disabled. The test-only reference is immutable commit 197cc239119b8bfb7df1b6921828dbb690adcd7f, the commit that adds the composer's saved blocks and their table through migration 031 (pipeline_blocks). It shares the current migration ledger, the ABERRANT 1.5.0 pin, frozen lock, generated catalog and field discovery metadata, so an already-open new browser keeps its controls while the backend rolls back. It also serves the current REST paths and field names, so the rehearsal calls and reads both releases the same way. The test uses the current API directly, without compatibility adapters or response rewriting. Monitors are started on the current release, by reading id. The previous reference, efe30f4a, has no migration 031; f02a9201 has no migration 030; 535b29cd has neither migration 028 nor 029; 934f8f93 has no migration 027; cbe2d839 before it has no migration 026, ad919d98 returns a design with its lane and without its calibration step, where the current release returns it as the editor reads it, 386eb796 lists no Timestral step in field discovery, 7194e94f has a catalog without the forecaster steps, and b03157ec locks ABERRANT 1.4.0, so the rehearsal accepts none of them. Releases before migration 031 refuse a database that has applied it. Rolling back past this schema boundary means restoring a verified backup, as recovery describes. The binary rollback floor for this release is ABERRANT 1.5.0; migration from earlier versions is checked separately and incompatible checkpoints relearn. Its migration ledger, numerical catalog and locked Aberrant version must match the current source before it can build. Other locked dependencies may differ, as they would in a real rollback: each release calibrates its own run, and detector checkpoints refuse a changed runtime. This fixture is not an eligible production release. Changes to those contracts require review of the rollback reference. CI retains nonsecret source/image identities and the asserted graph invariants as its compatible-binary-rollback artifact.

The production rollback rehearsal passed both integration tests with zero skips in 625.685 seconds on 2026-10-16. Its retained artifact records all three cutovers and verifies the deployed image digests and graph invariants; the following browser phase is a separate gate.

The initial compatibility release shipped with authoring off. The visual workbench is now enabled by default with TACTIC_PIPELINE_AUTHORING_ENABLED=true; numerical, real infrastructure, lifecycle, browser and resource gates must pass before release. An explicit operator override to false is preserved. Disabling authoring blocks new designs and workbench activations while the setup forms, existing graph runs, inspection, stopping and release recovery remain supported.

The software rollback floor is a pipeline-compatible release. Older binaries cannot interpret the migration ledger or graph configurations. Use a compatible binary, or the reviewed release database backup/restore procedure. Never erase migration history to make an old binary accept a newer database.