Testing and debugging¶

This page is the operational test and triage playbook for the current stack.

When to use this page¶

Use this page when validating local changes, checking runtime health, or diagnosing user-facing failures across API, telemetry, and monitoring.

Audience¶

• Developers debugging service behaviour.
• Operators triaging local or cluster-like environments.

Test commands¶

Backend¶

uv run --project backend ruff check .
uv run --project backend ty check --project backend
uv run --project backend python -m pytest -q

For a faster local loop, add -m "not slow" to skip the boundary-size runtime tests; CI always runs them.

For branch coverage, add --cov to the pytest command. The root .coveragerc measures every backend package; the database, collection and Swarm lanes skip unless their environment is set.

Backend tests live under backend/tests/, one folder per area: core, database (schema, migrations, DB sync), gateway, tactic, radar, proxy, pipelines, simulator (local broker fixtures) and contracts (cross-service, deployment and documentation contracts). Shared fakes and fixtures live in backend/tests/support/; test modules never import each other.

Expected result: lint and the type check exit with status 0, and tests complete without failures. The type check covers the packages listed under [tool.ty.src] in backend/pyproject.toml; RADAR joins once its lifecycle attributes are typed.

RADAR production integration¶

The radar job of the CI workflow runs as listed in Contributing and CI. It tests this run's production RADAR image (the images job's verified build, or the digest publish pushed on main), uses Python 3.14 and frozen dependencies, and selects the TimescaleDB image pinned by the production release manifest. Its isolated Ubuntu runner initializes Swarm, runs real MQTT/database/task replacement regressions and cleans labelled fixtures within a 45 minute job. Production release selection requires the commit's CI main-push run to succeed, which includes this job, and ci-gate requires it before merging. The dedicated job requires a nonempty JUnit report with zero skipped cases. Missing integration prerequisites therefore fail that job; the general backend suite still supports optional integration skips for local development.

Lanes select their tests by marker, never by file list:

Each lane fails on an empty report or a skipped case (require_executed_tests.py). test_suite_layout.py fails when a module reads a lane's environment gate without carrying that lane's marker, so a new integration test cannot silently skip everywhere.

To reproduce on a dedicated local Docker engine, initialize Swarm and provide a disposable PostgreSQL server whose test role can create and drop databases. Set its host URL in TEST_SCHEMA_DATABASE_URL; tests create uniquely named databases. The workflow contains the server bootstrap and readiness commands. Then run:

uv sync --project backend --all-packages --all-groups --frozen --python 3.14
export TEST_RADAR_DATABASE_IMAGE="$(python3 .github/scripts/release_image.py postgres)"
export TEST_RADAR_SWARM_IMAGE=off-key-radar-integration:local
export TEST_RADAR_DOCKER=1
docker pull eclipse-mosquitto:2.0
docker build -f backend/Dockerfile -t "$TEST_RADAR_SWARM_IMAGE" \
  --build-arg SERVICE_NAME=off-key-mqtt-radar \
  --build-arg SERVICE_ROOT=services/mqtt/radar backend
uv run --frozen --project backend python -m pytest -q --tb=short -m radar_integration

The two opt-in variables enable actual Docker lifecycle tests and current-image Swarm tests. These cover reconnect and ACL failures, scoped alarms, signed checkpoint recovery, overload, sensor silence, interrupted creation, concurrent heartbeats/catalogue changes, task shutdown, resource admission and observation time validation. Fixture teardown removes only resources owned by each run.

Frontend¶

npm --prefix frontend run lint
npm --prefix frontend test
npm --prefix frontend run build

Expected result: no lint errors, passing tests, and a successful production build. The build type-checks the application, Node tooling, unit tests and Playwright tests before producing the Vite bundle. npm --prefix frontend run test:coverage writes an HTML coverage report to frontend/coverage/.

Run the browser journeys against a running Compose stack, in two invocations because they need different TACTIC settings:

The browser lane of the deployment smoke runs both, one after the other. A run signs in once: the auth-setup project accepts the operator invitation, signs in and saves the remembered session to frontend/playwright/.auth/user.json, and every authenticated journey starts from that saved state. The gateway's login limit of 10 per account a minute therefore stays the production value. The pipeline invocation reuses the file the first one saved while authoring was still disabled, and writes to test-results/pipelines, because Playwright empties its output folder before a run and the first invocation's evidence must stay. The rehearsal lane runs only npx playwright test --project=auth-setup before its deployment tests, which take their access token from the same file.

CI browser journeys retain one retry for diagnostic evidence. Any test that passes only on retry is classified as flaky and fails CI.

The member-management browser journeys read invitation links from the local mail sink. Set MAILPIT_BASE_URL to its reachable HTTP endpoint when using custom ports. The disposable production Swarm rehearsal publishes its SMTP fixture's API on port 8025 and supplies MAILPIT_BASE_URL=http://127.0.0.1:8025 in smoke.env. The restore test reads the invitations of the members it creates from the same sink and honours the same variable.

Repository pre-commit gate¶

uv run --project backend pre-commit run --all-files

Runtime health checks¶

curl --fail http://localhost:8000/health
curl --fail http://localhost:8001/health
curl --fail http://localhost:8001/ready

# The backend image has no wget or curl; use its Python like the healthchecks do.
probe='import sys, urllib.request as u; print(u.urlopen(sys.argv[1]).read().decode())'
docker compose exec mqtt-proxy /app/bin/python -c "$probe" http://localhost:8010/health
docker compose exec mqtt-proxy /app/bin/python -c "$probe" http://localhost:8010/ready
docker compose exec db-sync /app/bin/python -c "$probe" http://localhost:8009/health
docker compose exec db-sync /app/bin/python -c "$probe" http://localhost:8009/ready/schema

/ready reports an applied collection revision. An empty selection is healthy idle.

Log inspection¶

Tail critical services:

docker compose logs -f api-gateway tactic-middleware db-sync mqtt-proxy

Inspect recent output for one service:

docker compose logs --tail 200 tactic-middleware

Debug flows by symptom¶

Frontend loads but charger data is empty¶

1. Confirm that API_PROXY_TARGET is reachable from the Vite dev server; built images proxy /api through nginx.
2. Check Gateway liveness at http://localhost:8000/health.
3. Check TACTIC readiness at http://localhost:8001/ready.
4. Inspect Gateway and TACTIC logs for the upstream error.

Monitoring start fails¶

1. Confirm that the model catalogue is served:

   curl --fail http://localhost:8000/v1/detectors
   

2. List existing services and look for sensors already claimed:

   curl --fail http://localhost:8000/v1/monitors
   

3. Confirm TACTIC readiness.

4. Inspect TACTIC logs for validation, Docker API, image, or network failures.

Telemetry is not arriving¶

1. Verify that the source publisher is active.
2. Check the Sources catalog, selected sensors, matching applied revisions and per-broker connection status.
3. Check proxy health and readiness.
4. Inspect proxy logs for parsing, authentication, or database-write errors.
5. Query the Gateway telemetry endpoint.

Evidence or anomalies are missing¶

1. Verify that the monitor workload exists.
2. Confirm that data reaches its EMQX topic.
3. Allow static calibration or adaptive warm-up to complete.

4. Query evidence and alarm counts:

   curl --fail "http://localhost:8000/v1/evidence?reading_id=<reading-id>"
   curl --fail http://localhost:8000/v1/alarms/count
   

5. Inspect RADAR logs for model or persistence errors.

Service addresses — local default¶

Related pages¶

• Getting started
• Developer setup
• Environment variables
• Deployment modes
• Backend API