Readiness, health checks, and process errors (middleware)

Operational reference for axiom-genesis-middleware startup: /healthz, /readyz, and /livez, how the readiness gate behaves, and how global error handlers affect production expectations.

Endpoints

Route	Status	Meaning
GET /healthz	Always 200 when the process is up	Liveness: Node is running, startupMode included in JSON. Does not prove database or Redis catalog is usable.
GET /readyz	200 iff all readiness gates are true; else 503	Combines three booleans from utils/startup-readiness.util.js (see below). Mounted at app root — not under /api.
GET /livez	200 iff /readyz would be 200 and SELECT 1 on the default MSSQL pool succeeds and Redis PING returns PONG; else 503	Stricter probe for load balancers that need an end-to-end check without hitting business APIs. App root — not under /api.

Application Gateway / load balancer: Point the backend health probe to /readyz (typical) or /livez (stricter). Path is on the Express root (e.g. https://<backend-host>:3001/readyz). Do not use /api/readyz unless you add that route explicitly.

Readiness gate (startup-readiness.util.js)

The app is considered ready only when:

1. redis — Set in index.js only after ensureRedisConnectedForStartup() succeeds (connect + PING). If Redis never connects, this stays false and /readyz stays 503.
2. externalDbs — Set only after connect_external_dbs() succeeds in startup-background.util.js. While Azure SQL is paused or waking, the loader retries until SQL_RESUME_MAX_WAIT_MS (default 180000) elapses; if it never succeeds, this flag is never set (no false “ready”).
3. cache — Set when the object catalog / cache bootstrap path finishes (including partial catalog outcomes so traffic is not blocked indefinitely if only catalog steps fail).

MSSQL client (config/db/mssql-connection.js): default MSSQL_CONNECTION_TIMEOUT is 120000 ms; transient Azure errors (e.g. 40613) are retried up to MSSQL_CONNECT_MAX_ATTEMPTS (default 25) with backoff.

So ready: true means: Redis answered PING, primary + external DB connect path completed, and cache gate ran to completion — it is stricter than before for Redis and SQL connect, but catalog / registry steps can still be partial while cache is flipped to avoid permanent 503. GET /livez adds a runtime SELECT 1 plus Redis PING on top of that (use when the load balancer must verify the pool and cache client respond).

Middleware ordering (startup-middleware.util.js)

After routes are registered, middleware returns 503 for any path except /healthz, /readyz, and /livez until readinessGate.isReady() is true. Plan probes so /readyz or /livez is healthy before generic /api/... traffic.

Environment knobs (optional)

Variable	Default	Role
MSSQL_CONNECTION_TIMEOUT	120000	Single connect attempt timeout (ms)
MSSQL_CONNECT_MAX_ATTEMPTS	25	Retries for transient SQL errors during pool connect
MSSQL_CONCURRENT_WAIT_TIMEOUT_MS	180000	Max wait if another coroutine is connecting
SQL_RESUME_MAX_WAIT_MS	180000	Total wall time for background connect_external_dbs retries
MSSQL_EXTERNAL_CONNECT_TIMEOUT_MS	falls back to MSSQL_CONNECTION_TIMEOUT	Per-attempt timeout inside that window
REDIS_STARTUP_MAX_ATTEMPTS	20	Connect + PING retries before Redis gate fails

uncaughtException / unhandledRejection (index.js)

• uncaughtException: Logs and often continues; only some heuristics trigger process.exit(1) (see source). Assumption: keep HTTP up for diagnostics where possible.
• unhandledRejection: Logged; process continues by policy.

Do not rely on these handlers for request-scoped recovery—fix the underlying promise/async chain.

Related files

File	Role
axiom-genesis-middleware/index.js	Redis verify, setRedisReady(), listen, loadBackgroundResources()
axiom-genesis-middleware/config/redis.config.js	ensureRedisConnectedForStartup()
axiom-genesis-middleware/config/db/mssql-connection.js	Pool connect, transient retries, timeouts
axiom-genesis-middleware/utils/startup-readiness.util.js	Gate state
axiom-genesis-middleware/utils/startup-background.util.js	DB retries, cache load, flips externalDbs / cache flags
axiom-genesis-middleware/utils/startup-middleware.util.js	/healthz, /readyz, /livez, 503 gate