diff --git a/docs/integrations/integration-toolkit/configuration.md b/docs/integrations/integration-toolkit/configuration.md index d85a97f..7f623e6 100644 --- a/docs/integrations/integration-toolkit/configuration.md +++ b/docs/integrations/integration-toolkit/configuration.md @@ -22,7 +22,7 @@ Integrations support two types: ### Creating an Integration ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v2/integrations' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -34,7 +34,7 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v2/integrations' \ **Creating a Connector Integration:** ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v2/integrations' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -68,14 +68,14 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v2/integrations' \ ### Listing Integrations ```bash -curl -X GET 'https://erp-integration.sls.epilot.io/v2/integrations' \ +curl -X GET 'https://integration-toolkit.sls.epilot.io/v2/integrations' \ -H 'Authorization: Bearer ' ``` ### Updating an Integration ```bash -curl -X PUT 'https://erp-integration.sls.epilot.io/v2/integrations/{integrationId}' \ +curl -X PUT 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -91,7 +91,7 @@ Deleting an integration also removes all associated use cases. ::: ```bash -curl -X DELETE 'https://erp-integration.sls.epilot.io/v2/integrations/{integrationId}' \ +curl -X DELETE 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}' \ -H 'Authorization: Bearer ' ``` @@ -100,7 +100,7 @@ curl -X DELETE 'https://erp-integration.sls.epilot.io/v2/integrations/{integrati ### Creating a Use Case ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -132,7 +132,7 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integration ### Enabling/Disabling a Use Case ```bash -curl -X PUT 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases/{useCaseId}' \ +curl -X PUT 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases/{useCaseId}' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -151,7 +151,7 @@ incoming events. Plan cutovers accordingly. View the change history for a use case: ```bash -curl -X GET 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases/{useCaseId}/history' \ +curl -X GET 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases/{useCaseId}/history' \ -H 'Authorization: Bearer ' ``` @@ -160,7 +160,7 @@ curl -X GET 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationI Outbound use cases deliver standardized epilot events (event-catalog events) to your external system. The configuration consists of an `event_catalog_event` and one or more `mappings`: ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -350,7 +350,7 @@ Managed call use cases define synchronous API operations against external partne ### Creating a Managed Call Use Case ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -377,7 +377,7 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integration Managed calls are executed via the `/v1/managed-call/{slug}/execute` endpoint, where the slug acts as the RPC method name: ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/managed-call/get-customer/execute' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/managed-call/get-customer/execute' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -433,7 +433,7 @@ Secure proxy use cases route HTTP requests through epilot's dedicated proxy infr ### Creating a Secure Proxy Use Case ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -458,7 +458,7 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integration ### Sending a Proxy Request ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/secure-proxy' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/secure-proxy' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -602,48 +602,22 @@ Include a correlation ID to track related events: - Filter monitoring data by batch - Trace event processing through the pipeline -## Monitoring Configuration +## Monitoring -### Query Events +Every inbound and outbound event the toolkit processes is recorded as a monitoring +event, queryable per integration and visible in the Integration Hub's **Monitoring** +tab. A one-off example — the last day's error events for one integration: ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/monitoring/events' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/monitoring/events' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ - -d '{ - "integration_id": "", - "from": "2024-01-15T00:00:00Z", - "to": "2024-01-15T23:59:59Z", - "status": ["error"], - "limit": 100 - }' + -d '{ "level": "error", "from_date": "2026-01-15T00:00:00Z", "limit": 50 }' ``` -### Query Statistics +Querying, stats and time series, event traces, replay, the code reference, and +alerting are documented in the [Monitoring section](./monitoring/overview.md). -```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/monitoring/stats' \ - -H 'Authorization: Bearer ' \ - -H 'Content-Type: application/json' \ - -d '{ - "integration_id": "", - "from": "2024-01-01T00:00:00Z", - "to": "2024-01-31T23:59:59Z" - }' -``` - -### Event Replay - -Reprocess failed or specific events: - -```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/events/replay' \ - -H 'Authorization: Bearer ' \ - -H 'Content-Type: application/json' \ - -d '{ - "event_ids": ["evt-001", "evt-002", "evt-003"] - }' -``` ## Rate Limits @@ -668,9 +642,28 @@ Each organization's data is fully isolated. Integration IDs are scoped to organi ### Permissions -| Action | Permission | -|--------|------------| -| Read integrations | `erp:read` | -| Create/Update integrations | `erp:write` | -| Send events | `erp:events` | -| View monitoring | `erp:monitoring` | +Three actions govern the Integration Toolkit. They are checked against the +organization the token belongs to, and — where the endpoint names one — against the +specific integration. + +| Permission | Grants | +|---|---| +| `integration:view` | Read integrations and use cases; read monitoring events, stats, time series, traces, access logs, notification history and status | +| `integration:manage` | Create, update and delete integrations and use cases; edit notification configuration; push external monitoring events; send test notifications; replay events | +| `integration:consume` | Poll and acknowledge the outbound queue. Scoped to [Pollable Outbound](./pollable-outbound.md) only — it grants nothing else | + +The `view` / `manage` split is the one that catches people out when minting a scoped +token for a middleware: + +| Endpoint | Needs | +|---|---| +| `POST /v2/integrations/{id}/monitoring/events`, `…/stats`, `…/time-series` | `integration:view` | +| `GET /v2/integrations/{id}/monitoring/traces/{correlationId}` | `integration:view` | +| `GET /v2/integrations/{id}/notifications/history`, `…/status` | `integration:view` | +| `POST /v2/integrations/{id}/monitoring/external-events` | `integration:manage` | +| `POST /v2/integrations/{id}/notifications/test` | `integration:manage` | +| `POST /v1/integrations/{id}/events/replay` | `integration:manage` | +| `POST /v1/integrations/{id}/outbound/messages/poll`, `…/ack` | `integration:consume` | + +Inbound event submission authenticates as the integration's own API token rather +than through these actions — see [Inbound Getting Started](./inbound/getting-started.md). diff --git a/docs/integrations/integration-toolkit/inbound/examples.md b/docs/integrations/integration-toolkit/inbound/examples.md index ee679c9..a7b63ec 100644 --- a/docs/integrations/integration-toolkit/inbound/examples.md +++ b/docs/integrations/integration-toolkit/inbound/examples.md @@ -93,7 +93,7 @@ Synchronize customer data from an ERP system into epilot contacts. ### API Request ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v3/erp/updates/events' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v3/erp/updates/events' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -480,7 +480,7 @@ Handle different entity types based on payload conditions. Before deploying your mapping, test it using the simulation endpoint: ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v2/erp/updates/mapping_simulation' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/erp/updates/mapping_simulation' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ diff --git a/docs/integrations/integration-toolkit/inbound/getting-started.md b/docs/integrations/integration-toolkit/inbound/getting-started.md index f99d2ca..efa8bef 100644 --- a/docs/integrations/integration-toolkit/inbound/getting-started.md +++ b/docs/integrations/integration-toolkit/inbound/getting-started.md @@ -19,7 +19,7 @@ This guide walks you through setting up an inbound integration to synchronize da Create a new integration to represent your ERP connection: ```bash title="Create an integration" -curl -X POST 'https://erp-integration.sls.epilot.io/v2/integrations' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -49,7 +49,7 @@ Save the `id` — you'll need it for subsequent API calls. A use case defines how specific data types are mapped and synchronized. Create an inbound use case for customer data: ```bash title="Create a use case" -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -78,7 +78,7 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integration Push data from your ERP system using the events endpoint: ```bash title="Send an inbound event" -curl -X POST 'https://erp-integration.sls.epilot.io/v3/erp/updates/events' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v3/erp/updates/events' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ diff --git a/docs/integrations/integration-toolkit/monitoring/_category_.json b/docs/integrations/integration-toolkit/monitoring/_category_.json new file mode 100644 index 0000000..37aa306 --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/_category_.json @@ -0,0 +1,4 @@ +{ + "label": "Monitoring & Alerting", + "position": 9 +} diff --git a/docs/integrations/integration-toolkit/monitoring/acks.md b/docs/integrations/integration-toolkit/monitoring/acks.md new file mode 100644 index 0000000..470b8e5 --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/acks.md @@ -0,0 +1,135 @@ +--- +sidebar_position: 5 +title: ACK Tracking +description: How ERPs acknowledge events they processed, the ACK lifecycle and timeout, and how to turn tracking off per use case +slug: /integrations/integration-toolkit/monitoring/acks +--- + +# ACK Tracking + +epilot knows it *delivered* an outbound event — the webhook returned 2xx, or the +message was polled. It does not know the ERP actually **processed** it. An +acknowledgement closes that gap: your middleware confirms the work is done, and the +event's status in the Integration Hub becomes end-to-end rather than delivery-only. + +## The flow + +```mermaid +sequenceDiagram + participant EP as epilot + participant MW as Your middleware + + EP->>MW: outbound event (payload carries _ack_id) + Note over EP: ACK_PENDING recorded + MW->>MW: process the event + MW->>EP: POST /v1/erp/tracking/acknowledgement { ack_id } + Note over EP: ACK_CONFIRMED — tracking record cleared +``` + +If the acknowledgement never arrives, the record goes stale and epilot records +`ACK_TIMEOUT` instead. + +## Where `ack_id` comes from + +It arrives **in the event payload**, as the `_ack_id` field, alongside the other +common metadata every core event carries (`_event_version`, `_event_source`, …). It is +not an HTTP header, and it is not something you construct. + +```jsonc +{ + "_event_source": "epilot", + "_ack_id": "ack_01HZY…", // ← acknowledge with this + "contract": { "...": "..." } +} +``` + +Each delivered event gets its own `_ack_id`. Acknowledge each one individually — an +id is consumed once, and acknowledging clears its tracking record. + +:::note +A JSONata outbound mapping controls what your ERP receives. If your mapping builds a +brand-new object rather than extending the event, carry `_ack_id` through explicitly, +or your middleware will never see the id it needs to acknowledge with. +::: + +## Sending the acknowledgement + +```bash title="Send ACK" +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/erp/tracking/acknowledgement' \ + -H 'Content-Type: application/json' \ + -d '{ "ack_id": "ack_01HZY…" }' +``` + +`ack_id` is the **only** field. There is no status to report: sending the +acknowledgement *is* the signal that processing succeeded. A failure is simply an ACK +that never arrives, which the timeout below turns into a visible warning. + +| Response | Meaning | +|---|---| +| `200` | Acknowledged; the tracking record is cleared | +| `400` | `ack_id` missing from the body | +| `404` | No tracking record — already acknowledged, already timed out, or an unknown id | + +A `404` after a successful `200` is normal if you retry: the record is gone because +the first call consumed it. Treat it as success. + +## The lifecycle and its codes + +Three [monitoring codes](./codes.md) tell the whole story, and all three are +filterable in the Monitoring tab: + +| Code | Level | When | +|---|---|---| +| `ACK_PENDING` | info | The event was delivered and epilot is waiting for the acknowledgement | +| `ACK_CONFIRMED` | info | Your acknowledgement arrived | +| `ACK_TIMEOUT` | warning | No acknowledgement within the timeout window | + +`ACK_PENDING` and `ACK_CONFIRMED` are **info**-level: they are lifecycle markers, not +outcomes, so they are counted in total events but deliberately excluded from the +success rate. `ACK_TIMEOUT` is a **warning** — the delivery itself worked, so it is +not an error on epilot's side, but something on yours needs attention. + +### The timeout window + +A checker runs every **10 minutes** and times out any record older than **15 +minutes**. In practice an unacknowledged event surfaces as `ACK_TIMEOUT` within about +25 minutes of delivery — so do not treat a missing ACK as final for roughly half an +hour. + +`ACK_TIMEOUT` is also promoted to its own figure in the stats response, +`ack_timeout_count`, so you can chart "how often is the ERP failing to confirm" without +filtering the event stream by code: + +```bash +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/monitoring/stats' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ "from_date": "2026-01-01T00:00:00Z", "use_case_type": "outbound" }' +``` + +## Turning tracking off + +ACK tracking is per outbound use case, via `ack_tracking`: + +| Value | Behaviour | +|---|---| +| `on` *(default)* | A tracking record is written, `ACK_PENDING` is recorded, and an unacknowledged event eventually raises `ACK_TIMEOUT` | +| `off` | No tracking record, no `ACK_PENDING`, and no `ACK_TIMEOUT` | + +Set it to `off` for consumers that will never acknowledge, and for deliveries that +already keep their own durable per-item record — [Pollable +Outbound](../pollable-outbound.md) queues, for instance, track consumption through the +queue's own `MSG_ACKED` lifecycle, so ACK tracking on top of it produces timeouts that +mean nothing. + +:::warning +Leaving `ack_tracking: on` for a consumer that never acknowledges generates a steady +stream of `ACK_TIMEOUT` warnings. That is noise on its own, and it will trip a +`warning_threshold` [alert rule](./alerting.md) if you enable one. +::: + +## Related + +- [Monitoring Codes](./codes.md) — the full code reference +- [Investigating events](./investigating.md) — traces and replay +- [Pollable Outbound](../pollable-outbound.md) — the queue's own delivery lifecycle diff --git a/docs/integrations/integration-toolkit/monitoring/alerting.md b/docs/integrations/integration-toolkit/monitoring/alerting.md new file mode 100644 index 0000000..bdafad3 --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/alerting.md @@ -0,0 +1,229 @@ +--- +sidebar_position: 4 +title: Alerting & Notifications +description: Configure per-integration alert rules, anomaly thresholds, digests and recipients so integration problems find you instead of the other way round +slug: /integrations/integration-toolkit/monitoring/alerting +--- + +# Alerting & Notifications + +Monitoring tells you what happened when you go and look. Alerting tells you without +looking. Each integration has its own notification configuration: who gets told, about +what, and how often. + +Open the integration in epilot 360 and use the **Notifications** tab. Everything below +is configurable there, and through the API on the integration's +`settings.notifications`. + +## Getting started + +Notifications are **off** until you turn them on. Switching them on gives you a +sensible starting set rather than a blank page: + +| Rule | On by default? | +|---|---| +| Critical error | ✅ | +| Error rate | ✅ | +| Success-rate drop | ✅ | +| Recovery — all clear | ✅ | +| Warning rate | ❌ opt-in | +| Silence / heartbeat | ❌ opt-in | + +Plus a **weekly digest**, Monday 08:00 in your organization's timezone, that skips +itself when nothing happened. + +Add at least one recipient and save. The defaults are deliberately quiet — the two +noisiest rules start off. + +:::tip +Use **Send test** on the Notifications tab to see exactly what an alert or digest looks +like. It renders a real notification and sends it **only to you**, and it is not +written to the notification history. +::: + +## The rules + +Six trigger types, each answering a different question. + +### Critical error + +Notifies the moment a single error occurs. This is the real-time path — it does not +wait for a sweep. + +Because one bad deployment on your side can produce thousands of identical errors in +a minute, identical alerts are collapsed: once an alert has fired for a given code, the +same code stays quiet for the rule's window (2 hours by default) instead of paging you +repeatedly for one incident. Suppressed decisions still appear in the history, marked +`debounced`, so you can see what was folded in. + +### Error rate and warning rate + +Notify when errors — or warnings — in a window climb above normal. Use these instead +of *Critical error* when a trickle of failures is expected and only a surge matters. + +Warning rate is off by default and, when enabled, is in-app only: warnings are common +enough that mailing every spike is rarely what you want. + +### Success-rate drop + +Notifies when the proportion of successful events falls below normal. This is the best +single health signal, because it catches "everything is slow and half of it is failing" +without you having to guess an error count. + +It only evaluates once there are enough events to be meaningful — a handful of events +with one failure is not a 50% outage. That minimum sample counts **outcomes**: `info` +events (queued messages, ignored duplicates) are excluded, exactly as they are from the +[success rate itself](./overview.md#levels). + +### Recovery — all clear + +Notifies when an alerting integration returns to healthy. Worth leaving on: without it, +you learn things broke but never that they resolved. + +### Silence / heartbeat + +Notifies when an integration goes unusually **quiet**. This is the rule that catches +what counts and rates cannot: a middleware that has stopped calling produces no errors, +so every other rule stays green while nothing is being synced at all. + +Off by default because a legitimately intermittent integration would cry wolf. If yours +sends continuously, turn it on — it is the difference between noticing a dead feed in +minutes and noticing it in a month. + +## Fixed thresholds or automatic + +Rate rules take either a number you choose or `auto`. + +**A fixed threshold** is predictable and right when you know your own numbers — "more +than 20 errors an hour is a problem." + +**`auto`** learns what normal looks like for *this* integration at *this* time of week, +and alerts on departures from it. It exists because integration traffic is not flat: a +nightly batch at 03:00 and a quiet Sunday afternoon have completely different normal +error counts, and one fixed number cannot be right for both. `auto` compares like with +like — this Tuesday 09:00 against previous Tuesday 09:00s — and is deliberately +resistant to one-off spikes, so a single bad hour does not teach it that bad hours are +normal. + +Two knobs shape it: + +| Setting | Effect | +|---|---| +| **Sensitivity** — low / medium / high | How far from normal counts as abnormal. Higher sensitivity alerts on smaller departures. Start at medium. | +| **Fallback threshold** | A plain number used while there is not yet enough history to know what normal is. | + +**Cold start:** a new integration has no history, so `auto` behaves as the fallback +threshold until roughly two weeks of data exist, then switches over on its own. You do +not have to do anything, but it does mean a brand-new integration alerts on the +fallback number — set one you would actually be happy with. + +The Notifications tab draws the learned range as a band with your current value marked +against it. The band is what the rule currently considers normal for this hour of the +week; a value outside it is what triggers the alert. A rule still in cold start shows +no band. + +## Choosing what to watch {#choosing-what-to-watch} + +By default a rule watches **every error-level code** across every use case. You can +narrow both axes. + +**By use case** — restrict the integration to specific use cases, so a known-noisy +import does not drown a critical sync. + +**By code** — either pick individual [codes](./codes.md), or use a group: + +| Group | Matches | +|---|---| +| `_error_` | Every error-level code | +| `_warning_` | Every warning-level code | +| `_success_` | Every success-level code | +| `_info_` | Every info-level code | +| `_any_` | Everything, any level | +| `_parent_` | Whatever the integration-level scope is set to | + +Groups and individual codes combine, so `_error_` plus `ACK_TIMEOUT` means "all errors, +and also that one warning I care about". Because [a code's level is +fixed](./overview.md#levels), a group scope keeps meaning the same thing as the +taxonomy grows. + +A maximum of 20 rules per integration keeps evaluation bounded. + +## Recipients and channels + +Recipients are epilot users in the same organization. Channels are **email** and +**in-app**, set once as a default and overridable per rule — for instance error rate by +email and in-app, warning rate in-app only. + +:::note +Channel settings are a ceiling, not an override. Each recipient's own notification +preferences still apply, so turning a channel **on** here never forces a delivery +someone has opted out of; turning it **off** does suppress it. A notification skipped +this way appears in the history as `recipient_opt_out`. +::: + +## Digests + +A digest is the scheduled counterpart to alerts: a periodic summary rather than an +interruption. + +| Setting | Options | +|---|---| +| Frequency | Daily, or weekly on a chosen day | +| Time | Any time of day, in a timezone you pick | +| Channels | Email, in-app | +| Include healthy | List every integration, or only ones with issues | +| Skip if empty | Suppress the digest entirely when nothing happened | + +Leaving **skip if empty** on is what keeps a weekly digest worth reading. + +## Muting + +**Mute** silences all non-digest alerts until a time you choose — for a planned ERP +migration, or while you work through a known backlog. Muted decisions are still +recorded in the history, marked `muted`, so muting never hides what happened. It is +time-boxed by design: there is no permanent mute to forget about. + +## Seeing what fired, and why nothing did + +The **Activity** list on the Notifications tab is the notification history: every real +decision, including ones that did **not** result in a message. That second part is what +makes it useful — when someone says "we should have been alerted", the history tells +you whether the rule never fired, or fired and was suppressed: + +| `suppressed_reason` | Meaning | +|---|---| +| `muted` | The integration was muted at the time | +| `debounced` | An alert for that code had already fired inside its window | +| `recipient_opt_out` | Every recipient had that channel switched off | + +Alongside it, the live status shows each rule's current state — `ok`, `alerting` or +`recovered` — with when it last fired and last cleared. + +## From the API + +The configuration lives on the integration itself, at `settings.notifications`, so it +round-trips through the normal integration `GET` and `PUT`. Three endpoints cover the +rest: + +| Endpoint | Purpose | Permission | +|---|---|---| +| `GET …/notifications/status` | Live per-rule state, and the learned band for `auto` rules. Add `?include=baseline_series` for the full picture. | `integration:view` | +| `GET …/notifications/history` | Cursor-paginated decisions, newest first; `?type=` filters by rule type | `integration:view` | +| `POST …/notifications/test` | Render and send one notification to the calling user only | `integration:manage` | + +```bash +curl 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/notifications/status' \ + -H 'Authorization: Bearer ' +``` + +:::note +Rules are evaluated on a sweep every few minutes, so status and rate-based alerts are +near-real-time rather than instant. *Critical error* is the exception — it fires on the +event itself. +::: + +## Related + +- [Monitoring Codes](./codes.md) — what to scope your rules to +- [Monitoring Overview](./overview.md) — levels, and why `info` is excluded from the success rate +- [Investigating events](./investigating.md) — what to do once an alert arrives diff --git a/docs/integrations/integration-toolkit/monitoring/codes.md b/docs/integrations/integration-toolkit/monitoring/codes.md new file mode 100644 index 0000000..57ae52e --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/codes.md @@ -0,0 +1,139 @@ +--- +sidebar_position: 2 +title: Monitoring Codes +description: Every monitoring code the Integration Toolkit emits, what it means, and what to do about it +slug: /integrations/integration-toolkit/monitoring/codes +--- + +{/* AUTOGENERATED — do not edit by hand. + Source: erp-integration-api packages/erp-utils/src/monitoring/__snapshots__/monitoring-codes.json + Regenerate: npm run update-monitoring-codes */} + +# Monitoring Codes + +Every monitoring event carries a **code** and a **level**. The code says what +happened; the level says how much you should care. Both are filterable in the +Integration Hub's [Monitoring tab](./overview.md) and through the events API. + +There are 80 codes. You are most likely here because you saw one in a failed +event — find it below. + +:::tip +A code's level is fixed. That is what makes alert rules scoped to `_error_` or +`_warning_` predictable — see [Alerting](./alerting.md#choosing-what-to-watch). +::: + +## Error + +Something failed and the event did not do what it was meant to do. These are what the default alert rules watch. + +| Code | What it means | +|---|---| +| `ATTACHMENT_NOT_FOUND` | The file no longer exists — it was removed between the event and the delivery | +| `ATTRIBUTE_TYPE_MISMATCH` | An attribute value did not match the type declared in the entity schema | +| `DEPRECATED_ENDPOINT` | This endpoint version is deprecated | +| `DIRECT_ENTITY_NOT_ALLOWED` | The entity is not permitted by the use case entity allowlist | +| `DIRECT_PAYLOAD_INVALID` | The direct mode payload failed validation against the versioned payload schema | +| `DIRECT_VERSION_UNSUPPORTED` | The direct mode payload version is not supported by the platform | +| `ENTITY_REFERENCE_NOT_FOUND` | A direct-by-id entity reference points at an entity that does not exist or has a different schema | +| `EVENT_NOT_CONFIGURED` | No mapping configuration found for this event type | +| `EXTERNAL_API_ERROR` | An external API returned an error | +| `EXTERNAL_ERROR` | An error span pushed by an external system (e.g. your integration middleware) via the external monitoring events endpoint. epilot assigns the code from the span's level; the middleware never sends one. | +| `FAN_OUT_INVALID_RESULT` | The split expression returned something other than a list, so no deliveries could be created — see split_expression_result_type | +| `FILE_EXTRACTION_FAILED` | Failed to extract file from the request | +| `FILE_FETCH_FAILED` | The file could not be fetched from epilot before it could be uploaded | +| `FILE_PROXY_UPLOAD_FAILED` | A file upload failed terminally, or every delivery attempt was exhausted | +| `FILE_TOO_LARGE` | The file exceeds the maximum size configured on the upload use case | +| `INTEGRATION_NOT_FOUND` | The inbound event referenced an integration_id that does not exist in the calling token's organization. Check the integration id and that the token belongs to the same org. | +| `INVALID_METER_READING_ATTRIBUTES` | Meter reading has invalid attributes | +| `MALFORMED_PAYLOAD` | The event payload could not be parsed (malformed JSON) | +| `MAPPING_EXPRESSION_FAILED` | A mapping expression failed to evaluate | +| `METERING_API_ERROR` | The metering API returned an error | +| `METER_READING_GROUP_FAILED` | A batch write of meter readings failed permanently after all retries — summary event alongside the per-reading errors | +| `METER_READING_GROUP_RETRYING` | A batch write of meter readings failed and will be retried automatically — one event per attempt covering the whole group (reading_count and external_ids in details) | +| `MISSING_REQUIRED_PARAM` | A required parameter is missing from the request | +| `MISSING_UNIQUE_IDENTIFIERS` | The event is missing the unique identifier field(s) required to match an entity | +| `OAUTH2_TOKEN_FAILURE` | Failed to obtain an OAuth2 access token | +| `PAYLOAD_TOO_LARGE` | The payload exceeded the maximum size accepted by the receiving system | +| `PRUNE_SCOPE_PARTIAL_FAILURE` | Scope pruning completed with some failures | +| `RECURSION_DEPTH_EXCEEDED` | Maximum recursion depth was exceeded during processing | +| `RELATION_REF_ITEM_NOT_FOUND` | The relation_ref target entity exists but the referenced item/value could not be matched — skipped as non-retryable. Check the mapping configuration and the entity data. | +| `RELATION_REF_VALUE_UNDEFINED` | A relation_ref mapping value resolved to undefined — check the mapping expression | +| `REQUIRED_PARAM_MISSING` | A param the use case marks as required resolved to nothing, so the delivery was stopped before anything was sent — see param_name | +| `SECURE_PROXY_DISABLED` | The secure proxy use case is disabled | +| `SECURE_PROXY_DOMAIN_BLOCKED` | The target domain is blocked | +| `SECURE_PROXY_DOMAIN_NOT_ALLOWED` | The target domain is not in the allowlist | +| `SECURE_PROXY_ERROR` | An error occurred in the secure proxy | +| `SECURE_PROXY_INVALID_CONFIG` | The secure proxy configuration is invalid | +| `SECURE_PROXY_INVALID_TYPE` | The secure proxy type is invalid | +| `SECURE_PROXY_INVALID_URL` | The target URL is invalid | +| `SECURE_PROXY_IP_BLOCKED` | The target IP address is blocked | +| `SECURE_PROXY_IP_NOT_ALLOWED` | The target IP address is not in the allowlist | +| `SECURE_PROXY_NOT_FOUND` | The secure proxy use case was not found | +| `SECURE_PROXY_UNAVAILABLE` | Secure proxy service is unavailable | +| `SIGNATURE_VERIFICATION_FAILED` | Request signature could not be verified | +| `SIGNATURE_VERIFICATION_UNAVAILABLE` | The file service could not be reached to verify the request signature | +| `STEP_DISABLED` | A request step's "run this step when" expression returned false, so this step and every step after it were skipped. This is the configuration working as written, not a fault. | +| `TIMEOUT` | The operation timed out | +| `UNIQUE_ID_MULTIPLE_MATCHES` | Multiple entities matched the unique ID | +| `UNIQUE_ID_NOT_IN_SCHEMA` | The unique ID attribute is not defined in the entity schema | +| `UNKNOWN_ERROR` | An unexpected error occurred during processing | +| `USE_CASE_DISABLED` | The use case is currently disabled | +| `USE_CASE_INVALID_TYPE` | The use case type is invalid or unsupported | +| `USE_CASE_MISSING_CONFIG` | The use case is missing required configuration | +| `USE_CASE_NOT_FOUND` | The use case could not be found | + +## Warning + +Processing continued, but something needs a human eye — often a retry in flight or a value nobody has mapped yet. + +| Code | What it means | +|---|---| +| `ACK_TIMEOUT` | Acknowledgement timed out waiting for the ERP system | +| `EXTERNAL_WARNING` | A warning span pushed by an external system via the external monitoring events endpoint. | +| `FILE_PROXY_UPLOAD_RETRYING` | A file upload failed with a retryable error and will be retried automatically — one event per attempt | +| `LOOKUP_UNMAPPED` | A value was not listed in a lookup table and its fallback was used — see lookup_name and lookup_key for the gap | +| `SOFT_DELETED_ENTITY_MATCHED` | A soft-deleted entity matched the unique ID — it will be resurrected on upsert, or referenced as-is by a relation. Investigate why the ERP source is sending events for a deleted entity. | + +## Success + +The event did what it was meant to do. Useful for confirming a sync actually landed. + +| Code | What it means | +|---|---| +| `ENTITY_CREATED` | A new entity was created in epilot | +| `ENTITY_DELETED` | An entity was deleted from epilot | +| `ENTITY_NO_OP` | No changes were needed for the entity | +| `ENTITY_UPDATED` | An existing entity was updated in epilot | +| `EXTERNAL_SUCCESS` | A success span pushed by an external system via the external monitoring events endpoint. | +| `FILE_PROXY_OK` | File proxy request completed successfully | +| `FILE_PROXY_UPLOADED` | The external system accepted the file | +| `METER_READING_DELETED` | One or more meter readings were deleted — emitted once per batch, not per reading (reading_count and external_ids in details) | +| `METER_READING_UPSERTED` | One or more meter readings were created or updated — emitted once per batch, not per reading (reading_count and external_ids in details) | +| `PRUNE_SCOPE_COMPLETED` | Scope pruning completed successfully | +| `WEBHOOK_DELIVERED` | Webhook was delivered successfully | + +## Info + +Lifecycle markers rather than outcomes: a message was queued, a duplicate was ignored, a step was skipped by configuration. Counted in total events, but excluded from the success rate. + +| Code | What it means | +|---|---| +| `ACK_CONFIRMED` | Acknowledgement was confirmed by the ERP system | +| `ACK_PENDING` | Acknowledgement is pending from the ERP system | +| `DUPLICATE_EVENT` | This event was already processed (duplicate) | +| `EXTERNAL_INFO` | An informational span pushed by an external system via the external monitoring events endpoint. | +| `FAN_OUT_EMPTY` | The split expression returned an empty list, so nothing was sent — expected for events that carry no relevant items | +| `FILE_PROXY_UPLOAD_ENQUEUED` | A per-file upload was accepted for delivery during fan-out | +| `MSG_ACKED` | Outbound message acknowledged by the polling consumer and removed from the queue | +| `MSG_DEAD_LETTERED` | Outbound message moved to the dead-letter queue after exhausting delivery attempts, or via an operator skip | +| `MSG_ENQUEUED` | Outbound message enqueued to the poll queue, awaiting consumption by the ERP | +| `MSG_EXPIRED_UNPOLLED` | Outbound message expired before being consumed — retention elapsed without a successful poll | +| `MSG_HEAD_BLOCKED` | Outbound stream halted by a poison head message (block policy) — requires operator unblock or consumer acknowledgement | +## Status-code families + +Some codes are generated from an upstream response rather than drawn from the fixed list above. + +| Pattern | Level | What it means | +|---|---|---| +| `HTTP_{status}` | warning | The upstream system answered the proxied request with this HTTP status. The proxy itself worked — it reached the target and returned its answer — so the refusal is recorded against the use case that owns the request. | diff --git a/docs/integrations/integration-toolkit/external-monitoring-events.md b/docs/integrations/integration-toolkit/monitoring/external-events.md similarity index 99% rename from docs/integrations/integration-toolkit/external-monitoring-events.md rename to docs/integrations/integration-toolkit/monitoring/external-events.md index 7214abf..0ae2764 100644 --- a/docs/integrations/integration-toolkit/external-monitoring-events.md +++ b/docs/integrations/integration-toolkit/monitoring/external-events.md @@ -1,5 +1,5 @@ --- -sidebar_position: 8 +sidebar_position: 6 title: External Monitoring Events description: Push monitoring events from an external system (e.g. an integration middleware) into epilot so the Integration Hub is your central monitoring point, with a cross-system event trace slug: /integrations/integration-toolkit/external-monitoring-events @@ -12,7 +12,7 @@ When part of your integration pipeline runs **outside epilot** — typically an :::info Topology This is for the case where epilot still does the bulk of the work (mapping, entity/metering updates). The middleware does two independent things: -1. **forwards the inbound data event** to the standard [inbound endpoint](./inbound/getting-started.md), and +1. **forwards the inbound data event** to the standard [inbound endpoint](../inbound/getting-started.md), and 2. **separately pushes its own monitoring events** (its processing steps) to the endpoint below. The two halves are linked into one trace by a shared `correlation_id`. diff --git a/docs/integrations/integration-toolkit/monitoring/investigating.md b/docs/integrations/integration-toolkit/monitoring/investigating.md new file mode 100644 index 0000000..818167a --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/investigating.md @@ -0,0 +1,181 @@ +--- +sidebar_position: 3 +title: Investigating Events +description: Trace a failed event across systems, read the captured request and response, replay it, and query the monitoring stream from the API +slug: /integrations/integration-toolkit/monitoring/investigating +--- + +# Investigating Events + +Something went wrong and you need to find out what. This page is the path from "the +Hub shows red" to "I know why, and I have fixed it". + +## Start with the event + +In the integration's **Monitoring** tab, filter the event table to `error` and pick +the event. The detail panel gives you the [code](./codes.md), the message, and the +`detail` object with everything the producer knew at the time. + +For anything that made an HTTP call — file proxy, managed call, secure proxy, outbound +delivery — `detail` also carries the **captured request and response**, rendered as +panels. This is usually where the answer is: you can see exactly what epilot sent and +exactly what came back. + +### What is captured, and what is not + +Two deliberate limits shape those panels. + +**Credentials are never stored.** Headers and body fields that carry secrets are +replaced with `` before anything is written — in both directions, and +wherever they appear in the structure. That covers the obvious names and the +credential-shaped ones a mapping might introduce, and it covers bodies as well as +headers, because an OAuth2 exchange puts the secret in the body and gets a token back +in the response. + +:::tip +If you configure a custom authentication header, prefer routing the call through the +[Secure Proxy](../configuration.md#secure-proxy-use-cases). It injects credentials +server-side, so the secret never travels through the part of the request that gets +captured at all — a stronger guarantee than relying on name matching. +::: + +**Long values are elided, not truncated.** A string over 256 characters is replaced +with its first 32 characters, a `✂ truncated` marker, and its original length. The +result is still valid, copyable JSON with every field in place — you can see that a +base64 document was there and how big it was, without the analytics store having to +hold it. A whole captured body is capped at 4KB. + +So a payload that looks cut off is working as intended. If you need the full body, +reproduce the call against your own system with the field values shown around the +elision. + +## Tracing {#tracing} + +One failure is rarely one row. Two endpoints group the stream, and they answer +different questions — see +[event_id vs correlation_id](./overview.md#event_id-and-correlation_id-are-different-groupings) +for why. + +**Everything one epilot event produced** — the fan-out: child entities, post-actions, +relation resolutions: + +```bash +curl 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/monitoring/events/{eventId}/associated' \ + -H 'Authorization: Bearer ' +``` + +It returns the associated events in chronological order, plus the original inbound +payload as the head of the trace when it is still within its 14-day window. + +**The whole business operation, across systems** — including spans your middleware +pushed: + +```bash +curl 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/monitoring/traces/{correlationId}' \ + -H 'Authorization: Bearer ' +``` + +This one rolls the spans up into a single `status` (`error` beats `warning` beats +`success` beats `info`), reports `started_at` / `ended_at`, and flags `truncated` when +the trace is longer than the returned window. Spans that came from outside epilot are +identifiable by their `EXTERNAL_*` code prefix. + +## Replaying events {#replaying-events} + +Once you have fixed the cause — corrected a mapping, mapped a missing value, brought +the ERP back up — replay the events that failed. + +```bash +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/events/replay' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ "event_ids": ["evt_91af", "evt_91b0"] }' +``` + +Things worth knowing before you use it: + +- **Inbound only.** Replay reconstructs a received inbound event. It does not re-send + outbound deliveries. +- **Maximum 100 event ids per request.** +- **14-day horizon.** The event is rebuilt from the stored inbound payload, which is + kept for 14 days — see [retention](./overview.md#how-long-data-is-kept). Older + events return `not_found`. +- **Always HTTP 200.** Per-event outcomes are in `results`, one entry per requested id + in request order, with a `status` of `success` / `queued`, `not_found`, `skipped`, + `ignored` or `error`. Compare `replayed` against the number of ids you sent rather + than trusting the status code. +- **A replay is a new event.** It is assigned a fresh `event_id` and a new + `correlation_id` prefixed `replay_`, so the replayed run is distinguishable from the + original in monitoring rather than overwriting it. + +## Querying from the API + +Everything the Monitoring tab shows is available directly. All three endpoints take a +POST body and are scoped to one integration. + +**The event stream** — filter by `level`, `code`, `use_case_id`, `use_case_type`, +`event_id`, `correlation_id` and a time range, with cursor pagination: + +```bash +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/monitoring/events' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ + "level": "error", + "use_case_type": "inbound", + "from_date": "2026-01-15T00:00:00Z", + "limit": 50 + }' +``` + +**Aggregates** — `…/monitoring/stats` for totals, success rate and `ack_timeout_count`, +with an optional `group_by` of `use_case_id`, `use_case_type`, `level`, `code` or +`date`. **Time series** — `…/monitoring/time-series` for bucketed counts at `5m`, +`10m`, `30m`, `1h`, `3h` or `1d`. + +### Which number am I looking at? {#which-number} + +Stats have two sources, and they answer genuinely different questions. Reading one as +the other is the most common way to reach a wrong conclusion from this data. + +| `source` | Counts | Use it for | +|---|---|---| +| `monitoring` *(default)* | Every row in the processing tree — fan-out children, post-actions, relation resolutions | "How much work happened, and how much of it failed" | +| `incoming` | Distinct events actually received, before any fan-out | "How many events did the ERP actually send us" | + +One received event routinely produces many monitoring rows, so `monitoring` totals are +legitimately much larger than `incoming` ones. Neither is wrong; they are different +denominators. + +:::warning +`source: incoming` reads the received-payload store, which is kept for **14 days**. A +query with a longer window silently under-reports — a 30-day chart will show a +suspicious cliff two weeks back, and a 30-day total will simply be too low. For +anything beyond 14 days, use the default `monitoring` source. +::: + +`source: incoming` also supports only `group_by: use_case_id`, and returns zero for the +level breakdown, because the received-payload store has no notion of an outcome. + +## Who called the API + +The **Access Token** tab, and `…/monitoring/access-logs`, show how the integration's +API tokens are actually being used — filterable by `token_id`, `service`, `method`, +`path`, `status` and time range, with cursor pagination. + +```bash +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/monitoring/access-logs' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ "status": 403, "from_date": "2026-01-15T00:00:00Z" }' +``` + +This is the place to answer "is the middleware still calling us", "which token is +being used for what", and "why is it getting 403s" — questions the monitoring stream +cannot answer, because a rejected call never becomes a monitoring event. + +## Related + +- [Monitoring Codes](./codes.md) — what the code you found means +- [Alerting](./alerting.md) — find out without looking +- [External Monitoring Events](./external-events.md) — make traces span your middleware diff --git a/docs/integrations/integration-toolkit/monitoring/overview.md b/docs/integrations/integration-toolkit/monitoring/overview.md new file mode 100644 index 0000000..14edf65 --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/overview.md @@ -0,0 +1,168 @@ +--- +sidebar_position: 1 +title: Monitoring Overview +description: How Integration Toolkit monitoring is structured — levels, codes, event and correlation ids, use case lanes, and how long data is kept +slug: /integrations/integration-toolkit/monitoring/overview +--- + +# Monitoring Overview + +Every event the Integration Toolkit processes leaves a trail. Inbound syncs, outbound +deliveries, file proxy fetches, managed calls and secure proxy requests all write to +one monitoring stream, visible in the Integration Hub's **Monitoring** tab and +queryable through the API. + +This page covers how that data is shaped, so the rest of the section makes sense. If +you are chasing a specific failure right now, go to +[Investigating events](./investigating.md) or look your code up in the +[code reference](./codes.md). + +## Where to look + +Open an integration in epilot 360 (**Integrations → your integration**) and use the +**Monitoring** tab. It gives you, in one place: + +- **Stat tiles** — total events, successes, errors, warnings, success rate, and + ACK timeouts for the selected period +- **Events over time** — a bucketed chart, optionally split by use case or lane +- **Use case breakdown** — which use cases produce the volume, and the errors +- **The event table** — filterable by level, code, use case and time, with a detail + panel per event +- **Access logs** — which API token called what + +The **Notifications** tab beside it is where you configure who gets told about all +this — see [Alerting](./alerting.md). + +## How an event is shaped + +Each row in the stream is one monitoring event: + +| Field | What it is | +|---|---| +| `level` | How much you should care: `success`, `error`, `warning` or `info` | +| `code` | What specifically happened — see the [code reference](./codes.md) | +| `message` | A human-readable line, usually the error text | +| `detail` | Free-form JSON with context for that code — including the captured request and response where there was one | +| `use_case_type` | Which lane produced it | +| `use_case_id` | Which configured use case, when one owns the event | +| `event_id` | The triggering event | +| `correlation_id` | The business operation | +| `created_at` | When it happened | + +### Levels + +There are four, and the distinction between the last two matters more than it looks: + +| Level | Meaning | +|---|---| +| `success` | The event did what it was meant to do | +| `error` | It failed | +| `warning` | It continued, but something needs a human eye — a retry in flight, an unmapped value | +| `info` | A lifecycle marker rather than an outcome: a message queued, a duplicate ignored, a step skipped by configuration | + +**`info` events are excluded from the success rate.** They are counted in +`total_events`, but a queued message or an ignored duplicate is not something that +could have succeeded or failed, so including them would drag the rate down for no +reason. An integration using ACK tracking or the poll queue emits a lot of them. + +A code's level is fixed — `ENTITY_CREATED` is always `success`, `ACK_TIMEOUT` is always +`warning`. That is what makes [alert rules](./alerting.md#choosing-what-to-watch) +scoped to a whole level predictable. + +### Use case lanes + +`use_case_type` separates the five kinds of work, so one noisy lane does not hide +another: + +| Lane | What it covers | +|---|---| +| `inbound` | ERP data arriving and being applied to entities and meter readings | +| `outbound` | epilot events delivered to your ERP by webhook or poll queue | +| `file_proxy` | Files fetched from, or delivered to, an external document system | +| `managed_call` | Synchronous calls to an external API | +| `secure_proxy` | Requests routed through the static-IP or VPN proxy | + +Two values in the use case column are not use cases: + +- **"General"** (an empty `use_case_id`) — system-level events that happened before + any use case was resolved, such as an event rejected at ingest. +- **`__unknown__`** — the event names a use case id that no longer exists on this + integration, usually because it was deleted or recreated. The history is real; the + configuration behind it is gone. + +### `event_id` and `correlation_id` are different groupings + +This is the single most useful thing to understand about the stream, because the two +answer different questions. + +One inbound event does not produce one monitoring row. It fans out — child entities, +post-actions, relation resolutions — and each step records its own event. All of those +share the **`event_id`** of the thing that triggered them. + +A **`correlation_id`** is wider: it identifies one *business operation*, and can span +several events and even several systems. If your middleware stamps the same +`correlation_id` on the spans it pushes and on the event it forwards, one trace covers +both halves — that is what [External Monitoring Events](./external-events.md) is for. + +``` +correlation_id: "bp-8f3a2c" ← one business operation +├── middleware span: received (external) +├── middleware span: mapped (external) +└── event_id: "evt_91af" ← one epilot event + ├── contact created + ├── billing account created + └── relation resolved +``` + +Two endpoints match the two groupings, and picking the wrong one is why a trace can +look incomplete: + +| To see | Use | +|---|---| +| Everything one epilot event produced | `GET …/monitoring/events/{eventId}/associated` | +| The whole business operation, across systems | `GET …/monitoring/traces/{correlationId}` | + +Both are covered in [Investigating events](./investigating.md#tracing). + +## How long data is kept + +| Data | Retained | What that limits | +|---|---|---| +| Monitoring events | **90 days** | How far back stats, charts and the event table can go | +| Received inbound payloads | **14 days** | How far back you can [replay](./investigating.md#replaying-events), and how long a trace can still show the original payload | + +The 14-day figure is the one that catches people out. Replay reconstructs the event +from the stored inbound payload, so **an event older than 14 days cannot be replayed** +— the monitoring record of it survives for the full 90 days, but the payload behind it +does not. If you are working through a backlog of failures, work through the oldest +first. + +## Is a specific entity up to date? + +Monitoring answers "what happened". A separate endpoint answers "is *this record* +current with the ERP": + +```bash +curl 'https://integration-toolkit.sls.epilot.io/v1/integrations/entities/{entityId}/sync-status' \ + -H 'Authorization: Bearer ' +``` + +It returns, per integration that has touched the entity, when it was last **synced** +and when it was last **changed**. + +The distinction matters: `last_synced_at` also advances on a no-op — an event that was +received and evaluated but changed nothing. Those deliberately leave no trace on the +entity itself: no activity feed entry, no `_updated_at` bump. So an entity that looks +untouched for weeks may be being checked constantly and simply not changing, and this +endpoint is the only way to tell those two states apart. An entity no inbound use case +has ever processed returns an empty list. + +Add `?integration_id=…` to narrow it to one integration. + +## Next + +- [Monitoring Codes](./codes.md) — look up what you saw +- [Investigating events](./investigating.md) — traces, captured payloads, replay +- [Alerting](./alerting.md) — get told without watching +- [ACK Tracking](./acks.md) — confirm the ERP processed what you sent +- [External Monitoring Events](./external-events.md) — bring your middleware's steps in diff --git a/docs/integrations/integration-toolkit/outbound-file-delivery.md b/docs/integrations/integration-toolkit/outbound-file-delivery.md index b0b59ee..2123ddf 100644 --- a/docs/integrations/integration-toolkit/outbound-file-delivery.md +++ b/docs/integrations/integration-toolkit/outbound-file-delivery.md @@ -280,5 +280,5 @@ Open an entry to see the use case, delivery, attachment, attempt, and available - [Core Events](/docs/integrations/core-events) - [Configuration](./configuration.md) - [File Proxy downloads](./file-proxy.md) -- [External Monitoring Events](./external-monitoring-events.md) +- [External Monitoring Events](./monitoring/external-events.md) - [Pollable Outbound](./pollable-outbound.md) diff --git a/docs/integrations/integration-toolkit/overview.md b/docs/integrations/integration-toolkit/overview.md index 39ea430..d03e1e5 100644 --- a/docs/integrations/integration-toolkit/overview.md +++ b/docs/integrations/integration-toolkit/overview.md @@ -24,7 +24,7 @@ The Integration Toolkit is composed of the following components. Each plays a sp | Component | Description | Status | |-----------|-------------|--------| -| **[Integration Hub](#integration-hub)** | Admin UI in epilot 360 to configure and monitor integrations | In progress | +| **[Integration Hub](#integration-hub)** | Admin UI in epilot 360 to configure and monitor integrations | Stable | | **[ERP Integration API](#erp-integration-api)** | CRUD API to manage integrations, use cases, and mappings | Stable | | **[ERP Inbound API](#inbound-api)** | Dedicated API to receive and simulate inbound ERP events | Stable | | **[Use Cases](./use-cases.md)** | Documented integration flows with testing support | Stable | @@ -32,13 +32,15 @@ The Integration Toolkit is composed of the following components. Each plays a sp | **[Changesets](/docs/entities/changesets)** | Pending attribute updates that wait for ERP confirmation or human approval | Stable | | **[Core Events](/docs/integrations/core-events)** | Standardized event payloads for outbound notifications | Stable | | **[Webhooks](/docs/integrations/webhooks)** | Push events from epilot to ERPs via core events | Stable | -| **[Pollable Outbound](./pollable-outbound.md)** | Pull-based outbound delivery — ERPs poll a queue instead of receiving webhooks | In progress | +| **[Pollable Outbound](./pollable-outbound.md)** | Pull-based outbound delivery — ERPs poll a queue instead of receiving webhooks | Stable | | **[JSONata Mapping](#jsonata-mapping)** | Transformation language for inbound and outbound data | Stable | | **[File Proxy](./file-proxy.md)** | Serve files from external archives on demand without migrating them into epilot | Stable | | **[Outbound File Delivery](./outbound-file-delivery.md)** | Deliver files referenced by epilot events to external document APIs | Stable | | **[Managed Calls](#managed-calls)** | Synchronous external API calls with JSONata mapping via connector integrations | Stable | | **[Secure Proxy](#secure-proxy)** | Route HTTP requests through epilot's secure proxy for static IP egress or VPN access | Stable | -| **[Monitoring and ACKs](#monitoring-and-acks)** | Central logging, error tracking, and event replay | In progress | +| **[Monitoring](./monitoring/overview.md)** | Central event logging, stats, cross-system traces, replay, and the code reference | Stable | +| **[Alerting & Notifications](./monitoring/alerting.md)** | Per-integration alert rules, anomaly baselines, digests, and delivery to email/in-app | Stable | +| **[ACK Tracking](./monitoring/acks.md)** | ERPs acknowledge processed events, closing end-to-end delivery visibility | Stable | | **[Blueprints](https://marketplace.epilot.cloud/en/blueprints)** | Packaged, installable integration setups | Stable | | **[Apps](https://marketplace.epilot.cloud/en/apps)** | Custom automation actions and portal extensions for ERP logic | In progress | @@ -120,14 +122,25 @@ See the [Configuration Guide](./configuration.md#secure-proxy-use-cases) for set - Outbound webhook payloads (epilot event to ERP format) - The Map Data flow building block -### Monitoring and ACKs - -All inbound and outbound events are centrally logged and surfaced in the Integration Hub. Key capabilities: - -- **Event replay** -- reprocess failed events -- **ACK tracking** -- ERPs acknowledge processed events via `v1/erp/tracking/acknowledgement`, enabling end-to-end visibility -- **Error alerting** -- per-use-case status indicators with actionable error details -- **Partner log shipping** -- middleware partners can send logs to epilot for centralized monitoring +### Monitoring and Alerting + +Every event the toolkit processes — inbound, outbound, file proxy, managed call and +secure proxy alike — is recorded as a monitoring event and surfaced in the Integration +Hub's **Monitoring** tab, with a matching API for automation. + +- **[Event stream and stats](./monitoring/overview.md)** — filter by level, code, use + case or correlation id; success rates and time series per lane +- **[Code reference](./monitoring/codes.md)** — every monitoring code, what it means + and what to do about it +- **[Traces and replay](./monitoring/investigating.md)** — follow one business + operation across systems, inspect the captured request and response, and reprocess + events +- **[Alerting](./monitoring/alerting.md)** — per-integration rules with anomaly + baselines, plus scheduled digests, delivered to email and in-app +- **[ACK tracking](./monitoring/acks.md)** — ERPs confirm they processed an event, + closing end-to-end visibility +- **[External monitoring events](./monitoring/external-events.md)** — push your + middleware's own processing steps in, so one trace spans both systems ## Architecture diff --git a/docs/integrations/integration-toolkit/pollable-outbound.md b/docs/integrations/integration-toolkit/pollable-outbound.md index 4c06335..f1752d2 100644 --- a/docs/integrations/integration-toolkit/pollable-outbound.md +++ b/docs/integrations/integration-toolkit/pollable-outbound.md @@ -52,7 +52,7 @@ Key properties: A poll mapping is configured on a regular outbound use case — same endpoint, same envelope as webhook mappings, only the `delivery` object differs. See [Outbound Use Case Configuration](./configuration.md#outbound-use-case-configuration) for the full use-case contract. ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ diff --git a/docs/integrations/integration-toolkit/use-cases.md b/docs/integrations/integration-toolkit/use-cases.md index 432e9c2..a2ed6dc 100644 --- a/docs/integrations/integration-toolkit/use-cases.md +++ b/docs/integrations/integration-toolkit/use-cases.md @@ -364,7 +364,7 @@ flowchart LR **What happens in the ERP:** - Middle layer receives the webhook and extracts meter reading data - Middle layer calls the ERP API to submit the reading (e.g., meter reading endpoint) -- Middle layer sends an [ACK](/docs/integrations/integration-toolkit/overview#monitoring-and-acks) back to epilot to confirm processing +- Middle layer sends an [ACK](./monitoring/acks.md) back to epilot to confirm processing **Core Event:** [`MeterReadingAdded`](/docs/integrations/core-events#MeterReadingAdded) @@ -768,18 +768,22 @@ flowchart LR ## ACK Tracking -All outbound use cases support [ACK tracking](/docs/integrations/integration-toolkit/overview#monitoring-and-acks). After processing a webhook, your middle layer should send an acknowledgment back to epilot: +All outbound use cases support [ACK tracking](./monitoring/acks.md). After processing a webhook, your middle layer should send an acknowledgment back to epilot: ```bash title="Send ACK" -curl -X POST 'https://erp-integration.sls.epilot.io/v1/erp/tracking/acknowledgement' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/erp/tracking/acknowledgement' \ -H 'Content-Type: application/json' \ - -d '{ - "ack_id": "", - "status": "processed" - }' + -d '{ "ack_id": "" }' ``` -This enables end-to-end monitoring in the Integration Hub: per-use-case status indicators show whether the ERP successfully processed each event. +`ack_id` is the only field. Acknowledging is the whole signal — there is no status to +report, because an ACK means "processed"; a failure is simply an ACK that never +arrives, and the event times out. + +This enables end-to-end monitoring in the Integration Hub: per-use-case status +indicators show whether the ERP processed each event. The full lifecycle, the timeout +behaviour and how to turn tracking off per use case are covered in +[ACK Tracking](./monitoring/acks.md). ## Next Steps diff --git a/package.json b/package.json index 9042608..042a9aa 100644 --- a/package.json +++ b/package.json @@ -23,6 +23,7 @@ "update-events": "cp ../event-catalog-api/packages/core/src/builtin-events/__snapshots__/* static/events", "update-sdk": "./scripts/update-sdk-docs.sh", "update-cli": "./scripts/update-cli-docs.sh", + "update-monitoring-codes": "node scripts/update-monitoring-codes.js", "update-pricing-playground": "./scripts/update-pricing-playground.sh" }, "dependencies": { diff --git a/scripts/update-monitoring-codes.js b/scripts/update-monitoring-codes.js new file mode 100644 index 0000000..08ff962 --- /dev/null +++ b/scripts/update-monitoring-codes.js @@ -0,0 +1,128 @@ +#!/usr/bin/env node +/** + * Generates the Integration Toolkit monitoring code reference from the snapshot + * emitted by erp-integration-api. + * + * The snapshot is produced and asserted by + * `packages/erp-utils/src/monitoring/code-catalog.test.ts` in that repo, so a code + * added to the taxonomy without a description fails CI there. This script turns + * that artifact into the published table — the docs never hand-maintain the list, + * which is what kept the old hub-only catalog drifting. + * + * Usage: npm run update-monitoring-codes + * Assumes erp-integration-api is checked out alongside this repo (../). + */ + +const fs = require('node:fs'); +const path = require('node:path'); + +const SNAPSHOT = path.resolve( + __dirname, + '../../erp-integration-api/packages/erp-utils/src/monitoring/__snapshots__/monitoring-codes.json', +); +const OUT = path.resolve( + __dirname, + '../docs/integrations/integration-toolkit/monitoring/codes.md', +); + +/** Rendering order and the operator-facing framing for each level. */ +const LEVELS = [ + { + key: 'error', + heading: 'Error', + blurb: + 'Something failed and the event did not do what it was meant to do. These are what the default alert rules watch.', + }, + { + key: 'warning', + heading: 'Warning', + blurb: + 'Processing continued, but something needs a human eye — often a retry in flight or a value nobody has mapped yet.', + }, + { + key: 'success', + heading: 'Success', + blurb: 'The event did what it was meant to do. Useful for confirming a sync actually landed.', + }, + { + key: 'info', + heading: 'Info', + blurb: + 'Lifecycle markers rather than outcomes: a message was queued, a duplicate was ignored, a step was skipped by configuration. Counted in total events, but excluded from the success rate.', + }, +]; + +function escapeCell(text) { + return text.replace(/\|/g, '\\|'); +} + +function render(snapshot) { + const { codes, families } = snapshot; + const byLevel = new Map(LEVELS.map((l) => [l.key, []])); + for (const entry of codes) { + if (!byLevel.has(entry.level)) { + throw new Error(`Unhandled level "${entry.level}" on code ${entry.code}`); + } + byLevel.get(entry.level).push(entry); + } + + const sections = LEVELS.map(({ key, heading, blurb }) => { + const rows = byLevel.get(key); + return [ + `## ${heading}`, + '', + blurb, + '', + '| Code | What it means |', + '|---|---|', + ...rows.map((r) => `| \`${r.code}\` | ${escapeCell(r.description)} |`), + '', + ].join('\n'); + }); + + const familySection = [ + '## Status-code families', + '', + 'Some codes are generated from an upstream response rather than drawn from the fixed list above.', + '', + '| Pattern | Level | What it means |', + '|---|---|---|', + ...families.map( + (f) => `| \`${f.pattern}\` | ${f.level} | ${escapeCell(f.description)} |`, + ), + '', + ].join('\n'); + + return `--- +sidebar_position: 2 +title: Monitoring Codes +description: Every monitoring code the Integration Toolkit emits, what it means, and what to do about it +slug: /integrations/integration-toolkit/monitoring/codes +--- + +{/* AUTOGENERATED — do not edit by hand. + Source: erp-integration-api packages/erp-utils/src/monitoring/__snapshots__/monitoring-codes.json + Regenerate: npm run update-monitoring-codes */} + +# Monitoring Codes + +Every monitoring event carries a **code** and a **level**. The code says what +happened; the level says how much you should care. Both are filterable in the +Integration Hub's [Monitoring tab](./overview.md) and through the events API. + +There are ${codes.length} codes. You are most likely here because you saw one in a failed +event — find it below. + +:::tip +A code's level is fixed. That is what makes alert rules scoped to \`_error_\` or +\`_warning_\` predictable — see [Alerting](./alerting.md#choosing-what-to-watch). +::: + +${sections.join('\n')}${familySection}`; +} + +const snapshot = JSON.parse(fs.readFileSync(SNAPSHOT, 'utf8')); +fs.writeFileSync(OUT, render(snapshot)); +console.log( + `Wrote ${path.relative(process.cwd(), OUT)} — ${snapshot.codes.length} codes, ${snapshot.families.length} famil${snapshot.families.length === 1 ? 'y' : 'ies'}`, +);