Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
335 changes: 335 additions & 0 deletions docs/send-guard.md

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions src/app/invite/award-referral-reward.ts
Original file line number Diff line number Diff line change
Expand Up @@ -179,11 +179,18 @@ export const awardReferralRewardOnKycApproval = async ({
const { intraledgerPaymentSendWalletIdForUsdWallet } = await import(
"@app/payments/send-intraledger"
)
const { SEND_GUARD_NOT_APPLICABLE } = await import(
"@app/payments/send-guard-optout"
)
const result = await intraledgerPaymentSendWalletIdForUsdWallet({
senderWalletId: rewardsWallet.id,
recipientWalletId,
amount: amountCents,
memo,
// A referral payout is a system credit out of the rewards wallet, not
// a user-initiated send: see SEND_GUARD_NOT_APPLICABLE. Guarding it
// would rate-limit a payout batch against itself.
authorize: SEND_GUARD_NOT_APPLICABLE,
})
if (result instanceof Error) {
baseLogger.error(
Expand Down
5 changes: 5 additions & 0 deletions src/app/payments/add-earn.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ import { AccountsIpsRepository } from "@services/mongoose/accounts-ips"

import { intraledgerPaymentSendWalletIdForBtcWallet } from "./send-intraledger"

import { SEND_GUARD_NOT_APPLICABLE } from "./send-guard-optout"

export const addEarn = async ({
quizQuestionId: quizQuestionIdString,
accountId,
Expand Down Expand Up @@ -90,6 +92,9 @@ export const addEarn = async ({
amount,
memo: quizQuestionId,
senderAccount: funderAccount,
// A quiz reward is a system credit out of the funder wallet, not a
// user-initiated send: see SEND_GUARD_NOT_APPLICABLE.
authorize: SEND_GUARD_NOT_APPLICABLE,
})
if (payment instanceof Error) return payment

Expand Down
609 changes: 609 additions & 0 deletions src/app/payments/authorize-send.ts

Large diffs are not rendered by default.

37 changes: 33 additions & 4 deletions src/app/payments/idempotency.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ type CachedPaymentSend = { fingerprint: string; result: PaymentSendStatus }
* - Completed result cached, same fingerprint → returns the stored result.
* `execute()` is never called, so no new IBEX invoice is minted, no second
* payment, and (because the ops-event notify lives inside `execute`) no duplicate
* ops event fires.
* ops event fires. `authorize()` is not called either — see below.
* - Completed result cached, DIFFERENT fingerprint → returns `IdempotencyKeyReuseError`
* and does NOT execute. Replaying the original result here would silently drop the
* new payment while reporting the old one's success. This check runs on both the
Expand All @@ -54,6 +54,23 @@ type CachedPaymentSend = { fingerprint: string; result: PaymentSendStatus }
* fresh attempt with the same key can retry. The lock still guards the concurrent
* window regardless of caching.
*
* ENG-573: the `authorize` hook (the send guard) runs INSIDE the lock, after
* the in-lock cache re-check and immediately before `execute` — i.e. only on
* the path that is about to execute a new payment. It is REQUIRED: while it was
* optional, wiring a new rail through this wrapper and forgetting the guard
* compiled, passed its tests and shipped unguarded — and this fork keeps adding
* rails that pay IBEX directly from a resolver (ENG-533). A caller that must
* not be guarded says so with `SEND_GUARD_NOT_APPLICABLE`
* (@app/payments/send-guard-optout), which is greppable; silence is not. Running it in the
* resolver ahead of this wrapper (the original wiring) made every retry of a
* timed-out send spend attempt budget and re-check the amount cap, so a client
* retrying past the burst budget got "Too many payment attempts" instead of the
* cached success of a payment that had already moved money — and a sats send
* near the cap could flip to "Cannot transfer more than $X" on retry because
* the mid price had ticked. A replay must cost nothing and must not be
* re-judged. An `authorize` rejection is an ApplicationError, so like any other
* error it is returned uncached and the key stays retryable.
*
* Reuses existing primitives: `RedisCacheService` for the result store and
* `LockService().lockPaymentIdempotencyKey` (a redlock `.using` lock that releases
* when `execute` finishes) for the in-flight guard.
Expand All @@ -72,19 +89,31 @@ export const withPaymentIdempotency = async ({
idempotencyKey,
senderWalletId,
requestFingerprint,
authorize,
execute,
}: {
idempotencyKey: string | null | undefined
senderWalletId: WalletId
requestFingerprint: string
// Runs only on the path that will actually execute a new payment; a rejection
// short-circuits without executing. Required — pass
// `SEND_GUARD_NOT_APPLICABLE` to opt a system credit out explicitly. See the
// ENG-573 note above.
authorize: SendGuardHook
execute: () => Promise<PaymentSendResult>
}): Promise<PaymentSendResult> => {
const authorizeThenExecute = async (): Promise<PaymentSendResult> => {
const authorized = await authorize()
if (authorized instanceof Error) return authorized
return execute()
}

// No key supplied → unchanged behavior.
if (!idempotencyKey) return execute()
if (!idempotencyKey) return authorizeThenExecute()

const trimmedKey = idempotencyKey.trim()
// A blank / whitespace-only key is treated as "no key" (unchanged behavior).
if (trimmedKey.length === 0) return execute()
if (trimmedKey.length === 0) return authorizeThenExecute()
if (trimmedKey.length > MAX_KEY_LENGTH) {
return new InvalidIdempotencyKeyError(idempotencyKey)
}
Expand Down Expand Up @@ -112,7 +141,7 @@ export const withPaymentIdempotency = async ({
const cachedInLock = await cache.get<CachedPaymentSend>({ key: cacheKey })
if (!(cachedInLock instanceof Error)) return resolveCached(cachedInLock)

const outcome = await execute()
const outcome = await authorizeThenExecute()

// Persist only a definitive payment outcome. Errors stay uncached so a
// fresh attempt with the same key can retry.
Expand Down
15 changes: 15 additions & 0 deletions src/app/payments/index.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,20 @@
export * from "./get-protocol-fee"
export * from "./idempotency"
// Named exports, not `export *`. The wildcard put
// `__resetOpsEventCoalescingForTest` — a test-only mutator of the ops-event
// coalescing windows — on the `Payments` public surface, one autocomplete away
// from a request handler; calling it there drops every accumulated `muted`
// count and silently makes the ops feed lossy in exactly the way the coalescing
// design exists to prevent. The reset stays importable from the module itself,
// which is all the spec needs.
export {
authorizeSend,
gateSend,
SendRejectionReasons,
OPS_EVENT_COALESCE_MS,
SEND_GUARD_SPAN_NAME,
} from "./authorize-send"
export type { SendKind, SendAmountInput, SendRejectionReason } from "./authorize-send"
export * from "./send-lightning"
export * from "./send-intraledger"
export * from "./update-pending-payments"
Expand Down
23 changes: 23 additions & 0 deletions src/app/payments/send-guard-optout.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/**
* ENG-573: the explicit opt-out from the send guard.
*
* `authorize` is REQUIRED on every arg type that carries it (see
* `SendGuardHook` in @app/wallets/index.types.d.ts). Optional meant a send path
* that forgot the hook compiled, passed its tests and shipped unguarded — and
* two such paths are already sitting in the tree with their bodies commented
* out, waiting for someone to re-enable them. Requiring the field turns that
* silent omission into a compile error.
*
* System credits are the legitimate exception: quiz rewards, referral payouts,
* card top-up credits and operator reimbursements are not user-initiated sends.
* They move money OUT of a Flash-owned funding wallet on our own instruction,
* so an account-scoped attempt budget and a per-account daily cap describe
* nothing about them — a 30-payment referral batch would rate-limit itself.
* They opt out by name, which is also the grep that answers "what still sends
* without the guard".
*
* Deliberately dependency-free: the callers below lazy-import the send
* functions specifically to keep the IBEX client and the Redis-backed rate
* limiter out of unrelated module graphs, and importing this must not undo it.
*/
export const SEND_GUARD_NOT_APPLICABLE: SendGuardHook = async () => true
5 changes: 5 additions & 0 deletions src/app/payments/send-intraledger.ts
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,9 @@ export const intraledgerPaymentSendWalletIdForBtcWallet = async (
idempotencyKey: args.idempotencyKey,
senderWalletId: args.senderWalletId,
requestFingerprint: `intraledger|${args.recipientWalletId}|${args.amount}`,
// ENG-573 send guard. Inside the wrapper, so a replayed key never spends
// attempt budget nor gets re-judged against a moved mid price.
authorize: args.authorize,
execute: async () => {
const validated = await validateIsBtcWallet(args.senderWalletId)
const result =
Expand All @@ -153,6 +156,8 @@ export const intraledgerPaymentSendWalletIdForUsdWallet = async (
idempotencyKey: args.idempotencyKey,
senderWalletId: args.senderWalletId,
requestFingerprint: `intraledger|${args.recipientWalletId}|${args.amount}`,
// ENG-573 send guard — see the BTC wrapper above.
authorize: args.authorize,
execute: async () => {
const validated = await validateIsUsdWallet(args.senderWalletId, {
includeUsdt: true,
Expand Down
9 changes: 9 additions & 0 deletions src/app/payments/send-lightning.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,10 @@ export const payInvoiceByWalletId = async (
idempotencyKey: args.idempotencyKey,
senderWalletId: args.senderWalletId,
requestFingerprint: `ln|${args.uncheckedPaymentRequest}`,
// ENG-573 send guard — see the no-amount wrappers below. Stubbed out of
// `lnInvoicePaymentSend` today (that resolver pays IBEX inline and guards
// it there), so this is what makes re-enabling it carry the guard.
authorize: args.authorize,
execute: async () => {
const result = await executePayInvoiceByWalletId(args)
notifyLightningSendResult(args, result)
Expand Down Expand Up @@ -225,6 +229,9 @@ export const payNoAmountInvoiceByWalletIdForBtcWallet = async (
idempotencyKey: args.idempotencyKey,
senderWalletId: args.senderWalletId,
requestFingerprint: `ln-noamount|${args.uncheckedPaymentRequest}|${args.amount}`,
// ENG-573 send guard. Inside the wrapper, so a replayed key never spends
// attempt budget nor gets re-judged against a moved mid price.
authorize: args.authorize,
execute: async () => {
const validated = await validateIsBtcWallet(args.senderWalletId)
const result =
Expand All @@ -241,6 +248,8 @@ export const payNoAmountInvoiceByWalletIdForUsdWallet = async (
idempotencyKey: args.idempotencyKey,
senderWalletId: args.senderWalletId,
requestFingerprint: `ln-noamount|${args.uncheckedPaymentRequest}|${args.amount}`,
// ENG-573 send guard — see the BTC wrapper above.
authorize: args.authorize,
execute: async () => {
const validated = await validateIsUsdWallet(args.senderWalletId, {
includeUsdt: true,
Expand Down
39 changes: 39 additions & 0 deletions src/app/wallets/index.types.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,27 @@ type GetOnChainFeeArgs = GetOnChainFeeWithoutCurrencyArgs & {
amountCurrency: WalletCurrency
}

// ENG-573: the send guard, as the idempotency wrapper's `authorize` hook. A
// resolver hands this down instead of awaiting the guard itself, so a replayed
// idempotency key returns the cached result without spending attempt budget or
// being re-judged against a moved mid price. See @app/payments/idempotency.
//
// REQUIRED — on `withPaymentIdempotency` itself and on every send-function arg
// type that reaches it. It was optional, and optional means a send path that
// forgets the hook compiles, passes its tests and ships unguarded. A caller that
// genuinely must not be guarded says so with `SEND_GUARD_NOT_APPLICABLE` below;
// for anything routed through the idempotency wrapper, silence is no longer an
// option the compiler accepts.
//
// What that does NOT cover: the on-chain rails do not go through the wrapper —
// `payOnChainByWalletId` (@app/wallets/send-on-chain) takes no hook, and the two
// live on-chain sends call `authorizeSend` in the resolver instead
// (`onchain-payment-send-all.ts`, `onchain-usd-payment-send.ts`). So whoever
// re-enables the stubbed `onchain-payment-send.ts` /
// `onchain-usd-payment-send-as-sats.ts` bodies has to add that call by hand;
// nothing here makes the compiler ask for it.
type SendGuardHook = () => Promise<true | ApplicationError>

type PaymentSendArgs = {
senderWalletId: WalletId
senderAccount?: Account
Expand All @@ -93,12 +114,25 @@ type PaymentSendArgs = {
type PayInvoiceByWalletIdArgs = PaymentSendArgs & {
uncheckedPaymentRequest: string
senderAccount: Account
// ENG-573 send guard, handed to `withPaymentIdempotency` as its `authorize`
// hook so it runs only on the path that will actually pay. See
// `SendGuardHook` above. Required — pass `SEND_GUARD_NOT_APPLICABLE` to opt a
// system credit out explicitly. This function is stubbed out of
// `lnInvoicePaymentSend` today (that resolver pays IBEX inline); the hook is
// required so re-enabling it cannot quietly drop the guard the inline path
// has.
authorize: SendGuardHook
}

type PayNoAmountInvoiceByWalletIdArgs = PaymentSendArgs & {
uncheckedPaymentRequest: string
amount: number
senderAccount: Account
// ENG-573 send guard, handed to `withPaymentIdempotency` as its `authorize`
// hook so it runs only on the path that will actually pay. See
// `SendGuardHook` above. Required — pass `SEND_GUARD_NOT_APPLICABLE` to opt a
// system credit out explicitly.
authorize: SendGuardHook
}

type IntraLedgerPaymentSendUsernameArgs = PaymentSendArgs & {
Expand All @@ -109,6 +143,11 @@ type IntraLedgerPaymentSendUsernameArgs = PaymentSendArgs & {
type IntraLedgerPaymentSendWalletIdArgs = PaymentSendArgs & {
recipientWalletId: WalletId
amount: number
// ENG-573 send guard, handed to `withPaymentIdempotency` as its `authorize`
// hook so it runs only on the path that will actually pay. See
// `SendGuardHook` above. Required — pass `SEND_GUARD_NOT_APPLICABLE` to opt a
// system credit out explicitly.
authorize: SendGuardHook
}

type PayOnChainByWalletIdResult = {
Expand Down
71 changes: 69 additions & 2 deletions src/config/schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,17 @@ const accountLimitConfigSchema = {
0: { type: "integer" },
1: { type: "integer" },
2: { type: "integer" },
// ENG-573: Business (L3) accounts exist in prod; without a limit here
// getAccountLimits({ level: 3 }) is NaN and the send guard fails closed.
// `required` like its siblings: the block-level default below carries a
// level 3, but a deployment that overrides `accountLimits` PARTIALLY
// replaces that default wholesale, and Ajv fills nothing back in. Levels
// 0-2 fail loudly at boot in that case; without this entry level 3 would
// instead resolve to NaN and silently block every Business account's
// sends at runtime. Boot-time failure is the cheaper of the two.
3: { type: "integer" },
},
required: ["0", "1", "2"],
required: ["0", "1", "2", "3"],
additionalProperties: false,
},
},
Expand Down Expand Up @@ -365,6 +374,23 @@ export const configSchema = {
invoiceCreateAttempt: rateLimitConfigSchema,
invoiceCreateForRecipientAttempt: rateLimitConfigSchema,
onChainAddressCreateAttempt: rateLimitConfigSchema,
// ENG-573 send guard: per-account budget on send *attempts*. Property-
// level defaults rather than `required` entries, because prod overrides
// the whole rateLimits block (deployments flash-values.tmpl.yaml) and a
// new required key there would fail config validation at boot.
paymentSendAttempt: {
...rateLimitConfigSchema,
default: { points: 10, duration: 60, blockDuration: 60 },
},
// `blockDuration` MUST match `duration` here. rate-limiter-flexible
// rewrites the key's TTL to blockDuration on the first breach
// (RateLimiterStoreAbstract._afterConsume -> _block), so a shorter block
// throws the daily counter away early and grants a fresh 200 points: a
// 3600 block against an 86400 window is really 200/hour, ~4,800/day.
paymentSendDailyAttempt: {
...rateLimitConfigSchema,
default: { points: 200, duration: 86400, blockDuration: 86400 },
},
},
required: [
"requestCodePerLoginIdentifier",
Expand Down Expand Up @@ -502,24 +528,65 @@ export const configSchema = {
"0": 12500,
"1": 100000,
"2": 5000000,
"3": 5000000, // ENG-573 placeholder: L3 inherits L2 until the ladder is decided
},
},
intraLedger: {
level: {
"0": 12500,
"1": 200000,
// ENG-573 decision (2026-09-08, operator): level 1 is ONE limit,
// $1,000, not $1,000 external / $2,000 internal. Galoy shipped them
// split; that split was the only place the guard's rail-vs-
// destination approximation could change an outcome, and closing it
// downward is the conservative resolution. Every level now carries
// equal withdrawal and intraLedger limits.
"1": 100000,
"2": 5000000,
"3": 5000000, // ENG-573 placeholder: L3 inherits L2 until the ladder is decided
},
},
tradeIntraAccount: {
level: {
"0": 200000,
"1": 5000000,
"2": 20000000,
"3": 20000000, // ENG-573 placeholder: L3 inherits L2 until the ladder is decided
},
},
},
},
// ENG-573 send guard (src/app/payments/authorize-send.ts). The guard is the
// first Flash-side amount cap that has ever rejected anything, so it ships
// with an operator switch rather than going straight to hard enforcement on
// 100% of sends:
//
// off — the guard returns immediately. No Redis, no price lookup, no
// ops event. Sends behave exactly as they did before ENG-573.
// log-only — DEFAULT. All three checks run and every would-be rejection
// posts a `transfer / would-reject` ops event, but the send is
// authorised. Read a day of those before flipping to enforce:
// nobody has yet measured what fraction of real traffic these
// numbers reject, and the ~300 prod accounts with no `level`
// field (174 with usernames) fall to the level-0 $125 cap.
// enforce — rejections are real.
//
// `mode` carries a property-level default AND the block carries one, so a
// partial yaml override can never leave it undefined (cf. accountLimits
// level 3 above). getSendGuardMode() coerces anything unrecognised back to
// log-only: a typo in a values file must not turn the guard into a
// fail-closed wall in front of every send.
sendGuard: {
type: "object",
properties: {
mode: {
type: "string",
enum: ["off", "log-only", "enforce"],
default: "log-only",
},
},
additionalProperties: false,
default: { mode: "log-only" },
},
spamLimits: {
type: "object",
properties: {
Expand Down
Loading
Loading