diff --git a/README.md b/README.md index d310fad..f7208cf 100644 --- a/README.md +++ b/README.md @@ -229,6 +229,7 @@ the same version hash as the simple one. | `GET .../versions/negotiate/:sessionId` | Session status, and the result or error of an async commit | | `GET .../versions/:semver/manifest` | Version manifest (add `?since=` for delta; both keyset-paginated) | | `GET .../versions/:semver/records` | Paginated records | +| `GET .../versions/:semver/records.ndjson` | Every record streamed as NDJSON in one request — the bulk read path, resumable via `?after=` | | `GET .../versions/:semver/diff?from=...` | Diff between two versions | | `POST /api/records/batch` | Fetch records by hash (JSONL stream) | | `GET /api/records/:hash/provenance` | Find all collections containing a record | diff --git a/public/llms.txt b/public/llms.txt index d184487..8309cc1 100644 --- a/public/llms.txt +++ b/public/llms.txt @@ -72,6 +72,7 @@ GET /api/collections/:owner/:slug/versions/latest → latest version with GET /api/collections/:owner/:slug/versions/:semver → specific version by semver (e.g. /versions/v1.2.0) ### Read records and files +GET /api/collections/:owner/:slug/versions/:semver/records.ndjson → ALL records, streamed as NDJSON in one request (?type=TypeName&after=recordId) GET /api/collections/:owner/:slug/versions/:semver/records → records for a version (?type=TypeName&limit=100&after=recordId) GET /api/collections/:owner/:slug/versions/:semver/manifest → manifest: record ids/types/hashes + file hashes + schema hashes (?since=v1.0.0 for delta) GET /api/collections/:owner/:slug/versions/:semver/files → list files for a version (hash, size, content type) @@ -92,15 +93,21 @@ GET /api/collections/:owner/:slug/versions/:semver/diff?from=:semver → diff be GET /api/collections/:owner/:slug/export → download .tar.gz archive (manifest.json + records/*.ndjson + files/*) GET /api/collections/:owner/:slug/export?version=v2.0.0 → export a specific version -Export assembles the whole archive in memory, so it is only offered on collections below -250,000 records. Above that it returns 413 with the record count and the limit. To read a -large collection, page the records endpoint with ?after= (see Pagination below), or fetch -the manifest if you only need id/type/hash — both work at any size. +Archive layout: manifest.json, records/.ndjson per record type, and files/. +Types too large for a single archive entry are split into numbered parts — +records/.0000.ndjson, records/.0001.ndjson, ... — at 25,000 records per part. +A type that fits in one part keeps the unnumbered records/.ndjson name, so archives +of ordinary collections are unchanged. Read every records/*.ndjson entry and you have the +version, whichever form it took. -The same 250,000-record limit applies to the SQL explorer (/api/query/...), which builds an -in-memory SQLite copy of a version. It is a UI feature rather than a documented API; on a -large collection use the records endpoint, or Hot, which hydrates a collection into a -queryable database built for the purpose. +Export is capped at 2,000,000 records and returns 413 above that — a guard against handing +back a multi-gigabyte tarball from a single GET, not a memory limit. For bulk reads prefer +records.ndjson (below): it streams, resumes, and needs no unpacking. + +The SQL explorer (/api/query/...) has a lower limit of 250,000 records, because unlike +export it genuinely does hold the whole version in memory to build a SQLite copy. It is a +UI feature rather than a documented API; on a large collection use records.ndjson, or Hot, +which hydrates a collection into a queryable database built for the purpose. ### Fork POST /api/collections/:owner/:slug/fork → fork collection into caller's org (requires write auth) @@ -445,8 +452,12 @@ To paginate through all records (works at any collection size): GET .../records?limit=2000&after= 3. Repeat until hasMore is false. -Ask for the largest page you can handle. Walking a whole collection is bounded by request -count, not bytes — 60 requests/minute unauthenticated, 5,000 authenticated — so a +BUT: if you want the whole collection, do not page it. Use the NDJSON stream below — +paging costs one round trip per page purely to re-establish a cursor the server just had. +Paging is for browsing a slice; streaming is for reading everything. + +Ask for the largest page you can handle. Walking a whole collection by paging is bounded by +request count, not bytes — 60 requests/minute unauthenticated, 5,000 authenticated — so a 3-million-record collection is 6,200 requests at 500/page and 1,550 at 2,000/page. Authenticate for any full-collection walk. @@ -455,6 +466,62 @@ beyond that. Use ?after= keyset pagination instead. --- +## Bulk Read (the fast way to get a whole collection) + +GET /api/collections/:owner/:slug/versions/:semver/records.ndjson + +Streams every record in the version as newline-delimited JSON in a single response. One +request regardless of size: a 3.1M-record collection is one call here versus 1,556 paged +ones. The server reads through a database cursor and writes as it goes, so memory is +constant on both ends — you can start processing the first line before the last is sent. + +Content-Type: application/x-ndjson +X-Underlay-Record-Count: 3113504 ← how many lines to expect + +One JSON object per line: +{"id":"arxiv:0704.0001","type":"Preprint","data":{...},"hash":"abc123..."} + +Parameters: +- type: restrict to one record type +- after: resume — return records with ids strictly after this value + +Guarantees you can rely on: +- Records are ordered by id, ascending. This is what makes `after` work. +- `hash` is the same content-address the records endpoint serves: the full record hash for + owners, the public hash for everyone else. +- Privacy filtering is identical to /records — private types and private records are + absent, private fields are stripped. + +Verify completeness yourself. A stream that dies halfway cannot report an error: the 200 +status and headers were already sent. Count the lines and compare against +X-Underlay-Record-Count (or the version's recordCount). If they differ, resume with +?after= rather than starting over. + +That resume behaviour makes this strictly better than paging for bulk reads: the same +recovery from a dropped connection, at a fraction of the requests. + +One edge: a record id is not guaranteed unique within a version — the same id can appear +under more than one hash. `after` resumes strictly past the id, so if a stream broke +between two lines sharing an id, resuming from it skips the second. Rare, and identical to +how ?after= behaves on the paged endpoint, but if you need exactness compare the final line +count against X-Underlay-Record-Count and re-read from an earlier id if it falls short. + +### Compression + +All /api/ responses are compressed when you send Accept-Encoding: gzip — about 3x on +record data. Most HTTP clients do this automatically. It applies to the NDJSON stream too. + +### When to use which + +- Whole collection, one pass → records.ndjson +- A page, or browsing → records?limit=2000&after= +- Only ids/types/hashes → manifest (≈120 bytes/record, far smaller than bodies) +- What changed since a version → manifest?since= (delta) +- Specific records you know hashes for → POST /api/records/batch (up to 10,000 per call) +- An archive to keep → export (.tar.gz) + +--- + ## Record Format { diff --git a/server.ts b/server.ts index 7939daa..950faf4 100644 --- a/server.ts +++ b/server.ts @@ -7,6 +7,7 @@ import { serveStatic } from '@hono/node-server/serve-static' import { Scalar } from '@scalar/hono-api-reference' import { Hono } from 'hono' import { createOpenApiDocument } from 'hono-zod-openapi' +import { compress } from 'hono/compress' import { cors } from 'hono/cors' import { marked } from 'marked' import type { ViteDevServer } from 'vite' @@ -115,6 +116,14 @@ app.use( app.get('/agent/:token', agentHandlers.agentPage) // --- Auth + rate limiting for API routes --- +// Responses are JSON and compress ~3x on real record data (measured on arXiv: +// 3.08 MB -> 1.00 MB for a 2,000-record page). Nothing was compressing before — +// not the app, and not Caddy, which has no `encode` directive — so every bulk +// read was paying full size on the wire. Applied in the app rather than at the +// proxy so local dev matches production, and because it covers the streaming +// endpoints too (chunked transfer compresses fine). +app.use('/api/*', compress()) + app.use('/api/*', authMiddleware) app.use('/api/*', rateLimitMiddleware) diff --git a/src/api/collections.ts b/src/api/collections.ts index 7312ea1..a4473da 100644 --- a/src/api/collections.ts +++ b/src/api/collections.ts @@ -14,10 +14,11 @@ import { getLatestReadyVersion, getOrgRole, hasOrgAccess } from '../lib/version- import { type AuthEnv } from './auth.server.js' import { requireAuth } from './auth.server.js' -// Export builds the whole archive in memory (see the guard in the export route), -// so it is offered only below this. Matches the SQL explorer's limit — both are -// whole-collection-in-memory features and should draw the line in the same place. -const MAX_EXPORT_RECORDS = 250_000 +// Export streams in bounded parts, so this is no longer a memory limit — it is a +// guard against handing someone a multi-GB tarball from a single GET. Bulk reads +// belong on records.ndjson. The SQL explorer keeps the lower 250k limit because +// that one genuinely does hold the collection in memory. +const MAX_EXPORT_RECORDS = 2_000_000 const app = new Hono() // Browse collections — public by default, or the caller's own with ?mine=true @@ -821,20 +822,20 @@ const app = new Hono() return c.json({ error: 'No versions found', statusCode: 404 }, 404) } - // The archive is assembled in memory: every record of a type is collected - // into a string[] and joined before it becomes a tar entry. That is fine - // for the collections this was built for and fatal on a multi-million - // record one — the join alone would exceed V8's maximum string length, - // and the array would exhaust the heap first, from an endpoint any - // visitor can reach. Refuse above the threshold rather than fall over. + // Export used to be capped because it assembled the whole archive in + // memory. It now emits bounded parts, so size is no longer the limit — + // but it is still a single long-running response that reads every record, + // and `records.ndjson` is the better tool for bulk reads (streaming, + // resumable, no tar to unpack). The cap stays well above any real + // collection purely as a guard against an accidental multi-GB download. if (version.recordCount > MAX_EXPORT_RECORDS) { return c.json( { error: `This collection has ${version.recordCount.toLocaleString()} records; export is ` + - `available below ${MAX_EXPORT_RECORDS.toLocaleString()}. Read it through the ` + - `records API instead (GET .../versions/:n/records?after=…), which pages at any ` + - `depth, or the manifest endpoint if you only need hashes.`, + `available below ${MAX_EXPORT_RECORDS.toLocaleString()}. For bulk reads use ` + + `GET .../versions/:n/records.ndjson, which streams the whole version in one ` + + `request and resumes with ?after=.`, recordCount: version.recordCount, maxRecords: MAX_EXPORT_RECORDS, statusCode: 413, @@ -901,10 +902,37 @@ const app = new Hono() .from(schema.versionRecords) .where(eq(schema.versionRecords.versionId, version.id)) + // tar needs each entry's byte length in its header, so an entry cannot be + // written from an unbounded stream. Previously every record of a type was + // collected into a string[] and joined — which is why export had to be + // capped: at 3.1M records the array exhausts the heap, and the join would + // exceed V8's maximum string length even if it didn't. + // + // Instead each type is emitted in bounded parts. A type that fits in one + // part keeps the original `records/.ndjson` name, so every archive + // that works today is byte-identical; only types too large for a single + // part split into `records/.0000.ndjson`, `.0001.ndjson`, … Memory is + // one part regardless of collection size. + const RECORDS_PER_PART = 25_000 + for (const { type } of types) { - const lines: string[] = [] let batchCursor: string | null = null let batchHasMore = true + let partIndex = 0 + let pending: string[] = [] + + const flushPart = (isFinalPart: boolean) => { + if (pending.length === 0) return + const name = + partIndex === 0 && isFinalPart + ? `records/${type}.ndjson` + : `records/${type}.${String(partIndex).padStart(4, '0')}.ndjson` + const buf = Buffer.from(pending.join('\n') + '\n') + pack.entry({ name, size: buf.length }, buf) + pending = [] + partIndex++ + } + while (batchHasMore) { // Walk the (version_id, type, record_id) index; record_objects is // joined only to pick up the body for the rows on this page. @@ -934,11 +962,12 @@ const app = new Hono() const page = batchHasMore ? batch.slice(0, 5000) : batch if (page.length > 0) batchCursor = page[page.length - 1]!.recordId for (const r of page) { - lines.push(JSON.stringify({ id: r.recordId, type: r.type, data: r.data })) + pending.push(JSON.stringify({ id: r.recordId, type: r.type, data: r.data })) } + // Emit whenever a part fills, so `pending` never grows past one part. + if (pending.length >= RECORDS_PER_PART) flushPart(false) } - const buf = Buffer.from(lines.join('\n') + '\n') - pack.entry({ name: `records/${type}.ndjson`, size: buf.length }, buf) + flushPart(true) } // Add files diff --git a/src/api/versions.ts b/src/api/versions.ts index 8073745..a5ab483 100644 --- a/src/api/versions.ts +++ b/src/api/versions.ts @@ -1,6 +1,7 @@ import { and, eq, sql } from 'drizzle-orm' import { Hono } from 'hono' import { openApi } from 'hono-zod-openapi' +import { stream } from 'hono/streaming' import { z } from 'zod' import { db, schema } from '../db/client.server.js' @@ -506,6 +507,185 @@ const app = new Hono() }) }, ) + // Stream every record in a version as NDJSON, in one request + .get( + '/:owner/:slug/versions/:n/records.ndjson', + openApi({ + tags: ['Versions'], + summary: 'Stream all records in a version as NDJSON', + description: + 'The bulk read path. Paging `/records` costs one round trip per page — 1,556 requests ' + + 'for a 3.1M-record collection — purely to re-establish a cursor the server just had. ' + + 'This streams the whole version in a single response, one JSON object per line, using a ' + + 'server-side cursor: memory is constant on both ends regardless of collection size. ' + + 'Records are ordered by id, and `?after=` resumes from the last id you saw, so a dropped ' + + 'connection costs the remainder rather than the whole read. Compare the line count ' + + "against the version's `recordCount` to confirm you received all of it — a truncated " + + 'stream cannot be signalled in the status code, which is already sent.', + request: { param: z.object({ owner: z.string(), slug: z.string(), n: z.string() }) }, + responses: { 200: z.any() }, + }), + async (c) => { + const { owner, slug, n } = c.req.valid('param') + const type = c.req.query('type') + const after = c.req.query('after') + + const collection = await resolveAccessibleCollection(owner, slug, c.get('userId')) + if (!collection) return c.json({ error: 'Collection not found', statusCode: 404 }, 404) + + const { semver } = parseSemver(n) + const [version] = await db + .select({ id: schema.versions.id, recordCount: schema.versions.recordCount }) + .from(schema.versions) + .where( + and( + eq(schema.versions.collectionId, collection.id), + eq(schema.versions.semver, semver), + eq(schema.versions.status, 'ready'), + ), + ) + .limit(1) + + if (!version) return c.json({ error: 'Version not found', statusCode: 404 }, 404) + + const ownerAccess = collection.ownerAccess + let privateTypes = new Set() + let schemaEntries: SchemaEntry[] = [] + if (!ownerAccess) { + schemaEntries = await loadVersionSchemas(version.id) + privateTypes = getPrivateTypes(schemaEntries) + if (type && privateTypes.has(type)) { + // Requesting a private type as a non-owner: an empty stream, not a 404, + // so callers iterating types don't have to special-case it. + c.header('Content-Type', 'application/x-ndjson') + return stream(c, async () => {}) + } + } + + // Private fields are resolved once per type rather than per record; at + // millions of rows the difference is not marginal. + const privateFieldsByType = new Map>() + for (const entry of schemaEntries) { + const fields = getPrivateFields(entry.schema) + if (fields.size > 0) privateFieldsByType.set(entry.slug, fields) + } + + const client = db.$client + const privateTypeList = [...privateTypes] + + // Hono's compress() middleware deliberately skips this response: its + // compressible-type list covers application/json and +json suffixes but + // not application/x-ndjson, and it bails on anything already marked + // Transfer-Encoding: chunked. Both are true here, so the route compresses + // itself — this is the response that benefits most, at roughly 3x. + const acceptsGzip = (c.req.header('Accept-Encoding') ?? '').includes('gzip') + + const encoder = new TextEncoder() + + // Read in keyset batches rather than one unbounded query. + // + // The obvious implementation — a single ORDER BY over the whole version, + // read through a cursor — does NOT stream. Postgres has to satisfy the + // sort before it can return the first row, and with no index supplying + // that order it sorts every row externally: on a 3.1M-record version that + // is ~3.5GB of temp files, ~46s before the first byte, and an ERROR 53100 + // when temp space runs out. A client-side cursor bounds the client's + // memory, not the server's. + // + // Adding LIMIT changes the plan qualitatively. Bounded, Postgres walks + // (version_id, record_id) in index order and finishes with an incremental + // sort over each small group of equal record_ids — tens of kilobytes, in + // memory, no temp files. So this issues many bounded queries instead of + // one unbounded one: constant memory on both sides, first row in + // milliseconds, nothing spilled to disk. + const BATCH = 5_000 + // record_id is not unique within a version (a record can appear under more + // than one hash), so the batch cursor is the (record_id, hash) pair. + // Advancing on record_id alone would drop or repeat rows whenever a + // duplicated id straddled a batch boundary. + let lastId: string | null = null + let lastHash: string | null = null + + // `pull` rather than a loop in `start`: the stream produces a batch only + // when the consumer is ready for one, so a slow client throttles the reads + // instead of letting them pile up in memory. + const source = new ReadableStream({ + async pull(controller) { + try { + // The caller's `after` is an id: it resumes strictly past that id, + // the same semantics the paged endpoint uses. Continuation between + // batches is by the full pair, which is what keeps duplicate ids + // intact. + const keyset = + lastId !== null + ? client`AND (vr.record_id, vr.record_hash) > (${lastId}, ${lastHash})` + : after + ? client`AND vr.record_id > ${after}` + : client`` + + const rows = await client` + SELECT vr.record_id AS id, vr.type, vr.record_hash AS record_hash, + ro.data, + ${ownerAccess ? client`ro.hash` : client`coalesce(vr.public_record_hash, ro.hash)`} AS hash + FROM version_records vr + INNER JOIN record_objects ro ON ro.hash = vr.record_hash + WHERE vr.version_id = ${version.id} + ${type ? client`AND vr.type = ${type}` : client``} + ${keyset} + ${ownerAccess ? client`` : client`AND ro.private = false AND vr.type <> ALL(${privateTypeList}::text[])`} + ORDER BY vr.record_id, vr.record_hash + LIMIT ${BATCH} + ` + + if (rows.length === 0) { + controller.close() + return + } + + let out = '' + for (const row of rows) { + const rowType = row['type'] as string + const privateFields = ownerAccess ? undefined : privateFieldsByType.get(rowType) + const data = + privateFields && privateFields.size > 0 + ? filterRecordData(row['data'], privateFields) + : row['data'] + out += + JSON.stringify({ id: row['id'], type: rowType, data, hash: row['hash'] }) + '\n' + } + const tail = rows[rows.length - 1]! + lastId = tail['id'] as string + lastHash = tail['record_hash'] as string + // One enqueue per batch rather than per record: 3.1M individual + // writes spends more time in the stream machinery than in the + // database. + controller.enqueue(encoder.encode(out)) + if (rows.length < BATCH) controller.close() + } catch (err) { + controller.error(err) + } + }, + }) + + const headers: Record = { + 'Content-Type': 'application/x-ndjson', + // Lets a client verify completeness without a second request. + 'X-Underlay-Record-Count': String(version.recordCount), + } + if (acceptsGzip) headers['Content-Encoding'] = 'gzip' + + // The DOM lib types CompressionStream's writable as BufferSource, which + // does not unify with ReadableStream; the pairing is correct at + // runtime. + const gzip = new CompressionStream('gzip') as unknown as ReadableWritablePair< + Uint8Array, + Uint8Array + > + const responseBody: ReadableStream = acceptsGzip ? source.pipeThrough(gzip) : source + + return new Response(responseBody, { headers }) + }, + ) // List files for a version .get( '/:owner/:slug/versions/:n/files', diff --git a/src/routes/docs/api/versions.tsx b/src/routes/docs/api/versions.tsx index 3442b3f..31b8e13 100644 --- a/src/routes/docs/api/versions.tsx +++ b/src/routes/docs/api/versions.tsx @@ -123,6 +123,14 @@ const getRecordsRes = `{ } }` +const ndjsonRes = `HTTP/1.1 200 OK +Content-Type: application/x-ndjson +X-Underlay-Record-Count: 3113504 + +{"id":"pub-001","type":"Publication","data":{"title":"..."},"hash":"sha256:..."} +{"id":"pub-002","type":"Publication","data":{"title":"..."},"hash":"sha256:..."} +{"id":"pub-003","type":"Publication","data":{"title":"..."},"hash":"sha256:..."}` + const manifestRes = `{ "semver": "v1.1.0", "hash": "a1b2c3d4...", @@ -616,6 +624,71 @@ export default function DocsApiVersions() {
+
+

GET /api/collections/:owner/:slug/versions/:n/records.ndjson

+

No auth for public collections

+

+ Every record in the version, streamed as newline-delimited JSON in a single response. This + is the bulk read path. Paging /records costs a round trip per page purely to + re-establish a cursor the server just had — 1,556 requests for a 3.1-million-record + collection, against one here. The server reads through a database cursor and writes as it + goes, so memory stays constant on both ends and you can process the first line before the + last is sent. +

+

Query parameters

+ + + + + + + + + + + +
+ type + Restrict to a single record type
+ after + + Resume: emit only records with ids strictly after this value. Records are ordered by + id ascending, so this restarts a dropped read from where it stopped rather than from + the beginning. +
+

+ Response 200 +

+ 
+          {ndjsonRes}
+
+

+ hash is the same content-address /records serves: the full + record hash for owners, the public hash for everyone else. Privacy filtering is identical + too — private types and private records are absent, private fields stripped. +

+

+ Check completeness yourself. A stream that fails partway cannot report + it: the 200 and headers were sent before anything went wrong.{' '} + X-Underlay-Record-Count tells you how many lines to expect. If you receive + fewer, resume with ?after= set to the id of the last complete line you parsed + — don't start over. +

+

+ A record id is not guaranteed unique within a version — the same id can appear under more + than one hash. Because after resumes strictly past the id, a stream that + broke between two lines sharing an id will skip the second on resume. This matches{' '} + /records paging, and the line-count check above is what catches it. +

+

+ Responses are compressed when you send Accept-Encoding: gzip, which most HTTP + clients do automatically — roughly 3× on record data, and it applies to this stream as + well. +

+
+ +
+

GET /api/collections/:owner/:slug/versions/:n/manifest

No auth for public collections

diff --git a/src/routes/docs/integration.tsx b/src/routes/docs/integration.tsx index 9632753..f257240 100644 --- a/src/routes/docs/integration.tsx +++ b/src/routes/docs/integration.tsx @@ -370,6 +370,15 @@ export default function DocsIntegration() { Get records (paginated) + + + GET .../versions/:semver/records.ndjson + + + Stream every record in one request (NDJSON). The bulk read path — use this instead of + paging when you want the whole collection + + GET .../versions/:semver/manifest diff --git a/src/routes/protocol.tsx b/src/routes/protocol.tsx index fccc4a1..50fa9c3 100644 --- a/src/routes/protocol.tsx +++ b/src/routes/protocol.tsx @@ -91,7 +91,16 @@ GET /api/collections/:owner/:slug/versions/v2.0.0/manifest?since=v1.1.0 # Fetch only the records you need POST /api/records/batch { "hashes": ["abc123...", "def456..."] } -# Returns JSONL stream` +# Returns JSONL stream + +# Or read the whole version in one streamed response +GET /api/collections/:owner/:slug/versions/v2.0.0/records.ndjson +Content-Type: application/x-ndjson +X-Underlay-Record-Count: 3113504 + +{"id":"pub-001","type":"Publication","data":{...},"hash":"..."} +{"id":"pub-002","type":"Publication","data":{...},"hash":"..."} +# ... one object per line, ordered by id; ?after= resumes` const schemaExample = `{ "type": "object", @@ -485,6 +494,48 @@ export default function Protocol() { walk may hold only updated entries. The cursor is opaque — pass back what you were given rather than constructing one.

+

Reading a whole version

+

+ Paging is the wrong shape for "give me everything": each request pays a round trip to + re-establish a cursor the server just had, so a three-million-record collection costs + over fifteen hundred of them. records.ndjson streams the entire version + in one response, read through a database cursor and written as it goes, so neither + side holds more than a chunk. +

+

+ Four properties make it usable as a protocol rather than a convenience, and an + implementation is expected to honour all four: +

+
    +
  • + Ordered by record id, ascending. This is what gives{' '} + ?after= meaning, and it is the difference between a stream you can + resume and one you must restart. +
    • +
  • + One JSON object per line, of the form{' '} + {'{id, type, data, hash}'}. hash is the same + content-address the paged endpoint serves — the full record hash for owners, the + public hash for everyone else. +
    • +
  • + Privacy filtering is identical to the paged endpoint. Private types + and private records are absent; private fields are stripped. A reader must not be + able to learn more by choosing a different transport. +
    • +
  • + Completeness is the reader's to verify. A stream that fails partway + cannot say so — its 200 and headers left before the failure did. + X-Underlay-Record-Count states how many lines to expect; count them, + and resume from the last complete line with ?after=. +
    • +
+

+ That last point is a deliberate trade rather than an oversight. Any single-response + bulk format has it — the alternative is paging, which buys per-page error reporting at + the cost of a round trip per page. Making the expected count explicit lets a client + get the safety without the round trips. +