Skip to content

Add export_records_chunked for memory-bounded exports of large projects#328

Open
oldrobotdev wants to merge 1 commit into
redcap-tools:masterfrom
oldrobotdev:stream-export-records
Open

Add export_records_chunked for memory-bounded exports of large projects#328
oldrobotdev wants to merge 1 commit into
redcap-tools:masterfrom
oldrobotdev:stream-export-records

Conversation

@oldrobotdev

Copy link
Copy Markdown

Closes #308.

export_records pulls the whole cohort in one API call and decodes the entire body into memory (request.py execute does one session.post, then response.json() or response.text), so a 20k-subject export sits in RAM all at once.

This adds export_records_chunked, an opt-in generator that implements the record-id batching you suggested in the issue: it exports just the record_id column once, then yields the records in batches of chunk_size, so the client never holds more than one batch plus the id list. export_records itself is untouched, so nothing existing changes.

for batch in project.export_records_chunked(chunk_size=1000):
    ...  # each batch is a normal export_records result (json/csv/xml/df)

It takes the same filtering and formatting arguments as export_records (fields, forms, events, filter_logic, date_begin/date_end, format_type, and so on) and passes them through on each batch. json and df batches concatenate cleanly; csv and xml batches each carry their own header row, which is noted in the docstring.

On the server-load concern raised later in the thread: this doesn't fetch more than the manual workaround would, it just automates it, and it's opt-in, so anyone worried about load can keep using export_records or page with date_begin/date_end. It's aimed at the memory-constrained client (small FaaS instances and the like), not at reducing server work or doing incremental sync.

Tests: 6 new tests using the existing mocked-API fixtures (no live server), full unit suite 125 passed. black, mypy and pylint clean.

export_records materializes the entire response in memory, which is a
problem for large cohorts (20k+ records). Add an opt-in generator that
exports the record IDs first, then exports them chunk_size at a time,
yielding one batch per iteration. Only a single batch is held in memory,
so large projects can be streamed to disk or processed incrementally.
The smaller requests are also easier on the REDCap server.

export_records is untouched, so existing behavior is unchanged. Each
yielded batch matches the shape export_records would return for the
requested format_type.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Streaming Response

1 participant