Add export_records_chunked for memory-bounded exports of large projects#328
Open
oldrobotdev wants to merge 1 commit into
Open
Add export_records_chunked for memory-bounded exports of large projects#328oldrobotdev wants to merge 1 commit into
oldrobotdev wants to merge 1 commit into
Conversation
export_records materializes the entire response in memory, which is a problem for large cohorts (20k+ records). Add an opt-in generator that exports the record IDs first, then exports them chunk_size at a time, yielding one batch per iteration. Only a single batch is held in memory, so large projects can be streamed to disk or processed incrementally. The smaller requests are also easier on the REDCap server. export_records is untouched, so existing behavior is unchanged. Each yielded batch matches the shape export_records would return for the requested format_type.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #308.
export_records pulls the whole cohort in one API call and decodes the entire body into memory (request.py execute does one session.post, then response.json() or response.text), so a 20k-subject export sits in RAM all at once.
This adds export_records_chunked, an opt-in generator that implements the record-id batching you suggested in the issue: it exports just the record_id column once, then yields the records in batches of chunk_size, so the client never holds more than one batch plus the id list. export_records itself is untouched, so nothing existing changes.
It takes the same filtering and formatting arguments as export_records (fields, forms, events, filter_logic, date_begin/date_end, format_type, and so on) and passes them through on each batch. json and df batches concatenate cleanly; csv and xml batches each carry their own header row, which is noted in the docstring.
On the server-load concern raised later in the thread: this doesn't fetch more than the manual workaround would, it just automates it, and it's opt-in, so anyone worried about load can keep using export_records or page with date_begin/date_end. It's aimed at the memory-constrained client (small FaaS instances and the like), not at reducing server work or doing incremental sync.
Tests: 6 new tests using the existing mocked-API fixtures (no live server), full unit suite 125 passed. black, mypy and pylint clean.