When a push or pull request payload is received, we need to determine the diff from the last sync, or from the current main branch.
This can be done with the /repos/{owner}/{repo}/compare/{basehead} API: https://docs.github.com/en/rest/commits/commits?apiVersion=2026-03-10#compare-two-commits
For push payloads...
Verify that the branch is main (if not, log & return). Use the before and after sha properties as the base/head of the comparison.
For pull_request payloads...
We will need a new table to record previous pull_request imports. This should track the branch name (indexed & unique) and the commit sha, with a uuid PK.
When a new event is received, find the last processed commit sha from the database, and use that as the base for the comparison. Otherwise, use the base.sha from the pull request payload.
After fetching the comparison (in both jobs):
Iterate through files[] and collect file paths that match any post, collection, or author contents. (note that posts and post subdirs are both counted, whereas authors/collections only count files directly within their folders)
content/{user}/* -> authorcontent/{user}/posts/{post}/** -> postcontent/{user}/collections/{collection}/* -> collectioncontent/{user}/collections/{collection}/posts/{post}/** -> post
Then, for each discovered ID, trigger and await the response of the sync-author, sync-collection, and sync-post jobs.
These jobs can be parallelized, but the logic must consider dependencies - if both a collection and a post within the collection need to be synced, the collection must finish syncing before the post. Likewise for authors -> collections, or authors -> posts.
When a push or pull request payload is received, we need to determine the diff from the last sync, or from the current main branch.
This can be done with the
/repos/{owner}/{repo}/compare/{basehead}API: https://docs.github.com/en/rest/commits/commits?apiVersion=2026-03-10#compare-two-commitsFor
pushpayloads...Verify that the branch is
main(if not, log & return). Use thebeforeandaftersha properties as the base/head of the comparison.For
pull_requestpayloads...We will need a new table to record previous pull_request imports. This should track the branch name (indexed & unique) and the commit sha, with a uuid PK.
When a new event is received, find the last processed commit sha from the database, and use that as the base for the comparison. Otherwise, use the
base.shafrom the pull request payload.After fetching the comparison (in both jobs):
Iterate through
files[]and collect file paths that match any post, collection, or author contents. (note that posts and post subdirs are both counted, whereas authors/collections only count files directly within their folders)content/{user}/*-> authorcontent/{user}/posts/{post}/**-> postcontent/{user}/collections/{collection}/*-> collectioncontent/{user}/collections/{collection}/posts/{post}/**-> postThen, for each discovered ID, trigger and await the response of the sync-author, sync-collection, and sync-post jobs.
These jobs can be parallelized, but the logic must consider dependencies - if both a collection and a post within the collection need to be synced, the collection must finish syncing before the post. Likewise for authors -> collections, or authors -> posts.