perf: parallel range-GET S3 downloads for large objects#225
Open
joshfriend wants to merge 1 commit intomainfrom
Open
perf: parallel range-GET S3 downloads for large objects#225joshfriend wants to merge 1 commit intomainfrom
joshfriend wants to merge 1 commit intomainfrom
Conversation
![chatgpt-codex-connector[bot]](https://avatars.gh.lumenfield.work/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: afbf08ad55
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
joshfriend
force-pushed
the
jfriend/parallel-s3-range-get
branch
3 times, most recently
from
7ba1a28 to
3294383
Compare
mikejuyoon
approved these changes
Mar 24, 2026
stuartwdouglas
approved these changes
Mar 24, 2026
alecthomas
reviewed
Mar 25, 2026
alecthomas
reviewed
Mar 25, 2026
joshfriend
force-pushed
the
jfriend/parallel-s3-range-get
branch
3 times, most recently
from
0bd3818 to
26dd9c5
Compare
Replace the single GetObject stream in S3.Open with parallel range-GET requests for objects larger than 32 MiB. Workers download chunks concurrently via errgroup and reassemble them in order via io.Pipe. All chunk requests are pinned to the ETag from the initial stat to prevent corruption if the key is overwritten mid-read. An errgroup with derived context ensures all workers are cancelled promptly on any error or early consumer close.
joshfriend
force-pushed
the
jfriend/parallel-s3-range-get
branch
from
26dd9c5 to
8662246
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace the single GetObject stream in S3.Open with parallel range-GET requests for objects larger than 32 MiB. 8 workers download chunks concurrently and reassemble them in order via io.Pipe, multiplying S3 throughput for cold snapshot downloads (observed ~100 MB/s single-stream → 400+ MB/s with parallel connections on staging hardware).