Skip to content

Don't buffer files in memory at send time#260

Merged
PaulSonOfLars merged 5 commits intov2from
paul/filestreams
Mar 21, 2026
Merged

Don't buffer files in memory at send time#260
PaulSonOfLars merged 5 commits intov2from
paul/filestreams

Conversation

@PaulSonOfLars
Copy link
Copy Markdown
Owner

What

Addresses #259

Previously, we would pipe files while creating the multipart reader.
This changed in #242, where i opted to avoid double-marshalling fields - but introduced a regression where files were double-buffered in memory in an attempt to maintain the fix from #231.

This PR introduces regression tests to cover those edge cases, and then updates the requests code to correctly use pipes and allow retries without double-buffering files.

Impact

  • Are your changes backwards compatible? Y
  • Have you included documentation, or updated existing documentation? Y
  • Do errors and log messages provide enough context? Y

@danielwaichcesl
Copy link
Copy Markdown

Hi @PaulSonOfLars,

Tested PR #260 with a 31 MB file upload - works perfectly! 🎉

Results
Memory usage:

Before: ~66 MB peak (buffered entire file)
After PR #260: ~8.8 MB peak
87% memory reduction ✅
Test details:

File: 31 MB
Upload: successful in ~17s
No memory leaks
GC reclaims all memory correctly
Impact
This solves our issue with large file uploads in containerized environments. The streaming implementation works great - files are no longer buffered in memory.

Thank you for the quick fix! 👏

@PaulSonOfLars PaulSonOfLars merged commit c3edd4d into v2 Mar 21, 2026
3 checks passed
@PaulSonOfLars PaulSonOfLars deleted the paul/filestreams branch March 21, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants