Response.content is wasteful in time and memory for large inputs

[tzickel]

https://github.com/requests/requests/blob/883caaf145fbe93bd0d208a6b864de9146087312/requests/models.py#L827

iter chunk is 10K, let's say we have binary content that is about 10MB, the .join first converts the iter_content to a list of 1000 items of 10K each (if it's not chunked), then it creates a new large 10MB string, copies all of them to it, then returns it.

This both creates pressure on the allocator and at the peak takes * 2 the memory of the input.

Without resorting to C, and if we know the input size (i.e. not chunked), we can create a bytearray and fill it with one iteration at a time. On my computer it is about *2-3 faster, takes half the memory and causes 1/2 of page reclaims.

The problem is that is of course that it returns a bytearray and not a bytes type, maybe this should be a helper method for large inputs if people care about this stuff ?

arr = bytearray(totalsize)
i = 0
for item in self.iter_content(CONTENT_CHUNK_SIZE):
  l = len(item)
  arr[i:i + l] = item
  i += l
return arr

[nateprewitt]

Thanks for taking the time start this discussion, @tzickel! My initial thoughts here are mostly about usability and expectations around Requests' API. We deal predominantly in byte strings and strings when handling data. Implementing the suggested change would break that, and converting the bytesarray back to bytes nullifies the memory savings.

The general idea here is correct though, this is what we recommended for large body content. Using iter_content on streamed responses is much faster/more efficient. This approach also allows the user to choose whatever data structure or allocation best fits their needs. We could look at adding a helper function to the Requests toolbelt, but since this is a 5-line function that can be optimized for different use cases, it may be easier to leave this to users to implement.

[tzickel]

My main issue is that I'm passing a large input (between a few MB to a few GB) to request toolbelt MultipartDecoder (because if it's not multipart I don't use .content, but stream it anyhow) and I'm looking for ways to optimize that workflow (considering I don't need the real input and that I need all the data in-memory). MultipartDecoder uses the request.content

https://github.com/requests/toolbelt/blob/6d74296f0f267d9e06ba7828585dac61276d4e2b/requests_toolbelt/multipart/decoder.py#L156

So I'm looking for ways to optimize it (without maintaining code outside).

I guess I should be using something like this:
requests/toolbelt#222
And when I know the size of a part, I can directly stream into a preallocated bytearray (almost)