Implement server push

[alekstorm]

Depends on #39

See the changes to docs/source/advanced.rst for an overview.

Apologies for not getting approval for this ahead of time; you did clearly state in the documentation that server push wasn't a planned feature. However, I find it one of the most compelling parts of HTTP/2.0, and I think this a workable API for a single-threaded library. What do you think?

P.S. I also snuck in a fix for a bug that caused hyper to crash when trying to send a WINDOW_UPDATE frame to a server that had gone away with the stream half-closed on the remote side.

[Lukasa]

Before we add this functionality we must provide an API for disabling SERVER_PUSH, if only for the sake of requests which does not expect them.

[Lukasa]

This is another seriously impressive chunk of work @alekstorm, thankyou! As with the h2-10 work, I'll need a little while to review the code properly. We'll also have to bikeshed the API for a while I'm afraid, because it's something I'd like to get right the first time. =)

[Lukasa]

You're no longer removing :status from the headers. We should do that because people don't expect to see it, and it's effectively a HTTP/2 implementation detail.

[alekstorm]

Actually, extract_from_key_value_set (above) does remove the specified keys from the key-value set - see the docstring in util.py. It's easy to miss, I know :). Any ideas for a better name?

[Lukasa]

The standard Python term for something that removes from a collection and returns would be pop, so pop_from_key_value_set is the suggested term. =) Obviously this'll need to be updated in #39.

[alekstorm]

Hokay, rebased on development (thanks for reviewing #39 so quickly!), reverted the changes to the requests adapter, and addressed most other comments.

Still TODO:

• Provide an API for disabling server push (is this still a priority if we're no longer hanging pushed resources off the Response object in the requests adapter?)
• Bikeshed the hyper API some more (I have some ideas here; hopefully they'll gel with yours)

[alekstorm]

A closer examination of the spec made me realize that PUSH_PROMISE frames can also be sent after the response header block for the original request; this prompted a reworking of the API to support two use cases: handling pushed resources that were promised before certain checkpoints (specifically, calls to HTTP20Connection.getresponse() and HTTP20Response.read()), and handling all pushed resources as they come in, regardless of the state of the original response.

These considerations are reflected in the new HTTP20Connection.getpushes() method; see docs/source/advanced.rst for an overview. Looking forward to your feedback.

[Lukasa]

I think we should reword this section of the docs slightly. What matters about getpushes() is that it returns an iterable of promises. The fact that the iterable is sometimes a list and sometimes a generator is mostly irrelevant. In fact, I think we should go further and always return a generator, even when we know we've got a list. That way we have a consistent API that prevents people doing silly things like calling len() on a generator.

[Lukasa]

Heh, reading through the code reveals that it is always a generator. In that case, just adjust the wording so that it's clearer that the return value is always a generator. Rather than the call itself blocking, it's whether the generator will block on future possible pushes. =)

[Lukasa]

Alright, more review comments inline. =)

As for the two TODOs:

1. Yeah, I still want a way to turn off Server Push. If you're not expecting them (most users won't be), we don't want to waste the bandwidth accepting them. One of my goals is to have requests users who use HTTP/2 through hyper get better performance in all (or almost all) cases than they would in HTTP/1.1. That means not letting endpoints send us extra data we're not going to use.

   This API doesn't have to be clever though: it's probably enough just to have it be a flag on the HTTP20Connection object.

2. I like the generator API for pushed streams. My only big worry at the moment is that pushed streams have a different API to the responses we get from streams we initiated (in that they have their own object with their own methods). I want to think about whether we can do better than that.

   In the meantime, if you have ideas for improving the API you should let me know!

[Lukasa]

Also, looks like the tests need more code coverage. =) This is small, so it should be possible to achieve without introducing #pragma no cover

[alekstorm]

Okey doke, thanks for the continued excellence in reviewing. I've addressed all comments (let me know if you're happy with the new wording of the getpushes() documentation), so the last TODO item is code coverage (I usually save reaching 100% for last, since the API may change under review).

[Lukasa]

This is fine for now, but it'd be an interesting enhancement to have it be a property that causes the appropriate SETTINGS frame to be emitted.

[alekstorm]

Agreed.

[Lukasa]

Alright, I pronounce this good-to-go. Let's get a green build and I'll merge the heck out of this. =D

[alekstorm]

Bam, done.

[Lukasa]

This is beautiful, I'm delighted to have this. Once again, you're just doing incredible work @alekstorm, thankyou so much. 🍰 👍

[Lukasa]

Just so you know @alekstorm, I'll be at PyCon from tomorrow through to Tuesday, which will reduce my responsiveness to pull requests and issues. Wanted to make sure you were aware. =)

[alekstorm]

Sure, thanks for letting me know. Have fun!

[alekstorm]

And thanks for the words of encouragement! They really do mean a lot :)

[Lukasa]

They're 100% genuine. =)

Based on your contributions over the last week or so, you've got everything you need to be a really valuable open source developer. You definitely have the technical skillset, you've demonstrated a willingness to work with maintainers to maintain style and project standards, you take code review well, and you're prepared to put in the work. You'd be a welcome contributor to any open source project you felt like.

You've done awesome work on hyper, and you should be proud of it. =)