hyper do not work on my Fedora

[michal-niklas]

I have problem with running Python hyper library that adds support for HTTP/2 protocol.

On my Fedora machine I installed it with pip and pip3 to use it with Python 2.7.8 and Python 3.4.1. Then I copied test script that connects with twitter:

from hyper import HTTP20Connection

conn = HTTP20Connection('twitter.com:443')
conn.request('GET', '/')
resp = conn.getresponse()

print(resp.read())

I run it using Python 2.7.8 and it ends with error:

[mn:/plib] 1 ‡ python hyper_test.py 
Traceback (most recent call last):
  File "hyper_test.py", line 4, in <module>
    conn.request('GET', '/')
  File "/usr/lib/python2.7/site-packages/hyper/http20/connection.py", line 149, in request
    self.endheaders(message_body=body, final=True, stream_id=stream_id)
  File "/usr/lib/python2.7/site-packages/hyper/http20/connection.py", line 310, in endheaders
    self.connect()
  File "/usr/lib/python2.7/site-packages/hyper/http20/connection.py", line 215, in connect
    self._recv_cb()
  File "/usr/lib/python2.7/site-packages/hyper/http20/connection.py", line 580, in _recv_cb
    self._consume_single_frame()
  File "/usr/lib/python2.7/site-packages/hyper/http20/connection.py", line 496, in _consume_single_frame
    frame, length = Frame.parse_frame_header(header)
  File "/usr/lib/python2.7/site-packages/hyper/http20/frame.py", line 52, in parse_frame_header
    frame = FRAMES[type](stream_id)
KeyError: 80

It ends with different error with Python 3.4.1:

[mn:/plib] 1 ‡ python3 hyper_test.py 
Traceback (most recent call last):
  File "hyper_test.py", line 4, in <module>
    conn.request('GET', '/')
  File "/usr/lib/python3.4/site-packages/hyper/http20/connection.py", line 149, in request
    self.endheaders(message_body=body, final=True, stream_id=stream_id)
  File "/usr/lib/python3.4/site-packages/hyper/http20/connection.py", line 310, in endheaders
    self.connect()
  File "/usr/lib/python3.4/site-packages/hyper/http20/connection.py", line 204, in connect
    sock = wrap_socket(sock, self.host)
  File "/usr/lib/python3.4/site-packages/hyper/http20/tls.py", line 44, in wrap_socket
    assert ssl_sock.selected_npn_protocol() in H2_NPN_PROTOCOLS
AssertionError

What can cause such problems on my environment?

PS I have also asked such question on StackOverflow: http://stackoverflow.com/questions/28990996/python-hyper-library-do-not-work-on-my-environment

[Lukasa]

Sadly, there are a number of tricky points here.

Firstly, on Python 2.7.8, this will almost always fail. This is because HTTP/2 requires that you use one of two SSL technologies, NPN or ALPN, to negotiate up to HTTP/2. Sadly, Python 2.7.8's standard library does not support NPN or ALPN. In principle PyOpenSSL could fix this problem, but my NPN pull request and my ALPN pull request have both been open for nearly 9 months with no prospect of getting merged.

For this reason, hyper does not really support Python 2.7.8 at this time, though I'm hoping to add support for it over time. If you want to use Python 2.7, I recommend using 2.7.9, which contains the SSL functions we need.

The Python 3.4 error shows that Twitter is not offering HTTP/2 on your connection. That's a little odd, though I've been seeing real trouble with Twitter recently.

Just to test, can you try using the hyper command-line tool to see if that's any better? Run hyper GET https://twitter.com/ and show me the output. =)

[Lukasa]

While we're here, can you also show me the output of python3 -c 'import ssl; print(ssl.OPENSSL_VERSON)'?

[t2y]

In my environment, I got the same error with Python 3.4.

(hyper)$ hyper GET https://twitter.com
...
  File "path/to/hyper/hyper/http20/tls.py", line 45, in wrap_socket
    assert ssl_sock.selected_npn_protocol() in H2_NPN_PROTOCOLS
AssertionError

But, other sites are available.

(hyper)$ hyper GET https://google.com
(hyper)$ hyper GET https://http2.golang.org

I guess http2 doesn't work now on twitter.com, but spdy/3.1.

[Lukasa]

@t2y Can you dump the TLS handshake with Wireshark and see what NPN protocols are being advertised?

When I connect, I see twitter advertising h2, spdy/3.1, and http/1.1.

[Lukasa]

@michal-niklas By the way, a good HTTP/2 testing tool is http2bin.org. I know it works because I manage it. =D

[t2y]

@Lukasa hmm, I just confirmed that the return value of ssl_sock.selected_npn_protocol() is http/1.1.

[dimaqq]

fyi, works for me:
python 2.7.9
hyper git
OpenSSL 1.0.2 22 Jan 2015
arch linux.

@michal-niklas please post openssl version on your system as visible to Python.

[Lukasa]

Looks like Twitter's got different logic in different places in the world. Weird!

[dimaqq]

@t2y and @michal-niklas please compare your dig twitter.com

Here's mine, from Europe:
twitter.com. 14 IN A 199.16.156.198
twitter.com. 14 IN A 199.16.156.38
twitter.com. 14 IN A 199.16.156.6
twitter.com. 14 IN A 199.16.156.70

[Lukasa]

Mine, from the UK:

twitter.com. 30 IN A 185.45.5.32
twitter.com. 30 IN A 185.45.5.43

[t2y]

That's funny, from Japan.

twitter.com. 19 IN A 199.16.158.168
twitter.com. 19 IN A 199.16.158.179

[Lukasa]

Well Twitter's DNS is super weird.

Can we all confirm that hyper GET https://http2bin.org/get returns appropriately?

[t2y]

Like this, from Japan.

(hyper)$ hyper GET https://http2bin.org/get
{'args': {},
 'headers': {'Connection': 'keep-alive', 'Host': 'http2bin.org'},
 'origin': '153.206.170.217',
 'url': 'https://http2bin.org/get'}

[Lukasa]

Looks to me like Twitter are doing some A/B testing of HTTP/2 versus other protocols. I recommend, then, that people trial HTTP/2 against other services, such as http2bin, Google, etc.