Importing requests is fairly slow

[cournape]

We are using requests in our projects, and it is working great. Unfortunately, for our CLI tools, using requests is an issue because it is slow to import. E.g. on my 2014 macbook:

python -c "import requests"

Takes close to 90 ms.

Is optimizing import time worth of consideration for the project ?

[Lukasa]

This is almost certainly the result of CFFI. Can you print what libraries you have installed in your environment (pip freeze)?

[cournape]

This is not linked to cffi (I don't have it installed). I did the test w/ a quasi empty virtualenv:

$ pip list
pip (8.1.2)
requests (2.10.0)
setuptools (21.1.0)
wheel (0.24.0)
$ time python -c "import requests"

real    0m0.090s
user    0m0.061s
sys 0m0.026s

[sigmavirus24]

I have a 2012 Macbook pro:, using a slightly different way of measuring the import time:

$ pip list
pip (8.0.2)
requests (2.10.0)
setuptools (19.6.2)
wheel (0.26.0)
$ python -m timeit -n 10000000 'import requests'
10000000 loops, best of 3: 0.571 usec per loop

I also upgraded pip and setuptools to match your environment:

$ pip list
pip (8.1.2)
requests (2.10.0)
setuptools (21.1.0)
wheel (0.26.0)
$ python -m timeit -n 10000000 'import requests'
10000000 loops, best of 3: 0.59 usec per loop

Further

$ time python -c 'import requests'
python -c 'import requests'  0.07s user 0.05s system 61% cpu 0.200 total

Which is still not ideal, but is probably higher than the simple import of requests due to Python's initialization.

[sigmavirus24]

That said, if you have information about specific things in our __init__.py (and elsewhere) that are causing slowness, I think we can fix them so long as they don't break backwards compatibility or our functional API.

[cournape]

timeit is not informative for imports, you have to create a new process every time because of python import caching.

The main cost is importing urllib3, so I suspect that should be addressed there.

[nicktimko]

@sigmavirus24 as @cournape mentions you can't time it like that; try using time on the command line. On my machine (ran each ~6-10 times to try to get a reliable average) for user+sys times (including sys because if there's any strange calls made into the kernel as a result of the import, that should be counted):

• time python -c "" (CPy 2.7.11) = 130 ms (time for Python to start up)
• time python -c "import requests" (2.10.0) = 240 ms (above + importing requests)
• time python -c "import urllib3" (1.15.1) = 210 ms (installed separately)

So urllib3 takes about 80 ms, then about 30 ms more for requests stuff.

[kennethreitz]

Neither of these numbers seem that high to me.

[nicktimko]

Yeah. Especially as your average request is going to take a few hundred milliseconds or so. If you reeeeally want to shave off milliseconds, could always back-translate to use the stdlib. 😜

Or you could shorten the name of your command line tool by 1-2 characters.

[kennethreitz]

Ah, I didn't realize you were working on a CLI tool. Well, it hasn't seemed to be an issue for httpie.

[cournape]

It is true that as soon as you do even one request on a remote server, the delay will not matter much. But the problem in python is that you pay the import cost whether you actually do a request or not: think doing tool --version, tool --help, etc... Imagine a tool like git or hg that only occasionally do http IO.

80 ms is pretty bad in this case: the rule of thumb for a CLI tool to have noticeable delay is ~100-150 ms. Try using e.g. git with a 200 ms delay, it is a bit annoying.

Now, I completely understand that it may not be a focus of the library, especially if it is not that easy to improve.

[kennethreitz]

I personally think you're using the wrong tool for the job — given that, on the metrics given above, Python itself is taking 120ms alone. Httpie and mercuial are well-loved tools that do their jobs very well, but they are not, nor ever will they be, curl or git.

[nicktimko]

Makes sense; you could code up some lazy loading bits. Found this though, which claims to have been excerpted from hg: https://github.com/bwesterb/py-demandimport (haven't ever used it, no guarantees).

Apparent origin: https://selenic.com/hg/file/tip/mercurial/demandimport.py

[cournape]

@kennethreitz python does not take 120 ms to start, unless you are on a seriously broken environment. It is much closer to 20 ms on decently modern hw (< 5 years), i.e. importing requests means 3x the cost of starting python.

FWIW, on my 2011 Desktop PC (Debian):

$ time python -c ""
real    0m0.020s
user    0m0.016s
sys 0m0.000s
$ time python -c "import requests"
real    0m0.107s
user    0m0.088s
sys 0m0.012s

On my 2014 macbook (OS X)

$ time python -c ""

real    0m0.022s
user    0m0.010s
sys 0m0.009s
$ time python -c "import requests"
real    0m0.096s
user    0m0.060s
sys 0m0.032s

A simple hg (for help) on my macbook takes ~ 100 ms

[nicktimko]

Oh, oops, yeah...my environment is a bit wonky because of the pyenv shim (adds about 100 ms because it's a bash script that leads to python with exec). Bypassing that, the incremental times are still the same and Python itself loads in 10 ms or so.

[kennethreitz]

haha, I thought that number looked high :)