Pulling models from behind firewall not possible even though pulling images works

[holger-stenzhorn]

I am running Docker Engine on a Rocky Linux VM behind a firewall and therefore I have provided a proxy configuration in /etc/systemd/system/docker.service.d/http-proxy.conf:

[Service]
Environment="HTTP_PROXY=http://some.proxy.server:3128"
Environment="HTTPS_PROXY=http://some.proxy.server:3128"
Environment="NO_PROXY=localhost,127.0.0.1"

With this configuration pulling of images works BUT pulling of models does not:

❯ docker model run ai/smollm2
353d9a3ec7ee: Pull complete Digest: sha256:00fba948428db7f9a750a7a74b3af55ab99801d0a6f1a1d7b68169a948661342
Status: Downloaded newer image for docker/model-runner:latest

Successfully pulled docker/model-runner:latest
Starting model runner container docker-model-runner...
Unable to find model 'ai/smollm2' locally. Pulling from the server.
Failed to pull model: pulling ai/smollm2 failed with status 500 Internal Server Error: error while pulling model: reading model from registry: failed to pull model "ai/smollm2": UNKNOWN - Get "https://index.docker.io/v2/": net/http: TLS handshake timeout

So seemingly the proxy configuration is not passed to the container docker-model-runner but there is also no possibility to manually specify it.

[xenoscopic]

Thanks for filing the report!

@thaJeztah do you know if there's an easy way for us to pass the Docker Engine HTTP proxy setting into our model runner container in Docker CE? I guess we could bind-mount / parse http-proxy.conf, but I don't know if the permissions / ownership on that file would allow it. Our other option would probably be adding a --proxy flag to docker model install-runner, but it would be nice from an integration standpoint if it could be inherited from dockerd's config.

[ericcurtin]

@holger-stenzhorn just curious we are putting in effort to make contributing to docker model-runner more contributor friendly, would you be interested in working on this?

[holger-stenzhorn]

@ericcurtin Yes, that sounds quite interesting for me.

[ericcurtin]

@ericcurtin Yes, that sounds quite interesting for me.

Might be worth going through docker code also, ideally we be as most similar to docker as possible, (just use their config stuff)

[ericcurtin]

@holger-stenzhorn all the code required is consolidated into this repo now, please take a look!

[holger-stenzhorn]

@ericcurtin As I was sick last week, I did not have the chance to look into this properly, and since I’m not a Go expert and a bit short on time, I’m worried I might break something accidentally — I can certainly help with testing, but probably not develop a fix on my own.

[ericcurtin]

@xenoscopic in the DMR case is it the model-runner daemon that pulls models rather than the docker daemon? If yes, it makes the problem a little clearer...

[xenoscopic]

Hey @ericcurtin, yeah, that's the case - the model distribution code is doing the pulling, and when running in a container (as with CE) it won't inherit any proxy settings. I'm not sure how to handle it - we'd need to pipe those in somehow, perhaps during the docker model install-runner stage?

[ericcurtin]

@xenoscopic I like the UX of just inheriting, it does seem a little messy to implement, but I think we can handle that under the hood for improved UX

[ericcurtin]

@holger-stenzhorn I think this fixes it, if you could build and test it, it would be appreciated, please reopen otherwise:

#263