Change 710218 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] docker_registry_ha: Increase nginx tmpfs size from 1gb to 2gb

https://gerrit.wikimedia.org/r/710218

Change 710218 merged by JMeybohm:

[operations/puppet@production] docker_registry_ha: Increase nginx tmpfs size from 1gb to 2gb

https://gerrit.wikimedia.org/r/710218

We're running into this again today. I tried the same workaround as before (removing a recent but inactive mediawiki checkout) but it didn't help this time.

Noting for the record that pushes magically started working again around 12:28 UTC (2021/01/19).

Another case of failing to push a large image: T342084. Is it possible to configure NGINX to use a different (large, on-disk) storage area for certain URLs?

In T288198#9035228, @dancy wrote:

Another case of failing to push a large image: T342084. Is it possible to configure NGINX to use a different (large, on-disk) storage area for certain URLs?

Use case from ML - we are porting the recommendation-api Python service from wmf-cloud to k8s. It uses a dict file containing embeddings (basically a serialized blob) that weights around 3GB (so one big layer when we add it) and that it is loaded when the application boostraps. We are working with Research to figure out how to handle cases like this one, since "embeddings" could get multiple shapes and sizes, and they'd need to have flexibility when they experiment (like, using a serialized blob as starter and then think about some specialized datastore). This is not an "experimentation" use case of course, but the recommendation-api's setting may vary in the future. We could think about other options if raising the tmpfs mount is not an option (like fetching from swift or similar).

In T288198#9035228, @dancy wrote:

Another case of failing to push a large image: T342084. Is it possible to configure NGINX to use a different (large, on-disk) storage area for certain URLs?

It certainly is possible to definite different caching areas directives for various nginx configuration stanzas but what exactly do you have in mind?

In T288198#9036874, @elukey wrote:

In T288198#9035228, @dancy wrote:

Another case of failing to push a large image: T342084. Is it possible to configure NGINX to use a different (large, on-disk) storage area for certain URLs?

Use case from ML - we are porting the recommendation-api Python service from wmf-cloud to k8s. It uses a dict file containing embeddings (basically a serialized blob) that weights around 3GB (so one big layer when we add it) and that it is loaded when the application boostraps. We are working with Research to figure out how to handle cases like this one, since "embeddings" could get multiple shapes and sizes, and they'd need to have flexibility when they experiment (like, using a serialized blob as starter and then think about some specialized datastore). This is not an "experimentation" use case of course, but the recommendation-api's setting may vary in the future. We could think about other options if raising the tmpfs mount is not an option (like fetching from swift or similar).

So, we 've met this embeddings question in the past, in the form of the word2vec package for ORES. See T188446 and T187217. I was against packaging that artifact in a Debian package or put in the git repo as is and I still think that was a good decision. git-lfs however, despite promising, didn't pan out to be that great, for a variety of reasons.

Despite talking about container images here, same principles apply. Putting, what is clearly data (and a lot of it) in containers is going to cause operational headaches (probably for multiple teams) down the line. If it can be done in any better way like fetching it from Swift or splicing it, storing it in some datastore and querying it over an API is definitely a way more sustainable path forward.

@akosiaris Hi! Getting back to the issue, this time in a different form T359067. We have followed your suggestion for the embeddings (now they live in swift), but Pytorch is now giving us some headaches so your opinion would be really appreciated :)