Closing this one after https://phabricator.wikimedia.org/T270177#6740607

Sounds good, thanks @Bstorm. I verified that I have access to the CloudVPS project and I can see the VM flavours.

Could this be a caching related issue that either got resolved or is only affecting specific cases?

It looks very interesting. Overall it seems like all the metrics between engines are similar except of the total blocking time.
I am wondering how the numbers would changed if the benchmark setup is not just a main div in the DOM but also includes other elements where a map is nested.

Ok, lets use "maps-experiments" instead

Closing this task after T271952

Some updates from our beta environment:

Regarding tegola config, I captured all the work done so far here: https://github.com/johngian/openmaptiles-tegola
It involves patches:

• On tegola to make the openmaptiles SQL queries compatible with the tegola config
• On openmaptiles-tools to add a helper tool to generate the tegola config

Is there a way to figure out what are the articles used for testing? Maybe we can exclude them from wikifeeds. Other than that it doesn't look like an issue in the wikifeeds service level.

It looks like the debian package was also able to be built on stretch if backports are enabled. I uploaded a debian package built for stretch here:
https://github.com/johngian/imposm3/releases/tag/debian-0.11.0

Can we easily re-use the diff script we had when we evaluated imposm to sample if the output of the new environment is valid ?

I think we should have an item to check if there are any scheduled tasks so when we disable replication and tile-pregeneration there are no in-flight tasks pending.

I wrapped up (partially out of curiosity) the packaging of imposm3 with deb build dependencies, in case we prefer to avoid vendoring:

In T238753#6668798, @akosiaris wrote:

In T238753#6668586, @MoritzMuehlenhoff wrote:

There's also an additional option:Postgres 9.6 is also available on Buster (We already use it for cescout, which has a strict dependency on 9.6 since OONI upstream publishes there datasets that way).

From a quick glance Buster has all the deps (src:leveldb, src:geos, Golang 1.11) required to build imposm3, so we can create an imposm 3 deb on Buster, setup a Ganeti instance on Buster (maps-import1001) with Postgres 9.6 and add it to the Maps Postgres setup. Then the OSM import can simply happen from that separate instance (until we eventually also migrate maps at large).

I think there is also a dated cassandra requirement somewhere in there, so it might not be that easy. But that's almost hearsay, so @MSantos could you confirm ?

That being said, I worked a bit on packaging imposm3 yesterday. I am happy to report success: https://people.wikimedia.org/~akosiaris/

I started with stretch in mind (since maps is stretrch), but turns out that it can't be built on stretch, so it requires buster after all. That being said, if we can backport leveldb and libgeos we might be able to run it on stretch.

For those not familiar with the state of maps and stumbling upon this task, getting a feeling of exasperation is normal. Trying to adopt a not well maintained in the past infrastructure means that some tech debt needs to be paid. It's

Regarding the packaging part, here is some context about the state of imposm3 and debian: https://phabricator.wikimedia.org/T238753
Just to add to what Mateus said, our ideas was to investigate how complicated would be to build a debian package for our needs or at least just for the interim for testing purposes.

I think that the SLO definition should be a subtask of this ticket. We need to know what kind of service metrics we maintain and what are the user facing / product driven aspects that we need to keep track to define the SLO.

Although chromium 87 is released, debian buster is still using an old version (83.0.4103.116-1~deb10u3) .
The puppeteer package that we install from npm comes with pre-bundled binaries for browsers. Should we consider using that instead of the debian package?
Usually we prefer software packaged for debian but it might take a while until packages for 87 reach buster.

Added links here: https://wikitech.wikimedia.org/wiki/Push_notifications#Dashboards

Here is the report of the spike addressing the points that the team raised after reviewing it.
https://docs.google.com/document/d/1_sSANm5MrwkrUE_ZtQs_LPLtORN8vQ66oWRVwEiID9s/edit?usp=sharing

In T266373#6617586, @akosiaris wrote:

Interestingly, proton returns transfer-encoding: chunked responses, that don't have a Content-Length obviously. So, for the internal service, cl-matches-bytes makes no sense and it's not there.

Just pointing out that Transfer-Encoding is a hop-by-hop HTTP header and hence it's restbase that switches to a non-chunked response and sets as required the Content-Length header

In T266373#6616778, @akosiaris wrote:

In T266373#6613038, @Jgiannelos wrote:

@akosiaris More from debugging on this issue:

• Querying directly the RESTBase service for a PDF render of an article doesn't reproduce the issue

Looking at the pastes above, I must say I don't see a direct RESTBase service (restbase.discovery.wmnet or restbase.svc.eqiad.wmnet or restbase.svc.codfw.wmnet) call. There is a single wget against the proton service, but I don't think that gives us enough data. However, I think we indeed need to test this more.

I used the same scripts pointing to restbase.svc.eqiad.wmnet and proton.svc.eqiad.wmnet but since it didn't show any issues I didn't paste the whole run output.

Some findings regarding localization alternatives

@akosiaris More from debugging on this issue: