Picking up the task

@Jhancock.wm I am afraid the server is dead again :(

Dear @Jhancock.wm or @Papaul, server can be shutdown and checked at your convenience, this part of the stack has failovers in place. Thank you!

In T344110#9143064, @Jhancock.wm wrote:

@jijiki is there a day this week that I can update the firmware on this server?

Additionally, while the key for both certificates are of the same size, during negotiation, envoy chooses the TLS_AES_256_GCM_SHA384 cipher suite, while previously was choosing TLS_AES_128_GCM_SHA256. We will investigate this further when we pick it up again.

@ArielGlenn please mark it as resolved if everything is as expected. (thanks @Urbanecm!)

observability we are still looking into how things could progress here on the Maps side of things. Please let me know if it is alright to stall this roll out for now.

In T344110#9117302, @Jhancock.wm wrote:

Turns out there is one more thing I need to do to. I missed a firmware update. Is it safe for me to reboot at this time?

@Jdforrester-WMF it will be extremely useful for the future to improve wikifunction's observability, for instance, there were logs emitted to logstash which logged that specific problem, and would be visible with a better curated dashboard.

Regarding the value "32531" opcache.max_accelerated_files, is that only numbers from a specific set of prime numbers are valid there. Using a different number, will result to the actual value used being the first number in that set, that is greater than or equal to what was configured.

In T344324#9112390, @Joe wrote:

In T344324#9112384, @JMeybohm wrote:

Could we just circumvent the problem by enabling the thanos-swift listener in tegolas service-proxy and configure it to use the plaintext localhost connection instead of connecting directly to thanos-swift?

That's always going to be our last resort, although I fear the s3 sdk will complain about various things and that will need tweaking. We need anyways to understand what's wrong as we might encounter similar issues somewhere else.

So: yes we could, but no we shouldn't.

Testing procedure:

From the 18th Aug rollout, tcpdumps can be found in my home directory on kubernetes2010 and thanos-fe2004. The pod IP on kubernetes2010 is 10.194.150.55 and thano's ip is 10.2.1.54

@Jhancock.wm thank you very much for the update!

In T341843#9073121, @ayounsi wrote:

Not sure i f it's a Netbox or Redis issue :(

@Tchanders we could potentially provide those resources, as long as they are for a limited amount of time, eg 2hrs.

In T339331#8992755, @Tchanders wrote:

Given that we have the ability to provide as many resources as the pod needs, in which case, we can make it possible to load the whole dump in memory, if that would help with our current challenges.

@jijiki Does this only apply to the initial import, or could we do something similar for the daily updates - i.e. have the two dumps uncompressed in memory? I'm wondering about faster ways to compare the two dumps without reading from the database. We have a diffing script that works quickly, but uses a lot of storage space, and I'm wondering if it's at all worth pursuing.

@STran answering your questions from the previous thread

When the chart was created, it was missing enabling a template, which was not evident until we were able to successfully start ipoid containers in the staging environment.

Done.

@Krinkle, since we are approaching the end of Q4, we will discuss internally and schedule this work for Q1.

In T325635#8937624, @STran wrote:

This ticket was created at a time when the script could not be finished on local instances at all and which has been somewhat resolved (for sufficiently powered computers with bare metal instances) "good enough" for development. It's a little less concerned with being production-ready, which the other ticket very much is and as such the discussion should be centralized there and this ticket closed out. For the questions @jijiki asked, I've moved those to a new ticket specifically focused on if an initial import needs additional development/management.

Dear Anti-Harassment for the time being, the chart is ready! What now?

In T305724#8931736, @sbassett wrote:

Uncompressed, the entire daily feed has recently been around 8 Gb.

Quick Note: Currently iPoid's database, m5-master.eqiad.wmnet is anchored to eqiad for writes, while in can read from m5-master.codfw.wmnet, that means that, the service will be active/active from the user perspective, but its cronjob, which is the only path where there are writes in the database, will run always from eqiad.

Since serviceops is done with T336163, we must consider how we are going to do the initial import in production. This is our suggestion:

In T305724#8929267, @sbassett wrote:

Unfortunately, the data missing from the export diff at this time is likely to be the data most relevant and interesting to the Foundation and Community.

In T305724#8928288, @kostajh wrote:

@jijiki to clarify, what do you mean by diff, do you mean the https://exports.spur.us/v1/feeds/anonymous?diff=true (docs)? AIUI, that doesn't have the necessary data that we need.

Or by diff do you mean the process I sketched in T305724#8917809?

To my understanding (please correct me if I am wrong) we have some tolerance regarding having stale data. If that statement is true, and give that the offered diff by the provider is missing some groups of data, my suggestion is the following:

Helm chart is ready. We will be exposing the following environmental variables, some of which will hold sensitive information: