Change deployed, apertium-fra-cat, apertium-separable is updated and apertium-apy has been rolling restarted

In T197246#4415306, @akosiaris wrote:

In T197246#4415080, @jcrespo wrote:

Do we need to do a dns update now? I was trying to fix labsdb1006 puppet run, but I found @akosiaris working on it at the moment and didn't want to modify anything without your permission. Is the OSM import running?

Yes, the OSM import is currently running. Coastlines and land polygons have been imported already

grafana-admin.wikimedia.org now redirects to grafana.wikimedia.org (preserving the url structure) in order to migrate the last few users to it.

In T198939#4408846, @Volans wrote:

In T198939#4408774, @akosiaris wrote:

There is a query tab but I get

What you were looking for has been disabled by the administrator.

We decided to disable access to the query tab because it allows to query everything, including catalogs, hence showing also secrets :(
See https://github.com/voxpupuli/puppetboard/blob/master/screenshots/query.png

The inventory page is customizable from the configuration with different facts, so basically is a static query:
https://puppetboard.wikimedia.org/inventory

What we could do is to try to make a patch to send upstream to allow for a more granular configuration for query, allowing to specify which kind of objects can be queried.

In T197246#4415080, @jcrespo wrote:

Do we need to do a dns update now? I was trying to fix labsdb1006 puppet run, but I found @akosiaris working on it at the moment and didn't want to modify anything without your permission. Is the OSM import running?

FWIW, servermon allows searching for specific facts (for specific hosts as well) under the fact query menu allowing to relatively easily slice information present in puppetdb (actually in mysql activerecord db, but this is unimportant), which is not something I see possible in puppetboard. There is a query tab but I get

Key updated, should make it to the cluster in the next 30 mins. I am resolving this, feel free to reopen/reach out if there's a problem.

I am setting this to Stalled and a low priority, pending @Braveheart 's response.

I am adding the cloud services team for the 2fa reset and removing the SRE-access-request tag per @Krenair 's comment above.

FWIW, +1 from me.

In T197246#4325654, @Bstorm wrote:

Bump up the switch move or the stretch upgrade? The switch move is scheduled for the 10th and 11th to avoid conflicts with holiday plans and so forth as well as to coincide with two other database clusters moving on the same days.

This has been completed successfully a few weeks ago.

In T197246#4325604, @Bstorm wrote:

A timeline for upgrading to stretch or this move event?

In T197246#4293564, @jcrespo wrote:

I know this is out of scope, but I wonder if we should not try to do an upgrade to stretch at the same time. I know this is out of scope, but I hate doing postgres maintenance, so I would prefer to do it only once. Because this is out of scope, I only ask you to consider it, not propsing it to do it striongly (I don't even know which version comes with stretch).

In T170150#4320025, @Quiddity wrote:

Re: announcement email of completion - https://lists.wikimedia.org/pipermail/wikitech-l/2018-June/090251.html
Are all the grafana-admin links directly replaceable with grafana? I.e. Can we just do a simple search & replace through the search results?

Things are definitely going way better now. I only see 1 alert in the last 24 hours.

In T170150#4318810, @GoranSMilovanovic wrote:

Hi,

• I can see the change on https://grafana.wikimedia.org/login
• However, I cannot login w. my LDAP username/password (LDAP username would be: GoranSMilovanovic): Error while trying to authenticate user.

In T186748#4301132, @pmiazga wrote:

Lowering priority to depict we currently have upgraded quite a bit the CPU count but the task is not yet resolved.

Agreed. While overall the proposed solution is probably the best, I went ahead with option A (increase the vCPU count to 10) alone for now in order to facilitate moving forward with this without blocking it on adding more machines to the cluster. Overall this gives us a total of 20 worker count per DC which is pretty close to the proposed one and I am hopeful this will resolve the issues experienced. I 'll however the task open in order to later on implement the proposed approach

In T186748#4282998, @pmiazga wrote:

@akosiaris - I did couple checks, looks like there are couple errors to investigate and fix (example: trying to 'length' of undefined) and I'll fix those. The majority of those errors come from operations-software-service-checker, the checker has a timelimit=5s and the service from time to time (to be honest it looks like most of the time) doesn't respond in 5 seconds. I'll try to use some easy/lightweight articles so it creates PDF much faster.

I can probably indeed help. I presently have no idea what's up with labs1006, but labsdb1007 is the alias osmdb.eqiad.wmnet which is directly used by the maps labs project. I 've added @Kolossos and @dschwen as 2 of the people I know use this infrastructure. As far as I know, a lot of map tiles are pregerenated and at least some functionality will not be directly impacted by a downtime of the service. I am willing to be there will be functionality that will be impacted however, but @Kolossos and @dschwen can probably shed more light into this.

In T170150#4280054, @Tgr wrote:

Any thoughts on whether this might make something like T189531: All Wikimedia developer services should use single sign-on easier or harder in the future?

In T183177#4278435, @fgiunchedi wrote:

I researched the "panic on uncorrectable errors" a bit and turns out not edac but the machine check framework already takes care of panicking (or SIGBUS'ing the process) in case uncorrectable errors are reported.

Details below, it looks like for UE the kernel already does the right thing.

I 've stalled adding LVS configuration for proton due to an instability we've been noticing. This instability is very obvious if one looks into the alert history of every one of the 4 services (there are 4 hosts powering the service right now)

Yeah, found something new, I 've reblocked some stuff, I 'll update P7249. Things do look normal again, this might just be a whack-a-mole game though

P7249 for the list of IPs

I 've banned a specific IP (I 'll share it in a private paste later on), restarted apache and everything seems to be ok now

All of my tests went fine. Scheduling this for Wednesday June 27th. I 'll send an email to wikitech-l as well