We actually had support for gitlab but it had a bug. That's fixed now.

That's fixed.

We probably should actually fix the gitlab part before running it again

In T363573#9748629, @Daimona wrote:

Lol, my first conflict in gitlab.

Fixed in https://gitlab.wikimedia.org/ladsgroup/Phabricator-maintenance-bot/-/commit/982cfc2f33f8190728363c5ecdc55fbbab1ea40e

In T363581#9748213, @daniel wrote:

How will this be kept in sync with tables.json in MW core and extensions?

Actually we made a lot of progress in this regard. Now people can disable 2FA via a special page rather than running a maint script in production. As results, the userright to disable 2FA can now be easily[1] expanded to potentially stewards and checkusers to reduce the work from T&S and slowly allow us to roll it out to larger groups (maybe extended confirmed) in some wikis. Persian Wikipedia is actually a good candidate wiki given the sensitivity of editing in Iran or related to Iran.

In T363581#9748076, @CDanis wrote:

I like the idea! A few questions

• Do you intend to have some monitoring checks or a dashboard to compare the catalog to actual live prod state?

I reviewed this, no objection from our side. I'd guess it was added for readability which is not a good reason. Will appreciate making these tables slightly smaller.

In T355730#9740772, @Eevans wrote:

In T355730#9740038, @Ladsgroup wrote:

I was asked to provide feedback from mariadb perspective (and how consistent we want to be across different technologies but in the same team).

We don't usually hand over dev accounts to the staging environment. Many development/staging work gets done on mariadb instance outside of production, most notably beta cluster (which has its own issues but I assume setting up a dedicated project for cassandra in cloud VPS and giving access to that wouldn't be too hard). Given that it's outside of prod, the impact of mistakes or compromise is quite limited, It also discourages "testing in production" situation. I know the staging cluster has different data but still it's in prod infra with all the complexities/downsides that it brings with itself.

Ok, so to be clear: This ticket was the result of needs arising from the Generated Data (née AQS) cluster, and not the RESTBase cluster. @Jgiannelos —who may yet weigh in with their requirements— is awaiting the outcome of this issue in relation to the latter. The cassandra-dev cluster is hosting use-cases for both, and I hadn't planned on differentiating when it came to providing developer access.

I was asked to provide feedback from mariadb perspective (and how consistent we want to be across different technologies but in the same team).

Removed. The bot won't update a closed ticket.

This will be messy. It's a private wiki.

Ready for data engineering people to take over

Ready for data engineering people to take over

Ready for data engineering people to take over

Ready for data engineering people to take over

Ready for data engineering people to take over

Ready for data engineering people to take over

Ready for data engineering people to take over

In T361041#9738119, @gerritbot wrote:

Change #1022447 merged by Ladsgroup:

[operations/puppet@production] Add sysop_plwiki to private wikis

https://gerrit.wikimedia.org/r/1022447

When the bot got created gitlab in wikimedia didn't exist. We need to add support to it. It shouldn't be hard.

It's not a bug of the optimizer.

I do it ASAP.

Running optimize table.

I know there is a backward compatibility layer added to wikireplicas but I highly recommend announcing the change and removing that layer a couple months later. The main use of views is to hide private information and combining b/c with that adds a lot of complexity and can and has caused private data to be leaked before (e.g. an extra schema change has been but didn't take the b/c view into account). Specially given that we hide private information via views in block data already.

oh thanks for looking into it.

It recovered on its own, might be a network issue. I will take a look.

nvm, it didn't recover

I was summoned: It's fine for now but some should be put in place after roll out.

The rebuild fixed the issue. The replag is zero.

Anyway, the lag is decreasing, It'll take two hours or so to fully recover (unless there is another replication crash). I will close this once the lag goes below one second.

In T363077#9732870, @Wbm1058 wrote:

Sorry to butt in, but, my understanding from the above is that the corrupted table coincidentally happens to be the same table which was just recently "normalized", correct?

In T363077#9732470, @Ladsgroup wrote:

I started an optimize table on pagelinks, It's unlikely it would fix the issue but worth a try.

Thanks I will try that!

FWIW, this is a data corruption issue. Last time it happened on sanitarium hosts the whole system went down for a week. I'm trying to figure out if I can avoid re-cloning the whole host (which would take a lot of time, potentially weeks) and only reclone the corrupted table but there is no easy way to do this AFAICS. I keep you posted.

I started an optimize table on pagelinks, It's unlikely it would fix the issue but worth a try.

In T352010#9732305, @Liz wrote:

Oh, no, this task isn't done yet? System lag again? I hope this one is shorter than the last one.

In T120242#9726131, @Ottomata wrote:

There is a lil discussion about this topic in T249745: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable". Moving that discussion to here.

@Ladsgroup wrote:

those data could be regenerated from canonical data on the wiki

@Ottomata wrote:

This is very expensive, and requires complicated logic and maintenance to do.

@Ladsgroup wrote:

if you have a way to find mismatch and redo for a small portion of changes (e.g. even 1% of changes), it should be fine.

Agree, but how?

How should one find out they are missing records, and which records to fetch?

To people who are subscribed: This automatic depools and repool comments are going to continue for weeks, feel free to unsubscribe

FWIW, it has already recovered and indeed it was caused by the schema change.

The idea is to send an email to -owner@ mail and give it a grace period, if they all are inactive, then ask "central admins" to do something about it. I did the exact same thing with the gendergap mailing list.

yeah, I'm honestly inclined to move that to behind misermode (=disabling it in production). Going through every link in a wiki doesn't make much sense, doesn't provide any benefit.

In T249745#9725953, @Ottomata wrote:

search index not getting updated in 0.001% of edits

Search is probably fine.

Nothing on your side, the wiki creation script was broken for a while. I'm planning to create wikis soon.

In T352010#9725011, @Liz wrote:

Looks like everything is back to normal. Thanks for all of the work you do for the project.

No comment for over a week after some outreach work as well. CycloneDX it is.

Avoiding cookie licking.

How can I get into a pod in job runners namespace(?) via shell.php? I want to try some stuff

In T249745#9723704, @Ottomata wrote:

For replicating state changes (T120242) [...]

Why though? Why is 99.9999% (or 99.999999% or 99.99%) not enough?

There is a "Why do we need this?" section in T120242's description. Let's keep this discussion there?

That task wants MW state propagation using events to be equally consistent with MariaDB. That way folks can trust the state they are getting to build read-model products outside of MW, where it is difficult to do so (search indexes, AI backed patrolling tools, WDQS, etc. etc.)

In T352010#9723975, @Stevietheman wrote:

Just wanted to note we're past the previous estimate and the lag is past 25 hours and seemingly getting longer. This is affecting a Quarry query I've been needing to run for literal days and database reports I have aren't keeping current. Any new estimates?

In T360029#9723602, @CDanis wrote:

I don't see why you couldn't do a simple subprocess.run to do a commit, probably checking first that dbctl diff returns empty.

In T362786#9723051, @Marostegui wrote:

We do need to list the spares somehow somewhere because otherwise we will forget where they are (or even their hoatnames)

First we need to add them to dbctl, I'm not sure how that can be done. We currently have this in mw config:

                'parsercache-dbs' => [
                        'pc1' => '10.64.0.57',   # pc1011, A1 8.8TB 512GB # pc1
                        'pc2' => '10.64.16.65',  # pc1012, B1 8.8TB 512GB # pc2
                        'pc3' => '10.64.32.163', # pc1013, C5 8.8TB 512GB # pc3
                        'pc4' => '10.64.32.53',  # pc1016, C6 8.6TB 512GB # pc4
                        # spare: '10.64.48.89',  # pc1014, D6 8.8TB 512GB
                        # spare: '10.64.0.17',   # pc1015, A6 8.8TB 512GB
                        # Use spare(s) to replace any of the above if needed
                ],
and
                'parsercache-dbs' => [ 
                        'pc1' => '10.192.0.72',   # pc2011, A5 8.8TB 512GB # pc1 
                        'pc2' => '10.192.16.55',  # pc2012, B5 8.8TB 512GB # pc2
                        'pc3' => '10.192.32.57',  # pc2013, C1 8.8TB 512GB # pc3
                        'pc4' => '10.192.48.92',  # pc2016, D3 8.8TB 512GB # pc4
                        # spare: '10.192.48.52',  # pc2014, D1 8.8TB 512GB
                        # spare: '10.192.32.132', # pc2015, C5 8.8TB 512GB
                        # Use spare(s) to replace any of the above if needed
                ],

In T360029#9658042, @CDanis wrote:

Just to make sure I understand, the request here is an easy-to-automate way of dbctl to change the instance IP address?

It probably wouldn't be too hard to add support for editing IP or just any single field in general, and then we could do that from a cookbook + a dbctl commit. Would that meet needs here?

In T352010#9721159, @Liz wrote:

Do you have any idea when this task will be done, the system will catch up to the 10 hour time lag and everything goes back to normal?