We have a patch for it ready even!

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1178899

My plan was to merge this this week. Any objections?

Nice, I missed that! :)

That should be fine, with the email to cloud-announce you already planned.

In T402787#11114471, @fnegri wrote:

That should be fine, with the email to cloud-announce you already planned.

I've sent that already last week, didn't I? With corresponding tech news entry and so on.

Change #1178899 had a related patch set uploaded (by FNegri; author: Zabe):

[operations/puppet@production] maintain-views: Stop providing rc_new and rc_type to replicas

https://gerrit.wikimedia.org/r/1178899

In T402787#11114475, @Ladsgroup wrote:

In T402787#11114471, @fnegri wrote:

That should be fine, with the email to cloud-announce you already planned.

I've sent that already last week, didn't I? With corresponding tech news entry and so on.

https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/HUMVKPDTDUSNAEP5YSRUYUGFLQJ7EPRI/

I've sent that already last week, didn't I? With corresponding tech news entry and so on.

Oh yes, I did even read it! :D Thanks, I've added this task id to the patch, you can merge at your convenience.

I'm actually waiting for this issue of tech news to reach people (later today) and then get to dropping it tomorrow-ish

Sounds good!

Sorry for the noise, I saw that T36320: Remove obsolete recentchanges.rc_new database field was resolved, so I thought this had been missed. I should've double checked the open patches!

Change #1178899 merged by Ladsgroup:

[operations/puppet@production] maintain-views: Stop providing rc_new and rc_type to replicas

https://gerrit.wikimedia.org/r/1178899

Cookbook cookbooks.sre.wikireplicas.update-views run by ladsgroup: Started updating wiki replica views

Cookbook cookbooks.sre.wikireplicas.update-views started by ladsgroup executed with errors:

• an-redacteddb1001.eqiad.wmnet (PASS)
  • Ran Puppet agent
  • Ran 'maintain-views --replace --auto-depool --all-databases --table recentchanges'

• clouddb1017.eqiad.wmnet (PASS)
  • Ran Puppet agent
  • Ran 'maintain-views --replace --auto-depool --all-databases --table recentchanges'

• clouddb1018.eqiad.wmnet (FAIL)
  • Ran Puppet agent
  • The maintain-views run failed, see OUTPUT of 'maintain-views ...' above for details

Cookbook cookbooks.sre.wikireplicas.update-views run by ladsgroup: Started updating wiki replica views

@Ladsgroup usual issue with table locks T300427: Automate maintain-views replica depooling

The workaround I used in the past is to run maintain-views manually on each host while keeping another terminal open to kill queries as needed.

I can take care of that.

If you do it, I'd be grateful. These days we are a bit under-staffed.

Cookbook cookbooks.sre.wikireplicas.update-views started by ladsgroup executed with errors:

• an-redacteddb1001.eqiad.wmnet (PASS)
  • Ran Puppet agent
  • Ran 'maintain-views --replace --auto-depool --all-databases --table recentchanges'

• clouddb1017.eqiad.wmnet (PASS)
  • Ran Puppet agent
  • Ran 'maintain-views --replace --auto-depool --all-databases --table recentchanges'

• clouddb1018.eqiad.wmnet (FAIL)
  • Ran Puppet agent
  • The maintain-views run failed, see OUTPUT of 'maintain-views ...' above for details

(I re-tried it immediately, also failed)

Thanks!!!