Change 586300 had a related patch set uploaded (by Nikerabbit; owner: Nikerabbit):
[mediawiki/extensions/ContentTranslation@master] Avoid failure on restoring draft with no categories

Change 586296 merged by Vgutierrez:
[operations/puppet@production] ATS: Remove host level codfw storage_elements config

Change 586299 had a related patch set uploaded (by Ema; owner: Ema):
[operations/debs/varnish4@debian-wmf] 5.1.3-1wm13: handle fetch error in vbf_stp_condfetch

Change 586298 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable wmf-analytics log

I'd rather avoid creating and managing inetnums at a per team or per route object granularity, but it's not possible to create inetnums for the whole allocation (whole /22 for example), only sub-allocations.

How is any of the assertion listed in this tasks description an issue? What do we win by disallowing one form?

assertFalse should be used instead of comparing to false

assertTrue should be used instead of comparing to true

assertNull should be used instead of comparing to null

Change 586297 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/WikibaseMediaInfo@master] Split API messages to a separate file

Change 586296 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Remove host level codfw storage_elements config

Mentioned in SAL (#wikimedia-operations) [2020-04-06T07:06:55Z] <marostegui> Rename wb_terms on codfw - T248086

Change 583613 merged by Vgutierrez:
[operations/puppet@production] partman: clean up cacheproxy selectors

Change 583613 merged by Vgutierrez:
[operations/puppet@production] partman: clean up cacheproxy selectors

@Aklapper have you tried https://turnilo.wikimedia.org/ or http://superset.wikimedia.org/ ? The latter allows you now to explore hive data via Presto (see SQLLab), I am wondering if it is sufficient for your use case. If so, it wouldn't require an explicit kerberos account (since you wouldn't access hadoop directly), let me know :)

Change 586208 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] admin: add kerberos flag for aklapper

In T249059#6030883, @jcrespo wrote:

While I agree with everyone here, I have a couple of thoughts here:

• Clearly the right option would be to have staging, but that is not available at the moment, users should move when that is available
• While analytics setup is not thought to run high load queries, those actually happen due to the fact that MySQL/InnoDB is not an OLAP engine, and that analytics-like queries tend to stress MySQL process more than productiony-like ones
• If OLTP queries are run on the Analytics replicas, those should be way less impacting than OLAP queries currently running by analytics users and researchers
• Analytics should not be the first option, wikireplicas, available to all users, should be the first option considered to run queries (but that may not work for all users)

@Dwisehaupt I changed the identityFile as per your reply (point 1 below) and got the warning (point 2). What am I missing?

@Aklapper Hi! I'm sorry for the long delay, I had trouble setting up with the way Wikimedia works with Git and I abandoned collaboration to python projects altogether.

Acked this morning warnings in icinga for commonswiki_content shard size for cloudelastic.

Change 586207 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Replace dbproxy1011 with dbproxy1019

Change 586206 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] wikireplicas_dns: Replace dbproxy1011 with dbproxy1019

Change 586204 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki: convert all API servers to use envoy for TLS termination.

Mentioned in SAL (#wikimedia-operations) [2020-04-06T06:30:22Z] <marostegui> Upgrade dbproxy1019 - T231520

While I agree with everyone here, I have a couple of thoughts here:

• Clearly the right option would be to have staging, but that is not available at the moment, users should move when that is available
• While analytics setup is not thought to run high load queries, those actually happen due to the fact that MySQL/InnoDB is not an OLAP engine, and that analytics-like queries tend to stress MySQL process more than productiony-like ones
• If OLTP queries are run on the Analytics replicas, those should be way less impacting than OLAP queries currently running by analytics users and researchers
• Analytics should not be the first option, wikireplicas, available to all users, should be the first option considered to run queries (but that may not work for all users)

Change 586205 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: convert all appserver to use envoy for TLS termination

Change 586204 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: convert all API servers to use envoy for TLS termination.

Change 586203 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] ci: Use docker.io on Buster

So what is the status of this? Which rights are needed and which are required?

Change 586167 merged by Marostegui:
[operations/puppet@production] install_server: Allow labsdb1011 to reimage with Buster

Change 586167 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow labsdb1011 to reimage with Buster

Ready for DC-Ops!

Change 586166 merged by Marostegui:
[operations/dns@master] wmnet: Remove production DNS entries for dbproxy1010

Change 586165 merged by Marostegui:
[operations/puppet@production] dbproxy1010: Remove all puppet references

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: dbproxy1010.eqiad.wmnet

• dbproxy1010.eqiad.wmnet (PASS)
  • Downtimed host on Icinga
  • Found physical host
  • Downtimed management interface on Icinga
  • Wiped bootloaders
  • Powered off
  • Set Netbox status to Decommissioning
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB

Change 586166 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Remove production DNS entries for dbproxy1010

Change 586165 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1010: Remove all puppet references

Mentioned in SAL (#wikimedia-operations) [2020-04-06T05:50:51Z] <vgutierrez> ats-tls restart in cp3056, cp3058 and cp3062 - T249335

Change 586164 had a related patch set uploaded (by Kevin Bazira; owner: Kevin Bazira):
[mediawiki/extensions/Jade@master] Added MW message key for jade-deleteproposal

I agree with Nuria - we are kinda converting the Analytis replica into a staging environment, and my fear is that it will be the norm here: "hey, just use the Analytics replica for testing that query".
The Analytics replica do have the same data as production (there might be a few indexes differences) but in general, they do have the same table structure and data.
On the other hand, the configuration (specially the buffer pool size - and warmness) is _way_ different from what we have in production, as it is way smaller.