In T264221#7075468, @Bstorm wrote:

It seems to me that adding something in the template to specify the port for the ingress is the simplest and safest solution. That should just be a profile-based template and adding a parameter with a sane default.

So tools ingress nodes are failing puppet since their ephemeral disk is only 20G while the patch has min_gb => 40,. Is there any reason not to remove that?

Updated the docs: https://wikitech.wikimedia.org/w/index.php?title=Portal:Toolforge/Admin/Kubernetes/Networking_and_ingress&curid=444438&diff=1911417&oldid=1886320

I just deployed the new ingress controller to toolsbeta:

Yes, the wiki replicas have been upgraded and you now need to connect to the correct shard. See https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign for details. This was announced multiple times on the cloud-announce@ mailing list that we strongly recommend that all cloud services users are subscribed to.

Untagging stewardbots, SULWatcher no longer uses this channel

Thanks! Deployed with two sets of fixes (https://gerrit.wikimedia.org/r/c/labs/tools/stewardbots/+/687591/ and https://gerrit.wikimedia.org/r/c/labs/tools/stewardbots/+/687594) that both were intended to be included in the initial change.

The last time this was done (T250874) was about 11 months ago, so we have another month remaining before the certificates expire.

I'm still unable to get the admission webhook working with v1 api, but the v1beta1 api still appears to be working with the ingress controller v0.46.0. Managed to find a spot where to put the default backend too, so I think it's ready for toolsbeta.

I have the ingress controller v0.46.0 as deployed by Helm running mostly fine locally. There are still a things to deal with (notably 404 handling and verifying the admission controller works), I'll likely deploy it to the new toolsbeta node either tomorrow or early next week.

In T23272#7071382, @Xaosflux wrote:

Would a non-enwiki OS'er screen shot the deleted lines here (https://en.wikipedia.org/w/index.php?title=User:Xaosflux/sandbox101&action=history) to illustrate?

Hopefully fixed for good this time.

Usual process for renaming global groups is something like this:

1. add messages for new group in WikimediaMessages, wait until they roll out everywhere with the train
2. ask a steward to rename the group
3. remove messages for the old group

(edit conflict)

Ingress and control currently use a smaller flavor (g2.cores2.ram4.disk40), we need an equivalent flavor for those. Is 140G ephemeral storage really needed? For example tools-k8s-worker-68 is currently using 40G in /docker, we could use smaller disks to preserve space.

We can be 100% certain there are no clashes by creating three new ingress nodes with a slightly different taint/label, deploy via helm, test and then redirect the load balancers via puppet

• tools-k8s-ingress-1 and tools-k8s-ingress-2
• Beware, has a server group

• cloud-admin@lists.wikimedia.org per https://wikitech.wikimedia.org/wiki/Help:Access_policies#Responsibilities

protecting per T281972

In T281625#7062453, @bd808 wrote:

@Majavah what new quota are you hoping to receive? This ask currently seems to be for "more", but it is not clear how much more. +100GiB? +300GiB? +500GiB?

I'm calling this resolved. The patches are deployed and I amended the documentation on https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Databases#Read_only for these situations and updated enwiki and metawiki error messages to link to it.

protecting due to the abuse potential of not being able to suppress an user with PII or other sensitive material in the username

This needs a proper solution for the CentralAuthGroupMembershipProxy/SpecialGlobalGroupMembership hack, see also T263207 T255309.

Maybe T280744: Mail to root@lists1001.wikimedia.org from noreply@lists1001.wikimedia.org doesn't work is related?

In T280807#7054326, @Aklapper wrote:

See https://admin.toolforge.org/tool/commonsarchive for a list of maintainers

Yeah, that works. Thanks!

In T110115#7050521, @jcrespo wrote:

Is there a way to have effective monitoring on beta

Now that we have cfssl I created a CNAME for a svc domain name and configured an envoy proxy in front of Swift. This means that it's now available from https://ms-fe.svc.deployment-prep.eqiad1.wikimedia.cloud. I haven't moved any clients yet, will do in the future if someone isn't faster than me.

So this is now working on deployment-mediawiki11 with the help of jbond to get CFSSL working with envoy. Next step is to get caches and other hosts to trust it.

In T206158#7048731, @Majavah wrote:

So Envoy will only be installed if profile::envoy::ensure is present, but if that's set to present you need to have at least one services_proxy enabled listener.

Not sure if intentional or not.

Hi! The existing enwiki tool, UTRS, has already had a significant amount of work put into it to make it support stewards and other non-enwiki wikis. While most required functionality is already present, it's not fully there yet, mostly due to lack of volunteer availability for code review.

Now that CFSSL is a thing, I took a look at this as the certs are much easier to generate. I added the following hiera to deployment-mediawiki11:

profile::envoy::ensure: present
profile::mediawiki::webserver::has_tls: true
profile::pki::client::ensure: present
profile::services_proxy::envoy::enabled_listeners:
- dummy-to-workaround-requirements
profile::services_proxy::envoy::listeners:
- name: dummy-to-workaround-requirements
  port: 24107
  service: dummy-to-workaround-requirements
  timeout: 1s
  upstream: example.com
profile::tlsproxy::envoy::access_log: true
profile::tlsproxy::envoy::capitalize_headers: true
profile::tlsproxy::envoy::cfssl_label: deployment-prep.eqiad1.wikimedia.cloud
profile::tlsproxy::envoy::global_cert_name: deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
profile::tlsproxy::envoy::retries: false
profile::tlsproxy::envoy::services:
- port: 80
  server_names:
  - '*'
profile::tlsproxy::envoy::sni_support: 'no'
profile::tlsproxy::envoy::ssl_provider: cfssl
profile::tlsproxy::envoy::tls_port: 443
profile::tlsproxy::envoy::upstream_response_timeout: 203.0
service::catalog:
  dummy-to-workaround-requirements:
    description: this only exists to workaround https://github.com/wikimedia/puppet/blob/d780c818b698f2f255f51f105c3d815099291c39/modules/profile/manifests/services_proxy/envoy.pp#L37
    encryption: false
    ip:
      eqiad:
        default: 192.0.2.1
    lvs:
      class: high-traffic1
      conftool:
        cluster: dummy-to-workaround-requirements
        service: dummy-to-workaround-requirements
      depool_threshold: '.5'
      enabled: true
      monitors:
        IdleConnection:
          max-delay: 300
          timeout-clean-reconnect: 3
      scheduler: wrr
    port: 24108
    sites: []
    state: production

Instead of tracking the current primary in hiera/etc, I'd propose to just set read_only = 1 by default on all beta database servers. The replicas should be read-only (which this task is all about), and the primary server isn't restarted often (especially now that Cloud VPS VMs can be live migrated between hypervisors without downtime), if they are it's usually a human who can set the primary to read-write using set global read_only = 0; and if it's a crash it's a good idea for a human to look at the logs before allowing writes.

Treating as a software bug and not a betacluster infrastructure one, this is a blocker for next train

https://github.com/wikimedia/Wikibase/commit/8c912634c1316be52653dc5514701b124809393d looks related, cc @Lucas_Werkmeister_WMDE

Aren't those supposed to exist on wikidatawiki and not on enwiki?

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/680821 looks related, cc @tstarling

After chatting with Leaderboard and DannyS712 on IRC I went ahead and granted +steward, given there is a valid use case (testing things, which beta is designed for) and they are a trusted user with some elevated access on production too.

(change visibility) 21:08, 28 April 2021 Majavah talk contribs block changed global group membership for User:Zabe from (none) to developer (phab:T281239)

Done, per multiple supports and no opposes above.