• https://wikitech.wikimedia.org/w/index.php?title=Fundraising/SAL&oldid=2377054
• https://wikimedia.social/@sal/115947349186922854
• https://sal.toolforge.org/log/AI9_7ZsB8tZ8Ohr05_mI
• https://sal.toolforge.org/fundraising

Stashbot's config is kept in private files on Toolforge because I have been too lazy to update things to read secrets from envvars instead of embedding them in a YAML document. I will log the changes here for posterity.

In T415389#11550633, @greg wrote:

Both are fine. And I assume the separate view on https://sal.toolforge.org/ is just going to be via having a fundraising entry on the /projects list?

Would https://wikitech.wikimedia.org/wiki/Fundraising/SAL be a reasonable place to record the !log messages?

The semi-secret magic here was that the #wikimedia-operations channel config matches all Phabricator tags and alternate tags that start with sre- (case insensitive). The case insensitive part is likely non-obvious from reading the current config file.

In T337570#11550325, @bd808 wrote:

I have a hunch that this configuration state is the problem, and that the Phorge integration being enabled at the instance level may be complicating disabling the legacy "Custom issue tracker" integration.

In T337570#11550285, @dancy wrote:

In T337570#11550214, @bd808 wrote:

Phabricator linking seems to be working on https://gitlab.wikimedia.org/toolforge-repos/versions/-/merge_requests/8. I wonder if there is something interesting going on in that wikifunctions MR description? Does the regex that is in use allow for trailing whitespace? Is there a newline requirement?

Interesting. By the way it was https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/641 that made me notice this.

In T411516#11550236, @bd808 wrote:

The train ran for a week and continued to run in the next week without any identified problems. I think we can call this experiment a success and proceed to the cleanup phase where we update docs on mediawiki.org to stop telling people that they need to rush to get their extension/skin on the train early.

We could technically close this as invalid because it turned out that the requested functionality was already in core, but resolved seems more correct because of the effort that was put into proving that the desired change was actually sufficient to keep scap working as desired. It would also be fair to toss a coin to decide if @thcipriani or I deserve the claim on resolution; I proved that core worked as desired and @thcipriani was able to show that scap worked in testing with a missing extension. The production test was a trivial, but necessary follow up. I used https://justflipacoin.com/ with Heads == @thcipriani, Tails == @bd808 to determine that @thcipriani gets the credit.

The train ran for a week and continued to run in the next week without any identified problems. I think we can call this experiment a success and proceed to the cleanup phase where we update docs on mediawiki.org to stop telling people that they need to rush to get their extension/skin on the train early.

The display values add up to 99.99% now. We could get incrementally closer to 100% by showing more digits, but hopefully this is good enough.

In T337570#11549528, @dancy wrote:

If a merge request description body consists of a single Bug: TXXXXX line, the link to Phabricator does not get created. I'm wondering if this is a new problem since the recent Gitlab UI changes. See the job link referenced in the task description for an example (https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/18).

Superseded by T415293: Shut down the API Portal

In T415360#11549593, @Psymetric wrote:

Alright, no worries, we'll check out a possible Toolforge implementation.

In T415360#11549410, @Psymetric wrote:

The wiki has not been created yet on Miraheze and is awaiting approval (https://meta.miraheze.org/wiki/Special:RequestWikiQueue/73470).

There is a user record for the https://ldap.toolforge.org/user/jacobhung Developer account in the https://toolsadmin.wikimedia.org/ database. That is also connected to the JacobHung SUL account which would keep toolsadmin from letting you create another Developer account using that same SUL account.

It is generally possible to host a MediaWiki instance as a Toolforge tool. Some examples:

• https://mwdemo.toolforge.org/wiki/Main_Page
• https://toolhub-extension-demo.toolforge.org/wiki/Main_Page

I think the thing to try here would be a branch of https://gitlab.wikimedia.org/toolforge-repos/bridgebot that pulls in a treeish from https://github.com/matterbridge-org/matterbridge instead of the one currently being used from https://gitlab.wikimedia.org/toolforge-repos/bridgebot-matterbridge.

Looking at P87548 I just realized that the current rproxy solution I have built really only works to provide a single reverse proxy per Toolforge tool. This is because the envvar based configuration only allows one set of config data to be supplied. I need to think a bit harder about reasonable ways to support N deployments per tool. It would be relatively simple to add support for a config file. I would like a solution that avoids using NFS if I can dream one up.

I have an initial working solution at https://gitlab.wikimedia.org/toolforge-repos/containers-rproxy. I have deployed it to replace the tool-bd808-test.proxy-scdn ingress only proxy from the task description. That looked something like:

bd808@laptop:~$ ssh dev.toolforge.org
bd808@tools-bastion-14:~$ become bd808-test
tools.bd808-test@tools-bastion-14:~$ kubectl delete ingress proxy-scdn
tools.bd808-test@tools-bastion-14:~$ kubectl delete service i-scdn-co
tools.bd808-test@tools-bastion-14:~$ toolforge envvars create RPROXY_UPSTREAM_URL 'https://i.scdn.co'
tools.bd808-test@tools-bastion-14:~$ toolforge envvars create RPROXY_PATH_REGEX '/scdn(/|$)(.*)'
tools.bd808-test@tools-bastion-14:~$ toolforge envvars create RPROXY_PATH_TEMPLATE '/$2'
tools.bd808-test@tools-bastion-14:~$ toolforge envvars create GO_LOG debug
tools.bd808-test@tools-bastion-14:~$ toolforge jobs run \
    --image tool-containers/rproxy:latest \
    --command web \
    --continuous \
    --port 8000 \
    --health-check-http '/healthz' \
    rproxy-scdn
tools.bd808-test@tools-bastion-14:~$ kubectl apply --validate=true -f - << EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rproxy-scdn
spec:
  rules:
    - host: bd808-test.toolforge.org
      http:
        paths:
          - path: /scdn
            pathType: Prefix
            backend:
              service:
                name: rproxy-scdn
                port:
                  number: 8000
EOF

In T413803#11539622, @IKhitron wrote:

And the Beta wikis are down now, there is just not enough time.

sudo journalctl --since "2026-01-20 10:05:00" --until "2026-01-20 10:15:00" turned up the kernel oom-killer going after cassandra in the time range where we had a data collection gap.

Cherry-pick has things running again. Hopefully @SLyngshede-WMF or @Joe can make time to review and merge my patch upstream in ops/puppet.git. Thanks for the nudge in the right direction @ssingh.

In T415113#11539251, @bd808 wrote:

I wonder if that lua.check_traffic_class method is coming from a private location in production?

bd808@mbp03:~/projects/wmf/operations/puppet$ git grep check_traffic_class
modules/profile/templates/cache/haproxy/tls_terminator.cfg.erb:    http-request lua.check_traffic_class

Running the config check with all of the config files gives a different error:

bd808@deployment-cache-text08.deployment-prep.eqiad1:~$ sudo haproxy -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -c
[NOTICE]   (24908) : haproxy version is 2.8.18-1~bpo11+1
[NOTICE]   (24908) : path to executable is /usr/sbin/haproxy
[ALERT]    (24908) : config : parsing [/etc/haproxy/conf.d/tls.cfg:218]: 'http-request' expects 'wait-for-handshake', 'set-log-level', 'set-nice', 'use-service', 'sc-add-gpc(*)', 'sc-inc-gpc(*)', 'sc-inc-gpc0(*)', 'sc-inc-gpc1(*)', 'sc-set-gpt(*)', 'sc-set-gpt0(*)', 'send-spoe-group', 'do-resolve(*)', 'cache-use', 'add-acl(*)', 'add-header', 'allow', 'auth', 'capture', 'del-acl(*)', 'del-header', 'del-map(*)', 'deny', 'disable-l7-retry', 'early-hint', 'normalize-uri', 'redirect', 'reject', 'replace-header', 'replace-path', 'replace-pathq', 'replace-uri', 'replace-value', 'return', 'set-header', 'set-map(*)', 'set-method', 'set-path', 'set-pathq', 'set-query', 'set-uri', 'strict-mode', 'tarpit', 'track-sc(*)', 'set-timeout', 'wait-for-body', 'set-var-fmt(*)', 'set-var(*)', 'unset-var(*)', 'set-dst', 'set-dst-port', 'set-mark', 'set-src', 'set-src-port', 'set-tos', 'silent-drop', 'set-priority-class', 'set-priority-offset', 'set-bandwidth-limit', 'lua.is_datacenter', 'lua.res_proxy', 'lua.set_contact_info', but got 'lua.check_traffic_class'.
[ALERT]    (24908) : config : Error(s) found in configuration file : /etc/haproxy/conf.d/tls.cfg

Looks to be a duplicate of the behavior from T412774: Project deployment-prep instance deployment-sessionstore06 is down. The instance is up, but something spiked it's load to a point where Prometheus scrapes failed causing a down time alert.

Already resolved.

bd808@deployment-puppetserver-1.deployment-prep.eqiad1:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(51d830ad4e) gitpuppet - varnish: Move error message from footer to body for HTTP 4xx responses'
Notice: /Stage[main]/Profile::Puppetserver::Volatile/File[/srv/puppet_fileserver/volatile/datacenter_vendors]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Profile::Puppetserver::Volatile/File[/srv/puppet_fileserver/volatile/datacenter_vendors]/ensure: removed (corrective)
Notice: Applied catalog in 11.95 seconds

In T414864#11538114, @dancy wrote:

For this to be feasible we would need to set up an egress gateway node on the D.O. side and use the DOKS routing agent to ensure that outbound traffic exits through the gateway.

The runners you would like unblocked are hosted on Digital Ocean. I do not think that it would be reasonable to open the Beta Cluster to the full DO IPv4 address space. If we have a fixed sub-space for egress we can unblock that. @dancy can you help me figure out if there is a restricted range that the runner egress through?

In T414920#11532819, @hashar wrote:

wmf/next might be needed though.

In T410730#11530540, @Amdrel wrote:

Effectively the script would do the following steps:

1. ssh into each encoder
2. Apply a patched Puppet manifest with credentials
3. Run systemctl --no-block reload v2ccelery.service to initiate the Celery restart (non-blocking)
4. Repeat until there are no more instances

In T414674#11529972, @taavi wrote:

In T414674#11529957, @bd808 wrote:

I will try to make a reverse proxy container in the same spirit as the redirect container. I can borrow from things I learned building https://gitlab.wikimedia.org/toolforge-repos/gitlab-content to make something relatively light weight I think.

That would be very helpful, thank you!

Do you have thoughs already on how that would deal with tools that currently host a "normal" web service in addition to those reverse proxies? (e.g. tool-moedata has four proxy ingresses in addition to the mail webservice, it seems.)

In T414674#11524714, @taavi wrote:

The current list of tools relying on this functionality is at P87548. Of those, a couple are simple redirects that can be replaced with https://wikitech.wikimedia.org/wiki/Tool:Containers#Redirect_container, but the ones proxying to external upstream servers will need an another solution.

fresh-node24 is also not in the most recently tagged 24.05.1 release, so I guess after the checksum is fixed a new release is needed as well.

In T142237#10449475, @bd808 wrote:

It would be nice though, so I will try not to loose track of this for years again.

It was only one year this time... I was about to finally request adoption, but I saw that Matt has had some edits on enwiki really recently. I bumped the threads on his enwiki and wikitech Talk pages one more time. I am setting a reminder for myself to go ahead and file the adoption request on 2026-01-29 if I haven't heard back.

I think this may just be a quirk of tunnelencabulator, but when I run it and use ssh gerrit -- gerrit show-connections --wide I see myself entering via IPv6 and not an edge proxy. For me ssh -4 is necessary to route traffic over the tunnel.

In https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment#Preparing_for_deployment we wrote:

IMPORTANT: Do this early! Ideally, at least three weeks prior to your target deployment date, to ensure that your extension is present as a submodule in the required branches. (The extension submodule must be present in all branches currently running on the cluster, or the localization cache builder will fail.)

https://wikitech.wikimedia.org/w/index.php?title=Tool:Gitlab-account-approval/Log&diff=prev&oldid=2374449

Yeah, it's going to be the same problem. The quick fix is adding profile::tlsproxy::envoy::upstream_sni: null and profile::tlsproxy::envoy::upstream_tls: false to the project's hiera. The longer fix is either https://gerrit.wikimedia.org/r/c/operations/puppet/+/702326 or moving the defaults into the lookup calls in the ::profile::tlsproxy::envoy module.

Welcome to the projects @Lwilson-ctr. I have added you to the Trusted-Contributors group here in Phabricator. After you link your https://ldap.toolforge.org/user/lwilson-ctr Developer account to your Phabricator profile in User Settings the Tool-gitlab-account-approval bot should approve your GitLab account within a few minutes. That group membership should also make using Phabricator a little bit nicer for you.

I just nudged https://gerrit.wikimedia.org/r/c/operations/puppet/+/702326 as a possible fix for problems like T414304: No Puppet resources found on instance deployment-shellbox01 on project deployment-prep which are caused by the failure to load /etc/puppet/hieradata/common/$classpath.yaml configuration in the Cloud VPS realm(s).

A local fix for Beta Cluster is copying the new hiera settings into the project global hiera:
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/e1aea5f956fbc945bd7bb34ede02b24e77efbaf5%5E%21/#F0

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index 2151103..5f2d748 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

Failure appears to be caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219770 where @ABran-WMF introduced new Hera settings profile::tlsproxy::envoy::upstream_tls and profile::tlsproxy::envoy::upstream_sni. These have been given Production global defaults in hieradata/common/profile/tlsproxy/envoy.yaml, but that config file is not in the puppetserver's hiera lookup scope within the Beta Cluster or other WMCS projects.

I added some user facing docs that someone might accidentally see at https://wikitech.wikimedia.org/wiki/Tool:Iw#URLs_with_query_parameters describing the problem and some workarounds.

@jeena will be shipping my risky patch with the 1.46-wmf.11 train (T413802: 1.46.0-wmf.11 deployment blockers). Let's hope for a smooth run like @thcipriani's local and Beta Cluster tests have already seen.

1. Risky Patch! 🚂🔥