The project appears to be a combination of PHP, HTML, and javascript. There is no obvious license attributed to the project in it's Toolforge home directory. The tool was created on 2015-06-17 which unfortunately predates the selection of a default license at tool creation time. Per the 2023-05-26 Cloud Services Terms of use revision, if necessary we can declare the code to be licensed under the GPL-3.0 license. It would be more ideal however for @DB111 to choose and document their own OSI approved license rather than relying on the TOU default licensing clause.

@valerio.bozzolan Are you invoking the Toolforge right to fork policy to obtain a copy of the source code for the wikimap tool? If so, lets make sure that @DB111 understands that their use of the Toolforge compute resources to host their tool at https://wikimap.toolforge.org/ are subject to this policy and that the Toolforge Standards Committee is tasked with facilitating this community right.

This is functionally the same issue as T434075: Reviewer-bot not adding reviewers due to SSH key change. The python code tries to use ssh gerrit gerrit ls-projects to get the list of known projects from gerrit. This fails because the ssh config is expecting the old RSA key that are removed on 2026-08-04.

The announcement sent to the cloud-announce list, which I assume was done so that tool maintainers know things are changing, does not mention that there is no way to access x4 via the Wiki Replicas yet, when that is expected, or what will be possible when the tables are in a new db that cannot be linked across sections to other tables. These are all reasonable questions for folks to have beyond "where did the data go" which is what has been documented in that email.

In T423980#12291649, @Novem_Linguae wrote:

This ticket is for putting x4 on the private mediawiki replicas (e.g. stat1008). It sounds like you also want to put it in the public wiki replicas?

In T423980#12291608, @bd808 wrote:

https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/thread/E4RXH3F3WTBWSGZDN6BZPJGEUY2IM4KP/ announces that on 2026-09-08 (the next WMF US working day after today) the commons link tables will only be available from the x4 cluster. This is going to break a bunch of tools and they will stay broken until the x4 cluster is available in the Wiki Replicas.

In T423980#12291608, @bd808 wrote:

https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/thread/E4RXH3F3WTBWSGZDN6BZPJGEUY2IM4KP/ announces that on 2026-09-08 (the next WMF US working day after today) the commons link tables will only be available from the x4 cluster. This is going to break a bunch of tools and they will stay broken until the x4 cluster is available in the Wiki Replicas.

I know that priority is subjective and should be controlled by people doing the work, but this is also the best signaling mechanism for community impact that we have at the moment.

https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/thread/E4RXH3F3WTBWSGZDN6BZPJGEUY2IM4KP/ announces that on 2026-09-08 (the next WMF US working day after today) the commons link tables will only be available from the x4 cluster. This is going to break a bunch of tools and they will stay broken until the x4 cluster is available in the Wiki Replicas.

I made a backup of the blocking related scripts from deployment-mediawiki14:~root/ and linked to them from the blocking instructions on wikitech.

This is being actively used in the work for T401839: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie. We are not likely to have all of the Bullseye replacements built with tofu, but we will get some. The rest can be imported later and/or replaced again via this gitops tooling.

New hosts:

• deployment-deploy06.deployment-prep.eqiad1.wikimedia.cloud
• deployment-mediawiki15.deployment-prep.eqiad1.wikimedia.cloud
• deployment-mediawiki16.deployment-prep.eqiad1.wikimedia.cloud
• deployment-jobrunner06.deployment-prep.eqiad1.wikimedia.cloud
• deployment-mwmaint04.deployment-prep.eqiad1.wikimedia.cloud

I have added this as a dependency of T436190: Inform relevant groups about implications of testwiki moving to pretrain. The initial Pretrain automated deployments will run at 02:00 UTC Tuesday through Thursday. The wmf/next image build step has been running in this time slot 7 days a week for many months now, but has not been treated as a critical event so if things blew up.

Cluster has existed for quite a while at this point. @dduvall did a lot of follow up work in T424061: Ensure zuul-executor can prepare workspaces and dispatch builds to keep up with OpenStack changes.

@dduvall Did this functionally happen when you moved things to the CAPI backend in T424061: Ensure zuul-executor can prepare workspaces and dispatch builds?

The problem that T436412: [tofu-cloudvps] cloudvps_puppet_prefix.hiera is not idempotent and should be discusses as "the second issue" is the use case that led me to submit this feature request to provide a work around.

Totally possible today. The necessary magic is knowing how to lookup an $ID value. The puppet-enc server can be queried from inside a Beta Cluster 'make shell' session:

bd808$ make shell
🖥️ Entering Docker container...
Identity added: /dev/fd/63 (betadevopsbot)
/srv/app/bin/start-socks-proxy.sh: connect: Connection refused
/srv/app/bin/start-socks-proxy.sh: line 11: /dev/tcp/127.0.0.1/1080: Connection refused
Starting SOCKS5 proxy at 127.0.0.1:1080...
Warning: Permanently added 'bastion.wmcloud.org' (ED25519) to the list of known hosts.
somebody@deployer:/srv/app$ curl -s -H "Accept: application/json" -H "X-Auth-Token: $(OS_DEBUG= openstack token issue -f yaml | yq .id)" "https://puppet-enc.cloudinfra.wmcloud.org/v1/deployment-prep/prefix?detailed" | jq '.prefixes[] | select(.prefix | startswith("deployment-mediawiki"))'
{
  "id": 14426,
  "prefix": "deployment-mediawiki"
}
{
  "id": 21178,
  "prefix": "deployment-mediawiki13.deployment-prep.eqiad1.wikimedia.cloud"
}
{
  "id": 21177,
  "prefix": "deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud"
}
{
  "id": 23620,
  "prefix": "deployment-mediawiki15.deployment-prep.eqiad1.wikimedia.cloud"
}
{
  "id": 23619,
  "prefix": "deployment-mediawiki16.deployment-prep.eqiad1.wikimedia.cloud"
}

To get things back on track in the tofu state I:

• Used Horizon to edit the hiera for deployment-mediawiki15 & deployment-mediawiki16 and remove the profile::mediawiki::php::absented_php_versions: [] empty array from each.
• Imported the cleaned hiera settings into the tofu state:
  • tofu import 'module.mediawiki.cloudvps_puppet_prefix.vm["15"]' 23620
  • tofu import 'module.mediawiki.cloudvps_puppet_prefix.vm["16"]' 23619
• Edited main.tf to also remove the profile::mediawiki::php::absented_php_versions: [] empty arrays.
• Ran make tofu-apply to verify that the main.tf, tofu state, and system state all matched.
• Submitted MR16 to make git state match tofu state.

In a make shell session:

somebody@deployer:/srv/app$ curl -s -H "Accept: application/json" -H "X-Auth-Token: $(OS_DEBUG= openstack token issue -f yaml | yq .id)" "https://puppet-enc.cloudinfra.wmcloud.org/v1/deployment-prep/prefix?detailed" | jq '.prefixes[] | select(.prefix | startswith("deployment-mediawiki"))'
{
  "id": 14426,
  "prefix": "deployment-mediawiki"
}
{
  "id": 21178,
  "prefix": "deployment-mediawiki13.deployment-prep.eqiad1.wikimedia.cloud"
}
{
  "id": 21177,
  "prefix": "deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud"
}
{
  "id": 23620,
  "prefix": "deployment-mediawiki15.deployment-prep.eqiad1.wikimedia.cloud"
}
{
  "id": 23619,
  "prefix": "deployment-mediawiki16.deployment-prep.eqiad1.wikimedia.cloud"
}
somebody@deployer:/srv/app$ curl -s -H "Accept: application/json" -H "X-Auth-Token: $(OS_DEBUG= openstack token issue -f yaml | yq .id)" "https://puppet-enc.cloudinfra.wmcloud.org/v1/deployment-prep/prefix/id/23620" | jq
{
  "hiera": {
    "profile::mediawiki::php::absented_php_versions": [],
    "profile::mediawiki::php::php_versions": [
      "8.5"
    ],
    "profile::mediawiki::webserver::default_php_version": "8.5"
  },
  "id": 23620,
  "prefix": "deployment-mediawiki15.deployment-prep.eqiad1.wikimedia.cloud",
  "roles": []
}

And this made me realize the error that is getting thrown is because of the empty arrays (T436412#12267798).

I think what is happening in the https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/jobs/943247 case is that my implementation of gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/internal/provider.ctyTypeForValue cannot handle an empty slice.

// Compute a cty.Type for an arbitrary value.
func ctyTypeForValue(value any) (cty.Type, error) {
	val := reflect.ValueOf(value)
	kind := val.Kind()
	switch kind {
	case reflect.Invalid:
		// Value is nil and the type is unknown.
		// Let's pretend it should be typed as a string
		return cty.String, nil
	case reflect.Bool:
		return cty.Bool, nil
	case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
		return cty.Number, nil
	case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
		return cty.Number, nil
	case reflect.Float32, reflect.Float64:
		return cty.Number, nil
	case reflect.Slice:
		// Assume that arrays are homogeneous
		listElmType, err := ctyTypeForValue(value.([]any)[0])
		if err != nil {
			return cty.NilType, fmt.Errorf("Failed to derive list element type: %w", err)
		}
		return cty.List(listElmType), nil
	case reflect.Map:
		// Assume that maps are string keyed
		return ctyTypeForMap(value.(map[string]any))
	case reflect.String:
		return cty.String, nil
	default:
		return cty.NilType, fmt.Errorf("Unexpected type %v%w", val.Kind(), ErrCtyTypeForValue)
	}
}

When value is an empty Slice, it calls ctyTypeForValue(value.([]any)[0]) which then will raise runtime error: index out of range [0] with length 0.

The tofu automation is currently stalled by T436412: [tofu-cloudvps] cloudvps_puppet_prefix.hiera is not idempotent and should be. My short term plan for getting unstuck is to use https://puppet-enc.cloudinfra.wmcloud.org/v1/deployment-prep/prefix?detailed to get the data needed to manually add the enc data that was not updated as hoped in the tofu state graph with tofu import.

Marking as in progress and high priority, but not claiming ownership as this work may not be completed before my sabbatical starts on September 1. I'm going to give it a fair shot though. :)

I think the work left here is putting the thresholds @dancy computed above into the config. Is that a patch you can prepare @dancy?

If the "oops there is already a prefix here" step failed with an error message that gave the upstream prefix's internal id it would then be possible to use tofu import to update the state graph and get things back on track.

Some of this discussion has started already.

group that grants editinterface (e.g. sysops and interface admins). If it is, the account should be automatically approved.

Next week's Deployment calendar should include Pretrain windows Monday-Thursday. I decided it would be better to list Monday even if we are not using it yet as we expect to use it once we have some confidence in the ability to update the pretrain container without updating other containers unnecessarily.

In T435869#12253864, @Jnanaranjan_sahu wrote:

2. Security Review: I've updated the description of T435890: Application Security Review for RecentPageStats extension to use the standard security template.

In T433998#12258671, @ssingh wrote:

Traffic will work on upgrading deployment-cache-(text|upload)08 if that's fine and we would like to coordinate about that. Is this task the right place to do that or is something else preferred?

With T426484: Add proxy support to GitHub actions implemented is the general problem solved for your needs now @Xqt?

To prove to myself that the PHP changes are working as intended I built a test instance using tofu.

module "mediawiki" {
  source = "./modules/service"

I seem to have fixed the opcache.ini problem with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328728/3..5/modules/php/manifests/default_extensions.pp

bd808@deployment-deploy06:~$ php --version
PHP 8.5.9 (cli) (built: Aug  6 2026 12:42:42) (NTS)
Copyright (c) The PHP Group
Built by Debian
Zend Engine v4.5.9, Copyright (c) Zend Technologies
    with Zend OPcache v8.5.9, Copyright (c), by Zend Technologies

I will try building another instance with PHP 8.5 enabled to see if there are more weird things that were just incidentally fixed as I poked around figuring out what was wrong.

@CCiufo-WMF This is basically the "what's next after the Pretrain MVP is rolled out" step that I was talking about recently.

Nearly there.

bd808@deployment-deploy06:~$ php --version
Warning: Failed loading Zend extension 'opcache.so' (tried: /usr/lib/php/20250925/opcache.so (/usr/lib/php/20250925/opcache.so: cannot open shared object file: No such file or directory), /usr/lib/php/20250925/opcache.so.so (/usr/lib/php/20250925/opcache.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP 8.5.9 (cli) (built: Aug  6 2026 12:42:42) (NTS)
Copyright (c) The PHP Group
Built by Debian
Zend Engine v4.5.9, Copyright (c) Zend Technologies
    with Zend OPcache v8.5.9, Copyright (c), by Zend Technologies

Puppet is still making a /etc/php/8.5/mods-available/opcache.ini and linking in th various conf.d places.

I'm not going to claim this task because it is very unlikely to finish before my sabbatical starts, but I was at least able to put up a patch to kickstart any debate about how we should actually setup the timer's command.

jouncebot is the main bot I can think of involved in things about deployments. It will "just work" once [[wikitech:Deployments]] has the window data.

Actually enabling this deployment should be considered blocked on communications out to SREs and deployer that the unsupervised deployment step is being enabled. Those communications should include updating the the automation for https://wikitech.wikimedia.org/wiki/Deployments so that the deployment window is documented. It would be perfectly fine for that update to document it as M-Th even though we are not enabling for Monday in this first increment.

bd808@deployment-cache-upload08.deployment-prep.eqiad1:~$ sudo -i run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-cache-upload08.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(5f3f5a9d2d) gitpuppet - P:mediawiki::php: add 8.5'
Error: /Stage[main]/Profile::Cache::Haproxy/Acme_chief::Cert[unified]/File[/etc/acmecerts/unified]: Failed to generate additional resources using 'eval_generate': certificate verify failed [unable to get local issuer certificate for CN=deployment-acme-chief07.deployment-prep.eqiad1.wikimedia.cloud]
Error: /Stage[main]/Profile::Cache::Haproxy/Acme_chief::Cert[unified]/File[/etc/acmecerts/unified]: Could not evaluate: Could not retrieve file metadata for puppet://deployment-acme-chief07.deployment-prep.eqiad1.wikimedia.cloud/acmedata/unified: certificate verify failed [unable to get local issuer certificate for CN=deployment-acme-chief07.deployment-prep.eqiad1.wikimedia.cloud]
Notice: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]: Dependency File[/etc/acmecerts/unified] has failures: true
Warning: /Stage[main]/Haproxy/Systemd::Service[haproxy]/Service[haproxy]: Skipping because of failed dependencies
Notice: Applied catalog in 10.99 seconds

This is fallout from the switch to the new acme-chief servers that @BCornwall built yesterday. Hopefully he has an idea of what needs to happen to fix it. We cannot modify the CDN edge blocking rules until we can run puppet on the deployment-cache-{text,upload} boxes, so I'm going to mark this as high priority.

In T421244#11847209, @bd808 wrote:

Booleans were coerced to strings somewhere???

• yamlencode({"a": true, "c": false}) works as expected in tofu console. Did I do something weird in the provider when I worked on clean round tripping?

This is {{done}} except for README.md updates that I forgot to make.

The interesting thing about a revert is that the commit is deployed, but the revert commit is also which then nullifies the original patch.

test

In T435393#12254809, @bd808 wrote:

OPcache is apparently built-in for our php8.5 packages. That is going to be annoying to special case in Puppet.

I think this was Brett working through things on deployment-acme-chief{07,08}. Hopefully he can confirm.