Page MenuHomePhabricator

taavi (Taavi Väänänen)
SREAdministrator

Today

  • No visible events.

Tomorrow

  • No visible events.

Friday

  • No visible events.

User Details

User Since
Feb 24 2019, 3:58 PM (373 w, 2 d)
Roles
Administrator
Availability
Available
IRC Nick
taavi
LDAP User
Majavah
MediaWiki User
Taavi [ Global Accounts ]

Recent Activity

Yesterday

taavi added a parent task for T422459: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet: T423998: The imagelinks replica in hewiki is not available.
Tue, Apr 21, 9:21 AM · cloud-services-team, Data-Services, Data-Engineering-Radar, DBA, Data-Engineering
taavi added a subtask for T423998: The imagelinks replica in hewiki is not available: T422459: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet.
Tue, Apr 21, 9:21 AM · Data-Services, cloud-services-team
taavi removed a subtask for T422459: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet: T423998: The imagelinks replica in hewiki is not available.
Tue, Apr 21, 9:21 AM · cloud-services-team, Data-Services, Data-Engineering-Radar, DBA, Data-Engineering
taavi removed a parent task for T423998: The imagelinks replica in hewiki is not available: T422459: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet.
Tue, Apr 21, 9:21 AM · Data-Services, cloud-services-team
taavi moved T423998: The imagelinks replica in hewiki is not available from Backlog to Wiki replicas on the Data-Services board.
Tue, Apr 21, 9:21 AM · Data-Services, cloud-services-team
taavi edited projects for T423998: The imagelinks replica in hewiki is not available, added: Data-Services; removed Quarry.
Tue, Apr 21, 9:20 AM · Data-Services, cloud-services-team
taavi closed T416677: Move internal dumps NFS clients to clouddumps1001 as Declined.

This does not seem relevant after moving dumps HTTPS behind LVS.

Tue, Apr 21, 8:32 AM · tools-infrastructure-team, SRE, Datasets-General-or-Unknown
taavi added a comment to T423970: [tofu-cloudvps] Add support for importing legacy cloudvps_puppet_prefix objects.

As far as I can tell, the provider already has code for this. Did you try that and find it not working, or was this task filed on the assumption that it wouldn't be there?

Tue, Apr 21, 8:27 AM · cloud-services-team, Cloud-VPS

Mon, Apr 20

taavi created T423921: repos/wme/pageviews-data-transfer is missing a software license.
Mon, Apr 20, 5:09 PM · Software-Licensing, Wikimedia Enterprise
taavi added projects to T423310: IPv6 non-functional in GitLab CI environments: GitLab (CI & Job Runners), IPv6.

(Partial) duplicate of T403746: Support IPv6 on WMCS hosted runners?

Mon, Apr 20, 3:48 PM · IPv6, GitLab (CI & Job Runners), collaboration-services
taavi added a project to T423870: jobs-api: set enable_storage to true in values.yaml: Toolforge.

@Raymond_Ndibe please remember to add a project tag like Toolforge in addition to a team tag

Mon, Apr 20, 1:44 PM · Patch-For-Review, Toolforge, tools-platform-team

Fri, Apr 17

taavi closed T423703: DISPLAYTITLE: allow capitalization shifted beyond first letter as Declined.

Yes, but only if $wgRestrictDisplayTitle is set to false, which it is not on most Wikimedia wikis. In the configuration, DISPLAYTITLE is meant for changing formatting in ways that are not possible otherwise. The added confusion this would create would hardly be worth the tiny benefit of this over renaming the page to match the title it would ve displayed as.

Fri, Apr 17, 10:50 PM · MediaWiki-Page-derived-data
taavi added a project to T422691: [builds-builder] use yq instead of tomljson/jq/jsontoml: Toolforge.
Fri, Apr 17, 5:23 PM · Toolforge, tools-platform-team
taavi edited projects for T422830: Openstack uwsgi logging to '<frozen importlib._bootstrap>.log', added: tools-infrastructure-team; removed tools-platform-team.
Fri, Apr 17, 5:22 PM · tools-infrastructure-team, Cloud-VPS
taavi edited projects for T422801: Consider allowing cumin access to all Cloud VPS VMs, added: tools-infrastructure-team; removed tools-platform-team.
Fri, Apr 17, 5:22 PM · tools-infrastructure-team, Cloud-VPS
taavi added a project to T423412: [jobs-cli] refactor to use job_type argument that is already available in jobs-api: Toolforge.
Fri, Apr 17, 5:22 PM · Patch-For-Review, Toolforge, tools-platform-team
taavi added a project to T423731: lima-kilo: remove git dir check for local deployment: Toolforge.
Fri, Apr 17, 5:22 PM · Toolforge, tools-platform-team
taavi edited projects for T423703: DISPLAYTITLE: allow capitalization shifted beyond first letter, added: MediaWiki-Page-derived-data; removed MediaWiki-extensions-DisplayTitle.
Fri, Apr 17, 2:27 PM · MediaWiki-Page-derived-data
taavi closed T423703: DISPLAYTITLE: allow capitalization shifted beyond first letter as Declined.

Such changes to the title can be done by renaming the page. Using DISPLAYTITLE for the first character exists as MediaWiki considers all pages to start with an uppercase letter.

Fri, Apr 17, 2:26 PM · MediaWiki-Page-derived-data

Thu, Apr 16

taavi closed T423610: Cannot contact service with port from buildservice as Invalid.

Clarified the docs with https://wikitech.wikimedia.org/w/index.php?diff=2403068.

Thu, Apr 16, 2:58 PM · cloud-services-team, Toolforge
taavi added a comment to T423610: Cannot contact service with port from buildservice.

That seems to indicate that you're trying to talk HTTPS to a service that's only listening on plaintext?

Thu, Apr 16, 2:57 PM · cloud-services-team, Toolforge
taavi edited projects for T423598: Migrate our use of osbpo away from mirrors.wikimedia.org, added: Cloud-VPS, tools-infrastructure-team; removed Cloud-Services, SRE.
Thu, Apr 16, 1:44 PM · Patch-For-Review, tools-infrastructure-team, Cloud-VPS
taavi added a project to T423544: jobs not getting loaded properly: Toolforge.
Thu, Apr 16, 7:06 AM · Patch-For-Review, Toolforge, tools-platform-team

Wed, Apr 15

taavi updated the task description for T417028: Reclaim public IPs from individual dumps distribution (clouddumps) hosts.
Wed, Apr 15, 10:46 AM · tools-infrastructure-team, Datasets-General-or-Unknown
taavi closed T422040: Migrate clouddumps https/rsync interfaces behind LVS as Resolved.
Wed, Apr 15, 10:44 AM · Traffic, Data-Services, tools-infrastructure-team, Datasets-General-or-Unknown
taavi closed T422040: Migrate clouddumps https/rsync interfaces behind LVS, a subtask of T417028: Reclaim public IPs from individual dumps distribution (clouddumps) hosts, as Resolved.
Wed, Apr 15, 10:44 AM · tools-infrastructure-team, Datasets-General-or-Unknown

Tue, Apr 14

taavi added a project to T423317: Allow user to add passkey/authenticator app without enabling 2FA: MediaWiki-extensions-OATHAuth.
Tue, Apr 14, 5:17 PM · Product Safety and Integrity, MediaWiki-extensions-OATHAuth

Mon, Apr 13

taavi closed T421386: cadvisor-reported Istio network usage is way too high, a subtask of T392356: Replace ingress-nginx before upstream EOL date, as Resolved.
Mon, Apr 13, 7:55 AM · Patch-For-Review, Toolforge, cloud-services-team (FY2025/2026-Q3-Q4)
taavi closed T421386: cadvisor-reported Istio network usage is way too high as Resolved.
Mon, Apr 13, 7:55 AM · Toolforge, cloud-services-team (FY2025/2026-Q3-Q4)
taavi renamed T423073: Migrate WMDE Tech Wishes gitlab.com repos to gitlab.wikimedia.org from Migrate WMDE Tech Wishes gitlab.org repos to gitlab.wikimedia.org to Migrate WMDE Tech Wishes gitlab.com repos to gitlab.wikimedia.org.
Mon, Apr 13, 7:44 AM · WMDE-TechWish-Maintenance

Sat, Apr 11

taavi merged T423010: PageImages extension isn't installed in Wikisources into T417538: Enable PageImages by default for Wikisource and Wikibooks.
Sat, Apr 11, 6:51 PM · PageImages, All-and-every-Wikisource, Wikimedia-Site-requests
taavi merged task T423010: PageImages extension isn't installed in Wikisources into T417538: Enable PageImages by default for Wikisource and Wikibooks.
Sat, Apr 11, 6:51 PM · All-and-every-Wikisource, PageImages
taavi changed the status of T422538: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error from Resolved to Invalid.

Filed T423005 for making the error message better, and marking this as invalid since nothing was actually changed about the infrastructure.

Sat, Apr 11, 5:10 PM · cloud-services-team, Toolforge
taavi created T423005: webservice start should give a better error message when a conflicting job exists.
Sat, Apr 11, 5:10 PM · tools-platform-team, Toolforge
taavi created T422992: Bump LibUp to Node 24.
Sat, Apr 11, 11:21 AM · LibUp

Fri, Apr 10

taavi closed T386559: X-spam-score header missing on obvious spam delivered to multiple Mailman3 lists via HyperKitty web ui as Resolved.

The patch is merged, and I don't see the button in the interface anymore, so closing.

Fri, Apr 10, 2:57 PM · collaboration-services, SRE, Wikimedia-Mailing-lists
taavi closed T422925: Add basic alerting for Toolforge Elasticsearch service as Resolved.
Fri, Apr 10, 2:06 PM · cloud-services-team, Toolforge
taavi removed a project from T422929: Continuous job failed to start due to missing envvar specified in secrets specification: cloud-services-team.
Fri, Apr 10, 10:51 AM · tools-platform-team, Toolforge
taavi edited projects for T422801: Consider allowing cumin access to all Cloud VPS VMs, added: tools-platform-team; removed cloud-services-team.
Fri, Apr 10, 10:51 AM · tools-infrastructure-team, Cloud-VPS
taavi edited projects for T422830: Openstack uwsgi logging to '<frozen importlib._bootstrap>.log', added: tools-platform-team; removed cloud-services-team.
Fri, Apr 10, 10:50 AM · tools-infrastructure-team, Cloud-VPS
taavi added a comment to T422929: Continuous job failed to start due to missing envvar specified in secrets specification.

This seems to be a reoccurance of T365048 (and thus a real bug). Something's caused the pod to exit and restart (but without recreating the Pod API object), and since the Secret references are injected at Pod creation time only it's blocking the pod from starting back up. I can think of a few different fixes:

  • envvars-api could use a single Secret for all of the envvars
  • envvars-api could restart all pods referencing a Secret that's being removed
  • something could watch for pods that end up in this restarting problem state and manually delete them via the API.
Fri, Apr 10, 10:34 AM · tools-platform-team, Toolforge
taavi created T422925: Add basic alerting for Toolforge Elasticsearch service.
Fri, Apr 10, 10:07 AM · cloud-services-team, Toolforge
taavi added a comment to T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble.

I've updated https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images#Testing_locally_%28optional%29 to use the new images. Is the locales warning there still relevant or does that need removing/changing?

Fri, Apr 10, 10:02 AM · Toolforge, tools-platform-team, cloud-services-team (FY2025/2026-Q3-Q4), Patch-For-Review
taavi added a comment to T422046: [builds-api] expose supported versions.

I think this would be useful. We already advertise pack at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images#Testing_locally_(optional), and this would let me extend the bot keeping the pre-built image lists up to date to keep the image lists on that page updated as well.

Fri, Apr 10, 9:33 AM · Patch-For-Review, cloud-services-team, Toolforge
taavi added a project to T422916: CSP violations with known domains in the blocked-uri are not collected by csp-report: Tools.
Fri, Apr 10, 9:23 AM · Tools
taavi closed T422829: Toolforge HTML head links sometimes are issued as http://<tool>.toolforge:443 as Resolved.
Fri, Apr 10, 8:40 AM · Toolforge, cloud-services-team
taavi closed T422829: Toolforge HTML head links sometimes are issued as http://<tool>.toolforge:443, a subtask of T392356: Replace ingress-nginx before upstream EOL date, as Resolved.
Fri, Apr 10, 8:40 AM · Patch-For-Review, Toolforge, cloud-services-team (FY2025/2026-Q3-Q4)

Thu, Apr 9

taavi added a comment to T422509: Cloud init and unattended upgrades while bootstrapping Trixie VMs.

We are attempting to only get the puppet package from the wikimedia repo (this is set by cloud-init at creation time)

Thu, Apr 9, 7:01 PM · Cloud-VPS, cloud-services-team
taavi edited projects for T409727: [builds-api,harbor,image-config] Move pre-built images to harbor, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:33 PM · Toolforge, cloud-services-team (FY2025/2026-Q3-Q4), Patch-For-Review
taavi edited projects for T359804: [jobs-api] Refactor before webservice support, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:33 PM · Toolforge, cloud-services-team, Patch-For-Review, User-Raymond_Ndibe
taavi edited projects for T402568: [components-api] Queue builds when the build queue is full, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:32 PM · cloud-services-team, Toolforge, Patch-For-Review
taavi edited projects for T407477: [docs] update all readmes with the same deployment docs, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:32 PM · cloud-services-team, Toolforge, Patch-For-Review
taavi edited projects for T404157: [builds-api, maintain-harbor] fix build/image cleanup, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:32 PM · cloud-services-team, Toolforge, Patch-For-Review
taavi edited projects for T379047: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.32, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:32 PM · Toolforge, cloud-services-team (FY2025/2026-Q3-Q4)
taavi edited projects for T402764: [components-api] allow specifying `source_repo`+`ref` for the config, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:32 PM · Toolforge, Patch-For-Review, cloud-services-team
taavi edited projects for T397949: [docs] enable docs linter in one of the repos, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · Toolforge, cloud-services-team
taavi removed hashtags from Toolforge (Toolforge iteration 26): #toolforgecurrent, #tfcurrent.
Thu, Apr 9, 3:31 PM
taavi edited projects for T368600: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · Toolforge, cloud-services-team (FY2025/2026-Q3-Q4), Epic
taavi archived Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM
taavi edited projects for T359650: [jobs-api] Create storage layer, and save business models in persistent storage, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · Toolforge, tools-platform-team, User-Raymond_Ndibe
taavi edited projects for T420425: [Toolforge Sustainability Framework]Percentage scoring of framework subcategories, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · cloud-services-team, Toolforge
taavi edited projects for T420559: [Toolforge Sustainability Framework] Create an inventory of Toolforge actions, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · Toolforge, tools-platform-team
taavi edited projects for T194332: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · tools-platform-team, Toolforge, cloud-services-team (FY2025/2026-Q3-Q4), Goal, User-dcaro, Cloud-Services-Origin-Team, Cloud-Services-Worktype-Project, Cloud Services Proposals, Epic
taavi edited projects for T418326: add more logs tests to toolforge-deploy, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · Toolforge, cloud-services-team, Patch-For-Review
taavi edited projects for T401172: [jobs-api] make job status an enum, with clearly defined states, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · Toolforge, tools-platform-team, Patch-For-Review, User-Raymond_Ndibe
taavi edited projects for T348755: [jobs-api,webservice] Run webservices via the jobs framework, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · Toolforge, tools-platform-team, Patch-For-Review, cloud-services-team (FY2025/2026-Q3-Q4), User-Raymond_Ndibe, Epic
taavi edited projects for T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · Toolforge, tools-platform-team, cloud-services-team (FY2025/2026-Q3-Q4), Patch-For-Review
taavi edited projects for T388092: [jobs-api] allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:31 PM · Toolforge, tools-platform-team, Patch-For-Review, cloud-services-team (FY2025/2026-Q3-Q4), User-Raymond_Ndibe, Epic
taavi edited projects for T422184: [general] upgrade all python repos to python >=3.13, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:30 PM · Toolforge, tools-platform-team
taavi edited projects for T418528: [harbor,tools] Harbor object usage in S3 is steadily increasing, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:30 PM · Toolforge, tools-platform-team, Patch-For-Review
taavi edited projects for T415322: [jobs-api] Use the same images as webservice, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:30 PM · Toolforge, tools-platform-team, Patch-For-Review
taavi edited projects for T422753: [components-api] failing deployment 422 from jobs-api, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:30 PM · tools-platform-team, cloud-services-team, Toolforge
taavi edited projects for T392356: Replace ingress-nginx before upstream EOL date, added: Toolforge; removed Toolforge (Toolforge iteration 26).
Thu, Apr 9, 3:30 PM · Patch-For-Review, Toolforge, cloud-services-team (FY2025/2026-Q3-Q4)
taavi added a comment to T422509: Cloud init and unattended upgrades while bootstrapping Trixie VMs.

The base image is based on a trixie VM with our puppet classes already applied (that happens at build time). So shouldn't /that/ have already downgraded puppet in the base image?

Thu, Apr 9, 2:25 PM · Cloud-VPS, cloud-services-team
taavi triaged T422829: Toolforge HTML head links sometimes are issued as http://<tool>.toolforge:443 as High priority.
Thu, Apr 9, 2:17 PM · Toolforge, cloud-services-team
taavi added a parent task for T422829: Toolforge HTML head links sometimes are issued as http://<tool>.toolforge:443: T392356: Replace ingress-nginx before upstream EOL date.
Thu, Apr 9, 2:17 PM · Toolforge, cloud-services-team
taavi added a subtask for T392356: Replace ingress-nginx before upstream EOL date: T422829: Toolforge HTML head links sometimes are issued as http://<tool>.toolforge:443.
Thu, Apr 9, 2:17 PM · Patch-For-Review, Toolforge, cloud-services-team (FY2025/2026-Q3-Q4)
taavi claimed T422829: Toolforge HTML head links sometimes are issued as http://<tool>.toolforge:443.

This is an issue with our Istio configuration:

1taavi@tools-k8s-haproxy-8:~$ curl -H "X-Forwarded-Port: 443" -H "X-Forwarded-Proto: https" --connect-to ::tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud:30000 http://sal.toolforge.org 2>&1 | grep stylesheet
2 <link rel="stylesheet" type="text/css" href="http://sal.toolforge.org:443/assets/main.css">
3taavi@tools-k8s-haproxy-8:~$ curl -H "X-Forwarded-Port: 443" -H "X-Forwarded-Proto: https" --connect-to ::tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud:30002 http://sal.toolforge.org 2>&1 | grep stylesheet
4 <link rel="stylesheet" type="text/css" href="https://sal.toolforge.org/assets/main.css">

Thu, Apr 9, 1:37 PM · Toolforge, cloud-services-team
taavi created P90341 (An Untitled Masterwork).
Thu, Apr 9, 1:34 PM
taavi edited projects for T422817: GraphQL frontend challenges: evaluate better integrations with PAWS (Jupyter), added: PAWS; removed Cloud Services Proposals.
Thu, Apr 9, 12:44 PM · PAWS, Wikidata, Data-Engineering-Jupyter, Data-Engineering, cloud-services-team, Wikibase GraphQL
taavi changed the visibility for T422779: clouddb1017 reports a new database created.
Thu, Apr 9, 10:09 AM · SecTeam-Processed, cloud-services-team, Data-Services, DBA
taavi closed T422672: Stop serving dumps.wikimedia.org port 80 as Resolved.
Thu, Apr 9, 7:25 AM · tools-infrastructure-team, Data-Services, Datasets-General-or-Unknown

Wed, Apr 8

taavi added a comment to T422509: Cloud init and unattended upgrades while bootstrapping Trixie VMs.

This is not an unattended-upgrades problem. It instead seems to be a problem with how the Puppet agent packages are installed - Trixie by default includes Puppet 8, but our codebase is not yet fully compatible with Puppet 8 (and particularly its removal of deprecated facts), so we need to pull in the Puppet 7 agent packages from a component on apt.wm.o.

Wed, Apr 8, 8:58 PM · Cloud-VPS, cloud-services-team
taavi added a comment to T400917: [jobs-api] Allow customizing time to request Loki logs for.

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_jobs#Job_logs needs updating for these changes.

Wed, Apr 8, 4:47 PM · tools-platform-team, Toolforge (Toolforge iteration 26), cloud-services-team (FY2025/2026-Q3-Q4), Patch-For-Review
taavi updated the task description for T417028: Reclaim public IPs from individual dumps distribution (clouddumps) hosts.
Wed, Apr 8, 4:31 PM · tools-infrastructure-team, Datasets-General-or-Unknown
taavi added a comment to T422672: Stop serving dumps.wikimedia.org port 80.

@taavi is this done by tools platform or tools infra?

Wed, Apr 8, 4:31 PM · tools-infrastructure-team, Data-Services, Datasets-General-or-Unknown
taavi claimed T422672: Stop serving dumps.wikimedia.org port 80.
Wed, Apr 8, 4:30 PM · tools-infrastructure-team, Data-Services, Datasets-General-or-Unknown
taavi edited projects for T422672: Stop serving dumps.wikimedia.org port 80, added: tools-infrastructure-team; removed tools-platform-team.
Wed, Apr 8, 4:30 PM · tools-infrastructure-team, Data-Services, Datasets-General-or-Unknown
taavi added a comment to T422452: clis: only create tag on merge of the release patch.

Oppose. Pushing a tag should be the action that triggers the release pipeline.

Wed, Apr 8, 2:47 PM · Toolforge, tools-platform-team
taavi created T422672: Stop serving dumps.wikimedia.org port 80.
Wed, Apr 8, 2:35 PM · tools-infrastructure-team, Data-Services, Datasets-General-or-Unknown
taavi closed T422287: Toolforge Prometheus instance is unstable as Resolved.

Declaring this a success, Prometheus has stayed up without an OOM for a significantly longer time than it did before:

image.png (851×1 px, 101 KB)

Wed, Apr 8, 11:49 AM · tools-infrastructure-team, cloud-services-team, Toolforge
taavi added a comment to T421719: Data persistance: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets.

A wrinkle here is that ferm doesn't get reloaded on the other swift nodes (presumably because the config for ferm hasn't actually changed, because the hostname of the node is unchanged), so you have to do that by cumin-hand before the reimaged node works again.

Wed, Apr 8, 11:16 AM · DBA, Ceph, SRE-swift-storage, User-Eevans, Data-Persistence
taavi created P90323 (An Untitled Masterwork).
Wed, Apr 8, 7:47 AM

Tue, Apr 7

taavi added a comment to T422559: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance.

Seems like mx-in*.wikimedia.org do not like these emails for whatever reason:

2026-04-07 19:39:51 1wACH9-00BqE5-1C ** phabricator-no-reply@wmcloud.org R=dnslookup_unsigned T=remote_smtp_unsigned H=mx-in2001.wikimedia.org [208.80.153.75] I=[172.16.2.248] X=TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256 CV=yes DN="CN=mx-in1001.wikimedia.org": SMTP error from remote mail server after RCPT TO:<phabricator-no-reply@wmcloud.org>: 550 5.1.1 <phabricator-no-reply@wmcloud.org>: Recipient address rejected: User unknown in relay recipient table DT=0s
Tue, Apr 7, 9:21 PM · Infrastructure-Foundations, Mail, collaboration-services, VPS-project-Phabricator
taavi updated the task description for T422287: Toolforge Prometheus instance is unstable.
Tue, Apr 7, 1:10 PM · tools-infrastructure-team, Toolforge, cloud-services-team
taavi created T422468: Gerrit load balancer services still in lvs_setup.
Tue, Apr 7, 8:47 AM · collaboration-services
taavi added a comment to T410883: Support HTTP QUERY method as standard alternative to Promise-Non-Write-API-Action header.

This should probably be split in two tasks, one for support in MediaWiki and an another for the required changes in the Wikimedia CDN?

Tue, Apr 7, 8:33 AM · MediaWiki-Action-API, MW-Interfaces-Team
taavi claimed T422040: Migrate clouddumps https/rsync interfaces behind LVS.

Per Traffic this should be a high-traffic2 service. I have allocated a VIP, namely

dumps-lb.eqiad.wikimedia.org has address 208.80.154.242
dumps-lb.eqiad.wikimedia.org has IPv6 address 2620:0:861:ed1a::3:242
Tue, Apr 7, 8:32 AM · Traffic, Data-Services, tools-infrastructure-team, Datasets-General-or-Unknown
taavi closed T422042: Dumps access log analytics should support multiple active hosts as Resolved.
Tue, Apr 7, 8:31 AM · Data-Platform-SRE, Data-Services, tools-infrastructure-team, Datasets-General-or-Unknown
taavi closed T422042: Dumps access log analytics should support multiple active hosts, a subtask of T422040: Migrate clouddumps https/rsync interfaces behind LVS, as Resolved.
Tue, Apr 7, 8:31 AM · Traffic, Data-Services, tools-infrastructure-team, Datasets-General-or-Unknown