Page MenuHomePhabricator

Stashbot (Stashbot)
Bot that likes to stash things away for laterBot

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Feb 10 2016, 12:29 AM (258 w, 12 h)
Roles
Bot
Availability
Available
IRC Nick
stashbot
LDAP User
Stashbot
MediaWiki User
Unknown

Recent Activity

Today

Stashbot added a comment to T272428: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist.

Mentioned in SAL (#wikimedia-operations) [2021-01-20T13:20:45Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.27/extensions/Translate/: 20decbd5cc3de0af655b9419cf69fc442ab056a4: Add flag to toggle the usage of the group synchronization cache (T272428; T182433) (duration: 01m 10s)

Wed, Jan 20, 1:21 PM · Security-Team, Language-Team (Language-2021-January-March), MediaWiki-extensions-Translate, DBA, User-brennen, Wikimedia-production-error
Stashbot added a comment to T182433: Implement a stronger synchronization in RepoNG and Translate.

Mentioned in SAL (#wikimedia-operations) [2021-01-20T13:20:45Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.27/extensions/Translate/: 20decbd5cc3de0af655b9419cf69fc442ab056a4: Add flag to toggle the usage of the group synchronization cache (T272428; T182433) (duration: 01m 10s)

Wed, Jan 20, 1:21 PM · MW-1.36-notes (1.36.0-wmf.27; 2021-01-19), Patch-For-Review, Language-Team (Language-2021-January-March), Developer Productivity, MediaWiki-extensions-Translate, translatewiki.net
Stashbot added a comment to T267767: Drop default of revactor_timestamp.

Mentioned in SAL (#wikimedia-operations) [2021-01-20T12:49:51Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2087.codfw.wmnet with reason: Schema change T267767

Wed, Jan 20, 12:49 PM · Blocked-on-schema-change, DBA
Stashbot added a comment to T267767: Drop default of revactor_timestamp.

Mentioned in SAL (#wikimedia-operations) [2021-01-20T12:49:47Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2087.codfw.wmnet with reason: Schema change T267767

Wed, Jan 20, 12:49 PM · Blocked-on-schema-change, DBA
Stashbot added a comment to T272008: Move wikireplicas under the new sanitarium hosts (db1154, db1155).

Mentioned in SAL (#wikimedia-operations) [2021-01-20T10:34:50Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1079 to stop replication T272008', diff saved to https://phabricator.wikimedia.org/P13842 and previous config saved to /var/cache/conftool/dbconfig/20210120-103449-marostegui.json

Wed, Jan 20, 10:35 AM · cloud-services-team (Kanban), DBA
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T10:05:10Z] <dcaro> Everything looks ok, created a new vm with a volume in ceph without issues, and on warnings/errors on ceph status, closing (T272303)

Wed, Jan 20, 10:05 AM · cloud-services-team (Kanban)
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:55:08Z] <dcaro> Eqiad ceph cluster uprgaded, doing sanity checks (T272303)

Wed, Jan 20, 9:55 AM · cloud-services-team (Kanban)
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:46:30Z] <dcaro> 75% of the eqiad cluster upgraded... continuing (T272303)

Wed, Jan 20, 9:46 AM · cloud-services-team (Kanban)
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:37:55Z] <dcaro> 25% of the eqiad cluster upgraded... continuing (T272303)

Wed, Jan 20, 9:37 AM · cloud-services-team (Kanban)
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:24:39Z] <dcaro> Mgr daemons upgraded and running, upgrading osd daemons on servers cloudcephosd1*, this make take a bit longer (T272303)

Wed, Jan 20, 9:24 AM · cloud-services-team (Kanban)
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:22:13Z] <dcaro> Mon daemons upgraded and running, upgrading mgr daemons on servers cloudcephmon1* (T272303)

Wed, Jan 20, 9:22 AM · cloud-services-team (Kanban)
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:16:02Z] <dcaro> Starting eqiad ceph upgrade, upgrading the mon servers cloudcephmon1* (T272303)

Wed, Jan 20, 9:16 AM · cloud-services-team (Kanban)
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:01:13Z] <dcaro> Will start the ceph upgrade in 15 min, no downtime nor performance impact is expected (T272303)

Wed, Jan 20, 9:01 AM · cloud-services-team (Kanban)
Stashbot added a comment to T270309: Instrument banner module on newcomer homepage.

Mentioned in SAL (#wikimedia-operations) [2021-01-20T00:43:29Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:656284|Update /analytics/legacy/homepagemodule/ schema version to 1.1.0 (T270309)]] (duration: 01m 03s)

Wed, Jan 20, 12:43 AM · MW-1.36-notes (1.36.0-wmf.25; 2021-01-05), GrowthExperiments-Homepage, Growth-Team (Current Sprint)
Stashbot added a comment to T261408: Add a link engineering: Maintenance script for retrieving, caching, and updating search index.

Mentioned in SAL (#wikimedia-operations) [2021-01-20T00:30:20Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:655863|(no-op) GrowthExperiments: Disable link recommendations (T261408)]] (duration: 01m 05s)

Wed, Jan 20, 12:30 AM · MW-1.36-notes (1.36.0-wmf.27; 2021-01-19), Patch-For-Review, Growth-Team (Current Sprint), Growth-Structured-Tasks
Stashbot added a comment to T179696: Homepage for https://docker-registry.wikimedia.org.

Mentioned in SAL (#wikimedia-operations) [2021-01-20T00:09:25Z] <legoktm> uploaded docker-report 0.0.4-1~deb9u1 to stretch-wikimedia (T179696)

Wed, Jan 20, 12:09 AM · serviceops, Patch-For-Review, SRE, MediaWiki-Containers

Yesterday

Stashbot added a comment to T272247: 2021-01-17: tools NFS share cleanup.

Mentioned in SAL (#wikimedia-cloud) [2021-01-19T23:32:30Z] <bstorm> truncated 34GB error log file that was full of warnings like "Only variables should be passed by reference in /data/project/geohack/public_html/geohack.php on line 192" T272247

Tue, Jan 19, 11:32 PM · cloud-services-team (Kanban)
Stashbot added a comment to T272247: 2021-01-17: tools NFS share cleanup.

Mentioned in SAL (#wikimedia-cloud) [2021-01-19T23:30:37Z] <bstorm> truncated 36GB mybot.out file T272247

Tue, Jan 19, 11:30 PM · cloud-services-team (Kanban)
Stashbot added a comment to T272247: 2021-01-17: tools NFS share cleanup.

Mentioned in SAL (#wikimedia-cloud) [2021-01-19T22:57:43Z] <bstorm> truncated 75GB error log /data/project/robokobot/virgule.err T272247

Tue, Jan 19, 10:57 PM · cloud-services-team (Kanban)
Stashbot added a comment to T272247: 2021-01-17: tools NFS share cleanup.

Mentioned in SAL (#wikimedia-cloud) [2021-01-19T22:48:30Z] <bstorm> truncated 100GB error log /data/project/magnus-toolserver/error.log T272247

Tue, Jan 19, 10:48 PM · cloud-services-team (Kanban)
Stashbot added a comment to T272247: 2021-01-17: tools NFS share cleanup.

Mentioned in SAL (#wikimedia-cloud) [2021-01-19T22:43:03Z] <bstorm> truncated 107GB log '/data/project/meetbot/logs/messages.log' T272247

Tue, Jan 19, 10:43 PM · cloud-services-team (Kanban)
Stashbot added a comment to T272247: 2021-01-17: tools NFS share cleanup.

Mentioned in SAL (#wikimedia-cloud) [2021-01-19T22:34:50Z] <bstorm> truncating 194 GB error log '/data/project/mix-n-match/mnm-microsync.err' T272247

Tue, Jan 19, 10:34 PM · cloud-services-team (Kanban)
Stashbot added a comment to T255973: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T21:46:20Z] <ottomata> wiping kafka-test cluster data and starting from scratch - T255973

Tue, Jan 19, 9:46 PM · Analytics-Clusters
Stashbot added a comment to T269713: Run the MigrateAflFilter script for AbuseFilter.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T21:00:36Z] <Urbanecm> Start of foreachwikiindblist group2 extensions/AbuseFilter/maintenance/MigrateAflFilter.php --batch-size=1000 (T269713)

Tue, Jan 19, 9:00 PM · User-Urbanecm, Wikimedia-maintenance-script-run, AbuseFilter
Stashbot added a comment to T271341: 1.36.0-wmf.27 deployment blockers.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T17:59:03Z] <brennen> starting deploy-promote to testwikis for 1.36.0-wmf.27 (T271341)

Tue, Jan 19, 5:59 PM · User-brennen, Patch-For-Review, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release, Train Deployments
Stashbot added a comment to T269713: Run the MigrateAflFilter script for AbuseFilter.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T17:30:09Z] <Urbanecm> Start of foreachwikiindblist group1 extensions/AbuseFilter/maintenance/MigrateAflFilter.php --batch-size=1000 (T269713)

Tue, Jan 19, 5:30 PM · User-Urbanecm, Wikimedia-maintenance-script-run, AbuseFilter
Stashbot added a comment to T271761: Create initial Function Orchestrator service.

Mentioned in SAL (#wikimedia-releng) [2021-01-19T17:11:06Z] <James_F> Zuul: [mediawiki/services/function-orchestrator] Add pipeline CI T271761

Tue, Jan 19, 5:11 PM · Patch-For-Review, Abstract Wikipedia (Phase δ)
Stashbot added a comment to T269713: Run the MigrateAflFilter script for AbuseFilter.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T17:08:48Z] <Urbanecm> Run extensions/AbuseFilter/maintenance/MigrateAflFilter.php for all group0 wikis (T269713)

Tue, Jan 19, 5:08 PM · User-Urbanecm, Wikimedia-maintenance-script-run, AbuseFilter
Stashbot added a comment to T269713: Run the MigrateAflFilter script for AbuseFilter.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T17:06:20Z] <Urbanecm> mwscript extensions/AbuseFilter/maintenance/MigrateAflFilter.php --wiki=test2wiki --batch-size=1000 # T269713

Tue, Jan 19, 5:06 PM · User-Urbanecm, Wikimedia-maintenance-script-run, AbuseFilter
Stashbot added a comment to T269713: Run the MigrateAflFilter script for AbuseFilter.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T17:04:40Z] <Urbanecm> mwscript extensions/AbuseFilter/maintenance/MigrateAflFilter.php --wiki=testwiki --batch-size=1000 # T269713

Tue, Jan 19, 5:04 PM · User-Urbanecm, Wikimedia-maintenance-script-run, AbuseFilter
Stashbot added a comment to T271341: 1.36.0-wmf.27 deployment blockers.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T16:46:29Z] <brennen> 1.36.0-wmf.27 was branched at fbb516d8e33924c6cb66c93bb6d42907558c31f3 for T271341

Tue, Jan 19, 4:47 PM · User-brennen, Patch-For-Review, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release, Train Deployments
Stashbot added a comment to T269713: Run the MigrateAflFilter script for AbuseFilter.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T15:15:44Z] <Urbanecm> Run foreachwikiindblist closed extensions/AbuseFilter/maintenance/MigrateAflFilter.php (T269713)

Tue, Jan 19, 3:15 PM · User-Urbanecm, Wikimedia-maintenance-script-run, AbuseFilter
Stashbot added a comment to T271261: Prepare and check storage layer for trwikivoyage.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T14:19:32Z] <marostegui> Sanitize trwikivoyage on db2094:3315, db1124:3315, db1154:3315 T271261

Tue, Jan 19, 2:19 PM · cloud-services-team (Kanban), Data-Services, DBA
Stashbot added a comment to T271264: Add Wikidata support for trwikivoyage.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T14:08:21Z] <Amir1> end of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T271264)

Tue, Jan 19, 2:08 PM · User-Ladsgroup, Wikidata, Wikidata-Campsite
Stashbot added a comment to T271264: Add Wikidata support for trwikivoyage.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T13:49:47Z] <Amir1> start of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T271264)

Tue, Jan 19, 1:49 PM · User-Ladsgroup, Wikidata, Wikidata-Campsite
Stashbot added a comment to T271260: Create Wikivoyage Turkish.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T13:36:23Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Creating trwikivoyage (T271260) (duration: 00m 55s)

Tue, Jan 19, 1:36 PM · Patch-For-Review, Turkish-Sites, MW-1.36-notes (1.36.0-wmf.26; 2021-01-12), Wiki-Setup (Create), User-Urbanecm
Stashbot added a comment to T271260: Create Wikivoyage Turkish.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T13:35:07Z] <urbanecm@deploy1001> Synchronized static/images/project-logos/: Creating trwikivoyage (T271260) (duration: 00m 55s)

Tue, Jan 19, 1:35 PM · Patch-For-Review, Turkish-Sites, MW-1.36-notes (1.36.0-wmf.26; 2021-01-12), Wiki-Setup (Create), User-Urbanecm
Stashbot added a comment to T271260: Create Wikivoyage Turkish.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T13:34:05Z] <urbanecm@deploy1001> rebuilt and synchronized wikiversions files: Creating trwikivoyage (T271260)

Tue, Jan 19, 1:34 PM · Patch-For-Review, Turkish-Sites, MW-1.36-notes (1.36.0-wmf.26; 2021-01-12), Wiki-Setup (Create), User-Urbanecm
Stashbot added a comment to T271260: Create Wikivoyage Turkish.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T13:32:36Z] <urbanecm@deploy1001> Synchronized dblists: Creating trwikivoyage (T271260) (duration: 00m 55s)

Tue, Jan 19, 1:32 PM · Patch-For-Review, Turkish-Sites, MW-1.36-notes (1.36.0-wmf.26; 2021-01-12), Wiki-Setup (Create), User-Urbanecm
Stashbot added a comment to T271260: Create Wikivoyage Turkish.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T13:31:37Z] <urbanecm@deploy1001> Synchronized wmf-config/db-codfw.php: Creating trwikivoyage (T271260) (duration: 00m 55s)

Tue, Jan 19, 1:31 PM · Patch-For-Review, Turkish-Sites, MW-1.36-notes (1.36.0-wmf.26; 2021-01-12), Wiki-Setup (Create), User-Urbanecm
Stashbot added a comment to T271260: Create Wikivoyage Turkish.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T13:30:36Z] <urbanecm@deploy1001> Synchronized wmf-config/db-eqiad.php: Creating trwikivoyage (T271260) (duration: 00m 56s)

Tue, Jan 19, 1:30 PM · Patch-For-Review, Turkish-Sites, MW-1.36-notes (1.36.0-wmf.26; 2021-01-12), Wiki-Setup (Create), User-Urbanecm
Stashbot added a comment to T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T12:44:57Z] <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:656842| Bumping portals to master (T128546)]] (duration: 00m 56s)

Tue, Jan 19, 12:45 PM · periodic-update, Patch-For-Review, Wikimedia-Portals, Discovery, Discovery-Portal-Sprint
Stashbot added a comment to T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T12:44:00Z] <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:656842| Bumping portals to master (T128546)]] (duration: 00m 56s)

Tue, Jan 19, 12:44 PM · periodic-update, Patch-For-Review, Wikimedia-Portals, Discovery, Discovery-Portal-Sprint
Stashbot added a comment to T269712: Migrate afl_filter to afl_filter_id and afl_global.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T12:32:16Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 338c0f9fe32512266c3030f7c9b7f8804ed30432: wgAbuseFilterAflFilterMigrationStage: Make WRITE_BOTH everywhere (T269712) (duration: 00m 56s)

Tue, Jan 19, 12:32 PM · Patch-For-Review, AbuseFilter
Stashbot added a comment to T270974: Temporarily change logo on the Finnish Wikipedia.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T12:06:56Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 6a4cbe662655edaa4f6c36e69877766a6a48d828: Revert "Switch fiwiki to their 500k temporary logo!" (T270974) (duration: 00m 56s)

Tue, Jan 19, 12:07 PM · User-Majavah, Release-Engineering-Team, Wikimedia-Site-requests
Stashbot added a comment to T271058: cloudnet1004/cloudnet1003: network hiccups because broadcom driver/firmware problem.

Mentioned in SAL (#wikimedia-cloud) [2021-01-19T10:17:24Z] <arturo> icinga-downtime cloudnet1004 for 1 week (T271058)

Tue, Jan 19, 10:17 AM · ops-eqiad, SRE, cloud-services-team (Kanban)
Stashbot added a comment to T272008: Move wikireplicas under the new sanitarium hosts (db1154, db1155).

Mentioned in SAL (#wikimedia-operations) [2021-01-19T09:01:00Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1112 to stop replication T272008', diff saved to https://phabricator.wikimedia.org/P13828 and previous config saved to /var/cache/conftool/dbconfig/20210119-090100-marostegui.json

Tue, Jan 19, 9:01 AM · cloud-services-team (Kanban), DBA
Stashbot added a comment to T272008: Move wikireplicas under the new sanitarium hosts (db1154, db1155).

Mentioned in SAL (#wikimedia-operations) [2021-01-19T08:58:57Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1078 to stop replication T272008', diff saved to https://phabricator.wikimedia.org/P13826 and previous config saved to /var/cache/conftool/dbconfig/20210119-085856-marostegui.json

Tue, Jan 19, 8:59 AM · cloud-services-team (Kanban), DBA
Stashbot added a comment to T269337: Add ms-be20[58-61] to swift.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T08:54:23Z] <godog> swift codfw-prod: more weight to ms-be20[58-61] - T269337

Tue, Jan 19, 8:54 AM · User-fgiunchedi, SRE-swift-storage
Stashbot added a comment to T272008: Move wikireplicas under the new sanitarium hosts (db1154, db1155).

Mentioned in SAL (#wikimedia-operations) [2021-01-19T07:02:47Z] <marostegui> Stop MySQL on db1082 T272008

Tue, Jan 19, 7:02 AM · cloud-services-team (Kanban), DBA
Stashbot added a comment to T272008: Move wikireplicas under the new sanitarium hosts (db1154, db1155).

Mentioned in SAL (#wikimedia-operations) [2021-01-19T06:57:49Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1082 to stop replication T272008', diff saved to https://phabricator.wikimedia.org/P13821 and previous config saved to /var/cache/conftool/dbconfig/20210119-065748-marostegui.json

Tue, Jan 19, 6:58 AM · cloud-services-team (Kanban), DBA
Stashbot added a comment to T272215: High latency on appservers.

Mentioned in SAL (#wikimedia-operations) [2021-01-19T04:39:34Z] <Krinkle> unlocked per ttps://phabricator.wikimedia.org/T272215#6755025

Tue, Jan 19, 4:39 AM · serviceops, SRE, Wikimedia-Incident
Stashbot added a comment to T272317: bodh: Implement backend.

Mentioned in SAL (#wikimedia-cloud) [2021-01-19T04:14:07Z] <Jayprakash12345> Deploying backend (T272317)

Tue, Jan 19, 4:14 AM · Tool-bodh

Mon, Jan 18

Stashbot added a comment to T272320: bodh: Add backed to CI/CD.

Mentioned in SAL (#wikimedia-releng) [2021-01-18T23:38:58Z] <James_F> Zuul: [labs/tools/bodh-backend] Provide CI with tox-docker T272320

Mon, Jan 18, 11:39 PM · Tool-bodh
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-18T16:00:34Z] <dcaro> Codfw1 ceph cluster uprgaded, will wait until tomorrow to see if there's any instability, but everything looks fine (T272303)

Mon, Jan 18, 4:00 PM · cloud-services-team (Kanban)
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-18T15:38:04Z] <dcaro> Upgraded mgr sevices on codfw ceph cluster, starting with osd ones (T272303)

Mon, Jan 18, 3:38 PM · cloud-services-team (Kanban)
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-18T15:35:00Z] <dcaro> Upgraded mon sevices on codfw ceph cluster, starting with mgr ones (T272303)

Mon, Jan 18, 3:35 PM · cloud-services-team (Kanban)
Stashbot added a comment to T272303: [ceph] Upgrade to 14.2.16 from 14.2.5.

Mentioned in SAL (#wikimedia-cloud) [2021-01-18T15:21:42Z] <dcaro> Starting upgrade of ceph mon nodes on codfw (T272303)

Mon, Jan 18, 3:21 PM · cloud-services-team (Kanban)
Stashbot added a comment to T272296: ceph: nautilus: decide on pending package upgrades.

Mentioned in SAL (#wikimedia-operations) [2021-01-18T14:30:26Z] <arturo> updating packages in buster-wikimedia/thirdparty/ceph-nautilus-buster (T272296)

Mon, Jan 18, 2:30 PM · cloud-services-team (Kanban)
Stashbot added a comment to T268457: Investigate possible optimizer regression on 10.4.17 with DELETE statements.

Mentioned in SAL (#wikimedia-operations) [2021-01-18T13:12:04Z] <marostegui> Upgrade db2071 to 10.4.17 - T268457

Mon, Jan 18, 1:12 PM · mariadb-optimizer-bug, DBA
Stashbot added a comment to T272008: Move wikireplicas under the new sanitarium hosts (db1154, db1155).

Mentioned in SAL (#wikimedia-operations) [2021-01-18T09:25:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 to stop replication T272008', diff saved to https://phabricator.wikimedia.org/P13795 and previous config saved to /var/cache/conftool/dbconfig/20210118-092546-marostegui.json

Mon, Jan 18, 9:26 AM · cloud-services-team (Kanban), DBA
Stashbot added a comment to T272008: Move wikireplicas under the new sanitarium hosts (db1154, db1155).

Mentioned in SAL (#wikimedia-operations) [2021-01-18T08:17:40Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1106 to stop replication, place db1105:3311 temporarily in vslow T272008', diff saved to https://phabricator.wikimedia.org/P13787 and previous config saved to /var/cache/conftool/dbconfig/20210118-081740-marostegui.json

Mon, Jan 18, 8:17 AM · cloud-services-team (Kanban), DBA
Stashbot added a comment to T272082: Reflected XSS on archiva.wikimedia.org (due to Jetty, likely CVE-2019-10241).

Mentioned in SAL (#wikimedia-operations) [2021-01-18T08:13:52Z] <elukey> clean up old archiva debs and upload 2.2.4-3 to buster-wikimedia - T272082

Mon, Jan 18, 8:13 AM · Patch-For-Review, Vuln-XSS, Security, Security-Team

Sun, Jan 17

Stashbot added a comment to T272247: 2021-01-17: tools NFS share cleanup.

Mentioned in SAL (#wikimedia-cloud) [2021-01-17T16:53:53Z] <arturo> icinga downtime labstore1004 /srv/tools space check for 3 days (T272247)

Sun, Jan 17, 4:53 PM · cloud-services-team (Kanban)
Stashbot added a comment to T248925: Make MediaWiki compatible with PHP 8.

Mentioned in SAL (#wikimedia-releng) [2021-01-17T03:44:33Z] <James_F> Zuul: [mediawiki/core] Add composer (not vendor) experimental PHP 8.0 job T248925

Sun, Jan 17, 3:44 AM · MW-1.36-notes (1.36.0-wmf.25; 2021-01-05), Patch-For-Review, MW-1.35-notes, PHP 8.0 support, MediaWiki-General, MediaWiki-Stakeholders-Group

Sat, Jan 16

Stashbot added a comment to T271428: Bump php-ast to 1.0.10 for all phan jobs.

Mentioned in SAL (#wikimedia-releng) [2021-01-16T23:24:19Z] <James_F> Docker: Building cascade of new php-ast image T271428

Sat, Jan 16, 11:24 PM · Patch-For-Review, Continuous-Integration-Config, Release-Engineering-Team, phan-taint-check-plugin, phan
Stashbot added a comment to T272215: High latency on appservers.

Mentioned in SAL (#wikimedia-operations) [2021-01-16T12:18:53Z] <elukey> elukey@cumin1001:~$ sudo cumin 'A:mw-app-canary and A:mw-eqiad' 'run-puppet-agent' -b 10 - T272215

Sat, Jan 16, 12:19 PM · serviceops, SRE, Wikimedia-Incident
Stashbot added a comment to T272215: High latency on appservers.

Mentioned in SAL (#wikimedia-operations) [2021-01-16T12:10:13Z] <elukey> 'elukey@cumin1001:~$ sudo cumin 'A:mw-eqiad' 'run-puppet-agent' -b 10' T272215)

Sat, Jan 16, 12:10 PM · serviceops, SRE, Wikimedia-Incident

Fri, Jan 15

Stashbot added a comment to T272075: Enable anniversary logo for fr.wikipedia - 20th birthday.

Mentioned in SAL (#wikimedia-operations) [2021-01-15T20:39:14Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 66e6be391ecfde7ca0604146ab978987ce472b5c: Set anniversary logo for frwiki (3/3; T272075) (duration: 00m 55s)

Fri, Jan 15, 8:39 PM · User-Urbanecm, Wikimedia-Site-requests
Stashbot added a comment to T272075: Enable anniversary logo for fr.wikipedia - 20th birthday.

Mentioned in SAL (#wikimedia-operations) [2021-01-15T20:37:56Z] <urbanecm@deploy1001> Synchronized static/images/mobile/copyright/wikipedia-tagline-fr-20.svg: 66e6be391ecfde7ca0604146ab978987ce472b5c: Set anniversary logo for frwiki (2/3; T272075) (duration: 00m 55s)

Fri, Jan 15, 8:38 PM · User-Urbanecm, Wikimedia-Site-requests
Stashbot added a comment to T272075: Enable anniversary logo for fr.wikipedia - 20th birthday.

Mentioned in SAL (#wikimedia-operations) [2021-01-15T20:36:41Z] <urbanecm@deploy1001> Synchronized static/images/mobile/copyright/wikipedia-fr-20.svg: 66e6be391ecfde7ca0604146ab978987ce472b5c: Set anniversary logo for frwiki (1/3; T272075) (duration: 00m 58s)

Fri, Jan 15, 8:36 PM · User-Urbanecm, Wikimedia-Site-requests
Stashbot added a comment to T272075: Enable anniversary logo for fr.wikipedia - 20th birthday.

Mentioned in SAL (#wikimedia-operations) [2021-01-15T20:21:50Z] <urbanecm@deploy1001> Synchronized static/images/mobile/copyright/wikipedia-fr-20.svg: 66e6be391ecfde7ca0604146ab978987ce472b5c: Set anniversary logo for frwiki (1/3; T272075) (duration: 01m 54s)

Fri, Jan 15, 8:21 PM · User-Urbanecm, Wikimedia-Site-requests
Stashbot added a comment to T272159: Jenkins Matrix builds containing a slash are not reachable.

Mentioned in SAL (#wikimedia-operations) [2021-01-15T17:17:44Z] <legoktm> legoktm@contint2001:~$ sudo systemctl reload apache2 # for T272159

Fri, Jan 15, 5:17 PM · Jenkins, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure
Stashbot added a comment to T272161: Toolforge: reFill has not processed two days.

Mentioned in SAL (#wikimedia-cloud) [2021-01-15T16:40:54Z] <bstorm> restarted the pod because it appears to be wedged after the network outage the other day T272161

Fri, Jan 15, 4:40 PM · Tool-refill
Stashbot added a comment to T272127: 2021-01-15: ** PROBLEM alert - labstore1004/Ensure mysql credential creation for tools users is running is CRITICAL **.

Mentioned in SAL (#wikimedia-operations) [2021-01-15T16:17:22Z] <bstorm> canceled downtime for maintain-dbusers on labstore1004 T272127

Fri, Jan 15, 4:17 PM · cloud-services-team (Kanban)
Stashbot added a comment to T272125: Memory errors on clouddb1019.

Mentioned in SAL (#wikimedia-cloud) [2021-01-15T13:41:57Z] <arturo> icinga downtime labstore1004 maintain-dbuser alert until 2021-01-19 (T272125)

Fri, Jan 15, 1:42 PM · DBA, SRE, ops-eqiad
Stashbot added a comment to T272103: [regression - wmf.26] frwiki Homepage SE module has 'cirrussearch-query-too-long' for default filters.

Mentioned in SAL (#wikimedia-operations) [2021-01-15T09:58:24Z] <reedy@deploy1001> Synchronized php-1.36.0-wmf.26/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php: T272103 (duration: 00m 57s)

Fri, Jan 15, 9:58 AM · MW-1.36-notes (1.36.0-wmf.27; 2021-01-19), Patch-For-Review, Growth-Team (Current Sprint), Regression
Stashbot added a comment to T272125: Memory errors on clouddb1019.

Mentioned in SAL (#wikimedia-cloud) [2021-01-15T09:47:36Z] <arturo> labstore1004 maintain-dbusers affected by T272127 and T272125

Fri, Jan 15, 9:48 AM · DBA, SRE, ops-eqiad
Stashbot added a comment to T272127: 2021-01-15: ** PROBLEM alert - labstore1004/Ensure mysql credential creation for tools users is running is CRITICAL **.

Mentioned in SAL (#wikimedia-cloud) [2021-01-15T09:47:36Z] <arturo> labstore1004 maintain-dbusers affected by T272127 and T272125

Fri, Jan 15, 9:47 AM · cloud-services-team (Kanban)
Stashbot added a comment to T269337: Add ms-be20[58-61] to swift.

Mentioned in SAL (#wikimedia-operations) [2021-01-15T09:18:59Z] <godog> swift codfw-prod: more weight to ms-be20[58-61] - T269337

Fri, Jan 15, 9:19 AM · User-fgiunchedi, SRE-swift-storage
Stashbot added a comment to T257324: Consolidate edge bastion server into ganeti.

Mentioned in SAL (#wikimedia-operations) [2021-01-15T09:07:58Z] <moritzm> installing bast5002 T257324

Fri, Jan 15, 9:08 AM · Patch-For-Review, Traffic, SRE
Stashbot added a comment to T257324: Consolidate edge bastion server into ganeti.

Mentioned in SAL (#wikimedia-operations) [2021-01-15T08:45:31Z] <moritzm> installing bast4003 T257324

Fri, Jan 15, 8:45 AM · Patch-For-Review, Traffic, SRE
Stashbot added a comment to T271527: [ceph] Disable write cache on all osds.

Mentioned in SAL (#wikimedia-cloud) [2021-01-15T08:19:52Z] <dcaro> Merging the patch to disable write caches on ceph osds (T271527)

Fri, Jan 15, 8:19 AM · cloud-services-team (Kanban)

Thu, Jan 14

Stashbot added a comment to T272094: Change EnWiki logo for Wikipedia's 20th anniversary.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T23:13:36Z] <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T272094 Change enwiki logo to 20th Birthday Celebration one (duration: 00m 56s)

Thu, Jan 14, 11:13 PM · Wikimedia-Site-requests
Stashbot added a comment to T272094: Change EnWiki logo for Wikipedia's 20th anniversary.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T23:10:57Z] <jforrester@deploy1001> Synchronized static/images/project-logos/enwiki20-2x.png: T272094 Sync out logo before going live, 3/3 (duration: 00m 55s)

Thu, Jan 14, 11:11 PM · Wikimedia-Site-requests
Stashbot added a comment to T272094: Change EnWiki logo for Wikipedia's 20th anniversary.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T23:09:51Z] <jforrester@deploy1001> Synchronized static/images/project-logos/enwiki20-1.5x.png: T272094 Sync out logo before going live, 2/3 (duration: 00m 55s)

Thu, Jan 14, 11:09 PM · Wikimedia-Site-requests
Stashbot added a comment to T272094: Change EnWiki logo for Wikipedia's 20th anniversary.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T23:07:25Z] <jforrester@deploy1001> Synchronized static/images/project-logos/enwiki20.png: T272094 Sync out logo before going live, 1/3 (duration: 01m 02s)

Thu, Jan 14, 11:07 PM · Wikimedia-Site-requests
Stashbot added a comment to T267090: Productionize clouddb10[13-20].

Mentioned in SAL (#wikimedia-operations) [2021-01-14T20:17:32Z] <mutante> ACKing all unhandled crit alerts about systemd on clouddb hosts - notifications are disabled but this cleans up Icinga web UI noise - T267090

Thu, Jan 14, 8:17 PM · Patch-For-Review, DBA, Data-Services, cloud-services-team (Kanban)
Stashbot added a comment to T272041: Close lrc.wikipedia.org for editing.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T19:31:53Z] <urbanecm@deploy1001> Synchronized dblists/closed.dblist: d3e274e9b953f5edda07fa3a016b7291a451ceb2: Close lrcwiki (T272041) (duration: 00m 58s)

Thu, Jan 14, 7:33 PM · User-Urbanecm, Wiki-Setup (Close)
Stashbot added a comment to T272041: Close lrc.wikipedia.org for editing.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T18:38:13Z] <Amir1> started mass deletion of lrcwiki (T272041) - https://w.wiki/uPV

Thu, Jan 14, 6:38 PM · User-Urbanecm, Wiki-Setup (Close)
Stashbot added a comment to T261134: upgrade cloud-vps openstack to Openstack version 'Stein'.

Mentioned in SAL (#wikimedia-cloud) [2021-01-14T17:58:24Z] <bstorm> hard rebooting tools-sgecron-01 following network issues during upgrade to stein T261134

Thu, Jan 14, 5:58 PM · Patch-For-Review, cloud-services-team (Kanban)
Stashbot added a comment to T264358: Investigate oozie banner monthly job timeouts.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T15:19:12Z] <otto@deploy1001> Finished deploy [analytics/refinery@1117f45]: Explicitly set timeout in banner_activity-druid-monthly-coord - T264358 (duration: 02m 16s)

Thu, Jan 14, 3:19 PM · Patch-For-Review, Analytics-Kanban, Analytics
Stashbot added a comment to T264358: Investigate oozie banner monthly job timeouts.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T15:16:55Z] <otto@deploy1001> Started deploy [analytics/refinery@1117f45]: Explicitly set timeout in banner_activity-druid-monthly-coord - T264358

Thu, Jan 14, 3:17 PM · Patch-For-Review, Analytics-Kanban, Analytics
Stashbot added a comment to T271519: codfw1dev: repurpose/rename labtestvirt2003.codfw.wmnet as cloudgw2001-dev.codfw.wmnet.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T14:28:47Z] <arturo> running homer in asw-b-codfw* (T271519)

Thu, Jan 14, 2:28 PM · cloud-services-team (Kanban)
Stashbot added a comment to T271519: codfw1dev: repurpose/rename labtestvirt2003.codfw.wmnet as cloudgw2001-dev.codfw.wmnet.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T14:24:19Z] <arturo> running homer in asw-b-codfw* (T271519)

Thu, Jan 14, 2:24 PM · cloud-services-team (Kanban)
Stashbot added a comment to T271978: PHP Notice: Undefined index: edit (CologneBlueHooks.php).

Mentioned in SAL (#wikimedia-operations) [2021-01-14T14:06:04Z] <hashar@deploy1001> Synchronized php-1.36.0-wmf.26/skins/CologneBlue/includes/CologneBlueHooks.php: Edit link may not be present, avoid undefined index notice T271978 (duration: 01m 07s)

Thu, Jan 14, 2:06 PM · MW-1.36-notes (1.36.0-wmf.27; 2021-01-19), Patch-For-Review, Readers-Web-Backlog (Tracking), CologneBlue, Wikimedia-production-error
Stashbot added a comment to T266487: Upgrade Netbox and accompanying to the 2.9 series (Tracking Task).

Mentioned in SAL (#wikimedia-operations) [2021-01-14T12:50:03Z] <volans> upgraded python3-pynetbox to 5.3.0-1 on all affected hosts - T266487

Thu, Jan 14, 12:50 PM · Patch-For-Review, DC-Ops, netbox
Stashbot added a comment to T266487: Upgrade Netbox and accompanying to the 2.9 series (Tracking Task).

Mentioned in SAL (#wikimedia-operations) [2021-01-14T12:14:04Z] <volans> built and uploaded python3-pynetbox 5.3.0-1 to apt.wikimedia.org - T266487

Thu, Jan 14, 12:14 PM · Patch-For-Review, DC-Ops, netbox
Stashbot added a comment to T272013: Remove QuickSurveys config for WMDE template topic.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T12:09:44Z] <awight@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:656116|Remove unused WMDE TeWü QuickSurveys (T253112, T272013)]] (duration: 01m 07s)

Thu, Jan 14, 12:10 PM · Readers-Web-Backlog (Tracking), Unplanned-Sprint-Work, Performance-Team (Radar), WMDE-TechWish, WMDE-QWERTY-Sprint-2021-01-06, QuickSurveys
Stashbot added a comment to T253112: Create survey for TechWish prototype announcements on dewiki and metawiki.

Mentioned in SAL (#wikimedia-operations) [2021-01-14T12:09:44Z] <awight@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:656116|Remove unused WMDE TeWü QuickSurveys (T253112, T272013)]] (duration: 01m 07s)

Thu, Jan 14, 12:09 PM · QuickSurveys (Surveys), WMDE-QWERTY-Sprint-2020-06-24, WMDE-QWERTY-Sprint-2020-06-10, Epic, MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), Patch-For-Review, WMDE-QWERTY-Sprint-2020-05-27, WMDE-Templates-FocusArea, WMDE-QWERTY-Sprint-2020-05-13