Page MenuHomePhabricator
Feed Advanced Search

Yesterday

colewhite triaged T221138: relocate/reimage cloudvirt1004 with 10G interfaces as Normal priority.
Wed, Apr 17, 6:48 PM · Patch-For-Review, ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
colewhite triaged T221139: relocate/reimage cloudvirt1003 with 10G interfaces as Normal priority.
Wed, Apr 17, 6:48 PM · Patch-For-Review, ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
colewhite triaged T221140: relocate/reimage cloudvirt1002 with 10G interfaces as Normal priority.
Wed, Apr 17, 6:48 PM · Patch-For-Review, ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
colewhite triaged T221141: relocate/reimage cloudvirt1001 with 10G interfaces as Normal priority.
Wed, Apr 17, 6:47 PM · Patch-For-Review, ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
colewhite triaged T221259: eqord - ulsfo Telia link down - IC-313592 as High priority.
Wed, Apr 17, 6:47 PM · Operations, netops

Tue, Apr 16

colewhite triaged T220860: access for foks to labweb (in one way or another) (or make changePassword.php work on mwmaint hosts) as Normal priority.
Tue, Apr 16, 6:11 PM · Operations, SRE-Access-Requests
colewhite triaged T220844: remove RT mail aliases as Normal priority.
Tue, Apr 16, 6:10 PM · Mail, Operations
colewhite triaged T220687: eqiad: (3) - zookeeper cluster for Analytics as Normal priority.
Tue, Apr 16, 6:10 PM · User-Elukey, hardware-requests, Operations
colewhite triaged T221125: cumin aliases not matching any hosts as Normal priority.
Tue, Apr 16, 6:09 PM · cloud-services-team, Operations, Operations-Software-Development
colewhite triaged T221115: labpuppetmaster logs 'cannot collect exported resources without storeconfigs being set' as Normal priority.
Tue, Apr 16, 6:08 PM · cloud-services-team, Operations
colewhite triaged T220853: VMs on cloudvirt1015 crashing as Normal priority.
Tue, Apr 16, 6:07 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)
colewhite triaged T220590: Decom ms-be101[345] as Normal priority.
Tue, Apr 16, 6:06 PM · User-fgiunchedi, media-storage, Operations
colewhite triaged T200297: Introduce a new namespace for collaborative judgements about wiki entities as Normal priority.
Tue, Apr 16, 6:05 PM · MW-1.33-notes (1.33.0-wmf.14; 2019-01-22), Patch-For-Review, Scoring-platform-team (Current), DBA, Operations, Jade, TechCom-RFC
colewhite triaged T220787: Fix RAID handler alert and puppet facter to work with Gen10 hosts and ssacli tool as Normal priority.
Tue, Apr 16, 6:04 PM · Patch-For-Review, Operations, Icinga, monitoring
colewhite triaged T220567: Wikitech page views sometimes default to MobileFrontend as Normal priority.
Tue, Apr 16, 6:03 PM · Traffic, Operations, wikitech.wikimedia.org
colewhite lowered the priority of T220500: logstash1012 lock up caused central logging stuck from High to Normal.
Tue, Apr 16, 6:02 PM · User-herron, Wikimedia-Logstash, Operations
colewhite triaged T220500: logstash1012 lock up caused central logging stuck as High priority.
Tue, Apr 16, 6:02 PM · User-herron, Wikimedia-Logstash, Operations
colewhite closed T220880: Degraded RAID on analytics1039 as Resolved.
Tue, Apr 16, 6:02 PM · ops-eqiad, Operations
colewhite added a comment to T220880: Degraded RAID on analytics1039.

we're pretty sure this is a false alarm

Tue, Apr 16, 6:02 PM · ops-eqiad, Operations
colewhite updated subscribers of T220880: Degraded RAID on analytics1039.
Tue, Apr 16, 6:01 PM · ops-eqiad, Operations
colewhite triaged T220681: Set `enable_dl` to 0 in php.ini as Normal priority.
Tue, Apr 16, 5:32 PM · Patch-For-Review, PHP 7.2 support, Performance-Team (Radar), Operations
colewhite triaged T220901: Elasticsearch nodes overloading in eqiad as High priority.
Tue, Apr 16, 3:55 PM · Patch-For-Review, Operations, Discovery-Search (Current work)
colewhite triaged T220907: Degraded RAID on ms-be1013 as High priority.
Tue, Apr 16, 3:42 PM · ops-eqiad, Operations
colewhite triaged T220982: maps hosts have bad permissions under /srv/deployment as High priority.
Tue, Apr 16, 3:41 PM · Operations
colewhite triaged T221047: relocate/reimage cloudvirt1007 with 10G interfaces as Normal priority.
Tue, Apr 16, 3:40 PM · Patch-For-Review, ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
colewhite triaged T221048: relocate/reimage cloudvirt1006 with 10G interfaces as Normal priority.
Tue, Apr 16, 3:40 PM · Patch-For-Review, ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
colewhite triaged T221049: relocate/reimage cloudvirt1005 with 10G interfaces as Normal priority.
Tue, Apr 16, 3:40 PM · Patch-For-Review, ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
colewhite triaged T221052: config file change canarying for logstash as Normal priority.
Tue, Apr 16, 3:39 PM · Operations, Wikimedia-Logstash
colewhite triaged T221068: decom ms-be201[345] as Normal priority.
Tue, Apr 16, 3:39 PM · User-fgiunchedi, Operations
colewhite triaged T221083: puppet fact: migrate away from the uniqueid fact as Normal priority.
Tue, Apr 16, 3:36 PM · Puppet, Operations

Mon, Apr 15

colewhite moved T219825: Update dashboards to node-exporter 0.16+ metric names from Backlog to In progress on the monitoring board.
Mon, Apr 15, 3:14 PM · Patch-For-Review, monitoring

Wed, Apr 3

colewhite added a comment to T219825: Update dashboards to node-exporter 0.16+ metric names.

fundraising database cannot be updated at this time. It looks like it may need upgrading or forwards-compatibility rules.

Wed, Apr 3, 10:51 PM · Patch-For-Review, monitoring

Mon, Apr 1

colewhite added a subtask for T213288: TEC6: Upgrade metrics monitoring infrastructure core components (Q3 2018/19 goal): T219825: Update dashboards to node-exporter 0.16+ metric names.
Mon, Apr 1, 6:44 PM · User-fgiunchedi, Goal, monitoring, Operations
colewhite added a parent task for T219825: Update dashboards to node-exporter 0.16+ metric names: T213288: TEC6: Upgrade metrics monitoring infrastructure core components (Q3 2018/19 goal).
Mon, Apr 1, 6:44 PM · Patch-For-Review, monitoring
colewhite closed T213708: Upgrade production prometheus-node-exporter to >= 0.16 as Resolved.
Mon, Apr 1, 6:43 PM · Patch-For-Review, Goal, monitoring, Operations
colewhite closed T213708: Upgrade production prometheus-node-exporter to >= 0.16, a subtask of T213288: TEC6: Upgrade metrics monitoring infrastructure core components (Q3 2018/19 goal), as Resolved.
Mon, Apr 1, 6:43 PM · User-fgiunchedi, Goal, monitoring, Operations
colewhite triaged T219825: Update dashboards to node-exporter 0.16+ metric names as Low priority.
Mon, Apr 1, 6:42 PM · Patch-For-Review, monitoring
colewhite created T219825: Update dashboards to node-exporter 0.16+ metric names.
Mon, Apr 1, 6:41 PM · Patch-For-Review, monitoring

Thu, Mar 28

colewhite updated the task description for T213708: Upgrade production prometheus-node-exporter to >= 0.16.
Thu, Mar 28, 11:13 PM · Patch-For-Review, Goal, monitoring, Operations

Fri, Mar 22

colewhite closed T216101: LDAP access to the WMF group for Angela Muigai as Resolved.
Fri, Mar 22, 6:18 PM · LDAP-Access-Requests
colewhite added a comment to T216101: LDAP access to the WMF group for Angela Muigai.

Thank you for following up!

Fri, Mar 22, 6:09 PM · LDAP-Access-Requests

Thu, Mar 21

colewhite added a comment to T217932: Change log routing to ELK cluster to use rsyslog->kafka rather than talking directly to the ELK cluster.

As I understand it, journald is already wired up to copy to rsyslog. The only change needed to get these logs onto Kafka is to whitelist the application in the lookup_table_output.json.

Thu, Mar 21, 5:24 PM · cloud-services-team (Kanban), Patch-For-Review, Striker

Mar 6 2019

colewhite closed T214594: node-exporter collector.diskstats.ignored-devices underescaped as Resolved.
Mar 6 2019, 6:34 PM · Patch-For-Review, monitoring

Mar 4 2019

colewhite claimed T214594: node-exporter collector.diskstats.ignored-devices underescaped.
Mar 4 2019, 4:10 PM · Patch-For-Review, monitoring

Feb 25 2019

colewhite closed T216120: LDAP access to the wmf group for Delphine Ménard (dmenard) as Resolved.
Feb 25 2019, 8:13 PM · Patch-For-Review, LDAP-Access-Requests
colewhite added a comment to T216120: LDAP access to the wmf group for Delphine Ménard (dmenard).

@Delphine_wmf is now in the wmf ldap group. Resolving task.

Feb 25 2019, 8:13 PM · Patch-For-Review, LDAP-Access-Requests

Feb 21 2019

colewhite created P8120 Smartmon Node Exporter comparison.
Feb 21 2019, 10:19 PM
colewhite placed T215940: Mailing list migration for Arbitration Committee to Google Group up for grabs.
Feb 21 2019, 6:23 PM · Operations, Office-IT, Wikimedia-Mailing-lists
colewhite updated the task description for T215940: Mailing list migration for Arbitration Committee to Google Group.
Feb 21 2019, 6:23 PM · Operations, Office-IT, Wikimedia-Mailing-lists
colewhite updated subscribers of T215940: Mailing list migration for Arbitration Committee to Google Group.

Mbox files shared with @eross .

Feb 21 2019, 6:23 PM · Operations, Office-IT, Wikimedia-Mailing-lists
colewhite closed T215576: Please add Runa Bhattacharjee to the `wmf` LDAP group as Resolved.
Feb 21 2019, 5:59 PM · Patch-For-Review, LDAP-Access-Requests
colewhite added a comment to T215576: Please add Runa Bhattacharjee to the `wmf` LDAP group.

@Arrbee is now in the wmf ldap group. Resolving task.

Feb 21 2019, 5:59 PM · Patch-For-Review, LDAP-Access-Requests
colewhite added a comment to T213708: Upgrade production prometheus-node-exporter to >= 0.16.

On further investigation, the log messages appear to be from the shebang of the ipmitool awk script.

Feb 21 2019, 4:51 PM · Patch-For-Review, Goal, monitoring, Operations

Feb 15 2019

colewhite added a comment to T216120: LDAP access to the wmf group for Delphine Ménard (dmenard).

I was unable to find your account in LDAP. Have you had an account created for you by OIT or created one on wikitech?

Feb 15 2019, 9:41 PM · Patch-For-Review, LDAP-Access-Requests
colewhite triaged T216235: cleanup reprepro configuration for elasticsearch-curator as Normal priority.
Feb 15 2019, 7:36 PM · Discovery-Search (Current work), Patch-For-Review, User-fgiunchedi, Elasticsearch, Operations
colewhite triaged T216226: GPU upgrade for stat1005 as Normal priority.
Feb 15 2019, 7:35 PM · Analytics, hardware-requests, Operations
colewhite triaged T216202: Disk failure on labsdb1005 as Normal priority.
Feb 15 2019, 7:34 PM · Operations, ops-eqiad
colewhite triaged T216243: cron spam for slow queries on mwmaint /usr/local/bin/foreachwiki initSiteStats.php --update > /dev/null as Normal priority.
Feb 15 2019, 7:33 PM · Operations, MediaWiki-Maintenance-scripts
colewhite triaged T216273: New cronspam from db clusters as Normal priority.
Feb 15 2019, 7:33 PM · Operations
colewhite added a subtask for T132324: Tracking and Reducing cron-spam to root@ : T216273: New cronspam from db clusters.
Feb 15 2019, 7:32 PM · Patch-For-Review, Operations
colewhite added a parent task for T216273: New cronspam from db clusters: T132324: Tracking and Reducing cron-spam to root@ .
Feb 15 2019, 7:32 PM · Operations
colewhite triaged T216223: Degraded RAID on labsdb1005 as Normal priority.
Feb 15 2019, 7:31 PM · cloud-services-team (Kanban), Toolforge, ops-eqiad, Operations
colewhite created T216273: New cronspam from db clusters.
Feb 15 2019, 7:22 PM · Operations
colewhite edited projects for T216223: Degraded RAID on labsdb1005, added: cloud-services-team (Kanban); removed cloud-services-team.
Feb 15 2019, 4:53 PM · cloud-services-team (Kanban), Toolforge, ops-eqiad, Operations

Feb 14 2019

colewhite triaged T216090: ensure httpd error logs from "misc apps" (krypton) end up in logstash as Normal priority.
Feb 14 2019, 11:12 PM · Wikimedia-Logstash, Operations, serviceops
colewhite updated subscribers of T216090: ensure httpd error logs from "misc apps" (krypton) end up in logstash.
Feb 14 2019, 11:12 PM · Wikimedia-Logstash, Operations, serviceops
colewhite triaged T216192: Update label and switch to rename labvirt1012 to cloudvirt1012 as Normal priority.
Feb 14 2019, 11:11 PM · ops-eqiad, Operations
colewhite claimed T215940: Mailing list migration for Arbitration Committee to Google Group.
Feb 14 2019, 10:51 PM · Operations, Office-IT, Wikimedia-Mailing-lists
colewhite claimed T216101: LDAP access to the WMF group for Angela Muigai.
Feb 14 2019, 8:56 PM · LDAP-Access-Requests
colewhite claimed T216120: LDAP access to the wmf group for Delphine Ménard (dmenard).
Feb 14 2019, 8:55 PM · Patch-For-Review, LDAP-Access-Requests
colewhite closed T215830: Requesting access to analytics-privatedata for esanders as Resolved.
Feb 14 2019, 8:55 PM · Patch-For-Review, Operations, SRE-Access-Requests
colewhite added a comment to T215830: Requesting access to analytics-privatedata for esanders.

The group membership change has been deployed.

Feb 14 2019, 8:54 PM · Patch-For-Review, Operations, SRE-Access-Requests
colewhite closed T215938: Access request: Ladsgroup to analytics-wmde-users as Resolved.
Feb 14 2019, 8:54 PM · Patch-For-Review, SRE-Access-Requests, Operations
colewhite added a comment to T215938: Access request: Ladsgroup to analytics-wmde-users.

The group membership change has been deployed.

Feb 14 2019, 8:53 PM · Patch-For-Review, SRE-Access-Requests, Operations
colewhite triaged T216183: Special:ProtectedPages times out on enwiki for Module namespace as High priority.
Feb 14 2019, 8:33 PM · User-Marostegui, Wikimedia-production-error, MediaWiki-Database, MediaWiki-Special-pages
colewhite added a comment to T216183: Special:ProtectedPages times out on enwiki for Module namespace.

The logs indicate that the request is timing out fetching data from the database.

Feb 14 2019, 8:32 PM · User-Marostegui, Wikimedia-production-error, MediaWiki-Database, MediaWiki-Special-pages
CDanis awarded T216088: Mapping of servers to stakeholders a Like token.
Feb 14 2019, 1:07 AM · Operations

Feb 13 2019

colewhite claimed T213708: Upgrade production prometheus-node-exporter to >= 0.16.
Feb 13 2019, 11:36 PM · Patch-For-Review, Goal, monitoring, Operations
colewhite claimed T215830: Requesting access to analytics-privatedata for esanders.
Feb 13 2019, 11:31 PM · Patch-For-Review, Operations, SRE-Access-Requests
colewhite triaged T216088: Mapping of servers to stakeholders as Normal priority.
Feb 13 2019, 11:28 PM · Operations
colewhite removed a project from T215938: Access request: Ladsgroup to analytics-wmde-users: LDAP-Access-Requests.
Feb 13 2019, 8:49 PM · Patch-For-Review, SRE-Access-Requests, Operations
colewhite claimed T215938: Access request: Ladsgroup to analytics-wmde-users.
Feb 13 2019, 8:48 PM · Patch-For-Review, SRE-Access-Requests, Operations
colewhite closed T216068: Degraded RAID on cloudvirt1024, a subtask of T215892: Degraded RAID on cloudvirt1024, as Resolved.
Feb 13 2019, 8:43 PM · cloud-services-team (Kanban), ops-eqiad, Operations
colewhite closed T216068: Degraded RAID on cloudvirt1024 as Resolved.
Feb 13 2019, 8:43 PM · ops-eqiad, Operations
colewhite added a comment to T216068: Degraded RAID on cloudvirt1024.

Resolving as duplicate of parent.

Feb 13 2019, 8:42 PM · ops-eqiad, Operations
colewhite added a parent task for T216068: Degraded RAID on cloudvirt1024: T215892: Degraded RAID on cloudvirt1024.
Feb 13 2019, 8:42 PM · ops-eqiad, Operations
colewhite added a subtask for T215892: Degraded RAID on cloudvirt1024: T216068: Degraded RAID on cloudvirt1024.
Feb 13 2019, 8:42 PM · cloud-services-team (Kanban), ops-eqiad, Operations
colewhite closed T215575: Please add Petar Petković to the `wmf` LDAP group as Resolved.
Feb 13 2019, 8:39 PM · Patch-For-Review, LDAP-Access-Requests
colewhite added a comment to T215575: Please add Petar Petković to the `wmf` LDAP group.

@Petar.petkovic is now in the wmf ldap group. Resolving task.

Feb 13 2019, 8:39 PM · Patch-For-Review, LDAP-Access-Requests
colewhite triaged T215892: Degraded RAID on cloudvirt1024 as Normal priority.
Feb 13 2019, 8:32 PM · cloud-services-team (Kanban), ops-eqiad, Operations
colewhite triaged T216004: Degraded RAID on cloudvirt1018 as Normal priority.
Feb 13 2019, 8:31 PM · cloud-services-team (Kanban), ops-eqiad, Operations
colewhite updated the task description for T213708: Upgrade production prometheus-node-exporter to >= 0.16.
Feb 13 2019, 3:04 AM · Patch-For-Review, Goal, monitoring, Operations
colewhite triaged T215848: icinga really needs to check puppet run success of passive icinga hosts as Normal priority.
Feb 13 2019, 2:38 AM · monitoring, Icinga, Operations

Feb 11 2019

colewhite closed T215574: Please add Natalia Harateh to the `wmf` LDAP group as Resolved.
Feb 11 2019, 10:55 PM · Patch-For-Review, LDAP-Access-Requests
colewhite added a comment to T215574: Please add Natalia Harateh to the `wmf` LDAP group.

@NHarateh_WMF is now in the wmf ldap group. Resolving task.

Feb 11 2019, 10:54 PM · Patch-For-Review, LDAP-Access-Requests
colewhite added a comment to T215576: Please add Runa Bhattacharjee to the `wmf` LDAP group.

@Arrbee your LDAP user does not have a WMF email address associated and this appears to be required for membership of the wmf group.

Feb 11 2019, 9:48 PM · Patch-For-Review, LDAP-Access-Requests
colewhite added a comment to T215575: Please add Petar Petković to the `wmf` LDAP group.

@Petar.petkovic your LDAP user does not have a WMF email address associated and this appears to be required for membership of the wmf group.

Feb 11 2019, 9:47 PM · Patch-For-Review, LDAP-Access-Requests
colewhite claimed T215576: Please add Runa Bhattacharjee to the `wmf` LDAP group.
Feb 11 2019, 8:08 PM · Patch-For-Review, LDAP-Access-Requests
colewhite claimed T215575: Please add Petar Petković to the `wmf` LDAP group.
Feb 11 2019, 8:07 PM · Patch-For-Review, LDAP-Access-Requests
colewhite claimed T215574: Please add Natalia Harateh to the `wmf` LDAP group.
Feb 11 2019, 7:32 PM · Patch-For-Review, LDAP-Access-Requests