Page MenuHomePhabricator

JHedden (Jason Hedden)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
May 28 2019, 6:09 PM (25 w, 3 d)
Availability
Available
LDAP User
Jhedden
MediaWiki User
JHedden (WMF) [ Global Accounts ]

Recent Activity

Yesterday

JHedden closed T238115: Prepare and check storage layer for shywiktionary, a subtask of T238105: Create Shawiya Wiktionary, as Resolved.
Thu, Nov 21, 4:04 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), User-Ladsgroup, User-Urbanecm, Wiki-Setup (Create)
JHedden closed T238115: Prepare and check storage layer for shywiktionary as Resolved.
Updates on labsdb10{09,10,11,12}
$ sudo /usr/local/sbin/maintain-replica-indexes --database shywiktionary --debug
$ sudo /usr/local/sbin/maintain-views --databases shywiktionary --debug
$ sudo /usr/local/sbin/maintain-meta_p --databases shywiktionary 
Thu, Nov 21, 4:04 PM · cloud-services-team (Kanban), Data-Services, DBA, Operations
JHedden closed T238114: Prepare and check storage layer for gcrwiki, a subtask of T238104: Create Guianan Creole Wikipedia, as Resolved.
Thu, Nov 21, 4:03 PM · User-Ladsgroup, Wiki-Setup (Create), User-Urbanecm
JHedden closed T238114: Prepare and check storage layer for gcrwiki as Resolved.
Updates on labsdb10{09,10,11,12}
$ sudo /usr/local/sbin/maintain-replica-indexes --database gcrwiki --debug
$ sudo /usr/local/sbin/maintain-views --databases gcrwiki --debug
$ sudo /usr/local/sbin/maintain-meta_p --databases gcrwiki 
Thu, Nov 21, 4:03 PM · cloud-services-team (Kanban), Data-Services, DBA, Operations
JHedden closed T237373: Prepare and check storage layer for szywiki, a subtask of T237369: Create Sakizaya Wikipedia, as Resolved.
Thu, Nov 21, 4:01 PM · User-Ladsgroup, User-Urbanecm, Wiki-Setup (Create)
JHedden closed T237373: Prepare and check storage layer for szywiki as Resolved.
Updates on labsdb10{09,10,11,12}
$ sudo /usr/local/sbin/maintain-replica-indexes --database szywiki --debug
$ sudo /usr/local/sbin/maintain-views --databases szywiki --debug
$ sudo /usr/local/sbin/maintain-meta_p --databases szywiki 
Thu, Nov 21, 4:00 PM · cloud-services-team (Kanban), Data-Services, DBA
JHedden closed T238522: Prepare and check storage layer for minwiktionary, a subtask of T236861: Create Minangkabau Wiktionary, as Resolved.
Thu, Nov 21, 3:56 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), User-Ladsgroup, Wiki-Setup (Create), User-Urbanecm
JHedden closed T238522: Prepare and check storage layer for minwiktionary as Resolved.
Updates on labsdb10{09,10,11,12}
$ sudo /usr/local/sbin/maintain-replica-indexes --database minwiktionary --debug
$ sudo /usr/local/sbin/maintain-views --databases minwiktionary --debug
$ sudo /usr/local/sbin/maintain-meta_p --databases minwiktionary 
Thu, Nov 21, 3:56 PM · cloud-services-team (Kanban), Data-Services, DBA, Operations
JHedden closed T236404: Prepare and check storage layer for ge.wikimedia.org, a subtask of T236389: Create a wiki for Wikimedia Community User Group Georgia, as Resolved.
Thu, Nov 21, 3:54 PM · User-Ladsgroup, MW-1.35-notes (1.35.0-wmf.4; 2019-10-29), Core Platform Team Workboards (Clinic Duty Team), User-MarcoAurelio, Wiki-Setup (Create), User-Urbanecm
JHedden closed T236404: Prepare and check storage layer for ge.wikimedia.org as Resolved.
Updates on labsdb10{09,10,11,12}
$ sudo /usr/local/sbin/maintain-replica-indexes --database gewikimedia --debug
$ sudo /usr/local/sbin/maintain-views --databases gewikimedia --debug
$ sudo /usr/local/sbin/maintain-meta_p --databases gewikimedia 
Thu, Nov 21, 3:54 PM · Data-Services, DBA, Operations

Tue, Nov 19

JHedden closed T230575: Degraded RAID on cloudvirt1018 as Resolved.

@Jclark-ctr replaced the drive in slot 4 with a new 1.9TB drive today. I've confirmed that the system and RAID set are both healthy.

Tue, Nov 19, 10:29 PM · ops-eqiad, Operations
JHedden closed T236331: Degraded RAID on cloudvirt1018 as Resolved.

@Jclark-ctr replaced this today with a new 1.9TB drive. No host errors were seen and the megaraid card looks clean.

Tue, Nov 19, 10:25 PM · ops-eqiad, Operations
JHedden closed T238222: Request creation of BishopFox VPS project as Resolved.

Hi @Mfrostbfox and @Danbishopfox The BishopFox CloudVPS project has been approved and created.

Tue, Nov 19, 7:35 PM · Cloud-VPS (Project-requests)
JHedden closed T237297: Request creation of WikiLoop VPS project as Resolved.

Hi @Xinbenlv and @zzn_google, The WikiLoop CloudVPS project has been approved and created.

Tue, Nov 19, 7:28 PM · Cloud-VPS (Project-requests)

Mon, Nov 18

JHedden created P9654 keystone changelog .
Mon, Nov 18, 4:19 PM

Fri, Nov 15

JHedden added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

I've update the task details with the current status. Should I leave the netbox status as staged or set it to active? These systems will be testing non-production workloads for the near future, and active seems to imply production status.

Fri, Nov 15, 11:01 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden updated the task description for T224188: rack/setup/install (3) new osd ceph nodes.
Fri, Nov 15, 10:56 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Looks like it's an issue with the virtual disk not getting assigned /dev/sda. Checking to see if I can work around this with our installation process and partman, but I may need to switch the controller to HBA mode and use software RAID for the operating system.

Fri, Nov 15, 7:54 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

I'm having an issue on cloudcephosd1002 and 1003.

Fri, Nov 15, 5:54 PM · Operations, cloud-services-team (Kanban), Cloud-Services

Thu, Nov 14

JHedden added a member for WMF-NDA: Phamhi.
Thu, Nov 14, 5:41 PM

Wed, Nov 13

JHedden updated the task description for T224188: rack/setup/install (3) new osd ceph nodes.
Wed, Nov 13, 11:07 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden updated the task description for T224188: rack/setup/install (3) new osd ceph nodes.
Wed, Nov 13, 7:00 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden updated the task description for T224188: rack/setup/install (3) new osd ceph nodes.
Wed, Nov 13, 6:57 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden committed rLPRI443330a8fe83: add fake k8s node_token for ceph storage cluster (authored by JHedden).
add fake k8s node_token for ceph storage cluster
Wed, Nov 13, 5:50 PM
JHedden closed T228102: rack/setup/install cloudcephmon100[123] as Resolved.
Wed, Nov 13, 4:02 PM · cloud-services-team (Kanban), Operations, Cloud-Services

Tue, Nov 12

JHedden added a comment to T228102: rack/setup/install cloudcephmon100[123].

@Jclark-ctr Could you help me with the cloudcephmon1002 and cloudcephmon1003 servers? I'm unable to power them on through iDRAC SSH, IPMI, or the web interface. I see the event logged in the lifecycle log but they never power on.

Tue, Nov 12, 6:06 PM · cloud-services-team (Kanban), Operations, Cloud-Services

Fri, Nov 8

JHedden updated the task description for T228102: rack/setup/install cloudcephmon100[123].
Fri, Nov 8, 9:18 PM · cloud-services-team (Kanban), Operations, Cloud-Services
JHedden added a comment to T236819: Identify container images and packages required for rook.io and ceph.

OS Packages for Debian GNU/Linux 10 (buster) pulled from http://apt.wikimedia.org/wikimedia stretch-wikimedia thirdparty/kubeadm-k8s

containerd.io
cri-tools
docker-ce
docker-ce-cli
ipset
kubeadm
kubectl
kubernetes-cni
Fri, Nov 8, 8:05 PM · cloud-services-team (Kanban)
JHedden added a comment to T228102: rack/setup/install cloudcephmon100[123].

@Jclark-ctr Could you help me with the cloudcephmon1002 and cloudcephmon1003 servers? I'm unable to power them on through iDRAC SSH, IPMI, or the web interface. I see the event logged in the lifecycle log but they never power on.

Fri, Nov 8, 5:28 PM · cloud-services-team (Kanban), Operations, Cloud-Services

Tue, Nov 5

JHedden added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

@Jclark-ctr thanks! confirmed that it's working from my end now.

Tue, Nov 5, 10:33 PM · Operations, cloud-services-team (Kanban), Cloud-Services

Mon, Nov 4

JHedden added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

The management interface for cloudcephosd1001.mgmt is currently unavailable, could we get someone take a look at it please?

Mon, Nov 4, 10:22 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden created P9521 /var/log/apache2/horizon_error.log.
Mon, Nov 4, 3:41 PM

Fri, Nov 1

JHedden closed T214297: cloudvps: neutron: investigate keepalived warning, a subtask of T214299: cloudvps: neutron: upgrade jessie -> stretch, as Resolved.
Fri, Nov 1, 3:36 PM · cloud-services-team (Kanban)
JHedden closed T214297: cloudvps: neutron: investigate keepalived warning as Resolved.

Confirmed that this warning can safely be ignored.

Fri, Nov 1, 3:36 PM · cloud-services-team (Kanban)

Tue, Oct 29

JHedden renamed T236819: Identify container images and packages required for rook.io and ceph from Identify container images required for rook.io and ceph to Identify container images and packages required for rook.io and ceph.
Tue, Oct 29, 9:08 PM · cloud-services-team (Kanban)
JHedden created T236819: Identify container images and packages required for rook.io and ceph.
Tue, Oct 29, 5:30 PM · cloud-services-team (Kanban)

Mon, Oct 28

JHedden updated the task description for T227538: b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC).
Mon, Oct 28, 5:25 PM · DC-Ops, Operations, ops-eqiad
JHedden placed T227538: b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC) up for grabs.
Mon, Oct 28, 5:24 PM · DC-Ops, Operations, ops-eqiad
JHedden added a comment to T236703: OpenStack server list shows outdated floating ip information.

This was originally noticed while working on T235627

Mon, Oct 28, 3:55 PM · cloud-services-team, Cloud-VPS
JHedden created T236703: OpenStack server list shows outdated floating ip information.
Mon, Oct 28, 3:53 PM · cloud-services-team, Cloud-VPS

Oct 23 2019

JHedden moved T236290: Deploy a Ceph testing environment using Rook.io on VMs from Inbox to Doing on the cloud-services-team (Kanban) board.
Oct 23 2019, 4:07 PM · cloud-services-team (Kanban)
JHedden placed T225320: Ceph Proof of Concept Build and Testing up for grabs.
Oct 23 2019, 4:06 PM · Epic, cloud-services-team (Kanban)
JHedden created T236290: Deploy a Ceph testing environment using Rook.io on VMs.
Oct 23 2019, 4:06 PM · cloud-services-team (Kanban)
JHedden claimed T225320: Ceph Proof of Concept Build and Testing.
Oct 23 2019, 4:04 PM · Epic, cloud-services-team (Kanban)
JHedden moved T231793: Remove systemd from openstack-mitaka from Doing to Watching on the cloud-services-team (Kanban) board.
Oct 23 2019, 4:03 PM · cloud-services-team (Kanban), Operations
JHedden closed T223905: HA for openstack services as Resolved.

Grafana dashboard for HAproxy metrics: https://grafana.wikimedia.org/d/tanisM2Zz/wmcs-openstack-eqiad1-api-stats

Oct 23 2019, 4:03 PM · Goal, cloud-services-team (Kanban)
JHedden updated the task description for T223905: HA for openstack services.
Oct 23 2019, 4:01 PM · Goal, cloud-services-team (Kanban)
JHedden updated the task description for T223905: HA for openstack services.
Oct 23 2019, 3:59 PM · Goal, cloud-services-team (Kanban)
JHedden moved T219079: Horizon - Not possible to remove A record from Record Set from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Oct 23 2019, 3:52 PM · Horizon, cloud-services-team (Kanban)
JHedden merged task T230537: keystone/horizon: character encoding issue in username into T229227: Usernames containing unicode characters unable to authenticate to Wikitech and Horizon.
Oct 23 2019, 3:50 PM · cloud-services-team (Kanban)
JHedden merged T230537: keystone/horizon: character encoding issue in username into T229227: Usernames containing unicode characters unable to authenticate to Wikitech and Horizon.
Oct 23 2019, 3:50 PM · cloud-services-team (Kanban), wikitech.wikimedia.org, Horizon
JHedden closed T142866: Upgrade qemu on labvirts, a subtask of T141673: Track labs instances hanging , as Resolved.
Oct 23 2019, 3:42 PM · RelEng-Archive-FY201718-Q1, Patch-For-Review, Cloud-Services
JHedden closed T142866: Upgrade qemu on labvirts as Resolved.
Oct 23 2019, 3:42 PM · cloud-services-team (Kanban), Cloud-Services
JHedden closed T149057: Designate seems very slow to delete records? as Resolved.

Record deletes are working as expected now, likely resolved from the OpenStack upgrades and service improvements.

Oct 23 2019, 3:40 PM · cloud-services-team (Kanban), Operations, Cloud-Services
JHedden moved T158103: Horizon Mitaka 'remember me' checkbox immune to keyboard focus from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Oct 23 2019, 3:37 PM · cloud-services-team (Kanban), Developer-notice, Cloud-Services
JHedden moved T192183: Horizon only saves user user settings in browser cookies from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Oct 23 2019, 3:34 PM · cloud-services-team (Kanban), Upstream, Horizon
JHedden moved T192182: Horizon instance list: Sorting should work regardless of paging from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Oct 23 2019, 3:34 PM · cloud-services-team (Kanban), Upstream, Horizon
JHedden closed T196210: Strange default for IP Address field on floating IP association form as Resolved.

This is no longer the case, most likely resolved from the OpenStack upgrades. The form to associate a floating IP address correctly shows IP addresses now.

Oct 23 2019, 3:32 PM · cloud-services-team (Kanban), Horizon
JHedden moved T216713: Allow use of custom error page behind domainproxy from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Oct 23 2019, 3:21 PM · Cloud-VPS, cloud-services-team (Kanban)
JHedden moved T218606: tools.wmflabs.org landing page ("admin" tool) has poor layout on devices with width <550px from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Oct 23 2019, 3:21 PM · cloud-services-team (Kanban), Mobile, Tools
JHedden moved T229660: Horizon warning for instance deletion does not include instance name from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Oct 23 2019, 3:19 PM · Upstream, Horizon, cloud-services-team (Kanban)

Oct 22 2019

JHedden closed T235863: Openstack: designate API issues as Resolved.

Patch has been merged and openstack zone list --all-projects is now working as expected.

Oct 22 2019, 7:16 PM · Patch-For-Review, cloud-services-team (Kanban)

Oct 21 2019

JHedden updated the task description for T227540: b4-eqiad pdu refresh (Thursday 10/24 @11am UTC).
Oct 21 2019, 7:09 PM · DC-Ops, Operations, ops-eqiad
JHedden updated the task description for T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC).
Oct 21 2019, 6:57 PM · DC-Ops, Operations, ops-eqiad

Oct 18 2019

JHedden closed T235906: Cannot access replica databases as Resolved.

@dom_walden I verified that your account is now working again, please let us know if you're still unable to connect to the replica databases.

Oct 18 2019, 7:19 PM · Toolforge
JHedden added a comment to T235863: Openstack: designate API issues.

The patch seems simple enough that perhaps we can do the patching ourselves? What do you think?

--- a/designateclient/v2/client.py
+++ b/designateclient/v2/client.py
@@ -58,7 +58,7 @@ class DesignateAdapter(adapter.LegacyJsonAdapter):
         if self.all_projects:
             kwargs['headers'].setdefault(
                 'X-Auth-All-Projects',
-                self.all_projects
+                str(self.all_projects)
             )
         if self.edit_managed:
Oct 18 2019, 4:18 PM · Patch-For-Review, cloud-services-team (Kanban)
JHedden added a comment to T235863: Openstack: designate API issues.
root@cloudcontrol1004:~# openstack zone list --all-projects
Unexpected exception for http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones?: Header value True must be of type str or bytes, not <type 'bool'>
Oct 18 2019, 2:43 PM · Patch-For-Review, cloud-services-team (Kanban)

Oct 17 2019

JHedden updated the task description for T223905: HA for openstack services.
Oct 17 2019, 3:44 PM · Goal, cloud-services-team (Kanban)
JHedden closed T223907: Set up HA endpoints for keystone, glance, nova, designate apis, a subtask of T223905: HA for openstack services, as Resolved.
Oct 17 2019, 3:43 PM · Goal, cloud-services-team (Kanban)
JHedden closed T223907: Set up HA endpoints for keystone, glance, nova, designate apis as Resolved.

All HA changes are now implemented in both regions codfw1dev and eqiad1.

Oct 17 2019, 3:43 PM · cloud-services-team (Kanban)
JHedden added a comment to T223907: Set up HA endpoints for keystone, glance, nova, designate apis.

New configuration

Oct 17 2019, 3:30 PM · cloud-services-team (Kanban)
JHedden added a comment to T223907: Set up HA endpoints for keystone, glance, nova, designate apis.

Updating the OpenStack endpoints in eqiad1 to the new domain.

Oct 17 2019, 3:17 PM · cloud-services-team (Kanban)
JHedden closed T235697: labsdb1011 missing tools user accounts as Resolved.
Oct 17 2019, 1:33 PM · Data-Services, cloud-services-team (Kanban)
JHedden claimed T235697: labsdb1011 missing tools user accounts.
Oct 17 2019, 12:13 PM · Data-Services, cloud-services-team (Kanban)

Oct 16 2019

Bstorm awarded T235697: labsdb1011 missing tools user accounts a Yellow Medal token.
Oct 16 2019, 11:10 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.

Changes pushed. Full list of accounts that were out of sync and created at: https://phabricator.wikimedia.org/P9366

Oct 16 2019, 9:57 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.
example command
maintain-dbusers --account-type user harvest-replicas
new account entry
$ mysql -h labsdb1011.eqiad.wmnet -u labsdbadmin -p -e 'SELECT COUNT(User) from mysql.user where User = "u21436"\G'
*************************** 1. row ***************************
COUNT(User): 1

and I'm able to connect from toolforge now :)

mysql --defaults-file=$HOME/replica.my.cnf -h enwiki.analytics.db.svc.eqiad.wmflabs enwiki_p
...
Oct 16 2019, 9:42 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.

I tested the above patch with my user account and verified it's working as expected.

Oct 16 2019, 9:29 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.

I believe the current implementation of harvest works in the opposite direction as what would be needed to fix the labsdb1011 auth tables. It populates the labsdbaccounts database on m5 which tracks account provisioning status based on the current filesystem ($HOME/replica.my.cnf) and remote database auth table status (labsdb10{09,10,11}, clouddb1001.clouddb).

Harvest has two functions:

  1. harvest_cnf_files finds tools and users replica.my.cnf file and insert/update m5's labsdbaccounts.account table
  2. harvest_replica_accts Determine the state of users defined in m5's labsdbaccounts.account table on each replica, then update the users state to either absent|present in labsdbaccounts.account_host table for each replica.
Oct 16 2019, 8:37 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.

Related task T235382

Oct 16 2019, 7:27 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.

I noticed there's a harvest function in maintain-dbusers https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/wmcs/nfs/maintain-dbusers.py#19

Oct 16 2019, 7:13 PM · Data-Services, cloud-services-team (Kanban)
JHedden created T235697: labsdb1011 missing tools user accounts.
Oct 16 2019, 7:12 PM · Data-Services, cloud-services-team (Kanban)
JHedden closed T220530: Ensure clouddb1001 is monitored appropriately from the tendril/prometheus side as Resolved.
Oct 16 2019, 6:03 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235627: Toolforge: upgrade main proxy servers to Debian Buster.

Please @Bstorm and @JHedden confirm the scheduled operation window works for you both, thanks!

Oct 16 2019, 1:31 PM · Toolforge, cloud-services-team (Kanban), Kubernetes

Oct 15 2019

JHedden closed T234770: Prepare and check storage layer for banwiki, a subtask of T234768: Create Balinese Wikipedia, as Resolved.
Oct 15 2019, 1:53 PM · MW-1.35-notes (1.35.0-wmf.2; 2019-10-15), User-Ladsgroup, Wiki-Setup (Create), User-Urbanecm
JHedden closed T234770: Prepare and check storage layer for banwiki as Resolved.
Updates on labsdb10{09,10,11,12}
$ sudo /usr/local/sbin/maintain-replica-indexes --database banwiki --debug
$ sudo /usr/local/sbin/maintain-views --databases banwiki --debug
$ sudo /usr/local/sbin/maintain-meta_p --databases banwiki 
Oct 15 2019, 1:53 PM · cloud-services-team (Kanban), Data-Services, DBA, Operations
JHedden claimed T234770: Prepare and check storage layer for banwiki.
Oct 15 2019, 1:28 PM · cloud-services-team (Kanban), Data-Services, DBA, Operations

Oct 11 2019

JHedden added a comment to T234617: Toolforge. introduce new domain toolforge.org.

The toolforge.org domain has been setup in designate

Oct 11 2019, 9:53 PM · Toolforge, cloud-services-team (Kanban), Kubernetes
JHedden added a comment to T235304: Create a service account to manage toolforge.org. from acme-chief.

Assigned roles for the tools-dns-manager user in eqiad1

Oct 11 2019, 9:26 PM · Toolforge, cloud-services-team (Kanban)

Oct 10 2019

JHedden added a comment to T231793: Remove systemd from openstack-mitaka.

After the Newton upgrade the Stretch hosts areall running the distro version: systemd-sysv 232-25+deb9u11

  • cloudcontrol[1003-1004].wikimedia.org
  • cloudnet[1003-1004].eqiad.wmnet
  • cloudvirt[1001-1009,1012-1014,1016-1030].eqiad.wmnet
Oct 10 2019, 8:12 PM · cloud-services-team (Kanban), Operations
JHedden closed T235127: wmfsink designate handler not running during vm deletes as Resolved.
Oct 10 2019, 4:34 PM · cloud-services-team (Kanban)
JHedden moved T235127: wmfsink designate handler not running during vm deletes from Inbox to Doing on the cloud-services-team (Kanban) board.
Oct 10 2019, 1:49 PM · cloud-services-team (Kanban)

Oct 9 2019

JHedden created T235127: wmfsink designate handler not running during vm deletes.
Oct 9 2019, 8:34 PM · cloud-services-team (Kanban)
JHedden closed T235070: Resolve keystone schema errors, a subtask of T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07, as Invalid.
Oct 9 2019, 2:44 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
JHedden closed T235070: Resolve keystone schema errors as Invalid.

This was a false alarm... The domain_id column was removed in a later revision: https://github.com/openstack/keystone/blob/master/keystone/common/sql/migrate_repo/versions/091_migrate_data_to_local_user_and_password_tables.py#L82

Oct 9 2019, 2:44 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
JHedden updated the task description for T235070: Resolve keystone schema errors.
Oct 9 2019, 2:21 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
JHedden claimed T235070: Resolve keystone schema errors.
Oct 9 2019, 2:15 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
JHedden created T235070: Resolve keystone schema errors.
Oct 9 2019, 1:53 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools

Oct 3 2019

JHedden added a comment to T234518: codfw1dev: glance_rsync_images is failing.

Puppet is not managing the permissions on the secondary controllers image directory.

cloudcontrol2003-dev:~# ls -lad /srv/glance/images
drwxr-xr-x 2 glance glance 4096 Oct  1 21:29 /srv/glance/images
Oct 3 2019, 1:54 PM · cloud-services-team (Kanban)

Oct 2 2019

JHedden added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

Honestly I have no idea where is that 208.80.152.0 coming from.

Oct 2 2019, 1:22 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)