Page MenuHomePhabricator

JHedden (Jason Hedden)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
May 28 2019, 6:09 PM (25 w, 20 h)
Availability
Available
LDAP User
Jhedden
MediaWiki User
JHedden (WMF) [ Global Accounts ]

Recent Activity

Yesterday

JHedden closed T230575: Degraded RAID on cloudvirt1018 as Resolved.

@Jclark-ctr replaced the drive in slot 4 with a new 1.9TB drive today. I've confirmed that the system and RAID set are both healthy.

Tue, Nov 19, 10:29 PM · ops-eqiad, Operations
JHedden closed T236331: Degraded RAID on cloudvirt1018 as Resolved.

@Jclark-ctr replaced this today with a new 1.9TB drive. No host errors were seen and the megaraid card looks clean.

Tue, Nov 19, 10:25 PM · ops-eqiad, Operations
JHedden closed T238222: Request creation of BishopFox VPS project as Resolved.

Hi @Mfrostbfox and @Danbishopfox The BishopFox CloudVPS project has been approved and created.

Tue, Nov 19, 7:35 PM · Cloud-VPS (Project-requests)
JHedden closed T237297: Request creation of WikiLoop VPS project as Resolved.

Hi @Xinbenlv and @zzn_google, The WikiLoop CloudVPS project has been approved and created.

Tue, Nov 19, 7:28 PM · Cloud-VPS (Project-requests)

Mon, Nov 18

JHedden created P9654 keystone changelog .
Mon, Nov 18, 4:19 PM

Fri, Nov 15

JHedden added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

I've update the task details with the current status. Should I leave the netbox status as staged or set it to active? These systems will be testing non-production workloads for the near future, and active seems to imply production status.

Fri, Nov 15, 11:01 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden updated the task description for T224188: rack/setup/install (3) new osd ceph nodes.
Fri, Nov 15, 10:56 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Looks like it's an issue with the virtual disk not getting assigned /dev/sda. Checking to see if I can work around this with our installation process and partman, but I may need to switch the controller to HBA mode and use software RAID for the operating system.

Fri, Nov 15, 7:54 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

I'm having an issue on cloudcephosd1002 and 1003.

Fri, Nov 15, 5:54 PM · Operations, cloud-services-team (Kanban), Cloud-Services

Thu, Nov 14

JHedden added a member for WMF-NDA: Phamhi.
Thu, Nov 14, 5:41 PM

Wed, Nov 13

JHedden updated the task description for T224188: rack/setup/install (3) new osd ceph nodes.
Wed, Nov 13, 11:07 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden updated the task description for T224188: rack/setup/install (3) new osd ceph nodes.
Wed, Nov 13, 7:00 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden updated the task description for T224188: rack/setup/install (3) new osd ceph nodes.
Wed, Nov 13, 6:57 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden committed rLPRI443330a8fe83: add fake k8s node_token for ceph storage cluster (authored by JHedden).
add fake k8s node_token for ceph storage cluster
Wed, Nov 13, 5:50 PM
JHedden closed T228102: rack/setup/install cloudcephmon100[123] as Resolved.
Wed, Nov 13, 4:02 PM · cloud-services-team (Kanban), Operations, Cloud-Services

Tue, Nov 12

JHedden added a comment to T228102: rack/setup/install cloudcephmon100[123].

@Jclark-ctr Could you help me with the cloudcephmon1002 and cloudcephmon1003 servers? I'm unable to power them on through iDRAC SSH, IPMI, or the web interface. I see the event logged in the lifecycle log but they never power on.

Tue, Nov 12, 6:06 PM · cloud-services-team (Kanban), Operations, Cloud-Services

Fri, Nov 8

JHedden updated the task description for T228102: rack/setup/install cloudcephmon100[123].
Fri, Nov 8, 9:18 PM · cloud-services-team (Kanban), Operations, Cloud-Services
JHedden added a comment to T236819: Identify container images and packages required for rook.io and ceph.

OS Packages for Debian GNU/Linux 10 (buster) pulled from http://apt.wikimedia.org/wikimedia stretch-wikimedia thirdparty/kubeadm-k8s

containerd.io
cri-tools
docker-ce
docker-ce-cli
ipset
kubeadm
kubectl
kubernetes-cni
Fri, Nov 8, 8:05 PM · cloud-services-team (Kanban)
JHedden added a comment to T228102: rack/setup/install cloudcephmon100[123].

@Jclark-ctr Could you help me with the cloudcephmon1002 and cloudcephmon1003 servers? I'm unable to power them on through iDRAC SSH, IPMI, or the web interface. I see the event logged in the lifecycle log but they never power on.

Fri, Nov 8, 5:28 PM · cloud-services-team (Kanban), Operations, Cloud-Services

Tue, Nov 5

JHedden added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

@Jclark-ctr thanks! confirmed that it's working from my end now.

Tue, Nov 5, 10:33 PM · Operations, cloud-services-team (Kanban), Cloud-Services

Mon, Nov 4

JHedden added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

The management interface for cloudcephosd1001.mgmt is currently unavailable, could we get someone take a look at it please?

Mon, Nov 4, 10:22 PM · Operations, cloud-services-team (Kanban), Cloud-Services
JHedden created P9521 /var/log/apache2/horizon_error.log.
Mon, Nov 4, 3:41 PM

Fri, Nov 1

JHedden closed T214297: cloudvps: neutron: investigate keepalived warning, a subtask of T214299: cloudvps: neutron: upgrade jessie -> stretch, as Resolved.
Fri, Nov 1, 3:36 PM · cloud-services-team (Kanban)
JHedden closed T214297: cloudvps: neutron: investigate keepalived warning as Resolved.

Confirmed that this warning can safely be ignored.

Fri, Nov 1, 3:36 PM · cloud-services-team (Kanban)

Tue, Oct 29

JHedden renamed T236819: Identify container images and packages required for rook.io and ceph from Identify container images required for rook.io and ceph to Identify container images and packages required for rook.io and ceph.
Tue, Oct 29, 9:08 PM · cloud-services-team (Kanban)
JHedden created T236819: Identify container images and packages required for rook.io and ceph.
Tue, Oct 29, 5:30 PM · cloud-services-team (Kanban)

Mon, Oct 28

JHedden updated the task description for T227538: b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC).
Mon, Oct 28, 5:25 PM · DC-Ops, Operations, ops-eqiad
JHedden placed T227538: b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC) up for grabs.
Mon, Oct 28, 5:24 PM · DC-Ops, Operations, ops-eqiad
JHedden added a comment to T236703: OpenStack server list shows outdated floating ip information.

This was originally noticed while working on T235627

Mon, Oct 28, 3:55 PM · cloud-services-team, Cloud-VPS
JHedden created T236703: OpenStack server list shows outdated floating ip information.
Mon, Oct 28, 3:53 PM · cloud-services-team, Cloud-VPS

Wed, Oct 23

JHedden moved T236290: Deploy a Ceph testing environment using Rook.io on VMs from Inbox to Doing on the cloud-services-team (Kanban) board.
Wed, Oct 23, 4:07 PM · cloud-services-team (Kanban)
JHedden placed T225320: Ceph Proof of Concept Build and Testing up for grabs.
Wed, Oct 23, 4:06 PM · Epic, cloud-services-team (Kanban)
JHedden created T236290: Deploy a Ceph testing environment using Rook.io on VMs.
Wed, Oct 23, 4:06 PM · cloud-services-team (Kanban)
JHedden claimed T225320: Ceph Proof of Concept Build and Testing.
Wed, Oct 23, 4:04 PM · Epic, cloud-services-team (Kanban)
JHedden moved T231793: Remove systemd from openstack-mitaka from Doing to Watching on the cloud-services-team (Kanban) board.
Wed, Oct 23, 4:03 PM · cloud-services-team (Kanban), Operations
JHedden closed T223905: HA for openstack services as Resolved.

Grafana dashboard for HAproxy metrics: https://grafana.wikimedia.org/d/tanisM2Zz/wmcs-openstack-eqiad1-api-stats

Wed, Oct 23, 4:03 PM · Goal, cloud-services-team (Kanban)
JHedden updated the task description for T223905: HA for openstack services.
Wed, Oct 23, 4:01 PM · Goal, cloud-services-team (Kanban)
JHedden updated the task description for T223905: HA for openstack services.
Wed, Oct 23, 3:59 PM · Goal, cloud-services-team (Kanban)
JHedden moved T219079: Horizon - Not possible to remove A record from Record Set from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Wed, Oct 23, 3:52 PM · Horizon, cloud-services-team (Kanban)
JHedden merged task T230537: keystone/horizon: character encoding issue in username into T229227: Usernames containing unicode characters unable to authenticate to Wikitech and Horizon.
Wed, Oct 23, 3:50 PM · cloud-services-team (Kanban)
JHedden merged T230537: keystone/horizon: character encoding issue in username into T229227: Usernames containing unicode characters unable to authenticate to Wikitech and Horizon.
Wed, Oct 23, 3:50 PM · cloud-services-team (Kanban), wikitech.wikimedia.org, Horizon
JHedden closed T142866: Upgrade qemu on labvirts, a subtask of T141673: Track labs instances hanging , as Resolved.
Wed, Oct 23, 3:42 PM · RelEng-Archive-FY201718-Q1, Patch-For-Review, Cloud-Services
JHedden closed T142866: Upgrade qemu on labvirts as Resolved.
Wed, Oct 23, 3:42 PM · cloud-services-team (Kanban), Cloud-Services
JHedden closed T149057: Designate seems very slow to delete records? as Resolved.

Record deletes are working as expected now, likely resolved from the OpenStack upgrades and service improvements.

Wed, Oct 23, 3:40 PM · cloud-services-team (Kanban), Operations, Cloud-Services
JHedden moved T158103: Horizon Mitaka 'remember me' checkbox immune to keyboard focus from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Wed, Oct 23, 3:37 PM · cloud-services-team (Kanban), Developer-notice, Cloud-Services
JHedden moved T192183: Horizon only saves user user settings in browser cookies from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Wed, Oct 23, 3:34 PM · cloud-services-team (Kanban), Upstream, Horizon
JHedden moved T192182: Horizon instance list: Sorting should work regardless of paging from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Wed, Oct 23, 3:34 PM · cloud-services-team (Kanban), Upstream, Horizon
JHedden closed T196210: Strange default for IP Address field on floating IP association form as Resolved.

This is no longer the case, most likely resolved from the OpenStack upgrades. The form to associate a floating IP address correctly shows IP addresses now.

Wed, Oct 23, 3:32 PM · cloud-services-team (Kanban), Horizon
JHedden moved T216713: Allow use of custom error page behind domainproxy from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Wed, Oct 23, 3:21 PM · Cloud-VPS, cloud-services-team (Kanban)
JHedden moved T218606: tools.wmflabs.org landing page ("admin" tool) has poor layout on devices with width <550px from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Wed, Oct 23, 3:21 PM · cloud-services-team (Kanban), Mobile, Tools
JHedden moved T229660: Horizon warning for instance deletion does not include instance name from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Wed, Oct 23, 3:19 PM · Upstream, Horizon, cloud-services-team (Kanban)

Tue, Oct 22

JHedden closed T235863: Openstack: designate API issues as Resolved.

Patch has been merged and openstack zone list --all-projects is now working as expected.

Tue, Oct 22, 7:16 PM · Patch-For-Review, cloud-services-team (Kanban)

Mon, Oct 21

JHedden updated the task description for T227540: b4-eqiad pdu refresh (Thursday 10/24 @11am UTC).
Mon, Oct 21, 7:09 PM · DC-Ops, Operations, ops-eqiad
JHedden updated the task description for T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC).
Mon, Oct 21, 6:57 PM · DC-Ops, Operations, ops-eqiad

Oct 18 2019

JHedden closed T235906: Cannot access replica databases as Resolved.

@dom_walden I verified that your account is now working again, please let us know if you're still unable to connect to the replica databases.

Oct 18 2019, 7:19 PM · Toolforge
JHedden added a comment to T235863: Openstack: designate API issues.

The patch seems simple enough that perhaps we can do the patching ourselves? What do you think?

--- a/designateclient/v2/client.py
+++ b/designateclient/v2/client.py
@@ -58,7 +58,7 @@ class DesignateAdapter(adapter.LegacyJsonAdapter):
         if self.all_projects:
             kwargs['headers'].setdefault(
                 'X-Auth-All-Projects',
-                self.all_projects
+                str(self.all_projects)
             )
         if self.edit_managed:
Oct 18 2019, 4:18 PM · Patch-For-Review, cloud-services-team (Kanban)
JHedden added a comment to T235863: Openstack: designate API issues.
root@cloudcontrol1004:~# openstack zone list --all-projects
Unexpected exception for http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones?: Header value True must be of type str or bytes, not <type 'bool'>
Oct 18 2019, 2:43 PM · Patch-For-Review, cloud-services-team (Kanban)

Oct 17 2019

JHedden updated the task description for T223905: HA for openstack services.
Oct 17 2019, 3:44 PM · Goal, cloud-services-team (Kanban)
JHedden closed T223907: Set up HA endpoints for keystone, glance, nova, designate apis, a subtask of T223905: HA for openstack services, as Resolved.
Oct 17 2019, 3:43 PM · Goal, cloud-services-team (Kanban)
JHedden closed T223907: Set up HA endpoints for keystone, glance, nova, designate apis as Resolved.

All HA changes are now implemented in both regions codfw1dev and eqiad1.

Oct 17 2019, 3:43 PM · cloud-services-team (Kanban)
JHedden added a comment to T223907: Set up HA endpoints for keystone, glance, nova, designate apis.

New configuration

Oct 17 2019, 3:30 PM · cloud-services-team (Kanban)
JHedden added a comment to T223907: Set up HA endpoints for keystone, glance, nova, designate apis.

Updating the OpenStack endpoints in eqiad1 to the new domain.

Oct 17 2019, 3:17 PM · cloud-services-team (Kanban)
JHedden closed T235697: labsdb1011 missing tools user accounts as Resolved.
Oct 17 2019, 1:33 PM · Data-Services, cloud-services-team (Kanban)
JHedden claimed T235697: labsdb1011 missing tools user accounts.
Oct 17 2019, 12:13 PM · Data-Services, cloud-services-team (Kanban)

Oct 16 2019

Bstorm awarded T235697: labsdb1011 missing tools user accounts a Yellow Medal token.
Oct 16 2019, 11:10 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.

Changes pushed. Full list of accounts that were out of sync and created at: https://phabricator.wikimedia.org/P9366

Oct 16 2019, 9:57 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.
example command
maintain-dbusers --account-type user harvest-replicas
new account entry
$ mysql -h labsdb1011.eqiad.wmnet -u labsdbadmin -p -e 'SELECT COUNT(User) from mysql.user where User = "u21436"\G'
*************************** 1. row ***************************
COUNT(User): 1

and I'm able to connect from toolforge now :)

mysql --defaults-file=$HOME/replica.my.cnf -h enwiki.analytics.db.svc.eqiad.wmflabs enwiki_p
...
Oct 16 2019, 9:42 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.

I tested the above patch with my user account and verified it's working as expected.

Oct 16 2019, 9:29 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.

I believe the current implementation of harvest works in the opposite direction as what would be needed to fix the labsdb1011 auth tables. It populates the labsdbaccounts database on m5 which tracks account provisioning status based on the current filesystem ($HOME/replica.my.cnf) and remote database auth table status (labsdb10{09,10,11}, clouddb1001.clouddb).

Harvest has two functions:

  1. harvest_cnf_files finds tools and users replica.my.cnf file and insert/update m5's labsdbaccounts.account table
  2. harvest_replica_accts Determine the state of users defined in m5's labsdbaccounts.account table on each replica, then update the users state to either absent|present in labsdbaccounts.account_host table for each replica.
Oct 16 2019, 8:37 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.

Related task T235382

Oct 16 2019, 7:27 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235697: labsdb1011 missing tools user accounts.

I noticed there's a harvest function in maintain-dbusers https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/wmcs/nfs/maintain-dbusers.py#19

Oct 16 2019, 7:13 PM · Data-Services, cloud-services-team (Kanban)
JHedden created T235697: labsdb1011 missing tools user accounts.
Oct 16 2019, 7:12 PM · Data-Services, cloud-services-team (Kanban)
JHedden closed T220530: Ensure clouddb1001 is monitored appropriately from the tendril/prometheus side as Resolved.
Oct 16 2019, 6:03 PM · Data-Services, cloud-services-team (Kanban)
JHedden added a comment to T235627: Toolforge: upgrade main proxy servers to Debian Buster.

Please @Bstorm and @JHedden confirm the scheduled operation window works for you both, thanks!

Oct 16 2019, 1:31 PM · Toolforge, cloud-services-team (Kanban), Kubernetes

Oct 15 2019

JHedden closed T234770: Prepare and check storage layer for banwiki, a subtask of T234768: Create Balinese Wikipedia, as Resolved.
Oct 15 2019, 1:53 PM · MW-1.35-notes (1.35.0-wmf.2; 2019-10-15), User-Ladsgroup, Wiki-Setup (Create), User-Urbanecm
JHedden closed T234770: Prepare and check storage layer for banwiki as Resolved.
Updates on labsdb10{09,10,11,12}
$ sudo /usr/local/sbin/maintain-replica-indexes --database banwiki --debug
$ sudo /usr/local/sbin/maintain-views --databases banwiki --debug
$ sudo /usr/local/sbin/maintain-meta_p --databases banwiki 
Oct 15 2019, 1:53 PM · cloud-services-team (Kanban), Data-Services, DBA, Operations
JHedden claimed T234770: Prepare and check storage layer for banwiki.
Oct 15 2019, 1:28 PM · cloud-services-team (Kanban), Data-Services, DBA, Operations

Oct 11 2019

JHedden added a comment to T234617: Toolforge. introduce new domain toolforge.org.

The toolforge.org domain has been setup in designate

Oct 11 2019, 9:53 PM · Toolforge, cloud-services-team (Kanban), Kubernetes
JHedden added a comment to T235304: Create a service account to manage toolforge.org. from acme-chief.

Assigned roles for the tools-dns-manager user in eqiad1

Oct 11 2019, 9:26 PM · Toolforge, cloud-services-team (Kanban)

Oct 10 2019

JHedden added a comment to T231793: Remove systemd from openstack-mitaka.

After the Newton upgrade the Stretch hosts areall running the distro version: systemd-sysv 232-25+deb9u11

  • cloudcontrol[1003-1004].wikimedia.org
  • cloudnet[1003-1004].eqiad.wmnet
  • cloudvirt[1001-1009,1012-1014,1016-1030].eqiad.wmnet
Oct 10 2019, 8:12 PM · cloud-services-team (Kanban), Operations
JHedden closed T235127: wmfsink designate handler not running during vm deletes as Resolved.
Oct 10 2019, 4:34 PM · cloud-services-team (Kanban)
JHedden moved T235127: wmfsink designate handler not running during vm deletes from Inbox to Doing on the cloud-services-team (Kanban) board.
Oct 10 2019, 1:49 PM · cloud-services-team (Kanban)

Oct 9 2019

JHedden created T235127: wmfsink designate handler not running during vm deletes.
Oct 9 2019, 8:34 PM · cloud-services-team (Kanban)
JHedden closed T235070: Resolve keystone schema errors, a subtask of T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07, as Invalid.
Oct 9 2019, 2:44 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
JHedden closed T235070: Resolve keystone schema errors as Invalid.

This was a false alarm... The domain_id column was removed in a later revision: https://github.com/openstack/keystone/blob/master/keystone/common/sql/migrate_repo/versions/091_migrate_data_to_local_user_and_password_tables.py#L82

Oct 9 2019, 2:44 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
JHedden updated the task description for T235070: Resolve keystone schema errors.
Oct 9 2019, 2:21 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
JHedden claimed T235070: Resolve keystone schema errors.
Oct 9 2019, 2:15 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
JHedden created T235070: Resolve keystone schema errors.
Oct 9 2019, 1:53 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools

Oct 3 2019

JHedden added a comment to T234518: codfw1dev: glance_rsync_images is failing.

Puppet is not managing the permissions on the secondary controllers image directory.

cloudcontrol2003-dev:~# ls -lad /srv/glance/images
drwxr-xr-x 2 glance glance 4096 Oct  1 21:29 /srv/glance/images
Oct 3 2019, 1:54 PM · cloud-services-team (Kanban)

Oct 2 2019

JHedden added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

Honestly I have no idea where is that 208.80.152.0 coming from.

Oct 2 2019, 1:22 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)

Oct 1 2019

JHedden closed T234373: Tune arp cache garbage collection for Neutron as Resolved.
Oct 1 2019, 9:46 PM · Cloud-VPS
JHedden added a comment to T234373: Tune arp cache garbage collection for Neutron .

to accommodate our current environment and growth I'd like to increase these values on the cloudnet servers to:

Oct 1 2019, 9:03 PM · Cloud-VPS
JHedden created T234373: Tune arp cache garbage collection for Neutron .
Oct 1 2019, 8:56 PM · Cloud-VPS

Sep 27 2019

JHedden added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

I suspect that for database and/or NFS access we need to know actual source IPs though... is that right?

Sep 27 2019, 2:16 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)

Sep 24 2019

JHedden added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

Subnet pools are a base requirement of address scopes, and unfortunately we cannot update existing subnets to use subnet pools on our version of OpenStack.

Sep 24 2019, 8:01 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)

Sep 23 2019

JHedden added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

It looks like the purpose of the routing hacks is to disable NAT between the dmz_cidr's source:destination:

profile::openstack::eqiad1::neutron::dmz_cidr:
 - 172.16.0.0/21:91.198.174.0/24
 - 172.16.0.0/21:198.35.26.0/23
 - 172.16.0.0/21:10.0.0.0/8
 - 172.16.0.0/21:208.80.152.0/22
 - 172.16.0.0/21:103.102.166.0/24
 - 172.16.0.0/21:172.16.0.0/21
Sep 23 2019, 10:30 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)

Sep 19 2019

JHedden closed T221272: Expose new ipblocks.ipb_sitewide column to the replicas as Resolved.

The ipblocks and ipblocks_ipindex views have been rebuilt with the ipb_sitewide field on all of the replicas.

Sep 19 2019, 8:32 PM · cloud-services-team (Kanban), Data-Services, Security-Team, Anti-Harassment
JHedden added a comment to T219374: Prepare and check storage layer for hi.wikisource.

So everything done?
From my side I can connect fine and query the views.

Sep 19 2019, 2:38 PM · Core Platform Team Workboards (Clinic Duty Team), cloud-services-team, Analytics, Data-Services, DBA
JHedden added a comment to T219374: Prepare and check storage layer for hi.wikisource.

@JHedden after your +1, I have merged the change. Can you run puppet and try again to see if that index error is no more?

Sep 19 2019, 2:26 PM · Core Platform Team Workboards (Clinic Duty Team), cloud-services-team, Analytics, Data-Services, DBA
JHedden added a comment to T219374: Prepare and check storage layer for hi.wikisource.

OK, the replica DNS entries are setup for hiwikisource now.

Sep 19 2019, 1:44 PM · Core Platform Team Workboards (Clinic Duty Team), cloud-services-team, Analytics, Data-Services, DBA