Andrew (Andrew Bogott)
User

Projects (11)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Nov 2 2014, 11:35 PM (202 w, 3 d)
Availability
Available
IRC Nick
andrewbogott
LDAP User
Unknown
MediaWiki User
Andrewbogott [ Global Accounts ]

Recent Activity

Tue, Sep 18

Andrew added a comment to T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses.

floating IPs should work now too.

Tue, Sep 18, 1:35 PM · Patch-For-Review, Cloud-Services
Andrew added a comment to T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses.

The above patch fixes PTRs for internal and floating eqiad1 addresses. Floating IPs still need work:

Tue, Sep 18, 3:38 AM · Patch-For-Review, Cloud-Services

Mon, Sep 17

Andrew added a comment to T196507: Degraded RAID on cloudvirt1019.

Thank you for keeping touch with HP about this. This is dumb :(

Mon, Sep 17, 4:09 PM · ops-eqiad, Operations
Andrew added a comment to T202636: Allow routing between eqiad and eqiad1 regions.

@arturo thanks! I'm pondering whether or not we should just do that cloud-wide... maybe just the bastions and the proxies.

Mon, Sep 17, 1:21 PM · Patch-For-Review, Cloud-VPS, cloud-services-team

Fri, Sep 14

Andrew reopened T202636: Allow routing between eqiad and eqiad1 regions as "Open".

This is working, but I note that the originating IP is getting natted when I go between regions:

Fri, Sep 14, 9:35 PM · Patch-For-Review, Cloud-VPS, cloud-services-team
Andrew reopened T202636: Allow routing between eqiad and eqiad1 regions, a subtask of T167293: Nova-network to Neutron migration, as Open.
Fri, Sep 14, 9:35 PM · Patch-For-Review, Epic, Cloud-Services
Andrew added a comment to T204349: Migrated VMs (possibly all VMs) don't appear in the project overview page on Horizon.

Update: VMs created in eqiad1 appear there, but migrated VMs do not.

Fri, Sep 14, 1:39 PM · Epic, Cloud-Services
Andrew triaged T204349: Migrated VMs (possibly all VMs) don't appear in the project overview page on Horizon as Normal priority.
Fri, Sep 14, 1:38 PM · Epic, Cloud-Services

Thu, Sep 13

Andrew closed T191790: Figure out how to migrate an instance from a nova-network region to a neutron region as Resolved.

The 'figure out' parts of this are done now; all that's left is to do it.

Thu, Sep 13, 4:21 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services
Andrew closed T191790: Figure out how to migrate an instance from a nova-network region to a neutron region, a subtask of T167293: Nova-network to Neutron migration, as Resolved.
Thu, Sep 13, 4:20 PM · Patch-For-Review, Epic, Cloud-Services
Andrew closed T204003: Move labvirt1019 and 1020 to eqiad1 as Resolved.
Thu, Sep 13, 12:59 PM · Patch-For-Review, Cloud-VPS

Wed, Sep 12

Andrew added a comment to T204004: Rename labvirt1019 and cloudvirt1020 to cloudvirt1019 and cloudvirt1020.

thanks chris!

Wed, Sep 12, 8:07 PM · Operations, DC-Ops, ops-eqiad

Tue, Sep 11

Andrew added a comment to T204059: Request creation of hashtags VPS project.

Approved, but let's try to hold off for a week so that we can start this in the new region and avoid a future migration

Tue, Sep 11, 3:33 PM · Cloud-VPS (Project-requests)
Andrew added a comment to T196507: Degraded RAID on cloudvirt1019.

I renamed these servers but they're still complaining about missing batteries.

Tue, Sep 11, 3:01 PM · ops-eqiad, Operations
Andrew reassigned T204004: Rename labvirt1019 and cloudvirt1020 to cloudvirt1019 and cloudvirt1020 from Andrew to Cmjohnson.

I reimaged these and made all the puppet/dns changes needed. All that remains is the datacenter bits.

Tue, Sep 11, 1:35 AM · Operations, DC-Ops, ops-eqiad
Andrew updated the task description for T204004: Rename labvirt1019 and cloudvirt1020 to cloudvirt1019 and cloudvirt1020.
Tue, Sep 11, 1:31 AM · Operations, DC-Ops, ops-eqiad
Andrew renamed T196507: Degraded RAID on cloudvirt1019 from Degraded RAID on labvirt1019 to Degraded RAID on cloudvirt1019.
Tue, Sep 11, 1:23 AM · ops-eqiad, Operations
Andrew renamed T194855: Degraded RAID on cloudvirt1020 from Degraded RAID on labvirt1020 to Degraded RAID on cloudvirt1020.
Tue, Sep 11, 1:23 AM · ops-eqiad, Operations

Mon, Sep 10

Andrew created T204004: Rename labvirt1019 and cloudvirt1020 to cloudvirt1019 and cloudvirt1020.
Mon, Sep 10, 10:17 PM · Operations, DC-Ops, ops-eqiad
Andrew created T204003: Move labvirt1019 and 1020 to eqiad1.
Mon, Sep 10, 10:16 PM · Patch-For-Review, Cloud-VPS
Andrew closed T203488: wmcs-roots group not granted access to new eqiad1 region bare metal servers as Resolved.

Do we need to add documentation somewhere for turning up new services/regions about this additional config need?

Mon, Sep 10, 9:54 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS, Puppet
Andrew edited projects for T201439: rename/reimage labnodepool1002.eqiad.wmnet as cloudservices1003.wikimedia.org, added: ops-eqiad; removed Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS.

This is all done except for the physical label changes in eqiad.

Mon, Sep 10, 9:37 PM · ops-eqiad, Operations
Andrew closed T203494: Requesting access to Root for Giovanni Tirloni as Resolved.
Mon, Sep 10, 5:10 PM · Patch-For-Review, SRE-Access-Requests, Operations
Andrew added a comment to T203494: Requesting access to Root for Giovanni Tirloni.

Approved during today's SRE meeting

Mon, Sep 10, 4:52 PM · Patch-For-Review, SRE-Access-Requests, Operations
Andrew updated the task description for T203494: Requesting access to Root for Giovanni Tirloni.
Mon, Sep 10, 4:52 PM · Patch-For-Review, SRE-Access-Requests, Operations
Andrew updated the task description for T203494: Requesting access to Root for Giovanni Tirloni.
Mon, Sep 10, 3:08 PM · Patch-For-Review, SRE-Access-Requests, Operations

Thu, Sep 6

Andrew updated the task description for T203489: Onboard gtirloni to WMF.
Thu, Sep 6, 2:03 PM · Patch-For-Review, Operations, cloud-services-team
Andrew updated the task description for T203489: Onboard gtirloni to WMF.
Thu, Sep 6, 2:03 PM · Patch-For-Review, Operations, cloud-services-team
Andrew added a member for WMF-NDA-Requests: GTirloni.
Thu, Sep 6, 2:02 PM
Andrew added a comment to T203374: Request increased quota for analytics Cloud VPS project.

I'm trying to keep a handle on VM growth right now because that will limit the things I have to migrate in a few weeks... ping me if you run out of quota in the meantime, otherwise let's revisit this post-neutron.

Thu, Sep 6, 1:38 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Wed, Sep 5

Andrew added a comment to T203374: Request increased quota for analytics Cloud VPS project.

I just now looked at this project and it looks to me like there's already enough headroom for 2 more m1.large VMs (and then some). So... we're all set for now, correct?

Wed, Sep 5, 8:44 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Andrew added a comment to T167973: Move wikitech and labstestwiki to s5.

In the end nothing is needed. m5 will not be read only in eqiad. Wikitech traffic will be redirected to eqiad and thus to m5. So nothing to worry about.

Wed, Sep 5, 5:38 PM · wikitech.wikimedia.org, DBA
Andrew added a comment to T201082: labtestweb2001 is sending updates to a read-only db host: db2037.

The context here is that a while ago I moved the local labtestwikitech database off of labtestweb2002 because Jaime asked me to 'productionize' it and I misunderstood his request. I could certainly just move it back there (although that would leave me with the puzzle of what 'productionize' means) or we could move labtestwikitech to db1073 (which would require some kind of tunnel encryption) or... I'm open to suggestions.

Wed, Sep 5, 5:36 PM · Release-Engineering-Team (Watching / External), cloud-services-team, wikitech.wikimedia.org, Wikimedia-production-error
Andrew added a comment to T167973: Move wikitech and labstestwiki to s5.

Hi all! I'm a bit lost because I think this task no longer has anything to do with its original post (which is about moving the databases off of the local wikitech server, long since done but to m5 rather than s5.) If I understand it, there are two different issues under discussion:

Wed, Sep 5, 5:34 PM · wikitech.wikimedia.org, DBA

Tue, Sep 4

Andrew updated the task description for T203489: Onboard gtirloni to WMF.
Tue, Sep 4, 8:43 PM · Patch-For-Review, Operations, cloud-services-team
Andrew updated the task description for T203489: Onboard gtirloni to WMF.
Tue, Sep 4, 8:31 PM · Patch-For-Review, Operations, cloud-services-team
Andrew updated the task description for T203489: Onboard gtirloni to WMF.
Tue, Sep 4, 7:30 PM · Patch-For-Review, Operations, cloud-services-team
Andrew updated the task description for T203489: Onboard gtirloni to WMF.
Tue, Sep 4, 7:21 PM · Patch-For-Review, Operations, cloud-services-team
Andrew updated the task description for T203489: Onboard gtirloni to WMF.
Tue, Sep 4, 7:18 PM · Patch-For-Review, Operations, cloud-services-team
Andrew updated the task description for T203489: Onboard gtirloni to WMF.
Tue, Sep 4, 7:15 PM · Patch-For-Review, Operations, cloud-services-team
Andrew updated subscribers of T203494: Requesting access to Root for Giovanni Tirloni.
Tue, Sep 4, 7:14 PM · Patch-For-Review, SRE-Access-Requests, Operations
Andrew updated the task description for T203494: Requesting access to Root for Giovanni Tirloni.
Tue, Sep 4, 7:13 PM · Patch-For-Review, SRE-Access-Requests, Operations
Andrew created T203494: Requesting access to Root for Giovanni Tirloni.
Tue, Sep 4, 7:12 PM · Patch-For-Review, SRE-Access-Requests, Operations
Andrew updated subscribers of T203489: Onboard gtirloni to WMF.
Tue, Sep 4, 6:58 PM · Patch-For-Review, Operations, cloud-services-team
Andrew claimed T203488: wmcs-roots group not granted access to new eqiad1 region bare metal servers.
Tue, Sep 4, 5:37 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS, Puppet
Andrew claimed T203489: Onboard gtirloni to WMF.
Tue, Sep 4, 5:36 PM · Patch-For-Review, Operations, cloud-services-team
Andrew created T203489: Onboard gtirloni to WMF.
Tue, Sep 4, 5:35 PM · Patch-For-Review, Operations, cloud-services-team
Andrew added a comment to T203374: Request increased quota for analytics Cloud VPS project.

The two new m1.larges are approved -- I'll handle this shortly.

Tue, Sep 4, 3:37 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Andrew added a comment to T203072: Request creation of ign2commons VPS project.

I'm reluctant to create big disks during the eqiad->eqiad1 transition; if y'all can wait a few weeks I'd like you to hold off until we have some hardware capacity in eqiad1 and then we can set you up there. Please ping me later in the month :)

Tue, Sep 4, 3:36 PM · Cloud-VPS (Project-requests)
Andrew added a comment to T202636: Allow routing between eqiad and eqiad1 regions.

Note: I would like to explore neutron firewalling options to limit which traffic is able to transit the virtual router (probably only bastion traffic)

Tue, Sep 4, 3:12 PM · Patch-For-Review, Cloud-VPS, cloud-services-team
Andrew added a comment to T132880: tools.jembot crashes and leaves orphan php-cgi processes regularly.

This bot was just now gobbling up CPU throughout the cluster.

Tue, Sep 4, 2:32 PM · Tools
Andrew moved T201439: rename/reimage labnodepool1002.eqiad.wmnet as cloudservices1003.wikimedia.org from Inbox to To-Do on the cloud-services-team (Kanban) board.
Tue, Sep 4, 3:10 AM · ops-eqiad, Operations

Mon, Sep 3

Andrew closed T202532: After migrating a VM to eqiad1, transfer DNS to point to the new VM as Resolved.

This is all working in the region-migrate script.

Mon, Sep 3, 5:36 PM · cloud-services-team (Kanban), Cloud-Services
Andrew closed T202532: After migrating a VM to eqiad1, transfer DNS to point to the new VM, a subtask of T191790: Figure out how to migrate an instance from a nova-network region to a neutron region, as Resolved.
Mon, Sep 3, 5:36 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services
Andrew closed T202533: During eqiad1 migration, transfer dynamic proxies to point to the post-migrate VM as Resolved.

If we have cross-region routing then we don't need to do this. The migration script already moves each proxy to the migrated VM; later when we actually migrate the proxy project everything should switch over just fine.

Mon, Sep 3, 5:35 PM · cloud-services-team (Kanban), Cloud-Services
Andrew closed T202533: During eqiad1 migration, transfer dynamic proxies to point to the post-migrate VM, a subtask of T191790: Figure out how to migrate an instance from a nova-network region to a neutron region, as Resolved.
Mon, Sep 3, 5:35 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services
Andrew closed Restricted Task, a subtask of T167293: Nova-network to Neutron migration, as Resolved.
Mon, Sep 3, 5:34 PM · Patch-For-Review, Epic, Cloud-Services
Andrew closed T202603: Request creation of graphql VPS project as Resolved.

I apologize -- I'm not ready to create your VMs in the new region, so you should go ahead and create things in eqiad as usual.

Mon, Sep 3, 5:31 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)
Andrew closed T202675: Request creation of antiharassment VPS project as Resolved.
Mon, Sep 3, 5:31 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)
Andrew closed T201035: Delete cloud VPS project mediawiki-docker as Resolved.

deleted!

Mon, Sep 3, 5:29 PM · cloud-services-team (Kanban), Cloud-VPS
Andrew closed T203104: VPS puppet enc 'prefix' field size too small as Resolved.
Mon, Sep 3, 4:10 PM · Patch-For-Review, Cloud-VPS, DBA
Andrew added a comment to T203104: VPS puppet enc 'prefix' field size too small.

Done. Thanks all!

Mon, Sep 3, 4:09 PM · Patch-For-Review, Cloud-VPS, DBA
Andrew updated subscribers of T199374: Delegate public IP range for Eqiad1-r OpenStack deployment to designate (neutron).

After https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/457460/ and fixing a silly typo, I now think that the ptr records should be updating correctly and promptly.

Mon, Sep 3, 3:19 PM · Patch-For-Review, Cloud-Services

Sat, Sep 1

D3r1ck01 awarded Blog Post: Neutron is (finally) coming a Mountain of Wealth token.
Sat, Sep 1, 5:10 PM · Cloud-VPS

Fri, Aug 31

Andrew claimed T199374: Delegate public IP range for Eqiad1-r OpenStack deployment to designate (neutron).

I merged https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/445310/ and the entries created look right to me. I can't actually dig -x yet though, presumable because we still need the delegation patch.

Fri, Aug 31, 8:35 PM · Patch-For-Review, Cloud-Services
Andrew added a comment to T201202: Replace bare-metal promethium.wikitextexp.eqiad.wmflabs with a VM or VMs.

I created a special flavor, 'parsingtest' and a new big VM, 'parsing-qa-01'. That should work as a promethium replacement. It might be a bit hard to migrate things over from the old promethium until T202636 is resolved.

Fri, Aug 31, 7:57 PM · Cloud-VPS (Quota-requests), Parsing-Team, cloud-services-team
Andrew added a comment to T202675: Request creation of antiharassment VPS project.

@dbarratt: I apologize; one of the blockers for moving forward with the new region didn't get done this week and will now be delayed until mid-September. You should go ahead and use the self-serve Horizon interface to get set up in the 'eqiad' region in the meantime; we'll have to migrate you over like everyone else when things are ready.

Fri, Aug 31, 3:53 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)

Wed, Aug 29

Andrew closed T201341: rack/setup/install cloudservices1004.wikimedia.org as Resolved.

puppet is running now. Thank you!

Wed, Aug 29, 7:59 PM · Patch-For-Review, Operations
Krenair awarded T203104: VPS puppet enc 'prefix' field size too small a Cookie token.
Wed, Aug 29, 4:48 PM · Patch-For-Review, Cloud-VPS, DBA
Andrew added a comment to T203104: VPS puppet enc 'prefix' field size too small.

(I altered it to 127 in the meantime so that @Krenair can get on with his work)

Wed, Aug 29, 4:44 PM · Patch-For-Review, Cloud-VPS, DBA
Andrew created T203104: VPS puppet enc 'prefix' field size too small.
Wed, Aug 29, 4:41 PM · Patch-For-Review, Cloud-VPS, DBA
Andrew added a comment to T203031: eqiad1 private IPs lacking PTRs in DNS.

Wouldn't be surprised actually if everything around the nova_fixed_multi stuff needs to be rethought. I have noticed in ./modules/openstack/templates/mitaka/designate/designate.conf.erb there is some commented out neutron settings, I wonder what it does.

Wed, Aug 29, 4:22 AM · Cloud-Services

Tue, Aug 28

Andrew added a comment to T202675: Request creation of antiharassment VPS project.

With the name 'antiharassment' this is approved. You can set up a web proxy with whatever http name you want :)

Tue, Aug 28, 3:44 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)
Andrew added a comment to T202603: Request creation of graphql VPS project.

Seems reasonable -- approved

Tue, Aug 28, 3:41 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)

Mon, Aug 27

chasemp awarded T202636: Allow routing between eqiad and eqiad1 regions a Love token.
Mon, Aug 27, 3:21 PM · Patch-For-Review, Cloud-VPS, cloud-services-team

Sun, Aug 26

Andrew added a comment to T202822: db2088 rebooted itself and came back sick.

I don't see anything in the syslog to warn about the coming crash... it just stops dead at 00:50:01

Sun, Aug 26, 1:50 AM · ops-codfw, Patch-For-Review, Operations, DBA
Andrew added projects to T202822: db2088 rebooted itself and came back sick: DBA, Operations.
Sun, Aug 26, 1:47 AM · ops-codfw, Patch-For-Review, Operations, DBA
Andrew created T202822: db2088 rebooted itself and came back sick.
Sun, Aug 26, 1:46 AM · ops-codfw, Patch-For-Review, Operations, DBA

Fri, Aug 24

Andrew claimed T164123: tools-k8s-master-01 has two floating IPs.
Fri, Aug 24, 7:59 PM · cloud-services-team (Kanban), Operations, Cloud-Services
Andrew closed Restricted Task, a subtask of T167293: Nova-network to Neutron migration, as Resolved.
Fri, Aug 24, 5:38 PM · Patch-For-Review, Epic, Cloud-Services
Krenair awarded Blog Post: Neutron is (finally) coming a Mountain of Wealth token.
Fri, Aug 24, 5:29 PM · Cloud-VPS
Krenair awarded Blog Post: Neutron is (finally) coming a Like token.
Fri, Aug 24, 5:29 PM · Cloud-VPS
Andrew added a comment to T201341: rack/setup/install cloudservices1004.wikimedia.org.

*bump* -- I'm happy to do the OS install &c. if that helps move this along. Thanks!

Fri, Aug 24, 2:28 PM · Patch-For-Review, Operations

Thu, Aug 23

Andrew created T202636: Allow routing between eqiad and eqiad1 regions.
Thu, Aug 23, 3:36 PM · Patch-For-Review, Cloud-VPS, cloud-services-team

Wed, Aug 22

Andrew published Blog Post: Neutron is (finally) coming.
Wed, Aug 22, 9:00 PM · Cloud-VPS
chasemp awarded Blog Post: Neutron is (finally) coming a Love token.
Wed, Aug 22, 1:59 PM · Cloud-VPS
Andrew updated the task description for T202533: During eqiad1 migration, transfer dynamic proxies to point to the post-migrate VM.
Wed, Aug 22, 1:45 PM · cloud-services-team (Kanban), Cloud-Services
Andrew triaged T202533: During eqiad1 migration, transfer dynamic proxies to point to the post-migrate VM as Normal priority.
Wed, Aug 22, 1:44 PM · cloud-services-team (Kanban), Cloud-Services
Andrew triaged T202532: After migrating a VM to eqiad1, transfer DNS to point to the new VM as Normal priority.
Wed, Aug 22, 1:42 PM · cloud-services-team (Kanban), Cloud-Services
Andrew added a comment to T202407: cloudcontrol1003/1004 cronspam.

Does this mean merging nova in both deployments?

Wed, Aug 22, 12:55 PM · Cloud-VPS
Andrew added a comment to T199125: rack/setup/install cloudvirt102[34].

@chasemp two questions: 1) was there a reason we requested these with 10G? (Or, did we?) 2) Is it important that these be in a particular rack for neutron purposes?

Wed, Aug 22, 2:04 AM · Patch-For-Review, cloud-services-team (Kanban), ops-eqiad, Cloud-VPS, Operations

Tue, Aug 21

Andrew added a comment to T201925: Server error when viewing web proxies.

I tested this myself with a few different configurations (including with an account with limiter rights) but I can't reproduce the issue. It looks to me like you're not project admin in any of those projects, so I'd expect you to be able to see a list of proxies but not modify them.

Tue, Aug 21, 7:32 PM · Cloud-Services
Andrew added a comment to T202407: cloudcontrol1003/1004 cronspam.

Ah, I'm wrong, nova still uses a local database. So, to restate: We should move everything currently using local mysql there to m5, then we can uninstall mysql and this issue will go away :)

Tue, Aug 21, 7:15 PM · Cloud-VPS
Andrew added a comment to T202407: cloudcontrol1003/1004 cronspam.

As far as I know, nothing running on that host still uses mysql. If that's right, then the solution is obvious :)

Tue, Aug 21, 7:14 PM · Cloud-VPS
Andrew created T202431: Move labweb hosts from nutcracker to mcrouter?.
Tue, Aug 21, 5:02 PM · Cloud-Services
Andrew assigned T202155: Approve https://packages.sury.org/php/ as an acceptable Debian package source for Toolforge to aborrero.
Tue, Aug 21, 3:32 PM · cloud-services-team, Toolforge
Andrew added a comment to T202155: Approve https://packages.sury.org/php/ as an acceptable Debian package source for Toolforge.

I think our preference would be to import these packages into our repo rather than point at the external repo. Arturo will follow up.

Tue, Aug 21, 3:31 PM · cloud-services-team, Toolforge
Andrew added a comment to T202155: Approve https://packages.sury.org/php/ as an acceptable Debian package source for Toolforge.

I think this is fine. It's not 100% obvious to me what we need to do to implement this; is it just editing a line on a wiki?

Tue, Aug 21, 3:29 PM · cloud-services-team, Toolforge
Andrew updated subscribers of T202407: cloudcontrol1003/1004 cronspam.
Tue, Aug 21, 2:39 PM · Cloud-VPS

Aug 21 2018

Andrew created T202407: cloudcontrol1003/1004 cronspam.
Aug 21 2018, 2:00 PM · Cloud-VPS