Page MenuHomePhabricator

nskaggs ( Nicholas Skaggs)
User

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Jun 16 2020, 4:12 PM (105 w, 6 d)
Availability
Available
IRC Nick
balloons
LDAP User
Nskaggs
MediaWiki User
NSkaggs (WMF) [ Global Accounts ]

Recent Activity

Wed, Jun 15

nskaggs committed rLTMK4fe2f282f225: Update tests for python3 (authored by nskaggs).
Update tests for python3
Wed, Jun 15, 12:37 PM

Tue, Jun 14

nskaggs updated the task description for T310640: wmcs-cinder-backup-manager: TypeError: 'TupleWithMeta' object is not callable.
Tue, Jun 14, 4:49 PM · cloud-services-team (Kanban)
nskaggs triaged T310640: wmcs-cinder-backup-manager: TypeError: 'TupleWithMeta' object is not callable as Medium priority.
Tue, Jun 14, 4:49 PM · cloud-services-team (Kanban)
nskaggs created T310640: wmcs-cinder-backup-manager: TypeError: 'TupleWithMeta' object is not callable.
Tue, Jun 14, 4:48 PM · cloud-services-team (Kanban)

Mon, Jun 13

nskaggs added a comment to T304888: Q4: (Need By: TBD) rack/setup/install 6 wmcs hosts.

Filed T310546 and T310547 to free ports and allow cloudnet1005 and cloudnet1006 connections to cloudsw1*.

Mon, Jun 13, 8:20 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs added a project to T304096: move cloudcephmon1002.eqiad.wmnet from rack B4 to rack D5: cloud-services-team (Hardware).
Mon, Jun 13, 8:19 PM · cloud-services-team (Hardware), SRE, ops-eqiad, User-dcaro, Cloud-Services, DC-Ops
nskaggs updated subscribers of T310547: Recable cloudcephosd1015 from cloudsw1-d5-eqiad to cloudsw2-d5-eqiad .
Mon, Jun 13, 8:18 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs renamed T310546: Recable cloudcephosd1021 from cloudsw1-c8-eqiad to cloudsw2-c8-eqiad from Move network on cloudcephosd1021 from cloudsw1-c8-eqiad to cloudsw2-c8-eqiad to Recable cloudcephosd1021 from cloudsw1-c8-eqiad to cloudsw2-c8-eqiad.
Mon, Jun 13, 8:18 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs renamed T310547: Recable cloudcephosd1015 from cloudsw1-d5-eqiad to cloudsw2-d5-eqiad from Move network connections on cloudcephosd1015 from cloudsw1-d5-eqiad to cloudsw2-d5-eqiad to Recable cloudcephosd1015 from cloudsw1-d5-eqiad to cloudsw2-d5-eqiad .
Mon, Jun 13, 8:18 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs updated the task description for T310547: Recable cloudcephosd1015 from cloudsw1-d5-eqiad to cloudsw2-d5-eqiad .
Mon, Jun 13, 8:17 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs updated the task description for T310546: Recable cloudcephosd1021 from cloudsw1-c8-eqiad to cloudsw2-c8-eqiad.
Mon, Jun 13, 8:17 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs moved T310546: Recable cloudcephosd1021 from cloudsw1-c8-eqiad to cloudsw2-c8-eqiad from Backlog to Lower Priority Items on the ops-eqiad board.
Mon, Jun 13, 8:16 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs edited projects for T310547: Recable cloudcephosd1015 from cloudsw1-d5-eqiad to cloudsw2-d5-eqiad , added: cloud-services-team (Hardware); removed cloud-services-team (Kanban).
Mon, Jun 13, 8:16 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs moved T310547: Recable cloudcephosd1015 from cloudsw1-d5-eqiad to cloudsw2-d5-eqiad from Backlog to Lower Priority Items on the ops-eqiad board.
Mon, Jun 13, 8:15 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs triaged T310547: Recable cloudcephosd1015 from cloudsw1-d5-eqiad to cloudsw2-d5-eqiad as Low priority.
Mon, Jun 13, 8:15 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs triaged T310546: Recable cloudcephosd1021 from cloudsw1-c8-eqiad to cloudsw2-c8-eqiad as Low priority.
Mon, Jun 13, 8:15 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs awarded T302981: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts a Stroopwafel token.
Mon, Jun 13, 2:20 PM · Patch-For-Review, Infrastructure-Foundations, SRE, ops-eqiad, netops, cloud-services-team (Hardware), DC-Ops

Thu, Jun 9

nskaggs added a comment to T304989: Finalise design extension of WMCS networks to new cloudsw in Eqiad rows E/F.

Thanks for the explanation. I just want to make sure if not a cookbook, then a runbook at least to make it very simple for an SRE to run. If the change ever needs to be executed, it would require a timely response and I want to ensure neither you nor any other one specific person would have to be required to do so. Still, I agree ideally if the change needed to be done you would be the one to execute it.

Thu, Jun 9, 8:13 PM · Patch-For-Review, SRE, Infrastructure-Foundations, netops

Wed, Jun 8

nskaggs added a comment to T304989: Finalise design extension of WMCS networks to new cloudsw in Eqiad rows E/F.

@cmooney , for the manual override, https://wikitech.wikimedia.org/wiki/Network_design_-_Eqiad_WMCS_Network_Infra#Manual_Intervention, who can perform the ovverride? Is it possible to have a cookbook to do this? Or otherwise make it "easy" or more accessible? Or would this always remain a more complex networking operation? I'm trying to understand how quickly we could recover should we lose a link. Thanks!

Wed, Jun 8, 4:45 PM · Patch-For-Review, SRE, Infrastructure-Foundations, netops

Tue, Jun 7

nskaggs added a comment to T310103: Cinder Backups: deleting snapshot failed, dependent volumes.

Note, the sql timeout which occurred after just over 24 hours of realtime.

Tue, Jun 7, 9:52 PM · cloud-services-team (Kanban)
nskaggs added a project to T310103: Cinder Backups: deleting snapshot failed, dependent volumes: cloud-services-team (Kanban).
Tue, Jun 7, 9:43 PM · cloud-services-team (Kanban)
nskaggs created T310103: Cinder Backups: deleting snapshot failed, dependent volumes.
Tue, Jun 7, 9:42 PM · cloud-services-team (Kanban)
nskaggs moved T304096: move cloudcephmon1002.eqiad.wmnet from rack B4 to rack D5 from Backlog to Lower Priority Items on the ops-eqiad board.
Tue, Jun 7, 7:36 PM · cloud-services-team (Hardware), SRE, ops-eqiad, User-dcaro, Cloud-Services, DC-Ops
nskaggs triaged T304096: move cloudcephmon1002.eqiad.wmnet from rack B4 to rack D5 as Low priority.
Tue, Jun 7, 7:35 PM · cloud-services-team (Hardware), SRE, ops-eqiad, User-dcaro, Cloud-Services, DC-Ops
nskaggs renamed T304096: move cloudcephmon1002.eqiad.wmnet from rack B4 to rack D5 from move cloudcephmon1003.eqiad.wmnet from rack B2 to rack C8 to move cloudcephmon1002.eqiad.wmnet from rack B4 to rack D5.
Tue, Jun 7, 7:33 PM · cloud-services-team (Hardware), SRE, ops-eqiad, User-dcaro, Cloud-Services, DC-Ops
nskaggs added a comment to T308925: Investigate/move renderer to ubuntu container.

Copying in question from IRC:

Tue, Jun 7, 7:16 PM · Wikimedia-Hackathon-2022, PAWS
nskaggs updated subscribers of T310097: Webservices broken on buster grid.
Tue, Jun 7, 6:54 PM · Tools, cloud-services-team (Kanban)
nskaggs renamed T310097: Webservices broken on buster grid from Tools broken on buster grid to Webservices broken on buster grid.
Tue, Jun 7, 6:52 PM · Tools, cloud-services-team (Kanban)
nskaggs changed the status of T310097: Webservices broken on buster grid from Open to In Progress.
Tue, Jun 7, 6:50 PM · Tools, cloud-services-team (Kanban)
nskaggs added a comment to P29377 Tools Broken on Buster.

Moved into ticket for work and tracking: T310097

Tue, Jun 7, 6:49 PM
nskaggs created T310097: Webservices broken on buster grid.
Tue, Jun 7, 6:49 PM · Tools, cloud-services-team (Kanban)

Mon, Jun 6

nskaggs closed T309978: sal.toolforge.org is down as Resolved.

Ran webservice restart, and sal.toolforge.org is up again.

Mon, Jun 6, 8:32 PM · Tools
nskaggs updated subscribers of T260223: Kiwix rsyncs not completing and stacking up on labstore1006,7.

Adding in @Andrew

Mon, Jun 6, 1:12 PM · affects-Kiwix-and-openZIM, Dumps-Generation, cloud-services-team (Kanban)
nskaggs added a comment to P29377 Tools Broken on Buster.

For posterity, to find broken candidates, look at the cron of started services.

Mon, Jun 6, 1:11 PM

Fri, Jun 3

nskaggs added a comment to P29377 Tools Broken on Buster.

More things that restarted in the last hour (when they should have been stable)... Likely broken candidates, but I didn't check

Fri, Jun 3, 9:31 PM
nskaggs closed T309821: Buster webservice grid went BOOM! as Resolved.

I'm going to try and set expectations here and say this part of the incident is closed / resolved. Hopefully we don't have to re-open!

Fri, Jun 3, 9:29 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
nskaggs closed T309821: Buster webservice grid went BOOM!, a subtask of T277653: Toolforge: add Debian Buster to the grid and eliminate Debian Stretch, as Resolved.
Fri, Jun 3, 9:29 PM · Patch-For-Review, Toolforge, cloud-services-team (Kanban)
nskaggs created P29378 (An Untitled Masterwork).
Fri, Jun 3, 7:11 PM
nskaggs created P29377 Tools Broken on Buster.
Fri, Jun 3, 6:55 PM
nskaggs edited P29375 (An Untitled Masterwork).
Fri, Jun 3, 6:51 PM
nskaggs edited P29375 (An Untitled Masterwork).
Fri, Jun 3, 6:47 PM
nskaggs edited P29375 (An Untitled Masterwork).
Fri, Jun 3, 6:46 PM
nskaggs edited P29375 (An Untitled Masterwork).
Fri, Jun 3, 6:44 PM
nskaggs created P29375 (An Untitled Masterwork).
Fri, Jun 3, 6:40 PM
nskaggs edited P29374 (An Untitled Masterwork).
Fri, Jun 3, 6:26 PM
nskaggs edited P29374 (An Untitled Masterwork).
Fri, Jun 3, 6:25 PM
nskaggs edited P29374 (An Untitled Masterwork).
Fri, Jun 3, 6:16 PM
nskaggs created P29374 (An Untitled Masterwork).
Fri, Jun 3, 6:13 PM
nskaggs created P29373 (An Untitled Masterwork).
Fri, Jun 3, 5:24 PM
nskaggs added a comment to T309821: Buster webservice grid went BOOM!.

Upon further review, after the new buster hosts began acting up again, with OOM errors, even though free -m showed memory. After investigation, dcaro noted that there was a 24Mb swap partition on each. Adding temporary 1G swap space seems to have removed the errors.

Fri, Jun 3, 3:54 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
nskaggs added a comment to T309821: Buster webservice grid went BOOM!.

Noted dpkg was broken, and tools-sgeweblight-10-9 and tools-sgeweblight-10-10 were missing grid service, etc. Fixed dpkg, re-ran puppet to bring back online.

Fri, Jun 3, 12:20 AM · Patch-For-Review, cloud-services-team (Kanban), Toolforge

Thu, Jun 2

nskaggs claimed T309340: maintain-kubeusers tests fail with python 3.10.
Thu, Jun 2, 10:12 PM · cloud-services-team (Kanban), Toolforge
nskaggs updated subscribers of T57503: Mirror more Kiwix downloads directories.

Looping in @Andrew. @Kelson note that yes, we are installing new, more capable machines that have more capacity than in years past. Once they are up and running, we can explore mirroring this additional data.

Thu, Jun 2, 9:29 PM · cloud-services-team (Kanban), Cloud-Services, affects-Kiwix-and-openZIM, SRE, Datasets-General-or-Unknown
nskaggs added a comment to T308748: [toolforge] Split toolforge cli from toolforge build subcommand.

Given this change, the expectation would be new subcommands (like toolforge-build) are implemented as separate binaries? Or is this only expected for toolforge-build? Any thoughts on toolforge-build also being written in golang?

Thu, Jun 2, 9:20 PM · User-Slst2020, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
nskaggs added a comment to T309342: Failed to bind the UNIX domain socket to '/var/run/ceph/guests/ceph-client*.asok.

Good catch! It does seem to be a simple permission error. I'm curious if any other behavior changes will be observed once this is corrected.

Thu, Jun 2, 9:01 PM · User-dcaro, cloud-services-team (Kanban)
nskaggs added a comment to T53434: Establish an internal system or a recommended external system for monitoring user-created Toolforge web services.

Might this explain the credential issues? https://wikitech.wikimedia.org/wiki/Grafana.wikimedia.org#Editing_dashboards. You need the right ldap group, all of which requires NDA. I believe T295296 mentions this. I can help with https://wikitech.wikimedia.org/wiki/Volunteer_NDA if this is the only blocker.

Thu, Jun 2, 8:53 PM · cloud-services-team (Kanban), User-Matthewrbowker, community-labs-monitoring, Toolforge
nskaggs added a project to T57503: Mirror more Kiwix downloads directories: cloud-services-team (Kanban).
Thu, Jun 2, 8:44 PM · cloud-services-team (Kanban), Cloud-Services, affects-Kiwix-and-openZIM, SRE, Datasets-General-or-Unknown
nskaggs added a comment to T304888: Q4: (Need By: TBD) rack/setup/install 6 wmcs hosts.

@cmooney Let's arrange to move some machines so we can have more optimal routing. @dcaro, do you think it would be easier to move a cephosd versus draining and migrating a cloudvirt? We could move cloudcephosd1015 and cloudcephosd1021. Otherwise, I would suggest moving cloudvirts. cloudvirt1046 and cloudvirt1035 respectively.

Thu, Jun 2, 8:43 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
nskaggs added a comment to T309576: Degraded RAID on cloudnet1004.

Yes, I agree. Let's focus on bringing the new machines online.

Thu, Jun 2, 8:35 PM · cloud-services-team (Kanban), SRE, ops-eqiad

May 26 2022

nskaggs closed T309202: Request increased quota for commtech Cloud VPS project as Resolved.
May 26 2022, 6:20 PM · Community-Tech, Cloud-VPS (Quota-requests)
nskaggs updated subscribers of T309342: Failed to bind the UNIX domain socket to '/var/run/ceph/guests/ceph-client*.asok.
May 26 2022, 6:18 PM · User-dcaro, cloud-services-team (Kanban)
nskaggs triaged T309342: Failed to bind the UNIX domain socket to '/var/run/ceph/guests/ceph-client*.asok as Lowest priority.
May 26 2022, 6:17 PM · User-dcaro, cloud-services-team (Kanban)
nskaggs created T309342: Failed to bind the UNIX domain socket to '/var/run/ceph/guests/ceph-client*.asok.
May 26 2022, 6:17 PM · User-dcaro, cloud-services-team (Kanban)
nskaggs added a comment to T309202: Request increased quota for commtech Cloud VPS project.

+1 from me.

May 26 2022, 5:31 PM · Community-Tech, Cloud-VPS (Quota-requests)
nskaggs reassigned T302981: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts from nskaggs to Andrew.
May 26 2022, 2:38 PM · Patch-For-Review, Infrastructure-Foundations, SRE, ops-eqiad, netops, cloud-services-team (Hardware), DC-Ops

May 25 2022

nskaggs added a project to T305280: Prepare and check storage layer for kcgwiki: Data-Engineering.
May 25 2022, 4:20 PM · cloud-services-team (Kanban), Data-Services, DBA
nskaggs added a comment to T299574: Q3:(Need By: TBD) rack/setup/install cloudvirt10[48-50].eqiad.wmnet.

I believe this task is also now cautiously ready to proceed with the finalization of the design in T304989. @cmooney can you confirm?

May 25 2022, 2:55 PM · cloud-services-team (Hardware), SRE, ops-eqiad, DC-Ops
nskaggs added a comment to T302981: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts.

Just wondering on the status of these machines. Anything I can help with?

May 25 2022, 2:51 PM · Patch-For-Review, Infrastructure-Foundations, SRE, ops-eqiad, netops, cloud-services-team (Hardware), DC-Ops

May 10 2022

nskaggs updated subscribers of T304888: Q4: (Need By: TBD) rack/setup/install 6 wmcs hosts.

I was wondering why some of them were spread the way they were (aka outside WMCS dedicated racks), but I see @cmooney updated the racking details with this intention. So yes, please proceed with racking.

May 10 2022, 4:38 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops

May 6 2022

nskaggs added a comment to T301949: ToolsDB upgrade => Bullseye, MariaDB 10.4.

@Ladsgroup Thank you very much for these insights! Please feel free to share any other thoughts on how to improve the service performance or our ability to maintain it.

May 6 2022, 3:39 PM · Data-Persistence (Consultation), Cloud-VPS (Debian Stretch Deprecation), cloud-services-team (Kanban), Toolforge, Data-Services
nskaggs awarded T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye a Party Time token.
May 6 2022, 2:59 PM · Patch-For-Review, Cloud-VPS (Debian Stretch Deprecation), Release-Engineering-Team (Seen), Continuous-Integration-Infrastructure
nskaggs added a comment to T305631: cloudvirt1016: sudden reboot.

You can file a troubleshooting ticket with https://phabricator.wikimedia.org/maniphest/task/edit/form/55/, after reviewing https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Hardware_Troubleshooting_Runbook

May 6 2022, 2:54 PM · cloud-services-team (Hardware)

May 4 2022

nskaggs added a comment to T305974: Provide wmf-pt-kill on Debian Bullseye.

@Marostegui Thank you for following up with upstream here! Perhaps they can ultimately produce a fix or resolution. Until then, yes, let's keep utilizing our existing version. Fingers crossed we can one day removing our custom package.

May 4 2022, 9:24 PM · cloud-services-team (Kanban), DBA

May 3 2022

nskaggs changed the status of T307482: Quarry running very slowly from Open to In Progress.
May 3 2022, 5:34 PM · Quarry, cloud-services-team (Kanban)

May 2 2022

nskaggs added a comment to T306324: Consider improving quota workflow.

@valhallasw Sorry to hear you ran into limitations during your migration! We can consider revising the default. What level would have worked for you?

May 2 2022, 7:53 PM · cloud-services-team (Kanban), Toolforge
nskaggs added a comment to T305831: Cloud VPS: evaluate if VM name global uniqueness enforcement can be dropped.

I think we can move towards dropping this restriction. It'll mean one less hack against upstream nova. Will need to look at and test some things about resolv.conf though.

May 2 2022, 5:42 PM · Cloud-VPS, cloud-services-team (Kanban)
nskaggs added a comment to T305974: Provide wmf-pt-kill on Debian Bullseye.

Would it be possible to use the upstream package at this point? pt-kill is in debian. I'm not sure what patches are being applied beyond the debian version and couldn't find anything beyond: https://phabricator.wikimedia.org/T183983#3983899. Can someone help explain some context for this package? Thanks!

May 2 2022, 5:27 PM · cloud-services-team (Kanban), DBA

Apr 28 2022

nskaggs closed T306130: Hypervisor hardware config for 2022 and beyond as Resolved.
Apr 28 2022, 2:56 PM · cloud-services-team (Kanban)
nskaggs added a comment to T306130: Hypervisor hardware config for 2022 and beyond.

@wiki_willy Thanks for confirming we can order the same spec machine in the R440 chassis. Feel free to update config G accordingly.

Apr 28 2022, 2:55 PM · cloud-services-team (Kanban)

Apr 21 2022

nskaggs updated subscribers of T306130: Hypervisor hardware config for 2022 and beyond.

@wiki_willy @RobH Can you and team confirm our most recent specification (T303446) could be ordered in an R440 chassis? In particular, paying attention to the previous issues (T201352#4671220) such as power requirements being met? If so, we can transition to an R440/450 chassis depending on pricing.

Apr 21 2022, 3:59 PM · cloud-services-team (Kanban)

Apr 20 2022

nskaggs added a comment to T306130: Hypervisor hardware config for 2022 and beyond.

As to why R640 was the standard default, see comments here: https://phabricator.wikimedia.org/T201352#4671220. In short power requirements were the reason.

Apr 20 2022, 4:49 PM · cloud-services-team (Kanban)

Apr 15 2022

nskaggs added a comment to T306101: Cloud VPS "traffic" project Stretch deprecation.

I opened T306245 for diffscan, it's actively in use so please don't delete it. 2 weeks head's up seems a bit short.

Apr 15 2022, 2:52 PM · Cloud-VPS (Debian Stretch Deprecation)

Apr 13 2022

nskaggs created T306130: Hypervisor hardware config for 2022 and beyond.
Apr 13 2022, 7:57 PM · cloud-services-team (Kanban)
nskaggs added a comment to T304881: Q3:(Need By: TBD) rack/setup/install 7 wmcs hosts.

@Papaul By default for HA purposes, we include language to spread servers out when needed. However, given these machines are in dev, and not production, you can safely ignore that request to share racks. Especially if it makes it easier / more convenient for you and team to manage. Thanks for asking. I hope relaxing this requirement helps!

Apr 13 2022, 6:29 PM · Patch-For-Review, SRE, cloud-services-team (Hardware), ops-codfw, DC-Ops

Mar 31 2022

nskaggs moved T299574: Q3:(Need By: TBD) rack/setup/install cloudvirt10[48-50].eqiad.wmnet from Backlog to Racking / Decom on the cloud-services-team (Hardware) board.
Mar 31 2022, 7:50 PM · cloud-services-team (Hardware), SRE, ops-eqiad, DC-Ops

Mar 29 2022

nskaggs added a comment to T304905: Request increased quota for giftbot Toolforge tool.

+1 from me. Thank you for migrating over from gridengine!

Mar 29 2022, 9:32 PM · Toolforge (Quota-requests)
nskaggs awarded T302855: cloudcontrol1005 - Check unit status of backup_cinder_volumes a Burninate token.
Mar 29 2022, 5:11 PM · Patch-For-Review, Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro

Mar 23 2022

nskaggs added a comment to T215217: deployment-prep: Code stewardship request.

Should we consider it necessary to have support for longer, https://deb.freexian.com/extended-lts/ could be an option. Note, both the timeframe and specific support would have to be defined. I would also caution this is NOT a "solution" to avoiding upgrading these instances. However, it could be part of a plan to upgrade if needed.

Mar 23 2022, 7:18 PM · Release-Engineering-Team (Radar), Beta-Cluster-Infrastructure, Code-Stewardship-Reviews

Mar 14 2022

nskaggs assigned T302981: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts to RobH.

Thanks Arzhel! I don't believe anything else is needed from me. Assigning back to @RobH. Feel free to ping again if I missed something!

Mar 14 2022, 7:43 PM · Patch-For-Review, Infrastructure-Foundations, SRE, ops-eqiad, netops, cloud-services-team (Hardware), DC-Ops
nskaggs placed T302981: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts up for grabs.
Mar 14 2022, 7:40 PM · Patch-For-Review, Infrastructure-Foundations, SRE, ops-eqiad, netops, cloud-services-team (Hardware), DC-Ops
nskaggs placed T260647: Rename account Zoranzoki21 to Kizule on Gerrit up for grabs.
Mar 14 2022, 7:35 PM · Gerrit, wikitech.wikimedia.org, LDAP

Mar 10 2022

nskaggs updated subscribers of T302981: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts.

@Jclark-ctr I would want confirmation from infa foundations that all the necessary network connectivity is present. From what I understand, these machines need the public1 VLAN. And need to serve public traffic for dumps, and NFS traffic to cloud and analytics (data engineering), amongst other things. Assuming the new rows are "the same" as the old rows, it should be fine. But I'll let others confirm. @cmooney @ayounsi can you help confirm?

Mar 10 2022, 6:20 PM · Patch-For-Review, Infrastructure-Foundations, SRE, ops-eqiad, netops, cloud-services-team (Hardware), DC-Ops

Mar 9 2022

nskaggs added a comment to T302981: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts.

Note, the existing machines are taking up a total of 12U (6U each) in D2 and A4.

Mar 9 2022, 8:48 PM · Patch-For-Review, Infrastructure-Foundations, SRE, ops-eqiad, netops, cloud-services-team (Hardware), DC-Ops
nskaggs added a comment to T302981: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts.

@RobH I updated the task to call out these should be installed into two different rows, as well as not installing in WMCS specific racks. These machines host dumps, and manage NFS exports for both cloud and data engineering. They don't utilize a cloud specific VLAN. The existing boxes are racked in non-WMCS specific racks.

Mar 9 2022, 8:46 PM · Patch-For-Review, Infrastructure-Foundations, SRE, ops-eqiad, netops, cloud-services-team (Hardware), DC-Ops
nskaggs updated the task description for T302981: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts.
Mar 9 2022, 8:44 PM · Patch-For-Review, Infrastructure-Foundations, SRE, ops-eqiad, netops, cloud-services-team (Hardware), DC-Ops
nskaggs closed T301592: what to do with wmfcloud.org as Resolved.

For now, we will hold the domain and prevent misuse.

Mar 9 2022, 8:30 PM · cloud-services-team (Kanban)
nskaggs awarded T302901: Request increased quota for ipfs Cloud VPS project a Stroopwafel token.
Mar 9 2022, 8:29 PM · IPFS, Cloud-VPS (Quota-requests)
nskaggs added a comment to T300784: Request creation of gsc-analysis VPS project.

@SCherukuwada Did Toolforge work for your needs? Or would you prefer a cloud vps project? Let us know and we'll create the project.

Mar 9 2022, 3:23 PM · Cloud-VPS (Project-requests)
nskaggs added a comment to T300160: Request increased quota for maps Cloud VPS project.

@dschwen How are things going? Anything further we can help with?

Mar 9 2022, 3:19 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
nskaggs added a comment to T293391: Q2:(Need By: TBD) rack/setup/install cloudvirt1047.eqiad.wmnet.

@Cmjohnson What's the status of imaging this box? Why did it fail?

Mar 9 2022, 3:07 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops