RobH (Rob Halsell)
Operations Engineer

Projects (23)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Nov 24 2014, 1:43 PM (186 w, 4 d)
Availability
Available
IRC Nick
RobH
LDAP User
RobH
MediaWiki User
RobH

My GPG Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7 0245 D22A

I am an Operations Engineer on Wikimedia's Datacenter Operations Team.

I also am the primary triage engineer for the hardware-requests project, as well as the private S4 procurement space and procurement project.

All questions involving allocation of hardware can be initially addressed on https://wikitech.wikimedia.org/wiki/Operations_requests.

Please note that private message via phabricator is not my preferred contact means. Please feel free to contact me (robh) directly via irc/freenode, or email my @wikimedia.org email address.

Recent Activity

Yesterday

RobH added a comment to T197857: Add @pmiazga @Niedzielski and @phuedx to the deploy-service group.

Thanks for feedback, duly noted and set in the proper column for SRE meeting review approval.

Thu, Jun 21, 10:47 PM · Patch-For-Review, Proton, Operations, SRE-Access-Requests
RobH renamed T197895: Remove shell access to analytics-privatedata-users for ironholds from Remove shell access to analytics-privatedata-users for DYNKM to Remove shell access to analytics-privatedata-users for ironholds.
Thu, Jun 21, 8:10 PM · Patch-For-Review, Operations, SRE-Access-Requests, Research-collaborations, Research-management, Research
RobH moved T197895: Remove shell access to analytics-privatedata-users for ironholds from Backlog to waiting period on the SRE-Access-Requests board.
Thu, Jun 21, 8:09 PM · Patch-For-Review, Operations, SRE-Access-Requests, Research-collaborations, Research-management, Research
RobH triaged T197895: Remove shell access to analytics-privatedata-users for ironholds as Normal priority.
Thu, Jun 21, 8:09 PM · Patch-For-Review, Operations, SRE-Access-Requests, Research-collaborations, Research-management, Research
RobH moved T197857: Add @pmiazga @Niedzielski and @phuedx to the deploy-service group from In Discussion to SRE Meeting Review Required on the SRE-Access-Requests board.
Thu, Jun 21, 7:15 PM · Patch-For-Review, Proton, Operations, SRE-Access-Requests
RobH updated the task description for T197237: Requesting access for mbsantos.
Thu, Jun 21, 6:39 PM · Analytics, SRE-Access-Requests, Operations
RobH updated subscribers of T197237: Requesting access for mbsantos.

Please note that next Monday's SRE team meeting has been canceled, as the SRE off-site is occurring this week.

Thu, Jun 21, 6:39 PM · Analytics, SRE-Access-Requests, Operations
RobH moved T197857: Add @pmiazga @Niedzielski and @phuedx to the deploy-service group from Backlog to In Discussion on the SRE-Access-Requests board.
Thu, Jun 21, 6:37 PM · Patch-For-Review, Proton, Operations, SRE-Access-Requests
RobH added a comment to T197857: Add @pmiazga @Niedzielski and @phuedx to the deploy-service group.

Just reviewing this as clinic duty this week, and this seems to be a deploy service, but doesn't list sudo rights in data.yaml. Does this include sudo rights deeper within the puppet code for this group, or does this group not act as any user other than the user's own?

Thu, Jun 21, 6:37 PM · Patch-For-Review, Proton, Operations, SRE-Access-Requests
RobH closed T197886: Add kharlan to `wmf` LDAF group as Resolved.

Turns out I was being overly cautious and was able to merge today.

Thu, Jun 21, 4:28 PM · Patch-For-Review, LDAP-Access-Requests
RobH added a comment to T197886: Add kharlan to `wmf` LDAF group.

So, we can add you to the group once we add you to that patchset above. Unfortuantely, we're on a code freeze this week, due to the SRE offsite.

Thu, Jun 21, 4:23 PM · Patch-For-Review, LDAP-Access-Requests

Wed, Jun 20

RobH placed T197237: Requesting access for mbsantos up for grabs.
Wed, Jun 20, 7:01 PM · Analytics, SRE-Access-Requests, Operations
RobH reassigned T197237: Requesting access for mbsantos from herron to greg.

@greg: Would you review and approve/deny deployers access for @mbsantos?

Wed, Jun 20, 6:57 PM · Analytics, SRE-Access-Requests, Operations

Fri, Jun 8

RobH closed T178392: Replacement hardware for cumin masters as Resolved.
Fri, Jun 8, 8:47 PM · hardware-requests, Operations
RobH edited projects for T196175: decom/reclaim tin, added: decommission; removed hardware-requests.
Fri, Jun 8, 8:45 PM · decommission, DC-Ops, ops-eqiad, Operations
RobH removed a project from T196651: rack upgraded storage capacity in labstore100[67].eqiad.wmnet: procurement.
Fri, Jun 8, 6:17 PM · ops-eqiad, Cloud-VPS, Operations
RobH updated subscribers of T196345: eqiad: (1) new stat box to offload users from stat1005.

Please note I'll be away all next week, so if this needs quotation before I return, please chat with @Cmjohnson & @faidon.

Fri, Jun 8, 6:16 PM · hardware-requests, Operations, Analytics
RobH moved T196345: eqiad: (1) new stat box to offload users from stat1005 from Backlog to In Discussion / Review on the hardware-requests board.
Fri, Jun 8, 6:14 PM · hardware-requests, Operations, Analytics
RobH shifted T196345: eqiad: (1) new stat box to offload users from stat1005 from the Restricted Space space to the S1 Public space.
Fri, Jun 8, 6:14 PM · hardware-requests, Operations, Analytics
RobH assigned T196345: eqiad: (1) new stat box to offload users from stat1005 to elukey.

So, the difference between this request, and our current dual cpu misc spec, is we no longer put in 4 * 4TB disks (like stat1006), but dual small SSDs.

Fri, Jun 8, 6:14 PM · hardware-requests, Operations, Analytics
RobH renamed T196345: eqiad: (1) new stat box to offload users from stat1005 from eqiad: hardware request a new stat analytics host to eqiad: (1) new stat box to offload users from stat1005.
Fri, Jun 8, 6:06 PM · hardware-requests, Operations, Analytics
RobH renamed T196345: eqiad: (1) new stat box to offload users from stat1005 from Site: eqiad | hardware request a new stat analytics host to eqiad: hardware request a new stat analytics host.
Fri, Jun 8, 6:05 PM · hardware-requests, Operations, Analytics
RobH triaged T196507: Degraded RAID on labvirt1019 as Normal priority.
Fri, Jun 8, 6:05 PM · ops-eqiad, Operations
RobH updated the task description for T193736: Decommission db1056.
Fri, Jun 8, 3:11 PM · ops-eqiad, Operations, decommission, DBA
RobH reassigned T196651: rack upgraded storage capacity in labstore100[67].eqiad.wmnet from faidon to Cmjohnson.
Fri, Jun 8, 3:01 PM · ops-eqiad, Cloud-VPS, Operations

Thu, Jun 7

RobH added a comment to T196709: replace bad disk in db2059.

I've set this to high priority due to the looming end of fiscal. If this new SAS disk works fine and the raid rebuilds without incident, we'll be ordering a dozen (or more) additional disks for shelf spares.

Thu, Jun 7, 10:56 PM · DBA, Operations, ops-codfw
RobH triaged T196709: replace bad disk in db2059 as High priority.
Thu, Jun 7, 10:54 PM · DBA, Operations, ops-codfw
RobH triaged T196702: Add thcipriani and hashar to gerrit-root as Normal priority.
Thu, Jun 7, 10:42 PM · Patch-For-Review, Operations, Release-Engineering-Team (Kanban), SRE-Access-Requests
RobH triaged T196703: Add thcipriani to phabricator-roots as Normal priority.
Thu, Jun 7, 10:42 PM · Patch-For-Review, SRE-Access-Requests, Release-Engineering-Team (Kanban), Operations
RobH moved T196703: Add thcipriani to phabricator-roots from Backlog to SRE Meeting Review Required on the SRE-Access-Requests board.

This requires SRE team meeting review. Moving to the appropriate column. The next meeting is Monday, June 11th, 2018.

Thu, Jun 7, 10:23 PM · Patch-For-Review, SRE-Access-Requests, Release-Engineering-Team (Kanban), Operations
RobH moved T196702: Add thcipriani and hashar to gerrit-root from Backlog to SRE Meeting Review Required on the SRE-Access-Requests board.

This requires SRE team meeting review. Moving to the appropriate column for that. Next meeting is Monday, June 11th, 2018.

Thu, Jun 7, 10:21 PM · Patch-For-Review, Operations, Release-Engineering-Team (Kanban), SRE-Access-Requests
RobH triaged T196701: rack/setup/install torrelay1001.wikimedia.org as Normal priority.
Thu, Jun 7, 9:13 PM · ops-eqiad, Operations
RobH triaged T196698: rack/setup/install auth1002 as Normal priority.
Thu, Jun 7, 8:57 PM · ops-eqiad, Operations
RobH lowered the priority of T196697: rack/setup/add to spares tracking 2 single cpu misc class systems from High to Normal.
Thu, Jun 7, 8:55 PM · ops-eqiad, Operations
RobH triaged T196697: rack/setup/add to spares tracking 2 single cpu misc class systems as High priority.
Thu, Jun 7, 8:55 PM · ops-eqiad, Operations
RobH triaged T196693: rack/setup/install authdns1001.wikimedia.org as Normal priority.
Thu, Jun 7, 8:35 PM · Traffic, DNS, ops-eqiad, Operations
RobH triaged T196691: rack/setup/install dns100[12].wikimedia.org as Normal priority.
Thu, Jun 7, 8:33 PM · DNS, ops-eqiad, Traffic, Operations
RobH updated the task description for T196489: upgrade all codfw switch stacks to include additional 10G switch per row.
Thu, Jun 7, 8:16 PM · ops-codfw, netops, Operations
RobH triaged T196690: rack/setup/install dbproxy101[2-7].eqiad.wmnet as Normal priority.
Thu, Jun 7, 8:11 PM · ops-eqiad, DBA, Operations
RobH updated the task description for T196685: rack/setup/install rdb10[09|10].eqiad.wmnet.
Thu, Jun 7, 7:41 PM · ops-eqiad, User-Joe, User-Elukey, Operations
RobH added a comment to T196685: rack/setup/install rdb10[09|10].eqiad.wmnet.

I've emailed both @Joe and @elukey regarding the racking locations of these, email below:

Thu, Jun 7, 7:41 PM · ops-eqiad, User-Joe, User-Elukey, Operations
RobH triaged T196685: rack/setup/install rdb10[09|10].eqiad.wmnet as Normal priority.
Thu, Jun 7, 7:36 PM · ops-eqiad, User-Joe, User-Elukey, Operations
RobH triaged T196666: rack/setup/add to spares tracking 2 single cpu misc class systems as Normal priority.
Thu, Jun 7, 5:38 PM · Patch-For-Review, ops-codfw, Operations
RobH triaged T196665: rack/setup/install bast2002.wikimedia.org as Normal priority.
Thu, Jun 7, 5:34 PM · Patch-For-Review, ops-codfw, Operations
RobH triaged T196664: rack/setup/install authdns2001.wikimedia.org as Normal priority.
Thu, Jun 7, 5:33 PM · ops-codfw, Operations
RobH reassigned T196651: rack upgraded storage capacity in labstore100[67].eqiad.wmnet from RobH to Cmjohnson.
Thu, Jun 7, 4:23 PM · ops-eqiad, Cloud-VPS, Operations
RobH updated the task description for T196651: rack upgraded storage capacity in labstore100[67].eqiad.wmnet.
Thu, Jun 7, 4:23 PM · ops-eqiad, Cloud-VPS, Operations
RobH triaged T196651: rack upgraded storage capacity in labstore100[67].eqiad.wmnet as Lowest priority.
Thu, Jun 7, 4:22 PM · ops-eqiad, Cloud-VPS, Operations
RobH added a comment to T195690: Degraded RAID on labsdb1009.

We had next day support until it expired on May 25th. However, if this case was open before hten, they should honor the warranty.

Thu, Jun 7, 3:15 PM · DBA, ops-eqiad, Operations
RobH closed T196616: Reset admin password for wikimk-l as Resolved.

I've gone ahead and reset the password, which emails it out automatically to all the list admins for wikimk-l.

Thu, Jun 7, 3:06 PM · Operations, Wikimedia-Mailing-lists
RobH added a comment to T195690: Degraded RAID on labsdb1009.

We had next day support until it expired on May 25th. However, if this case was open before hten, they should honor the warranty.

Thu, Jun 7, 3:04 PM · DBA, ops-eqiad, Operations

Wed, Jun 6

RobH reassigned T196030: troubleshoot cr3/cr4 link from RobH to ayounsi.

Ok, I'm back onsite today, and I've taken the following steps:

Wed, Jun 6, 10:58 PM · Operations, ops-ulsfo, netops, Traffic
RobH updated the task description for T196566: Requesting deployment access for jforrester.
Wed, Jun 6, 8:23 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Operations, SRE-Access-Requests
RobH added a comment to T196566: Requesting deployment access for jforrester.

I forgot to note that @Jdforrester-WMF put 'deployers' but I assume he meant 'deployment'

Wed, Jun 6, 8:20 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Operations, SRE-Access-Requests
RobH reassigned T196000: move/setup/install labtestnet2003(WMF6469) from RobH to chasemp.

This system is now ready for cloud team to take over. Feel free to use or resolve this task as needed.

Wed, Jun 6, 8:18 PM · Patch-For-Review, Cloud-VPS, Operations
RobH updated the task description for T196000: move/setup/install labtestnet2003(WMF6469).
Wed, Jun 6, 8:17 PM · Patch-For-Review, Cloud-VPS, Operations
RobH removed a project from T196000: move/setup/install labtestnet2003(WMF6469): Patch-For-Review.
Wed, Jun 6, 7:17 PM · Patch-For-Review, Cloud-VPS, Operations
RobH reassigned T190424: modify labs-hosts1-vlans for http load of installer kernel from faidon to ayounsi.

So, this bit me again today, and it would be nice to make these tftp loads via http like the rest of production.

Wed, Jun 6, 7:16 PM · cloud-services-team, netops, Operations
RobH updated the task description for T196566: Requesting deployment access for jforrester.
Wed, Jun 6, 6:08 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Operations, SRE-Access-Requests
RobH moved T196566: Requesting deployment access for jforrester from Backlog to SRE Meeting Review Required on the SRE-Access-Requests board.

Any additions to deployers requires approval by both @greg (for RI) plus review in the SRE weekly meetings.

Wed, Jun 6, 5:53 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Operations, SRE-Access-Requests
RobH triaged T196566: Requesting deployment access for jforrester as Normal priority.
Wed, Jun 6, 5:51 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Operations, SRE-Access-Requests
RobH updated the task description for T196566: Requesting deployment access for jforrester.
Wed, Jun 6, 5:51 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Operations, SRE-Access-Requests
RobH updated the task description for T196566: Requesting deployment access for jforrester.
Wed, Jun 6, 5:50 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Operations, SRE-Access-Requests
RobH added a comment to T196478: rack/setup/install backup1001.

Ok, I'd hold off on OS install until AFTER we get the shelves, just in case we have any issues we see it then.

Wed, Jun 6, 4:42 PM · Patch-For-Review, Operations, ops-eqiad
RobH closed T196120: Create new editing-team mailing list as Resolved.

I've gone ahead and created this list, setting it to private and requiring approval to join it. Since it is a private team list, I turned the archives to private, and then disabled archiving of messages. (Team lists inherently tend to be more informal and do not require archival.) If you decide you want archiving, you can turn it on within the admin panel, click on the archiving options page and enable it.

Wed, Jun 6, 4:30 PM · Operations, Wikimedia-Mailing-lists
RobH closed T192865: Create mediawiki-debian mailing list as Resolved.

This seems to have sat, and the new list is working fine without the old list archives from the third party server. Resolving.

Wed, Jun 6, 4:26 PM · Operations, MediaWiki-Debian, Wikimedia-Mailing-lists
RobH moved T196041: New closed communication public policy mailing list needed from Backlog to List creation on the Wikimedia-Mailing-lists board.
Wed, Jun 6, 4:07 PM · Wikimedia-Mailing-lists, Operations
RobH closed T196438: Reset admin password for wikimedia-mk as Resolved.

The admin password for wikiemdia-mk has been reset and automatically emailed to the list admins.

Wed, Jun 6, 4:06 PM · Operations, Wikimedia-Mailing-lists
RobH added a comment to T196556: Give access to recommender-feedback@wikimedia.org.

WMF OIT handles the actual @wikimedia.org address allocations for staff and a google alias. That isn't a @lists.wikimedia.org address, which is handled via this project.

Wed, Jun 6, 4:04 PM · Office-IT, Mail
RobH added a comment to T196556: Give access to recommender-feedback@wikimedia.org.

I don't see this actual list already created, but it mentions that Ori was the admin? This seems to be asking for a modification, not a new list.

Wed, Jun 6, 3:55 PM · Office-IT, Mail
RobH closed T196192: Add Reedy to contint-docker group as Resolved.

This has been merged live. All affected servers will call into puppet and get the change within the next 30 minutes or so.

Wed, Jun 6, 3:49 PM · Patch-For-Review, SRE-Access-Requests, Operations, Continuous-Integration-Infrastructure

Tue, Jun 5

RobH added a comment to T196192: Add Reedy to contint-docker group.

If no objections are noted, I'll be merging this live tomorrow, Wednesday, 2018-06-06.

Tue, Jun 5, 6:06 PM · Patch-For-Review, SRE-Access-Requests, Operations, Continuous-Integration-Infrastructure
RobH triaged T196493: rack/setup/install dns200[12].wikimedia.org as Normal priority.
Tue, Jun 5, 6:00 PM · Patch-For-Review, Traffic, DNS, ops-codfw, Operations
RobH added a parent task for T196417: Rack/Setup frbast2001.frack.codfw.wmnet: Unknown Object (Task).
Tue, Jun 5, 5:48 PM · Patch-For-Review, ops-codfw, Operations, fundraising-tech-ops
RobH renamed T196489: upgrade all codfw switch stacks to include additional 10G switch per row from upgrade all codfw switch stacks to include additional 10G to upgrade all codfw switch stacks to include additional 10G switch per row.
Tue, Jun 5, 5:46 PM · ops-codfw, netops, Operations
RobH triaged T196489: upgrade all codfw switch stacks to include additional 10G switch per row as Normal priority.
Tue, Jun 5, 5:46 PM · ops-codfw, netops, Operations
RobH triaged T196487: upgrade row d to have 3 10G switches as Normal priority.
Tue, Jun 5, 5:37 PM · ops-eqiad, netops, Operations
RobH triaged T196484: rack/setup/install graphite1004 as Normal priority.
Tue, Jun 5, 5:30 PM · ops-eqiad, Operations
RobH triaged T196483: rack/setup/install graphite2003 as Normal priority.
Tue, Jun 5, 5:30 PM · ops-codfw, Operations
RobH triaged T196478: rack/setup/install backup1001 as Normal priority.
Tue, Jun 5, 4:42 PM · Patch-For-Review, Operations, ops-eqiad
RobH triaged T196477: rack/setup/install backup2001 as Normal priority.
Tue, Jun 5, 4:40 PM · Patch-For-Review, ops-codfw, Operations

Mon, Jun 4

RobH moved T196192: Add Reedy to contint-docker group from Backlog to waiting period on the SRE-Access-Requests board.
Mon, Jun 4, 5:48 PM · Patch-For-Review, SRE-Access-Requests, Operations, Continuous-Integration-Infrastructure
RobH closed T195797: Stas needs root access on WDQS test cluster as Resolved.

These patches have been merged, and should be live shortly on hosts!

Mon, Jun 4, 5:24 PM · Patch-For-Review, Discovery, SRE-Access-Requests, Wikidata, Wikidata-Query-Service, Operations
RobH updated subscribers of T195797: Stas needs root access on WDQS test cluster.

Please note this was approved in today's SRE team meeting. I'll go ahead and start merging @MoritzMuehlenhoff's patches.

Mon, Jun 4, 5:08 PM · Patch-For-Review, Discovery, SRE-Access-Requests, Wikidata, Wikidata-Query-Service, Operations

Sun, Jun 3

RobH added a comment to T196080: Site: eqiad | hardware request for a dedicated stat analytics host for the Research team.

That is not a bad idea. Although moving folks between stat boxes is not the easiest thing to do... :)

Sun, Jun 3, 4:39 PM · Operations, Analytics, hardware-requests

Fri, Jun 1

RobH triaged T196189: rack/setup/install snapshot1009 as Normal priority.
Fri, Jun 1, 7:23 PM · User-ArielGlenn, Patch-For-Review, ops-eqiad, Datasets-General-or-Unknown, Operations
RobH removed a parent task for T195385: Rack and setup snapshot1008: Unknown Object (Task).
Fri, Jun 1, 7:10 PM · Operations, Datasets-General-or-Unknown, Dumps-Generation
RobH moved T196080: Site: eqiad | hardware request for a dedicated stat analytics host for the Research team from Backlog to In Discussion / Review on the hardware-requests board.
Fri, Jun 1, 6:15 PM · Operations, Analytics, hardware-requests
RobH assigned T196080: Site: eqiad | hardware request for a dedicated stat analytics host for the Research team to elukey.

We CANNOT move the GPU between hosts. It is in that chassis (stat1005), specifically ordered to house it. It cannot move into another host. It has to live in that host, due to both warranty and chassis formfactor reasons. The GPU simply won't fit into any other chassis but this singular R730. We have no other R730s.

Fri, Jun 1, 6:04 PM · Operations, Analytics, hardware-requests
Jdforrester-WMF awarded T175288: setup/install/deploy deploy1001 as deployment server a Yellow Medal token.
Fri, Jun 1, 6:03 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Operations

Thu, May 31

RobH closed Unknown Object (Task), a subtask of T181936: Give misc dump crons their own host, as Resolved.
Thu, May 31, 4:57 PM · Patch-For-Review, hardware-requests, Operations, Datasets-General-or-Unknown, Dumps-Generation
RobH closed Unknown Object (Task), a subtask of T184293: rack/setup/install lvs101[3-6], as Resolved.
Thu, May 31, 4:40 PM · Patch-For-Review, ops-eqiad, Operations, Traffic
RobH closed Unknown Object (Task), a subtask of T189822: Replace 5 Samsung SSD 850 devices w/ 4 1.6T Intel or HP SSDs, as Resolved.
Thu, May 31, 4:40 PM · User-fgiunchedi, Services (blocked), Operations, Cassandra, User-Eevans
RobH closed Unknown Object (Task), a subtask of T185494: Degraded RAID on restbase-dev1006, as Resolved.
Thu, May 31, 4:38 PM · User-Eevans, ops-eqiad, Operations
RobH closed Unknown Object (Task), a subtask of T175738: Long term storage for frack prometheus data, as Resolved.
Thu, May 31, 4:32 PM · Operations, fundraising-tech-ops
RobH closed Unknown Object (Task), a subtask of T184704: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw, as Resolved.
Thu, May 31, 4:31 PM · Patch-For-Review, DBA
RobH closed Unknown Object (Task), a subtask of T183145: Refresh SWAP notebook hardware, as Resolved.
Thu, May 31, 4:30 PM · Patch-For-Review, Analytics-Kanban
RobH renamed T196000: move/setup/install labtestnet2003(WMF6469) from move/setup/install labtestnet2002(WMF6469) to move/setup/install labtestnet2003(WMF6469).
Thu, May 31, 4:05 PM · Patch-For-Review, Cloud-VPS, Operations
RobH added a comment to T196000: move/setup/install labtestnet2003(WMF6469).

My bad, updating task and lets call this labtestnet2003. This is to replace labtestnet2001, and I assumed it would be 2002, my bad!

Thu, May 31, 4:04 PM · Patch-For-Review, Cloud-VPS, Operations
RobH added a comment to T196030: troubleshoot cr3/cr4 link.

Probably still need to swap RX/TX.

Thu, May 31, 3:16 PM · Operations, ops-ulsfo, netops, Traffic