RobH (Rob Halsell)Administrator
Operations Engineer

Projects (23)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Nov 24 2014, 1:43 PM (212 w, 1 d)
Roles
Administrator
Availability
Available
IRC Nick
RobH
LDAP User
RobH
MediaWiki User
RobH [ Global Accounts ]

My GPG Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7 0245 D22A

I am an Operations Engineer on Wikimedia's Datacenter Operations Team.

I also am the primary triage engineer for the hardware-requests project, as well as the private S4 procurement space and procurement project.

All questions involving allocation of hardware can be initially addressed on https://wikitech.wikimedia.org/wiki/Operations_requests.

Please note that private message via phabricator is not my preferred contact means. Please feel free to contact me (robh) directly via irc/freenode, or email my @wikimedia.org email address.

Recent Activity

Yesterday

RobH added a comment to T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use.

Oh, the firmware needs to be updated. We've done this without downtime in the past on other servertechs, but since I'll be onsite to install the other brackets, I'll flash then. (Better safe than sorry and if something went wrong I'd be in the car for an hour before arriving there due to traffic.)

Tue, Dec 18, 11:46 PM · Patch-For-Review, Operations, ops-ulsfo
RobH removed a project from T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use: Patch-For-Review.
Tue, Dec 18, 9:03 PM · Patch-For-Review, Operations, ops-ulsfo
RobH added a comment to T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use.

Both PDU sets are online via mgmt interfaces and can be remotely administered. I'm getting the same error code on serial, will troubleshoot it remotely. Since mgmt network works, serial working is low priority.

Tue, Dec 18, 8:33 PM · Patch-For-Review, Operations, ops-ulsfo
RobH added a comment to T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use.

I'm keeping this open to track the additional steps of adding the brackets to the ps1 in each cabinet (only ps2 has them presently) and also then auditing and labeling every power connection for remote port administration on the PDUs.

Tue, Dec 18, 8:13 PM · Patch-For-Review, Operations, ops-ulsfo
RobH updated the task description for T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use.
Tue, Dec 18, 8:13 PM · Patch-For-Review, Operations, ops-ulsfo
RobH removed a project from T209618: rack/setup/install ms-be10[44-50].eqiad.wmnet: ops-eqiad.
Tue, Dec 18, 5:23 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
RobH closed T207140: Add maint-announce@ to Equinix's recipient list for eqsin incidents as Resolved.

So yesterday eqsin sent out the email subject: COMPLETED - Non-Critical Upstream Provider Maintenance-SG Metro Area Network Maintenance -16-DEC-2018 [5-179367337587] and it arrived to the maint announce group successfully. Resolving this task since now we get these notices to both my inbox (like before) and now properly receive independently to maint-announce group.

Tue, Dec 18, 4:45 PM · Wikimedia-Incident, Traffic, Operations
RobH closed T207140: Add maint-announce@ to Equinix's recipient list for eqsin incidents, a subtask of T206861: Power incident in eqsin, as Resolved.
Tue, Dec 18, 4:45 PM · Wikimedia-Incident, Traffic, Operations
RobH claimed T199321: Return graphite200[12] to spares pool.
Tue, Dec 18, 4:32 PM · decommission, User-fgiunchedi, Operations
RobH closed T199673: eqiad | (14 + 6) hadoop hardware refresh and expansion as Resolved.

This has been filled via procurement task T204177, resolving.

Tue, Dec 18, 4:31 PM · hardware-requests, Operations, User-Elukey, Analytics
RobH closed T199673: eqiad | (14 + 6) hadoop hardware refresh and expansion, a subtask of T198694: Q1 2018/19 Analytics procurement, as Resolved.
Tue, Dec 18, 4:31 PM · Analytics-Kanban, User-Elukey

Mon, Dec 17

RobH reassigned T209618: rack/setup/install ms-be10[44-50].eqiad.wmnet from Cmjohnson to fgiunchedi.

ms-be1048 has been relocated into row d & reimaged with its row d ip address. should be all set for you

Mon, Dec 17, 8:26 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
RobH added a comment to T209618: rack/setup/install ms-be10[44-50].eqiad.wmnet.

TL;DR / IRC Summary Update:

Mon, Dec 17, 4:59 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations

Fri, Dec 14

RobH moved T209738: decom einsteinium from Backlog to Decommission on the ops-eqiad board.
Fri, Dec 14, 10:31 PM · ops-eqiad, decommission, Operations
RobH reassigned T209738: decom einsteinium from RobH to Cmjohnson.

ready for disk wipe and remainder of steps

Fri, Dec 14, 10:30 PM · ops-eqiad, decommission, Operations
RobH updated the task description for T209738: decom einsteinium.
Fri, Dec 14, 10:29 PM · ops-eqiad, decommission, Operations
RobH removed a project from T209738: decom einsteinium: Patch-For-Review.
Fri, Dec 14, 10:29 PM · ops-eqiad, decommission, Operations
RobH moved T191362: decom promethium/WMF3571 from Backlog to Ready for Decommission on the decommission board.
Fri, Dec 14, 10:22 PM · decommission, Operations, DC-Ops, ops-eqiad
RobH moved T209738: decom einsteinium from Backlog to Ready for Decommission on the decommission board.
Fri, Dec 14, 10:22 PM · ops-eqiad, decommission, Operations
RobH reassigned T209642: Remove labnodepool1001.eqiad.wmnet from RobH to Cmjohnson.

This is ready for disk wipe and remainder of steps to decom the system.

Fri, Dec 14, 10:22 PM · DC-Ops, ops-eqiad, decommission, Operations
RobH updated the task description for T209642: Remove labnodepool1001.eqiad.wmnet.
Fri, Dec 14, 10:21 PM · DC-Ops, ops-eqiad, decommission, Operations
RobH updated the task description for T209642: Remove labnodepool1001.eqiad.wmnet.
Fri, Dec 14, 10:20 PM · DC-Ops, ops-eqiad, decommission, Operations
RobH claimed T209738: decom einsteinium.
Fri, Dec 14, 10:16 PM · ops-eqiad, decommission, Operations
RobH updated the task description for T209642: Remove labnodepool1001.eqiad.wmnet.
Fri, Dec 14, 10:08 PM · DC-Ops, ops-eqiad, decommission, Operations
RobH removed a project from T209642: Remove labnodepool1001.eqiad.wmnet: Patch-For-Review.
Fri, Dec 14, 10:07 PM · DC-Ops, ops-eqiad, decommission, Operations
RobH added a comment to T209642: Remove labnodepool1001.eqiad.wmnet.

labnodepool1001 asw2-b-eqiad ge-3/0/18

Fri, Dec 14, 9:59 PM · DC-Ops, ops-eqiad, decommission, Operations
RobH removed a project from T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech: ops-codfw.

As this is no longer awaiting any onsite work from ops-codfw, I've removed it from the projects.

Fri, Dec 14, 9:37 PM · Patch-For-Review, Operations, fundraising-tech-ops
RobH added a comment to T201342: rack/setup/install puppetmaster1003.eqiad.wmnet.

So, there is already a puppetmaster1001 and puppetmaster1002, do we need a third puppetmaster? Perhaps this was not needed, since rhodium is a third online puppetmaster?

Fri, Dec 14, 9:34 PM · ops-eqiad, Operations
RobH reassigned T212011: migrate netinsights from rhenium to sulfur from RobH to faidon.

So the setup task noted that @faidon is familar with the services on this box, assigning him for input on best way to migrate.

Fri, Dec 14, 9:32 PM · User-Elukey, netops, Operations
RobH updated the task description for T212011: migrate netinsights from rhenium to sulfur.
Fri, Dec 14, 8:25 PM · User-Elukey, netops, Operations
RobH assigned T212010: Degraded RAID on sodium to Cmjohnson.

So failed disk, but under warranty until June 17, 2019.

Fri, Dec 14, 8:22 PM · ops-eqiad, Operations
RobH closed T201364: rack/setup/install sulfur.wikimedia.org as Resolved.

So this now has the netinsights role running. T212011 will track the migration of services.

Fri, Dec 14, 8:20 PM · ops-eqiad, Operations
RobH triaged T212011: migrate netinsights from rhenium to sulfur as Normal priority.
Fri, Dec 14, 8:19 PM · User-Elukey, netops, Operations
RobH removed a project from T201364: rack/setup/install sulfur.wikimedia.org: Patch-For-Review.
Fri, Dec 14, 8:15 PM · ops-eqiad, Operations
RobH added a comment to T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use.

I have the codfw switch maintenance from 8am to 11am (where codfw will be depooled). And a dentist apt at 1pm.
I think it's better to repool codfw before depooling ulsfo. I can take care of it.

Fri, Dec 14, 7:05 PM · Patch-For-Review, Operations, ops-ulsfo
RobH updated subscribers of T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use.

@BBlack: This work is now scheduled for Tuesday, December 18th, at 11:00-12:00 Pacific.

Fri, Dec 14, 4:41 PM · Patch-For-Review, Operations, ops-ulsfo
RobH updated the task description for T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use.
Fri, Dec 14, 4:40 PM · Patch-For-Review, Operations, ops-ulsfo

Thu, Dec 13

RobH moved T211070: decommission of restbase200[1-6] (lease return in December 2018) from Backlog to High Priority Tasks on the ops-codfw board.
Thu, Dec 13, 10:26 PM · Operations, ops-codfw, DC-Ops, decommission
RobH moved T211070: decommission of restbase200[1-6] (lease return in December 2018) from Ready for Decommission to pending onsite steps (codfw) on the decommission board.
Thu, Dec 13, 10:26 PM · Operations, ops-codfw, DC-Ops, decommission
RobH reassigned T211070: decommission of restbase200[1-6] (lease return in December 2018) from RobH to Papaul.
Thu, Dec 13, 10:26 PM · Operations, ops-codfw, DC-Ops, decommission
RobH removed a project from T211070: decommission of restbase200[1-6] (lease return in December 2018): Patch-For-Review.
Thu, Dec 13, 10:20 PM · Operations, ops-codfw, DC-Ops, decommission
RobH added a comment to T211070: decommission of restbase200[1-6] (lease return in December 2018).

Network ports for later label removal:

Thu, Dec 13, 10:10 PM · Operations, ops-codfw, DC-Ops, decommission
RobH removed a project from T211070: decommission of restbase200[1-6] (lease return in December 2018): Patch-For-Review.
Thu, Dec 13, 10:04 PM · Operations, ops-codfw, DC-Ops, decommission
RobH moved T209642: Remove labnodepool1001.eqiad.wmnet from Backlog to Ready for Decommission on the decommission board.
Thu, Dec 13, 6:24 PM · DC-Ops, ops-eqiad, decommission, Operations
RobH moved T211070: decommission of restbase200[1-6] (lease return in December 2018) from Blocked on Service Owners to Ready for Decommission on the decommission board.
Thu, Dec 13, 6:23 PM · Operations, ops-codfw, DC-Ops, decommission
RobH changed the status of T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) from Open to Stalled.

We will be returning these to Farnam sometime this or next month. Go ahead and unrack/prepare these to be boxed up, we should have info on where the boxes will come from via Farnam sometime this month.

Thu, Dec 13, 6:22 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA

Wed, Dec 12

RobH closed T179192: Check analytics1037 power supply status as Resolved.

This hasn't reoccured in a very long time, none since this task creation, resolving.

Wed, Dec 12, 11:38 PM · ops-eqiad, Operations, User-Elukey, Analytics
RobH moved T209029: cloudelastic1004: SMART/disk error from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Wed, Dec 12, 11:36 PM · ops-eqiad, Operations, DC-Ops, cloud-services-team (Kanban)
RobH moved T209139: Broken memory on mw1239 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Wed, Dec 12, 11:36 PM · ops-eqiad, Operations
RobH moved T211668: mw1272 crashed: Bad page map in process hhvm from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Wed, Dec 12, 11:36 PM · serviceops, ops-eqiad, Operations, HHVM
RobH moved T211796: Degraded RAID on ms-be1045 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Wed, Dec 12, 11:36 PM · ops-eqiad, Operations
RobH moved T211807: Degraded RAID on cloudvirt1019 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Wed, Dec 12, 11:36 PM · cloud-services-team, ops-eqiad, Operations
RobH moved T208734: Decommission asw-c-eqiad from Backlog to Decommission on the ops-eqiad board.
Wed, Dec 12, 11:34 PM · Operations, ops-eqiad, netops
RobH moved T209642: Remove labnodepool1001.eqiad.wmnet from Backlog to Decommission on the ops-eqiad board.
Wed, Dec 12, 11:34 PM · DC-Ops, ops-eqiad, decommission, Operations
RobH removed a project from T209618: rack/setup/install ms-be10[44-50].eqiad.wmnet: ops-eqiad.
Wed, Dec 12, 11:33 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
RobH moved T209616: rack/setup/install cloudvirt10[25-30].eqiad.wmnet from Backlog to Racking Tasks on the ops-eqiad board.
Wed, Dec 12, 11:32 PM · ops-eqiad, Operations, DC-Ops, cloud-services-team
RobH changed the status of T209389: rack/setup/install sessionstore200[123].codfw.wmnet from Open to Stalled.

Ok, I've chatted with @Eevans about both T209393 and T209389. The sessionstore servers were put in to order this quarter, but the actual role and software won't be ready until next. (When the request went in, it was taking a LOT longer for Dell to get things to us.)

Wed, Dec 12, 11:28 PM · Services (watching), Core Platform Team Backlog (Watching / External), Operations, Core Platform Team (Session Management Service (CDP2))
RobH changed the status of T209393: rack/setup/install sessionstore100[123].eqiad.wmnet from Open to Stalled.

Ok, I've chatted with @Eevans about both T209393 and T206017. The sessionstore servers were put in to order this quarter, but the actual role and software won't be ready until next. (When the request went in, it was taking a LOT longer for Dell to get things to us.)

Wed, Dec 12, 11:27 PM · Core Platform Team Kanban (Doing), Operations, Core Platform Team (Session Management Service (CDP2)), User-Eevans
RobH removed projects from T209393: rack/setup/install sessionstore100[123].eqiad.wmnet: Patch-For-Review, ops-eqiad.
Wed, Dec 12, 11:26 PM · Core Platform Team Kanban (Doing), Operations, Core Platform Team (Session Management Service (CDP2)), User-Eevans
RobH closed T206017: Hardware for session storage service as Resolved.
Wed, Dec 12, 11:25 PM · Core Platform Team Kanban (Done with CPT), hardware-requests, Operations, Core Platform Team (Session Management Service (CDP2)), User-Eevans
RobH reassigned T209389: rack/setup/install sessionstore200[123].codfw.wmnet from RobH to Eevans.

So I think there is some confusion here, we don't know what role to apply to these hosts, so they don't apply anything but spare and don't allow you to login.

Wed, Dec 12, 10:55 PM · Services (watching), Core Platform Team Backlog (Watching / External), Operations, Core Platform Team (Session Management Service (CDP2))
RobH added a comment to T207760: setup/install weblog1001/WMF4750 as oxygen replacement.

Ahh, due to past discussions via linked tasks, I assumed he was part of the refresh-replace, so I made assumptions!

Wed, Dec 12, 10:33 PM · Operations, Analytics
RobH moved T205507: Decommission analytics100[1,2] from Backlog to Blocked on Service Owners on the decommission board.
Wed, Dec 12, 10:13 PM · Patch-For-Review, Operations, ops-eqiad, decommission, User-Elukey, Analytics
RobH moved T199321: Return graphite200[12] to spares pool from Backlog to Blocked on Service Owners on the decommission board.
Wed, Dec 12, 10:12 PM · decommission, User-fgiunchedi, Operations
RobH moved T209357: Return graphite100[13] to spares pool (or decom) from Backlog to Blocked on Service Owners on the decommission board.
Wed, Dec 12, 10:12 PM · decommission, Patch-For-Review, User-fgiunchedi, Operations
RobH removed a parent task for T199321: Return graphite200[12] to spares pool: T209357: Return graphite100[13] to spares pool (or decom).
Wed, Dec 12, 10:12 PM · decommission, User-fgiunchedi, Operations
RobH removed a subtask for T209357: Return graphite100[13] to spares pool (or decom): T199321: Return graphite200[12] to spares pool.
Wed, Dec 12, 10:12 PM · decommission, Patch-For-Review, User-fgiunchedi, Operations
RobH removed a project from T209357: Return graphite100[13] to spares pool (or decom): hardware-requests.
Wed, Dec 12, 10:11 PM · decommission, Patch-For-Review, User-fgiunchedi, Operations
RobH moved T178592: decommission/replace bast4001.wikimedia.org from Backlog to pending onsite steps (ulsfo) on the decommission board.
Wed, Dec 12, 10:09 PM · decommission, Operations, ops-ulsfo
RobH moved T200209: Decom graphite2001 from Backlog to Blocked on Service Owners on the decommission board.
Wed, Dec 12, 10:00 PM · decommission, ops-codfw, Operations, monitoring
RobH moved T211070: decommission of restbase200[1-6] (lease return in December 2018) from Backlog to Blocked on Service Owners on the decommission board.
Wed, Dec 12, 10:00 PM · Operations, ops-codfw, DC-Ops, decommission
RobH moved T200210: Decom graphite2002 from Backlog to Blocked on Service Owners on the decommission board.
Wed, Dec 12, 10:00 PM · decommission, monitoring, Operations, ops-codfw
RobH moved T211826: decommission oxygen.eqiad.wmnet from Backlog to Blocked on Service Owners on the decommission board.
Wed, Dec 12, 10:00 PM · DC-Ops, decommission
RobH reassigned T207760: setup/install weblog1001/WMF4750 as oxygen replacement from RobH to elukey.

So this should either go to @elukey or @Ottomata, as this is ready to go into serivice and replace oxygen, then we can decommission oxygen on T211826.

Wed, Dec 12, 10:00 PM · Operations, Analytics
RobH added a subtask for T207760: setup/install weblog1001/WMF4750 as oxygen replacement: T211826: decommission oxygen.eqiad.wmnet.
Wed, Dec 12, 9:59 PM · Operations, Analytics
RobH added a parent task for T211826: decommission oxygen.eqiad.wmnet: T207760: setup/install weblog1001/WMF4750 as oxygen replacement.
Wed, Dec 12, 9:59 PM · DC-Ops, decommission
RobH created T211826: decommission oxygen.eqiad.wmnet.
Wed, Dec 12, 9:59 PM · DC-Ops, decommission
RobH removed a project from T207760: setup/install weblog1001/WMF4750 as oxygen replacement: ops-eqiad.
Wed, Dec 12, 9:56 PM · Operations, Analytics
RobH reassigned T201364: rack/setup/install sulfur.wikimedia.org from RobH to Cmjohnson.

This system is behaving poorly, when I send a racadm serveraction powercycle, it takes a very long time to process, and then I never see any output on the serial console.

Wed, Dec 12, 6:12 PM · ops-eqiad, Operations
RobH updated the task description for T201364: rack/setup/install sulfur.wikimedia.org.
Wed, Dec 12, 5:44 PM · ops-eqiad, Operations
RobH reassigned T209618: rack/setup/install ms-be10[44-50].eqiad.wmnet from RobH to fgiunchedi.

Turns out when you enable puppet on a new install with the cert signed, you must still manually run the first run. Fixed.

Wed, Dec 12, 5:36 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
RobH added a project to T200209: Decom graphite2001: decommission.
Wed, Dec 12, 4:39 PM · decommission, ops-codfw, Operations, monitoring
RobH added a project to T200210: Decom graphite2002: decommission.
Wed, Dec 12, 4:39 PM · decommission, monitoring, Operations, ops-codfw
RobH triaged T200210: Decom graphite2002 as Normal priority.
Wed, Dec 12, 4:36 PM · decommission, monitoring, Operations, ops-codfw
RobH triaged T200209: Decom graphite2001 as Normal priority.
Wed, Dec 12, 4:36 PM · decommission, ops-codfw, Operations, monitoring
RobH updated the task description for T200210: Decom graphite2002.
Wed, Dec 12, 4:36 PM · decommission, monitoring, Operations, ops-codfw
RobH updated the task description for T200209: Decom graphite2001.
Wed, Dec 12, 4:35 PM · decommission, ops-codfw, Operations, monitoring

Tue, Dec 11

RobH added a comment to T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use.

Put in a ticket with DR techs:

Tue, Dec 11, 10:49 PM · Patch-For-Review, Operations, ops-ulsfo
RobH reassigned T209618: rack/setup/install ms-be10[44-50].eqiad.wmnet from RobH to fgiunchedi.
Tue, Dec 11, 7:01 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
RobH assigned T211613: rack/setup/install db11[26-38].eqiad.wmnet to Cmjohnson.
Tue, Dec 11, 4:33 PM · Patch-For-Review, DBA, ops-eqiad, User-Marostegui, Operations

Mon, Dec 10

RobH updated the task description for T178592: decommission/replace bast4001.wikimedia.org.
Mon, Dec 10, 11:40 PM · decommission, Operations, ops-ulsfo
RobH updated subscribers of T178592: decommission/replace bast4001.wikimedia.org.

So, this is on asw2-ulsfo:ge-2/0/12

Mon, Dec 10, 11:39 PM · decommission, Operations, ops-ulsfo
RobH added a comment to T178592: decommission/replace bast4001.wikimedia.org.

wipe is in progress via usb live image boot, it'll take 24-48 hours to complete, so I'll just check it when I'm onsite next.

Mon, Dec 10, 10:44 PM · decommission, Operations, ops-ulsfo
RobH claimed T178592: decommission/replace bast4001.wikimedia.org.
Mon, Dec 10, 9:40 PM · decommission, Operations, ops-ulsfo
RobH removed projects from T178592: decommission/replace bast4001.wikimedia.org: Patch-For-Review, Traffic.
Mon, Dec 10, 9:37 PM · decommission, Operations, ops-ulsfo
RobH added a project to T178592: decommission/replace bast4001.wikimedia.org: decommission.
Mon, Dec 10, 9:30 PM · decommission, Operations, ops-ulsfo
RobH closed T179050: setup bast4002/WMF7218 as Resolved.
Mon, Dec 10, 9:28 PM · Patch-For-Review, Traffic, Operations, ops-ulsfo
RobH closed T179050: setup bast4002/WMF7218, a subtask of T164327: replace ulsfo aging servers, as Resolved.
Mon, Dec 10, 9:28 PM · Traffic, Operations, ops-ulsfo
RobH added a comment to T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use.

So some bad news: I only ordered enough brackets for half the PDUs. T211632 has been created to order the other half. However, this will NOT block the migration to the new PDUs, I'll just temp affix the ST PDUs in place with zip ties rather than brackets. I can move them to brackets after the fact (when the new brackets arrive) with no downtime.

Mon, Dec 10, 9:10 PM · Patch-For-Review, Operations, ops-ulsfo
RobH updated the task description for T209101: ulsfo: install new PDUs in racks / phase out APC loaner PDU use.
Mon, Dec 10, 8:58 PM · Patch-For-Review, Operations, ops-ulsfo