Page MenuHomePhabricator

fgiunchedi (Filippo Giunchedi)
/* No comment */

Projects (15)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 8:06 AM (320 w, 4 d)
Availability
Available
IRC Nick
godog
LDAP User
Filippo Giunchedi
MediaWiki User
Filippo Giunchedi [ Global Accounts ]

Recent Activity

Yesterday

fgiunchedi moved T267650: LibreNMS supports more than one Alertmanager address from Backlog to Doing on the User-fgiunchedi board.
Tue, Nov 24, 3:10 PM · User-fgiunchedi, observability
fgiunchedi added a comment to T267650: LibreNMS supports more than one Alertmanager address.

Upstream PR by yours truly https://github.com/librenms/librenms/pull/12346

Tue, Nov 24, 1:05 PM · User-fgiunchedi, observability
fgiunchedi added a project to T267186: alerts.w.o / idp.w.o interaction and CORS: CAS-SSO.
Tue, Nov 24, 8:21 AM · CAS-SSO, Patch-For-Review, User-fgiunchedi, observability
fgiunchedi added a comment to T267186: alerts.w.o / idp.w.o interaction and CORS.

Was able to get an error this morning when the session needed refreshing and thus a new ticket:

Tue, Nov 24, 8:20 AM · CAS-SSO, Patch-For-Review, User-fgiunchedi, observability

Mon, Nov 23

fgiunchedi updated the task description for T268435: Add ms-be106[0-3] to swift.
Mon, Nov 23, 3:31 PM · Patch-For-Review, User-fgiunchedi, SRE-swift-storage
fgiunchedi updated the task description for T266017: Implement alerting roadmap phase 2.
Mon, Nov 23, 9:23 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi closed T261281: Improve performance of Thanos (+ Prometheus) as Resolved.

This is complete! I've left out trying memcache since the in-memory caching seems to work well for now

Mon, Nov 23, 9:14 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi updated the task description for T261281: Improve performance of Thanos (+ Prometheus).
Mon, Nov 23, 9:13 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi moved T268435: Add ms-be106[0-3] to swift from Backlog to Doing on the User-fgiunchedi board.
Mon, Nov 23, 9:13 AM · Patch-For-Review, User-fgiunchedi, SRE-swift-storage
fgiunchedi updated the task description for T268435: Add ms-be106[0-3] to swift.
Mon, Nov 23, 9:02 AM · Patch-For-Review, User-fgiunchedi, SRE-swift-storage
fgiunchedi updated the task description for T268435: Add ms-be106[0-3] to swift.
Mon, Nov 23, 8:13 AM · Patch-For-Review, User-fgiunchedi, SRE-swift-storage
fgiunchedi created T268435: Add ms-be106[0-3] to swift.
Mon, Nov 23, 8:08 AM · Patch-For-Review, User-fgiunchedi, SRE-swift-storage
fgiunchedi added a comment to T265093: (Need By: ASAP) rack/setup/install ms-be106[0-3].

The hosts all look good, thanks a lot @RobH and @Cmjohnson !

Mon, Nov 23, 8:07 AM · Operations, ops-eqiad, DC-Ops

Thu, Nov 19

fgiunchedi added a comment to T267186: alerts.w.o / idp.w.o interaction and CORS.

Ran into an error just now, although I'm not sure why CORS wouldn't allow this and might not be on the IDP side?

Thu, Nov 19, 4:19 PM · CAS-SSO, Patch-For-Review, User-fgiunchedi, observability
fgiunchedi added a comment to T267186: alerts.w.o / idp.w.o interaction and CORS.

Different error message this time in the browser's console, but something has definitely changed!

I have just pushed another change which i think should fix this are you able to test again

Thu, Nov 19, 3:18 PM · CAS-SSO, Patch-For-Review, User-fgiunchedi, observability
fgiunchedi added a comment to T268176: Allow kafka network access to cloudvirt hosts.

The full list of hosts is the logging-eqiad kafka cluster on tcp port 9093, namely:

Thu, Nov 19, 11:43 AM · cloud-services-team (Kanban), Cloud-VPS
fgiunchedi added a comment to T267186: alerts.w.o / idp.w.o interaction and CORS.

Different error message this time in the browser's console, but something has definitely changed!

Thu, Nov 19, 11:34 AM · CAS-SSO, Patch-For-Review, User-fgiunchedi, observability

Wed, Nov 18

fgiunchedi merged T268123: Degraded RAID on ms-be1022 into T267870: ms-be1022 smart storage battery failure; disk sdb possibly bad.
Wed, Nov 18, 10:31 AM · SRE-swift-storage, ops-eqiad, Operations
fgiunchedi merged task T268123: Degraded RAID on ms-be1022 into T267870: ms-be1022 smart storage battery failure; disk sdb possibly bad.
Wed, Nov 18, 10:31 AM · ops-eqiad, Operations
fgiunchedi added a comment to T268123: Degraded RAID on ms-be1022.

This was me manually failing sdb

Wed, Nov 18, 10:31 AM · ops-eqiad, Operations
fgiunchedi merged T268071: Degraded RAID on ms-be1022 into T267870: ms-be1022 smart storage battery failure; disk sdb possibly bad.
Wed, Nov 18, 9:37 AM · SRE-swift-storage, ops-eqiad, Operations
fgiunchedi merged task T268071: Degraded RAID on ms-be1022 into T267870: ms-be1022 smart storage battery failure; disk sdb possibly bad.
Wed, Nov 18, 9:36 AM · ops-eqiad, Operations

Tue, Nov 17

fgiunchedi added a comment to T267870: ms-be1022 smart storage battery failure; disk sdb possibly bad.

Update: the host isn't coming back (both mgmt and ssh) but yes given we'll need a BBU for ms-be1030 (T268036) too I'd say let's order some (?). The host will be fully decom'd in maybe 8-10 weeks but I think we can keep the bbu anyway post-decom

Tue, Nov 17, 5:38 PM · SRE-swift-storage, ops-eqiad, Operations
fgiunchedi updated subscribers of T268036: Degraded RAID on ms-be1030.

@Cmjohnson @Jclark-ctr looks like BBU replacement for this host (out of warranty AFAICS), thank you!

Tue, Nov 17, 4:53 PM · ops-eqiad, Operations
fgiunchedi added a comment to T268036: Degraded RAID on ms-be1030.

This host went down earlier today, it is missing a hw raid firmware upgrade so I'll do that just in case. This looks like a BBU needing a replacement tho

Tue, Nov 17, 4:44 PM · ops-eqiad, Operations
fgiunchedi closed T267645: Wrong redirect when logging into grafana-rw from a grafana.w.o dashboard, a subtask of T262512: Enable CAS authentication for Grafana, as Resolved.
Tue, Nov 17, 1:17 PM · User-fgiunchedi, Patch-For-Review, observability, Operations
fgiunchedi closed T267645: Wrong redirect when logging into grafana-rw from a grafana.w.o dashboard as Resolved.

This is fixed now \o/

Tue, Nov 17, 1:17 PM · User-fgiunchedi, observability, Operations
fgiunchedi closed T267972: Redirect grafana.wm.o/explore to grafana-rw as Resolved.

I can confirm that the short link above works now, thanks for reaching out!

Tue, Nov 17, 8:30 AM · Performance-Team (Radar), observability, Graphite
fgiunchedi added a comment to T267901: SMART data dump healthy metric can contain None.

Indeed the underlying disk was failed and marked as such by the controller. To be clear in this case IIRC we have the disk name from hpssacli already and I think we should use it in reporting at least device_smart_healthy without requiring smartctl to return valid output

Tue, Nov 17, 8:17 AM · observability
fgiunchedi added a comment to T267870: ms-be1022 smart storage battery failure; disk sdb possibly bad.

@fgiunchedi The server is out of warranty, I have some decom'd HP servers and most likely can steal a bbu from one of them. I also have decom'd host w/3TB disks that we can take from. This server will require downtime, also worth noting the 4 new ms-be hosts are here and in the rack and will be ready for you by the end of the week (at the latest). In case you want to decom ms-be1022.

Tue, Nov 17, 7:54 AM · SRE-swift-storage, ops-eqiad, Operations
fgiunchedi closed T267748: Degraded RAID on ms-be2031 as Resolved.

Thank you @Papaul ! SSD is rebuilding

Tue, Nov 17, 7:49 AM · Operations, ops-codfw

Mon, Nov 16

fgiunchedi moved T267645: Wrong redirect when logging into grafana-rw from a grafana.w.o dashboard from Backlog to Doing on the User-fgiunchedi board.
Mon, Nov 16, 10:27 AM · User-fgiunchedi, observability, Operations
fgiunchedi moved T267018: LibreNMS sends its alerts to Alertmanager, resulting in email notifications to network operations from Up next to Doing on the User-fgiunchedi board.
Mon, Nov 16, 10:27 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi added a comment to T184086: Add prometheus exporter to Gerrit.

The patch is live, unfortunately due to how our Prometheus puppetization works it means we're scraping metrics from gerrit from all sites. Given that the number of metrics is not high (~2k ballpark) we're ok for now but we definitely need to tune the configuration so that only Prometheus hosts in eqiad scrape gerrit metrics

Mon, Nov 16, 10:03 AM · Patch-For-Review, observability, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Gerrit, Operations
fgiunchedi added a comment to T267872: Degraded RAID on ms-be1022.

Nevermind, let's follow up on T267870: ms-be1022 smart storage battery failure; disk sdb possibly bad

Mon, Nov 16, 9:35 AM · ops-eqiad, Operations
fgiunchedi updated subscribers of T267872: Degraded RAID on ms-be1022.

Replacement BBU needed, looks like the host is OOW. cc @Cmjohnson @Jclark-ctr

Mon, Nov 16, 9:29 AM · ops-eqiad, Operations
fgiunchedi assigned T267748: Degraded RAID on ms-be2031 to Papaul.

@Papaul looks like the SSD is busted on this host. Host is OOW I think, we'll need a replacement SSD, thanks!

Mon, Nov 16, 9:28 AM · Operations, ops-codfw
fgiunchedi created T267901: SMART data dump healthy metric can contain None.
Mon, Nov 16, 9:26 AM · observability
fgiunchedi added a project to T265435: codfw: Testing Out Sample PDUs: observability.

Once the PDU are installed please let observability know. At minimum we'd need to test librenms discovery and their SNMP MIB to snmp-exporter for pulling power data into Prometheus

Mon, Nov 16, 9:09 AM · observability, ops-codfw, Operations, DC-Ops

Tue, Nov 10

fgiunchedi added a comment to T184086: Add prometheus exporter to Gerrit.

The patch is live, unfortunately due to how our Prometheus puppetization works it means we're scraping metrics from gerrit from all sites. Given that the number of metrics is not high (~2k ballpark) we're ok for now but we definitely need to tune the configuration so that only Prometheus hosts in eqiad scrape gerrit metrics

Tue, Nov 10, 4:31 PM · Patch-For-Review, observability, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Gerrit, Operations
fgiunchedi updated subscribers of T267650: LibreNMS supports more than one Alertmanager address.
Tue, Nov 10, 4:11 PM · User-fgiunchedi, observability
fgiunchedi created T267650: LibreNMS supports more than one Alertmanager address.
Tue, Nov 10, 4:07 PM · User-fgiunchedi, observability
aborrero awarded T267645: Wrong redirect when logging into grafana-rw from a grafana.w.o dashboard a The World Burns token.
Tue, Nov 10, 2:58 PM · User-fgiunchedi, observability, Operations
fgiunchedi created T267645: Wrong redirect when logging into grafana-rw from a grafana.w.o dashboard.
Tue, Nov 10, 2:57 PM · User-fgiunchedi, observability, Operations
fgiunchedi added a comment to T267135: smart-data-dump should fail loudly when it can't gather metrics.

The related problem to this one is also that the hp raid controller name changed when we changed the facter invocation, and ssacli isn't recognized as such by smart_data_dump

Tue, Nov 10, 11:24 AM · Operations, observability
fgiunchedi added a comment to T267135: smart-data-dump should fail loudly when it can't gather metrics.

I think it is fair to say that if no disks are detected then that's always an error condition (?) In that case I think a simple(r) solution would be to exit non-zero if no disks are detected so the systemd service/timer fails loudly.

Tue, Nov 10, 11:01 AM · Operations, observability
fgiunchedi closed T266535: Silencing alerts through the alerts dashboard is supported and functional, a subtask of T266017: Implement alerting roadmap phase 2, as Resolved.
Tue, Nov 10, 10:50 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi closed T266535: Silencing alerts through the alerts dashboard is supported and functional as Resolved.
Tue, Nov 10, 10:50 AM · User-fgiunchedi, observability

Mon, Nov 9

fgiunchedi updated the task description for T266535: Silencing alerts through the alerts dashboard is supported and functional.
Mon, Nov 9, 2:57 PM · User-fgiunchedi, observability
fgiunchedi added a comment to T266535: Silencing alerts through the alerts dashboard is supported and functional.

Status update: alerts now can be acknowledged via silences by prefixing the silence text with ACK!

Mon, Nov 9, 2:57 PM · User-fgiunchedi, observability
fgiunchedi added a comment to T265857: Update CAS to 6.2.

I was looking at prometheus jobs down alert today and idp shows up there, I'm assuming because the prometheus endpoint has been removed in Ia4b089af. Please remove the IDP prometheus job as well, thanks!

Mon, Nov 9, 9:47 AM · Patch-For-Review, CAS-SSO, Operations
fgiunchedi updated the task description for T261281: Improve performance of Thanos (+ Prometheus).
Mon, Nov 9, 9:17 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi added a comment to T261281: Improve performance of Thanos (+ Prometheus).

Status update: query-frontend is serving queries with in-memory caching (1GB, to start with)

Mon, Nov 9, 9:15 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi updated the task description for T261281: Improve performance of Thanos (+ Prometheus).
Mon, Nov 9, 9:13 AM · Patch-For-Review, User-fgiunchedi, observability

Thu, Nov 5

fgiunchedi closed T265712: Sync users and permissions from LDAP to Grafana, a subtask of T262512: Enable CAS authentication for Grafana, as Resolved.
Thu, Nov 5, 1:18 PM · User-fgiunchedi, Patch-For-Review, observability, Operations
fgiunchedi closed T265712: Sync users and permissions from LDAP to Grafana as Resolved.

This is live now, LDAP users are synced daily to Grafana.

Thu, Nov 5, 1:18 PM · Patch-For-Review, User-fgiunchedi, observability, Operations

Wed, Nov 4

fgiunchedi reopened T242819: ores.wmflabs.org - 503 icinga alerts as "Open".

Reopening, this is alerting again

Wed, Nov 4, 10:49 AM · Patch-For-Review, Machine Learning Platform (Current), Operations, ORES
fgiunchedi updated the task description for T261281: Improve performance of Thanos (+ Prometheus).
Wed, Nov 4, 9:09 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi moved T267186: alerts.w.o / idp.w.o interaction and CORS from Backlog to Radar on the User-fgiunchedi board.
Wed, Nov 4, 8:56 AM · CAS-SSO, Patch-For-Review, User-fgiunchedi, observability
fgiunchedi created T267186: alerts.w.o / idp.w.o interaction and CORS.
Wed, Nov 4, 8:53 AM · CAS-SSO, Patch-For-Review, User-fgiunchedi, observability
fgiunchedi added a comment to T266017: Implement alerting roadmap phase 2.

Noticed this today on alerts.w.o and IDP interaction, we're missing CORS:

Wed, Nov 4, 8:46 AM · Patch-For-Review, User-fgiunchedi, observability

Tue, Nov 3

fgiunchedi added a project to T267135: smart-data-dump should fail loudly when it can't gather metrics: Operations.
Tue, Nov 3, 3:15 PM · Operations, observability
fgiunchedi created T267135: smart-data-dump should fail loudly when it can't gather metrics.
Tue, Nov 3, 3:13 PM · Operations, observability
fgiunchedi moved T266019: Several days of metrics not uploaded to Thanos object storage from Prometheus on PoPs from Backlog to Radar on the User-fgiunchedi board.
Tue, Nov 3, 10:06 AM · User-fgiunchedi, observability
fgiunchedi moved T262512: Enable CAS authentication for Grafana from Up next to Doing on the User-fgiunchedi board.
Tue, Nov 3, 10:06 AM · User-fgiunchedi, Patch-For-Review, observability, Operations
fgiunchedi moved T267018: LibreNMS sends its alerts to Alertmanager, resulting in email notifications to network operations from Backlog to Up next on the User-fgiunchedi board.
Tue, Nov 3, 10:06 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi added a comment to T266800: VictorOps ~5min delay from email received to incident paging.

@fgiunchedi should we consider converting out transport from email to API calls at this point? Should give us an immediate feedback that we relayed the alert to VO infrastructure.

Tue, Nov 3, 8:56 AM · observability, Operations

Mon, Nov 2

fgiunchedi added a comment to T266800: VictorOps ~5min delay from email received to incident paging.

This is now a case with VO support. They'll be following up with their transactional email provider.

Mon, Nov 2, 1:40 PM · observability, Operations
fgiunchedi updated the task description for T261281: Improve performance of Thanos (+ Prometheus).
Mon, Nov 2, 1:29 PM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi created T267019: Alert design guidelines for teams are produced.
Mon, Nov 2, 1:10 PM · observability
fgiunchedi created T267018: LibreNMS sends its alerts to Alertmanager, resulting in email notifications to network operations.
Mon, Nov 2, 1:09 PM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi created T267017: Grafana sends its alerts to Alertmanager, resulting in email notifications routed to the appropriate teams.
Mon, Nov 2, 1:09 PM · observability
fgiunchedi moved T266535: Silencing alerts through the alerts dashboard is supported and functional from Up next to Doing on the User-fgiunchedi board.
Mon, Nov 2, 1:09 PM · User-fgiunchedi, observability
fgiunchedi moved T266017: Implement alerting roadmap phase 2 from Up next to Doing on the User-fgiunchedi board.
Mon, Nov 2, 1:09 PM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi committed rODTHc64da0954bfe: Pass SyslogIdentifier=%N to systemd services (authored by fgiunchedi).
Pass SyslogIdentifier=%N to systemd services
Mon, Nov 2, 11:13 AM
fgiunchedi created T267002: Rename swift.discovery.wmnet to ms-fe.discovery.wmnet.
Mon, Nov 2, 9:55 AM · User-fgiunchedi, SRE-swift-storage
fgiunchedi closed T261633: Put ms-be2057 (Dell R740xd2) in service as Resolved.

Host is fully in service now

Mon, Nov 2, 8:33 AM · Patch-For-Review, User-fgiunchedi, Operations, SRE-swift-storage
fgiunchedi closed T261633: Put ms-be2057 (Dell R740xd2) in service, a subtask of T266016: Refresh and expand Swift hardware capacity, as Resolved.
Mon, Nov 2, 8:32 AM · User-fgiunchedi, SRE-swift-storage

Thu, Oct 29

fgiunchedi created T266800: VictorOps ~5min delay from email received to incident paging.
Thu, Oct 29, 4:28 PM · observability, Operations

Wed, Oct 28

fgiunchedi removed a project from T196994: Open Phab tasks on SMART failure: User-fgiunchedi.
Wed, Oct 28, 2:58 PM · Patch-For-Review, Operations, observability
fgiunchedi added a project to T239458: Mediawiki logging indexing conflict: Wikimedia-Logstash.
Wed, Oct 28, 2:58 PM · Wikimedia-Logstash, Editing-team (Tracking), MediaWiki-Logevents, VisualEditor, MediaWiki-General
fgiunchedi removed a project from T239458: Mediawiki logging indexing conflict: User-fgiunchedi.
Wed, Oct 28, 2:57 PM · Wikimedia-Logstash, Editing-team (Tracking), MediaWiki-Logevents, VisualEditor, MediaWiki-General
fgiunchedi removed a project from T86552: Monitor and alarm on SMART attributes [tracking]: User-fgiunchedi.
Wed, Oct 28, 2:57 PM · Epic, Operations, observability
fgiunchedi removed a project from T239090: Restbase logging indexing conflict on 'res' and 'body' logging fields: User-fgiunchedi.
Wed, Oct 28, 2:57 PM · Platform Team Workboards (Clinic Duty Team), Wikimedia-Logstash, RESTBase

Tue, Oct 27

fgiunchedi placed T178690: Better organization for SRE grafana dashboards up for grabs.
Tue, Oct 27, 3:39 PM · User-CDanis, User-fgiunchedi, observability, Operations
fgiunchedi placed T126574: puppet should try to mount all mountable swift filesystems up for grabs.
Tue, Oct 27, 3:39 PM · Patch-Needs-Improvement, Operations
fgiunchedi closed T205974: logrotate cronspam on ms-be1040 as Invalid.

IIRC this was fixed, boldly resolving

Tue, Oct 27, 3:38 PM · Patch-Needs-Improvement, Operations
fgiunchedi created T266570: Two close pages for idle workers api + appserver didn't auto-resolve on recovery.
Tue, Oct 27, 2:42 PM · observability, Operations
fgiunchedi added a comment to T249311: Deploy Prometheus Push Gateway.

Very cool! Could this be used for custom metrics for batch jobs in Hadoop?

Tue, Oct 27, 1:14 PM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi moved T266535: Silencing alerts through the alerts dashboard is supported and functional from Backlog to Up next on the User-fgiunchedi board.
Tue, Oct 27, 9:32 AM · User-fgiunchedi, observability
fgiunchedi created T266535: Silencing alerts through the alerts dashboard is supported and functional.
Tue, Oct 27, 9:29 AM · User-fgiunchedi, observability
fgiunchedi moved T266019: Several days of metrics not uploaded to Thanos object storage from Prometheus on PoPs from Up next to Backlog on the User-fgiunchedi board.
Tue, Oct 27, 9:18 AM · User-fgiunchedi, observability
fgiunchedi closed T249311: Deploy Prometheus Push Gateway as Resolved.

This is complete!

Tue, Oct 27, 9:08 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi committed rODTH203ec840f282: New upstream release (authored by fgiunchedi).
New upstream release
Tue, Oct 27, 8:37 AM
fgiunchedi committed rODTH04da0d9b6f6a: Merge tag 'v0.16.0' into debian/buster-wikimedia (authored by fgiunchedi).
Merge tag 'v0.16.0' into debian/buster-wikimedia
Tue, Oct 27, 8:37 AM
fgiunchedi added a comment to T211661: Automatically clean up unused thumbnails in Swift.

From my POV as Swift maintainer I'm ok to go ahead with testing etc.

Tue, Oct 27, 8:34 AM · Patch-For-Review, Traffic, SRE-swift-storage, Performance-Team, Operations

Mon, Oct 26

fgiunchedi closed T266214: Degraded RAID on ms-be2017 as Resolved.

Handler reenabled, disk reenabled and rebuilding! Thanks @Papaul @Marostegui !

Mon, Oct 26, 9:00 AM · SRE-swift-storage, Operations, ops-codfw

Oct 22 2020

fgiunchedi updated the task description for T266019: Several days of metrics not uploaded to Thanos object storage from Prometheus on PoPs.
Oct 22 2020, 3:56 PM · User-fgiunchedi, observability
fgiunchedi added a comment to T246004: Spike: Can/should Swift be used as Flink checkpoint backend?.

Thank you all for swift (pun intended) action!

Oct 22 2020, 1:29 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service, Epic
fgiunchedi reopened T221904: swift backend decomms / rebalances are noisy as "Open".

Unfortunately reopening, we've been seeing failures (e.g. systemd, ssh) during latest codfw rebalances

Oct 22 2020, 12:14 PM · SRE-swift-storage, Operations