In T402316#11441929, @Dwisehaupt wrote:

This is the updated panel: https://frmon.wikimedia.org/goto/ff6iecgkug2rka?orgId=1 on the Payment Transaction Outcomes dashboard. This is something we could alert on with the new alertmanager rules if we wish.

There are two tables in the metabase mariadb database that could be useful:

MariaDB [metabase]> desc login_history;
+--------------------+--------------+------+-----+----------------------+----------------+
| Field              | Type         | Null | Key | Default              | Extra          |
+--------------------+--------------+------+-----+----------------------+----------------+
| id                 | int(11)      | NO   | PRI | NULL                 | auto_increment |
| timestamp          | timestamp(6) | NO   | MUL | current_timestamp(6) |                |
| user_id            | int(11)      | NO   | MUL | NULL                 |                |
| session_id         | varchar(254) | YES  | MUL | NULL                 |                |
| device_id          | char(36)     | NO   |     | NULL                 |                |
| device_description | longtext     | YES  |     | NULL                 |                |
| ip_address         | longtext     | YES  |     | NULL                 |                |
+--------------------+--------------+------+-----+----------------------+----------------+

In T367820#11299299, @VRiley-WMF wrote:

Hey @Jgreen Is this okay to close, or would you like this assigned to you?

I think we're close to being able to get this from prometheus based on the metric "logged_payments". This is a counter collected from logs on one-minute intervals and sub-categorized by several labels. The labels are my best guess of how we'd want things categorized, and probably need refinement.

In T405982#11281892, @Jhancock.wm wrote:

@Jgreen we got these drives in. <edit> When is a good time to install the drives in 2001-2003?

These are pretty rare so it took a while to catch anything, but the prometheus collector is working now.

Ok, I'm reopening this so I don't forget.

@Eileenmcnaughton I haven't changed anything yet, for now users should only be able to read from the live replica smashpig. Yeah dev_smashpig is probably ancient.

Ok, fixed I think. The issue was that the view was created with DEFINER=civicrm@10.64.40.% and user civicrm was missing select access on the target table.

The create view syntax is the same for the view in civicrm vs dev_civicrm, so I think it's probably a permissions issue, looking at that now.

In T367820#11254703, @Jclark-ctr wrote:

@Jgreen i have installed the 5th drive T405983 was there anything left on dcops side @VRiley-WMF

This seems to have cleared up around 9:20 on 2025-09-30, nothing significant in the logs after that. I don't see anything in the logs, server admin logs, etc that would explain the resolution or cause.

In T404243#11248259, @Ejegg wrote:

We want to know how many people click that but then choose one of the alternatives like switching to annual, pausing, or reducing the amount.

@Ejegg, is this something we want to alert about or just make it graphable?

Closing b/c I think we're close enough, although we aren't alerting for HMS outages directly we'll know quickly if there is an outage because trino will complain.

Could we get the info we need from the API side? Looking at this https://civicrm.org/extensions/api-rest-log -- As long as the login method is to request a link, I think the request would go app -> portal->civiproxy->api and the event could be loggable into DB from the API side.

I noticed the regex that was supposed to catch these wasn't quite right, hopefully fixed now.

Trino has been stable after the last upgrade and tuning limits on the client side, a restarter service doesn't seem needed anymore.

AFAICT this is done

The issue was caused by a missing s3.region setting.

Workaround is committed and will apply next time we rebuild the venv.