@cmooney this happens to be a FR maintenance week, is it feasible to plan it sometime tomorrow-Friday?

In T407927#12098110, @RKumar_WMF wrote:

In Gr4vy the Insights shows the data with the time lag of 15 mins or so. We can drill down to information like which country, BIN, issuer etc is causing the decline in auth rate.
The dashboard shared looks very nice. Does this tell about the declines or can I drill down to it? I think it would be good to get into a call to understand the dashboard.

In T407927#12093267, @RKumar_WMF wrote:

It would be helpful to understand what payment success rate here means. Also, I understood Eileen is also working on Authorization rates data/dashboard.

In T407927#12092137, @AKanji-WMF wrote:

Thanks @Dwisehaupt and @AnnWF - this data is avail in grafana if we ever want to dashboard it out etc. cc @RKumar_WMF

Done!

In T428093#11982872, @ssingh wrote:

In T428093#11982863, @taavi wrote:

In T428093#11982860, @ssingh wrote:

So payments is a CNAME and hence we can't add a CAA record for it.

Can we not add a CAA record to the CNAME targets (payments-eqiad/codfw)?

We can yeah but I am not sure if there is something in the fr-tech setup that complicates that. @Jgreen can comment on those bits.

I added the frdeploy user's key for read access. The key setup should be done, and once T429251 is done we should be able to work with f_c_u on frpm1002

frdb1004 (soon-to-be-decommed reader) has BIOS adjustment and mariadb tuning applied

Looking at the data collected, these seem rare enough (19 total in the past 2 days) I suspect it's attributable to routine network disturbances. Closing the task b/c I don't think there's anything we can fix.

Closing this, afaik there's nothing left to do.

In T421558#11955970, @Lars wrote:

@Dwisehaupt @Jgreen The above removed about 250GB from the activities table.

frdb1006 (staging) has BIOS adjustment and mariadb tuning applied

In T426824#11942813, @Jhancock.wm wrote:

i can get this one in the morning if Jeff or Dallas is around and want to coordinate.

What we can do with this from the SRE side is already done, unless there's feedback on how to improve it. It would be more precise/accurate to do reporting from the database or analytics side.

We've disabled all the passive checks, so there should be nothing incoming to the nsca server from FR anymore.

frdb1007 has all the same tuning stuff applied

In T425424#11909128, @tappof wrote:

I believe you can go ahead with the plan you shared.

frmon2002 is done, /srv is 1.7T, 42% full

frmon1002 is done, /srv/ is 1.2T and 58% full

frdb1008 first tuning attempt (all settings are in hosts.yaml):

frdb1008:
    mariadb_innodb_flush_neighbors: 0
    mariadb_innodb_io_capacity: 20000
    mariadb_innodb_io_capacity_max: 30000
    mariadb_innodb_log_file_size: '4G'
    mariadb_innodb_read_io_threads: 16
    mariadb_innodb_write_io_threads: 16
    mariadb_open_files_limit: 15000
    mariadb_table_open_cache: 8000
    mariadb_table_open_cache_instances: 16

DNS is updated