Adding @aaron as the person most familiar with that code.

@aaron Is this a task Performance is going to take on? I'm going untag CPT for now but please retag us if that is incorrect.

Change 534268 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] [WIP] rdbms: add query timeout support to Database::select()

https://gerrit.wikimedia.org/r/534268

BTW, I consider this a smaller issue once replication control was migrated to heartbeat- I am guessing some show slave status are still in place. The query timeout, however, will be useful in multiple other cases like limiting DOS due to api performance spikes.

Change 534268 merged by jenkins-bot:

[mediawiki/core@master] rdbms: add query timeout support to Database::select()

https://gerrit.wikimedia.org/r/534268

Change 747692 had a related patch set uploaded (by Ladsgroup; author: Aaron Schulz):

[mediawiki/core@wmf/1.38.0-wmf.12] rdbms: add query timeout support to Database::select()

https://gerrit.wikimedia.org/r/747692

Change 747692 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.12] rdbms: add query timeout support to Database::select()

https://gerrit.wikimedia.org/r/747692

Mentioned in SAL (#wikimedia-operations) [2021-12-16T15:03:25Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.12/includes/libs/rdbms/database/: Backport: [[gerrit:747692|rdbms: add query timeout support to Database::select() (T129093 T195792)]] (duration: 01m 11s)

According to db-production.php, only the "es" servers use Seconds_Behind_Master (except for the 'is static' servers, which don't need any lag methods). Is there a reason those servers cannot use pt-heartbeat as well?

In T129093#8583806, @aaron wrote:

According to db-production.php, only the "es" servers use Seconds_Behind_Master (except for the 'is static' servers, which don't need any lag methods). Is there a reason those servers cannot use pt-heartbeat as well?

I don't think there should be any reason for es4 and es5 (non RO es hosts) not to use pt-heartbeat.

In T129093#8584245, @Marostegui wrote:

In T129093#8583806, @aaron wrote:

According to db-production.php, only the "es" servers use Seconds_Behind_Master (except for the 'is static' servers, which don't need any lag methods). Is there a reason those servers cannot use pt-heartbeat as well?

I don't think there should be any reason for es4 and es5 (non RO es hosts) not to use pt-heartbeat.

Is this just a matter of wmf-config then?

I believe so. At least from our side it's all ready

Tables seems fine. The "es*" vs "cluster*" part of config will be annoying though (the "shard" option needs the "es*" name):

> $lb = \MediaWiki\MediaWikiServices::getInstance()->getDBLoadBalancerFactory()->getExternalLB( 'cluster26' );
= Wikimedia\Rdbms\LoadBalancer {#3349}

> iterator_to_array( $lb->getConnection( DB_REPLICA )->query( "select * from heartbeat.heartbeat" ) );
= [
    {#5769
      +"ts": "2023-02-06T20:31:12.000950",
      +"server_id": "171970708",
      +"file": "es1021-bin.008190",
      +"position": "230205834",
      +"relay_master_log_file": null,
      +"exec_master_log_pos": null,
      +"shard": "es4",
      +"datacenter": "eqiad",
    },
    {#5784
      +"ts": "2023-02-06T20:31:12.001070",
      +"server_id": "180359316",
      +"file": "es2021-bin.008236",
      +"position": "823987058",
      +"relay_master_log_file": "es1021-bin.008190",
      +"exec_master_log_pos": "230205834",
      +"shard": "es4",
      +"datacenter": "codfw",
    },
  ]

> 
> $lb = \MediaWiki\MediaWikiServices::getInstance()->getDBLoadBalancerFactory()->getExternalLB( 'cluster27' );
= Wikimedia\Rdbms\LoadBalancer {#5789}

> iterator_to_array( $lb->getConnection( DB_REPLICA )->query( "select * from heartbeat.heartbeat" ) );
= [
    {#5782
      +"ts": "2023-02-06T20:31:17.000970",
      +"server_id": "171966666",
      +"file": "es1024-bin.008220",
      +"position": "772610538",
      +"relay_master_log_file": null,
      +"exec_master_log_pos": null,
      +"shard": "es5",
      +"datacenter": "eqiad",
    },
    {#3350
      +"ts": "2023-02-06T20:31:17.001170",
      +"server_id": "180367499",
      +"file": "es2023-bin.008269",
      +"position": "503583733",
      +"relay_master_log_file": "es1024-bin.008220",
      +"exec_master_log_pos": "772610538",
      +"shard": "es5",
      +"datacenter": "codfw",
    },
  ]

>

Change 893835 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[operations/mediawiki-config@master] Use pt-heartbeat for all non-static external clusters

https://gerrit.wikimedia.org/r/893835