The haproxy logs on cloudcontrol100x nodes are somewhat full of messages like this:
Jun 14 19:23:38 cloudcontrol1004 haproxy[370057]: [WARNING] 164/192338 (370057) : Server mysql/cloudcontrol1005.wikimedia.org is DOWN, reason: Socket error, info: "Connection reset by peer", check duration: 56ms. 0 active and 2 backup servers left. Running on backup. 851 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:23:43 cloudcontrol1004 haproxy[370057]: Backup Server mysql/cloudcontrol1003.wikimedia.org is DOWN, reason: Socket error, info: "Connection reset by peer", check duration: 41ms. 0 active and 1 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:23:43 cloudcontrol1004 haproxy[370057]: [WARNING] 164/192343 (370057) : Backup Server mysql/cloudcontrol1003.wikimedia.org is DOWN, reason: Socket error, info: "Connection reset by peer", check duration: 41ms. 0 active and 1 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:23:43 cloudcontrol1004 haproxy[370057]: Server mysql/cloudcontrol1005.wikimedia.org is UP, reason: Layer7 check passed, code: 200, check duration: 52ms. 1 active and 1 backup servers online. 0 sessions requeued, 0 total in queue. Jun 14 19:23:43 cloudcontrol1004 haproxy[370057]: [WARNING] 164/192343 (370057) : Server mysql/cloudcontrol1005.wikimedia.org is UP, reason: Layer7 check passed, code: 200, check duration: 52ms. 1 active and 1 backup servers online. 0 sessions requeued, 0 total in queue. Jun 14 19:23:45 cloudcontrol1004 haproxy[370057]: Backup Server mysql/cloudcontrol1003.wikimedia.org is UP, reason: Layer7 check passed, code: 200, check duration: 38ms. 1 active and 2 backup servers online. 0 sessions requeued, 0 total in queue.
This is concerning for a few reasons. It might reflect actual galera/mysql outages, or it might reflect haproxy being overly sensitive and causing service interruptions where there should be none.
I don't see any similar behavior in codfw1dev.