Page MenuHomePhabricator
Create Task
Maniphest T381690

pt-heartbeat updates table even if read_only=1
Open, LowPublic
Actions

Description

While working on T352206: [toolsdb] Upgrade to MariaDB 10.6, I needed to restart the old primary temporarily to check something. MariaDB restarted with read_only=1 but I was surprised by the fact that the GTID was increasing.

I then realized I forgot to stop pt-hearbeat and it was still updating the heartbeat table even if read_only=1.

I think it would be nicer if pt-heartbeat respected the read_only setting, this should be possible if it connects with a user other than root, or maybe we can remove the READ_ONLY ADMIN privilege from the root user.

Related Objects

Event Timeline

fnegri created this task.Dec 6 2024, 6:48 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptDec 6 2024, 6:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui subscribed.Dec 6 2024, 7:00 PM

Keep in mind that for now we do need it as root needs to perform schema changes on replicas which are always on RO.

Also a RO system with pt-heartbeat running allow us to perform operations and leave reads unaffected as MW sees no lag on the hosts. If pt-heartbeat would also stop replicas would lag (MW uses pt-heartbeat to measure lag). Of course I'm talking about production here. For other infra we can see what options and cons/pros

fnegri triaged this task as Low priority.Dec 9 2024, 8:27 AM

@Marostegui thanks for the info!

For ToolsDB, I'm not sure we want to remove READ_ONLY ADMIN for root, but having a dedicated user for pt-heartbeat would probably work.

I'll keep this open at low priority, feel free to remove the Data-Persistence tag.

Marostegui added a comment.Dec 10 2024, 8:34 AM

I will remove the tag but I will keep subscribed.

Marostegui removed a project: Data-Persistence.Dec 10 2024, 8:34 AM
taavi edited projects, added Toolforge; removed Data-Services.Jan 25 2025, 2:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits