Page MenuHomePhabricator
Create Task
Maniphest T415609

reconfigure codfw minio cluster after additional devices added
Closed, ResolvedPublic
Actions

Description

Once all of the drives are added from T405982, we need to reconfigure the minio setup so that it will use all 4 hosts and all 4 drives.

Steps required:

  • all drives installed
  • all services referencing writes to minio are stopped/put into maintenance
  • data copied across from codfw cluster to eqiad cluster
    • example command: mcli mirror --insecure franio-codfw/ franio-eqiad/
  • deploy config
  • on codfw franio hosts, wipe all /minio-0*/* including .minio.sys dir
  • ensure all directories are minio:minio perms wise
  • all 4 machines must be online and able to talk to each other on port 9100
  • stop minio
  • start minio
  • copy data back

Rough timing on initial data copy (not catch up syncs) was about 35-40 mins in the two tests. Keep that in mind when planning downtime notifications

Related Objects

Event Timeline

Dwisehaupt created this task.Jan 26 2026, 11:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 26 2026, 11:45 PM
Dwisehaupt mentioned this in T364185: FR-Tech FY2526Q3 maintenance window.Jan 26 2026, 11:46 PM
Dwisehaupt moved this task from Triage to Up Next on the fundraising-tech-ops board.Jan 27 2026, 10:50 PM
Dwisehaupt added a comment.Jan 29 2026, 10:06 PM

Drive addition is scheduled for tomorrow 2026-01-30. We can then do the expansion earlier in the maintenance week.

Dwisehaupt added a comment.Jan 30 2026, 11:59 PM

Drives have been added. New devices have been formatted with XFS filesystem, added to fstab, mount points created, permissions adjusted, and drive mounted. Ready for reconfiguration.

Dwisehaupt claimed this task.Feb 2 2026, 9:25 PM
Dwisehaupt moved this task from Up Next to In Progress on the fundraising-tech-ops board.

one last data copy was done. config was pushed out. data wiped. cluster reconfigured with 4 hosts and 4 drives. cluster came up clean. data copy back to codfw in progress.

Dwisehaupt closed this task as Resolved.Feb 2 2026, 11:54 PM
Dwisehaupt moved this task from In Progress to Done on the fundraising-tech-ops board.
Dwisehaupt added a subscriber: AStein-WMF.

For reference, the data copy this last time took 78 minutes as compared to the previous runs that were reliably in the 35-40 min range each time I did a fresh reclone. Something to be aware of if we do it in the future.

Also, there was an oversight in the planning for this. In all of the tests and VB setups I didn't have access tokens for multiple users set up. Due to this, I missed the fact that they were not retained in the process. @AStein-WMF was able to help and recreate the production access tokens. This had to be done on the command line since minio removed the ability to do anything but bucket management through the web UI.

We will write up steps on how to back up these access tokens for future backup/recover/transfer processes.

As of now, minio is back up in codfw with all 4 hosts in a 16 drive config. Access is restored, metabase is rendering data, and jobs are running.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits