Page MenuHomePhabricator

Re-enable puppet on tools-exec-{cyberbot,catscan,gift,wmt}
Closed, ResolvedPublic

Description

12:16 <yuvipanda> valhallasw: if they are large, I'd suggest: 1. force a puppet run, 2. restart them to make sure that files are in /tmp properly, 3. bind mount root to access old /tmp and clean it out

Details

Related Gerrit Patches:
operations/puppet : productiontools: make seperate /tmp hiera-configurable

Event Timeline

valhallasw raised the priority of this task from to Needs Triage.
valhallasw updated the task description. (Show Details)
valhallasw added a project: Toolforge.
valhallasw added a subscriber: valhallasw.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2015, 10:42 AM

@Magnus (for catscan), @revi/@JohnLewis (for wmt), @Cyberpower678 and @Giftpflanze: we need to reboot these -exec hosts to make sure they get updates again.

  • Do you have a preference for when this should happen?
  • Do you have any files in /tmp that need to be retained?
  • Do you expect any issues with completely rebuilding the instances? e.g. the files in /tmp above.
valhallasw updated the task description. (Show Details)May 14 2015, 1:42 PM
valhallasw set Security to None.

tools-exec-gift can be rebooted when there aren't any jobs running (there is a cascade running for some days every 1st and 15th of the month). The best point of time for rebooting it now would be before 2015-05-15T00:00 (UTC). No files in /tmp. I do not expect any issues with rebuilding the instances (but you never know; I also think there is some special sge configuration (?) that would have to be retained).

Change 210918 had a related patch set uploaded (by Merlijn van Deen):
tools: make seperate /tmp hiera-configurable

https://gerrit.wikimedia.org/r/210918

Rebooting the wmt node is fine. I don't foresee any issues with a rebuild either.

valhallasw added a comment.EditedMay 14 2015, 8:44 PM

@yuvipanda noted these nodes are actually all toollabs::node::compute::dedicated (as opposed to toollabs::node::compute::general, where a change was applied that needed a reboot), so we can actually safely re-enable puppet without a reboot!

Applied to:

  • tools-exec-gift
  • tools-exec-wmt
  • tools-exec-cyberbot
  • tools-exec-catscan
valhallasw closed this task as Resolved.May 14 2015, 8:58 PM
valhallasw claimed this task.

And puppet is enabled again on all hosts now!

Change 210918 abandoned by Yuvipanda:
tools: make seperate /tmp hiera-configurable

Reason:
Not needed because the compute::general role isn't applied on dedicated instances

https://gerrit.wikimedia.org/r/210918

Restricted Application added a project: Cloud-Services. · View Herald TranscriptJun 7 2017, 6:51 PM