Page MenuHomePhabricator

Sunset our use of Salt
Closed, ResolvedPublic

Description

cumin is now at a place where it's sufficiently mature enough (or at least as mature) and battle-tested to be able to serve as a SaltStack replacement for our limited use cases, or at least where it's close to becoming that.

For infrastructure simplicity, as well as security reasons (Salt is yet another way for root-level access), we should devise a plan forward to fully deprecate Salt for all our use cases.

At least the following need to happen for this to complete:

  • Migrate everything that's still on trebuchet to scap3
  • Remove trebuchet from everywhere installed + its puppet manifests
  • Migrate debdeploy to cumin, potentially adding features that are missing
  • Retire and/or replace custom salt grains from puppet
  • Find and replace all salt references in documentation (wikitech)
  • Uninstall salt-minion and salt masters from everywhere in production
  • Migrate Cloud VPS to be able to use Cumin as distributed command execution
  • Uninstall salt-minion and salt masters from everywhere in Labs?
  • Remove Icinga monitoring of salt-minions in prod
  • Remove wmf-auto-reimage / wmf-reimage dependencies on salt
  • Search for all Trebuchet-related and Salt/SaltStack-related tasks on Phabricator and resolve them (free-text search too, there are a lot). Archive the Trebuchet and Salt components

Details

Related Gerrit Patches:

Related Objects

StatusAssignedTask
ResolvedMoritzMuehlenhoff
Resolvedthcipriani
InvalidNone
Resolvedthcipriani
DeclinedNone
Resolvedakosiaris
Resolvedelukey
ResolvedYurik
Resolvedthcipriani
Resolvedthcipriani
Resolvedthcipriani
Resolvedthcipriani
ResolvedbearND
ResolvedYurik
Resolvedori
ResolvedSmalyshev
ResolvedDzahn
Resolvedmobrovac
DeclinedNone
Resolvedmobrovac
Resolvedfgiunchedi
Resolvedakosiaris
ResolvedNiharika
Resolvedmmodell
Resolvedthcipriani
Resolvedmmodell
Resolvedmmodell
Resolvedmmodell
Resolvedfgiunchedi
Resolvedmmodell
DeclinedNone
Resolvedthcipriani
Resolvedmobrovac
Resolveddduvall
Resolvedmobrovac
InvalidNone
Resolveddemon
DeclinedKrinkle
ResolvedOttomata
ResolvedOttomata
ResolvedKrinkle
Resolvedmobrovac
ResolvedMoritzMuehlenhoff
ResolvedVolans
ResolvedGehel
Resolveddcausse
ResolvedVolans
ResolvedVolans
Resolvedhashar
ResolvedMoritzMuehlenhoff

Event Timeline

faidon created this task.May 8 2017, 7:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 8 2017, 7:00 PM
jcrespo added a subscriber: jcrespo.May 8 2017, 7:04 PM
  • retire/and or substitute custom salt grains from puppet (?)
Andrew added a subscriber: Andrew.May 8 2017, 7:06 PM
faidon updated the task description. (Show Details)May 8 2017, 7:07 PM
greg added a subscriber: greg.May 8 2017, 7:08 PM
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Peachey88 reopened this task as Open.May 9 2017, 10:09 AM
jcrespo updated the task description. (Show Details)May 10 2017, 4:39 PM
hashar added a subscriber: hashar.

Per T151996#3300615 the deployment of ElasticSearch will not be migrated from Trebuchet to Scap but instead use Debian packages which is T158560. Now a sub task of this sunset task.

Volans updated the task description. (Show Details)Jul 3 2017, 1:28 PM

Change 370993 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Remove custom salt grains due to salt deprecation

https://gerrit.wikimedia.org/r/370993

Change 370993 merged by Jcrespo:
[operations/puppet@production] mariadb: Remove custom salt grains due to salt deprecation

https://gerrit.wikimedia.org/r/370993

demon updated the task description. (Show Details)Sep 18 2017, 9:23 PM
Volans updated the task description. (Show Details)Sep 21 2017, 7:07 PM

Change 380493 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] wmf-auto-reimage: remove Salt actions

https://gerrit.wikimedia.org/r/380493

Change 380493 merged by Volans:
[operations/puppet@production] wmf-auto-reimage: remove Salt actions

https://gerrit.wikimedia.org/r/380493

Dzahn updated the task description. (Show Details)Sep 25 2017, 4:07 PM
Dzahn added a subscriber: Dzahn.

Saw Icinga alerts for salt-minions not running (on argon and chlorine)

"PROCS CRITICAL: 0 processes with regex args '^/usr/bin/python /usr/bin/salt-minion'" (https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=argon&service=salt-minion+processes)

Made me think we should remove all of these monitoring checks now, as part of removing salt. Added a checkbox above.

Dzahn updated the task description. (Show Details)Sep 25 2017, 5:32 PM
Dzahn added a comment.Sep 25 2017, 5:38 PM

Nevermind, it was already removed by Moritz. These alerts i saw were just still there because puppet was disabled on these hosts for some Kubernetes tests by Alex. He fixed these and thye are gone now.

The only ones left now are some UNKNOWNs that are explained by other issues: cp4024 (T174891) , lvs1008 (T150256), db1022/db1023 (T166486)

Once salt is removed in labs we can also remove the "unaccepted salt keys" icinga check on labcontrol1001/1002 (@Andrew) and then that's all and Icinga is free of any salt checks.

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=salt

hashar added subscribers: elukey, MoritzMuehlenhoff.

deployment-prep / integration WMCS projects no more rely on salt (thank you @Volans / @MoritzMuehlenhoff / @bd808 / @elukey and others).

MoritzMuehlenhoff closed this task as Resolved.Sep 28 2017, 12:18 PM
MoritzMuehlenhoff claimed this task.
MoritzMuehlenhoff updated the task description. (Show Details)

This is done (there's a few pending openstack puppet cleanups, but nothing really blocking to close this bug).