Page MenuHomePhabricator

Sunset our use of Salt
Closed, ResolvedPublic

Description

cumin is now at a place where it's sufficiently mature enough (or at least as mature) and battle-tested to be able to serve as a SaltStack replacement for our limited use cases, or at least where it's close to becoming that.

For infrastructure simplicity, as well as security reasons (Salt is yet another way for root-level access), we should devise a plan forward to fully deprecate Salt for all our use cases.

At least the following need to happen for this to complete:

  • Migrate everything that's still on trebuchet to scap3
  • Remove trebuchet from everywhere installed + its puppet manifests
  • Migrate debdeploy to cumin, potentially adding features that are missing
  • Retire and/or replace custom salt grains from puppet
  • Find and replace all salt references in documentation (wikitech)
  • Uninstall salt-minion and salt masters from everywhere in production
  • Migrate Cloud VPS to be able to use Cumin as distributed command execution
  • Uninstall salt-minion and salt masters from everywhere in Labs?
  • Remove Icinga monitoring of salt-minions in prod
  • Remove wmf-auto-reimage / wmf-reimage dependencies on salt
  • Search for all Trebuchet-related and Salt/SaltStack-related tasks on Phabricator and resolve them (free-text search too, there are a lot). Archive the Trebuchet and Salt components

Related Objects

StatusSubtypeAssignedTask
ResolvedMoritzMuehlenhoff
Resolvedthcipriani
InvalidNone
Resolvedthcipriani
DeclinedNone
Resolvedakosiaris
Resolvedelukey
ResolvedYurik
Resolvedthcipriani
Resolvedthcipriani
Resolvedthcipriani
Resolvedthcipriani
Resolved bearND
ResolvedYurik
Resolvedori
ResolvedSmalyshev
ResolvedDzahn
Resolved mobrovac
DeclinedNone
Resolved mobrovac
Resolvedfgiunchedi
Resolvedakosiaris
ResolvedNiharika
Resolved mmodell
Resolvedthcipriani
Resolved mmodell
Resolved mmodell
Resolved mmodell
Resolvedfgiunchedi
Resolved mmodell
DeclinedNone
Resolvedthcipriani
Resolved mobrovac
Resolveddduvall
Resolved mobrovac
InvalidNone
Resolved demon
DeclinedKrinkle
ResolvedOttomata
ResolvedOttomata
ResolvedKrinkle
Resolved mobrovac
ResolvedMoritzMuehlenhoff
ResolvedVolans
ResolvedGehel
Resolved dcausse
ResolvedVolans
ResolvedVolans
Resolvedhashar
ResolvedMoritzMuehlenhoff

Event Timeline

  • retire/and or substitute custom salt grains from puppet (?)
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
hashar subscribed.

Per T151996#3300615 the deployment of ElasticSearch will not be migrated from Trebuchet to Scap but instead use Debian packages which is T158560. Now a sub task of this sunset task.

Change 370993 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Remove custom salt grains due to salt deprecation

https://gerrit.wikimedia.org/r/370993

Change 370993 merged by Jcrespo:
[operations/puppet@production] mariadb: Remove custom salt grains due to salt deprecation

https://gerrit.wikimedia.org/r/370993

Change 380493 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] wmf-auto-reimage: remove Salt actions

https://gerrit.wikimedia.org/r/380493

Change 380493 merged by Volans:
[operations/puppet@production] wmf-auto-reimage: remove Salt actions

https://gerrit.wikimedia.org/r/380493

Dzahn subscribed.

Saw Icinga alerts for salt-minions not running (on argon and chlorine)

"PROCS CRITICAL: 0 processes with regex args '^/usr/bin/python /usr/bin/salt-minion'" (https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=argon&service=salt-minion+processes)

Made me think we should remove all of these monitoring checks now, as part of removing salt. Added a checkbox above.

Nevermind, it was already removed by Moritz. These alerts i saw were just still there because puppet was disabled on these hosts for some Kubernetes tests by Alex. He fixed these and thye are gone now.

The only ones left now are some UNKNOWNs that are explained by other issues: cp4024 (T174891) , lvs1008 (T150256), db1022/db1023 (T166486)

Once salt is removed in labs we can also remove the "unaccepted salt keys" icinga check on labcontrol1001/1002 (@Andrew) and then that's all and Icinga is free of any salt checks.

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=salt

hashar added subscribers: elukey, MoritzMuehlenhoff.

deployment-prep / integration WMCS projects no more rely on salt (thank you @Volans / @MoritzMuehlenhoff / @bd808 / @elukey and others).

MoritzMuehlenhoff claimed this task.
MoritzMuehlenhoff updated the task description. (Show Details)

This is done (there's a few pending openstack puppet cleanups, but nothing really blocking to close this bug).