| MoritzMuehlenhoff | |
| Jul 10 2019, 10:58 AM |
We recently introduced two new LDAP readonly replicas which are behind LVS, the Cloud VPS instances are in the process of migrating fully towards those. This task tracks the migration of our web-based services (where applicable).
Once all readonly consumers are migrated we can doublecheck via tcpdump for a few days and restrict access to serpens/seaborgium to the services performing actual write changes.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | MoritzMuehlenhoff | T227650 Migrate web services using LDAP authentication towards the readonly LDAP replicas | |||
| Resolved | MoritzMuehlenhoff | T227778 Create an LDAP replica in codfw (using LVS) |
Change 524479 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] piwik: Read LDAP servers from Hiera and switch to read-only replicas
Change 524487 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] puppetboard: Read LDAP servers from Hiera and switch to read-only replicas
Change 524489 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] debmonitor: Read LDAP servers from Hiera and switch to read-only replicas
Change 524494 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] netbox: Read LDAP server from Hiera and switch to read-only replicas
Change 524479 merged by Elukey:
[operations/puppet@production] piwik: Read LDAP servers from Hiera and switch to read-only replicas
Change 524519 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] grafana: Read LDAP servers from Hiera and switch to read-only replicas
Change 524540 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] icinga: Read LDAP servers from Hiera and switch to read-only replicas
Change 523992 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] static-rt: LDAP config, use ro, Hiera and new password classes
Change 523994 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] librenms: use ldap-ro, stop using ldap-labs, use Hiera
Change 523991 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] microsites/transparency: use ldap-ro, stop using ldap-labs, use Hiera
Change 523995 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] xhgui::app: use ldap-ro, stop using ldap-labs, use Hiera
Change 523993 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] tendril: use ldap-ro, use Hiera, refactor to profile
Change 524583 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] netbox: stop using ::passwords::ldap:wmf_cluster
Change 524584 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: stop using ::passwords::ldap:wmf_cluster
Change 524585 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] grafana: stop using ldap-labs, use ldap-ro
Change 524586 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] graphite: use ldap-ro, stop using ldap-labs, use Hiera
Change 524585 abandoned by Dzahn:
grafana: stop using ldap-labs, use ldap-ro
Reason:
duplicate of https://gerrit.wikimedia.org/r/c/operations/puppet/ /524519/
Change 524584 merged by Dzahn:
[operations/puppet@production] icinga: stop using ::passwords::ldap:wmf_cluster
Change 524487 merged by Muehlenhoff:
[operations/puppet@production] puppetboard: Read LDAP servers from Hiera and switch to read-only replicas
Change 524586 merged by Muehlenhoff:
[operations/puppet@production] graphite: use ldap-ro, stop using ldap-labs, use Hiera
Change 524494 merged by Muehlenhoff:
[operations/puppet@production] netbox: Read LDAP server from Hiera and switch to read-only replicas
Change 523994 merged by Muehlenhoff:
[operations/puppet@production] librenms: use ldap-ro, stop using ldap-labs, use Hiera
Change 524742 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] yarn: Read LDAP servers from Hiera and switch to read-only replicas
Change 524749 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Superset: Read LDAP servers from Hiera and switch to read-only replicas
Change 524752 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] turnilo: Read LDAP servers from Hiera and switch to read-only replicas
Change 524489 merged by Muehlenhoff:
[operations/puppet@production] debmonitor: Read LDAP servers from Hiera and switch to read-only replicas
Change 524540 merged by Muehlenhoff:
[operations/puppet@production] icinga: Read LDAP servers from Hiera and switch to read-only replicas
Change 524752 merged by Elukey:
[operations/puppet@production] turnilo: Read LDAP servers from Hiera and switch to read-only replicas
Change 524742 merged by Elukey:
[operations/puppet@production] yarn: Read LDAP servers from Hiera and switch to read-only replicas
Change 524749 merged by Elukey:
[operations/puppet@production] Superset: Read LDAP servers from Hiera and switch to read-only replicas
Change 524519 merged by Muehlenhoff:
[operations/puppet@production] grafana: Read LDAP servers from Hiera and switch to read-only replicas
Change 523991 merged by Muehlenhoff:
[operations/puppet@production] microsites/transparency: use ldap-ro, stop using ldap-labs, use Hiera
Change 524791 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] profile::webperf::xhgui: Read LDAP servers from Hiera and switch to read-only replicas
Change 524798 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] librenms: Read LDAP servers from Hiera and switch to read-only replicas
Change 524810 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] maintain_dbusers: Read LDAP servers from Hiera and switch to read-only replicas
Change 523995 merged by Dzahn:
[operations/puppet@production] xhgui::app: use ldap-ro, stop using ldap-labs
Change 523992 merged by Dzahn:
[operations/puppet@production] static-rt: LDAP config, use ro, Hiera and new password classes
Change 524583 merged by Dzahn:
[operations/puppet@production] netbox: stop using ::passwords::ldap:wmf_cluster
Change 523993 merged by Dzahn:
[operations/puppet@production] tendril: use ldap-ro, use Hiera, refactor to profile
Change 524798 merged by Dzahn:
[operations/puppet@production] librenms: Read LDAP servers from Hiera and switch to read-only replicas
Change 524791 merged by Muehlenhoff:
[operations/puppet@production] xhgui: Read LDAP servers from Hiera and switch to read-only replicas
Change 525057 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] kibana: Read LDAP servers from standard Hiera and switch to read-only replicas
Change 525069 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] kibana: Switch to read-only LDAPreplicas
Change 525069 merged by Muehlenhoff:
[operations/puppet@production] kibana: Switch to read-only LDAPreplicas
Change 524810 merged by Muehlenhoff:
[operations/puppet@production] maintain_dbusers: Read LDAP servers from Hiera and switch to read-only replicas
The following services have been converted to use the read-only replicas:
Once the remaining Cloud VPS instances are switched to use the eqiad r/o servers, we can narrow down access to the r/w servers in Ferm.
Change 525057 abandoned by Muehlenhoff:
kibana: Read LDAP servers from standard Hiera and switch to read-only replicas
The last remaining patch should be https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/525220/ (and some semi-related refactoring to switch more prod hosts to the new ldap Hiera settings.
Once that is done, I'm planning to tcpdump remaining accesses to the r/w servers to doublecheck and when that is all tightened down, ferm rules can be added to restrict access to the r/w hosts.
Removing task assignee due to inactivity, as this open task has been assigned for more than two years. See the email sent to the task assignee on February 06th 2022 (and T295729).
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome.
If this task has been resolved in the meantime, or should not be worked on ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator.
I ran tcpdump for the LDAP ports on seaborgium and serpens for a little over an hour:
On serpens:
On seaborgium:
This looks promising, the only cases which a closer look at contint1001 and archiva, which should both be doing just fine by only using the replicas. Then we'd be close to restricting access to the r/w servers via Ferm.