Now that we have a more comprehensive solution for installing python (and other) packages for ad-hoc usage on analytics clients (AKA stat boxes), we should remove all of the python-* deb packages from profile::analytics::cluster::packages::statistics, and ask that users who need them install them into their conda envs instead.
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Ottomata | T272313 Newpytyer python spark kernels | |||
| Resolved | Ottomata | T275786 Remove all debian python-* and other user requested packages installed for analytics clients, use conda instead | |||
| Open | None | T286743 Refinery python code should use anaconda-wmf |
Event Timeline
Comment Actions
This will allow us to stop including python specific versions of packages with our spark2 distribution.
Comment Actions
Change 692682 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] Remove profile::analytics::cluster::packages::hadoop
Comment Actions
Change 692682 merged by Ottomata:
[operations/puppet@production] Remove profile::analytics::cluster::packages::hadoop
Comment Actions
Mentioned in SAL (#wikimedia-analytics) [2021-05-18T18:46:17Z] <ottomata> removing extraneous python-kafka and python-confluent-kafka deb packages from analytics cluster - T275786
Comment Actions
Change 697804 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] Remove more python packages from stat boxes
Comment Actions
Change 697804 merged by Ottomata:
[operations/puppet@production] Remove more python packages from stat boxes
Comment Actions
Mentioned in SAL (#wikimedia-analytics) [2021-06-02T18:09:39Z] <ottomata> remove .deb packages from stat boxes: python3-mysqldb python3-boto python3-ua-parser python3-netaddr python3-pymysql python3-protobuf python3-unidecode python3-oauth2client python3-oauthlib python3-requests-oauthlib python3-ua-parser - T275786
Comment Actions
Change 698556 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] analytics cluster - Remove bigtop and stretch overrides where not needed
Comment Actions
Change 698556 merged by Ottomata:
[operations/puppet@production] analytics cluster - Remove bigtop and stretch overrides where not needed
Comment Actions
Change 698575 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] analytics cluster - Remove more deb packages that sbould not be needed
Comment Actions
Change 698575 merged by Ottomata:
[operations/puppet@production] analytics cluster - Remove more deb packages that sbould not be needed
Comment Actions
Mentioned in SAL (#wikimedia-analytics) [2021-06-07T17:07:41Z] <ottomata> remove packages from an clsuter nodes: sudo apt-get -y remove r-cran-rmysql python3-matplotlib python3-sklearn python3-enchant python3-nltk gfortran liblapack-dev libopenblas-dev - T275786
Comment Actions
Change 699792 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] analytics cluster - Remove more deb packages that sbould not be needed
Comment Actions
Change 699792 merged by Ottomata:
[operations/puppet@production] analytics cluster - Remove more deb packages that sbould not be needed
Comment Actions
Mentioned in SAL (#wikimedia-analytics) [2021-06-14T18:45:03Z] <ottomata> remove packges from hadoop common nodes: sudo cumin 'R:Class = profile::analytics::cluster::packages::common' 'apt-get -y remove python3-pandas python3-pycountry python3-numpy python3-tz' - T275786
Comment Actions
Change 702647 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] Require python3-pandas for performance asoranking
Comment Actions
Change 702647 merged by Ottomata:
[operations/puppet@production] Require python3-pandas for performance asoranking
Comment Actions
Change 722368 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] Add comment about 'statistics' packages
Comment Actions
Change 722368 merged by Ottomata:
[operations/puppet@production] Add comment about 'statistics' packages
Comment Actions
I'm going to close this task. Remaining work can be done as part of T286743: Refinery python code should use anaconda-wmf