Right now, the wmfdata python package uses spark2 to submit jobs to the spark cluster.
With the recent upgrade to spark3 on our analytics environment, this task is to upgrade these libraries to submit jobs using spark3
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T302819 Replace anaconda-wmf with smaller, non-stacked Conda environments | |||
| Resolved | xcollazo | T300442 Release Wmfdata-Python 2.0 | |||
| Resolved | xcollazo | T318587 Upgrade WMFData Python Package to use Spark3 |
Based on the activity above, it looks like Data-Engineering is planning to do this work.
Should we sync up on this @Ottomata and @Antoine_Quhen ?
From my reading of https://wikitech.wikimedia.org/wiki/Analytics/Systems/Jupyter/Administration and from looking at the jupyter puppet scripts, it looks like getting rid of anaconda-wmf is significantly more complex than to just make wmfdata work with Spark3.
It is more work, but not significantly.
What needs changed is
These need to work with /opt/conda-analytics instead of /usr/lib/anaconda-wmf. Instead of running the custom conda-create-stacked script, the jupyterhub-singleuser-conda-env.sh will just need to run a conda clone (I think) if env name == __NEW__
If we need to be able to use both /usr/lib/anaconda-wmf and /opt/conda-analytics for a while, then it will be a little more work to make jupyterhub-singleuser-conda-env.sh smart enough to do that.
But, I think we can probably avoid that. User's existent conda envs will continue to work as is (as long as we don't uninstall /usr/lib/anaconda-wmf), and after modification, all new envs created by jupyterhub-singleuser-conda-env.sh will just clone /opt/conda-analytics instead. We should probably verify that product analytics will be okay with this change. If they really need to create new enviroments stack-cloned from anaconda-wmf, we could provide them with documentation on how to make them manually.
Yes, the first steps look clear.
We also need to:
About conda stack vs clone:
Some picks from anaconda-wmf conda-create-stacked & conda-activate-stacked:
mkdir -p "${HOME}/.conda"
echo "
always_yes: true
envs_dirs:
- ${HOME}/.conda/envs
pkgs_dirs:
- ${HOME}/.conda/pkgs
- /opt/conda-analytics/pkgs
" > "${HOME}/.conda/condarc"/opt/conda-analytics/bin/conda create \ --offline \ --override-channels \ --use-local \ --clone base \ --name test4 source /opt/conda-analytics/etc/profile.d/conda.sh conda activate test4
add SPARK_CONF_DIR=/etc/spark3/conf in the target environment
Hm, I'm not sure this is quite the right thing to do. I think it would be better to add this to wmfdata defaults, and/or set it via puppet in user's `profile.d` somewhere?
Fetch pkgs from the pkgs/ dir
I don't think should be necessary if we aren't trying to stack (which I don't think we should anymore). I think the pkgs will automatically be used while cloning conda-analytics. Not 100% here, need to verify this.
Cloning conda-analytics
Looks good!
I don't see why we should keep the custom activate/deactivate scripts.
Agree, let's get rid of them!
Fetch pkgs from the pkgs/ dir
Oo! It looks like we can make the conda-analytics .deb package just have an /opt/conda-analytics/condarc file that has the pkgs_dirs set with /opt/conda-analytics/pkgs, then we don't need to modify user .conda/condarc files at all!
(For completeness, all the issues we had discussed upthread regarding the conda-analytics conda environment have been worked on elsewhere: T321088 and T321736.)
Now returning to wmfdata:
A Simple PR to make Spark3 be the default, and instructions on how to override back to Spark2 are here: https://github.com/wikimedia/wmfdata-python/pull/34
Will be testing these changes on the analytics test cluster soon (hopefully tomorrow).
@xcollazo's code has been merged, so I think this is done. Work continues on T300442: Release Wmfdata-Python 2.0.