Context: 2021-11-24 PA⇄DE sync meeting
The Data-Engineering team would like Product-Analytics to provide a Jupyter notebook that they (DE) can execute to test configuration & stack changes and check if those changes break end-user querying capability.
There should be at least one query run with wmfdata's spark module.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | BTullis | T295346 Move spark.local.dir to /srv on stat100x | |||
| Duplicate | nshahquinn-wmf | T296420 [REQUEST] Notebook for testing with wmfdata |
@BTullis: would it make sense to have it query via Hive as well? (by the way, wmfdata's hive module is a wrapper for hive CLI) and if yes, would you rather have two separate notebooks – one for each engine?
I already have a open pull request for this in the wmfdata-python repo (although it hasn't been touched in about 10 months). I'll work on getting that finished in the other ticket!