On stat1007:
```
$ ls -l /srv/published/datasets/discovery/metrics/external_traffic/
total 5548
-rw-rw-r-- 1 analytics-search analytics-search-users 3110291 Feb 9 05:02 referer_data.tsv
-rw-rw-r-- 1 analytics-search analytics-search-users 2565465 Feb 9 05:04 referer_nonbot_data.tsv
$ tail -n 1 /srv/published/datasets/discovery/metrics/external_traffic/referer_data.tsv
2021-02-08 TRUE external (search engine) Daum mobile web 141338
$ ls -l /srv/published/datasets/discovery/metrics/wdqs/
total 848
-rw-rw-r-- 1 analytics-search analytics-search-users 864510 Feb 9 05:07 basic_usage.tsv
$ tail -n 1 /srv/published/datasets/discovery/metrics/wdqs/basic_usage.tsv
2021-02-08 /bigdata/ldf TRUE FALSE 14
```
These files are generated with the help of [[ https://wikitech.wikimedia.org/wiki/Analytics/Systems/Reportupdater | Reportupdater ]], run by [[ https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/discovery/golden/+/refs/heads/master/main.sh | main.sh ]] that is scheduled through `kerberos::systemd_timer` in this manifest: [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/statistics/manifests/discovery.pp | statistics/discovery.pp ]]. That belongs to the miscellaneous jobs manifest ([[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/statistics/explorer/misc_jobs.pp | profile::statistics::explorer::misc_jobs ]]).
**Notes**: Both reports are non-R scripts that run `hive`:
- [[ https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/discovery/golden/+/refs/heads/master/modules/metrics/external_traffic/referer_data | modules/metrics/external_traffic/referer_data ]] (and [[ https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/discovery/golden/+/refs/heads/master/modules/metrics/external_traffic/referer_nonbot_data | referer_nonbot_data ]])
- [[ https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/discovery/golden/+/refs/heads/master/modules/metrics/wdqs/basic_usage | modules/metrics/wdqs/basic_usage ]]
These are necessary for the still-used External Traffic (https://discovery.wmflabs.org/external/) and Wikidata Query Service (https://discovery.wmflabs.org/wdqs/) dashboards.
-----
Steps taken so far:
1. updated the reportupdater submodule to the latest commit (https://gerrit.wikimedia.org/r/c/wikimedia/discovery/golden/+/677244)
2. `sudo -u analytics-search git submodule update` in stat1007:/srv/discovery/golden
3. reset my venv on stat1007 via https://wikitech.wikimedia.org/wiki/Analytics/Systems/Jupyter-SWAP#Resetting_user_virtualenvs
4. ran `pip install -U pid python-dateutil pymysql PyYAML Jinja2 dnspython` (pip install -r reportupdater/requirements.txt` failed)
5. Created /srv/discovery/venv and installed reportupdater's dependencies there as analytics-search (see T279443#6994725), so that this process is no longer dependent on me and my venv
6. Uploaded patch that sets PYTHONPATH in discovery.pp (see [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/678864 | 678864 ]])
-----
Reportupdater requirements.txt error:
```
ERROR: Command errored out with exit status 1:
command: /home/bearloga/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-2bcyzcbl/jinja2_24af26b243e3489b8cb7f52ad83b2dfc/setup.py'"'"'; __file__='"'"'/tmp/pip-install-2bcyzcbl/jinja2_24af26b243e3489b8cb7f52ad83b2dfc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-mqwesini
cwd: /tmp/pip-install-2bcyzcbl/jinja2_24af26b243e3489b8cb7f52ad83b2dfc/
Complete output (5 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-2bcyzcbl/jinja2_24af26b243e3489b8cb7f52ad83b2dfc/setup.py", line 40, in <module>
from setuptools import setup, Extension, Feature
ImportError: cannot import name 'Feature' from 'setuptools' (/home/bearloga/venv/lib/python3.7/site-packages/setuptools/__init__.py)
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/25/c8/212b1c2fd6df9eaf536384b6c6619c4e70a3afd2dffdd00e5296ffbae940/Jinja2-2.6.tar.gz#sha256=a85e185375db75c109f99ecde3d55415f035e7068a82252cf4f16c667ac38f49 (from https://pypi.org/simple/jinja2/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement Jinja2==2.6
ERROR: No matching distribution found for Jinja2==2.6
```