statistics::discovery runs stats from analytics/discovery-stats which is deprecated and formerly owned by Max from before the tune-up.
We need to repurpose that puppet configuration to run wikimedia/discovery/golden scripts instead, which are currently set up as a cronjob under my account on stat1002:
0 5 * * * cd /a/discovery/golden/ && sh main.sh >> /home/bearloga/discovery-golden.log 2>&1
Also vaguely related to T129260
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Resolved | None | T157807 Reinstall Analytics Hadoop Cluster with Debian Jessie | ||
| Declined | Ottomata | T159392 Update jq to v1.4.0 or higher | ||
| Resolved | Ottomata | T152712 Replacement of stat1002 and stat1003 | ||
| Resolved | Ottomata | T170471 Move statistics::discovery jobs from stat1002 -> stat1005 | ||
| Resolved | mpopov | T170494 [EPIC] Reconfigure Discovery-Stats on Analytics Cluster | ||
| Resolved | mforns | T173333 Reportupdater outputs files with restricted permissions | ||
| Resolved | Ottomata | T174110 Private data access for non-person user that calculates metrics | ||
| Resolved | Ottomata | T174465 Puppet admin module should support adding system users to managed groups |
@mpopov, it is live! :)
It just isn't announced yet. I'm trying to migrate all puppetized/automated jobs before I start the cat herding with some annoucements. So, ya, you can delay, but that makes it harder for me to get the puppetization of the boxes completely ready for the annoucement.
This isn't blocking anything yet, but the sooner it is done, the easier it is for me to keep cleaning up.
@mpopov, do I need to bother migrating the existing statistics::discovery stuff then? If possible we should probably remove this class, and then create a new more discovery specific one.
It is specific to Discovery :) Max (when he was with Discovery and when Discovery still existed) made it for some of Discovery's stats but it's deprecated, so we can either remove it or repurpose it.
Right, ha, I mean, I'd probably refactor your new stuff out of the statistics::discovery class, instead something else in puppet.
If you don't plan on moving over the stuff in statistics::discovery to stat1005, then great! I can just not migrate it, and we can make something new for your newer discovery jobs.
I just realized that reworking discovery-stats properly will require R package installation stuff from https://gerrit.wikimedia.org/r/#/c/366170/
Change 367930 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[operations/puppet@production] statistics::discovery: Reconfigure for Golden data retrieval
Change 367931 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] Compatibility with Puppetized runs
Change 367930 merged by Ottomata:
[operations/puppet@production] statistics::discovery: Reconfigure for Golden data retrieval
Change 368829 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Include statistics::discovery just in private profile
Change 368829 merged by Ottomata:
[operations/puppet@production] Include statistics::discovery just in private profile
Change 368831 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove re-declaration of 'wikidev' group in discovery class
Change 368831 merged by Ottomata:
[operations/puppet@production] Remove re-declaration of 'wikidev' group in discovery class
Change 367931 merged by Bearloga:
[wikimedia/discovery/golden@master] Compatibility with Puppetized runs
Change 369438 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[operations/puppet@production] statistics::discovery: Fix scheduled command
Change 369438 merged by Ottomata:
[operations/puppet@production] statistics::discovery: Fix scheduled command
Change 371769 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[operations/puppet@production] statistics::discovery: Manage datasets dir
Change 371769 abandoned by Bearloga:
statistics::discovery: Manage datasets dir
Reason:
patching reportupdater to write files with correct permissions instead (I2f5c0138f0df7b19ff0658322b9e7c989e58a7ac)
Change 373938 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[operations/puppet@production] statistics::private: Disable Discovery
Change 373938 abandoned by Bearloga:
statistics::private: Disable Discovery
Reason:
Abandoning in favor of Gehel's I2ae9d3f333d364128f82457680508da287f76477
Change 438125 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[operations/puppet@production] statistics::discovery: re-enable cron job
Change 438125 merged by Ottomata:
[operations/puppet@production] statistics::discovery: re-enable cron job
Disabled the cron job in my account's crontab on stat1005. Will check tomorrow if everything works as it should and the metric calculation by the analytics-search (system) user goes without problems then this task will be done.
Yay! Everything is working correctly now. We ran into a problem with file ownership but the ever-excellent @Ottomata fixed that up and we are now fully puppetized.