This seems to have fixed the issue:

In T318063#8266120, @BTullis wrote:

Yep, as @otto indicated, could you check by running the following please?

sudo -u analytics-research kerberos-run-command analytics-researh yarn logs -appOwner analytics-research -applicationId application_1663082229270_26382

Great!

@xcollazo thanks for the ping. If you mean the research instance on deploy1002, then I've pulled your changes, rebased on main and deployed. If you mean something else, please let me know. Thanks!

Solved by running the following commands on airflow1002:

It seems the majority of the issues described in the task have been resolved. Closing it for now. Feel free to re-open it.

@BBlack thank you. Who would be a good person to review the patch? @Dzahn, would you be able to review the above patch? Thank you.

Remaining items:

1. Implement https://wikitech.wikimedia.org/wiki/Tool:Event_Metrics. I looked more into it. Unfortunately, it does not fit our use case as it's tied to specific Wikimedia projects (Wikipedia, Wiktionary, and Wikivoyage). Before looking more into it, I was under the false impression that we could use it as a pageview counter.

Creating a spark session directly (as opposed to using wmfdata) seems to have fixed the issue.

@mforns, thanks! I'm able to push.

Thank you, both!

Hi @JAllemandou! Thanks for the reply and fixing the tag. I have followed the instructions from the link you shared. I've created a new stacked environment and activated it before running.

Hi @razzi, thanks for the instructions. I got my access.

Thanks!

OK. Thanks!

Indeed. Those schema tables are all empty. I visited the eventgate-validation dashboard on Logstash, but I couldn't find any such requests. Where can I find more about those missing events?

I just saw this task. The above described bug was fixed in https://gerrit.wikimedia.org/r/c/research/recommendation-api/+/669595 .

I think so. For example, here's what was sent:

@Ottomata thanks for reviewing the patch. I've merged it and deployed it.

@Isaac thanks for the link. I've been working on this patch. This is the main file. Feel free to add comments to the patch with new additions or fixes.

Thanks, both.

I think I created an Oozie job once ;) Would you say this is a good starting point: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Oozie? Any other links? Maybe you can share a similar job that I can take an inspiration from?

@akosiaris Thanks for reviewing my patchsets. I was wondering if you've seen my last comment. My understanding is that the patch is very close to being merged, which would be great. Could you please take a look at my benchmark numbers and let me know what you think? Thanks. Here's the patch: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/602527

Auto submit is working.

@hashar o/ Yes, we're still working on it. I'll try to speed it up.

@Reedy I've submitted a patch above. Could you review it? If I understood the tutorial correctly, this should be enough to auto-submit on +2.

Are you still having to submit patches manually? If so, we should probably fix that before calling this bug done (as enabling CI config was done in this task)

Eslint has been added.

As soon as T254211: mostviewed generator not returning any results is fixed, the issue will go away.

@santhosh The mostviewed API started working again. Recommendations also are working for the language wikis where the mostviewed API is working. For example, I see some results on for Russian: https://en.wikipedia.org/api/rest_v1/data/recommendation/article/creation/translation/ru

Hmm, it's still not working on French Wikipedia: https://fr.wikipedia.org/w/api.php?action=query&generator=mostviewed&prop=pageviews

o/ @Milimetric Thanks for looking into it. It's working for me too.

Thanks, @akosiaris!

@santhosh the issue is with the MediaWiki API. The mostviewed generator is not returning anything: https://en.wikipedia.org/w/api.php?action=query&generator=mostviewed&prop=pageviews

@JFishback_WMF, issues mentioned at T251732#6158467 have been addressed.

Just a little status update: I've removed YUI and working on upgrading bootstrap. Since a lot changed between versions 2 and 4, it'll take some time to fully upgrade years 2016 through 2020. I'll let you know when I'm done.

Thanks, @leila!

@leila that sounds good.

@Dzahn we have about 1.7 GB of video recordings of the recent workshop. We're trying to upload those files. Currently, there are only 4 files.

@akosiaris could you please re-review this patch: https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/565788/

@leila head's up that one of the patches is removing the Facebook button. Another is removing statcounter. We probably want to use https://wikitech.wikimedia.org/wiki/Tool:Event_Metrics instead. What do you think?

@elukey I've deployed the fix. Let me know if you still see the issue.

Here's a pull request: https://github.com/wikimedia/restbase/pull/1259

OK, I see. Then I'll not merge that patch. I'll handle the response code directly, possibly returning a 404 for failed health check requests.

@elukey Please take a look at the above patch. Rather than stubbing requests to MW API (and thus making a lot of code changes) I decided not to log health check error messages when the MW API is under stress. If the patch looks good to you, let me know and I'll deploy it.

All changes have been deployed. Feel free to re-open when you see the issue again.

@elukey With the above patch some 503 errors will be logged correctly with informative message. I'll deploy the patch as soon as possible.

In T247732#6064844, @elukey wrote:

@bmansurov thanks for following up! What I'd start doing is to log 50x errors from the MW api in the service logs if possible, so people can easily get what is happening when the recommendation api starts flapping.

This is already happening. See /srv/log/recommendation_api/main.log in scb1001.eqiad.wmnet, for example.

@elukey from the logs I see that both 404 and 503 come in pairs. In the recommendation API we ping the MediaWiki API, which sometimes returns 503. We then return a 404 here. So, the error has really to do with the MW API. Below I'm pasting a sample response from the MW API (prettified for readibility):

Thanks, I can login now.

Thanks both. @leila I'm on it. I need to access logstash.wikimedia.org to see the logs. According to the documentation:

Yes.

@elukey how can I access http://scb1001.eqiad.wmnet:9632? Should I be on some host to ping that URL? Also, where can I see the logs? Thanks!

Oh I see. The new text should be live now.

@Isaac, Can you clear your cache? This is what I see:

The campaign info has been updated.

@Isaac the campaign info has been added. http://recommend-large.wmflabs.org/?campaign=WikiGapFinder

@BBlack is the switchover complete? Can the previous content host stop hosting?

In T244587#5916280, @Isaac wrote:

Thanks for the updates @bmansurov -- code makes sense. I don't see the language overrides on my end so I assume you're waiting to push to the server until we get verification of the other lingering questions.

In case I'm doing something wrong, example: https://recommend-large.wmflabs.org/?campaign=WikiGapFinder&t=de still gives me English -> Swedish even with a full refresh (cmd+shift+R on mac) but https://recommend-large.wmflabs.org/?t=de appropriately makes German the target.

• Per Eric's comments above, I think we will be extending the list of acceptable Wikidata values for sex-or-gender here to include at least transgender female (Q1052281).

Done

@Isaac I've made some changes and created a large labs instance. Here it is: http://recommend-large.wmflabs.org/?campaign=WikiGapFinder

In T244587#5891759, @Isaac wrote:

And a few other questions / comments:

• Thinking about the increased delay for search results, do we know what is causing it? I suspect it's the addition of the claims parameter in the Wikidata query (as opposed to any additional processing that is going on).

Yes, that's true. Also, the site was already slow. I did a quick profiling. Here's the data.

@Astinson if I'm not mistaken the campaigns are already being tracked with eventlogger and also content translation. I can look into it if you need the details.

@Isaac The labs instance has been updated with the latest code. The WikiGapFinder campaign users will see articles about women on the landing page. Subsequent searches will also filter out articles that are not about women. Below are some of the before and after screenshots.

It seems like if the post-hoc filtering that you are suggesting doesn't return enough results, we would have to consider making queries to WDQS instead?

@BBlack the site has been updated. Please turn on the DNS.

@BBlack I'll let you know when we get the latest code into Gerrit.

@leila what do you think about T242374#5872212. Can we get the latest code changes?

@Isaac thanks. I made the change (which will be deployed once we're done with the remaining parts).

My bad. Apparently code runs on Python 3 too. Thanks for the input @Legoktm. I'll look into flake.

@Isaac this is how the new interface will look like on page load. Let me know if you want me to change anything in it.

@Mholloway thanks for the heads up. I'll look into it.