@akosiaris when you get some time, can you please take another look at https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/635074

I've looked into this and unless we want to do a check on rules, then everything is working as expected as pointed out by @kostajh . However, as our master branch is failing on the quality gate, it seems that we need change something about the way we are running Sonar. Perhaps some of the rules that have been disabled should be re-enabled?

@EBernhardson I think the commenting for python/scala would be included as a part of T264877

The original Flink dockerfile exposes ports, which is not an option in blubber. I'm hoping that any networking/ports issues can be resolved in the helm chart

In order to test the image created by the blubberfile, do the following in the rdf repo
cd .pipeline
blubber blubber.yaml production | docker build --tag blubber-flink-test-<version> --file - . -> this builds the image and tags it
docker run blubber-flink-test-<version> -> runs image

After the reindex started, Trey discovered during testing that the homoglypgh plugin was not working in Afrikaans. Erik noted that both tokens were being output: https://phabricator.wikimedia.org/P13023. Cyrillic and Latin are only in text and not text_search, so not getting indexed at query time. We need to ensure that text_search gets a correct copy of the config. After that, there will need to be another reindex.

All patches are merged and items can be deleted successfully

We're having an in person meeting soon, but I just wanted to outline some things I think will need to happen from a high level. This is based on reading https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Components.

I don't see a schema for suppressed deletes in the list of page schemas (https://schema.wikimedia.org/repositories/primary/jsonschema/mediawiki/page/). Is that information intentionally private?
Also, I can't see how having the page namespace and other information in the event meta compromises the event.
Do we want to default the page namespace to 0?
I think we could just listen to the suppressed deletes topic and pass them as regular deletes. Please let me know why we cannot do that

Delete functionality is complete, optimization work in progress to ensure that patches created on the streaming updater consumer that get sent to blazegraph do not contain statements from entities that will be deleted

Wrapping up the last phase of deleting an entity from blazegraph. I have verified that items to be deleted are present in the streaming updater output.

Wrapping up the flink pipeline work, a patch is out for the integration tests and there's some more test cases to be added for clarity on the Decide Mutation Operation to ensure that all delete cases are covered. Pipeline consumer work is up next

Clearing the state is out of the question, I think. Right now, we mark the revision we see a delete at and then we don't allow a Rev Create Event. However, this means that any regular RevCreate events don't allow the item to be reimported ever again. I think it should be something like, if RevCreateEvent revision > DelRev, then go ahead and do a full import.

What do we want to do if we get a PageDelete event for the same item multiple times? Right now we're not tracking deletes, so I'm assuming multiple deletes for the same item probably can't do that much harm.

Shared model delete functionality is complete, currently in the process of adding delete functionality to the flink pipeline

Since the gui directory is overwritten by scap via symlinks, Guillaume proposed that the config files live in /etc/config and that they be symlinked to the gui directory. See more discussion here:https://gerrit.wikimedia.org/r/c/operations/puppet/+/606297. I want to remove the custom config file that lives in the gui-deploy repo to avoid any confusion of config files: https://gerrit.wikimedia.org/r/c/wikidata/query/gui-deploy/+/606545

We need to disable testing profiles on the beta cluster. Could be changed in SearchSatisfaction or don't distribute the testing config to the beta cluster

It looks like this might be an issue with the translation, https://translatewiki.net/w/i.php?title=MediaWiki:Searchresults-title/wa&diff=9392337&oldid=1189285

Here is the nginx example -> https://github.com/nginxinc/NGINX-Demos/tree/master/oauth2-token-introspection-oss

Some notes for when/if we do oauth
Docs -> https://www.mediawiki.org/wiki/OAuth/For_Developers
we might be able to use this npm package for auth with mediawiki
This is an example of a tools project that uses auth and restricts who can log into the tool

As discussed in email with @Zbyszko the script should do the following

Deployed and the example links work. Marking this as done

Docs are updated -> https://wikitech.wikimedia.org/wiki/Wikidata_query_service#Production_Deployment and I verified that the zip file from the latest version can be downloaded from Archiva

Talked to @Gehel and the issue is probably that the archiva credentials that come from analytics have different server ID's than the ones we use. https://github.com/wikimedia/wikimedia-discovery-discovery-parent-pom/blob/master/pom.xml#L917 vs https://github.com/wikimedia/analytics-refinery-source/blob/master/pom.xml#L96. We can either change our pom to match analytics or add a separate credential in jenkins. @Gehel prefers changing our pom to match analytics for uniformity

I know what that is, I forgot to add the archiva deployment profile. I'll put a patch out

@hashar the jenkins job failed due to no git auth to push. you can see it here: https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-docker-wdqs/3/