people born in Africa

last step is to make sure that the airflow dags work next week and then this ticket should be good to be closed

Hey @Miriam do you think you could take a look at this? What information would you need that would be helpful?

Not being done since no longer needed

@akosiaris I just learned that there are archive links that have all of the Flink packages. I'm proposing that we close both this ticket and https://phabricator.wikimedia.org/T266495 and just use the Flink archive links where we won't have to worry about the packages no longer being available.

The problem is that we need more than just the jars. We need the entire tar file. @dcausse suggested people.wikimedia.org but that was considered not a good place for CI to hit

Updating this task to remove the helmfile requirement. That is more of a final step that doesn't include the helm chart. The benchmarking that's been highlighted is not possible on a local machine, we will need to do proper load testing in a staging environment. Logs and monitoring are addressed in the current helm chart, so I'm going to go ahead and mark this ticket as done.

The swift plugin has been added to the docker image and is properly configured in the helm chart.

The rdf spark tools changes have been merged. The airflow work is in progress and waiting on @elukey to merge the puppet patch.

In the interim (until we have the deb package) @dcausse suggested putting the Flink packages on people.wikimedia.org since packages on the official Flink download sites get updated and removed pretty frequently.

also @Miriam both queries still return wikipedia pages that do have images

@dcausse and I discussed getting the page view counts from https://pageviews.toolforge.org/?project=en.wikipedia.org&platform=all-access&agent=user&redirects=0&range=latest-20&pages=Cat|Dog unless you have another suggestion/idea @Miriam

On the wcqs beta host, curl -d "query=select * { sdc:M8979671 wdt:P571 ?o . }" localhost/bigdata/namespace/wcq/sparql returns an item with the correct data and curl -d "query=select * { sdc:M8979671 wdt:P571 ?o . }" localhost:9999/bigdata/namespace/wcq/sparql returns the 1010 date

The dumps have been investigated and the data is correct there, the problem is definitely on the search team's side. It could be with the reload scripts or nginx. It was possible to see the incorrect data on the wcqs server, so the problem is most likely with nginx.

I definitely agree that pipeline lib could be used for Java projects, but adding that current functionality with this project is out of scope. We are moving forward with downloading the jar from archiva as discussed in the meeting on 11/2/2020, but if a Java process for pipeline library is created, we would open to moving towards that in the future.

@Ottomata also suggested via IRC to consider using the event platform instead of kafka

blubberfile is done and the docker image is present in the wikimedia docker repository

Has been deployed and is activated in wikis that do not have issues with the filter. Issues to be addressed in T268788 and T268730

The projects are all related and there are dependencies. Separating the streaming-updater-producer would actually be a significant task. I think that would be a separate project to possibly consider in the future.

For #1, that's is correct. We are currently downloading Flink from the internet. I don't think that's the best idea long term, but it seems fine for now.

Instead of trying to skip CI for every commit, the easiest thing to do would be to move the pipeline directory into its own repo. It's not using any of the code in the current repo anyways.

@akosiaris it was unclear to me whether we need the promote section in the pipeline config. I'm referring to this: https://wikitech.wikimedia.org/wiki/PipelineLib/Reference#Promote and I saw it in a couple of configs here: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/mathoid/+/refs/heads/master/.pipeline/config.yaml#34. Additionally, just so I'm clear, we don't need to do the Jenkins configuration unless we want this to run on every commit (we do not want that). I'm referring to what I saw in the docs: https://wikitech.wikimedia.org/wiki/PipelineLib/Guides/How_to_configure_CI_for_your_project. We just want to be able to rebuild the image whenever we have a new release of the service (on average, once a week)

Sonar should definitely comment on this patch: https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/642203 but the quality gate passed. There's definitely an issue, will be investigating further

@akosiaris I started using the new Java images that you uploaded. I wasn't able to install gpg in the build process. There are some conflicts. We can skip gpg verification of the Flink tar, but I don't think that's a good idea. I will continue to do some debugging.