@thcipriani can you approve my account for gitlab, so I can complete https://phabricator.wikimedia.org/T344746

I did quite an overhaul of the code here: https://github.com/martaannaj/RecommenderServer to comply with the security and deployment related issues. (see https://phabricator.wikimedia.org/T292110#7613470 )

@thcipriani Can you help me out? We recently updated to a new golang version, and want to go forward getting this deployed properly.

@thcipriani Development is taking place here: https://github.com/martaannaj/RecommenderServer

@thcipriani could you help me forward with this? If preferred, we could set up a 20 minute call to get this all set and done.

@thcipriani I would like to take care of this. Since the version on gerrit is pretty outdated by now, is it possible to directly work with the latest which we develop on github?
I'd also be happy to move development completely to gitlab, if we can get our CI ported. Can we have runners for that on the gitlab installation?

We do our development on github. Does it make more sense to restart with a new repository on gitlab to mirror that, or better to migrate?

The testing code is now implemented, and we found two small issues with it. These have now been resolved and the code is simplified further.

Do we have a timeline for this move already?
Is it better to not update the gerrit repo at the moment? Our main development takes place on github and we have quite a few changes in the pipeline.

Hi @akosiaris , apologies for the slow progress.
We have been updating several things, but I didn't follow up with the testing code needed for the new parts. I have blocked time in the coming days to proceed with this and then we can merge it in.

Regarding the option of using a batch of queries to an external database; the issue is that what we are creating is a specialized index specifically for what we need. What we perform is a tree traversal were at each node a new decision is made. To do something like this with a database, one would have to fire a series of queries where each of the queries is dependent on all the results of the previous ones.

An update on the current status, mainly regarding the index file:

I merged the pull request on github now.

@Joe We now made the changes to use the bullseye distribution and the provided image with go installed See: https://github.com/martaannaj/RecommenderServer/pull/22/files
@QChris We just went ahead and added that file in our repository.

@Joe : would it be fine if we include the index file into the image for the A/B testing phase? If that is successful, we can see whether it makes sense to move it to a different location.

@QChris I noticed the addition of the .gitreview file on gerrit. Is this file needed? If so, we would merge it into our github repository, so we can keep the active development there and synchronize with gerrit.

We'd be happy to receive a patch or pull request.

@Joe for the base image, would you recommend our current approach of starting from an 'empty' image and downloading the latest go distribution ourselves, or should we take this one: https://docker-registry.wikimedia.org/golang1.17/tags/
I am not sure how stable these images are.

@Joe we have created files for blubber before, I assume what is needed is very similar to that? I am not sure I know what the deployment-charts repository is . The code is under development on https://github.com/martaannaj/RecommenderServer/ also the blubber configuration files are there.

@sbassett @Reedy thank you very much for your effort!

Ok, I added myself and @Martaannaj to MediaWiki-extensions-PropertySuggester and change the status to declined (not sure that is the right status, should it be resolved?)

@sbassett Is that something which should be checked now, during the security readiness review, or only later upon deployment?

Thanks @sbassett . I didn't realize it was possible to run semgrep without posting the results to their service. They actually have a configuration available which can be used without that feature. I have now configured that.

I have now added gokart. The github action was not working out of the box, because of some missing configuration parameters in the example. I opened a pull request for that.

• Currently development is ongoing on https://github.com/martaannaj/RecommenderServer Down the line the canonical repository might move to gerrit.
• The hosted service is currently at https://recommender.wmcloud.org/recommender but also this might still change.

@Aklapper : thanks. Process initiated. https://phabricator.wikimedia.org/T296599

We went forward fixing multiple aspects:

For the security policy, I created a first draft. I would prefer to use phabricator for any security issues. Smaller issues can go as issues on github. https://github.com/martaannaj/RecommenderServer/blob/main/SECURITY.md

I can see the property suggester events.

The example I was looking at fits the case you describe.

With much appreciated help from @Ottomata , we got to see some events using kafkacat on deployment-kafka-jumbo-2 now. To check whether the events are correctly generated, this seems sufficient.
One thing which got done on the way is including the config of these streams from this patch https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/708592 into the main config.

Updated a patch to resolve the issues.

@Ottomata Is this a patch you can review?

Turned out the instructions which we had found before are outdated (now marked as deprecated). New ones are here: https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To#Deployment
Accordingly, I created a patch to get our streams activated. https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/708592
At the moment, they are only in the -labs configuration. After making sure that works, we can move the configuration to main config and get it merged during a backport window.

We (@Michaelcochez and @Martaannaj ) should have access now. We still have to figure out where the events are logged. The /srv/log/eventlogging directory were logging should happen according to TestingOnBetaCluster seems empty.