I drafted a plan for deploying Eventgate-wikimedia and Eventstreams services to staging and production.

Thanks!

Sure. I wil try to do that right away.

I have removed the non PipelineLib repos from this task. Migrating those will be tracked in T366611: Migrate Data Engineering NodeJS library repos to GitLab.

@dancy Thank you! I have just done that. See merge request here.

Thanks for the reminder @tchin . Is there some restriction on the gitlab trusted runners repo. I an unable to push to the repo. @tchin @dancy

@Jdforrester-WMF thanks for the response. How do we go about this? DO we submit a ticket to release engineering or we do ourselves? DO you mind helping me with a guide for the process while I confirm if this is what we want from my the data engineering team.

In regards to node-rdkafka-statsd. @thcipriani what does archive mean for release engineering? Will you be deleting or will the repo be kept as read only?

In T344730#9838248, @Ottomata wrote:

Nit:

Can we rename https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-services-eventstreams to just https://gitlab.wikimedia.org/repos/data-engineering/eventstreams ? The "mediawiki-services" bit is vestigial and old gerrit namespacing.

https://medal.ctb.upm.es/internal/gitlab/help/user/project/settings/index.md#renaming-a-repository

I have imported the following repos to gitlab:

Based on the Mediawiki History checker use case, the RowLevelSchemaValidator has some Limitations that may not allow us to use it for our use cases:

1. The method has only String, Integer, Decimal and Timestamps Column definition. We need other column definitions like Double, Map, etc.
2. Even if we decide to use the Decimal Column Definition, this definition does not support Maximum value and Minimum Value customisations of the column. In the MediawikiHistory Checker the growth column needs to have a Max and Min value set.

In the code implementation for change in size detection, I have compared previous snapshot and current snapshot by reading previous snapshot into a dataframe.
One TODO will be to use the AWS Deequ Anomaly detectiom and filesystem repository capability to implement this check.

We added the following:

The suggested approach for this will be to use spark to run the queries after which result will be saved in the cluster. However, spark saves files in folder and we don't want to have different folders for each querry result. We want to put all the output files (report) in one location with is already rsynced to report server. Thus we would use our hdfsarchive operator to move the generated output from the spark output path to the final destination.
To start, we would migrate the queries in the browser folder first.

Update a patch containing 2 hql files required to create and update iceberg version of browser_general tables respectively.

RefineSanitize still was failing on Friday, so Ben and Joseph helped to successfully revert RefineSanitize to spark2 by reverting the refinery jar version from v0.2.16 back to v0.1.15. https://gerrit.wikimedia.org/r/930765 to allow it work during the weekend.

Thank you @hashar

Currently getting the error below when running refine jobs with spark3: Yet to get a solution for this but will update once I get one.

I got similar error when deploying analytics refinery:

See wikitech documentation here.

Here is a google doc containing a documentation on the data loss

Regarding the new column, I like to get suggestions on the name to use for the new field. I am thinking referer_data. Anyone has a better name?

@Mayakp.wiki I ran an analysis on the UDF which would be used to populate the data of the new field and posted the result in the parent ticket T309769 and there is a comment thread on it.

I ran the UDF on a day's data and extracted the top 1000 referer's for that day to show the impact of the GetRefererDataUDF on referers. You can check the spreadsheet and a little doc on it.

Traffic Can you please confirm that there were cases of pages served in eqsin but not reported in webrequest logs.

@taavi. done

@bd808 and @Platonides . I have been now have access to cloud bastion. Here is the result.

@Platonides Here is the result when I run on a production host.

@Mayakp.wiki We are introducing a new new column to wmf.webrequest table of a struct data type that would contain same data in existing referer_class column as well as the referer’s name. However the referer_class column won't be removed now. It would only be removed after all the downstream have been changed.

Before now I haven't ssh to any cloud or toolforge instance. Is there another verification method?

Please I have been unable to login to my wikitech account and do an important editing because of this issue. I would appreciate any form of assistance as this is urgent.

@Aklapper I am unable to 'ssh bastion.wmcloud.org' or ssh login.toolforge.org