Thank you @hashar

Currently getting the error below when running refine jobs with spark3: Yet to get a solution for this but will update once I get one.

I got similar error when deploying analytics refinery:

See wikitech documentation here.

Here is a google doc containing a documentation on the data loss

Regarding the new column, I like to get suggestions on the name to use for the new field. I am thinking referer_data. Anyone has a better name?

@Mayakp.wiki I ran an analysis on the UDF which would be used to populate the data of the new field and posted the result in the parent ticket T309769 and there is a comment thread on it.

I ran the UDF on a day's data and extracted the top 1000 referer's for that day to show the impact of the GetRefererDataUDF on referers. You can check the spreadsheet and a little doc on it.

Traffic Can you please confirm that there were cases of pages served in eqsin but not reported in webrequest logs.

@taavi. done

@bd808 and @Platonides . I have been now have access to cloud bastion. Here is the result.

@Platonides Here is the result when I run on a production host.

@Mayakp.wiki We are introducing a new new column to wmf.webrequest table of a struct data type that would contain same data in existing referer_class column as well as the referer’s name. However the referer_class column won't be removed now. It would only be removed after all the downstream have been changed.

Before now I haven't ssh to any cloud or toolforge instance. Is there another verification method?

Please I have been unable to login to my wikitech account and do an important editing because of this issue. I would appreciate any form of assistance as this is urgent.

@Aklapper I am unable to 'ssh bastion.wmcloud.org' or ssh login.toolforge.org

• Indeed this will alter the referer_class field as some rows previously labelled as external will now be labelled as external (media sites) class.

In the current patch we have a updated our referer classifier to include "external (media sites)" class to represent the list of sites to track. This is in addition to the previous classes: unknown, internal, external (search engine) and external. The classifier would also identify the Name of the site if it's a search engine or a media site (eg Youtube, Facebook, etc.).
Next step:

1. Test for performance and optimise to include caching if necessary.
2. Create a new UDF that will Identify the Names of the search engine and media sites by using the referer classifier.

We now have a custom AuthConfFactory that will be passed as a parameter to the job using spark.cassandra.auth.conf.factory. This should be deployed today. @Ottomata @Eevans The next step would be to have the file containing the Cassandra password loaded to HDFS.

There were 2 fix made in this repo:

Sure @JArguello-WMF I can take it. Would sync with @BTullis.

The HdfsArchiver Operator fails to run successfully on skein because we do not have Scala 2.12.10 version installed on the workers yet. For now, the Scala version 2.12.10 is provided by the spark 3 assembly which can only be found on an-launcher.

The airflow dags have been update to version 2.3.2. However, before this change can be merged, we'll need to perform

1. a puppet change on the airflow.cfg file.
2. upgrade the airflow deb used by all airflow instances.

There will be seperate tickets to track these steps above.