@Stefan_Schneider_WMDE I'm on it - T192659.

• Spark launches in the Dynamic Allocation mode - checked;
• With different settings (e.g. spark.executor.instances, spark.dynamicAllocation.minExecutors), it will use several executors, but from what I see from the Spark UI on yarn.wikimedia.org it always uses only one executor at the time; however
• that was a test on a rather small data set (~ 330K rows, 5 columns or so).

@Ottomata Everything seems to be fine thus far. However, I am not testing any really large datasets yet ( nycflights13, and that would be some 336776 rows, 16 columns).

@elukey The ideal situation would be to have the same version of R installed across all worker nodes. At this point, we have R 3.1.1 (2014) on Jessie hosts and R 3.3.3 (2017) on Stretch hosts. However, there were no backwards compatibility issues in R for years already, so let's not worry about this now. If it turns out that the discrepancy causes any trouble upon testing, I will report back - but I don't think it will. Thanks for support!

• Elementary tests with {SparkR} conducted: works.
• In the following hours I will be testing MLlib procedures with larger data sets.
• Reporting back here and documenting on Analytics/Systems/Cluster/Spark on Wikitech.
• Finally, I will try to make {sparklyr} connect (T139487) too; we will probably learn something even if it fails to connect again.

@Ottomata Fixed with the arguments to sparkR.session() and the preceding Sys.setenv() R call as in the following:

@Ottomata Andrew, seems like we have R on workers, thank you.
However, any operation, for example:

Discard, please. All the WDCM dashboards are running nicely now. Closing the ticket.

The first version of this Dashboard is finally deployed: http://wdcm.wmflabs.org/WDCM_BiasesDashboard/, and can be announced.

This is fixed. A bug was related to the leaflet maps implementation in R.

• All dashboards are updated and running.

• The WDCM system update is currently rolled-back to December 2017.
• The dashboards will be available with old data until the current maintenance operation is completed.

Done.

Just to make sure, since the explanatory text still mentions "90 days", we are passing the 90 days, right? Maybe we should adapt the text then.

@Addshore Thanks a lot!

Please take a look at the Dashboard now. As of the update problem ("All the graphs seem to consider the last day to be March 1st. ") - someone has introduced a new eventLogging SQL schema for the AdvancedSearchExtension without notifying me; the Dashboard now relies on a concatenation of data from two consecutive eventLogging schemes on the first data intake, and the proceeds to work with the most recent version. All in all, everything should be fixed now.

@Stefan_Schneider_WMDE Got it.

-> "growing graphs"

• I think I know the solution that will fit what you are describing. Reporting back as soon as implement it on the Dashboard.

"Also, would it be possible to display all lines at the same time, and only focus on one, if wanted?"

• Depends on (a) whether the above mentioned visualization can support something like that, and (b) whether the data scales so that it makes sense to have different keywords represented on the same chart. I need to experiment a bit with this and than I can let you know. I will do my best to make it happen, of course.

... on the first tab (where you did not actually change anything) it looks like something is broken...

• That means that something is wrong with the update module - fixing it is now a priority.

@Lea_WMDE Please review: T186697#4099426.

@Lea_WMDE Please review: T186697.

@Lea_WMDE The Advanced Search Extension Dashboard now presents the data described in:

• A. The scale will be fixed.
• B. Not all graphs should fit on the same panel, because the Advanced Search Extension are on a different scale (hundreds of observations vs. many thousands of observations for Special:Search). You won't be able to read anything from such charts.Q: did you mean: have one chart and a drop-down menu on the Dashboard to change the metric that is being presented?

@Lea_WMDE Please, you still did not clarify what do you mean by ...within the wikimedia cluster.

The charts refer to the previous 90 days.

@Lea_WMDE Initial data acquisition is completed, you can expect the first visualization examples later during the day.

(Finally - this really took a lot to complete) the Dashboard is ready to go public.

@Lea_WMDE Ok, the procedure will be run on daily basis as of today, and I will develop a feature on the Advanced Search Extension dashboard for you to track the progress and visualize the data. Resources: storage is no problem, human could be (namely, me: I am doing this by combining HiveQL and R processing, and I am constantly in a doubt on whether a person with a better knowledge of HiveQL than mine could improve the current workflow; that being said, I'll see to optimize this as much as I can).

@Lea_WMDE Sorry, my bad, a technical thing: ETL (abbr.) = Extract-Transform-Load, a set of procedures used to re-structure the data from a database table (typically) into another table that will be used for analytics.

One question to be sure: In this data, we are not collecting searches that land on special:search, but only searches that start (and land) on the special:search form, right?

@Lea_WMDE Please have a look and let me know do you really want any analysis based on the following data.

@Lydia_Pintscher @Jan_Dittrich What is the status here?

@Verena The answer to your question is: nine (9). Now, to make this sound less like "42", a table is attached with the following columns: (1) the timestamp of the 10th edit, (2) the Banner Campaign, (3) the timestamp of user registration. Please let me know if you need additional information.

@Stefan_Schneider_WMDE @Verena by the way

@Verena I will take care of that particular question

@Lea_WMDE if the userAgent field contains something like isAnon, then yes, otherwise, no. I don't think we have that (rather common in our RDBS tables) field in this Hive table.

@Lea_WMDE The following:

the asExtensionUpdate.csv file - that one that needs to be made public to support our Dashboard that runs from the CloudVPS service - is now sanitized as per your request:

@Lea_WMDE Please see T187908#4040814.

• Assuming (probably correctly) that the pageviews of Special:Search are not the same as executed searches, this task faces the same constraints as T187907#4040761:

@Lea_WMDE Please see T187907.

@dcausse @EBernhardson @Milimetric I will need your advise on this. Please let me know if I am missing something (I was relying on the Analytics/Data Lake/Traffic/Cirrus page to study the schema).