In T199340#4670813, @Milimetric wrote:

Ok, the dashboards are offline, data generation is stopped, there are some placeholder pages in place, and I've looked around on the wikis and updated any references I see to edit-analysis. Help with that is appreciated.

@Catrope's current plan:

• Migrate VE and WikiEditor to the mw.eventLog.Schema class, which was created after the VE instrumentation code was written
• Make the sampling rate configurable (like in this WIP change, but as an actual number rather than a weird hacky hex thing)
• Add oversampling support to mw.eventLog.Schema, which adds isOversampled=true to the event data for oversampled events (i.e. events that bypassed sampling and would not have been logged otherwise)
• Add code to WikimediaEvents that triggers oversampling in the cases where we need it

These metrics now use the Analytics Query Service—this didn't reduce the delay as we hoped, but we have at least removed our dependency on Wikistats 1.

I've started using a Google spreadsheet to format the metrics properly—that's a big step forward.

Done, finally!

In T206279#4644644, @JAllemandou wrote:

Some more info on the issue as I understand it: Given the data size to work and the fact that hive.auto.convert.join is set to true by default, Hive converts the join to happen into a Map side join, and even more, into a LOCAL map side join. This means that the join happens locally on the client instead of happening onto a cluster container. When using the hive client, local memory can be configured using HADOOP_HEAPSIZE en variable. However when using hue or beeline, local actually means in HiveServer2.
The solution provided above (SET hive.auto.convert.join=false;) will work in any case, but it could interesting to try to raise HiveServer2 available memory and see if it solves the issue.

In T206279#4649749, @mforns wrote:

@Neil_P._Quinn_WMF, can you please use hive client instead of beeline.
We know it says it's deprecated in favor of beeline, but we still believe that hive is better for now.
At least better than the beeline version that we have installed.
We'll follow up with that.

For the record, I actually think this is unblocked now!

@Ottomata, from my perspective, we can close this since @JAllemandou gave me a workaround and I documented it on Wikitech.

I'm now changing the parent tasks back—the lack of this data means we can't produce any metrics about the load performance of mobile editors, which will be crucial for our annual plan goal.

Changing the parent tasks here since this doesn't block our immediate metrics work, since event timestamps are precise enough to calculate overall session duration.

Changing the parent tasks here since this doesn't block our immediate metrics work, since event timestamps are precise enough to calculate overall session timestamps

Looks good to me!

This data has been accumulating successfully for a month now (for example, see P7612). Thanks, everyone!

Reopening so I don't forget to check the data to verify the fix.