I just tested this and some queries still get log entries included. Here is a sample:

May 6, 2021 5:04:25 PM WARNING: org.apache.parquet.hadoop.ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl

May 6, 2021 5:00:25 PM INFO: org.apache.parquet.CorruptStatistics: Ignoring statistics because this file was created prior to 1.8.0, see PARQUET-251

The question is still in Legal's queue. I've just sent them a message asking for a status update.

Still blocked on Legal's advice.

Legal has given final approval!

We think we've found and fixed the experiment bug! It seems like a single wiki edit was all that was needed.

Since this is work for the engineers, I think it shouldn't be in the analytics project.

Since this is work for the engineers, I think it shouldn't be in the analytics project.

I will wait until the implementation is close to being merged before doing this, as there's a small chance that the engineers could encounter issues that require changing the schema.

All merged now!

To check the possibility that KaiOS app pageviews were not being recorded properly in the pageviews_hourly stream, @SBisson used the app to viewed the Ugandan recommendations from Canada over the weekend.

I've completed the check; you can check out my notebook for the full details.

I talked to @AMuigai and we decided to spend the time on a deeper data bug search instead (T280178).

I've emailed Legal with the question.

Pulling this data is extremely easy. The main issue is the uncertainty about how we can share it internally—see the extensive discussion in T271202. I will check with Legal this week.

For what it's worth, I would recommend not reporting the opt-out rate for registered users at all. Since most of those users are long vanished, it's not really a fair metric. You've already dealt with that by providing a better metric (opt-out rate for active users), but in this case, it's so much better that I think it should just stand on its own.

This is now in Legal's queue. I requested that they do it by 30 April; they said they'll do their best but can't commit. I've offered to do anything that would help expedite the process, such as briefing one of the their team members face-to-face.

In T278597#7002929, @DAbad wrote:

So we have two time_stamps: Server_dt and Client_dt. We should validate using the Server_dt to validate if the client_dt is off and then determine how to handle it...

After speaking with Jason and Michael today, we likely don't need milliseconds and if we don't have fractional sections we will interpret as a round number.

The schema draft and diagram have been fully updated with the changes I made in response to the engineering review. There should be no more changes unless I get high-priority feedback or something changes in the product itself.

In the end, we didn't need this task. We went with an announce-the-switch-then-test model rather than the other way around.

I discussed the schema some time ago with @ngkountas and @santhosh. They gave me some helpful suggestions and said it looked good otherwise. I've just finished adjusting the draft to incorporate their suggestions.

In T280178#7000128, @SBisson wrote:

I understand it's early but given the early numbers you shared, ~3.5k devices per group per day, we could start seeing a difference between the groups.

I've put in the request to Legal.

@AMuigai @SBisson what do you hope to learn from this query? Is it just to confirm that the experiment isn't broken?

In T277348#6959373, @SNowick_WMF wrote:

will try the spark-sql query next time I need to run these queries.

The job has now run correctly for two days in a row, so I think we can now rely on it for the experiment.

In T216294#6957007, @elukey wrote:

Neil I think that this can be closed, what do you think?

In T275212#6958479, @razzi wrote:

@nshahquinn-wmf any luck with that setting?

It looks like the job ran just fine today! I'll check again tomorrow before closing this.