@kai.nissen I have had a look at the code with Gabriel and it looks fine to us at first glance. We went back to the old Matomo logs from a year ago, but since the data has been purged by now, it's a bit hard to follow what the actual problem is here. Since the relevant data is only tracked when the mbt flag is set in the URL, the last relevant data points are from September 2018 (after that we removed it from all banner links). If you still want us to look into this, could you please elaborate and let's talk about how to proceed here (as no recent sample data exists).

@Tim_WMDE if that helps: there was another test on mobile (test 7) in 2018 which should include that feature. that test was in 12/2018.

on the other hand @kai.nissen, since we now track banner submits via EventLogging that bug perhaps is not so much a problem anymore.

@Tobias_Schumann_WMDE Unfortunately not, we would need some recent logs to really get started with it as most data from December has been purged by now.

With test #07 giving us a new opportunity for gathering data, please have a close eye on the tracking numbers.
At the moment I could explain how I expect problematic numbers could look like, but I'm not sure what the actual differences/problems are:

• Which banner has lower numbers, CTRL or VAR?
• Does the lower number correlate with amount of PPL/MCP donations?

Please elaborate, @tmletzko, @JanJaquemot, @Tobias_Schumann_WMDE-ext when you have more data.

@gabriel-wmde You can see the effect of the bug if you compare the number of donations and visitors of in a test group.

In the VAR banner we have a direct redirect to the payment provider or confirmation page when you choose not to state an address with ppl, cc and ueb. What we want is that when that happens this is still counted as a visit (even if there is no recognizable visit on the donation page for the user). If the bug exists there is no visit counted in these cases. Hence, if you look into the data visitors should be missing but the donation, of course, is counted in the analysis section. This leads to a situation where we should have more donations than visitors in the test group of the var banner.

At the moment it's not possible to check in the data if the problem occurs because of: https://phabricator.wikimedia.org/T236732
I will upade this ticket when we have more clean data in the coming days.

@gabriel-wmde

So now, after some days it is possible to see the erroneous pattern: If you look at the data from oct 29th-30th you can see that in var there are more donations than visits, hence there is a portion of visits not being tracked correctly. In ctrl that problem doesn't occur which is not surprising because there is no direct redirect to payment providers.

Does that help you to identify the cause?

When looking at the analysis I don't see much discrepancy, is there another place you're looking at?

most recent data with VAR having more donations than visits.

@gabriel-wmde In your screenshot you can see that there are more donations than visitors. Even if the difference is not big that is not supposed to happen. Also, in ctrl you can see that there are quite a few more visits than donations - as it is supposed to be. Moreover like @tmletzko is showing the same applies to the current test on mobile (which tests the same thing)