Hi @AndyRussG and @Pcoombe; Peter, we've talked a bit about this, but I'm finally getting around to creating a task. Any help understanding this issue would be appreciated!
On December 7, 2020, PCoombe put up an a/b test in the large banner buckets (A&B) of our FY2021 en6C desktop CentralNotice campaign.
We've since realized that one of the banners in that test, B2021_120716_en6C_dsk_p1_lg_dsn_squareCorners, registered about a million more impressions than the other banner, B2021_120716_en6C_dsk_p1_lg_dsn_cnt.
Here's a Turnilo chart illustrating the issue; blue line = _squareCorners.
I've looked at the logs for that campaign and haven't caught any human error (e.g., someone leaving _squareCorners in a bucket when setting up the next test).
Finally and interestingly, donations did not continue to accumulate for _squareCorners, just impressions, making it look like an enormous loss from a testing perspective. Note the discrepancy in trials (impressions).
The obvious anxiety is that impressions could be unevenly distributed in more/most tests. I've done some spot-checking of other tests from that time period, and recently, but haven't caught other alarming impression imbalances.