I made some tiny changes to the task description and tagged WMDE-Fundraising-Tech who among others might already be tracking the risks posed by this "page experience" rollout.

PMs and managers have meetings in place to monitor SEO. This task isn't needed.

In T280476#7028456, @DStrine wrote:

PMs and managers have meetings in place to monitor SEO. This task isn't needed.

I would like to suggest that even with PMs and managers discussing options, this topic still needs open discussion in a space like this, please. Thanks so much.

Just another quick thought... if there are concerns that the banner bump is somehow needed to get users' attention, and that's causing people to worry about an engineering solution... there would be a way to achieve a similar "bump" effect without impacting SEO.

As pointed out by @Gilles in T279034#7012796, after users start interacting with the page, further changes are not taken into account by the LCP metric. So, any time after users start scrolling, overlay animations of any size are fine. For example, immediately after scrolling starts, or potentially on a delay if desired, banners could get bigger, overlaying some content, to get the user's attention. And many other sorts of attention-getting animation, besides increasing banner size, would also be fine at that time, from an SEO perspective.

Hope this is helpful!! :) Thanks so much!

Thank you, @AndyRussG this information is very helpful. For the desktop banners of WMDE we have a static 7-second delay before bumping the page content down. I can propose adding an interaction check before displaying point to this discussion. The decision will be made by other people in WMDE.
If CentralNotice would provide such an interaction check that would be nice, if it doesn't I think we can come up with something.

In T280476#7042031, @gabriel-wmde wrote:

Thank you, @AndyRussG this information is very helpful. For the desktop banners of WMDE we have a static 7-second delay before bumping the page content down. I can propose adding an interaction check before displaying point to this discussion. The decision will be made by other people in WMDE.

Hi @gabriel-wmde! Right... Ah, many apologies, I think I wasn't clear... What I was referring to with "a way to achieve a similar bump effect", above, unfortunately doesn't include anything that bumps the content down, but rather other kinds of animation, like overlays. That's because the two metrics involved work a bit differently.

So, it's only in the case of the LCP metric that changes are not taken into account after users start interacting with the page: "The browser will stop reporting new entries as soon as the user interacts with the page (via a tap, scroll, or keypress) [...]" This is the metric that can cause problems for overlays. So, once interactions start, overlays are OK, as regards SEO metrics.

On the other hand, the CLS metric, which measures layout shift, will continue to report entries. Certain kinds of user interactions, like clicks or taps, will prevent a layout shift from being reported, but only if the shift occurs within 500 ms of the interaction. "Layout shifts that occur within 500 milliseconds of user input will have the hadRecentInput flag set, so they can be excluded from calculations." So, if a click or a tap causes a layout shift, it's OK, but otherwise, it's not.

You can check the scores generated by banners (in lab conditions) with PageSpeed Insights. For example, a typical WMF mobile Fundraising banner causes a CLS score of 0.916, which is only... 9 times higher than the recommended maximum score of 0.1:

Since the WMDE banners push content down on a delay, I think, unfortunately, they'll have to change, even if we get an engineering solution to the current JS/background-loading banner bump. I couldn't get an actual score result with that tool for a WMDE banner, probably because the delay was too long. But you could make a test banner with a shorter delay to verify what score it would get. In the end, I think the most similar option for this type of banner would indeed be a small overlay or other animation of some sort after the user starts scrolling. In that case, an engineering solution to the CentralNotice banner bump would still be helpful (assuming you wish to show at least some banner content as soon as the page loads, in case the user navigates away without scrolling or interacting at all).

Another option (one that doesn't depend on a fix for the banner bump) could be to show no banner content initially, and then always show a small overlay banner on a delay, using JavaScript to measure the viewport and ensure the banner is small enough that it doesn't impact the LCP metric.

See also T280477: Study options and designs for banner formats to prevent search ranking decline for more notes on possible banner formats and Google's new metrics.

I hope this is useful! :)

If CentralNotice would provide such an interaction check that would be nice, if it doesn't I think we can come up with something.

Yeah, great idea, thanks so much!!!

Great to hear that discussions are happening—it would be nice if anyone could link to meeting minutes or other outcomes, since the mid-June deadline is approaching quickly and it still looks like a technical solution is required. It's worth noting that a quarterly planning cycle will be too slow in this case, no resources are dedicated at the moment and two weeks is too little time for development.

Meanwhile, it seems like next steps could be the investigations in T280477: Study options and designs for banner formats to prevent search ranking decline and subtasks, to determine:

• How CentralNotice banners affect LCP and CLS metrics. This can already be measured directly.
• How the metrics decline will affect our search ranking. Estimate how this will impact readership.

Maybe the results of that investigation will inform the various stakeholders and will give us a more accurate sense of urgency.

I missed your comment when writing, sorry! Of course you're already making the exact measurements of how we're impacting metrics... That's a scary initial finding that our CLS is 9x the recommended maximum. If you find the time, could you also show us how big of a jump this is over the baseline, a wiki page with no banners displayed?

I also started wondering whether our variable treatment of different visitors will be detected as "cloaking"? In other words, if the spider doesn't support cookies so we suppress banners automatically, while delivering banners to nearly 100% of Chrome users... we are de facto cloaking.

In T280476#7044910, @awight wrote:

I also started wondering whether our variable treatment of different visitors will be detected as "cloaking"? In other words, if the spider doesn't support cookies so we suppress banners automatically, while delivering banners to nearly 100% of Chrome users... we are de facto cloaking.

The Core Web Vitals scores are not measured by a spider/googlebot, but are based on data collected from Chrome users visiting the pages https://support.google.com/webmasters/thread/104436075

Regarding cloaking,

In T280476#7045799, @Pcoombe wrote:

The Core Web Vitals scores are not measured by a spider/googlebot, but are based on data collected from Chrome users visiting the pages https://support.google.com/webmasters/thread/104436075

I think we're on the same page—the new metrics are a separate issue, but I confused things by mentioning Chrome users. The reason I bring it up in this context is that, as we consider potential alternatives to improve MediaWiki it's important for SEO that we don't accidentally build a system which serves banners to normal browsers, but no banners to spiders because they're rejecting cookies for example. I think we might already have a bit of this going on, and I wonder if we have a secret "exception" hardcoded by Google, etc. The nondetermistic banner choice already brings up interesting questions about how cloaking is detected.

Just want to share that we have dashboards for the Google Web Vitals:
https://grafana.wikimedia.org/d/t_bhsNGMk/chrome-user-experience-report

In T280476#7530776, @Peter wrote:

Just want to share that we have dashboards for the Google Web Vitals:
https://grafana.wikimedia.org/d/t_bhsNGMk/chrome-user-experience-report

Fantastic, thanks! Just to call out one detail from these graphs, the Cumulative Layout Shift has been getting steadily worse, the "poor" rated pageviews have tripled from roughly 5% to 15% in the last two weeks. Could this be related to a fundraising campaign? Do we understand why the downward trend is moving so linearly, is there some type of moving window average disguising a step change?

Thanks so much @Peter and and @awight!

In T280476#7530776, @Peter wrote:

Just want to share that we have dashboards for the Google Web Vitals:

Wow that's great... Any quick pointers to code/doc on how that's generated? Specifically, wondering if there are ways to break this down further by country and site, and include at least some amount of past data in the breakdown?

In T280476#7533117, @awight wrote:

Fantastic, thanks! Just to call out one detail from these graphs, the Cumulative Layout Shift has been getting steadily worse, the "poor" rated pageviews have tripled from roughly 5% to 15% in the last two weeks. Could this be related to a fundraising campaign?

Just to add: ...or a combination of fundraising and non-fundraising ones (since the latter have also shown more than a few banners recently)?

Do we understand why the downward trend is moving so linearly, is there some type of moving window average disguising a step change?

Not sure, but Google Search console does show rolling 28-day or 30-day window averages.

Also, this topic has come to the fore again just now... Please see T296875.

Thanks again!!

Fantastic, thanks! Just to call out one detail from these graphs, the Cumulative Layout Shift has been getting steadily worse, the "poor" rated pageviews have tripled from roughly 5% to 15% in the last two weeks. Could this be related to a fundraising campaign? Do we understand why the downward trend is moving so linearly, is there some type of moving window average disguising a step change?

Yes it is the campaign. These metrics are collected form Google, the data is rolling averages over 28 days, so we use it to see trends, then we have tests that runs everyday so we can see exact when tests happens (and there we can see the impact of the campaigns).

Wow that's great... Any quick pointers to code/doc on how that's generated?

This is what we have, please let me know if anything is missing ssh I can add it!

Just to add: ...or a combination of fundraising and non-fundraising ones (since the latter have also shown more than a few banners recently)?

Yes all type of banners as it is now will decrease our score for cumulative layout shift. However it isn't so easy to "just" fix the layout shift as Gilles pointed out before: That will also impact the largest contentful paint (when the largest part of the screen is painted). Checkout what happens to LCP with the quicksurvey: https://phabricator.wikimedia.org/T294568#7491710

One last thing @AndyRussG that I forget to mention on out chat: You asked about if we could get this data per country instead of per domain. Not from Google (but we are working on collecting the data ourselves). Google "judge" the score per domain. When Google Web Vitals was announced Gilles gave the feedback back to Google that this would hurt sites with a global audience since we will have a hard time to get good metrics for global users. It's much easier to focus on one country to get faster time to first byte etc.

In T280476#7548083, @Peter wrote:

Wow that's great... Any quick pointers to code/doc on how that's generated?

This is what we have, please let me know if anything is missing ssh I can add it!

Thanks!! I do have another question: is there any way to access the back-end data for additional analysis? (I'd like to bring it into in a Jupyter notebook and run some automated change detection, to check/demonstrate alignment of change in banners displayed per pageview, change in CWV scores, and change in search engine referrals. Also, apologies if the answer is available somewhere.)

However it isn't so easy to "just" fix the layout shift

Heheh no it isn't easy! So far, I think the most promising option to completely remove the banner bump is this one: T283521.

(Just updated the task description to mention that CWV has so far only been included in the signal for search result rankings on mobile devices, with rollout to desktop planned for February and March, 2022. Details here. Also, just to note, I think a decline in search engine traffic on mobile devices would still hurt pageviews on both mobile and desktop, since links obtained from searches on mobile can be shared and then re-clicked on either platform.)

is there any way to access the back-end data for additional analysis

We get the data from https://web.dev/chrome-ux-report-api/ once a day and store it in a Graphite instance that the performance team run. We have that instance because our test servers that runs outside on our network doesn't have access to the rest of our network. Maybe it's easiest to get the data from https://web.dev/chrome-ux-report-bigquery/ for you? Need to take it easy though since you pay by number of rows that you access. I haven't used bug query so much myself.

We collect layout shift today for our own users but not the cumulative one and Google has changed the algorithm (to only measure the first 5 seconds), so that isn't perfect. When we have fixed that we collect the data the same way as Google it will be interesting to compare (and you can use that data in your investigations).

removing inactive/unrelated project tags