In T280621 I started to make sure we tested the same URLs in all our synthetic tools and compared them to the data we get from the Chrome User Experience report.

One thing that looked strange was that on mobile first contentful paint was much slower on our start page than the rest of the pages we test. Looking at the 75 percentile from Chrome User Experience report it looks like this:

The looking at the same URLs when we run on emulated mobile together with WebPageReplay it looks like this:

The main page is much faster.

Running with WebPageTest:

And sitespeed.io without WebPageReplay:

However running on a Moto G5 doing five runs gives us the same story as the Chrome User Experience report:
Median FCP for Barack Obama: 1.13s
Median FCP Main_page: 1.49s

I think this gives us two things:

1. The importance to test on real devices
2. We made have an opportunity to make the Main page faster