Page MenuHomePhabricator
Create Task
Maniphest T290138

navtiming prometheus scrape timeout and metric spamming
Closed, ResolvedPublic
Actions

Description

I noticed webperf2001's navtiming was alarming with "job unavailable" and indeed it is hitting the scrape timeout. The timeout reason is that navtiming is returning ~400k metrics back to prometheus, which is far too many

webperf2001:~$ curl -s localhost:9230/metrics | wc -l
383850

@dpifke could you take a look? thank you!

Event Timeline

fgiunchedi created this task.Sep 1 2021, 10:34 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 1 2021, 10:34 AM
fgiunchedi added a comment.Sep 1 2021, 10:41 AM

I scraped manually a few minutes apart and it looks like the recent battery level addition makes cardinality explode and metrics keep adding up. Below is the comparison of metrics added between the two scrapes:

webperf2001:~$ comm -13 <( awk '{print $1}' navtiming_metrics | sort ) <( awk '{print $1}' navtiming_metrics2 | sort )
webperf_cpubenchmark_seconds_bucket{battery_level="0",is_anon="True",is_oversample="False",le="+Inf",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_bucket{battery_level="0",is_anon="True",is_oversample="False",le="0.05",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_bucket{battery_level="0",is_anon="True",is_oversample="False",le="0.1",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_bucket{battery_level="0",is_anon="True",is_oversample="False",le="0.2",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_bucket{battery_level="0",is_anon="True",is_oversample="False",le="0.3",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_bucket{battery_level="0",is_anon="True",is_oversample="False",le="0.4",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_bucket{battery_level="0",is_anon="True",is_oversample="False",le="0.5",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_bucket{battery_level="0",is_anon="True",is_oversample="False",le="0.75",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_bucket{battery_level="0",is_anon="True",is_oversample="False",le="1.0",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_bucket{battery_level="70",is_anon="True",is_oversample="True",le="+Inf",origin_country="Japan",ua_family="Chrome",ua_version="93"}
webperf_cpubenchmark_seconds_bucket{battery_level="70",is_anon="True",is_oversample="True",le="0.05",origin_country="Japan",ua_family="Chrome",ua_version="93"}
webperf_cpubenchmark_seconds_bucket{battery_level="70",is_anon="True",is_oversample="True",le="0.1",origin_country="Japan",ua_family="Chrome",ua_version="93"}
webperf_cpubenchmark_seconds_bucket{battery_level="70",is_anon="True",is_oversample="True",le="0.2",origin_country="Japan",ua_family="Chrome",ua_version="93"}
webperf_cpubenchmark_seconds_bucket{battery_level="70",is_anon="True",is_oversample="True",le="0.3",origin_country="Japan",ua_family="Chrome",ua_version="93"}
webperf_cpubenchmark_seconds_bucket{battery_level="70",is_anon="True",is_oversample="True",le="0.4",origin_country="Japan",ua_family="Chrome",ua_version="93"}
webperf_cpubenchmark_seconds_bucket{battery_level="70",is_anon="True",is_oversample="True",le="0.5",origin_country="Japan",ua_family="Chrome",ua_version="93"}
webperf_cpubenchmark_seconds_bucket{battery_level="70",is_anon="True",is_oversample="True",le="0.75",origin_country="Japan",ua_family="Chrome",ua_version="93"}
webperf_cpubenchmark_seconds_bucket{battery_level="70",is_anon="True",is_oversample="True",le="1.0",origin_country="Japan",ua_family="Chrome",ua_version="93"}
webperf_cpubenchmark_seconds_count{battery_level="0",is_anon="True",is_oversample="False",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_count{battery_level="70",is_anon="True",is_oversample="True",origin_country="Japan",ua_family="Chrome",ua_version="93"}
webperf_cpubenchmark_seconds_sum{battery_level="0",is_anon="True",is_oversample="False",origin_country="Russia",ua_family="Opera",ua_version="48"}
webperf_cpubenchmark_seconds_sum{battery_level="70",is_anon="True",is_oversample="True",origin_country="Japan",ua_family="Chrome",ua_version="93"}
Krinkle added a project: NavigationTiming.Sep 1 2021, 10:52 AM
Stashbot added a comment.Sep 1 2021, 12:47 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-01T12:47:53Z] <godog> bounce webperf on webperf2001 - T290138

fgiunchedi added a subscriber: Krinkle.Sep 2 2021, 7:21 AM

Thank you for the fixes @dpifke @Krinkle, metric count is looking much better now. It is still creeping up over time but I think that's to be expected, see below for metrics added in 10 minutes

webperf2001:~$ comm -13 <( awk '{print $1}' navtiming_metrics_2021-09-02T07:00+00:00 | sort ) <( awk '{print $1}' navtiming_metrics_2021-09-02T07:10+00:00 | sort )
webperf_cpubenchmark_seconds_bucket{battery_level="low",is_oversample="False",le="+Inf",origin_country="India",ua_family="Chromium"}
webperf_cpubenchmark_seconds_bucket{battery_level="low",is_oversample="False",le="0.05",origin_country="India",ua_family="Chromium"}
webperf_cpubenchmark_seconds_bucket{battery_level="low",is_oversample="False",le="0.1",origin_country="India",ua_family="Chromium"}
webperf_cpubenchmark_seconds_bucket{battery_level="low",is_oversample="False",le="0.2",origin_country="India",ua_family="Chromium"}
webperf_cpubenchmark_seconds_bucket{battery_level="low",is_oversample="False",le="0.3",origin_country="India",ua_family="Chromium"}
webperf_cpubenchmark_seconds_bucket{battery_level="low",is_oversample="False",le="0.4",origin_country="India",ua_family="Chromium"}
webperf_cpubenchmark_seconds_bucket{battery_level="low",is_oversample="False",le="0.5",origin_country="India",ua_family="Chromium"}
webperf_cpubenchmark_seconds_bucket{battery_level="low",is_oversample="False",le="0.75",origin_country="India",ua_family="Chromium"}
webperf_cpubenchmark_seconds_bucket{battery_level="low",is_oversample="False",le="1.0",origin_country="India",ua_family="Chromium"}
webperf_cpubenchmark_seconds_count{battery_level="low",is_oversample="False",origin_country="India",ua_family="Chromium"}
webperf_cpubenchmark_seconds_sum{battery_level="low",is_oversample="False",origin_country="India",ua_family="Chromium"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="+Inf",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="+Inf",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.001",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.001",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.005",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.005",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.01",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.01",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.025",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.025",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.05",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.05",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.075",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.075",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.1",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.1",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.5",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="0.5",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="1.0",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="1.0",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="10.0",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="10.0",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="5.0",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_bucket{group="group2",le="5.0",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_count{group="group2",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_count{group="group2",ua_family="Opera",ua_version="12"}
webperf_firstinputdelay_seconds_sum{group="group2",ua_family="Android",ua_version="4"}
webperf_firstinputdelay_seconds_sum{group="group2",ua_family="Opera",ua_version="12"}
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="+Inf",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="0.005",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="0.01",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="0.025",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="0.05",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="0.075",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="0.1",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="0.25",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="0.5",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="0.75",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="1.0",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="10.0",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="2.5",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="5.0",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_bucket{cache_host="cp3064",cache_response_type="hit-local",le="7.5",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_count{cache_host="cp3064",cache_response_type="hit-local",transfer_size="0
webperf_navtiming_responsestart_by_cache_host_seconds_sum{cache_host="cp3064",cache_response_type="hit-local",transfer_size="0
webperf_painttiming_seconds_bucket{group="group1",le="+Inf",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="+Inf",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="0.1",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="0.1",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="0.5",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="0.5",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="0.75",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="0.75",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="1.0",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="1.0",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="1.25",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="1.25",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="1.5",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="1.5",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="1.75",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="1.75",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="10.0",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="10.0",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="2.0",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="2.0",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="2.5",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="2.5",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="3.0",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="3.0",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="4.0",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="4.0",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group1",le="5.0",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_bucket{group="group1",le="5.0",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_bucket{group="group2",le="+Inf",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="+Inf",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="+Inf",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="+Inf",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="+Inf",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="+Inf",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="0.1",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="0.1",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="0.1",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="0.1",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="0.1",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="0.1",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="0.5",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="0.5",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="0.5",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="0.5",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="0.5",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="0.5",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="0.75",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="0.75",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="0.75",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="0.75",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="0.75",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="0.75",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="1.0",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="1.0",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="1.0",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="1.0",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="1.0",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="1.0",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="1.25",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="1.25",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="1.25",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="1.25",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="1.25",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="1.25",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="1.5",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="1.5",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="1.5",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="1.5",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="1.5",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="1.5",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="1.75",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="1.75",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="1.75",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="1.75",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="1.75",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="1.75",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="10.0",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="10.0",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="10.0",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="10.0",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="10.0",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="10.0",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="2.0",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="2.0",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="2.0",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="2.0",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="2.0",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="2.0",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="2.5",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="2.5",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="2.5",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="2.5",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="2.5",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="2.5",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="3.0",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="3.0",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="3.0",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="3.0",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="3.0",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="3.0",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="4.0",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="4.0",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="4.0",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="4.0",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="4.0",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="4.0",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="5.0",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="5.0",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="5.0",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="group2",le="5.0",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_bucket{group="group2",le="5.0",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_bucket{group="group2",le="5.0",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_bucket{group="other",le="+Inf",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="+Inf",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="+Inf",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="+Inf",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="0.1",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="0.1",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="0.1",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="0.1",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="0.5",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="0.5",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="0.5",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="0.5",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="0.75",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="0.75",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="0.75",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="0.75",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="1.0",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="1.0",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="1.0",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="1.0",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="1.25",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="1.25",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="1.25",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="1.25",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="1.5",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="1.5",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="1.5",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="1.5",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="1.75",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="1.75",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="1.75",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="1.75",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="10.0",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="10.0",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="10.0",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="10.0",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="2.0",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="2.0",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="2.0",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="2.0",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="2.5",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="2.5",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="2.5",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="2.5",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="3.0",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="3.0",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="3.0",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="3.0",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="4.0",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="4.0",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="4.0",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="4.0",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="5.0",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_bucket{group="other",le="5.0",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_bucket{group="other",le="5.0",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_bucket{group="other",le="5.0",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_count{group="group1",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_count{group="group1",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_count{group="group2",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_count{group="group2",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_count{group="group2",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_count{group="group2",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_count{group="group2",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_count{group="group2",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_count{group="other",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_count{group="other",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_count{group="other",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_count{group="other",metric="first-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_sum{group="group1",metric="first-contentful-paint",ua_family="iOS_other",ua_version="93"}
webperf_painttiming_seconds_sum{group="group1",metric="first-paint",ua_family="Chrome",ua_version="54"}
webperf_painttiming_seconds_sum{group="group2",metric="first-contentful-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_sum{group="group2",metric="first-contentful-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_sum{group="group2",metric="first-contentful-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_sum{group="group2",metric="first-paint",ua_family="Android",ua_version="6"}
webperf_painttiming_seconds_sum{group="group2",metric="first-paint",ua_family="Chromium",ua_version="73"}
webperf_painttiming_seconds_sum{group="group2",metric="first-paint",ua_family="Opera",ua_version="12"}
webperf_painttiming_seconds_sum{group="other",metric="first-contentful-paint",ua_family="Chrome_Mobile_iOS",ua_version="77"}
webperf_painttiming_seconds_sum{group="other",metric="first-contentful-paint",ua_family="Edge",ua_version="75"}
webperf_painttiming_seconds_sum{group="other",metric="first-paint",ua_family="Chrome_Mobile",ua_version="47"}
webperf_painttiming_seconds_sum{group="other",metric="first-paint",ua_family="Edge",ua_version="75"}
fgiunchedi closed this task as Resolved.Sep 2 2021, 7:22 AM
fgiunchedi claimed this task.

Optimistically resolving

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL