We might want to split this task or rename to focus on the follow-up work. The field now has a search indicator, which was the only "must-have" requirement in the original list.

Moving back to the backlog. The guard condition is in a POC branch, ready for review.

Confirmed with Elisha that we want the full "SearchInputField" behavior including the clear action, if possible.

I've "boldly" edited the title to reflect how I understand the consensus above. We aren't trying to shift the source of truth, the point is to make events more useful for external system and service integrations.

I took the extremely scientific approach of opening the dialog several times and recording how long it took. If anything, there's a small improvement rather than a regression. One follow-up detail to look at is that the dialog is never destroyed, so there might be some memory leakage or penalty for a previous dialog instance having lots of parameters.

These are the messages to be customized on dewiki (as a local override, not as a general translation):

• https://de.wikipedia.org/wiki/MediaWiki:Visualeditor-dialog-transclusion-help-page-help
• https://de.wikipedia.org/wiki/MediaWiki:Visualeditor-dialog-transclusion-help-page-feedback

In T286992#7520249, @Johan wrote:

@awight For Tech News purposes, it would be good to know today – @Quiddity will deliver the newsletter in a few hours.

@ECohen_WMDE @Sarai-WMDE Please pull the data from here again, there were some significant changes. Most importantly, the total number of maps is now bracketed by an upper and lower estimate. But I found that the upper estimate "upper_limit_mapframes" (direct search for "mapframe" in all pages) was usually lower than the number I had been calculating as the lower bound (now called "checksum_total_transclusions", adding together all hits for each template, meaning that we were double-counting), so you should take the minimum of these two columns. I've decided that proportion_map_pages will just be upper_limit_mapframes / wiki_size, to simplify our ranking.

Script should be tweaked:

Script should be tweaked:

• Search for "mapframe
• Collect number of activeusers for each wiki
• Retry each wiki 5 times with a backoff. Failures are preventing script completion.

@JStrodt_WMDE @Lena_WMDE @Quiddity Wednesday's deployment is in jeopardy due to a production bug T296098, which caused the train to be rolled back last week. We can either wait to see whether the bug is fixed and the train released, or we can postpone until next week.

Starting to look better. I'm slightly supicious that enwikivoyage is only showing 0.09 maps per page, but scientifically clicking "random page" a couple-o-times makes this seem plausible at least.

We can search for mapframe in the rendered page output, at the same time as checking hastemplate:. This should reject the false positives.

Preliminary output: https://gitlab.com/wmde/search-mapframe-insource/-/tree/with-output/out

The mapframe tag is emitted by Lua code in some cases, notably in wikivoyage Module:Map. Matching 'mapframe' as well as <mapframe seemed to catch these. Sample data from en.wikivoyage.org:

usage_count	url
0	https://en.wikivoyage.org/wiki/Module:Map/sandbox
0	https://en.wikivoyage.org/wiki/Module:Map/sandbox2
0	https://en.wikivoyage.org/wiki/Module:Sandbox/Matroc/Inapoly
0	https://en.wikivoyage.org/wiki/Module:Sandbox/Matroc/Inapoly/doc
24973	https://en.wikivoyage.org/wiki/Module:Map
0	https://en.wikivoyage.org/wiki/Module:Mapdraw
0	https://en.wikivoyage.org/wiki/Module:Sandbox/Matroc/Mapdraw
0	https://en.wikivoyage.org/wiki/Module:Sandbox/Matroc
0	https://en.wikivoyage.org/wiki/Module:Mapdraw/doc
0	https://en.wikivoyage.org/wiki/Module:Sandbox/Wrh2
0	https://en.wikivoyage.org/wiki/Module:Sandbox/Matroc/Mapdraw2

I think output can be a single CSV with total counts from each wiki, then a directory full of JSON files detailing template names and counts, one file for each wiki. Templates should be listed as a title but also as a link for convenient clicking.

Basic queries are ready, the remaining work is to output in a reasonable way and parallelize if the run time is too long. Sample results:

Major gap in what we've done so far: dynamic maps cause a large number of tile loads, which would have been picked up in the above metrics as "static map load with no overlays"

Looking at this for half an hour...

@ECohen_WMDE my answer was a bit scattered yesterday, but we might not be able to postpone the template search if we want to understand the number of maps on pages—it seems like this is the only way to count the transcluded maps? I don't expect this to be a huge additional effort, if that's the concern.

In T295775#7506674, @ECohen_WMDE wrote:

I feel like this ticket maybe goes a bit beyond that?

Putting together some of the above to get a granular look,

with map_request as (
    select
        coalesce(
            url_extract_parameter(concat('http://bla.org/woo/', uri_query), 'domain'),
            nullif(regexp_replace(url_extract_host(referer), '\.m'), ''),
            'unknown'
        ) as normalized_host,
        regexp_like(url_extract_host(referer), '\.m\.') as is_mobile,
        url_extract_parameter(concat('http://bla.org/woo/', uri_query), 'title') as title,
        url_extract_parameter(concat('http://bla.org/woo/', uri_query), 'title') != '' as has_mapdata
    from webrequest
    where
        uri_host = 'maps.wikimedia.org'
        and regexp_like(uri_path, 'osm.*png$')
        and not regexp_like(http_status, '4..')
        and year = 2021
        and month = 11
        and day = 1
        and hour = 1
),

While we're here, let's see how many requests come from mobile web vs. desktop web:

select
  count(*) as sample_size,
  regexp_like(url_extract_host(referer), '\.m\.') as is_mobile
from webrequest
WHERE 
  uri_host = 'maps.wikimedia.org'
  and year = 2021
  and month = 11
  and day = 1
  and hour = 1
group BY 
  regexp_like(url_extract_host(referer), '\.m\.')
;

In T291120#7503969, @Ottomata wrote:

which becomes problematic if a revision is deleted

If it is deleted without a corresponding revision-delete state change event, that is problematic. Otherwise, fine, no?

While we're here, let's see how many requests come from mobile web vs. desktop web:

select
  count(*) as sample_size,
  regexp_like(url_extract_host(referer), '\.m\.') as is_mobile
from webrequest
WHERE 
  uri_host = 'maps.wikimedia.org'
  and year = 2021
  and month = 11
  and day = 1
  and hour = 1
group BY 
  regexp_like(url_extract_host(referer), '\.m\.')
;

Just following up because I may have been wrong to introduce the idea of "a" source of truth above. I did this to try to show where the confusion in this task is coming from, but it's important to note that "single source of truth" has a very strict definition, if we had a SSOT it would be the MediaWiki primary database and there are no other sources of truth. For example, if you want to make an update it must be to this database, not in an event stream, etc.

Digging into the huge number of requests with no domain parameter (in other words, not map image requests):

select
  count(*) as sample_size,
  regexp_like(uri_path, 'osm.*png$') as is_map
from webrequest
WHERE 
  uri_host = 'maps.wikimedia.org'
  and year = 2021
  and month = 11
  and day = 1
  and hour = 1
  and url_extract_parameter(concat('http://bla.org/woo/', uri_query), 'domain') is null
group BY 
  regexp_like(uri_path, 'osm.*png$')
;

Unique titles requested for each wiki, over a random 24-hour period:

select
  count(*) as sample_size,
  url_extract_parameter(concat('http://bla.org/woo/', uri_query), 'domain') as domain,
  count(distinct url_extract_parameter(concat('http://bla.org/woo/', uri_query), 'title')) as unique_titles
from webrequest
WHERE 
  uri_host = 'maps.wikimedia.org'
  and year = 2021
  and month = 11
  and day = 1
--  and hour = 1
group BY 
  url_extract_parameter(concat('http://bla.org/woo/', uri_query), 'domain')
order by
  unique_titles desc
;

Performance stats:

select
  count(*) as sample_size,
  cache_status,
  approx_percentile(time_firstbyte, 0.50) as time_firstbyte_p50,
  approx_percentile(time_firstbyte, 0.70) as time_firstbyte_p70,
  approx_percentile(time_firstbyte, 0.95) as time_firstbyte_p95  
from webrequest
WHERE 
  uri_host = 'maps.wikimedia.org'
  and year = 2021
  and month = 11
  and day = 1
  and hour = 1
group BY 
  cache_status;

-> truncated for clarity:

95751	int-front	0.0001	0.0001	0.0002
1060387	hit-front	0.0002	0.0002	0.0003
542485	hit-local	0.0012	0.0039	1.9999
49256	pass		0.1681	0.2514	3.0624
551351	miss		0.3021	0.4873	3.1874

First pass:

select
  count(*) as sample_size,
  histogram(cache_status),
  histogram(http_status),
  histogram(access_method),
  histogram(agent_type)
from webrequest
WHERE 
  uri_host = 'maps.wikimedia.org'
  and year = 2021
  and month = 11
  and day = 1
  and hour = 1;

Random note from a discussion about instrumentation: we're interested in how often users click a static map image to launch the dynamic, full-screen mode.

Remaining questions are moved to T295675.

This is just a walk-by comment so feel free to ignore, but didn't we have a nearly identical issue last year, and the solution was to put all these little files in a tmpfs which could simply be unmounted when the container is destroyed? If this sounds like something to pursue, I could track down the details...

In T295363#7500443, @ECohen_WMDE wrote:

Sample of distribution across wikis, from Nov 1, from 1-2am UTC (N=131997): P17738

@awight What is this showing? Is this already enough to point to wikis with frequent Kartographer views? For example ja.wiki seems to really stand out as a high number, though de wikivoyage is really low. So maybe I'm misinterpreting this or shouldn't be looking at this number at all? Sorry if these questions are premature.

In T294839#7500493, @thiemowmde wrote:

It turns out the position of the close button is always on the left in OOUI. I had to remove the CSS hack that moved it to the right because that caused other style problems.