Investigation: Which data source to use for GeoPoints
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To

Authored By

	lilients_WMDE
	Feb 22 2022, 2:44 PM

Referenced Files

None

Subscribers

Description

Following up T300042 and based on the POC developed there, we should add a caching mechanism for the query results.

We could use the same approach like Geoshapes or something else, if there is something better.

Open questions

How does SPARQL endpoint caching work? If the same query is made multiple times, is it executed again or not?
TBD: Ongoing request to the search team

~~[] How does geosearch caching work?~~ (invalid for our use case here)

Is there caching for the geoshapes endpoint?
- Looks like there is front caching for shape result ~24 hours.
- Shape data locally is retrieved directly form a local DB.
If a geoshapes query returns a wikidata item with no geoshape (maybe among a list of other items which do have shapes), is there a visible error? How is this handled?
- Looks like an empty set is returned, https://maps.wikimedia.org/geoshape?getgeojson=1&ids=Q102, https://maps.wikimedia.org/geoshape?getgeojson=1&ids=Q1028,Q102
Research existing traffic: What proportion of SPARQL requests are coming from the geoshapes service?
- What proportion of these queries return successfully?
seems super low, a few request per million SPARQL requests
- How long do geoshapes queries take when SPARQL is being used (in contrast to QID input)?
might be irrelevant regarding amount of requests
- Could/Should we improve the current approach to reduce traffic?
might be irrelevant regarding amount of requests
How many existing maps are there already using sparql queries for geoshapes? (Maybe interesting to find good pilot wikis for geopoints.)
also seems so low that it's probably irrelevant

Notes: https://docs.google.com/document/d/1HM5uo8onOVUws5zAT6taswRW_R0pSF3cf3IosmUcVkM

Relevant links

Wikidata:SPARQL query service/query optimization
ST_AsGeoJSON, part of the postgis suite of pgsql functions
Caching overview - Wikitech

Details

Other Assignee: Andrew-WMDE

Related Objects
Search...

Status	Assigned	Task
Open	None	T67626 [Epic] Support for queries on-wiki (automated list generation)
Resolved	ECohen_WMDE	T188291 Display point features (coordinates) from the results of a SPARQL query on a map in Wikipedia
Resolved	None	T302297 Display geopoint SPARQL queries in a map in Wikipedia (GeoPoints)
Resolved	Andrew-WMDE	T302291 Investigation: Which data source to use for GeoPoints
Resolved	None	T308218 Discuss WDQS geopoint use case with WMF Search team

Event Timeline

lilients_WMDE created this task.Feb 22 2022, 2:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 22 2022, 2:44 PM

lilients_WMDE mentioned this in T300042: Investigation: display SPARQL queries in a map in Wikipedia.Feb 22 2022, 2:45 PM

lilients_WMDE mentioned this in T302297: Display geopoint SPARQL queries in a map in Wikipedia (GeoPoints).Feb 22 2022, 2:57 PM

lilients_WMDE added a parent task: T302297: Display geopoint SPARQL queries in a map in Wikipedia (GeoPoints).

lilients_WMDE moved this task from Backlog to Ready for story time on the WMDE-GeoInfo-FocusArea board.Mar 2 2022, 9:48 AM

lilients_WMDE added a project: WMDE-TechWish-Sprint-2022-03-02.Mar 2 2022, 3:36 PM

lilients_WMDE renamed this task from Use caching mechanism for GeoPoints to Investigation: Use caching mechanism for GeoPoints.Mar 3 2022, 9:23 AM

lilients_WMDE set the point value for this task to 8.Mar 9 2022, 2:35 PM

lilients_WMDE claimed this task.Mar 10 2022, 2:37 PM

lilients_WMDE moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2022-03-02 board.

lilients_WMDE updated Other Assignee, added: awight.Mar 10 2022, 2:45 PM

lilients_WMDE updated the task description. (Show Details)

lilients_WMDE removed lilients_WMDE as the assignee of this task.Mar 10 2022, 3:52 PM

lilients_WMDE moved this task from Doing to Sprint Backlog on the WMDE-TechWish-Sprint-2022-03-02 board.

awight moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2022-03-02 board.Mar 11 2022, 2:57 PM

(I'm looking at the webrequest logs now, starting with this example query from @Tarrow.)

awight removed a subscriber: Tarrow.Mar 11 2022, 3:04 PM

Empty-handed for the moment. I went into superset and this presto query comes back with no data. I've experimented with the regexp query and it cannot find "kartotherian" anywhere (".*" is just out of exasperation).

SELECT time_firstbyte, cache_status, http_status, response_size, http_method, uri_path, uri_query, content_type, dt, hour, user_agent  
from webrequest
where
  regexp_like(user_agent, '.*Maps.*') and
  uri_host='query.wikidata.org'
  and webrequest_source='text'
  and year=2022 and month=3 and day=1

ECohen_WMDE moved this task from Ready for story time to In sprint on the WMDE-GeoInfo-FocusArea board.Mar 14 2022, 5:05 PM

WMDE-Fisch added a project: WMDE-TechWish-Sprint-2022-03-16.Mar 16 2022, 9:35 AM

WMDE-Fisch changed the point value for this task from 8 to 5.

Also found no matches for uri_host='wdqs.discovery.wmnet'.

lilients_WMDE moved this task from Sprint Backlog to Done on the WMDE-TechWish-Sprint-2022-03-16 board.Mar 18 2022, 10:24 AM

lilients_WMDE moved this task from Done to Sprint Backlog on the WMDE-TechWish-Sprint-2022-03-16 board.

lilients_WMDE claimed this task.Mar 18 2022, 10:48 AM

lilients_WMDE moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2022-03-16 board.

lilients_WMDE updated Other Assignee, added: Andrew-WMDE; removed: awight.

lilients_WMDE removed lilients_WMDE as the assignee of this task.Mar 18 2022, 4:06 PM

lilients_WMDE moved this task from Doing to Sprint Backlog on the WMDE-TechWish-Sprint-2022-03-16 board.

So, here are the counts for WDQS requests coming from Karotherian vs. other sources. I would say we don't have to be concerned about the load we cause with any new geopoints features. (Caching is still an open question, of course.)

select
  count(*) as total_requests,
  sum(case when user_agent rlike 'kartotherian' then 1 else 0 end) as from_kartotherian
from wmf.webrequest
where
  uri_host='query.wikidata.org'
  and webrequest_source='text'
  and year=2022 and month=3
group by day;

total_requests  from_kartotherian
10020708        0
10935084        4
16188054        42
13376930        40
11559587        0
11205092        0
12956225        0
13969256        0
13340032        0
19021548        0
15284965        0
12968692        0
12982497        0
13393441        0
10390427        1
8516103 0
7841070 0
8576772 0
7612512 0
8044851 0
4621804 0
21 rows selected (1840.988 seconds)

awight added a project: WMDE-TechWish-Sprint-2022-03-30.Mar 30 2022, 8:10 AM

awight updated the task description. (Show Details)Apr 12 2022, 4:35 PM

thiemowmde added a project: WMDE-TechWish-Sprint-2022-04-13.Apr 13 2022, 12:09 PM

WMDE-Fisch added a project: WMDE-TechWish-Sprint-2022-04-27.Apr 27 2022, 9:27 AM

WMDE-Fisch updated the task description. (Show Details)Apr 27 2022, 12:08 PM

WMDE-Fisch updated the task description. (Show Details)Apr 27 2022, 12:19 PM

WMDE-Fisch updated the task description. (Show Details)

WMDE-Fisch moved this task from Sprint Backlog to Watching / Epic / Stalled on the WMDE-TechWish-Sprint-2022-04-27 board.

Question: Is this ticket only about the server-side rendering of static map images? We would need this as well for the (enlarged) dynamic map. That should show the same markers, shouldn't it?

thiemowmde added a project: WMDE-TechWish-Sprint-2022-05-11.May 11 2022, 12:14 PM

thiemowmde moved this task from Sprint Backlog to Watching / Epic / Stalled on the WMDE-TechWish-Sprint-2022-05-11 board.

thiemowmde added a project: WMDE-TechWish-Sprint-2022-05-25.May 25 2022, 8:16 AM

thiemowmde moved this task from Sprint Backlog to Watching / Epic / Stalled on the WMDE-TechWish-Sprint-2022-05-25 board.

awight moved this task from In sprint to Ready for pickup on the WMDE-GeoInfo-FocusArea board.Jun 22 2022, 8:09 AM

thiemowmde renamed this task from Investigation: Use caching mechanism for GeoPoints to Investigation: Which data source to use for GeoPoints.Jul 20 2022, 8:41 AM

thiemowmde added a project: WMDE-TechWish-Sprint-2022-07-20.

thiemowmde moved this task from Ready for pickup to In sprint on the WMDE-GeoInfo-FocusArea board.

thiemowmde updated the task description. (Show Details)Jul 20 2022, 8:45 AM

The next step is to refine this task according to what we learned in our meeting with the WMF Search Platform team. Create tasks to adapt our code to the new data source.

Andrew-WMDE moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2022-07-20 board.Jul 21 2022, 2:44 PM

lilients_WMDE mentioned this in T307707: Markers added by QID to have auto-generated title and description.Jul 28 2022, 3:59 PM

lilients_WMDE closed subtask T308218: Discuss WDQS geopoint use case with WMF Search team as Resolved.Aug 2 2022, 12:15 PM

WMDE-Fisch assigned this task to Andrew-WMDE.Aug 3 2022, 8:08 AM

WMDE-Fisch added a project: WMDE-TechWish-Sprint-2022-08-03.

WMDE-Fisch moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2022-08-03 board.Aug 3 2022, 8:12 AM

For future reference, @Andrew-WMDE wrote:

[…] chat about the Wikibase REST API […]. It along with other APIs are supposed to replace commonly used queries in order to reduce the load on the WDQS. However, it's still in an early experimental phase without any concrete timeline on when it will be production ready. This means we also don't yet know what SLO to expect when it is deployed. Additionally, for our GeoPoints use case, we would have to implement the appropriate routes ourselves.
Looking back at the notes from our meeting with [WMF]. I think we should be able to continue using WDQS, as long as we're happy with an SLO of 95% and we don't cause any cascading failures.

Andrew-WMDE moved this task from Doing to Done on the WMDE-TechWish-Sprint-2022-07-20 board.Aug 10 2022, 8:12 AM

lilients_WMDE moved this task from Doing to Done on the WMDE-TechWish-Sprint-2022-08-03 board.Aug 11 2022, 12:24 PM

We're going for WDQS for now.

Investigation: Which data source to use for GeoPointsClosed, ResolvedPublic5 Estimated Story PointsActions