Page MenuHomePhabricator
Create Task
Maniphest T114017

Map current use of Wikimedia web APIs
Closed, ResolvedPublic
Actions

Description

  • List of known users of Wikimedia web APIs specifying main uses.
  • Metrics of quantitative use of these APIs

There are two ways to discover this

  • WMF staff and community update page(s) about users of the API based on what they know, e.g. "I met with LoveMyMusic and they show artist details" or "Here's a screenshot of Wikipedia lookup on Kindle"
  • we analyze the user agents and possibly hosts from the improved action API logs and other API logs

Note (also on the public page) there are also people building services from Wikipedia dumps, the RecentChanges stream, and (possibly?) the Wikimedia update feed services. They are not clients of our web APIs.

Issue how much of this is private, on an office.wikimedia.org page rather than a public page [[ API:Clients of Wikimedia APIs]]? It seems mentioning use of Wikimedia by a public service or app is uncontroversial, but if a client uses Wikimedia APIs for a private internal use, is it OK to list it? And should the quantitative metrics be public?

See also:

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT114017 Map current use of Wikimedia web APIs
 Resolvedbd808T102079 Metrics about the use of the Wikimedia web APIs
 ResolvedTgrT106457 Sample StatsD requests
 OpenNoneT108414 Load API request count and latency data from Hadoop to a dashboard
 ResolvedArielGlennT108417 stat1002 access for tgr
 Resolvedbd808T108618 Publish detailed Action API request information to Hadoop
 ResolvedNoneT114733 Determine proper encoding for structured log data sent to Kafka by MediaWiki
 ResolvedDzahnT115548 Requesting access to analytics-privatedata-users for Bryan Davis
 Resolvedbd808T118592 Create user defined function to classify network origin of an IP address
 ResolvedJoeT125084 MediaWiki monolog doesn't handle Kafka failures gracefully
 ResolvedJoeT119637 Update HHVM package to recent release
 ResolvedJoeT129467 HHVM 3.12 has a race-condition when starting up
 ResolvedJAllemandouT129886 Create wmf_raw.ApiAction table
 ResolvedOttomataT129889 Create mediawiki_ApiAction Kafka topic
 Resolvedbd808T113672 api.log does not indicate errors and exceptions
 Resolvedbd808T116065 Design aggregate tables to drive Action API reports
 OpenNoneT116658 Add Application errors for Mediawiki API to x-analytics
DeclinedNoneT122245 REST API entry point web request statistics at the Varnish level
 Resolvedbd808T132283 Determine which Action API parameters to whitelist/blacklist for action_param_hourly aggregate table
 OpenNoneT137321 Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables
 DeclinedNoneT154912 Is User-Agent data PII when associated with Action API requests?

Event Timeline

• Spage created this task.Sep 28 2015, 8:06 PM
• Spage raised the priority of this task from to Medium.
• Spage updated the task description. (Show Details)
• Spage added a project: Developer-Advocacy.
• Spage added a subscriber: Qgil.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 28 2015, 8:06 PM
• Spage removed a parent task: T102079: Metrics about the use of the Wikimedia web APIs.Sep 28 2015, 8:08 PM
• Spage added a subtask: T102079: Metrics about the use of the Wikimedia web APIs.
• Spage mentioned this in T109829: Developer Relations quarterly goals for October-December 2015.
• Elitre subscribed.Sep 29 2015, 4:54 PM
Qgil moved this task from To triage to October - December 2015 on the Developer-Advocacy board.Sep 29 2015, 6:31 PM
Qgil mentioned this in T114133: Survey on 3rd party syndication / data reuse.Sep 30 2015, 10:13 AM
Qgil updated the task description. (Show Details)
Qgil set Security to None.
Qgil added a comment.Oct 13 2015, 10:26 PM

Someone mentioned http://histography.io/, and I wonder where could we store these URLs of interesting projects allegedly using our Web APIs.

Qgil added a project: DevRel-October-2015.Oct 13 2015, 10:26 PM
• Elitre added a comment.Oct 15 2015, 3:54 PM

When the privacy thing is sorted out, and if there's no better way to do this, maybe we could ask the community to add tools they know/use/authored to that page (putting such a request on wikitech-l and Tech News for example)?

• Spage mentioned this in T115933: Write four showcase/inspire articles for the Web APIs hub.Oct 19 2015, 10:10 PM
• Spage claimed this task.Oct 27 2015, 7:23 PM
Qgil added a project: DevRel-November-2015.Nov 3 2015, 7:30 PM
• Spage raised the priority of this task from Medium to High.Nov 3 2015, 7:33 PM
Qgil removed • Spage as the assignee of this task.Jan 12 2016, 7:21 AM
Qgil lowered the priority of this task from High to Low.
Qgil moved this task from October - December 2015 to Proposed: April-June 2016 on the Developer-Advocacy board.
Qgil added a subscriber: • Spage.
Qgil added a subscriber: • SVentura.
Qgil added a comment.Jan 12 2016, 3:40 PM

This task was one of the T109829: Developer Relations quarterly goals for October-December 2015. The success criteria was

List of known users of Wikimedia web APIs specifying main uses. Metrics of quantitative use of these APIs

Due to personal problems, S Page could not work on this goal and we didn't have any substitute for him.

@SVentura, does your team have a list of known users of Wikimedia web APIs specifying main uses? You have been working on a survey and more, so hopefully you have an initial list to work with. (I understand that a different question is whether we can publish this information). Our quarterly review is next week, and any information you can provide will be useful.

About quantitative metrics, as far as I'm aware this is still blocked by T102079: Metrics about the use of the Wikimedia web APIs and out of the control / skill of the Developer-Advocacy team.

Aklapper added a comment.Jan 20 2016, 4:43 PM

@SVentura: Could you answer the last comment, please?

• SVentura added a comment.Jan 20 2016, 8:20 PM

@Qgil, @Aklapper, Sorry guys, just seeing these.

Yes, we ran a small survey (160 respondents) which gave us limited insight on our current API users - big a small. This is of course a fraction of total 450 million API requests we get a day but it's a start. Here's the live form of the survey FYI https://docs.google.com/a/wikimedia.org/forms/d/1yUrHzyLABN419RCDbzepjoRWCbaWYV4wbtbKPa95C4o/viewform

I am not sharing the actual raw responses here because of existing personal contact information in the forms, we are finalizing a report which I'll share with the product teams. The findings are not actionable because of the small sample size, but the survey confirmed that there is a wealth of information we can get from speaking with our users that will help inform our organization strategy.

Qgil moved this task from Proposed: April-June 2016 to Team radar on the Developer-Advocacy board.Jan 27 2016, 8:43 AM
Qgil added a project: Reading-Admin.Apr 26 2016, 8:52 AM

I added this task to Reading-Admin with the hope that someone there can file it under the appropriate project. Currently this task is only under Developer-Advocacy team radar, even if the question seems to be still relevant.

dr0ptp4kt moved this task from Backlog to Tracking on the Reading-Admin board.Jul 20 2017, 9:45 PM
Quiddity updated the task description. (Show Details)Aug 22 2019, 10:09 AM
dr0ptp4kt removed a project: Reading-Admin.Mar 19 2020, 7:37 PM
Aklapper moved this task from Team radar to To triage on the Developer-Advocacy board.Mar 20 2020, 11:23 AM
bd808 closed this task as Resolved.Apr 2 2020, 5:10 PM
bd808 subscribed.

Closing as "resolved" rather than "declined". The initial goals of this task were ambitious and it turns out also somewhat in conflict with technical and legal realities in the Wikimedia MediaWiki deployments. We learned quite a bit, but really can not ever reach the initial stated goal without either requiring API tokens which can be counted or using User-Agent + IP information as a rough substitute. 4.5 years seems long enough to leave an unsolvable task open. ;)

bd808 closed subtask T102079: Metrics about the use of the Wikimedia web APIs as Resolved.Apr 2 2020, 5:12 PM
bd808 mentioned this in T102079: Metrics about the use of the Wikimedia web APIs.
• Mholloway mentioned this in T285561: API Usage Metrics.Aug 19 2021, 2:33 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL