Page MenuHomePhabricator
Create Task
Maniphest T108618

Publish detailed Action API request information to Hadoop
Closed, ResolvedPublic
Actions

Description

Instrument MediaWiki on the WMF production cluster to send structured Action API request information to Hadoop via Kafka.

Data to collect:

DataNameType
Timestamptsint (unix epoch seconds)
Requesting IPipstring
Requesting User-AgentuserAgentstring
Wiki (wfWikiId())wikistring
Time spent processing request (ms resolution)timeSpentBackendint
Were errors encountered?hadErrorboolean
List of error codeserrorCodesarray<string>
Request parameters (name=value pairs)paramsmap<string,string>

Data will be collected by adding a new debug logging channel (ApiRequest) with structured data in the the PSR-3 context. Any MediaWiki deployment can then choose where and how to route these log messages.

For the WMF production cluster, introduce configuration to route this log channel to the local Kafka cluster in a topic that can be loaded into Hadoop.

Deployment Checklist

These steps are intended to be done in-order.

  1. Commit schema to mediawiki/event-schemas repository (gerrit)
  2. Commit submodule bump to analytics/refinery/source repository (gerrit)
  3. Commit Oozie job to create partitions to analytics/refinery repository (gerrit)
  4. Commit property changes for Camus to operations/puppet repository (gerrit)
  5. Wait for analytics to deploy new versions of refinery and refinery-source to analytics cluster
  6. T129889: Create mediawiki_ApiAction Kafka topic
  7. Go back and fix things that were done incorrectly (T108618#2132875)
  8. Commit submodule bump along with proper configuration to operations/mediawiki-config repository (gerrit)
  9. Deploy initial mediawiki-config patch to production with a sampling rate of a few events per minute for testing
  10. Verify events in Kafka are as expected. Check mediawiki logs for errors.
  11. After enough time has passed (Camus runs once per hour) verify events are showing up in HDFS
  12. Create table in Hive pointing at the events in HDFS (T129886: Create wmf_raw.ApiAction table)
  13. Submit coordinator to Oozie to auto-create partitions
  14. Adjust (or remove) sampling of events in operations/mediawiki-config repository

Original task description

log user agent in api.log
We tell clients to use an interesting user agent, but don't log it to api.php.

In T102079#1417411 Anomie commented

User agent could be included easily enough, but would need to be run by Ops for the text logfile and @bd808 for logstash (if it wouldn't already be there) to verify that it wouldn't make a prohibitive difference to the storage requirements.

Seems a simple change to ApiMain->logRequest(), ApiBase->logFeatureUsage() already logs user agent to api-feature-usage.log.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedQgilT153007 Technical Collaboration annual plan FY2017-18
 ResolvedKeeganT131689 Second iteration of the Technical Collaboration strategy
 DeclinedNoneT926 Engage with established technical communities
 ResolvedQgilT102790 Engineering Community quarterly review (as part of the Community Engagement review)
 ResolvedQgilT93770 Engineering Community quarterly goals for April-June 2015
 InvalidNoneT98348 Implement the Wikimedia Foundation Call to Action 2015
 InvalidNoneT98359 Create spaces for future community-led innovations and new knowledge creation
 ResolvedNoneT98361 Strengthen partnerships with organizations that use or contribute free content, or are aligned with the WMF in the free-knowledge movement
 DeclinedQgilT96013 Identify Wikimedia's top technical partners
 ResolvedQgilT97283 Plan to focus on the Developer audience
 ResolvedNoneT114017 Map current use of Wikimedia web APIs
 Resolvedbd808T102079 Metrics about the use of the Wikimedia web APIs
 DuplicateNoneT117203 [WD] External usage KPI
 ResolvedAddshoreT119071 Track the access of Wikidata data through the API
 DeclinedAddshoreT117205 [WD] Partnerships usage KPI
 Resolvedbd808T108618 Publish detailed Action API request information to Hadoop
 ResolvedNoneT114733 Determine proper encoding for structured log data sent to Kafka by MediaWiki
 ResolvedDzahnT115548 Requesting access to analytics-privatedata-users for Bryan Davis
 Resolvedbd808T118592 Create user defined function to classify network origin of an IP address
 ResolvedJoeT125084 MediaWiki monolog doesn't handle Kafka failures gracefully
 ResolvedJoeT119637 Update HHVM package to recent release
 ResolvedJoeT129467 HHVM 3.12 has a race-condition when starting up
 ResolvedJAllemandouT129886 Create wmf_raw.ApiAction table
 ResolvedOttomataT129889 Create mediawiki_ApiAction Kafka topic

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Nov 17 2015, 6:33 PM

Change 240614 merged by jenkins-bot:
Add structured API request debug logging

https://gerrit.wikimedia.org/r/240614

ReleaseTaggerBot added projects: MW-1.27-release-notes, MW-1.27-release (WMF-deploy-2015-11-17_(1.27.0-wmf.7)).Nov 17 2015, 7:00 PM
Addshore added a parent task: T119071: Track the access of Wikidata data through the API.Nov 19 2015, 2:21 PM
Addshore mentioned this in T119071: Track the access of Wikidata data through the API.Nov 26 2015, 1:19 PM
• SVentura subscribed.Dec 2 2015, 6:46 PM
bd808 closed subtask T118592: Create user defined function to classify network origin of an IP address as Resolved.Dec 23 2015, 4:50 PM
Anomie moved this task from In Dev to Done on the MediaWiki-Action-API board.Jan 4 2016, 7:02 PM
• ggellerman edited projects, added Analytics; removed Analytics-Backlog.Jan 12 2016, 7:32 PM
Milimetric moved this task from Incoming to Radar on the Analytics board.Jan 12 2016, 7:32 PM
• SVentura unsubscribed.Jan 12 2016, 7:33 PM
gerritbot added a comment.Jan 19 2016, 11:49 PM

Change 265164 had a related patch set uploaded (by BryanDavis):
Avro schema for Action API request logging

https://gerrit.wikimedia.org/r/265164

bd808 updated the task description. (Show Details)Jan 21 2016, 4:08 PM
bd808 added a comment.Jan 21 2016, 4:10 PM

I updated the data table in the description to match guidance given during Avro schema code review: event time changed from ISO8601 to unix timestamp and camelCase naming for fields to match EventLogging conventions.

gerritbot added a comment.Jan 21 2016, 4:30 PM

Change 265507 had a related patch set uploaded (by BryanDavis):
Update ApiAction logging channel values

https://gerrit.wikimedia.org/r/265507

bd808 moved this task from Backlog to In Dev/Progress on the Reading-Infrastructure-Team-Old (Don't use) board.Jan 25 2016, 5:37 PM
leila removed a project: Research.Jan 28 2016, 11:28 PM
bd808 added a subtask: T125084: MediaWiki monolog doesn't handle Kafka failures gracefully.Feb 2 2016, 9:06 PM
bd808 moved this task from In Dev/Progress to Needs Review/Feedback on the User-bd808 board.Feb 19 2016, 12:00 AM
bd808 mentioned this in T124420: Technical Collaboration narratives and budget for strategic work.Feb 23 2016, 4:43 PM
gerritbot added a comment.Feb 26 2016, 7:38 PM

Change 265164 merged by Nuria:
Avro schema for Action API request logging

https://gerrit.wikimedia.org/r/265164

bd808 updated the task description. (Show Details)Feb 27 2016, 12:32 AM
gerritbot added a comment.Feb 27 2016, 12:43 AM

Change 273556 had a related patch set uploaded (by BryanDavis):
Add ApiAction avro schema

https://gerrit.wikimedia.org/r/273556

bd808 updated the task description. (Show Details)Feb 27 2016, 12:45 AM
bd808 closed subtask T114733: Determine proper encoding for structured log data sent to Kafka by MediaWiki as Resolved.
gerritbot added a comment.Feb 27 2016, 1:09 AM

Change 273557 had a related patch set uploaded (by BryanDavis):
Add initial oozie job for ApiAction

https://gerrit.wikimedia.org/r/273557

bd808 updated the task description. (Show Details)Feb 27 2016, 1:10 AM
gerritbot added a comment.Feb 27 2016, 1:16 AM

Change 273558 had a related patch set uploaded (by BryanDavis):
Camus: specify latest schema for ApiAction

https://gerrit.wikimedia.org/r/273558

gerritbot added a comment.Feb 27 2016, 1:18 AM

Change 273559 had a related patch set uploaded (by BryanDavis):
Logging: Add ApiRequest kafka logging

https://gerrit.wikimedia.org/r/273559

bd808 updated the task description. (Show Details)Feb 27 2016, 1:19 AM
Qgil unsubscribed.Feb 29 2016, 9:00 AM
gerritbot added a comment.Feb 29 2016, 4:41 PM

Change 265507 merged by jenkins-bot:
Update ApiAction logging channel values

https://gerrit.wikimedia.org/r/265507

ReleaseTaggerBot added a project: MW-1.27-release (WMF-deploy-2016-03-01_(1.27.0-wmf.15)).Feb 29 2016, 5:00 PM
gerritbot added a comment.Feb 29 2016, 9:05 PM

Change 273556 merged by Nuria:
Add ApiAction avro schema

https://gerrit.wikimedia.org/r/273556

gerritbot added a comment.Mar 1 2016, 3:54 PM

Change 273558 merged by Ottomata:
Camus: specify latest schema for ApiAction

https://gerrit.wikimedia.org/r/273558

bd808 updated the task description. (Show Details)Mar 5 2016, 3:23 PM
bd808 mentioned this in T108599: Communicate the WikimediaBot convention {hawk}.Mar 7 2016, 4:43 PM
gerritbot added a comment.Mar 14 2016, 10:20 AM

Change 273557 merged by Joal:
Add initial oozie job for ApiAction

https://gerrit.wikimedia.org/r/273557

bd808 updated the task description. (Show Details)Mar 14 2016, 4:07 PM
bd808 added a comment.Mar 14 2016, 4:12 PM

Now at the "Wait for analytics to deploy new versions of refinery and refinery-source to analytics cluster" stage of the checklist.

EBernhardson added a comment.Mar 14 2016, 4:37 PM

keep intending to work on a way for camus to read the schema's from outside of the jar resources ... just havn't gotten around to it yet :(

bd808 created subtask T129886: Create wmf_raw.ApiAction table.Mar 14 2016, 4:59 PM
bd808 created subtask T129889: Create mediawiki_ApiAction Kafka topic.Mar 14 2016, 5:18 PM
bd808 updated the task description. (Show Details)Mar 14 2016, 5:22 PM
bd808 updated the task description. (Show Details)
bd808 updated the task description. (Show Details)Mar 17 2016, 5:35 PM
bd808 updated the task description. (Show Details)Mar 17 2016, 5:38 PM
Ottomata closed subtask T129889: Create mediawiki_ApiAction Kafka topic as Resolved.Mar 17 2016, 7:39 PM
gerritbot added a comment.Mar 17 2016, 9:51 PM

Change 273559 merged by jenkins-bot:
Logging: Add ApiRequest kafka logging

https://gerrit.wikimedia.org/r/273559

Stashbot subscribed.Mar 17 2016, 10:04 PM

Mentioned in SAL [2016-03-17T22:04:21Z] <bd808@tin> Synchronized wmf-config/event-schemas: Add ApiRequest kafka logging (T108618) (duration: 00m 38s)

Stashbot added a comment.Mar 17 2016, 10:05 PM

Mentioned in SAL [2016-03-17T22:05:17Z] <bd808@tin> Synchronized wmf-config/InitialiseSettings.php: Add ApiRequest kafka logging (T108618) (duration: 00m 34s)

Stashbot added a comment.Mar 17 2016, 10:10 PM

Mentioned in SAL [2016-03-17T22:10:29Z] <bd808@tin> Synchronized wmf-config/InitialiseSettings.php: Disable ApiRequest kafka logging (T108618) (duration: 00m 31s)

gerritbot added a comment.Mar 17 2016, 10:17 PM

Change 278136 had a related patch set uploaded (by BryanDavis):
Cast API timeSpentBackend to an int

https://gerrit.wikimedia.org/r/278136

gerritbot added a comment.Mar 17 2016, 10:23 PM

Change 278170 had a related patch set uploaded (by BryanDavis):
Cast API timeSpentBackend to an int

https://gerrit.wikimedia.org/r/278170

gerritbot added a comment.Mar 17 2016, 10:32 PM

Change 278180 had a related patch set uploaded (by BryanDavis):
Disable ApiRequest properly

https://gerrit.wikimedia.org/r/278180

bd808 added a comment.Mar 17 2016, 10:35 PM

So I messed up and named the logging channel "ApiRequest" in some places (MediaWiki, events-schemas) and "ApiAction" in others (oozie, hdfs). I'm going to have to make some patches to fix that. I think the "ApiRequest" parts will be easiest to change.

gerritbot added a comment.Mar 17 2016, 10:37 PM

Change 278136 merged by jenkins-bot:
Cast API timeSpentBackend to an int

https://gerrit.wikimedia.org/r/278136

gerritbot added a comment.Mar 17 2016, 10:38 PM

Change 278180 merged by jenkins-bot:
Disable ApiRequest properly

https://gerrit.wikimedia.org/r/278180

Stashbot added a comment.Mar 17 2016, 10:42 PM

Mentioned in SAL [2016-03-17T22:42:18Z] <bd808@tin> Synchronized wmf-config/InitialiseSettings.php: Disable ApiRequest properly (T108618) (duration: 00m 27s)

gerritbot added a comment.Mar 17 2016, 10:45 PM

Change 278187 had a related patch set uploaded (by BryanDavis):
Rename ApiRequest to ApiAction

https://gerrit.wikimedia.org/r/278187

gerritbot added a comment.Mar 17 2016, 10:49 PM

Change 278188 had a related patch set uploaded (by BryanDavis):
Rename ApiRequest to ApiAction

https://gerrit.wikimedia.org/r/278188

gerritbot added a comment.Mar 17 2016, 10:51 PM

Change 278170 merged by jenkins-bot:
Cast API timeSpentBackend to an int

https://gerrit.wikimedia.org/r/278170

gerritbot added a comment.Mar 17 2016, 11:00 PM

Change 278193 had a related patch set uploaded (by BryanDavis):
Fix ApiAction record name

https://gerrit.wikimedia.org/r/278193

bd808 added a comment.EditedMar 17 2016, 11:06 PM

New checklist:

gerritbot added a comment.Mar 17 2016, 11:57 PM

Change 278207 had a related patch set uploaded (by BryanDavis):
Rename ApiRequest to ApiAction

https://gerrit.wikimedia.org/r/278207

gerritbot added a comment.Mar 18 2016, 12:09 AM

Change 278187 merged by jenkins-bot:
Rename ApiRequest to ApiAction

https://gerrit.wikimedia.org/r/278187

ReleaseTaggerBot added projects: MW-1.27-release (WMF-deploy-2016-03-15_(1.27.0-wmf.17)), MW-1.27-release (WMF-deploy-2016-03-22_(1.27.0-wmf.18)).Mar 18 2016, 2:01 AM
gerritbot added a comment.Mar 18 2016, 3:25 PM

Change 278193 merged by Ottomata:
Fix ApiAction record name

https://gerrit.wikimedia.org/r/278193

gerritbot added a comment.Mar 18 2016, 3:41 PM

Change 278188 merged by Ottomata:
Rename ApiRequest to ApiAction

https://gerrit.wikimedia.org/r/278188

gerritbot added a comment.Mar 18 2016, 8:15 PM

Change 278346 had a related patch set uploaded (by BryanDavis):
Update mediawiki/event-schemas submodule

https://gerrit.wikimedia.org/r/278346

gerritbot added a comment.Mar 18 2016, 8:25 PM

Change 278347 had a related patch set uploaded (by BryanDavis):
Logging: add ApiAction kafka logging

https://gerrit.wikimedia.org/r/278347

gerritbot added a comment.Mar 21 2016, 2:55 PM

Change 278346 merged by Ottomata:
Update mediawiki/event-schemas submodule

https://gerrit.wikimedia.org/r/278346

bd808 updated the task description. (Show Details)Mar 22 2016, 11:32 PM
gerritbot added a comment.Mar 23 2016, 4:48 PM

Change 278207 merged by jenkins-bot:
Rename ApiRequest to ApiAction

https://gerrit.wikimedia.org/r/278207

gerritbot added a comment.Mar 23 2016, 4:52 PM

Change 278347 merged by jenkins-bot:
Logging: add ApiAction kafka logging

https://gerrit.wikimedia.org/r/278347

Stashbot added a comment.Mar 23 2016, 4:57 PM

Mentioned in SAL [2016-03-23T16:57:30Z] <bd808@tin> Synchronized wmf-config/InitialiseSettings.php: Logging: add ApiAction kafka logging (34f236c) (T108618) (duration: 00m 28s)

bd808 added a comment.Mar 23 2016, 5:16 PM
In T108618#2132875, @bd808 wrote:

New checklist:

bd808 updated the task description. (Show Details)Mar 23 2016, 5:17 PM
bd808 added a comment.Mar 23 2016, 5:19 PM

Verified that data is reaching Kafka from MediaWiki via kafkacat -b kafka1012 -t mediawiki_ApiAction -c 1 on stat1002. Next step is to check back after ~1h to verify that Camus is copying the events to HDFS.

bd808 updated the task description. (Show Details)Mar 25 2016, 2:51 PM
hive (wmf_raw)> describe ApiAction;
OK
col_name        data_type       comment
ts                      int
ip                      string
useragent               string
wiki                    string
timespentbackend        int
haderror                boolean
errorcodes              array<string>
params                  map<string,string>
year                    string
month                   string
day                     string
hour                    string

# Partition Information
# col_name              data_type               comment

year                    string
month                   string
day                     string
hour                    string
hive (wmf_raw)> select count(*) from ApiAction where year = 2016;
OK
_c0
133739
gerritbot added a comment.Mar 25 2016, 2:54 PM

Change 279614 had a related patch set uploaded (by BryanDavis):
Logging: Remove sampling from ApiAction kafka channel

https://gerrit.wikimedia.org/r/279614

gerritbot added a comment.Mar 25 2016, 3:01 PM

Change 279614 merged by jenkins-bot:
Logging: Remove sampling from ApiAction kafka channel

https://gerrit.wikimedia.org/r/279614

bd808 removed a project: Patch-For-Review.Mar 25 2016, 3:05 PM
bd808 updated the task description. (Show Details)
bd808 closed subtask T129886: Create wmf_raw.ApiAction table as Resolved.
bd808 removed a subtask: T116065: Design aggregate tables to drive Action API reports.
bd808 mentioned this in T119015: Create "so you want to push new data into Hive" tutorial.Mar 25 2016, 5:38 PM
bd808 closed this task as Resolved.Apr 6 2016, 11:14 PM

Schema documented on wiki at https://wikitech.wikimedia.org/wiki/Analytics/Data/ApiAction

bd808 moved this task from Needs Review/Feedback to Done on the User-bd808 board.Apr 6 2016, 11:14 PM
Qgil awarded a token.Apr 7 2016, 6:22 AM
bd808 mentioned this in T132283: Determine which Action API parameters to whitelist/blacklist for action_param_hourly aggregate table.Apr 10 2016, 5:48 AM
bd808 moved this task from Done to Archive on the User-bd808 board.Apr 13 2016, 4:08 PM
gerritbot added a comment.Apr 21 2016, 9:02 PM

Change 240617 abandoned by EBernhardson:
Send the api request log to kafka

Reason:
this patch was more an example, the feature has been written and deployed since.

https://gerrit.wikimedia.org/r/240617

Aklapper edited projects, added Analytics-Radar; removed MW-1.27-release (WMF-deploy-2016-03-22_(1.27.0-wmf.18)), MW-1.27-release (WMF-deploy-2016-03-15_(1.27.0-wmf.17)), MW-1.27-release (WMF-deploy-2016-03-01_(1.27.0-wmf.15)), Analytics.Jun 10 2020, 6:44 AM
Aklapper removed subscribers: Anomie, • DarTar, • Tbayer.Oct 16 2020, 5:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits