Page MenuHomePhabricator

Refine issues with page links change event
Closed, ResolvedPublic2 Story Points

Description

One of the most recent failures:

The following 1 of 1 dataset partitions for table event.mediawiki_page_links_change failed refinement:

Failure(org.wikimedia.analytics.refinery.job.refine.RefineTargetException: Failed refinement of hdfs://analytics-hadoop/wmf/data/raw/event/eqiad_mediawiki_page-links-change/hourly/2019/06/21/11 -> `event`.`mediawiki_page_links_change` (datacenter="eqiad",year=2019,month=6,day=21,hour=11). Original exception: java.lang.RuntimeException: Failed reading JSON/YAML data from http://schema.svc.eqiad.wmnet:8190/repositories/mediawiki/jsonschemanull)

Event Timeline

Nuria created this task.Jun 21 2019, 3:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 21 2019, 3:48 PM

Looking at events:

kafkacat -C -b kafka-jumbo1001.eqiad.wmnet -t eqiad.mediawiki.page-links-change

{"added_links": [{"external": false, "link": "/wiki/Property:P792"}], "database": "wikidatawiki", "meta": {"domain": "www.wikidata.org", "dt": "2019-06-21T15:31:36+00:00",
"id": "a7c10fa5-9439-11e9-9a57-1866da994aad", "request_id": "XQz4VwpAAK4AACniPh4AAABP", "schema_uri": "mediawiki/page/links-change/1", "topic": "mediawiki.page-links-chan
ge", "uri": "https://www.wikidata.org/wiki/Q11774134"}, "page_id": 12910767, "page_is_redirect": false, "page_namespace": 0, "page_title": "Q11774134", "performer": {"user
_edit_count": 109577, "user_groups": ["*", "user", "autoconfirmed"], "user_id": 5295, "user_is_bot": false, "user_registration_dt": "2012-10-30T21:23:44Z", "user_text": "S
hisma"}, "rev_id": 966331018}

Seems like schema is there: mediawiki/page/links-change/1

Also curl http://schema.svc.eqiad.wmnet:8190/repositories/mediawiki/jsonschema/mediawiki/page/links-change/1.yaml renders proper schema

elukey added a subscriber: elukey.Jun 21 2019, 3:50 PM

More verbose stacktrace from the yarn logs:

java.lang.RuntimeException: Failed reading JSON/YAML data from http://schema.svc.eqiad.wmnet:8190/repositories/mediawiki/jsonschemanull
        at org.wikimedia.analytics.refinery.core.jsonschema.JsonSchemaLoader.load(JsonSchemaLoader.java:59)
        at org.wikimedia.analytics.refinery.core.jsonschema.EventSchemaLoader.load(EventSchemaLoader.java:69)
        at org.wikimedia.analytics.refinery.core.jsonschema.EventSchemaLoader.getEventSchema(EventSchemaLoader.java:98)
        at org.wikimedia.analytics.refinery.core.jsonschema.EventSchemaLoader.getEventSchema(EventSchemaLoader.java:108)
        at org.wikimedia.analytics.refinery.job.refine.EventSparkSchemaLoader.loadSchema(SchemaLoaders.scala:75)
        at org.wikimedia.analytics.refinery.job.refine.RefineTarget.inputDataFrame(RefineTarget.scala:398)
        at org.wikimedia.analytics.refinery.job.refine.Refine$$anonfun$refineTargets$1.apply(Refine.scala:497)
        at org.wikimedia.analytics.refinery.job.refine.Refine$$anonfun$refineTargets$1.apply(Refine.scala:493)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at org.wikimedia.analytics.refinery.job.refine.Refine$.refineTargets(Refine.scala:493)
        at org.wikimedia.analytics.refinery.job.refine.Refine$$anonfun$8.apply(Refine.scala:376)
        at org.wikimedia.analytics.refinery.job.refine.Refine$$anonfun$8.apply(Refine.scala:370)
        at scala.collection.parallel.AugmentedIterableIterator$class.map2combiner(RemainsIterator.scala:115)
        at scala.collection.parallel.immutable.ParHashMap$ParHashMapIterator.map2combiner(ParHashMap.scala:76)
        at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1054)
        at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49)
        at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
        at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
        at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51)
        at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1051)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:152)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443)
        at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.io.FileNotFoundException: http://schema.svc.eqiad.wmnet:8190/repositories/mediawiki/jsonschemanull
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1890)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at java.net.URL.openStream(URL.java:1045)
        at com.fasterxml.jackson.core.JsonFactory._optimizedStreamFromURL(JsonFactory.java:1479)
        at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:313)
        at org.wikimedia.analytics.refinery.core.jsonschema.JsonSchemaLoader.load(JsonSchemaLoader.java:56)
        ... 30 more
Nuria renamed this task from Refine issues with page to Refine issues with page links change event.Jun 21 2019, 3:50 PM
Nuria added a comment.Jun 21 2019, 3:52 PM

Looking at raw data for hours that failed: hdfs://analytics-hadoop/wmf/data/raw/event/eqiad_mediawiki_page-links-change/hourly/2019/06/21/11

Nuria moved this task from Next Up to In Progress on the Analytics-Kanban board.Jun 21 2019, 4:02 PM

Strangely, looking at revision/create raw events, their meta: property is the same, there are no structural differences between that and the page-links-change events.

Nuria added a comment.Jun 21 2019, 4:54 PM

Looking at table_whitelist_regex = ^(mediawiki_api_request|mediawiki_cirrussearch_request|mediawiki_user_blocks_change|mediawiki_page_links_change|mediawiki_page_restrictions_change|mediawiki_revision_tags_change)$

We are refining " tags change" events that contain $schema":"/mediawiki/revision/tags-change/1.0.0 the mediawiki_page_links_change events does not have the $schema field and thus schema is not parsed by refine

Change 518299 had a related patch set uploaded (by Nuria; owner: Nuria):
[operations/puppet@production] Removing page_links events from the ones that get refined

https://gerrit.wikimedia.org/r/518299

Per @JAllemandou we could use a better error message that gives us a bit more info of what refine is actually failing maybe? (as there are several versions of refine in place for different streams)

Change 518299 merged by Elukey:
[operations/puppet@production] Removing page_links events from refine whitelist

https://gerrit.wikimedia.org/r/518299

Nuria assigned this task to Ottomata.Jun 21 2019, 6:04 PM
mforns triaged this task as High priority.Jun 24 2019, 3:58 PM
mforns moved this task from Incoming to Operational Excellence on the Analytics board.

Change 518770 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Refine page_links_change data using old Refine job

https://gerrit.wikimedia.org/r/518770

Change 518770 merged by Ottomata:
[operations/puppet@production] Refine page_links_change data using old Refine job

https://gerrit.wikimedia.org/r/518770

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Jun 26 2019, 3:02 PM
Ottomata set the point value for this task to 2.

Change 519486 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move page_links_change back to new schema aware refine job

https://gerrit.wikimedia.org/r/519486

Change 519486 merged by Ottomata:
[operations/puppet@production] Move page_links_change back to new schema aware refine job

https://gerrit.wikimedia.org/r/519486

Nuria moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Jun 28 2019, 4:05 PM
Nuria closed this task as Resolved.Jul 3 2019, 10:24 PM