JAllemandou (joal)
Data Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Feb 11 2015, 6:02 PM (145 w, 13 h)
Availability
Available
IRC Nick
joal
LDAP User
Unknown
MediaWiki User
JAllemandou (WMF)

Recent Activity

Yesterday

JAllemandou added a comment to T176785: Add action api counts to graphite-restbase job.

@Pchelolo is correct, idea is to have both restbase and mw-action-api hourly varnish-requests counts in graphite.

Wed, Nov 22, 6:46 PM · Patch-For-Review, Services (watching), Analytics-Kanban
JAllemandou updated subscribers of T176785: Add action api counts to graphite-restbase job.

@Krinkle and @Addshore: After looking once more in graphite, would mw.api.varnish_requests be ok for you guys? This would allow us to keep the restbase metric and have the same naming scheme for both.
If you prefer, we can also rename both, but we'll need to convince @Pchelolo .

Wed, Nov 22, 5:02 PM · Patch-For-Review, Services (watching), Analytics-Kanban
JAllemandou updated subscribers of T78405: SPIKE: experimenting with importing Revision history from XML dumps into an easier to use format, Avro.

@ggellerman : I think we can close this task, just want to confirm with you.

Wed, Nov 22, 4:18 PM · Analytics, Spike, Analytics-Cluster
JAllemandou added a comment to T78405: SPIKE: experimenting with importing Revision history from XML dumps into an easier to use format, Avro.

Some code exists for converting XML dump in hadoop: https://gerrit.wikimedia.org/r/#/c/361440/
It is not yet merged, but could be soon.

Wed, Nov 22, 4:13 PM · Analytics, Spike, Analytics-Cluster
JAllemandou added a comment to T176785: Add action api counts to graphite-restbase job.

@Krinkle: I'm happy to use a less functional approach, and have for instance analytics.varnish_requests.restbase and analytics.varnish_requests.mw_api.
Only concern is that it changes the existing metric for restbase. @mobrovac and @Pchelolo, is that a big deal?

Wed, Nov 22, 12:08 PM · Patch-For-Review, Services (watching), Analytics-Kanban

Tue, Nov 21

JAllemandou moved T176785: Add action api counts to graphite-restbase job from In Progress to In Code Review on the Analytics-Kanban board.
Tue, Nov 21, 8:26 PM · Patch-For-Review, Services (watching), Analytics-Kanban
JAllemandou updated subscribers of T176785: Add action api counts to graphite-restbase job.

Ping @mobrovac, @Pchelolo, @Gilles , @Anomie, @Krinkle for naming confirmation:
The existing metric is named restbase.requests.varnish_requests, and we plan to go with MediaWiki.api.varnish_request for the new one, reusing the existing graphite domain MediaWiki.api.
Thanks all.

Tue, Nov 21, 10:18 AM · Patch-For-Review, Services (watching), Analytics-Kanban
JAllemandou added a comment to T176785: Add action api counts to graphite-restbase job.

Check in which webrequext_source partition data lives (and in the mean time check for api/rest too):

Tue, Nov 21, 9:57 AM · Patch-For-Review, Services (watching), Analytics-Kanban
JAllemandou claimed T176785: Add action api counts to graphite-restbase job.
Tue, Nov 21, 9:45 AM · Patch-For-Review, Services (watching), Analytics-Kanban
JAllemandou moved T176785: Add action api counts to graphite-restbase job from Next Up to In Progress on the Analytics-Kanban board.
Tue, Nov 21, 9:45 AM · Patch-For-Review, Services (watching), Analytics-Kanban

Mon, Nov 20

JAllemandou moved T179689: Rename historical fields in mediawiki-history from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Mon, Nov 20, 4:28 PM · Analytics-Kanban
JAllemandou moved T179690: Fix mediawiki-history page reconstruction bug (restores) from In Code Review to Paused on the Analytics-Kanban board.
Mon, Nov 20, 4:28 PM · Analytics-Kanban

Mon, Nov 13

JAllemandou removed a project from T180266: Wikistats2: The granularity selector does not work for tops metrics: Analytics-Wikistats.
Mon, Nov 13, 4:26 PM · Analytics-Kanban
JAllemandou edited projects for T180266: Wikistats2: The granularity selector does not work for tops metrics, added: Analytics-Kanban; removed Analytics.
Mon, Nov 13, 4:26 PM · Analytics-Kanban
JAllemandou merged task T180310: wmf.mediawiki_history: page_is_redirect/page_is_redirect_latest into T161146: Provide historical redirect flag in Data Lake edit data.
Mon, Nov 13, 1:20 PM · Analytics-Data-Quality, Analytics
JAllemandou merged T180310: wmf.mediawiki_history: page_is_redirect/page_is_redirect_latest into T161146: Provide historical redirect flag in Data Lake edit data.
Mon, Nov 13, 1:20 PM · Analytics

Fri, Nov 3

JAllemandou claimed T179690: Fix mediawiki-history page reconstruction bug (restores).
Fri, Nov 3, 1:58 PM · Analytics-Kanban
JAllemandou moved T179690: Fix mediawiki-history page reconstruction bug (restores) from Next Up to In Code Review on the Analytics-Kanban board.
Fri, Nov 3, 1:58 PM · Analytics-Kanban
JAllemandou set the point value for T179690: Fix mediawiki-history page reconstruction bug (restores) to 5.
Fri, Nov 3, 1:58 PM · Analytics-Kanban
JAllemandou updated the task description for T179690: Fix mediawiki-history page reconstruction bug (restores).
Fri, Nov 3, 1:57 PM · Analytics-Kanban
JAllemandou created T179692: Fix mediawiki-history page reconstruction bug (restores final).
Fri, Nov 3, 1:57 PM · Analytics-Kanban
JAllemandou created T179690: Fix mediawiki-history page reconstruction bug (restores).
Fri, Nov 3, 1:56 PM · Analytics-Kanban
JAllemandou moved T179074: Fix mediawiki history page reconstruction bug (similar timestamps) from In Progress to In Code Review on the Analytics-Kanban board.
Fri, Nov 3, 1:53 PM · Analytics-Kanban
JAllemandou renamed T179074: Fix mediawiki history page reconstruction bug (similar timestamps) from Fix mediawiki history page reconstruction bug to Fix mediawiki history page reconstruction bug (similar timestamps).
Fri, Nov 3, 1:53 PM · Analytics-Kanban
JAllemandou moved T179689: Rename historical fields in mediawiki-history from Next Up to In Code Review on the Analytics-Kanban board.
Fri, Nov 3, 1:53 PM · Analytics-Kanban
JAllemandou claimed T179689: Rename historical fields in mediawiki-history.
Fri, Nov 3, 1:52 PM · Analytics-Kanban
JAllemandou created T179689: Rename historical fields in mediawiki-history.
Fri, Nov 3, 1:52 PM · Analytics-Kanban

Tue, Oct 31

JAllemandou moved T175844: Provide oozie job running ClickStream spark job regularly from In Progress to Paused on the Analytics-Kanban board.
Tue, Oct 31, 3:05 PM · Patch-For-Review, Analytics-Kanban
JAllemandou renamed T179074: Fix mediawiki history page reconstruction bug (similar timestamps) from Fix mediawiki history page reconstruction bug on move-conflict to Fix mediawiki history page reconstruction bug.
Tue, Oct 31, 3:05 PM · Analytics-Kanban
JAllemandou moved T179074: Fix mediawiki history page reconstruction bug (similar timestamps) from Next Up to In Progress on the Analytics-Kanban board.
Tue, Oct 31, 3:03 PM · Analytics-Kanban

Mon, Oct 30

JAllemandou added a comment to T177257: ArticlePlaceholder hit counts from bnwiki seem bogus.

Hi folks,
Not a bug for me:

SELECT access_method, count(1) from wmf.webrequest WHERE is_pageview AND pageview_info['project'] = 'bn.wikipedia' AND year = 2017 AND month = 9 AND day = 30 AND webrequest_source = 'text' AND x_analytics_map['ns'] = '-1' AND x_analytics_map['special'] = 'AboutTopic' group by access_method;

gives result:

access_method	_c1
desktop	13
mobile web	448

which is coherent with the request @Addshore did that doesn't include bn.m.wikipedia.org.
I let you either close or ping us back :)

Mon, Oct 30, 4:15 PM · User-Addshore, WMDE-Analytics-Engineering, Wikidata, ArticlePlaceholder

Thu, Oct 26

JAllemandou created T179074: Fix mediawiki history page reconstruction bug (similar timestamps).
Thu, Oct 26, 2:53 PM · Analytics-Kanban

Oct 23 2017

JAllemandou created T178832: Investigate AQS cassandra schema hash warninga.
Oct 23 2017, 7:06 PM · Analytics-Kanban
JAllemandou moved T178312: Upgrade AQS restbase-modules from In Code Review to Done on the Analytics-Kanban board.
Oct 23 2017, 7:03 PM · Patch-For-Review, Analytics-Kanban

Oct 20 2017

JAllemandou moved T178504: Update mediawiki_history_reduced oozie job loading AQS druid backend from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Oct 20 2017, 9:55 AM · Analytics-Kanban

Oct 19 2017

JAllemandou created T178587: Fix wikimedia-history revision-deleted data.
Oct 19 2017, 4:24 PM · Analytics-Kanban

Oct 18 2017

JAllemandou moved T178504: Update mediawiki_history_reduced oozie job loading AQS druid backend from Ready to Deploy to In Code Review on the Analytics-Kanban board.
Oct 18 2017, 3:05 PM · Analytics-Kanban
JAllemandou moved T178504: Update mediawiki_history_reduced oozie job loading AQS druid backend from Next Up to Ready to Deploy on the Analytics-Kanban board.
Oct 18 2017, 3:05 PM · Analytics-Kanban
JAllemandou claimed T178504: Update mediawiki_history_reduced oozie job loading AQS druid backend.
Oct 18 2017, 3:04 PM · Analytics-Kanban
JAllemandou created T178504: Update mediawiki_history_reduced oozie job loading AQS druid backend.
Oct 18 2017, 3:04 PM · Analytics-Kanban
JAllemandou moved T178478: Check data from new API endpoints agains existing sources from Next Up to In Progress on the Analytics-Kanban board.
Oct 18 2017, 10:47 AM · Analytics-Kanban
JAllemandou claimed T178478: Check data from new API endpoints agains existing sources .
Oct 18 2017, 10:47 AM · Analytics-Kanban
JAllemandou created T178478: Check data from new API endpoints agains existing sources .
Oct 18 2017, 10:47 AM · Analytics-Kanban

Oct 17 2017

JAllemandou moved T178312: Upgrade AQS restbase-modules from In Progress to In Code Review on the Analytics-Kanban board.
Oct 17 2017, 9:34 AM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T175805: Add mediawiki-history metrics to AQS from Ready to Deploy to Done on the Analytics-Kanban board.
Oct 17 2017, 9:34 AM · Patch-For-Review, Analytics-Kanban

Oct 16 2017

JAllemandou closed T164194: Unique Devices on Pivot, initial screen should not add values by default, is this configurable? as Declined.
Oct 16 2017, 5:25 PM · Analytics-Kanban
JAllemandou added a comment to T164194: Unique Devices on Pivot, initial screen should not add values by default, is this configurable?.

Declinig since we're moving away from pivot because of it being non open-source.
We'll be using supset instead, which will allow to une that at dashboard level.

Oct 16 2017, 5:25 PM · Analytics-Kanban
JAllemandou moved T178312: Upgrade AQS restbase-modules from Next Up to In Progress on the Analytics-Kanban board.
Oct 16 2017, 3:34 PM · Patch-For-Review, Analytics-Kanban
JAllemandou claimed T178312: Upgrade AQS restbase-modules.
Oct 16 2017, 3:33 PM · Patch-For-Review, Analytics-Kanban
JAllemandou created T178312: Upgrade AQS restbase-modules.
Oct 16 2017, 3:33 PM · Patch-For-Review, Analytics-Kanban

Oct 13 2017

JAllemandou moved T175805: Add mediawiki-history metrics to AQS from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Oct 13 2017, 10:41 AM · Patch-For-Review, Analytics-Kanban
JAllemandou reassigned T176223: Create Druid public cluster such AQS can query druid public data from JAllemandou to elukey.
Oct 13 2017, 10:41 AM · Patch-For-Review, Analytics-Kanban, Analytics-Wikistats

Oct 9 2017

JAllemandou added a comment to T174640: Invalid "wikimedia" family in unique devices data due to misplaced WMF-Last-Access-Global cookie .

The change above doesn't change the behavior of cookies, but at least removes wikimedia project-family from the ones available in Druid. The only place it'll still be visible is in hive (it was already removed from to-be-externaly-published datasets).

Oct 9 2017, 6:55 PM · Patch-For-Review, Analytics-Kanban, Traffic, Operations
JAllemandou moved T174640: Invalid "wikimedia" family in unique devices data due to misplaced WMF-Last-Access-Global cookie from Next Up to Ready to Deploy on the Analytics-Kanban board.
Oct 9 2017, 6:52 PM · Patch-For-Review, Analytics-Kanban, Traffic, Operations
JAllemandou moved T163327: Add monthly unique devices dataset to Druid from Ready to Deploy to Done on the Analytics-Kanban board.
Oct 9 2017, 1:56 PM · Analytics-Kanban
JAllemandou moved T175248: Chose how to deal with "Infinity" value for Banners from Ready to Deploy to Done on the Analytics-Kanban board.
Oct 9 2017, 1:06 PM · Patch-For-Review, Analytics-Kanban
JAllemandou added a comment to T164497: Cleaning scheme for banner data _SUCCESS files.

Interestingly, checking this patch allowed me to notice we don't currently use the new version of druid loading for banners ... Mwarf

Oct 9 2017, 11:59 AM · Patch-For-Review, Analytics-Kanban
JAllemandou added a comment to T164497: Cleaning scheme for banner data _SUCCESS files.

Base script merged - Now it needs a puppet companion to lauh the script :)

Oct 9 2017, 11:56 AM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T175248: Chose how to deal with "Infinity" value for Banners from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Oct 9 2017, 11:45 AM · Patch-For-Review, Analytics-Kanban

Oct 6 2017

JAllemandou moved T175248: Chose how to deal with "Infinity" value for Banners from Paused to In Code Review on the Analytics-Kanban board.
Oct 6 2017, 7:16 PM · Patch-For-Review, Analytics-Kanban

Sep 28 2017

JAllemandou added a comment to T175268: Stub new mediawiki history-based metrics.

URIs to be mocked are defined as swagger-config in Restbase pull request: https://github.com/wikimedia/restbase/pull/875

Sep 28 2017, 3:40 PM · Patch-For-Review, Analytics-Kanban, Analytics-Wikistats
JAllemandou moved T163327: Add monthly unique devices dataset to Druid from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Sep 28 2017, 2:53 PM · Analytics-Kanban

Sep 26 2017

JAllemandou created T176785: Add action api counts to graphite-restbase job.
Sep 26 2017, 7:29 PM · Patch-For-Review, Services (watching), Analytics-Kanban
JAllemandou moved T176599: Correct typo in oozie mobile_apps_session from Ready to Deploy to Done on the Analytics-Kanban board.
Sep 26 2017, 2:58 PM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T175707: Add "PhantomJS" to the list of bots in webrequest definition. from Ready to Deploy to Done on the Analytics-Kanban board.
Sep 26 2017, 11:22 AM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T176600: Correct (AGAIN ??!!) mediawiki_history cumulative count names from Ready to Deploy to Done on the Analytics-Kanban board.
Sep 26 2017, 11:20 AM · Analytics-Kanban
JAllemandou moved T176599: Correct typo in oozie mobile_apps_session from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Sep 26 2017, 10:24 AM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T175707: Add "PhantomJS" to the list of bots in webrequest definition. from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Sep 26 2017, 10:24 AM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T176600: Correct (AGAIN ??!!) mediawiki_history cumulative count names from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Sep 26 2017, 10:16 AM · Analytics-Kanban

Sep 25 2017

JAllemandou moved T175805: Add mediawiki-history metrics to AQS from In Progress to In Code Review on the Analytics-Kanban board.
Sep 25 2017, 12:17 PM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T163327: Add monthly unique devices dataset to Druid from Next Up to In Code Review on the Analytics-Kanban board.
Sep 25 2017, 9:13 AM · Analytics-Kanban
JAllemandou claimed T163327: Add monthly unique devices dataset to Druid.
Sep 25 2017, 9:13 AM · Analytics-Kanban
JAllemandou merged task T174174: Add edits endpoint to AQS using druid as a backend into T175805: Add mediawiki-history metrics to AQS.
Sep 25 2017, 8:53 AM · Patch-For-Review, Analytics-Kanban, Analytics-Wikistats
JAllemandou merged T174174: Add edits endpoint to AQS using druid as a backend into T175805: Add mediawiki-history metrics to AQS.
Sep 25 2017, 8:53 AM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T176600: Correct (AGAIN ??!!) mediawiki_history cumulative count names from Next Up to In Code Review on the Analytics-Kanban board.
Sep 25 2017, 8:52 AM · Analytics-Kanban
JAllemandou claimed T176600: Correct (AGAIN ??!!) mediawiki_history cumulative count names.
Sep 25 2017, 8:52 AM · Analytics-Kanban
JAllemandou created T176600: Correct (AGAIN ??!!) mediawiki_history cumulative count names.
Sep 25 2017, 8:52 AM · Analytics-Kanban
JAllemandou moved T175707: Add "PhantomJS" to the list of bots in webrequest definition. from Next Up to In Code Review on the Analytics-Kanban board.
Sep 25 2017, 8:50 AM · Patch-For-Review, Analytics-Kanban
JAllemandou claimed T175707: Add "PhantomJS" to the list of bots in webrequest definition..
Sep 25 2017, 8:47 AM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T176599: Correct typo in oozie mobile_apps_session from Next Up to In Code Review on the Analytics-Kanban board.
Sep 25 2017, 8:45 AM · Patch-For-Review, Analytics-Kanban
JAllemandou claimed T176599: Correct typo in oozie mobile_apps_session.
Sep 25 2017, 8:43 AM · Patch-For-Review, Analytics-Kanban
JAllemandou created T176599: Correct typo in oozie mobile_apps_session.
Sep 25 2017, 8:43 AM · Patch-For-Review, Analytics-Kanban

Sep 13 2017

JAllemandou added a comment to T158972: Spark job to produce clickstream dataset .

Actually @Nuria this task is only the spark, not the oozie that will make the saprk job run regularly.
I'll modify docs once we have the other one (T175844) done.

Sep 13 2017, 5:27 PM · Analytics-Kanban, Research
JAllemandou created T175844: Provide oozie job running ClickStream spark job regularly.
Sep 13 2017, 5:27 PM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T174915: Productionize mediawiki-history-reduced druid ingestion from Ready to Deploy to Done on the Analytics-Kanban board.
Sep 13 2017, 11:17 AM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T174484: Add redirect and pagelinks tables for partition repair in sqoop job for mediawiki history from Ready to Deploy to Done on the Analytics-Kanban board.
Sep 13 2017, 11:17 AM · Analytics-Kanban, Patch-For-Review
JAllemandou moved T161824: Add zero carrier to pageview_hourly data on druid from Ready to Deploy to Done on the Analytics-Kanban board.
Sep 13 2017, 11:12 AM · Analytics-Kanban, Patch-For-Review
JAllemandou moved T175163: Move GraphiteClient from refinery-core to refinery-job module from Ready to Deploy to Done on the Analytics-Kanban board.
Sep 13 2017, 10:55 AM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T158972: Spark job to produce clickstream dataset from Ready to Deploy to Done on the Analytics-Kanban board.
Sep 13 2017, 10:55 AM · Analytics-Kanban, Research
JAllemandou moved T158972: Spark job to produce clickstream dataset from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Sep 13 2017, 9:36 AM · Analytics-Kanban, Research
JAllemandou moved T175805: Add mediawiki-history metrics to AQS from Next Up to In Progress on the Analytics-Kanban board.
Sep 13 2017, 9:36 AM · Patch-For-Review, Analytics-Kanban
JAllemandou claimed T175805: Add mediawiki-history metrics to AQS.
Sep 13 2017, 9:36 AM · Patch-For-Review, Analytics-Kanban
JAllemandou set the point value for T175805: Add mediawiki-history metrics to AQS to 13.
Sep 13 2017, 9:36 AM · Patch-For-Review, Analytics-Kanban
JAllemandou created T175805: Add mediawiki-history metrics to AQS.
Sep 13 2017, 9:35 AM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T174484: Add redirect and pagelinks tables for partition repair in sqoop job for mediawiki history from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Sep 13 2017, 8:03 AM · Analytics-Kanban, Patch-For-Review
JAllemandou moved T174915: Productionize mediawiki-history-reduced druid ingestion from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Sep 13 2017, 8:02 AM · Patch-For-Review, Analytics-Kanban

Sep 12 2017

JAllemandou created T175707: Add "PhantomJS" to the list of bots in webrequest definition..
Sep 12 2017, 3:28 PM · Patch-For-Review, Analytics-Kanban

Sep 8 2017

JAllemandou moved T175163: Move GraphiteClient from refinery-core to refinery-job module from In Code Review to Ready to Deploy on the Analytics-Kanban board.
Sep 8 2017, 2:31 PM · Patch-For-Review, Analytics-Kanban
JAllemandou moved T174174: Add edits endpoint to AQS using druid as a backend from In Progress to In Code Review on the Analytics-Kanban board.
Sep 8 2017, 2:31 PM · Patch-For-Review, Analytics-Kanban, Analytics-Wikistats
JAllemandou added a comment to T166689: Provide top domain and data to truly test superset .

I updated the datasources to contain interesting metrics and created a new user able to use and create visualisation, but not mess with the config:

Sep 8 2017, 10:55 AM · Patch-For-Review, Analytics