In T215819#4955223, @chelsyx wrote:

If this is about the fund raising banner impression in the app, I think @JoeWalsh has done some calculations for Dec 2018. But I don't think we have any data back to Dec 2017, @mpopov do you know?

Tried using analytics-mysql on stat1007 and got "permission denied". Follow-up question: will it be made available on SWAP?

Working on acquiring data for this and it turned out to be much, much harder and more involved than I anticipated. The analytics data we get from the app just has the notification ID, so I'm in the process of getting the Echo extension tables into Hive (that's the part that's causing problems & delays) so that I can get editing activity for users who got the notifications on Android.

@Charlotte: thanks for the ping! Right now my priorities are: the notifications analysis, SEO sameAs analysis, some Search query migration, and then this. I'm working on acquiring data for T213458 and it turned out to be much, much harder than I anticipated so that's creating some delays.

@Charlotte: thanks for the ping! I'm working on acquiring data for T213458 and it turned out to be much, much harder than I anticipated.

In T172410#4965227, @Marostegui wrote:

In T172410#4965217, @mpopov wrote:

I just noticed that the tables related to the Echo extension are (surprisingly) not yet available in the enwiki shard (s1-analytics-replica.eqiad.wmnet), but are in analytics-store.eqiad.wmnet. Is there a page we can refer to to check on parity/status of data availability?

The echo tables are on x1. x1 is a separate instance in production, and hence on the new dbstore model.
They are available on the old analytics because it has all the instances mixed.

I just noticed that the tables related to the Echo extension are (surprisingly) not yet available in the enwiki shard (s1-analytics-replica.eqiad.wmnet), but are in analytics-store.eqiad.wmnet. Is there a page we can refer to to check on parity/status of data availability?

In T212386#4964982, @elukey wrote:

In T212386#4964927, @Ottomata wrote:

Sure, we can definitely work on a shared sqoop wrapper

I don't even mean a sqoop wrapper! Just I think the script should be able to output the proper hostname:port, maybe in a 'dry-run'n mode, rather than always connecting via mysql CLI.

analytics-mysql --output (or --dry-run?) enwiki

Would just output the hostname:port for easy use with other tools.

Didn't read it carefully, this seems a great idea, going to work on it asap!

@elukey: is there a recommendation for how to sqoop with the shards? since a shell command would look like:

@jcrespo: is it safe to assume that the current config of s3 (default) will stay that way? and if not, can I assume that the shard which is designated as the default one will have "(default)" in the comment?

In T212386#4951229, @jcrespo wrote:

Important, dblists are not the canonical place for database distribution (while it tries to be in sync). The canonical method is the array at the sectionsByDB array:

https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/db-eqiad.php;c178d2ccf57d28434c8acf06b5b793125ff25e1b$28?as=source&blame=off

Actually, I would like to request for https://github.com/wikimedia/operations-mediawiki-config/tree/master/dblists to have a single file I can download which has a mapping.

Oh, cool, thanks!

More reports: https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=10982794

Query & scripts: https://github.com/wikimedia-research/SEO-Experiment-Sitemaps

Just like with sameAs (T211191#4885005), there is no visible change in traffic due to the intervention:

1. Can we see whether people have made accounts through the app?

In T213702#4908349, @Gilles wrote:

The folks who (may) still work on sitemaps might be interested in that data, though. @ovasileva @mpopov ?

In T211366#4900240, @JKatzWMF wrote:

@Charlotte @mpopov

1. i saw in the ticket about the bug that hundreds of duplicate events were being sent.
   1. If we can detect them is there a way to correct for them? and
   2. if there are hundreds, what % of total events is that? The data doesn't have to be perfect, so if it's less than 5% of events of any given event-type, then I wouldn't let it get in the way of analysis.

Done

@JMinor @Charlotte: after speaking with @kzimmerman we decided I should manage this project and that it has priority (at least on our team). I'll follow up with you on next steps.

Cool! Thank you!

Done

@Abit @Ramsey-WMF in addition to T213597#4900741, here's the history of that metric with a 7-day rolling average to smooth the daily data a bit:

In T213597#4900903, @Neil_P._Quinn_WMF wrote:

True, but its revisions do have revision_is_deleted set, so you've already filtered them out of your query.

Okay, here are the numbers which were calculated with the following conditions:

In T213597#4893765, @Neil_P._Quinn_WMF wrote:

I noticed once big thing: it seems like your counts of file page edits (n_edits_total, n_additions_total, etc.) include the initial edit that creates the pages, so in the end you're getting the proportion of files which have metadata added in the first 2 months, including during the initial upload.

I tried excluding those initial creations (event_timestamp != page_creation_timestamp), and it looks like the proportion goes from 99% to 50%.

Thanks for clarifying! Okay, one more question for @Abit & @Ramsey-WMF just so everyone is on the same page. The statistic you want is: the % of all uploaded files which have had additions to their pages in the first 2 months after upload.

Just updated my database of sampled pages (using December 2018 snapshot) and recounted pageviews from 2018-11-01 to 2019-01-15 (code & data over at GitHub). There has not been any change, up or down since the rollout:

@Ramsey-WMF @Abit: hi, I would like to clarify what "metadata" includes. Here's my initial list:

Yup, I just double checked that I could still reproduce it in 6.1.4 (yes) as a consistency check and then installed 6.2.0, tried it, and this has been fixed. Erasing articles & lists works correctly in v6.2! Thanks & good job @NHarateh_WMF & @JoeWalsh!

A huge chunk of my analysis got invalidated when Dmitry & I found out that the underlying data was faulty (T213190). Specifically, all of the analysis related to session length, number of sessions, number of pages read per session. Unfortunately the nature of the bug means that we won't be able to compare those metrics before & after the update.

Nevermind, per T170022#4800915 & T170022#4866564 I guess there's nobody actually managing Maps and RI is just doing maintenance and fixing critical bugs.

By the way, on ouR side, the package 'wmf' (which I maintain) that we use for querying databases from R can be augmented to have an internal map of dbs to shards, and I can easily add a way to update that map from an external file. We would just require that the JSON/YAML/CSV/whatever file with the mapping is publicly accessible.

In T212172#4853129, @chelsyx wrote:

Here're some use cases from my work for the iOS app team:

• Of course, as @Neil_P._Quinn_WMF mentioned in T161149, I will need change_tag and change_tag_def to figure out which edits are made through the iOS app.
• For Wikidata short description edits made through the app, revision_comment_temp and comment tables are needed to figure out which language this edits is.

Done

Instances deleted

Done. Just tested everything and it's all good, so I've deleted the instances running Ubuntu Trusty. The only instances up are running Debian Stretch.

@EBjune Hiya! Even though I'm an admin for the project I can't do much because Horizon thinks there are 2 more instances than there actually are.

We seem to have a few ghost instances that are preventing me from launching a new Stretch instance which would replace the existing discovery-production one that runs on Trusty:

It works! :D Thanks, @aborrero!

Thank you, @Nuria!

In T211366#4820766, @Charlotte wrote:

If you can suggest ways to redesign the reading list analytics (separate ticket for us) let's do that based on the research questions we memorialised in the design deck for that feature.

In T211606#4814078, @elukey wrote:

@mpopov please also keep in mind that things like T211605#4814020 could happen with a project that is still developing fast and does not care much about breaking existing users, so upgrades might not be easy :D

@EBjune I am waiting for someone with access to the internal WMF apt repository to make shiny-server available.

1. Plan of action

Finished what I could and the findings are available over at https://people.wikimedia.org/~bearloga/reports/wikidata-incompleteness.html

I'll check with the team at our meeting later today and let you know :)

Alright, dependencies for Shiny Server resolved. Now on to the problem of Shiny Server itself:

I think the window on this closed.

Updating status to reflect reality.

Here's a draft of the report I sent out privately back in July: https://commons.wikimedia.org/wiki/File:Wikipedia_Android_app_multilingual_update_post-release_report.pdf

@ovasileva feel free to re-open if you have additional questions

@kzimmerman, @Neil_P._Quinn_WMF, and I need to schedule a discussion with @MNovotny_WMF to follow-up on the results

So @Gehel told me we might actually be fine. I'm testing out this theory but ran into a problem with the current puppet config for shiny_server module:

Things are looking okay so far: