In T144100#5035139, @JAllemandou wrote:

Reading HIVE-5672, the root bug has been fixed

True, but the Hive version we have is old enough so that we don't have that fix ... <shame/>

@Milimetric Would you mind pointing me to the definition this task will update? If there are formatting changes to how fields are delimited and escaped, we will need to find documentation to update, and write release notes for downstream consumers of the dump files.

Reading HIVE-5672, the root bug has been fixed and we're in theory welcome to use ROW FORMAT without (or with) LOCAL. I think that's worth a try because it gives us more flexibility to escape or delimit in common variants. I would lean towards double-quoting the title field as well as escaping whitespace other than <space>, but want to make a case for not escaping <space>, that it makes our format more human-readable.

In https://phabricator.wikimedia.org/diffusion/ANRE/browse/master/oozie/pageview/hourly/transform_pageview_to_legacy_format.hql$25 , we're doing a raw string concatenation in order to produce the output format. The code comment has some suggestions,

INSERT OVERWRITE DIRECTORY "${destination_directory}"
    -- Since "ROW FORMAT DELIMITED DELIMITED FIELDS TERMINATED BY ' '" only
    -- works for exports to local directories (see HIVE-5672), we have to
    -- prepare the lines by hand through concatenation :-(
    -- Set 0 as volume column since we don't use it.
    SELECT
        CONCAT_WS(" ", qualifier, page_title, cast(view_count AS string), "0") line

This is pedantic of me, but I want to break the subtask link to T152628 and make it a "mentioned in", for the reason @Tgr gives in T144100#2982645 . I'm renaming this task to make it more clear that the scope is to fix escaping in the dumps files.

1. With invalid title

I'll take a look at this and its subtasks.

In T216425#5004126, @MoritzMuehlenhoff wrote:

He probably used his wikimedia.org address for Phabricator notifications and can't access that any longer. I've just sent him a quick ping to his private address.

In T161907#4952554, @Ejegg wrote:

diagonal banner list on Special:CentralNoticeBanners in Firefox:

After talking to @joal at All Hands, I'm now thinking that T211069 should be the first priority for integrating the scoring pipeline into Hadoop. We should design a stream transformer to calculate features for each revision, and store the resulting values partitioned by feature name. This involves very different challenges than what's being discussed here, and in the end we might be able to entirely circumvent this minor glitch about merging the data centers.

Didn't have time to do this, regrettably.

I've only had very general conversations about this project so far, and didn't realize there was working code :-)

In T214874#4925938, @mobrovac wrote:

If I understand this task correctly, it seems to be a duplicate of T107196: Set up revscoring entry points in RESTBase.

Turns out this is really annoying in a stock mediawiki-vagrant, it obscures the form:

WIP: https://github.com/wikimedia/revscoring/compare/roots?expand=1

@fr-tech: Okay, just the main patch left and it's in your court! Let me know if I can do anything more.

Just for fun, I elaborated on the quick estimate based on existing w_cache files. Note that these are not the "root" data sources, these are the final, calculated values ready to be input into a model. Storing the calculated values is one alternative we might consider, the tradeoff is that the values are compact and can be used for existing models, but lack flexibility and completeness, so additional features added in the future will probably require another full MW API extraction.

Just discovered some good documentation about this exact problem, here: https://www.mediawiki.org/wiki/Manual:Extension_registration#Requirements_(dependencies)

@Nikerabbit Do you know of any mechanism to enable Extension:Translate when unit testing other extensions? ExtensionRegistry#isLoaded tells me that Translate isn't available from WMF CI tests for FundraisingTranslateWorkflow.

In T200297#4903879, @Milimetric wrote:

This wikitext-in-JSON thing seems really complicated.

Important change of plans—We're discussing backfilling, and it might be best to allow mismatched model versions in the dumps for now. In other words, go ahead and backfill with whatever the current model version is. Normalized data will continue to be segregated by model version, but the monthly "current" and "historical" dumps will patch together whatever scores are available, simply taking the newest model version used to score each revision.

@Nikerabbit Hi, are we unblocked now that MLEB 2019.01 is released?

In T213943#4888027, @Nikerabbit wrote:

All we have to do is rely on a namespace-based message group, and configure the group with:

Just to clarify, you have a code that gets run when the state of a language of message group's translations is changed to published?