Page MenuHomePhabricator

Get ContentTranslation to anticipate and handle AbuseFilter warnings and blocks
Open, HighPublic

Description

When user submit something through CX and they hit an AbuseFilter warning, they can't just submit the content again and let it go through.

An example failure is an external link from the source article that was transferred to the translation (example report). The issue is hard to spot for the user since the problematic content was not created explicitly.

Currently, we are showing an error when the user tries to publish (with a cryptic debug message). We can consider better approaches for (a) communicating the issue and (b) nor preventing publishing content.

For example, we can highlight the problematic content (e.g., the external link) for the user to fix. If the issues is not fixed when the user tries to publish, a warning informs the user that the content will be published without the problematic elements. In this way, most of the translation can be published instead of getting blocked because a small piece of content.

Proposed approach

Some considerations:

  • There are different kinds of issues: warnings and errors.
  • We can identify them at different moments: once the article is loaded, as content is edited or when publishing.
  • For some of the issues we can identify in which section/paragraph they are happening while others affect the whole document (or is not possible to identify the specific problematic content).
  • It is possible to have more than one issue at a time.

An overview is provided below illustrating how to represent some of the above aspects:

Based on the above, here are some representations for different cases:

Error on load that can be mapped to a section: title blacklist
In this case, as soon as the content is loaded, an error preventing from publishing is identified on one section:

  • An error message is shown in red.
  • The publish button remains disabled until the title gets edited.
  • The affected paragraph (the title) gets highlighted with a red line next to it. When the paragraph gets the editing focus, a contextual card will be shown (more below).

  • A card is shown with the specific error, and an indication that it prevents from publishing.
  • A "Learn more" link points to the local page with Abuse filter information.
  • A specific action that helps to resolve the issue can be provided. In this case, using MT to translate the title could be helpful. For other cases, other actions (e.g., a quick access to clear the content) or no action may be preferred.

Warning while editing
In this case, an issue is identified in a paragraph. Thus, the rest of the document can be published.

  • Paragraph gets highighted in yellow.
  • The button for publishing remains enabled.
  • A card with a yellow flag is shown when the paragraph has editing focus.
  • The card describes the issue (in the example above, in a generic way, more on this later) and explains that the content will be skipped when publishing.
  • A "learn more" link provides access to the local Abuse filter information.
  • A quick action allows to clear the paragraph.

When the user publishes a translation with content issues, a warning is shown indicating that the article was published but some parts were skipped:

Considering that spam links account for a high percentage of the abusefilter issues, a more specific card can be defined for this case:

Warnings while publishing
In this case, warnings should not prevent the user from publishing but they should surface the issues to encourage users to fix them.

Errors when publishing about the whole article: short article, protected page
In this case, the issue is discovered when trying to publish and it is not attached to a specific part of the document.

  • A message is shown after the user clicked on "Publish article".

Multiple issues
Since we are extending the existing error/warning mechanism, it is possible that several issues apply.
For those cases, instead of showing multiple bars, it is better to show one at a time (errors having precedence over warnings) and allowing users to navigate them:

Details

Related Gerrit Patches:
mediawiki/extensions/ContentTranslation : masterDisplay abusefilter errors nicely

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

The easiest solution for this is to send the user to the action=edit or veaction=edit form as soon as the click "publish", so that they can handle everything in the normal editing interface.

The easiest solution for this is to send the user to the action=edit or veaction=edit form as soon as the click "publish", so that they can handle everything in the normal editing interface.

It isn't easy: we need to record the fact that article has been published and add the correct tag. Yes, this can be done, but it is not easy.

I would also consider ending up on a page full of wikitext a slap on the face after we have worked hard to create a tool (which should be easy to use) that doesn't show any markup.

While looking at this, I thought I could prepare for my presentation by finding "green" (translatable) articles in beforehand. So, I found this one: https://no.wikipedia.org/wiki/Barnas_supershow. (translating no->nn) Checked the first paragraph, link is green, good. I then pressed "Interrupt" ("Avbryt"), and then started again, with the same article. This time it did not turn green, but stayed gray. Is it my interrupted article which blocks the second attempt? If so, a notification telling just that would have helped.

Hi Trond, I don't understand your problem clearly. Can you describe the actions you take? I am assuming you mean "Start translation" and "Cancel" buttons on the page selection dialog on Special:CX, because there is no Cancel anywhere in the translation page, but as this issue is about saving of completed translations, I am confused.

Nemo_bis added a comment.EditedNov 30 2015, 1:18 PM

It isn't easy: we need to record the fact that article has been published and add the correct tag. Yes, this can be done, but it is not easy.

Maybe some code reuse is possible. VisualEditor does it in some 25 lines total, AFAICT.
https://phabricator.wikimedia.org/diffusion/EVED/browse/master/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.js;e24eaea22cce151ff72fa8876589cdb5f4d583fe$1343-1348
https://phabricator.wikimedia.org/diffusion/EVED/browse/master/modules/ve-mw/init/ve.init.mw.Target.js;e24eaea22cce151ff72fa8876589cdb5f4d583fe$1472-1492

Certainly it's easier than reinventing integration with abusefilter, captcha, edittools and dozens or hundreds of extensions which modify the editor, EditPage and friends.

VisualEditor does it in some 25 lines total, AFAICT.

That is not the same thing. VE doesn't need to record the information we do.

Certainly it's easier than reinventing integration with abusefilter,

If by integration you mean displaying the error messages, that is easy. Actually highlighting the thing that causes the error is the difficult part, and to my knowledge neither VE nor the wikitext editor does this. Do correct me if I am wrong.

captcha,

Hasn't been an issue to us.

edittools and dozens or hundreds of extensions which modify the editor, EditPage and friends.

Probably not needed anyway.

I am assuming you mean "Start translation" and "Cancel" buttons on the page selection dialog on Special:CX,

Yes, I do, on this page:
https://nn.wikipedia.org/wiki/Spesial:Innhaldsomsetjing
Here I choose the page, and press "Start a new translation".
Then I get the page (two columns), I click (doubleclick, actually) to the right to get the first paragraph. In the good cases, the button "Publiser omsetjing" ("publish translation") turns grean, and I can go on.
In the bad cases, it stays gray, and I give in.
In this special case (article "Barnas supershow") it turned green, and I thought I could save the article for my demo.
So I pressed the "Alle omsetjingar" ("All translations") in the upper left corner, and got back to this page:
https://nn.wikipedia.org/wiki/Spesial:Innhaldsomsetjing
But when I then tried the same article again (starting over by suggesting the same article + translating the first paragraph on the two-column page), the "Publiser omsetjing" did not turn green.

as this issue is about saving of completed translations, I am confused.

Yes, and I am already sorry that I brought up this second issue (of not being able to try again), thereby putting the process off the track. To me, this was a plan B: If I do not know which article will work (be translatable), I may at least collect a bunch of articles that _are_ translatable.

Now, this is not (yet) possible, since the "green button" is not a guarantee it will work the next time. So we should probably concentrate upon discussing the more constructive solutions:

a. temporal solution: Mark the offensive paragraph or spot, so that I may delete it from the translation
b. permanent solution: make the MT platform more robust, so that it will not refuse publishing what has been published elsewhere.

@Trondtr in your case there seems to be a specific issue of "Publish translation/Publiser omsetjing" not getting enabled (= turning green). It should turn green always when a change is made to the translation and only turn back gray when you click it. We do not doo any checks whether the publishing will actually succeed before you click the button (and the failures which happen after clicking is what we are investigating here). To investigate your issue, can you please tell what is your browser name and version. If you know how to access the developer console (also known as JavaScript console) of your browser, please check if you see any errors or warnings in there.

As far as I have seen, it is a 1-1 relation between the "PT" button turning green when clicking the first paragraph, and the possibility of publishing anything. I cannot understand that you "do not doo any checks whether the publishing will actually succeed before you click the button " -- since even though "you" do not do it, evidently the computer does (green = I may continue translating the article; grey = I better give in -- this generalisation holds 100%).

Procedure: Press the paragraph, see the button turn

  • green ==> it will publish in the end
  • grey ==> it will not publish in the end

So yes the test for publishability is immediately being conducted by the program.

. Browser: Safari 9.0.1, Chrome (46.0.2490.86), Firefox (41.0) (same behaviour), OS: Mac El Capitan. Let me have a look at the JavaScript console.

Ok, Show JavaScript --

... I notice this line:

var d = "body .adblock-blacklist-dialog";

In any case, I do not know JavaScript, but here goes:

// Requires clickwatcher.js and elementchain.js and jQuery

// Create a selector that matches an element.
function selector_from_elm(el) {

var attrs = ['id', 'class', 'name', 'src', 'href', 'data'];
var result = [el.prop('nodeName')];
for (var i = 0; i < attrs.length; i++) {
  var attr = attrs[i];
  var val = el.attr(attr);
  if (val)
    result.push('[' + attr + '=' + JSON.stringify(val) + ']');
}
return result.join('');

}

Wizard that walks the user through selecting an element and choosing
properties to block.
clicked_item: the element that was right clicked, if any.
advanced_user:bool
function BlacklistUi(clicked_item, advanced_user) {

// If a dialog is ever closed without setting this to false, the
// object fires a cancel event.
this._cancelled = true;

// steps through dialog - see _preview()
this._current_step = 0;

this._callbacks = { 'cancel': [], 'block': [] };

this._clicked_item = clicked_item;
this._advanced_user = advanced_user;

}

// TODO: same event framework as ClickWatcher
BlacklistUi.prototype.cancel = function(callback) {

this._callbacks.cancel.push(callback);

}
BlacklistUi.prototype.block = function(callback) {

this._callbacks.block.push(callback);

}
BlacklistUi.prototype._fire = function(eventName, arg) {

var callbacks = this._callbacks[eventName];
for (var i = 0; i < callbacks.length; i++)
  callbacks[i](arg);

}
BlacklistUi.prototype._onClose = function() {

if (this._cancelled == true) {
  this._ui_page1.empty().remove();
  this._ui_page2.empty().remove();
  $(".adblock-ui-stylesheet").remove();
  this._chain.current().show();
  this._fire('cancel');
}

}
BlacklistUi.prototype.handle_change = function() {

this._last.show();
this._chain.current().hide();
this._last = this._chain.current();
this._redrawPage1();
this._redrawPage2();
this._preview(selector_from_elm(this._chain.current()));

}

BlacklistUi.prototype.show = function() {

// If we don't know the clicked element, we must find it first.
if (this._clicked_item == null) {
  var clickWatcher = new ClickWatcher();
  var that = this;
  clickWatcher.cancel(function() {
    that._preview(null);
    that._fire('cancel');
  });
  clickWatcher.click(function(element) {
    that._clicked_item = element;
    that.show();
  });
  this._preview("*");
  clickWatcher.show();
  return;
}

// If we do know the clicked element, go straight to the slider.
else {
  this._chain = new ElementChain(this._clicked_item);

  this._ui_page1 = this._build_page1();
  this._ui_page2 = this._build_page2();

  this._last = this._chain.current();
  this._chain.change(this, this.handle_change);
  this._chain.change();

  this._redrawPage1();
  this._ui_page1.dialog('open');
}

}

BlacklistUi.prototype._build_page1 = function() {

var that = this;

var page = $("<div>").
  append(translate("sliderexplanation")).
  append("<br/>").
  append("<input id='slider' type='range' min='0' value='0'/>").
  append("<div id='selected_data'></div>");

var btns = {};
var adblock_default_button_text = translate("buttonlooksgood");
btns[adblock_default_button_text] = {
  text: adblock_default_button_text,
  'class': 'adblock_default_button',
  click: function() {
    that._cancelled = false;
    that._ui_page1.dialog('close');
    that._cancelled = true;
    that._redrawPage2();
    that._ui_page2.dialog('open');
  }
}
btns[translate("buttoncancel")] =
    function() {
      that._ui_page1.dialog('close');
    }

page.dialog({
    dialogClass: "adblock-blacklist-dialog",
    position: [50, 50],
    width: 410,
    autoOpen: false,
    title: translate("slidertitle"),
    buttons: btns,
    open: function() {
      that._current_step = 1;
      that._preview(selector_from_elm(that._chain.current()));
    },
    close: function() {
      that._preview(null);
      that._onClose();
    }
  });
page.dialog("widget").css("position", "fixed");
changeTextDirection($("body .adblock-blacklist-dialog"));

var depth = 0;
var guy = this._chain.current();
while (guy.length > 0 && guy[0].nodeName != "BODY") {
  guy = guy.parent();
  depth++;
}
$("#slider", page).
  attr("max", Math.max(depth - 1, 1)).
  on("input change", function() {
    that._chain.moveTo(this.valueAsNumber);
  });

return page;

}

BlacklistUi.prototype._build_page2 = function() {

var that = this;

var page = $("<div>" + translate("blacklisteroptions1") +
  "<div>" +
    "<div id='adblock-details'></div><br/>" +
    "<div id='count'></div>" +
  "</div>" +
  "<div>" +
    "<br/>" + translate("blacklisternotsure") +
    "<br/><br/></div>" +
  "<div style='clear:left; font-size:smaller; margin-top: -20px;'>" +
    "<br/>" + translate("blacklisterthefilter") +
    "<div style='margin-left:15px;margin-bottom:15px'>" +
      "<div>" +
        "<div id='summary'></div><br/>" +
        "<div id='filter_warning'></div>" +
      "</div>" +
    "</div>" +
  "</div>" +
  "</div>");

var btns = {};
var adblock_default_button_text = translate("buttonblockit");
btns[adblock_default_button_text] = {
  text: adblock_default_button_text,
  'class': 'adblock_default_button',
  click: function() {
    var rule = $("#summary", that._ui_page2).text();
    if (rule.length > 0) {
      var filter = getUnicodeDomain(document.location.hostname) + "##" + rule;
      BGcall('add_custom_filter', filter, function() {
        block_list_via_css([rule]);
        that._ui_page2.dialog('close');
        that._fire('block');
      });
    } else {alert(translate("blacklisternofilter"));}
  }
}
if (that._advanced_user)
  btns[translate("buttonedit")] =
    function() {
      var custom_filter = getUnicodeDomain(document.location.hostname) + '##' + $("#summary", that._ui_page2).text();
      that._ui_page2.dialog('close');
      custom_filter = prompt(translate("blacklistereditfilter"), custom_filter);
      if (custom_filter) {//null => user clicked cancel
        if (!/\#\#/.test(custom_filter))
          custom_filter = "##" + custom_filter;
        BGcall('add_custom_filter', custom_filter, function(ex) {
          if (!ex) {
            block_list_via_css([custom_filter.substr(custom_filter.indexOf('##') + 2)]);
            that._fire('block');
          } else
            alert(translate("blacklistereditinvalid1", ex));
        });
      }
    }
btns[translate("buttonback")] =
    function() {
      that._cancelled = false;
      that._ui_page2.dialog('close');
      that._cancelled = true;
      that._redrawPage1();
      that._ui_page1.dialog('open');
    }
btns[translate("buttoncancel")] =
    function() {
      that._ui_page2.dialog('close');
    }

page.dialog({
    dialogClass: "adblock-blacklist-dialog ui-page-2",
    position:[50, 50],
    width: 500,
    autoOpen: false,
    title: translate("blacklisteroptionstitle"),
    buttons: btns,
    open: function() {
      that._current_step = 2;
      that._preview($('#summary', that._ui_page2).text());
    },
    close: function() {
      that._preview(null);
      that._onClose();
    }
  });
page.dialog("widget").css("position", "fixed");
changeTextDirection($("body .adblock-blacklist-dialog"));

return page;

}
BlacklistUi.prototype._redrawPage1 = function() {

var el = this._chain.current();

var selected_data = $("#selected_data", this._ui_page1);
selected_data.html("<b>" + translate("blacklisterblockedelement") + "</b><br/>");

selected_data.append($("<i></i>").text("<" + el[0].nodeName));
var attrs = ["id", "class", "name", "src", "href", "data"];
for (var i in attrs) {
  var val = BlacklistUi._ellipsis(el.attr(attrs[i]));
  if (val)
    selected_data.append("<br/>").
                append($("<i></i>").
                         text(attrs[i] + '="' + val + '"').
                         css("margin-left", "10px"));
}
selected_data.append("<i>&nbsp;&gt;</i>");

}

Return the CSS selector generated by the blacklister. If the
user has not yet gotten far enough through the wizard to
// determine the selector, return an empty string.
BlacklistUi.prototype._makeFilter = function() {

var result = [];

var el = this._chain.current();
var detailsDiv = $("#adblock-details", this._ui_page2);

if ($("input[type='checkbox']#cknodeName", detailsDiv).is(':checked')) {
  result.push(el.prop('nodeName'));
  // Some iframed ads are in a bland iframe.  If so, at least try to
  // be more specific by walking the chain from the body to the iframe
  // in the CSS selector.
  if (el.prop('nodeName') == 'IFRAME' && el.attr('id') == '') {
    var cur = el.parent();
    while (cur.prop('nodeName') != 'BODY') {
      result.unshift(cur.prop('nodeName') + " ");
      cur = cur.parent();
    }
  }
}
var attrs = ['id', 'class', 'name', 'src', 'href', 'data'];
for (var i in attrs) {
  if ($("input[type='checkbox']#ck" + attrs[i], detailsDiv).is(':checked'))
    result.push('[' + attrs[i] + '=' + getUnicodeUrl(JSON.stringify(el.attr(attrs[i]))) + ']');
}

var warningMessage;
if (result.length == 0)
  warningMessage = translate("blacklisterwarningnofilter");
else if (result.length == 1 && $("input[type='checkbox']#cknodeName", detailsDiv).is(':checked'))
  warningMessage = translate("blacklisterblocksalloftype", [result[0]]);
$("#filter_warning", this._ui_page2).
  css("display", (warningMessage ? "block" : "none")).
  text(warningMessage);
return result.join('');

}

BlacklistUi.prototype._redrawPage2 = function() {

var el = this._chain.current();
var that = this;

var detailsDiv = $("#adblock-details", that._ui_page2);

var summary = $("#summary", that._ui_page2);

function updateFilter() {
  var theFilter = that._makeFilter();

  summary.text(theFilter);

  var matchCount = $(theFilter).not(".ui-dialog").not(".ui-dialog *").length;

  $("#count", that._ui_page2).
    html("<center>" + ((matchCount == 1) ?
        translate("blacklistersinglematch") :
        translate("blacklistermatches", ["<b>" + matchCount + "</b>"]))
        + "</center>");
}

detailsDiv.empty();
var attrs = ['nodeName', 'id', 'class', 'name', 'src', 'href', 'data'];
for (var i = 0; i < attrs.length; i++) {
  var attr = attrs[i];
  var longVal = (attr == "nodeName" ? el.prop("nodeName") : el.attr(attr));
  var val = BlacklistUi._ellipsis(longVal);

  if (!val)
    continue;

  // Check src, data and href only by default if no other identifiers are
  // present except for the nodeName selector.
  var checked = true;
  if (attr == 'src' || attr == 'href' || attr == 'data')
    checked = $("input", detailsDiv).length == 1;

  var italic = $("<i></i>").text(val);
  var checkboxlabel = $("<label></label>").
    html(translate("blacklisterattrwillbe",
         ["<b>" + (attr == 'nodeName' ? translate("blacklistertype") : attr) +
          "</b>", "<i></i>"])).
    attr("for", "ck" + attr);
  $('i', checkboxlabel).replaceWith(italic);

  var checkbox = $("<div></div>").
    append("<input type=checkbox " + (checked ? 'checked="checked"': '') +
           " id=ck" + attr + " /> ").
    append(checkboxlabel);

  checkbox.find("input").change(function() {
    updateFilter();
    that._preview($("#summary", that._ui_page2).text());
  });

  detailsDiv.append(checkbox);
}

updateFilter();

}

Change the appearance of a CSS selector on the page, or if null, undo the change.
Inputs: selector:string - the selector generated by the blacklist wizard
BlacklistUi.prototype._preview = function(selector) {

$("#adblock_blacklist_preview_css").remove();
if (!selector) return;

var css_preview = document.createElement("style");
css_preview.type = "text/css";
css_preview.id = "adblock_blacklist_preview_css";

var d = "body .adblock-blacklist-dialog";

switch (this._current_step) {
case 0:
  // Raise highlight.
  css_preview.innerText = "body .adblock-highlight-node,";
  break;
case 1:
  // Show ui_page1.
  css_preview.innerText = d + ", " + d + " * {opacity:1!important;} ";
  // Fade the selector, while skipping any matching children.
  css_preview.innerText += selector + " {opacity:.1!important;} " +
    selector + " " + selector + " {opacity:1!important;}";
  break;
case 2:
  // Show ui_page2.
  css_preview.innerText = d + " input, " + d +
    " button {display:inline-block!important;} " + d + ".ui-page-2, " + d +
    " div:not(#filter_warning), " + d + " .ui-icon, " + d + " a, " + d +
    " center {display:block!important;} " +  d + " #adblock-details, " + d +
    " span, " + d + " b, " + d + " i {display:inline!important;} ";
  // Hide the specified selector.
  css_preview.innerText += selector + " {display:none!important;}";
}

// Finally, raise the UI above *all* website UI, using max 32-bit signed int.
css_preview.innerText += " " + d + " {z-index:2147483647!important;}";

document.documentElement.appendChild(css_preview);

}

Return a copy of value that has been truncated with an ellipsis in
the middle if it is too long.
Inputs: value:string - value to truncate
size?:int - max size above which to truncate, defaults to 50
BlacklistUi._ellipsis = function(value, size) {

if (value == null)
  return value;

if (size == undefined)
  size = 50;

value = getUnicodeUrl(value);

var half = size / 2 - 2; // With ellipsis, the total length will be ~= size

if (value.length > size)
  value = (value.substring(0, half) + "..." +
           value.substring(value.length - half));

return value;

}

//@ sourceURL=/uiscripts/blacklisting/blacklistui.js

Ok, Show JavaScript --

... I notice this line:

var d = "body .adblock-blacklist-dialog";

In any case, I do not know JavaScript, but here goes:

// Requires clickwatcher.js and elementchain.js and jQuery

// Create a selector that matches an element.
function selector_from_elm(el) {

var attrs = ['id', 'class', 'name', 'src', 'href', 'data'];
var result = [el.prop('nodeName')];
for (var i = 0; i < attrs.length; i++) {
  var attr = attrs[i];
  var val = el.attr(attr);
  if (val)
    result.push('[' + attr + '=' + JSON.stringify(val) + ']');
}
return result.join('');

}

Wizard that walks the user through selecting an element and choosing
properties to block.
clicked_item: the element that was right clicked, if any.
advanced_user:bool
function BlacklistUi(clicked_item, advanced_user) {

// If a dialog is ever closed without setting this to false, the
// object fires a cancel event.
this._cancelled = true;

// steps through dialog - see _preview()
this._current_step = 0;

this._callbacks = { 'cancel': [], 'block': [] };

this._clicked_item = clicked_item;
this._advanced_user = advanced_user;

}

// TODO: same event framework as ClickWatcher
BlacklistUi.prototype.cancel = function(callback) {

this._callbacks.cancel.push(callback);

}
BlacklistUi.prototype.block = function(callback) {

this._callbacks.block.push(callback);

}
BlacklistUi.prototype._fire = function(eventName, arg) {

var callbacks = this._callbacks[eventName];
for (var i = 0; i < callbacks.length; i++)
  callbacks[i](arg);

}
BlacklistUi.prototype._onClose = function() {

if (this._cancelled == true) {
  this._ui_page1.empty().remove();
  this._ui_page2.empty().remove();
  $(".adblock-ui-stylesheet").remove();
  this._chain.current().show();
  this._fire('cancel');
}

}
BlacklistUi.prototype.handle_change = function() {

this._last.show();
this._chain.current().hide();
this._last = this._chain.current();
this._redrawPage1();
this._redrawPage2();
this._preview(selector_from_elm(this._chain.current()));

}

BlacklistUi.prototype.show = function() {

// If we don't know the clicked element, we must find it first.
if (this._clicked_item == null) {
  var clickWatcher = new ClickWatcher();
  var that = this;
  clickWatcher.cancel(function() {
    that._preview(null);
    that._fire('cancel');
  });
  clickWatcher.click(function(element) {
    that._clicked_item = element;
    that.show();
  });
  this._preview("*");
  clickWatcher.show();
  return;
}

// If we do know the clicked element, go straight to the slider.
else {
  this._chain = new ElementChain(this._clicked_item);

  this._ui_page1 = this._build_page1();
  this._ui_page2 = this._build_page2();

  this._last = this._chain.current();
  this._chain.change(this, this.handle_change);
  this._chain.change();

  this._redrawPage1();
  this._ui_page1.dialog('open');
}

}

BlacklistUi.prototype._build_page1 = function() {

var that = this;

var page = $("<div>").
  append(translate("sliderexplanation")).
  append("<br/>").
  append("<input id='slider' type='range' min='0' value='0'/>").
  append("<div id='selected_data'></div>");

var btns = {};
var adblock_default_button_text = translate("buttonlooksgood");
btns[adblock_default_button_text] = {
  text: adblock_default_button_text,
  'class': 'adblock_default_button',
  click: function() {
    that._cancelled = false;
    that._ui_page1.dialog('close');
    that._cancelled = true;
    that._redrawPage2();
    that._ui_page2.dialog('open');
  }
}
btns[translate("buttoncancel")] =
    function() {
      that._ui_page1.dialog('close');
    }

page.dialog({
    dialogClass: "adblock-blacklist-dialog",
    position: [50, 50],
    width: 410,
    autoOpen: false,
    title: translate("slidertitle"),
    buttons: btns,
    open: function() {
      that._current_step = 1;
      that._preview(selector_from_elm(that._chain.current()));
    },
    close: function() {
      that._preview(null);
      that._onClose();
    }
  });
page.dialog("widget").css("position", "fixed");
changeTextDirection($("body .adblock-blacklist-dialog"));

var depth = 0;
var guy = this._chain.current();
while (guy.length > 0 && guy[0].nodeName != "BODY") {
  guy = guy.parent();
  depth++;
}
$("#slider", page).
  attr("max", Math.max(depth - 1, 1)).
  on("input change", function() {
    that._chain.moveTo(this.valueAsNumber);
  });

return page;

}

BlacklistUi.prototype._build_page2 = function() {

var that = this;

var page = $("<div>" + translate("blacklisteroptions1") +
  "<div>" +
    "<div id='adblock-details'></div><br/>" +
    "<div id='count'></div>" +
  "</div>" +
  "<div>" +
    "<br/>" + translate("blacklisternotsure") +
    "<br/><br/></div>" +
  "<div style='clear:left; font-size:smaller; margin-top: -20px;'>" +
    "<br/>" + translate("blacklisterthefilter") +
    "<div style='margin-left:15px;margin-bottom:15px'>" +
      "<div>" +
        "<div id='summary'></div><br/>" +
        "<div id='filter_warning'></div>" +
      "</div>" +
    "</div>" +
  "</div>" +
  "</div>");

var btns = {};
var adblock_default_button_text = translate("buttonblockit");
btns[adblock_default_button_text] = {
  text: adblock_default_button_text,
  'class': 'adblock_default_button',
  click: function() {
    var rule = $("#summary", that._ui_page2).text();
    if (rule.length > 0) {
      var filter = getUnicodeDomain(document.location.hostname) + "##" + rule;
      BGcall('add_custom_filter', filter, function() {
        block_list_via_css([rule]);
        that._ui_page2.dialog('close');
        that._fire('block');
      });
    } else {alert(translate("blacklisternofilter"));}
  }
}
if (that._advanced_user)
  btns[translate("buttonedit")] =
    function() {
      var custom_filter = getUnicodeDomain(document.location.hostname) + '##' + $("#summary", that._ui_page2).text();
      that._ui_page2.dialog('close');
      custom_filter = prompt(translate("blacklistereditfilter"), custom_filter);
      if (custom_filter) {//null => user clicked cancel
        if (!/\#\#/.test(custom_filter))
          custom_filter = "##" + custom_filter;
        BGcall('add_custom_filter', custom_filter, function(ex) {
          if (!ex) {
            block_list_via_css([custom_filter.substr(custom_filter.indexOf('##') + 2)]);
            that._fire('block');
          } else
            alert(translate("blacklistereditinvalid1", ex));
        });
      }
    }
btns[translate("buttonback")] =
    function() {
      that._cancelled = false;
      that._ui_page2.dialog('close');
      that._cancelled = true;
      that._redrawPage1();
      that._ui_page1.dialog('open');
    }
btns[translate("buttoncancel")] =
    function() {
      that._ui_page2.dialog('close');
    }

page.dialog({
    dialogClass: "adblock-blacklist-dialog ui-page-2",
    position:[50, 50],
    width: 500,
    autoOpen: false,
    title: translate("blacklisteroptionstitle"),
    buttons: btns,
    open: function() {
      that._current_step = 2;
      that._preview($('#summary', that._ui_page2).text());
    },
    close: function() {
      that._preview(null);
      that._onClose();
    }
  });
page.dialog("widget").css("position", "fixed");
changeTextDirection($("body .adblock-blacklist-dialog"));

return page;

}
BlacklistUi.prototype._redrawPage1 = function() {

var el = this._chain.current();

var selected_data = $("#selected_data", this._ui_page1);
selected_data.html("<b>" + translate("blacklisterblockedelement") + "</b><br/>");

selected_data.append($("<i></i>").text("<" + el[0].nodeName));
var attrs = ["id", "class", "name", "src", "href", "data"];
for (var i in attrs) {
  var val = BlacklistUi._ellipsis(el.attr(attrs[i]));
  if (val)
    selected_data.append("<br/>").
                append($("<i></i>").
                         text(attrs[i] + '="' + val + '"').
                         css("margin-left", "10px"));
}
selected_data.append("<i>&nbsp;&gt;</i>");

}

Return the CSS selector generated by the blacklister. If the
user has not yet gotten far enough through the wizard to
// determine the selector, return an empty string.
BlacklistUi.prototype._makeFilter = function() {

var result = [];

var el = this._chain.current();
var detailsDiv = $("#adblock-details", this._ui_page2);

if ($("input[type='checkbox']#cknodeName", detailsDiv).is(':checked')) {
  result.push(el.prop('nodeName'));
  // Some iframed ads are in a bland iframe.  If so, at least try to
  // be more specific by walking the chain from the body to the iframe
  // in the CSS selector.
  if (el.prop('nodeName') == 'IFRAME' && el.attr('id') == '') {
    var cur = el.parent();
    while (cur.prop('nodeName') != 'BODY') {
      result.unshift(cur.prop('nodeName') + " ");
      cur = cur.parent();
    }
  }
}
var attrs = ['id', 'class', 'name', 'src', 'href', 'data'];
for (var i in attrs) {
  if ($("input[type='checkbox']#ck" + attrs[i], detailsDiv).is(':checked'))
    result.push('[' + attrs[i] + '=' + getUnicodeUrl(JSON.stringify(el.attr(attrs[i]))) + ']');
}

var warningMessage;
if (result.length == 0)
  warningMessage = translate("blacklisterwarningnofilter");
else if (result.length == 1 && $("input[type='checkbox']#cknodeName", detailsDiv).is(':checked'))
  warningMessage = translate("blacklisterblocksalloftype", [result[0]]);
$("#filter_warning", this._ui_page2).
  css("display", (warningMessage ? "block" : "none")).
  text(warningMessage);
return result.join('');

}

BlacklistUi.prototype._redrawPage2 = function() {

var el = this._chain.current();
var that = this;

var detailsDiv = $("#adblock-details", that._ui_page2);

var summary = $("#summary", that._ui_page2);

function updateFilter() {
  var theFilter = that._makeFilter();

  summary.text(theFilter);

  var matchCount = $(theFilter).not(".ui-dialog").not(".ui-dialog *").length;

  $("#count", that._ui_page2).
    html("<center>" + ((matchCount == 1) ?
        translate("blacklistersinglematch") :
        translate("blacklistermatches", ["<b>" + matchCount + "</b>"]))
        + "</center>");
}

detailsDiv.empty();
var attrs = ['nodeName', 'id', 'class', 'name', 'src', 'href', 'data'];
for (var i = 0; i < attrs.length; i++) {
  var attr = attrs[i];
  var longVal = (attr == "nodeName" ? el.prop("nodeName") : el.attr(attr));
  var val = BlacklistUi._ellipsis(longVal);

  if (!val)
    continue;

  // Check src, data and href only by default if no other identifiers are
  // present except for the nodeName selector.
  var checked = true;
  if (attr == 'src' || attr == 'href' || attr == 'data')
    checked = $("input", detailsDiv).length == 1;

  var italic = $("<i></i>").text(val);
  var checkboxlabel = $("<label></label>").
    html(translate("blacklisterattrwillbe",
         ["<b>" + (attr == 'nodeName' ? translate("blacklistertype") : attr) +
          "</b>", "<i></i>"])).
    attr("for", "ck" + attr);
  $('i', checkboxlabel).replaceWith(italic);

  var checkbox = $("<div></div>").
    append("<input type=checkbox " + (checked ? 'checked="checked"': '') +
           " id=ck" + attr + " /> ").
    append(checkboxlabel);

  checkbox.find("input").change(function() {
    updateFilter();
    that._preview($("#summary", that._ui_page2).text());
  });

  detailsDiv.append(checkbox);
}

updateFilter();

}

Change the appearance of a CSS selector on the page, or if null, undo the change.
Inputs: selector:string - the selector generated by the blacklist wizard
BlacklistUi.prototype._preview = function(selector) {

$("#adblock_blacklist_preview_css").remove();
if (!selector) return;

var css_preview = document.createElement("style");
css_preview.type = "text/css";
css_preview.id = "adblock_blacklist_preview_css";

var d = "body .adblock-blacklist-dialog";

switch (this._current_step) {
case 0:
  // Raise highlight.
  css_preview.innerText = "body .adblock-highlight-node,";
  break;
case 1:
  // Show ui_page1.
  css_preview.innerText = d + ", " + d + " * {opacity:1!important;} ";
  // Fade the selector, while skipping any matching children.
  css_preview.innerText += selector + " {opacity:.1!important;} " +
    selector + " " + selector + " {opacity:1!important;}";
  break;
case 2:
  // Show ui_page2.
  css_preview.innerText = d + " input, " + d +
    " button {display:inline-block!important;} " + d + ".ui-page-2, " + d +
    " div:not(#filter_warning), " + d + " .ui-icon, " + d + " a, " + d +
    " center {display:block!important;} " +  d + " #adblock-details, " + d +
    " span, " + d + " b, " + d + " i {display:inline!important;} ";
  // Hide the specified selector.
  css_preview.innerText += selector + " {display:none!important;}";
}

// Finally, raise the UI above *all* website UI, using max 32-bit signed int.
css_preview.innerText += " " + d + " {z-index:2147483647!important;}";

document.documentElement.appendChild(css_preview);

}

Return a copy of value that has been truncated with an ellipsis in
the middle if it is too long.
Inputs: value:string - value to truncate
size?:int - max size above which to truncate, defaults to 50
BlacklistUi._ellipsis = function(value, size) {

if (value == null)
  return value;

if (size == undefined)
  size = 50;

value = getUnicodeUrl(value);

var half = size / 2 - 2; // With ellipsis, the total length will be ~= size

if (value.length > size)
  value = (value.substring(0, half) + "..." +
           value.substring(value.length - half));

return value;

}

//@ sourceURL=/uiscripts/blacklisting/blacklistui.js

Update: (... still do not know whether I should be ashamed or still baffled): When testing earlier, I have tried both with and without the adblock filter, both ways with bad results (but in hindsight, I cannot promise I restarted the browser). Looking at the "warning, adblock" above, I again turned it off, and ensured that JavaScript was on. After that I translated perhaps 6 articles, All worked, except one (which also worked when I tried it a second time). In Chrome, I had had JavaScript _on_ the whole time (with the earlier reported sad results), now I am not able to repeat them. I will thus comtinue translating, and report possible errors. At the moment they are hard to find.

Update again: Well, definitely not ashamed, but baffled, perhaps. The improved behaviour turned out to be short-lived. Even with JavaScript _on_ and adblock off, I get the same unpredictable behaviour as before, even with the "fail" percentage rising, Tonight I have still not found a single page working (getting translation). Examples of failed pages:
https://no.wikipedia.org/wiki/Narve_Skarpmoen
https://no.wikipedia.org/wiki/Grensevakt

In the short run, my presentation is tomorrow, and I just close my eyes and hope for the best. In the slightly longer run, this problem is a blocker.

Trondtr, on what wiki are you trying to publish the translations? I don't see anything in Special:AbuseLog/Trondtr on any of the wikis you are active on.

I try to publish on nn. Today (when I had the presentation) I managed to publish one. At the moment my success rate for publishing is appr 20% (8 out of 10 attempts just give a grey "publish translation" button. You see my translations here:

https://nn.wikipedia.org/wiki/Spesial:Bidrag/Trondtr

I know nothing about the AbuseLog. When I am not able to translate, the button stay gray. I click it, but nothing happens. I do not know what is (should be) logged or not. If it helps, I may start listing the failed articles (if the problems is in the article). If others are able to translate my failed attempts, we should instead look into my browser settings.

I know nothing about the AbuseLog. When I am not able to translate, the button stay gray.

The button staying grey is something that happens after you got an error publishing or is it happening in the first place?

If the button is just grey, that is an issue we have not been able to reproduce. In any case, the list of articles you experienced problems with (or some of them) would be helpful for the developers to identify the potential issues.

@Trondtr, thanks for your reports and your help testing the tool despite the issues.

Here is the procedure:

  1. I choose text + title for the translated article
  2. I click the "Start a new translation"
  3. The page I arrive at has the original to the left, an empty field to the right, and a grey "Publish translation" button
  4. I click the paragraphs, one by one, and get translations from Apertium. Now, things go good or bad:

GOOD:
After my first click, I get translated text, and the grey button turn green. Now I know I may continue, I will be able to publish in the end.
BAD:
After my first click, I get translated text, but the grey button does NOT turn green. Now I know I have to abandon this translation. No matter what I do, I will not be able to translate on any point. The button will stay gray, and nothing will happend when I press the grey button.

No other error message turns up. The button either starts out gray and continues as gray, or it turns green when the first paragraph is translated (I imagine the program scans the text for errors)..

Failed articles: I have reportet appr 7-10 of them, more to come.

Here is the procedure:
...

Thanks. This is really useful. It seems that the issue is not related to the abusefilter since the issue happens before trying to publish. I created a separate ticket for this issue: T120837: Publish button does not become active when editing the translation. That will allow to collect details about it as it is investigated.

So, I turned off the Enable AdBlock, and the success rate went up from 0% to 20%:

Ok, published:
https://no.wikipedia.org/wiki/Avis_i_Skolen

(still) not publishable, even with the AdBlock filter turned off:
https://no.wikipedia.org/wiki/Henry_Boucha
https://no.wikipedia.org/wiki/Redaktørplakaten
https://no.wikipedia.org/wiki/Eli_Arnstad
https://no.wikipedia.org/wiki/Elevrådsleder

Arrbee added a subscriber: Arrbee.Dec 10 2015, 10:47 AM

Screenshot provided by User:Sthelen.Aqua . She also mentions that after each time that she gets this error while publishing, whenever she tried a second time she could publish successfully.

Change 258960 had a related patch set uploaded (by Santhosh):
Display abusefilter errors nicely

https://gerrit.wikimedia.org/r/258960

Comment to User:SThelen.Aqua:
I am glad to hear that you get the translation published by clicking twice on the green (gray?) button.
For the record: I have tried to press the gray button twice (many times), and the only solid correspondence I find is "gray button after first paragraph translated = gray button no matter how many times I press or how much or little I translate".
Here is my last try, checked a minute ago, button staying gray:
https://no.wikipedia.org/wiki/Limonesisk_kreol

https://gerrit.wikimedia.org/r/#/c/258960/ improves the way the errors are displayed. Instead of JSON dumps that are very difficult to understand now, CX will display abusefilter errors just like the normal wikitext save action will display.

santhosh moved this task from Backlog to In Progress on the LE-CX7-Sprint 4 board.Dec 15 2015, 5:06 AM

Change 258960 merged by jenkins-bot:
Display abusefilter errors nicely

https://gerrit.wikimedia.org/r/258960

After a meeting today with Runa and Niklas, it turned out that I had an old .js page under my user page area (having to do with author counting of WP pages). After deleting it, all pages tested so far work. I will return with more info, but at present it seems that my bug (the grau button not turning green for nb->nn) is solved.

Arrbee raised the priority of this task from Normal to High.Jan 18 2016, 7:32 AM
Pginer-WMF added a comment.EditedJan 18 2016, 12:20 PM

I have explored some options, which depend on when can we discover the issues, how many issues can be discovered and how precise we can identify the problematic elements.

Here is an example of the general idea:

  • As soon a paragraph is adapted we highlight it is problematic (with a yellow bar next to it), and highlight the specific problematic element (in red).
  • A card will warn the user that problematic paragraphs were found, and provide the description of the issue (just one line that can be expanded if longer).


If the user tries to publish content with issues:

  • The content will be published by skipping the problematic parts.
  • A warning will indicate that the content was partially published (or not published if issues prevent any content to be published).
  • A warning card will inform of the total number of problematic paragraphs and let the user to move through them.

Feel free to provide feedback, especially about the constraints of surfacing ad identifying abusefilter errors.

Feel free to provide feedback, especially about the constraints of surfacing ad identifying abusefilter errors.

Attempting partial submissions seems an unwarranted complication of everyone's workflows and could easily be avoided by using the standard editing interface, see T114621#1836599.

Nemo_bis removed a subscriber: Nemo_bis.Jan 18 2016, 2:28 PM

Feel free to provide feedback, especially about the constraints of surfacing ad identifying abusefilter errors.

Attempting partial submissions seems an unwarranted complication of everyone's workflows and could easily be avoided by using the standard editing interface, see T114621#1836599.

Let's imagine the following scenario: a user adds several paragraphs to the translation, one of them has a link to a Youtube, which triggers the abuse filter of the target wiki. The user gets an error and needs to figure out which is the problematic link to remove it. If the user does not figure out which is the problem, the content cannot be published.

The publishing process is shared between the general editing tools and the translation-specific one. If the user was creating the article from scratch with a general editing tool (VE or Wikitext), she would hit the same abuse filter error. So it is not clear to me how this can be avoided by using the "standard user interface". Can you provide more details on how the user could avoid the abuse filter issues?

Nemo_bis removed a subscriber: Nemo_bis.Jan 18 2016, 6:24 PM

Interesting measurements about common causes producing abusefilter errors at T123911#1961228:

From Dec 12 2015 onwards, till today, there are 878 abusefilter errors
174 - spamblacklist - youtube, amazon and such links
109 - title blacklisted because of script issue, originating from non-latin wikis mostly
65- very short articles
30- edit protected pages

Pginer-WMF updated the task description. (Show Details)Jan 25 2016, 1:31 PM
Pginer-WMF updated the task description. (Show Details)Jan 26 2016, 10:55 AM

Another comment describing issues with Abusefilter.

From mid-sprint refinement meeting today:
ST: Main theme of this sprint is to deal with AbuseFilter
ST: PG has provided designs for handling AbuseFilter
ST: AA has provided statistics – many users are not able to proceed.
ST: NL pointed that there are warnings (ignorable) and errors (must fix before publishing)
ST: I added comments to T114621#1968507
PG: I was thinking more specific handling for some errors and more generic for others
PG: I am not sure if it possible to target specific cases (if they change a lot between wikis - which seems to be the case)
PG: Would it be possible to automatically retry publishing for AbuseFilter warning and then show warning to the user to ask to fix them?
ST: Technically that is possible, but community might become unhappy
PG: Then we can (showing from screen) allow users after showing a clear warning to click "Publish anyway".
PG: These two steps are connected together, but we can decide whether we can ask the user confirmation before or only show the warning afterwards
PG: These users have translated a lot, so that is for removing any blockers from publishing and letting them fix afterwards (like many users already like to do)
ST: There doesn't seem to be any human readable location to explain specific abuse filter messages.
PG: en:Wikipedia:Edit filter
ST: That page has versions in ~30 languages or so
NL: We could also write our own to customize to this use case
PG: There are pros and cons. Projects should already have this information.
ST: We can explain in more friendly way and give specific examples such as youtube.
PG: I will create a ticket where we can plan what that page should contain and see if it works out.
ST: Re multiple issues: AbuseFilter only returns the first issue it finds, not a list of all issues in the page
PG: That simplifies the design, but this might still apply in other situations (for example displaying messages when we already show that page currently exists).
PG: The main thing is to properly communicate errors - the issues of multiple simultaneous issues is something to watch for to see how big problem it is.
PG: For issues that get solved - the messages for those should be disappear immediately when fixed.
ST: Small question about the design, the colors: why were those colors chosen?
PG: It is often unclear what are the current guidelines. I am okay using the current colors currently in code (MediaWiki?) and have everything change at one go.
ST: Then I guess I don't need to worry too much about the colors.
NL: Suggesting that ST works on "frontend" issues and I can start figuring out how we can identify the problematic sections

For the record: a couple more examples of users experiencing issues with Abuse filter for Spanish and Portuguese.

santhosh moved this task from Backlog to In Progress on the LE-CX8-Sprint 1 board.Feb 2 2016, 6:32 AM
Arrbee assigned this task to santhosh.Feb 2 2016, 6:38 AM
Arrbee moved this task from Backlog to In Progress on the LE-CX8-Sprint 2 board.
Amire80 moved this task from Bugs to CX8 on the ContentTranslation board.Feb 14 2016, 4:32 PM
Arrbee moved this task from In Progress to Done on the LE-CX8-Sprint 2 board.Feb 16 2016, 6:48 AM
Arrbee moved this task from Done to In Progress on the LE-CX8-Sprint 2 board.

We have section validation code merged. Title validation code is not yet merged because it is mixed with parallel corpora refactoring, which we did not want put into train this week.

santhosh moved this task from Backlog to In Review on the LE-CX8-Sprint 3 board.Feb 16 2016, 6:58 AM
santhosh moved this task from In Review to In Progress on the LE-CX8-Sprint 3 board.

This ticket is to be used for tracking the data from abusefilter related events logged during the current sprint

The data being monitored are:

  1. Number of publishing failures due to abusefilter issues
  2. Recovery from failed errors and successful publishing of articles on retry
Amire80 moved this task from CX8 to Bugs on the ContentTranslation board.Apr 20 2016, 1:12 PM
He7d3r added a subscriber: He7d3r.Jun 2 2016, 11:36 AM
Arrbee moved this task from Bugs to Enhancements on the ContentTranslation board.Jun 26 2018, 6:27 PM