Changes to properties are only injected to client recent changes for pages with X aspect
Closed, InvalidPublic
Actions

Assigned To

None

Authored By

	matej_suchanek
	Dec 29 2017, 1:23 PM

Description

Today, I was running my bot over people in Czech Wikipedia to import data from categories. Most of them have Wikidata infobox, so they should be tracking the associated Wikidata item (with O aspect). However, when I look into recent changes and enable only Wikidata changes, I cannot see my bot. Interestingly, all articles that have some changes to properties track all aspects (X) of the entity. The injecting for O aspect must have been broken.

It's possible that this is just a temporary problem which may get solved with the new branch.

Event Timeline

matej_suchanek created this task.Dec 29 2017, 1:23 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 29 2017, 1:23 PM

One note is that we virtually disabled RC injection due to problems it caused for ruwiki and commonswiki so we inject one record and then we drop injecting. Missing RC records is an expected behavior but all pages using "O" aspects don't see any RC record, that sounds like a problem.
I did some queries and it seems some of the bot edits are indeed being injected into rc table of cswiki:

MariaDB [cswiki_p]> select * from recentchanges where rc_source = 'wb' limit 5;
+----------+----------------+---------+-------------------+--------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+--------+-----------+---------------+---------------+---------+-----------+--------------+-------+------------+------------+------------+----------+-------------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| rc_id    | rc_timestamp   | rc_user | rc_user_text      | rc_namespace | rc_title                       | rc_comment                                                                                                                                                                                                                               | rc_minor | rc_bot | rc_new | rc_cur_id | rc_this_oldid | rc_last_oldid | rc_type | rc_source | rc_patrolled | rc_ip | rc_old_len | rc_new_len | rc_deleted | rc_logid | rc_log_type | rc_log_action | rc_params                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+----------+----------------+---------+-------------------+--------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+--------+-----------+---------------+---------------+---------+-----------+--------------+-------+------------+------------+------------+----------+-------------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 32704253 | 20171129152034 |  393433 | Reinheitsgebot    |            0 | Eugene_Paul_Wigner             | /* wbsetreference-add:2| */ [[Property:P569]]: 17 November 1902, #quickstatements; invoked by Mix'n'match:References; /* wbsetreference-add:2| */ [[Property:P570]]: 1 January 1995, #quickstatements; invoked by Mix'n'match:References |        1 |      1 |      0 |    164737 |      15595845 |      15595845 |       5 | wb        |            1 |  NULL |      11476 |      11476 |          0 |        0 | NULL        |               | a:1:{s:20:"wikibase-repo-change";a:14:{s:2:"id";N;s:9:"object_id";s:7:"Q155790";s:4:"type";s:20:"wikibase-item~update";s:11:"revision_id";s:9:"600635524";s:7:"user_id";s:6:"940976";s:4:"time";s:14:"20171129152034";s:11:"entity_type";s:4:"item";s:7:"page_id";i:156958;s:9:"parent_id";i:590928337;s:7:"comment";s:114:"/* wbsetreference-add:2| */ [[Property:P570]]: 1 January 1995, #quickstatements; invoked by Mix'n'match:References";s:6:"rev_id";i:600635524;s:9:"user_text";s:14:"Reinheitsgebot";s:15:"central_user_id";i:31590885;s:3:"bot";i:1;}}      |
| 32704254 | 20171129152048 |  393433 | Reinheitsgebot    |            0 | Jean_Cocteau                   | /* wbsetreference-add:2| */ [[Property:P569]]: 5 July 1889, #quickstatements; invoked by Mix'n'match:References                                                                                                                          |        1 |      1 |      0 |     69018 |      15137268 |      15137268 |       5 | wb        |            1 |  NULL |       2334 |       2334 |          0 |        0 | NULL        |               | a:1:{s:20:"wikibase-repo-change";a:14:{s:2:"id";i:597527437;s:4:"time";s:14:"20171129152048";s:7:"user_id";s:6:"940976";s:11:"revision_id";s:9:"600635615";s:9:"object_id";s:6:"Q83158";s:4:"type";s:20:"wikibase-item~update";s:11:"entity_type";s:4:"item";s:7:"page_id";i:85561;s:9:"parent_id";i:593404289;s:7:"comment";s:111:"/* wbsetreference-add:2| */ [[Property:P569]]: 5 July 1889, #quickstatements; invoked by Mix'n'match:References";s:6:"rev_id";i:600635615;s:9:"user_text";s:14:"Reinheitsgebot";s:15:"central_user_id";i:31590885;s:3:"bot";i:1;}} |
| 32704255 | 20171129152057 |  393433 | Reinheitsgebot    |            0 | Adnan_Menderes                 | /* wbsetreference-add:2| */ [[Property:P569]]: 1899, #quickstatements; invoked by Mix'n'match:References                                                                                                                                 |        1 |      1 |      0 |   1274613 |      14780813 |      14780813 |       5 | wb        |            1 |  NULL |       4925 |       4925 |          0 |        0 | NULL        |               | a:1:{s:20:"wikibase-repo-change";a:14:{s:2:"id";i:597527491;s:4:"time";s:14:"20171129152057";s:7:"user_id";s:6:"940976";s:11:"revision_id";s:9:"600635670";s:9:"object_id";s:7:"Q238745";s:4:"type";s:20:"wikibase-item~update";s:11:"entity_type";s:4:"item";s:7:"page_id";i:232110;s:9:"parent_id";i:583044250;s:7:"comment";s:104:"/* wbsetreference-add:2| */ [[Property:P569]]: 1899, #quickstatements; invoked by Mix'n'match:References";s:6:"rev_id";i:600635670;s:9:"user_text";s:14:"Reinheitsgebot";s:15:"central_user_id";i:31590885;s:3:"bot";i:1;}}      |
| 32704256 | 20171129152108 |       0 | imported>Liamanka |            0 | José_Luis_Rodríguez_Zapatero   | Přidán mezijazykový odkaz: [[:zh-yue:薩帕特羅]]                                                                                                                                                                                          |        1 |      0 |      0 |    342089 |      15255602 |      15255602 |       5 | wb        |            1 |  NULL |       7887 |       7887 |          0 |        0 | NULL        |               | a:1:{s:20:"wikibase-repo-change";a:14:{s:2:"id";i:597527550;s:4:"time";s:14:"20171129152108";s:7:"user_id";s:7:"2902048";s:11:"revision_id";s:9:"600635729";s:9:"object_id";s:6:"Q41395";s:4:"type";s:20:"wikibase-item~update";s:11:"entity_type";s:4:"item";s:7:"page_id";i:43682;s:9:"parent_id";i:600160080;s:7:"comment";s:49:"/* wbsetsitelink-add:1|zh_yuewiki */ 薩帕特羅";s:6:"rev_id";i:600635729;s:9:"user_text";s:8:"Liamanka";s:15:"central_user_id";i:52437732;s:3:"bot";i:0;}}                                                                          |
| 32704257 | 20171129152109 |  393433 | Reinheitsgebot    |            0 | Jagdish_Bhagwati               | /* wbcreateclaim-create:1| */ [[Property:P570]]: 1980, #quickstatements; invoked by Mix'n'match:References; /* wbsetreference-add:2| */ [[Property:P570]]: 1980, #quickstatements; invoked by Mix'n'match:References                     |        1 |      1 |      0 |    940667 |      15226601 |      15226601 |       5 | wb        |            1 |  NULL |       1967 |       1967 |          0 |        0 | NULL        |               | a:1:{s:20:"wikibase-repo-change";a:14:{s:2:"id";N;s:9:"object_id";s:7:"Q559903";s:4:"type";s:20:"wikibase-item~update";s:11:"revision_id";s:9:"600635738";s:7:"user_id";s:6:"940976";s:4:"time";s:14:"20171129152109";s:11:"entity_type";s:4:"item";s:7:"page_id";i:527371;s:9:"parent_id";i:583893300;s:7:"comment";s:104:"/* wbsetreference-add:2| */ [[Property:P570]]: 1980, #quickstatements; invoked by Mix'n'match:References";s:6:"rev_id";i:600635738;s:9:"user_text";s:14:"Reinheitsgebot";s:15:"central_user_id";i:31590885;s:3:"bot";i:1;}}                |
+----------+----------------+---------+-------------------+--------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+--------+-----------+---------------+---------------+---------+-----------+--------------+-------+------------+------------+------------+----------+-------------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
5 rows in set (0.01 sec)

MariaDB [cswiki_p]> select * from recentchanges where rc_user_text = 'MatSuBot' and rc_source = 'wb' limit 5;
+----------+----------------+---------+--------------+--------------+----------------------+---------------------------------------------------------------------------+----------+--------+--------+-----------+---------------+---------------+---------+-----------+--------------+-------+------------+------------+------------+----------+-------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| rc_id    | rc_timestamp   | rc_user | rc_user_text | rc_namespace | rc_title             | rc_comment                                                                | rc_minor | rc_bot | rc_new | rc_cur_id | rc_this_oldid | rc_last_oldid | rc_type | rc_source | rc_patrolled | rc_ip | rc_old_len | rc_new_len | rc_deleted | rc_logid | rc_log_type | rc_log_action | rc_params                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+----------+----------------+---------+--------------+--------------+----------------------+---------------------------------------------------------------------------+----------+--------+--------+-----------+---------------+---------------+---------+-----------+--------------+-------+------------+------------+------------+----------+-------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 32751772 | 20171203125820 |  204676 | MatSuBot     |            0 | Souostroví           | /* wbsetclaimvalue:1| */ [[Property:P279]]: [[Q16887380]], fixed redirect |        1 |      1 |      0 |     31528 |      15393424 |      15393424 |       5 | wb        |            1 |  NULL |        847 |        847 |          0 |        0 | NULL        |               | a:1:{s:20:"wikibase-repo-change";a:14:{s:2:"id";i:599589522;s:4:"time";s:14:"20171203125820";s:7:"user_id";s:7:"1433337";s:11:"revision_id";s:9:"602708568";s:9:"object_id";s:6:"Q33837";s:4:"type";s:20:"wikibase-item~update";s:11:"entity_type";s:4:"item";s:7:"page_id";i:36659;s:9:"parent_id";i:592829310;s:7:"comment";s:73:"/* wbsetclaimvalue:1| */ [[Property:P279]]: [[Q16887380]], fixed redirect";s:6:"rev_id";i:602708568;s:9:"user_text";s:8:"MatSuBot";s:15:"central_user_id";i:15348099;s:3:"bot";i:1;}}   |
| 32751778 | 20171203125835 |  204676 | MatSuBot     |            0 | Množné_číslo         | /* wbsetclaimvalue:1| */ [[Property:P180]]: [[Q16887380]], fixed redirect |        1 |      1 |      0 |     23030 |      15392655 |      15392655 |       5 | wb        |            1 |  NULL |       1959 |       1959 |          0 |        0 | NULL        |               | a:1:{s:20:"wikibase-repo-change";a:14:{s:2:"id";i:599589610;s:4:"time";s:14:"20171203125835";s:7:"user_id";s:7:"1433337";s:11:"revision_id";s:9:"602708655";s:9:"object_id";s:7:"Q146786";s:4:"type";s:20:"wikibase-item~update";s:11:"entity_type";s:4:"item";s:7:"page_id";i:148388;s:9:"parent_id";i:596701241;s:7:"comment";s:73:"/* wbsetclaimvalue:1| */ [[Property:P180]]: [[Q16887380]], fixed redirect";s:6:"rev_id";i:602708655;s:9:"user_text";s:8:"MatSuBot";s:15:"central_user_id";i:15348099;s:3:"bot";i:1;}} |
| 32751779 | 20171203125839 |  204676 | MatSuBot     |            0 | Poklad               | /* wbsetclaimvalue:1| */ [[Property:P279]]: [[Q16887380]], fixed redirect |        1 |      1 |      0 |    623461 |      14417285 |      14417285 |       5 | wb        |            1 |  NULL |        922 |        922 |          0 |        0 | NULL        |               | a:1:{s:20:"wikibase-repo-change";a:14:{s:2:"id";i:599589635;s:4:"time";s:14:"20171203125839";s:7:"user_id";s:7:"1433337";s:11:"revision_id";s:9:"602708680";s:9:"object_id";s:7:"Q272937";s:4:"type";s:20:"wikibase-item~update";s:11:"entity_type";s:4:"item";s:7:"page_id";i:264076;s:9:"parent_id";i:574051220;s:7:"comment";s:73:"/* wbsetclaimvalue:1| */ [[Property:P279]]: [[Q16887380]], fixed redirect";s:6:"rev_id";i:602708680;s:9:"user_text";s:8:"MatSuBot";s:15:"central_user_id";i:15348099;s:3:"bot";i:1;}} |
| 32751803 | 20171203125850 |  204676 | MatSuBot     |            0 | Konvoj               | /* wbsetclaimvalue:1| */ [[Property:P279]]: [[Q16887380]], fixed redirect |        1 |      1 |      0 |     26036 |      15392911 |      15392911 |       5 | wb        |            1 |  NULL |       3692 |       3692 |          0 |        0 | NULL        |               | a:1:{s:20:"wikibase-repo-change";a:14:{s:2:"id";i:599589707;s:4:"time";s:14:"20171203125850";s:7:"user_id";s:7:"1433337";s:11:"revision_id";s:9:"602708752";s:9:"object_id";s:7:"Q370330";s:4:"type";s:20:"wikibase-item~update";s:11:"entity_type";s:4:"item";s:7:"page_id";i:352365;s:9:"parent_id";i:590421607;s:7:"comment";s:73:"/* wbsetclaimvalue:1| */ [[Property:P279]]: [[Q16887380]], fixed redirect";s:6:"rev_id";i:602708752;s:9:"user_text";s:8:"MatSuBot";s:15:"central_user_id";i:15348099;s:3:"bot";i:1;}} |
| 32751810 | 20171203125910 |  204676 | MatSuBot     |            0 | Vaubanova_opevnění   | /* wbsetclaimvalue:1| */ [[Property:P31]]: [[Q16887380]], fixed redirect  |        1 |      1 |      0 |    972924 |      15441218 |      15441218 |       5 | wb        |            1 |  NULL |       1914 |       1914 |          0 |        0 | NULL        |               | a:1:{s:20:"wikibase-repo-change";a:14:{s:2:"id";i:599589825;s:4:"time";s:14:"20171203125910";s:7:"user_id";s:7:"1433337";s:11:"revision_id";s:9:"602708871";s:9:"object_id";s:7:"Q833781";s:4:"type";s:20:"wikibase-item~update";s:11:"entity_type";s:4:"item";s:7:"page_id";i:786455;s:9:"parent_id";i:543854951;s:7:"comment";s:72:"/* wbsetclaimvalue:1| */ [[Property:P31]]: [[Q16887380]], fixed redirect";s:6:"rev_id";i:602708871;s:9:"user_text";s:8:"MatSuBot";s:15:"central_user_id";i:15348099;s:3:"bot";i:1;}}  |
+----------+----------------+---------+--------------+--------------+----------------------+---------------------------------------------------------------------------+----------+--------+--------+-----------+---------------+---------------+---------+-----------+--------------+-------+------------+------------+------------+----------+-------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
5 rows in set (0.47 sec)

This didn't get better with this week's new version. I can only see changes to statements when they are together with other changes, eg. changes to labels.

I will look into this in more depth when I'm back from vacation (next Monday)

The task is invalid. For example we enabled xkill for hewiki which means now there is no x aspects usages there at all but still the wiki has 20% of its rows injected from Wikidata (daily):

MariaDB [hewiki_p]> select left(rc_timestamp,8), sum(if(rc_source='wb',1,0)) as WB, count(*),sum(if(rc_source='wb',1,0))/count(*) as RATIO from recentchanges where rc_bot = 0 group by left(rc_timestamp,8);
+----------------------+------+----------+--------+
| left(rc_timestamp,8) | WB   | count(*) | RATIO  |
+----------------------+------+----------+--------+
| 20180120             |  855 |     4337 | 0.1971 |
| 20180121             | 1939 |     6996 | 0.2772 |
| 20180122             | 4083 |     9665 | 0.4225 |
| 20180123             | 1961 |     7237 | 0.2710 |
| 20180124             | 1715 |     7555 | 0.2270 |
| 20180125             | 1803 |     7329 | 0.2460 |
| 20180126             | 1601 |     7099 | 0.2255 |
| 20180127             | 1868 |     7145 | 0.2614 |
| 20180128             | 2262 |     8869 | 0.2550 |
| 20180129             | 3680 |     9262 | 0.3973 |
| 20180130             | 1897 |     8780 | 0.2161 |
| 20180131             | 1655 |     9342 | 0.1772 |
| 20180201             | 1612 |     8753 | 0.1842 |
| 20180202             | 1713 |     7270 | 0.2356 |
| 20180203             | 1630 |     7648 | 0.2131 |
| 20180204             | 1623 |     9053 | 0.1793 |
| 20180205             | 1426 |     7397 | 0.1928 |
| 20180206             | 1641 |     8030 | 0.2044 |
| 20180207             |  953 |     7327 | 0.1301 |
| 20180208             | 1556 |     7645 | 0.2035 |
| 20180209             | 1597 |     8049 | 0.1984 |
| 20180210             | 1672 |     8119 | 0.2059 |
| 20180211             | 1598 |     7916 | 0.2019 |
| 20180212             | 1641 |     8596 | 0.1909 |
| 20180213             | 1667 |     8063 | 0.2067 |
| 20180214             | 1540 |     7503 | 0.2053 |
| 20180215             | 1900 |     8673 | 0.2191 |
| 20180216             | 1447 |     7301 | 0.1982 |
| 20180217             | 1731 |     8041 | 0.2153 |
| 20180218             | 1614 |     8586 | 0.1880 |
| 20180219             |  600 |     3483 | 0.1723 |
+----------------------+------+----------+--------+
31 rows in set (13.12 sec)

As I explained above, we don't inject rc records too much and drop it after first batch in order to avoid database explosion when there is bad lua.

So I looked into hewiki's recent changes and can see the same problem: no changes to statements unless sent together with other changes.

I admit I didn't bring any examples. So randomly on cswiki:

Wikidata change, cswiki registered, O aspect tracked, the date of death from Wikidata is shown in the article but the change is niether in recent changes nor watchlist
Wikidata changes, cswiki registered, O aspect tracked, the employer is shown in the infobox but none of those changes can be found in recent changes or watchlist
Wikidata change, cswiki registered, O aspect tracked, the father is shown in the infobox but the change is not shown in the recent changes and watchlist

I believe this is a strong evidence.

As I explained above, we don't inject rc records too much and drop it after first batch in order to avoid database explosion when there is bad lua.

I hope sending a change to a single page on a wiki is not "too much".

Changes to properties are only injected to client recent changes for pages with X aspectClosed, InvalidPublicActions

Description

Event Timeline

Changes to properties are only injected to client recent changes for pages with X aspect
Closed, InvalidPublic
Actions