Page MenuHomePhabricator

It is possible to construct a list of all globally suppressed users
Closed, ResolvedPublicSecurity

Description

This is more of a proof of concept. I never automated this.

Steps to reproduce
  1. Find a newuser log entry of a globally suppressed user.

There are probably multiple ways of doing this. One way is to use the log on loginwiki:

MariaDB [loginwiki_p]> select * from logging where log_deleted>7 order by log_id desc limit 10;
+----------+----------+------------+----------------+-----------+---------------+-----------+----------------+------------+-------------+----------+
| log_id   | log_type | log_action | log_timestamp  | log_actor | log_namespace | log_title | log_comment_id | log_params | log_deleted | log_page |
+----------+----------+------------+----------------+-----------+---------------+-----------+----------------+------------+-------------+----------+
| 33790398 | newusers | NULL       | 20211005205654 |         0 |          NULL | NULL      |             11 | NULL       |          13 |     NULL |
| 33789838 | newusers | NULL       | 20211005194103 |         0 |          NULL | NULL      |             11 | NULL       |          13 |     NULL |
| 33780782 | newusers | NULL       | 20211004221045 |         0 |          NULL | NULL      |             11 | NULL       |          13 |     NULL |
| 33723340 | newusers | NULL       | 20210928131321 |         0 |          NULL | NULL      |             11 | NULL       |          13 |     NULL |
| 33718768 | newusers | NULL       | 20210928011022 |         0 |          NULL | NULL      |             11 | NULL       |          13 |     NULL |
| 33702716 | newusers | NULL       | 20210926071222 |         0 |          NULL | NULL      |             11 | NULL       |          13 |     NULL |
| 33700891 | newusers | NULL       | 20210925233523 |         0 |          NULL | NULL      |             11 | NULL       |          13 |     NULL |
| 33648877 | newusers | NULL       | 20210920081317 |         0 |          NULL | NULL      |             11 | NULL       |          13 |     NULL |
| 33619486 | newusers | NULL       | 20210916190637 |         0 |          NULL | NULL      |             11 | NULL       |          13 |     NULL |
| 33619478 | newusers | NULL       | 20210916190554 |         0 |          NULL | NULL      |             11 | NULL       |          13 |     NULL |
+----------+----------+------------+----------------+-----------+---------------+-----------+----------------+------------+-------------+----------+
10 rows in set (0.32 sec)

MariaDB [loginwiki_p]>

Let's choose 33619486 for this example.

  1. Find the global user id of that suppressed user.

We need to trick a little bit here. A way of doing this is to first get the context of that log entry. What I mean is basically to find out which user got created before and/or after that globally suppressed user. For this this example:

MariaDB [loginwiki_p]> select * from logging where log_id between 33619481 and 33619491 order by log_id desc;
+----------+----------+------------+----------------+-----------+---------------+----------------+----------------+-----------------------------------+-------------+----------+
| log_id   | log_type | log_action | log_timestamp  | log_actor | log_namespace | log_title      | log_comment_id | log_params                        | log_deleted | log_page |
+----------+----------+------------+----------------+-----------+---------------+----------------+----------------+-----------------------------------+-------------+----------+
| 33619491 | newusers | autocreate | 20210916190717 |  34171769 |             2 | Estebanibou    |             11 | a:1:{s:9:"4::userid";i:71853295;} |           0 |        0 |
| 33619490 | newusers | autocreate | 20210916190659 |  34171768 |             2 | SmartINFS315   |             11 | a:1:{s:9:"4::userid";i:71853288;} |           0 |        0 |
| 33619489 | newusers | autocreate | 20210916190648 |  34171767 |             2 | Jazzfan15      |             11 | a:1:{s:9:"4::userid";i:71853287;} |           0 |        0 |
| 33619488 | newusers | autocreate | 20210916190647 |  34171766 |             2 | Jgolf35        |             11 | a:1:{s:9:"4::userid";i:71853286;} |           0 |        0 |
| 33619487 | newusers | autocreate | 20210916190640 |  34171765 |             2 | MStearns21     |             11 | a:1:{s:9:"4::userid";i:71853285;} |           0 |        0 |
| 33619486 | newusers | NULL       | 20210916190637 |         0 |          NULL | NULL           |             11 | NULL                              |          13 |     NULL |
| 33619485 | newusers | autocreate | 20210916190637 |  34171763 |             2 | Vip_kuatrai    |             11 | a:1:{s:9:"4::userid";i:71853282;} |           0 |        0 |
| 33619484 | newusers | autocreate | 20210916190637 |  34171762 |             2 | Informacionvip |             11 | a:1:{s:9:"4::userid";i:71853277;} |           0 |        0 |
| 33619483 | newusers | autocreate | 20210916190636 |  34171761 |             2 | Deedo7         |             11 | a:1:{s:9:"4::userid";i:71853272;} |           0 |        0 |
| 33619482 | newusers | autocreate | 20210916190630 |  34171760 |             2 | Bluehairedb7   |             11 | a:1:{s:9:"4::userid";i:71853267;} |           0 |        0 |
| 33619481 | newusers | autocreate | 20210916190629 |  34171759 |             2 | Efe_Çelebi     |             11 | a:1:{s:9:"4::userid";i:71853266;} |           0 |        0 |
+----------+----------+------------+----------------+-----------+---------------+----------------+----------------+-----------------------------------+-------------+----------+
11 rows in set (0.00 sec)

MariaDB [loginwiki_p]>

The globally suppressed user is clearly visible in the middle. We know take the user that has been created before the globally suppressed user has been created, Vip kuatrai. Now we abuse the fact that global user ids are incremental (Note, however, that we have several wikis and so it will not be the exact same order as in the loginwiki), so the global user id of the globally suppressed user should be within a small range of the global user id of Vip kuatrai. Let's get the global user id of Vip kuatrai.

MariaDB [centralauth_p]> select gu_id, gu_name from globaluser where gu_name="Vip kuatrai";
+----------+-------------+
| gu_id    | gu_name     |
+----------+-------------+
| 67601590 | Vip kuatrai |
+----------+-------------+
1 row in set (0.01 sec)

MariaDB [centralauth_p]>

Now let's take a look of a range of global user ids around 67601590.

MariaDB [centralauth_p]> select gu_id, gu_name from globaluser where gu_id between 67601585 and 67601595;
+----------+----------------------------+
| gu_id    | gu_name                    |
+----------+----------------------------+
| 67601585 | عمر تقي الرسام             |
| 67601586 | Alebrex99                  |
| 67601587 | Bluehairedb7               |
| 67601588 | Efe Çelebi                 |           <------- Note that 67601589 is missing
| 67601590 | Vip kuatrai                |
| 67601591 | Deedo7                     |
| 67601592 | Informacionvip             |
| 67601593 | MStearns21                 |
| 67601594 | Jazzfan15                  |
| 67601595 | Jgolf35                    |
+----------+----------------------------+
10 rows in set (0.00 sec)

MariaDB [centralauth_p]>

We can see that 67601589 must be the global user id of the globally suppressed user since it is missing here.

  1. Find out the username of the globally suppressed user from the global user id (the actual issue here)
MariaDB [centralauth_p]> select * from localuser where lu_global_id=67601589;
+------------+------------+-----------------------+--------------------+-------------+--------------+
| lu_wiki    | lu_name    | lu_attached_timestamp | lu_attached_method | lu_local_id | lu_global_id |
+------------+------------+-----------------------+--------------------+-------------+--------------+
| enwiki     | <redacted> | 20210916190628        | new                |    42513886 |     67601589 |
| loginwiki  | <redacted> | 20210916190637        | login              |    71853284 |     67601589 |
| metawiki   | <redacted> | 20210916190640        | login              |    33138860 |     67601589 |
| simplewiki | <redacted> | 20210916191633        | login              |     1137097 |     67601589 |
+------------+------------+-----------------------+--------------------+-------------+--------------+
4 rows in set (1 min 13.33 sec)

MariaDB [centralauth_p]>

The <redacted> here is placed by me in order to prevent this task getting PermanentlyPrivate. I'm going to publish the query result in a permanently private counterpart of this task (T292595).

  1. Repeat the same thing for all other globally suppressed users. (In theory to get all of them an attacker would need to go to other wikis aswell). I think this can be automated, which makes this kinda scary.

Details

Risk Rating
Informational
Author Affiliation
Wikimedia Communities

Event Timeline

Let us post all private stuff to {T292595} in order to avoid this task getting permanently private.

This issue itself lies in the wiki replicas of the cloud services.

So basically the localuser table should be redacted?
The specific user in this case doesn't appear to have been lock+suppressed, just lock+hide, since they still show up as having contributions on enwiki. That being said, there does not currently appear to be any filtering on the localuser table in the replicas, https://gerrit.wikimedia.org/g/operations/puppet/+/4efea44df3994d25a2132e0823e3d3e4b4c65ec7/modules/profile/templates/wmcs/db/wikireplicas/maintain-views.yaml#151, so yeah, this could definitely be used to leak suppressed users too

Maybe instead filter it to where the relevant global user isn't hidden? Though that doesn't address users hidden locally by a wiki, that would be almost impossible to do since it would require checking each wiki for the view. Proposed view:

localuser:
  source: localuser
  source:
    - localuser
    - globaluser
  view: >
    select lu_wiki, lu_name, lu_attached_timestamp, lu_attached_method, lu_local_id, lu_global_id
  where: lu_global_id = gu_id AND gu_hidden=''

Also, much simpler attack vector to just search based on users already filtered out from the globaluser table due to being hidden:

MariaDB [centralauth_p]> SELECT lu_wiki, lu_name, lu_local_id, lu_global_id FROM localuser WHERE NOT EXISTS (SELECT 1 FROM globaluser WHERE gu_id = lu_global_id) LIMIT 5;

Works to show some suppressed users unfortunately and is a fairly trivial query to run. Escalating priority to high

The specific user in this case doesn't appear to have been lock+suppressed, just lock+hide, since they still show up as having contributions on enwiki.

It appears to be suppressed, lock+hide does not suppress the user creation log at loginwiki. I think the fact that the constributions show up at enwiki is an occurence of T25310, you can see the user has also been blocked locally at enwiki when you take a look at Special:BlockList page for that user.

I guess this is kind of a part of T169097, which has been going on for a long time now.

Proposed view:

localuser:
  source: localuser
  source:
    - localuser
    - globaluser
  view: >
    select lu_wiki, lu_name, lu_attached_timestamp, lu_attached_method, lu_local_id, lu_global_id
  where: lu_global_id = gu_id AND gu_hidden=''

This makes sense at a quick glance, but we've gone with this where clause in the past to check for hidden and/or suppressed users, though I know that's a slightly different concept than being globally-hidden.

This makes sense at a quick glance, but we've gone with this where clause in the past to check for hidden and/or suppressed users, though I know that's a slightly different concept than being globally-hidden.

This query is in the centralauth database so we can't check for local blocks (cross-db queries). gu_hidden is the best thing we have for global users, but I suspect there may be some weird edge cases with the localuser table.

In T292594#7412586, @Majavah wrote:

This query is in the centralauth database so we can't check for local blocks (cross-db queries). gu_hidden is the best thing we have for global users, but I suspect there may be some weird edge cases with the localuser table.

Yes, I wasn't sure though, given the way maintain-views seems to work, if it were possible to do synthetic cross-db joins via the sql syntax within the config file used to build other views. I don't really see any existing views doing that, so maybe not.

Yes, I wasn't sure though, given the way maintain-views seems to work, if it were possible to do synthetic cross-db joins via the sql syntax within the config file used to build other views. I don't really see any existing views doing that, so maybe not.

In theory it would be possible for s7 wikis. It's not for wikis on other sections since the last redesign where each section was given its own separate MariaDB instances.

localuser:
  source:
    - localuser
    - globaluser
  view: >
    select lu_wiki, lu_name, lu_attached_timestamp, lu_attached_method, lu_local_id, lu_global_id
  where: lu_global_id = gu_id AND gu_hidden=''

Will submit a patch shortly, in line with the above.

The patch looks good and was submitted; while deploying we ran into a lock error on clouddb1014:

2021-11-24 19:40:25,189 INFO [fiwiki_p.imagelinks]
Traceback (most recent call last):
  File "/usr/local/sbin/maintain-views", line 719, in <module>
    sys.exit(main())
  File "/usr/local/sbin/maintain-views", line 707, in main
    all_tables,
  File "/usr/local/sbin/maintain-views", line 507, in dbrun
    ops.execute(fullviews, customviews)
  File "/usr/local/sbin/maintain-views", line 392, in execute
    self.do_fullview(view)
  File "/usr/local/sbin/maintain-views", line 138, in do_fullview
    """
  File "/usr/local/sbin/maintain-views", line 55, in write_execute
    self.cursor.execute(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.InternalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

So the patch is still waiting to be deployed via maintain-views. I'll try deploying it again next week.

razzi added a subscriber: Andrew.

At cloud services / data engineering sync, @Andrew said he finished the deploy. This can probably be closed.

Yes, looks good.

MariaDB [centralauth_p]> SELECT lu_wiki, lu_name, lu_local_id, lu_global_id FROM localuser WHERE NOT EXISTS (SELECT 1 FROM globaluser WHERE gu_id = lu_global_id) LIMIT 5;
Empty set (9 min 31.87 sec)

MariaDB [centralauth_p]>

@sbassett Could you evaluate whether it is fine to make this public?

sbassett added a project: Security-Team.

@sbassett Could you evaluate whether it is fine to make this public?

I'll re-triage for the Security-Team clinic this Monday. I think it's likely fine to make public as-is, but just want to double-check with Privacy and IR folks.

sbassett changed the visibility from "Custom Policy" to "Public (No Login Required)".
sbassett changed the edit policy from "Custom Policy" to "All Users".
sbassett changed Risk Rating from N/A to Informational.
sbassett moved this task from Watching to Our Part Is Done on the Security-Team board.