Page MenuHomePhabricator

Notification emails repeated every day
Closed, ResolvedPublic

Description

Users receive the same notification lots of times over a longer period, at a rate of about one per day. The notification is about an old edit (often several years old, although always the last edit to the page), with no obvious reason for being triggered now. Seems specific to plwiki.

T257714: User is receiving the same Echo web notification over and over again is a similar but possibly distinct issue - a notification about a current event being repeated several times, in rapid succession.

Some people report only getting these spurious notifications via email (despite having configured both types) and seeing them as already marked read in Special:Notifications, others seem to get the web notifications as well. Multiple notification types (talk page message, mention), multiple edit types (revert, page move). Marking some of the notifications as read doesn't stop new ones from being triggered.

Some examples:

reporter@slestatyJD (via @MBq)Igi^MB (via @Tufor)Koreanovsky (via @Tufor)Hektor Absurdus (via @Tufor)
editdiffdiffdiffdiffdiff
screenshot
plwiki.png (768×806 px, 44 KB)
Pl.wp.nerv.jpg (704×497 px, 94 KB)
notesemails only

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
This comment was removed by slestaty.

Today (again) a few minutes ago...

Delivered-To: oversightedmail@gmail.com
Received: by 2002:ab3:5d02:0:0:0:0:0 with SMTP id j2csp3205290ltc;
        Mon, 13 Jul 2020 17:15:07 -0700 (PDT)
X-Received: by 2002:a17:906:a253:: with SMTP id bi19mr2025243ejb.338.1594685707426;
        Mon, 13 Jul 2020 17:15:07 -0700 (PDT)
ARC-Seal: i=2; a=rsa-sha256; t=1594685707; cv=pass;
        d=google.com; s=arc-20160816;
        b=QNmLDfK8eiyQgrnBdRWU3SovCrBZkrot0woMNVjFm8ggPfXbRgpyVOy/IPPCs09Sft
         Ff+SyMn4xuolxVYt3+IBNrYap8PBJFg4DsgEmYKgG1i1k6FKdD9duSZkkXG8jyhvmZnQ
         W0yusTEcEJ+CY3H/gXx35R3ar3P8bxf1wO1MJT8bg+VC2znwen6pch5Axw/taFi1Xa4I
         wAYIHZ6lGhyvtI9cIDPRUlPBKPmpCdfEipb08rO8f1M3SF5FLZ/kdvqdzqQeqo8Rg+oS
         76IcwahoUKuFv48jMvcRZDW7oOIT5MSFDAuci9WoQ6e/nimlnlGtTnai7b0Po/yZwqXT
         3jig==
ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=list-unsubscribe:message-id:date:reply-to:from:mime-version:subject
         :to:dkim-signature:delivered-to;
        bh=RN0mXGdS7mYSKcXI7shtZvkJAYUj0Idf53nrKVjbPBM=;
        b=wYyCB40PDAAHYng4LqDO1LgydviexPEMhZ/KGW7w592AvVQxkaTSW2+mvjNnUNsbPc
         18MGRBtk4uhUoTbI9yiTmpz2mTMgkuHPKEzoyeBMNFm36AKAM1un4Xu7wKfJqS3mbnXW
         9osEdQcNWGOBcwefI8SsPL2EeVz7zIsNSGfGJ6aE86EHnJf6BIEEevsoFh+Phpi2A1k7
         UmLbV45ZUUFOdinfMMtiQd+txuiIf0sId7CJ1kBJChggnXgRcNWg6l5sGPbArk+1jdwm
         aFhpS4zEoaceddfxFJmFGXcBBR1oVWOfwt7o03nkPOnhbCuR/disL2viJWZRYbH9g0dI
         7VbA==
ARC-Authentication-Results: i=2; mx.google.com;
       dkim=pass header.i=@wikimedia.org header.s=wiki-mail header.b=BZEVzhW5;
       arc=pass (i=1 spf=pass spfdomain=wikimedia.org dkim=pass dkdomain=wikimedia.org dmarc=pass fromdomain=wikimedia.org);
       spf=pass (google.com: domain of oversightedmail@gmail.com designates 209.85.220.41 as permitted sender) smtp.mailfrom="oversightedmail@gmail.com";
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=wikimedia.org
Return-Path: <oversightedmail@gmail.com>
Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41])
        by mx.google.com with SMTPS id gq21sor9040121ejb.75.2020.07.13.17.15.07
        for <oversightedmail@gmail.com>
        (Google Transport Security);
        Mon, 13 Jul 2020 17:15:07 -0700 (PDT)
Received-SPF: pass (google.com: domain of oversightedmail@gmail.com designates 209.85.220.41 as permitted sender) client-ip=209.85.220.41;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@wikimedia.org header.s=wiki-mail header.b=BZEVzhW5;
       arc=pass (i=1 spf=pass spfdomain=wikimedia.org dkim=pass dkdomain=wikimedia.org dmarc=pass fromdomain=wikimedia.org);
       spf=pass (google.com: domain of oversightedmail@gmail.com designates 209.85.220.41 as permitted sender) smtp.mailfrom="oversightedmail@gmail.com";
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=wikimedia.org
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:delivered-to:dkim-signature:to:subject
         :mime-version:from:reply-to:date:message-id:list-unsubscribe;
        bh=RN0mXGdS7mYSKcXI7shtZvkJAYUj0Idf53nrKVjbPBM=;
        b=HdtJPxm/Lhj2EuL6EzRzTCggQL4gmMEw7I/NDE8X9i3w8c6oerpY8smwGbHlpkOPnN
         NgPXKZ2Jhsxct+NIjPMYs+oo5a/xGX40e9OkiTq0rNhGuxh13LeWRHDdgafjwNZPGHoM
         EHN+VLbNEE05iYjAXDrRYhVGKX2uGpevrfdfcZr2fh3dpTUHyoqZIZh5NCPnwxn1iW8i
         X7b0oFsPKu4KyKIpJX5OIfl5IVQmNL3DAmzBcRDZTMMU7/o3E2pDLP3wxW19LRvKUE//
         gBbW4Vnq2kgtWzh+inZvcJCJelch/SiaqRUguWIT9HaNWA5WkO/W5HQPP8bg+hvjowAe
         vcIQ==
X-Gm-Message-State: AOAM5337y2fwDN/sFqlHQ64/I82ie+CnQrcsT5xLbLvvq3/j9ommSCNw GUkAA4Rf2PGFETNDjgG9qKvOTCc52W08uHsqim2hRS7SVMaQUrQ=
X-Received: by 2002:a17:906:2cd5:: with SMTP id r21mr2064238ejr.20.1594685707017;
        Mon, 13 Jul 2020 17:15:07 -0700 (PDT)
X-Forwarded-To: oversightedmail@gmail.com
X-Forwarded-For: oversightedmail@gmail.com oversightedmail@gmail.com
Delivered-To: oversightedmail@gmail.com
Received: by 2002:aa7:c143:0:0:0:0:0 with SMTP id r3csp3158116edp;
        Mon, 13 Jul 2020 17:15:05 -0700 (PDT)
X-Google-Smtp-Source: ABdhPJxYTLFDWJ5Aqfj9mRxCA20iCfYPTgpLJPSTuxOoAvmmxWiZBcDbcsFn/omPAbd7h7Jpowa+
X-Received: by 2002:ad4:57b2:: with SMTP id g18mr1922686qvx.207.1594685704862;
        Mon, 13 Jul 2020 17:15:04 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1594685704; cv=none;
        d=google.com; s=arc-20160816;
        b=Jcd2EsC9ihMleTNtiXfEk/+jwYNK3CaRkf7Aoy8WdGSixocBa/lVHluS+QQrHoZUdZ
         PfqIezSZ8u6gIACtDW8LYcEJ4oVNoxob1vyCnw8sAsPOGOrzMzJRhA52N/tKSwb6Q8nt
         O/jrwMdmiuFpz4haMEZrwq1Nvns4UGdWvAzsArvyx+9oMxtEECYdGN1NTFlmvG0tkpCf
         vKLvBoegnOV+Tte8lgEIqccEVB2ND2TeVvfjDzrMQK7w28JTGx4rjG30PMP+oM+R36fV
         bqbT3fksykRFAmKZibwrGfocNQZ4/cvf+U73ihDu/fHGLPmAI57oeGaExvXpZFVmDc6Q
         yOpg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=list-unsubscribe:message-id:date:reply-to:from:mime-version:subject
         :to:dkim-signature;
        bh=RN0mXGdS7mYSKcXI7shtZvkJAYUj0Idf53nrKVjbPBM=;
        b=rcBwO+LOoGdbkptykpmxSM2sI5LMaCcI89WOD5yArtzymQzhqMy3eqw6PVOFHQRzS2
         Ac4LgzJAtaRNWGpe4QVC6GWwt7C5RNK6Q+swkW/APKOmhR3ItWRQpNwZDGius51v0ph8
         RgG66WM7k3fr3BIy6hWDUH9ZPt6NjMbWntw4W7cX4RG+JMdIbHDFDnVhCNdockZtAbfh
         nNOBVXZ3d35OiSEZFS/zOmX8QQFr2JMRSmZcn5sYaiXfcVNnNYlrAcz9p49NAjUSNZfA
         9WgkpXJIvWGC9npgciKlZvwVip9Qpyn2VfWtiO4qsXzRrF5Z0zAQ7ZL5TMmgwAE2sgHu
         +ihg==
ARC-Authentication-Results: i=1; mx.google.com;
       dkim=pass header.i=@wikimedia.org header.s=wiki-mail header.b=BZEVzhW5;
       spf=pass (google.com: domain of wiki-plwiki-3q1g-qdfnd4-ud1zfk4agoiv+kqi@wikimedia.org designates 208.80.154.91 as permitted sender) smtp.mailfrom=wiki-plwiki-3q1g-qdfnd4-uD1zfK4AGOiV+kqI@wikimedia.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=wikimedia.org
Return-Path: <wiki-plwiki-3q1g-qdfnd4-uD1zfK4AGOiV+kqI@wikimedia.org>
Received: from wiki-mail-eqiad.wikimedia.org (wiki-mail-eqiad.wikimedia.org. [208.80.154.91])
        by mx.google.com with ESMTPS id u6si10019541qvy.87.2020.07.13.17.15.04
        for <oversightedmail@gmail.com>
        (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256);
        Mon, 13 Jul 2020 17:15:04 -0700 (PDT)
Received-SPF: pass (google.com: domain of wiki-plwiki-3q1g-qdfnd4-ud1zfk4agoiv+kqi@wikimedia.org designates 208.80.154.91 as permitted sender) client-ip=208.80.154.91;
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=wikimedia.org; s=wiki-mail; h=Content-Type:List-Unsubscribe:Message-ID:Date :Reply-To:From:MIME-Version:Subject:To:Sender:Cc:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Subscribe:List-Post:List-Owner:List-Archive; bh=RN0mXGdS7mYSKcXI7shtZvkJAYUj0Idf53nrKVjbPBM=; b=BZEVzhW50WglQPIQNoTmAPSMPM 2n1C+zBj3RqwNNFm68HEb42/JkOi3UDkhDyXz0Ro3vrr6hmk1sWKWvdpOWkEG6ZpYm9D3IgIpAP2i mza5uM7RwbsXCNxfOvi3zWf8z7Tmz6TUmzFVi9uMT6AaZesCMPeaQHjNhMlPmRFGI7Xg=;
Received: from [2620:0:861:107:10:64:48:224] (port=60518 helo=mw1382.eqiad.wmnet) by mx1001.wikimedia.org with esmtp (Exim 4.89) (envelope-from <wiki-plwiki-3q1g-qdfnd4-uD1zfK4AGOiV+kqI@wikimedia.org>) id 1jv8bA-0003yW-Cd for oversightedmail@gmail.com; Tue, 14 Jul 2020 00:15:04 +0000
Received: from www-data by mw1382.eqiad.wmnet with local (Exim 4.89) (envelope-from <wiki-plwiki-3q1g-qdfnd4-uD1zfK4AGOiV+kqI@wikimedia.org>) id 1jv8bA-000sRe-Bo for oversightedmail@gmail.com; Tue, 14 Jul 2020 00:15:04 +0000
To: Sir Lestaty de Lioncourt <oversightedmail@gmail.com>
Subject: ‪Pathoschild‬ deixou uma mensagem para você em Wikipedia
MIME-Version: 1.0
From: Wikipedia <wiki@wikimedia.org>
Reply-To: wiki@wikimedia.org
Date: Tue, 14 Jul 2020 02:15:04 +0200
Message-ID: <plwiki.5f0cf90854d7b8.41366174@pl.wikipedia.org>
X-Mailer: MediaWiki mailer
List-Unsubscribe: <https://pl.wikipedia.org/wiki/Specjalna:Preferencje>
Content-Type: multipart/alternative; boundary="=_90658cc273d0f134e0e5038bb0ff92fb"

--=_90658cc273d0f134e0e5038bb0ff92fb
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8

=E2=80=AAPathoschild=E2=80=AC deixou uma mensagem na sua p=C3=A1gina de dis=
cuss=C3=A3o.



Ver mensagem:  <https://pl.wikipedia.org/wiki/Dyskusja_wikipedysty:Sir_Lest=
aty_de_Lioncourt?markasread=3D17548839&markasreadwiki=3Dplwiki>

=E2=80=AAPathoschild=E2=80=AC:  <https://pl.wikipedia.org/wiki/Wikipedysta:=
Pathoschild>

Ver mudan=C3=A7as:  <https://pl.wikipedia.org/w/index.php?title=3DDyskusja_=
wikipedysty:Sir_Lestaty_de_Lioncourt&oldid=3Dprev&diff=3D41495542>

--

Para controlar quais e-mails enviamos para voc=C3=AA, verifique suas prefer=
=C3=AAncias:
https://pl.wikipedia.org/wiki/Specjalna:Preferencje#mw-prefsection-echo

Wikimedia Foundation, 1 Montgomery Street, Suite 1600, San Francisco, CA 94=
104, USA
--=_90658cc273d0f134e0e5038bb0ff92fb
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8

<html><head>
=09<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-8"=
 />
        <meta name=3D"viewport" content=3D"width=3Ddevice-width, initial-sc=
ale=3D1.0" />
=09<style>
=09=09@media only screen and (max-width: 480px){
=09=09=09table[id=3D"email-container"]{max-width:600px !important; width:10=
0% !important;}
=09=09}
=09</style>
=09<base href=3D"https://pl.wikipedia.org">
</head><body>
<table cellspacing=3D"0" cellpadding=3D"0" border=3D"0" width=3D"100%" alig=
n=3D"center" lang=3D"pt-BR" dir=3D"ltr">
<tr>
=09<td bgcolor=3D"#EAECF0"><center>
=09=09<br /><br />
=09=09<table cellspacing=3D"0" cellpadding=3D"0" border=3D"0" width=3D"600"=
 id=3D"email-container">
=09=09=09<tr>
=09=09=09=09<td bgcolor=3D"#FFFFFF" width=3D"5%">&nbsp;</td>
=09=09=09=09<td bgcolor=3D"#FFFFFF" width=3D"10%">&nbsp;</td>
=09=09=09=09<td bgcolor=3D"#FFFFFF" width=3D"80%" style=3D"line-height:40px=
;">&nbsp;</td>
=09=09=09=09<td bgcolor=3D"#FFFFFF" width=3D"5%">&nbsp;</td>
=09=09=09</tr><tr>
=09=09=09=09<td bgcolor=3D"#FFFFFF" rowspan=3D"2">&nbsp;</td>
=09=09=09=09<td bgcolor=3D"#FFFFFF" align=3D"center" valign=3D"top" rowspan=
=3D"2"><img src=3D"https://pl.wikipedia.org/w/load.php?modules=3Dext.echo.e=
mailicons&amp;image=3Dedit-user-talk&amp;lang=3Dpt-br&amp;format=3Drasteriz=
ed" alt=3D"" height=3D"30" width=3D"30"></td>
=09=09=09=09<td bgcolor=3D"#FFFFFF" align=3D"left" style=3D"font-family: Ar=
ial, Helvetica, sans-serif; font-size:13px; line-height:20px; color:#72777D=
;">=E2=80=AAPathoschild=E2=80=AC deixou uma mensagem na <b>sua p=C3=A1gina =
de discuss=C3=A3o</b>.</td>
=09=09=09=09<td bgcolor=3D"#FFFFFF" rowspan=3D"2">&nbsp;</td>
=09=09=09</tr><tr>
=09=09=09=09<td bgcolor=3D"#FFFFFF" align=3D"left" style=3D"font-family: Ar=
ial, Helvetica, sans-serif; line-height: 20px; font-weight: 600;">
=09=09=09=09=09<table cellspacing=3D"0" cellpadding=3D"0" border=3D"0">
=09=09=09=09=09=09<tr>
=09=09=09=09=09=09=09<td bgcolor=3D"#FFFFFF" align=3D"left" style=3D"font-f=
amily: Arial, Helvetica, sans-serif; padding-top: 8px; font-size:13px; font=
-weight: bold; color: #54595D;">
=09=09=09=09=09=09=09=09
=09=09=09=09=09=09=09</td>
=09=09=09=09=09=09</tr>
=09=09=09=09=09</table>
=09=09=09=09=09<table cellspacing=3D"0" cellpadding=3D"0" border=3D"0">
=09=09=09=09=09=09<tr>
=09=09=09=09=09=09=09<td bgcolor=3D"#FFFFFF" align=3D"left" style=3D"font-f=
amily: Arial, Helvetica, sans-serif; font-size:14px; padding-top: 25px;">
=09=09=09=09=09=09=09=09<a href=3D"https://pl.wikipedia.org/wiki/Dyskusja_w=
ikipedysty:Sir_Lestaty_de_Lioncourt?markasread=3D17548839&amp;markasreadwik=
i=3Dplwiki" style=3D"cursor:pointer; text-align:center; text-decoration:non=
e; padding:.45em 0.6em .45em; color:#FFF; background:#36C; font-family: Ari=
al, Helvetica, sans-serif;font-size: 13px;">Ver mensagem</a>&nbsp;&nbsp;<a =
href=3D"https://pl.wikipedia.org/wiki/Wikipedysta:Pathoschild" style=3D"tex=
t-decoration: none;font-size: 10px;font-family: Arial, Helvetica, sans-seri=
f; color: #72777D;">=E2=80=AAPathoschild=E2=80=AC</a>&nbsp;&nbsp;<a href=3D=
"https://pl.wikipedia.org/w/index.php?title=3DDyskusja_wikipedysty:Sir_Lest=
aty_de_Lioncourt&amp;oldid=3Dprev&amp;diff=3D41495542" style=3D"text-decora=
tion: none;font-size: 10px;font-family: Arial, Helvetica, sans-serif; color=
: #72777D;">Ver mudan=C3=A7as</a>
=09=09=09=09=09=09=09</td>
=09=09=09=09=09=09</tr>
=09=09=09=09=09</table>
=09=09=09=09</td>
=09=09=09</tr><tr>
=09=09=09=09<td bgcolor=3D"#FFFFFF">&nbsp;</td>
=09=09=09=09<td bgcolor=3D"#FFFFFF">&nbsp;</td>
=09=09=09=09<td bgcolor=3D"#FFFFFF" style=3D"line-height:40px;">&nbsp;</td>
=09=09=09=09<td bgcolor=3D"#FFFFFF">&nbsp;</td>
=09=09=09</tr><tr>
=09=09=09=09<td>&nbsp;</td>
=09=09=09=09<td>&nbsp;</td>
=09=09=09=09<td align=3D"left" style=3D"font-family: Arial, Helvetica, sans=
-serif; font-size:10px; line-height:13px; color:#72777D; padding:10px 20px;=
"><br />
=09=09=09=09=09Para controlar quais e-mails enviamos voc=C3=AA, <a href=3D"=
https://pl.wikipedia.org/wiki/Specjalna:Preferencje#mw-prefsection-echo" st=
yle=3D"text-decoration: none; color: #36C;">verifique suas prefer=C3=AAncia=
s</a>.<br />Wikimedia Foundation, 1 Montgomery Street, Suite 1600, San Fran=
cisco, CA 94104, USA
=09=09=09=09=09<br /><br />
=09=09=09=09</td>
=09=09=09=09<td>&nbsp;</td>
=09=09=09</tr><tr>
=09=09=09=09<td colspan=3D"4">&nbsp;</td>
=09=09=09</tr>
=09=09</table>
=09=09<br><br></center>
=09</td>
</tr>
</table>
</body></html>
--=_90658cc273d0f134e0e5038bb0ff92fb--

Oh, this might not be exactly the same issue I see now, hence reopening.

Two Polish Wikipedia users reported that they receive the same notifications day after day about events that happened years ago. One user reports continuous notifications about the last edit on his talk page (message notification).

https://pl.wikipedia.org/wiki/Wikipedia:Kawiarenka/Kwestie_techniczne#Dziwne_wzmiankowanie

There is an user from the Finnish WIkipedia who said that they got mentioned 5 days in a row on Polish Wikipedia:
https://fi.wikipedia.org/w/index.php?title=Wikipedia:Kahvihuone_(tekniikka)&oldid=19041003#Kerran_vuorokaudessa_toistuva_h%C3%A4lytysilmoitus_plwikist%C3%A4

They all involve one notification per day, about an edit that happened a long time ago (although the one in the task description seems to be a message notification and these ones are mentions).

All the edits mentioned here (the fiwiki report is the only one which did not specify the edit) are the latest revision of the respective page. One was a page move, the others are edits or reverts.

@slestaty do you receive only email notifications without any on-wiki notifications? If it's not only email then I think that this task should be renamed (per what @Tgr wrote). If it is then I think that's yet another issue.

We also have three new reports regarding:

  • this edit (latest revision, user mentioned in edit summary)
  • this edit (latest revision when user reported; page was modified today; after username change, user mentioned in edit summary)
  • this edit (latest revision, talk page edit)

Also, it's quite baffling that all those notifications are being sent because of edits on Polish Wikipedia.

@Tufor did the other users get on-wiki notifications? It wasn't clear to me from the reports.

Em T257766#6311786, @Tufor escreveu:

@slestaty do you receive only email notifications without any on-wiki notifications?

@Tufor I'm receiving only email notifications without any on-wiki notifs.

@Tgr no one explicitly wrote that they received any notifications via e-mail therefore I assume that they meant a on-wiki notification. Also I asked one user to check via Special:Notifications when exactly the notifications started to pop up (answer: July 9th), so that's definitely issue with the on-wiki system. Also, one more thing worth mentioning: all of these edits are in the User: or User talk: namespaces.

Yeah, this is also on-wiki. @slestaty should you have normally received a web notification, or is that just how you configured your notifications?

@Tgr you cannot un-check the "Web" checkbox in a "Talk page message" row in your Preferences, so that's kind of weird that he receives only this type of notification.

@Tgr Yes, I configured to receive both, on the wiki and by email (globally). I checked on plwiki now and got Pathoschild's message notifications (see the image), but those notifications on the wiki I didn't get, just by mail. Being more clear, when I enter plwiki I don't see any notice that there are new messages, I only see it if I enter the notifications page and even then, these notifications are all marked as read as you can see in the image.

plwiki.png (768×806 px, 44 KB)

Moving the original report by @slestaty from the task description:

Since July 10th I'm receiving a mail from wiki@wikimedia.org about a new message from Pathoschild on my talk page on plwiki, but this message from Pathoschild is from 2015. If it was just a stucked job, the message should be marked as read on the 10th when I opened the message and looked at my user talk, but yesterday and today I again received the email warning of the message. Has anyone an idea about it? Thanks.

Tgr added a subscriber: mb.
Tgr removed a subscriber: mb.
MMiller_WMF added a subscriber: Catrope.
MMiller_WMF subscribed.

@Catrope will be looking into this during the week.

Notes as I investigate:

  • User:Sir Lestaty de Lioncourt on plwiki (@slestaty)
    • Received an edit-user-talk notification about this edit (revision 41495542) by User:Pathoschild to their user talk page at 2015-01-11T23:27:22Z.
    • Received duplicates of this notification at the following timestamps:
      • 2020-07-11T00:24:25Z
      • 2020-07-12T00:22:16Z
      • 2020-07-12T00:24:24Z
      • 2020-07-13T00:16:13Z
      • 2020-07-14T00:15:04Z
      • 2020-07-15T04:01:16Z
      • 2020-07-16T00:15:12Z
      • 2020-07-17T00:20:49Z
      • 2020-07-18T00:23:59Z
      • 2020-07-19T00:22:17Z
      • 2020-07-20T00:17:47Z
    • The edit in question is currently the latest version of the page
    • Most of these look like they were marked as read by the user themselves: they were marked as read either a few minutes or a few hours after receipt, and the last few are unread
      • The exception is the second one on July 12th (2020-07-12T00:24:24Z) which was marked as read instantly (00:24:24). That looks like it might have been done automatically, although it could possibly have been a well-timed human interaction: the user had just received a notification at 00:22 and marked it as read at 00:23, so they were likely still active.
      • Note that marking as read can happen either through the web UI, or by clicking the link in the email; we don't record in the database which mechanism was used
    • The 2020 duplicates of this notification have identical event_extra values, but they're different from the 2016 notification's event_extra in that they're more complete: the 2016 one has an empty section-text and no target-page field, whereas the 2020 ones have the edit summary in the section-text field and do have a target-page field
    • The email this user posted here (at T257766#6303078) is dated Date: Tue, 14 Jul 2020 02:15:04 +0200, which matches 2020-07-14T00:15:04Z above. It was received by their email provider at Mon, 13 Jul 2020 17:15:07 -0700 (PDT), which is 3 seconds later.
  • User:JD on plwiki
    • Original edit: revision 15025515 by User:Vuvar1, renaming JD's user page at 2008-12-01T14:58:32Z. The edit summary links to JD's user page twice. No original notification about this edit, because Echo didn't exist until 2013
    • Received mention-summary notifications about this edit at the following timestamps:
      • 2020-07-10T18:54:18Z
      • 2020-07-11T18:47:58Z
      • 2020-07-12T18:47:54Z
      • 2020-07-13T18:48:11Z
      • 2020-07-14T22:31:36Z
      • 2020-07-15T18:47:46Z
      • 2020-07-16T16:36:24Z
      • 2020-07-16T16:36:53Z
      • 2020-07-16T16:37:06Z
      • 2020-07-16T16:37:17Z
      • 2020-07-16T16:37:32Z
      • 2020-07-16T16:38:23Z
      • 2020-07-16T16:40:20Z
    • The edit in question is not the latest version of the page; the next edit happened at 2020-07-16T16:41:08Z
  • User:Botev on plwiki
    • Original edit: revision 18591489 by User:Beau.bot on User:Igi^MBE at 2009-09-24T20:20:00Z. The edit summary mentions Botev (= contains a link to Botev's user page). No original notification about this edit, because Echo didn't exist until 2013
    • Received mention-summary notifications about this edit at the following timestamps:
      • 2020-07-10T18:51:06Z
      • 2020-07-11T18:46:14Z
      • 2020-07-12T18:45:04Z
      • 2020-07-13T18:44:33Z
      • 2020-07-14T22:26:31Z
      • 2020-07-15T18:45:21Z
      • 2020-07-16T18:45:10Z
      • 2020-07-17T18:44:51Z
      • 2020-07-18T18:44:25Z
      • 2020-07-19T18:46:02Z
      • 2020-07-20T18:43:31Z
    • The edit in question is currently the latest version of the page
  • User:Koreanovsky on plwiki
    • Original edit: revision 56113359 by User:Steinsplitter, renaming Koreanovsky's user page at 2019-03-10T06:28:51Z. No original notification, unclear why: the edit summary mentions Koreanovsky twice, and edit summary mention notifications were already enabled by then.
    • Received mention-summary notifications about this edit at the following timestamps:
      • 2020-07-09T19:26:22Z
      • 2020-07-10T19:32:06Z
      • 2020-07-11T19:25:09Z
      • 2020-07-12T19:25:47Z
      • 2020-07-13T19:25:02Z
      • 2020-07-14T23:08:38Z
      • 2020-07-15T19:25:14Z
    • The edit in question is not the latest version of the page; the next edit was at 2020-07-16T08:55:49Z
  • User:Hektor Absurdum on plwiki
    • Original edit: revision 60151849 by User:Euraczek on Hektor Absurdum's user talk page, at 2020-06-22T19:54:07Z. An edit-user-talk notification was sent at this time, as expected
    • Received duplicates of this notification at the following timestamps:
      • 2020-07-11T00:13:27Z
      • 2020-07-12T00:11:28Z
      • 2020-07-13T00:05:18Z
      • 2020-07-14T00:03:51Z
      • 2020-07-15T03:50:14Z
      • 2020-07-16T00:03:55Z
      • 2020-07-17T00:09:21Z
      • 2020-07-18T00:11:56Z
      • 2020-07-19T00:10:26Z
      • 2020-07-20T00:05:48Z
    • The edit in question is currently the latest version of the page
  • User:Amir smt on fawiki (reported on T257714)
    • Original edit: revision 28916760 which moved a page while mentioning Amir smt in the edit summary at 2020-05-02T12:00:15Z. No original notification, not sure why.
    • Received mention-summary notifications about this edit at the following timestamps:
      • 2020-07-10T19:34:17Z
      • 2020-07-10T19:40:25Z
      • 2020-07-10T19:44:58Z
      • 2020-07-10T21:44:00Z
      • 2020-07-10T21:51:01Z
      • 2020-07-10T21:57:08Z
      • 2020-07-10T22:01:47Z
      • 2020-07-10T22:06:27Z
      • 2020-07-10T22:11:09Z
      • 2020-07-10T22:18:06Z
      • 2020-07-10T22:26:12Z
      • 2020-07-10T22:32:17Z
      • 2020-07-10T22:38:20Z
      • 2020-07-10T22:43:00Z
      • 2020-07-10T22:47:43Z
      • 2020-07-10T22:52:24Z
      • 2020-07-10T22:57:07Z
      • 2020-07-10T23:03:41Z
      • 2020-07-10T23:12:40Z
      • 2020-07-10T23:17:33Z
      • 2020-07-10T23:22:32Z
      • 2020-07-10T23:27:57Z
      • 2020-07-10T23:33:45Z
      • 2020-07-10T23:38:51Z
    • The edit in question is not the latest version of the page; the next edit was at 2020-07-10T23:56:39Z
  • User:Paranaja on plwiki (reported on T257714)
    • Original edit: revision 55514112 by User:Ejs-80 renaming Paranaja's user page at 2019-01-05T23:41:24Z. The edit summary links to Paranaja's user page. No edit summary mention notification at the time, not sure why.
    • Received mention-summary notifications about this edit at the following timestamps:
      • 2020-07-09T21:44:39Z
      • 2020-07-10T21:49:14Z
      • 2020-07-11T21:43:04Z
      • 2020-07-12T21:44:24Z
      • 2020-07-13T21:40:03Z
      • 2020-07-15T01:24:54Z
    • The edit in question is the latest version of the page
  • User:1997kB on enwiki (reported on T257714)
    • Original edit: revision 968402952 on Wikipedia:Changing username/Simple by User:Cyberbot I at 2020-07-19T05:20:05Z, which mentions User:1997kB three times in three different sections. Accordingly, this user received three separate mention notifications for this same edit at 05:20:07 (two seconds later), with different section headings displayed for each notification
    • The user then received duplicates of all three notifications at the following timestamps:
      • 2020-07-19T05:30:05Z
      • 2020-07-19T05:40:05Z
      • 2020-07-19T05:50:05Z
      • 2020-07-19T06:00:05Z
      • 2020-07-19T06:10:05Z
      • 2020-07-19T06:20:05Z
      • 2020-07-19T06:30:05Z
      • 2020-07-19T06:40:05Z
      • 2020-07-19T06:50:05Z
      • 2020-07-19T07:00:05Z
      • 2020-07-19T07:10:05Z
      • 2020-07-19T07:20:05Z
      • 2020-07-19T07:30:05Z
      • 2020-07-19T07:40:04Z
      • 2020-07-19T07:50:05Z
    • The edit in question is not the latest version of the page; the next edit was at 2020-07-19T07:56:34Z
  • User:Nadzik on enwiki (reported on T257714)
    • Original edit: revision 967551312 on Wikipedia:Changing username/Simple by User:Cyberbot I at 2020-07-13T22:40:07Z. This edit mentions Nadzik, and they received a mention notification one second later.
    • They then received duplicate mention notifications at the following timestamps:
      • 2020-07-13T22:50:05Z
      • 2020-07-13T23:00:06Z
      • 2020-07-13T23:10:06Z
      • 2020-07-13T23:20:05Z
      • 2020-07-13T23:30:06Z
      • 2020-07-13T23:40:05Z
      • 2020-07-13T23:50:06Z
      • 2020-07-14T00:10:06Z
      • 2020-07-14T00:20:05Z
      • 2020-07-14T00:30:06Z
      • 2020-07-14T00:40:05Z
      • 2020-07-14T00:50:05Z
      • 2020-07-14T01:00:06Z
    • The edit in question is not the latest version of the page; the next edit was at 2020-07-14T01:04:00Z
  • User:Nadzik on enwiki (again! this unlucky user fell victim to this bug twice)
    • Original edit: revision 968007692 on Wikipedia:Changing username/Simple by User:Cyberbot I at 2020-07-16T17:10:04Z. This edit mentions Nadzik, and they received a mention notification two seconds later.
    • They then received duplicate mention notifications at the following timestamps:
      • 2020-07-16T17:20:05Z
      • 2020-07-16T17:30:05Z
      • 2020-07-16T17:40:05Z
      • 2020-07-16T17:50:06Z
      • 2020-07-16T18:00:06Z
      • 2020-07-16T18:10:05Z
      • 2020-07-16T18:20:06Z
      • 2020-07-16T18:30:06Z
      • 2020-07-16T18:40:05Z
      • 2020-07-16T18:50:06Z
      • 2020-07-16T19:00:05Z
      • 2020-07-16T19:10:05Z
      • 2020-07-16T19:20:06Z
      • 2020-07-16T19:30:05Z
      • 2020-07-16T19:40:06Z
      • 2020-07-16T19:50:05Z
      • 2020-07-16T20:00:06Z
      • 2020-07-16T20:10:05Z
      • 2020-07-16T20:20:05Z
      • 2020-07-16T20:30:06Z
      • 2020-07-16T20:40:05Z
      • 2020-07-16T20:50:06Z
      • 2020-07-16T21:00:06Z
      • 2020-07-16T21:10:06Z
    • The edit in question is not the latest version of the page; the next edit was at 2020-07-16T21:11:22Z

Things I noticed overall:

  • Most of these notifications repeat in consistent patterns
    • On plwiki, they repeat daily around the same time (except that the one on the late 14th / early 15th happens 4 hours later). Sometimes they happen multiple times on the same day in rapid succession
    • On enwiki they repeat every 10 minutes, and on fawiki approximately every 5 minutes
    • Could this be related to the pages' parser cache expirations?
  • All but one of these kept happening as long as the edit in question was the latest revision of the page, and then stopped once the page was edited again. The only exception is Paranaja's case, which stopped on its own without the page being edited. H/T to @Tgr's comment at T257766#6310288 for noticing that all the reported cases at the time were latest revisions.
  • The more complete event_extra fields, and the fact that this happens for edits from before Echo existed, suggest that these new notifications are not duplicate deliveries, but are really being generated anew. Perhaps some edits are being processed as if they're new
  • wmf.40 was deployed to group2 at 2020-07-09T19:05:47Z, and the first of these notifications started 21 minutes later
  • Could this be related to the pages' parser cache expirations?

I think I might be onto something here. The parser cache expirations of these pages are all reported as 2592000 (=30 days), but:

  • pl:User talk:Sir Lestaty de Lioncourt was saved in parser cache at 2020-07-20T00:17:46Z, and the most recent duplicate notification for that page was sent 1 second later
  • pl:User:Igi^MBE was saved in parser cache at 2020-07-20T18:43:30Z, and the most recent duplicate notification for that page was sent 1 second later
  • pl:User:Hektor Absurdus was saved in parser cache at 2020-07-20T00:05:47Z, and the most recent duplicate notification for that page was sent 1 second later
  • pl:User:Paranaja was saved in parser cache at 2020-07-20T21:48:35Z; this is the outlier whose duplicate notifications stopped on their own, but the timestamp is similar to that of the previous notifications for that page (which mostly happened between 21:40 and 21:50)

(and all the other pages were edited later)

So it looks like recaching pages in the parser cache is triggering a hook that causes Echo to think the edit is new.

In T257714#6316032 the user also independently came to a similar conclusion as @Tgr and in order to stop the annoying notifications, he ended up editing those pages to make the triggering edit not be the latest revision.

I think I figured it out: null edits are no longer being ignored by Echo.

I reproduced this bug by leaving a message for my staff account from my volunteer account here. then making a null edit. This resulted in two notifications, one with the timestamp of the original edit, and one with the timestamp of the null edit. I suspect the regularity of the timestamps I observed may be due to bots that make null edits.

I dont know if this is important, but all the emails arrived to me around 20h10-25h20(BRT/UTC-3) except on 15th that the mail arrived 01h01(BRT/UTC-3). So the notification is about to happen in a few minutes.

Change 614910 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/core@master] PageUpdater: Make EditResult::isNullEdit() return true for null edits

https://gerrit.wikimedia.org/r/614910

Change 615151 had a related patch set uploaded (by Ostrzyciel; owner: Ostrzyciel):
[mediawiki/core@master] PageUpdater: fix handling of null edits

https://gerrit.wikimedia.org/r/615151

Change 615151 merged by jenkins-bot:
[mediawiki/core@master] PageUpdater: fix handling of null edits

https://gerrit.wikimedia.org/r/615151

Change 615186 had a related patch set uploaded (by Kosta Harlan; owner: Ostrzyciel):
[mediawiki/core@wmf/1.35.0-wmf.40] PageUpdater: fix handling of null edits

https://gerrit.wikimedia.org/r/615186

Change 615187 had a related patch set uploaded (by Kosta Harlan; owner: Ostrzyciel):
[mediawiki/core@wmf/1.35.0-wmf.41] PageUpdater: fix handling of null edits

https://gerrit.wikimedia.org/r/615187

Change 615188 had a related patch set uploaded (by Kosta Harlan; owner: Ostrzyciel):
[mediawiki/core@REL1_35] PageUpdater: fix handling of null edits

https://gerrit.wikimedia.org/r/615188

Change 614910 abandoned by Kosta Harlan:
[mediawiki/core@master] PageUpdater: Make EditResult::isNullEdit() return true for null edits

Reason:
let's go with I04bb058c64483967617958d86aa40a67c31071cb

https://gerrit.wikimedia.org/r/614910

Change 615186 abandoned by Catrope:
[mediawiki/core@wmf/1.35.0-wmf.40] PageUpdater: fix handling of null edits

Reason:
Not needed, wmf.41 is already deployed to all wikis

https://gerrit.wikimedia.org/r/615186

Change 615196 had a related patch set uploaded (by Catrope; owner: Ostrzyciel):
[mediawiki/core@wmf/1.36.0-wmf.1] PageUpdater: fix handling of null edits

https://gerrit.wikimedia.org/r/615196

Change 615187 merged by jenkins-bot:
[mediawiki/core@wmf/1.35.0-wmf.41] PageUpdater: fix handling of null edits

https://gerrit.wikimedia.org/r/615187

Change 615196 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.1] PageUpdater: fix handling of null edits

https://gerrit.wikimedia.org/r/615196

Mentioned in SAL (#wikimedia-operations) [2020-07-21T19:01:32Z] <catrope@deploy1001> Synchronized php-1.35.0-wmf.41/includes/Storage/PageUpdater.php: Fix handling of null edits (T257766) (duration: 01m 11s)

Mentioned in SAL (#wikimedia-operations) [2020-07-21T19:02:51Z] <catrope@deploy1001> Synchronized php-1.36.0-wmf.1/includes/Storage/PageUpdater.php: Fix handling of null edits (T257766) (duration: 01m 06s)

The fix is deployed, this should now stop happening.

Change 615188 merged by jenkins-bot:
[mediawiki/core@REL1_35] PageUpdater: fix handling of null edits

https://gerrit.wikimedia.org/r/615188

  • User:Nadzik on enwiki (reported on T257714)
    • Original edit: revision 967551312 on Wikipedia:Changing username/Simple by User:Cyberbot I at 2020-07-13T22:40:07Z. This edit mentions Nadzik, and they received a mention notification one second later.
    • They then received duplicate mention notifications at the following timestamps:
      • 2020-07-13T22:50:05Z
      • 2020-07-13T23:00:06Z
      • 2020-07-13T23:10:06Z
      • 2020-07-13T23:20:05Z
      • 2020-07-13T23:30:06Z
      • 2020-07-13T23:40:05Z
      • 2020-07-13T23:50:06Z
      • 2020-07-14T00:10:06Z
      • 2020-07-14T00:20:05Z
      • 2020-07-14T00:30:06Z
      • 2020-07-14T00:40:05Z
      • 2020-07-14T00:50:05Z
      • 2020-07-14T01:00:06Z
    • The edit in question is not the latest version of the page; the next edit was at 2020-07-14T01:04:00Z
  • User:Nadzik on enwiki (again! this unlucky user fell victim to this bug twice)
    • Original edit: revision 968007692 on Wikipedia:Changing username/Simple by User:Cyberbot I at 2020-07-16T17:10:04Z. This edit mentions Nadzik, and they received a mention notification two seconds later.
    • They then received duplicate mention notifications at the following timestamps:
      • 2020-07-16T17:20:05Z
      • 2020-07-16T17:30:05Z
      • 2020-07-16T17:40:05Z
      • 2020-07-16T17:50:06Z
      • 2020-07-16T18:00:06Z
      • 2020-07-16T18:10:05Z
      • 2020-07-16T18:20:06Z
      • 2020-07-16T18:30:06Z
      • 2020-07-16T18:40:05Z
      • 2020-07-16T18:50:06Z
      • 2020-07-16T19:00:05Z
      • 2020-07-16T19:10:05Z
      • 2020-07-16T19:20:06Z
      • 2020-07-16T19:30:05Z
      • 2020-07-16T19:40:06Z
      • 2020-07-16T19:50:05Z
      • 2020-07-16T20:00:06Z
      • 2020-07-16T20:10:05Z
      • 2020-07-16T20:20:05Z
      • 2020-07-16T20:30:06Z
      • 2020-07-16T20:40:05Z
      • 2020-07-16T20:50:06Z
      • 2020-07-16T21:00:06Z
      • 2020-07-16T21:10:06Z
    • The edit in question is not the latest version of the page; the next edit was at 2020-07-16T21:11:22Z

Today another one happened, also by the Cyberbot. Cheers! ~~~~

Looking into the last report - the comment above.
Just trying to see some patterns to narrow down what is happening with two users User: Nadzik =28985363 and User:Cyberbot I=16266655. I did not limit the query by the time - all all available notifications are displayed:

select event_type, event_agent_id, notification_timestamp, notification_read_timestamp from  echo_notification join echo_event on event_id=notification_event where notification_user=28985363 and event_page_id=7535778 ;
repeating_notifications.query
+------------+----------------+------------------------+-----------------------------+
| event_type | event_agent_id | notification_timestamp | notification_read_timestamp |
+------------+----------------+------------------------+-----------------------------+
| mention    |       16266655 | 20200713224008         | 20200713225144              |
| mention    |       16266655 | 20200713225005         | 20200713225142              |
| mention    |       16266655 | 20200713230006         | 20200713230728              |
| mention    |       16266655 | 20200713231006         | 20200713231051              |
| mention    |       16266655 | 20200713232005         | 20200714080403              |
| mention    |       16266655 | 20200713233006         | 20200714080403              |
| mention    |       16266655 | 20200713234005         | 20200714080403              |
| mention    |       16266655 | 20200713235006         | 20200714080403              |
| mention    |       16266655 | 20200714001006         | 20200714080403              |
| mention    |       16266655 | 20200714002005         | 20200714080403              |
| mention    |       16266655 | 20200714003006         | 20200714080403              |
| mention    |       16266655 | 20200714004005         | 20200714080403              |
| mention    |       16266655 | 20200714005005         | 20200714080403              |
| mention    |       16266655 | 20200714010006         | 20200714080403              |
| mention    |       16266655 | 20200716171006         | 20200716193625              |
| mention    |       16266655 | 20200716172005         | 20200718132349              |
| mention    |       16266655 | 20200716173005         | 20200718132349              |
| mention    |       16266655 | 20200716174005         | 20200718132349              |
| mention    |       16266655 | 20200716175006         | 20200718132349              |
| mention    |       16266655 | 20200716180006         | 20200718132349              |
| mention    |       16266655 | 20200716181005         | 20200718132349              |
| mention    |       16266655 | 20200716182006         | 20200718132349              |
| mention    |       16266655 | 20200716183006         | 20200718132349              |
| mention    |       16266655 | 20200716184005         | 20200718132349              |
| mention    |       16266655 | 20200716185006         | 20200718132349              |
| mention    |       16266655 | 20200716190005         | 20200718132349              |
| mention    |       16266655 | 20200716191005         | 20200718132349              |
| mention    |       16266655 | 20200716192006         | 20200718132349              |
| mention    |       16266655 | 20200716193005         | 20200718132349              |
| mention    |       16266655 | 20200716194006         | 20200718132349              |
| mention    |       16266655 | 20200716195005         | 20200718132349              |
| mention    |       16266655 | 20200716200006         | 20200718132349              |
| mention    |       16266655 | 20200716201005         | 20200718132349              |
| mention    |       16266655 | 20200716202005         | 20200718132349              |
| mention    |       16266655 | 20200716203006         | 20200718132349              |
| mention    |       16266655 | 20200716204005         | 20200718132349              |
| mention    |       16266655 | 20200716205006         | 20200718132349              |
| mention    |       16266655 | 20200716210006         | 20200718132349              |
| mention    |       16266655 | 20200716211006         | 20200718132349              |
| mention    |       16266655 | 20200721132009         | 20200721132052              |
| mention    |       16266655 | 20200721133007         | 20200721141144              |
| mention    |       16266655 | 20200721134007         | 20200721141144              |
| mention    |       16266655 | 20200721135008         | 20200721141144              |
| mention    |       16266655 | 20200721140007         | 20200721141144              |
| mention    |       16266655 | 20200721141008         | 20200721141144              |
| mention    |       16266655 | 20200721142007         | 20200721142252              |
| mention    |       16266655 | 20200721143007         | 20200721144625              |
| mention    |       16266655 | 20200721144009         | 20200721144625              |
| mention    |       16266655 | 20200721145009         | 20200721145108              |
+------------+----------------+------------------------+-----------------------------+
49 rows in set (0.00 sec)

In fact, I checked other notifications (other pages and other users who initiated notifications), there were not noticeable patterns of repetitions.

The fix was deployed at 2020-07-21T19:02:51Z. The last duplicate notification that @Nadzik received according to @Etonkovidova's DB query was at 2020-07-21T14:50:09Z, so that's expected: they happened a few hours before I deployed the fix.

@Nadzik: did you receive any duplicate notifications after 19:03 UTC (21:03 your local time) on July 21st?

Until this moment I have not received any email from the wiki. Considering that emails arrived every day around 20h10-25h20(BRT/UTC-3) I imagine that the correction made by @Catrope solved the problem.

Trizek-WMF subscribed.

Can someone sum up this bug in one easy to understand for global audience sentence? Thanks!

I will do that; indeed, I will update the tech news draft.

@Nadzik: did you receive any duplicate notifications after 19:03 UTC (21:03 your local time) on July 21st?

Nope, I didn't. I will let you know if anything happens. Cheers!

Re-checked - the notifications seemed to be stopped. Resolving by now.

Until this moment I have not received any email from the wiki. Considering that emails arrived every day around 20h10-25h20(BRT/UTC-3) I imagine that the correction made by @Catrope solved the problem.

Great to hear that! Credit goes to @Ostrzyciel for writing the proper version of the fix that we ended up using.