This is worrying. Can you give more information about where and when this is/was happening? Thanks.

In T139536#2437550, @SBisson wrote:

This is worrying. Can you give more information about where and when this is/was happening? Thanks.

In production, right now.

Is there a stacktrace in the log?

No.

Change 297831 had a related patch set uploaded (by Sbisson):
Undefined property: stdClass::$event_deleted

https://gerrit.wikimedia.org/r/297831

@SBisson EchoEvent::loadFromRow <- EchoEvent::newFromRow:

1. EventMapper::fetchById() - Uses select *.
2. EmailBatch::getEvents() - Uses select *.
3. EventMapper::fetchByUserBundleHash
   • email: select ... event_deleted ...
   • web: select ... , but missing event_deleted.

@SBisson asked me to check all the tables for this field (this would only be an issue if you used SELECT * and it was missing in the table; if you wrote SELECT [...] event_deleted [...] but there was no event_deleted in the table you would see a really obvious exception).

Most are on extension1. All of these have event_deleted except for the nonecho.dblist wikis:

mattflaschen@terbium:~$ pass=$(echo 'echo $wgDBadminpassword;' | mwscript eval.php --wiki="enwiki"); while read line; do echo "SELECT '$line', event_deleted FROM echo_event LIMIT 1"|mysql -u wikiadmin -p"$pass" -h 10.64.16.18 "$line"; done < /srv/mediawiki-staging/dblists/all.dblist > /dev/null
ERROR 1049 (42000): Unknown database 'labswiki'
ERROR 1049 (42000): Unknown database 'labtestwiki'
ERROR 1049 (42000): Unknown database 'legalteamwiki'
ERROR 1049 (42000): Unknown database 'zerowiki'

The ones not on extension1 are:

• labswiki - event_deleted present
• labtestwiki - event_deleted present.
• mediawikiwiki - event_deleted present
• metawiki - event_deleted present
• officewiki - event_deleted present

In T139536#2440347, @Mattflaschen-WMF wrote:

The ones not on extension1 are:

• labswiki - event_deleted present
• labtestwiki - Can't access this DB.

I think you just need to SSH to labtestweb2001.codfw.wmnet and try from there... It should theoretically work like labswiki does with silver.eqiad.wmnet, and I'd expect the same result but it's a good idea to check.

Also:

4. EventMapper->fetchByTypesAndPage (echo_event.*).
5. EventMapper->fetchUnreadByUserAndPage (*)
6. EchoNotification::newFromRow

6a. ApiEchoNotifications->makeForeignNotification (I think this is an actual trigger).
6b. NotificationMapper->fetchByUserInternal (*)
6c. NotificationMapper->fetchNewestByUserBundleHash (*)
6d. NotificationMapper->fetchByUserEvents (*)
6e. NotificationMapper->fetchByUserOffset (*)
6f. NotificationPager->getNotifications (*).
6 etc. (tests)

The fetchByUserBundleHash triggers in theory, but may not trigger in practice, since this code should no longer be reachable (yeah, if so it should be removed).

In T139536#2440387, @Krenair wrote:

I think you just need to SSH to labtestweb2001.codfw.wmnet and try from there... It should theoretically work like labswiki does with silver.eqiad.wmnet, and I'd expect the same result but it's a good idea to check.

It won't let me ssh to that. Anyway, I think I found the trigger (ApiEchoNotifications->makeForeignNotification).

In T139536#2440394, @Mattflaschen-WMF wrote:

In T139536#2440387, @Krenair wrote:

I think you just need to SSH to labtestweb2001.codfw.wmnet and try from there... It should theoretically work like labswiki does with silver.eqiad.wmnet, and I'd expect the same result but it's a good idea to check.

It won't let me ssh to that.

Strange. You have deployment access, you should be able to:

krenair@labtestweb2001:~$ cat /etc/ssh/userkeys/mattflaschen 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHESn/vYQ5F7M1fOWjQMKnTe83DpT0GSlDmJW9rhJO4GEX4VT2P/csQsNPjYHD4zqJyYf49mk1idldxg7QFjoZCeLvlw0QHcg+pTOh91IE2fP+V6qLErIs0VHW7pzPb4slevKqxYpKJKY2P4qvviulfENg4XoqkH8szfPdN0L0iouVZVLxIB2jjD561Zc0ytc1Kg4QW9dXZi3Jb7ukxdVC7v7Sipz/H8ge4urc4Y4BCbTPciRf30Ne5sRjqlR0qDvmeUhi9AOz/qmLqj4lmYRJ+OaaLat2ejyAFF0HAI/uZjtNgpvkaV85FlIahqKgVGKlCaV0fXOQP9oRUkvLPKsd mflaschen@wikimedia.org

Are you sure your SSH config is correct to allow connecting to hosts outside eqiad? Maybe try mira.codfw.wmnet or something.

Change 297943 had a related patch set uploaded (by Mattflaschen):
Fix E_NOTICE due to event_delete missing on foreign psuedo-notification

https://gerrit.wikimedia.org/r/297943

In T139536#2440398, @Krenair wrote:

Are you sure your SSH config is correct to allow connecting to hosts outside eqiad? Maybe try mira.codfw.wmnet or something.

Yeah, I tried that earlier. mira works, and mira is not special-cased (or mentioned at all) in my ssh config.

However, for labtestweb2001.codfw.wmnet I get:

channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

Ah, turns out this is my fault. Despite being in eqiad and codfw respectively, silver and labtestweb2001 have public IPs, and therefore are silver.wikimedia.org and labtestweb2001.wikimedia.org, they don't fall under eqiad.wmnet/codfw.wmnet in DNS. Because of their firewalls you still need to proxy in via a production bastion host though.

Change 297831 abandoned by Sbisson:
Undefined property: stdClass::$event_deleted

Reason:
Not the root cause.

https://gerrit.wikimedia.org/r/297831

Change 297943 merged by jenkins-bot:
Fix E_NOTICE due to event_deleted missing on foreign psuedo-notification

https://gerrit.wikimedia.org/r/297943

• Selectively checked x1-analytics-slave for event_deleted column to present
• betalabs has labswiki which does not have echo tables
• did some general regression testing -no error msgs appear

Change 298400 had a related patch set uploaded (by MaxSem):
Fix E_NOTICE due to event_deleted missing on foreign psuedo-notification

https://gerrit.wikimedia.org/r/298400

Change 298400 merged by jenkins-bot:
Fix E_NOTICE due to event_deleted missing on foreign psuedo-notification

https://gerrit.wikimedia.org/r/298400