Page MenuHomePhabricator

Undefined property: stdClass::$event_deleted in Event.php
Closed, ResolvedPublic

Description

Seen in prod:

Notice: Undefined property: stdClass::$event_deleted in /srv/mediawiki/php-1.28.0-wmf.9/extensions/Echo/includes/model/Event.php on line 259

Event Timeline

MaxSem created this task.Jul 6 2016, 10:20 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 6 2016, 10:20 PM
SBisson claimed this task.Jul 7 2016, 1:58 PM
SBisson moved this task from Untriaged to In Development on the Collab-Team-2016-Apr-Jun-Q4 board.

This is worrying. Can you give more information about where and when this is/was happening? Thanks.

MaxSem added a comment.Jul 7 2016, 3:45 PM

This is worrying. Can you give more information about where and when this is/was happening? Thanks.

In production, right now.

Is there a stacktrace in the log?

Change 297831 had a related patch set uploaded (by Sbisson):
Undefined property: stdClass::$event_deleted

https://gerrit.wikimedia.org/r/297831

Krinkle added a subscriber: Krinkle.EditedJul 7 2016, 11:47 PM

@SBisson EchoEvent::loadFromRow <- EchoEvent::newFromRow:

  1. EventMapper::fetchById() - Uses select *.
  2. EmailBatch::getEvents() - Uses select *.
  3. EventMapper::fetchByUserBundleHash
    • email: select ... event_deleted ...
    • web: select ... , but missing event_deleted.

@SBisson asked me to check all the tables for this field (this would only be an issue if you used SELECT * and it was missing in the table; if you wrote SELECT [...] event_deleted [...] but there was no event_deleted in the table you would see a really obvious exception).

Most are on extension1. All of these have event_deleted except for the nonecho.dblist wikis:

mattflaschen@terbium:~$ pass=$(echo 'echo $wgDBadminpassword;' | mwscript eval.php --wiki="enwiki"); while read line; do echo "SELECT '$line', event_deleted FROM echo_event LIMIT 1"|mysql -u wikiadmin -p"$pass" -h 10.64.16.18 "$line"; done < /srv/mediawiki-staging/dblists/all.dblist > /dev/null
ERROR 1049 (42000): Unknown database 'labswiki'
ERROR 1049 (42000): Unknown database 'labtestwiki'
ERROR 1049 (42000): Unknown database 'legalteamwiki'
ERROR 1049 (42000): Unknown database 'zerowiki'

The ones not on extension1 are:

  • labswiki - event_deleted present
  • labtestwiki - event_deleted present.
  • mediawikiwiki - event_deleted present
  • metawiki - event_deleted present
  • officewiki - event_deleted present

The ones not on extension1 are:

  • labswiki - event_deleted present
  • labtestwiki - Can't access this DB.

I think you just need to SSH to labtestweb2001.codfw.wmnet and try from there... It should theoretically work like labswiki does with silver.eqiad.wmnet, and I'd expect the same result but it's a good idea to check.

Also:

  1. EventMapper->fetchByTypesAndPage (echo_event.*).
  2. EventMapper->fetchUnreadByUserAndPage (*)
  3. EchoNotification::newFromRow

6a. ApiEchoNotifications->makeForeignNotification (I think this is an actual trigger).
6b. NotificationMapper->fetchByUserInternal (*)
6c. NotificationMapper->fetchNewestByUserBundleHash (*)
6d. NotificationMapper->fetchByUserEvents (*)
6e. NotificationMapper->fetchByUserOffset (*)
6f. NotificationPager->getNotifications (*).
6 etc. (tests)

The fetchByUserBundleHash triggers in theory, but may not trigger in practice, since this code should no longer be reachable (yeah, if so it should be removed).

I think you just need to SSH to labtestweb2001.codfw.wmnet and try from there... It should theoretically work like labswiki does with silver.eqiad.wmnet, and I'd expect the same result but it's a good idea to check.

It won't let me ssh to that. Anyway, I think I found the trigger (ApiEchoNotifications->makeForeignNotification).

I think you just need to SSH to labtestweb2001.codfw.wmnet and try from there... It should theoretically work like labswiki does with silver.eqiad.wmnet, and I'd expect the same result but it's a good idea to check.

It won't let me ssh to that.

Strange. You have deployment access, you should be able to:

krenair@labtestweb2001:~$ cat /etc/ssh/userkeys/mattflaschen 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHESn/vYQ5F7M1fOWjQMKnTe83DpT0GSlDmJW9rhJO4GEX4VT2P/csQsNPjYHD4zqJyYf49mk1idldxg7QFjoZCeLvlw0QHcg+pTOh91IE2fP+V6qLErIs0VHW7pzPb4slevKqxYpKJKY2P4qvviulfENg4XoqkH8szfPdN0L0iouVZVLxIB2jjD561Zc0ytc1Kg4QW9dXZi3Jb7ukxdVC7v7Sipz/H8ge4urc4Y4BCbTPciRf30Ne5sRjqlR0qDvmeUhi9AOz/qmLqj4lmYRJ+OaaLat2ejyAFF0HAI/uZjtNgpvkaV85FlIahqKgVGKlCaV0fXOQP9oRUkvLPKsd mflaschen@wikimedia.org

Are you sure your SSH config is correct to allow connecting to hosts outside eqiad? Maybe try mira.codfw.wmnet or something.

Change 297943 had a related patch set uploaded (by Mattflaschen):
Fix E_NOTICE due to event_delete missing on foreign psuedo-notification

https://gerrit.wikimedia.org/r/297943

Are you sure your SSH config is correct to allow connecting to hosts outside eqiad? Maybe try mira.codfw.wmnet or something.

Yeah, I tried that earlier. mira works, and mira is not special-cased (or mentioned at all) in my ssh config.

However, for labtestweb2001.codfw.wmnet I get:

channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host
Krenair added a comment.EditedJul 8 2016, 1:42 AM

Ah, turns out this is my fault. Despite being in eqiad and codfw respectively, silver and labtestweb2001 have public IPs, and therefore are silver.wikimedia.org and labtestweb2001.wikimedia.org, they don't fall under eqiad.wmnet/codfw.wmnet in DNS. Because of their firewalls you still need to proxy in via a production bastion host though.

Change 297831 abandoned by Sbisson:
Undefined property: stdClass::$event_deleted

Reason:
Not the root cause.

https://gerrit.wikimedia.org/r/297831

SBisson moved this task from Needs Review to QA Review on the Collab-Team-2016-Apr-Jun-Q4 board.
SBisson added a subscriber: SBisson.

Change 297943 merged by jenkins-bot:
Fix E_NOTICE due to event_deleted missing on foreign psuedo-notification

https://gerrit.wikimedia.org/r/297943

  • Selectively checked x1-analytics-slave for event_deleted column to present
  • betalabs has labswiki which does not have echo tables
  • did some general regression testing -no error msgs appear

Change 298400 had a related patch set uploaded (by MaxSem):
Fix E_NOTICE due to event_deleted missing on foreign psuedo-notification

https://gerrit.wikimedia.org/r/298400

Change 298400 merged by jenkins-bot:
Fix E_NOTICE due to event_deleted missing on foreign psuedo-notification

https://gerrit.wikimedia.org/r/298400

jmatazzoni closed this task as Resolved.Jul 12 2016, 6:03 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM