iegreview: missing grants@ sender address (was: login failing with csrf token missing warning)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	May 14 2020, 7:46 PM

Description

Reported via email on 2020-05-14 by @Mjohnson_WMF.

Login attempts at https://iegreview.wikimedia.org/campaigns are failing with an "invalid request" message indicating CSRF token mismatch. Reproducible by other users as well.

Log output on mwlog1001.eqiad.wmnet shows errors like:

$ tail -1 /srv/mw-log/iegreview.log | python -mjson.tool
{
    "@timestamp": "2020-05-14T19:31:40.779714+00:00",
    "@version": 1,
    "channel": "iegreview",
    "expected": "7f82ce00b2f7189cfe9929ccce5683e58b097478",
    "got": "76576ac736dcd65b720c37b07f36b75e74b571e2",
    "host": "miscweb1002",
    "http_method": "POST",
    "ip": "**REDACTED**",
    "level": "ERROR",
    "message": "Missing or invalid CSRF token",
    "process_id": 9274,
    "referrer": "https://iegreview.wikimedia.org/campaigns",
    "server": "iegreview.wikimedia.org",
    "type": "iegreview",
    "uid": "99d43ae",
    "url": "/login.post"
}

Possibly some session storage retrieval issues causing this? CSRF tokens are actually in the html form and being posted to the backend.

@Mjohnson_WMF reports that authentication had been working recently (within the last week) and certainly after the application was migrated to miscweb1002 (T247648: miscweb1001/2001 - upgrade to buster or decom).

Details

	Subject	Repo	Branch	Lines +/-
	change email sender address from grants@ to projectcom@	wikimedia/iegreview	master	+1 -1

Customize query in gerrit

Related Objects

Mentioned Here: T247648: miscweb1001/2001 - upgrade to buster or decom

Event Timeline

bd808 created this task.May 14 2020, 7:46 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2020, 7:46 PM

bd808 triaged this task as High priority.May 14 2020, 7:49 PM

Proxying a report from a kind SRE who looked at the host, the problem is likely that the root partition is full. This makes sense with the symptoms as PHP would by default be storing PHP session data on the local disk.

@Volans moved some files off the disk which has logins working again!

@Dzahn there was some trace that you may have created the files that filled the disk here. Just pinging you for awareness.

The host was having the disk full. I've moved temporarily the following files to cumin1001:/root/:

-rw-rw-r-- 1 2552 wikidev 579684081 May 13 03:19 codereview.tar.gz
-rw-r--r-- 1 root root    579447575 May 14 12:24 with_r.tar.gz

@Dzahn I think you were working on this, I didn't move the rest of the files (more than 100k) for now.

Current status of the disk is:

/dev/vda1      ext4       18G   16G  780M  96% /

@Volans @bd808 Arr, yea, i had moved those files and it was meant to be a temp. thing. I just cleaned up there and it now has 5G space again. Thanks a lot for taking care of it, volans.

Sorry for inconvenience @Mjohnson_WMF. Since everything should work again i am closing the ticket as resolved.

Also deleted files from cumin1001 again.

@Dzahn and @Volans

Thank you both for your help with the grants scoring tool last week. I am following up because I am still struggling to log in. It's not recognizing my password. I had tried to reset my password last week and I tried again just now. But I couldn't then and still can't find an email providing a link to set up the new password. I use the tool only a couple of times a year (when it becomes very important), so there may be something I'm doing wrong, but I'm not sure what. Do you have any suggestions for me for resetting my password?

Thank you so much for your help.

Warm regards,

Marti

In T252815#6156289, @Mjohnson_WMF wrote:

@Dzahn and @Volans

But I couldn't then and still can't find an email providing a link to set up the new password.

I can see log events in mwlog1001:/srv/mw-log/iegreview.log for password reset tokens being created. I do not see any errors in send email logged. But I triggered a reset for my account and I have not received it.

I do not have access to this server to debug further.

Dzahn reopened this task as Open.May 22 2020, 6:29 AM

@Mjohnson_WMF @bd808 I just tested sending the password recovery mail to you while watching log files on the server and I found:

SMTP error from remote mail server after RCPT TO:<mjohnson@wikimedia.org>: 550-Verification failed for <grants@wikimedia.org>\n550-Address grants@wikimedia.org does not exist\n550 Sender verify failed

So it is because the app is trying to send from grants@wikimedia.org but that does not exist and nowadays the mail servers reject that kind of mail.

It seems like the previously existing grants@wikimedia.org Google group has been deleted as part of T191881 or otherwise.

@Mjohnson_WMF I temporarily added an alias for grants@ to myself on the mail servers and then hit the password recovery form again. I see it has now delivered an email to you. Please check if you received it and you are now unblocked.

@bd808 ^ Just a temp. unblocker on my side, the real fix will have to be on the OIT / Google side.grants@ used to be a Google group but has apparently been deleted at some point. I will open a Zendesk ticket for that.

Dzahn renamed this task from Iegreview login failing with csrf token missing warning to iegreview: missing grants@ sender address (was: login failing with csrf token missing warning).May 22 2020, 8:15 AM

Change 598952 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[wikimedia/iegreview@master] change email sender address from grants@ to projectcom@

https://gerrit.wikimedia.org/r/598952

gerritbot added a project: Patch-For-Review.May 27 2020, 7:09 AM

@bd808 I uploaded a change to the sender email address in the iegreview repo. ^ Could you possibly deploy it?

In T252815#6168498, @Dzahn wrote:

@bd808 I uploaded a change to the sender email address in the iegreview repo. ^ Could you possibly deploy it?

I live hacked the prod deploy to use the projectcom@wikimedia.org sender account and tested a reset email to make sure that it is being used correctly. Doing a deploy today is not ideal as there is currently an active campaign and deploy is out of sync with the HEAD of the repo. We should fix that for sure!, but ideally not when folks are using the app as it may break badly.

Change 598952 abandoned by Dzahn:
change email sender address from grants@ to projectcom@

Reason:
live hacked and per comments above

https://gerrit.wikimedia.org/r/598952

Gotcha @bd808 Alright, thanks, the live hack will do it for now.

So the iegreview app is fixed and I am just keeping this open now until I have removed myself again from receiving grants@ email, pending a decision about an alias added by OIT.

Maintenance_bot removed a project: Patch-For-Review.May 28 2020, 8:10 AM

I removed the temp. hack to forward grants@ to myself in exim aliases again. I suggested to OIT to add an alias for it to Alex Wang since there was still legit mail arriving there.

iegreview: missing grants@ sender address (was: login failing with csrf token missing warning)Closed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

iegreview: missing grants@ sender address (was: login failing with csrf token missing warning)
Closed, ResolvedPublic
Actions