Mentioned in SAL (#wikimedia-operations) [2018-09-19T19:40:59Z] <hashar> web request 60 second timeout when deploying is filled as https://phabricator.wikimedia.org/T204871

The web request times out are enforced since September 10th (see T97192#4561879 and https://lists.wikimedia.org/pipermail/wikitech-l/2018-September/090803.html ). We have not deployed a train since then.

I have not paid attention to those errors when promoting group0 yesterday. Looking at group0 the same issue happened:

19:20	<hashar@deploy1001>	rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.22

Maybe we always had the issue and it is now showing up due to the timeout limit being now respected.

Same happened with group2:

14:38:57 Finished sync-apaches (duration: 00m 38s)

Maybe we always had the issue and it is now showing up due to the timeout limit being now respected.

I believe this to be true.

Logs are full of this error message.

(Starting to add subtasks to this that are either instances of or other teams tracking their portions of the problem.)

Making this a follow-up/actionable to track this work better/more realistically.

Just a note that for today's promotion of group1 to 1.33.0-wmf.13 (T206667), I segmented the group1 error-log dashboard to have a view of just these timeout errors and a view excluding them. It was very helpful in keeping on both the rise in timeouts and side effects or unrelated errors. I plan on saving the dashboards and adding links in the train docs.

The issue about app servers having the first few requests time out after a deploy, naturally, also affects the canaries. As such, this is sometimes causing the endpoint checks, whhich Scap performs against canaries during a deployment, to sometimes fail.

@hashar wrote at T203664:

Spotted while promoting all wikis:

13:05:12 Check 'Check endpoints for mwdebug1002.eqiad.wmnet' failed:
  /wiki/{title} (Main Page) timed out before a response was received;
  /wiki/{title} (Special Version) timed out before a response was received;
  /w/api.php (Main Page pageprops) timed out before a response was received

The above report is about mwdebug (which is slower in general due to being a VM), but I've seen it happen on other canaries as well.

In T204871#4930433, @Krinkle wrote:

The issue about app servers having the first few requests time out after a deploy, naturally, also affects the canaries. As such, this is sometimes causing the endpoint checks, whhich Scap performs against canaries during a deployment, to sometimes fail.

I've seen that as well, and just as problematic is that it's rather hard to determine the difference between incidental slowness due to the deployment itself vs breakage actually caused by the new code being deployed.