Page MenuHomePhabricator

cron fail on tools-submit
Closed, ResolvedPublic

Description

No job submits for flr task since 3:10 UTC; manual jsub works.

Event Timeline

zhuyifei1999 raised the priority of this task from to Needs Triage.
zhuyifei1999 updated the task description. (Show Details)
zhuyifei1999 added a project: Toolforge.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptDec 12 2015, 9:46 AM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
yuvipanda triaged this task as High priority.Dec 12 2015, 9:47 AM

Seems LDAP related:

Dec 12 09:42:02 tools-submit CRON[24437]: Permission denied
Dec 12 09:42:02 tools-submit CRON[24428]: Permission denied
Dec 12 09:42:02 tools-submit CRON[24427]: Permission denied
Dec 12 09:42:02 tools-submit CRON[24432]: Permission denied
Dec 12 09:42:03 tools-submit kernel: [10433656.107362] init: updatetools main process (24281) terminated with status 1
Dec 12 09:42:03 tools-submit kernel: [10433656.107399] init: updatetools main process ended, respawning
Dec 12 09:42:03 tools-submit nslcd[29853]: [18444b] <passwd(all)> passwd entry uid=80686,ou=people,dc=wikimedia,dc=org denied by validnames option: "80686"
Dec 12 09:42:04 tools-submit nslcd[29853]: [18444b] <passwd(all)> ldap_result() failed: Size limit exceeded
Dec 12 09:42:05 tools-submit nslcd[29853]: [d0381c] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:05 tools-submit nslcd[29853]: [2e04fe] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:05 tools-submit nslcd[29853]: [f1aba9] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [6fdd76] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [a5b684] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [81d615] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [19e6bb] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [42a984] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [bb3346] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [6e2ba1] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [e2cfa2] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [17d3bb] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [cb496f] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:06 tools-submit nslcd[29853]: [d9da9a] <group=50062> error writing to client: Broken pipe
Dec 12 09:42:07 tools-submit nslcd[29853]: [ca56c1] <group=550> error writing to client: Broken pipe
Dec 12 09:42:07 tools-submit nslcd[29853]: [b80a03] <group=550> error writing to client: Broken pipe
Dec 12 09:42:07 tools-submit nslcd[29853]: [22100a] <group=550> error writing to client: Broken pipe
Dec 12 09:42:10 tools-submit kernel: [10433662.887394] init: updatetools main process (24445) terminated with status 1
Dec 12 09:42:10 tools-submit kernel: [10433662.887433] init: updatetools main process ended, respawning
Dec 12 09:42:10 tools-submit nslcd[29853]: [e85ae0] <passwd(all)> passwd entry uid=80686,ou=people,dc=wikimedia,dc=org denied by validnames option: "80686"
Dec 12 09:42:10 tools-submit nslcd[29853]: [e85ae0] <passwd(all)> ldap_result() failed: Size limit exceeded
Dec 12 09:42:14 tools-submit puppet-agent[22485]: Finished catalog run in 66.85 seconds
Dec 12 09:42:17 tools-submit kernel: [10433670.428106] init: updatetools main process (24574) terminated with status 1
Dec 12 09:42:17 tools-submit kernel: [10433670.428142] init: updatetools main process ended, respawning
Dec 12 09:42:17 tools-submit nslcd[29853]: [53ed2b] <passwd(all)> passwd entry uid=80686,ou=people,dc=wikimedia,dc=org denied by validnames option: "80686"
Dec 12 09:42:18 tools-submit nslcd[29853]: [53ed2b] <passwd(all)> ldap_result() failed: Size limit exceeded
Dec 12 09:42:25 tools-submit kernel: [10433677.902346] init: updatetools main process (24725) terminated with status 1
Dec 12 09:42:25 tools-submit kernel: [10433677.902389] init: updatetools main process ended, respawning
Dec 12 09:42:25 tools-submit nslcd[29853]: [354bbb] <passwd(all)> passwd entry uid=80686,ou=people,dc=wikimedia,dc=org denied by validnames option: "80686"
Dec 12 09:42:25 tools-submit nslcd[29853]: [354bbb] <passwd(all)> ldap_result() failed: Size limit exceeded
(END)
Stigmj added a subscriber: Stigmj.Dec 12 2015, 10:10 AM

Same problem for my account. Last task run was 03:10.

https://gerrit.wikimedia.org/r/#/c/258663/ was hand-reverted on tools-submit, bringing cron back. Will need more thorough investigation when I'm more awake...

KTC added a subscriber: KTC.Dec 12 2015, 11:11 AM

No cron jobs have run for approximately the last six hours. The error messages I've been getting vary quite a bit; the most recent one was:

error: commlib error: got select error (Connection refused)
Unable to run job: unable to send message to qmaster using port 6444 on host "tools-grid-master.tools.eqiad.wmflabs": got send error.
Exiting.
valhallasw closed this task as Resolved.May 27 2016, 12:23 PM