Looking at the beta cluster scap time trend, the runtime went from less than a minute up to more than ten minutes.
Success #45591 12 min deployment-bastion.eqiad Success #45590 12 min deployment-bastion.eqiad Success #45589 13 min deployment-bastion.eqiad Success #45588 12 min deployment-bastion.eqiad Success #45587 12 min deployment-bastion.eqiad Success #45586 12 min deployment-bastion.eqiad Success #45585 14 min deployment-bastion.eqiad Failed #45584 0.32 sec deployment-bastion.eqiad Failed #45583 0.86 sec deployment-bastion.eqiad Success #45582 35 sec deployment-bastion.eqiad Success #45581 44 sec deployment-bastion.eqiad Success #45580 42 sec deployment-bastion.eqiad Success #45579 40 sec deployment-bastion.eqiad Success #45578 46 sec deployment-bastion.eqiad
The two failures are trying to unlink() /var/lock/scap but failling due to a permission error:
File "/mnt/srv/deployment/scap/scap/scap/utils.py", line 256, in lock
os.unlink(filename)
Operation not permitted: '/var/lock/scap'The next build #45585 happened after those two failures and ran for 14 minutes. It occured on March 17th 16:54:06 UTC. Looking at the console log with elapsed time (hh:mm:ss.micro):
00:01:44.958 16:55:51 Updating LocalisationCache for master using 2 thread(s) 00:13:55.018 17:08:01 Generating JSON versions and md5 files
All subsequent builds rebuild the LocalisationCache as well. So something is broken and cause scap / l10n updater to always consider the cache to be outdated thus rebuilding it everytime :(
Seems some operation / change has been made at that time which is the slowness root cause.