Looking at the [[ https://integration.wikimedia.org/ci/job/beta-scap-eqiad/buildTimeTrend | beta cluster scap time trend ]], the runtime went from less than a minute up to more than ten minutes.
```
Success #45591 12 min deployment-bastion.eqiad
Success #45590 12 min deployment-bastion.eqiad
Success #45589 13 min deployment-bastion.eqiad
Success #45588 12 min deployment-bastion.eqiad
Success #45587 12 min deployment-bastion.eqiad
Success #45586 12 min deployment-bastion.eqiad
Success #45585 14 min deployment-bastion.eqiad
Failed #45584 0.32 sec deployment-bastion.eqiad
Failed #45583 0.86 sec deployment-bastion.eqiad
Success #45582 35 sec deployment-bastion.eqiad
Success #45581 44 sec deployment-bastion.eqiad
Success #45580 42 sec deployment-bastion.eqiad
Success #45579 40 sec deployment-bastion.eqiad
Success #45578 46 sec deployment-bastion.eqiad
```
The two failures are trying to unlink() /var/lock/scap but failling due to a permission error:
```
File "/mnt/srv/deployment/scap/scap/scap/utils.py", line 256, in lock
os.unlink(filename)
Operation not permitted: '/var/lock/scap'
```
The next build #45585 happened after those two failures and ran for 14 minutes. It occured on March 17th 16:54:06 UTC. Looking at the console log with elapsed time (hh:mm:ss.micro):
```
00:01:44.958 16:55:51 Updating LocalisationCache for master using 2 thread(s)
00:13:55.018 17:08:01 Generating JSON versions and md5 files
```
All subsequent builds rebuild the LocalisationCache as well. So something is broken and cause scap / l10n updater to always consider the cache to be outdated thus rebuilding it everytime :(
Seems some operation / change has been made at that time which is the slowness root cause.