Page MenuHomePhabricator
Create Task
Maniphest T179963

Workspaces for mwgate-php55lint / mwgate-php70lint are getting huge
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ du -hs *
311M	analytics-refinery-release
4.0K	analytics-refinery-release@tmp
873M	apps-android-wikipedia-publish
913M	apps-android-wikipedia-test
114M	commit-message-validator
227M	composer-package-validate
42M	debian-glue
472K	debian-glue-non-voting
4.0K	fail-archived-repositories
88M	integration-zuul-layoutdiff
16M	integration-zuul-layoutvalidation-gate
876K	labs-tools-ZppixBot-php55lint
788M	mediawiki-core-code-coverage
1.3G	mediawiki-core-jsduck
3.1G	mediawiki-core-php55lint
82M	mediawiki-vendor-composer-security
96M	mwext-CirrusSearch-whitespaces
163M	mwext-VisualEditor-jsduck
696K	mwgate-composer-validate
50M	mwgate-jsduck
5.4G	mwgate-php55lint
24M	mwgate-php56lint
15M	operations-dns-lint
1.6M	operations-dns-tabs
192M	operations-mw-config-php55lint
97M	operations-mw-config-typos
169M	phabricator-jessie-commits
3.7M	phabricator-jessie-debs
170M	phabricator-jessie-diffs
4.8M	php55lint
144M	php56lint
14M	php-compile-hhvm
15M	php-compile-php55
15M	php-compile-php70
88M	selenium-Wikibase-T167432
100K	test-csteipp-sensiolabs-securityadvisorieschecker
502M	wikimedia-fundraising-civicrm

High prio because this is filling up /srv

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Legoktm added a comment.Nov 29 2017, 3:55 AM

Just had to clean this up on integration-slave-jessie-1003.

Stashbot subscribed.Dec 22 2017, 10:39 PM

Mentioned in SAL (#wikimedia-releng) [2017-12-22T22:38:59Z] <thcipriani> integration-slave-jessie-1004 removed mediawiki-core-jsduck, mwgate-php55lint, mediawikicore-php55lint as /srv mount was full T179963

hashar mentioned this in T184269: integration.integration-slave-jessie-1001 disk space full.Jan 6 2018, 9:28 AM
hashar mentioned this in T181899: mwgate-php55lint fails to fetch origin.Jan 22 2018, 2:10 PM
hashar merged a task: T181899: mwgate-php55lint fails to fetch origin.
hashar added a subscriber: Bawolff.
hashar added a comment.EditedJan 22 2018, 2:15 PM

That is happening once per week or so. The workaround is to clean out the workspaces manually but that is terrible.

We apparently two different templates to generate the PHP lint jobs. Comparing php55lint and mwgate-php55lint:

--- php55lint	2017-12-04 09:26:58.000000000 +0100
+++ mwgate-php55lint	2017-12-04 09:26:58.000000000 +0100
@@ -4,3 +4,4 @@
   <description>&lt;p&gt;Job is managed by &lt;a href=&quot;https://www.mediawiki.org/wiki/CI/JJB&quot;&gt;Jenkins Job Builder&lt;/a&gt;.&lt;/p&gt;
-&lt;p&gt;This job is triggered by Zuul&lt;/p&gt;
+&lt;p&gt;This job is triggered by Zuul.&lt;/p&gt;
+&lt;p&gt;Git submodules are NOT processed.&lt;/p&gt;
 &lt;!-- Managed by Jenkins Job Builder --&gt;</description>
@@ -15,3 +16,3 @@
       <strategy class="hudson.tasks.LogRotator">
-        <daysToKeep>30</daysToKeep>
+        <daysToKeep>15</daysToKeep>
         <numToKeep>-1</numToKeep>
@@ -113,5 +114,3 @@
     <extensions>
-      <hudson.plugins.git.extensions.impl.CloneOption>
-        <shallow>true</shallow>
-      </hudson.plugins.git.extensions.impl.CloneOption>
+      <hudson.plugins.git.extensions.impl.CleanCheckout/>
       <hudson.plugins.git.extensions.impl.SubmoduleOption>
@@ -124,3 +123,2 @@
       </hudson.plugins.git.extensions.impl.SubmoduleOption>
-      <hudson.plugins.git.extensions.impl.WipeWorkspace/>
     </extensions>

EDIT: mixed it up

php55lint

Wipes the workspace and does a shallow clone. That is the version we would want to use everywhere. I am not sure how git-changed-in-head.sh behave with a shallow clone but hopefully it will be fine.

mwgate-php55lint

Apparently does a full clone and a clean checkout. It does NOT wipe the workspace hence the .git directory keeps growing up as different repositories trigger that job (see my earlier comment T179963#3742837 ).

hashar mentioned this in T179740: mwgate-php55lint failing with "Failed to fetch from git://contint1001.wikimedia.org/mediawiki/core".Jan 22 2018, 2:21 PM
hashar merged a task: T185378: fatal: write error: No space left on device on php55lint.Jan 22 2018, 2:49 PM
hashar merged a task: T184269: integration.integration-slave-jessie-1001 disk space full.
hashar merged a task: T179740: mwgate-php55lint failing with "Failed to fetch from git://contint1001.wikimedia.org/mediawiki/core".
hashar added subscribers: MarcoAurelio, Reedy.
hashar added a subscriber: valerio.bozzolan.
gerritbot subscribed.Jan 22 2018, 3:16 PM

Change 405722 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Use shallow clone for phplint jobs

https://gerrit.wikimedia.org/r/405722

gerritbot added a project: Patch-For-Review.Jan 22 2018, 3:16 PM
hashar mentioned this in T185821: integration-slave-jessie-1002 out of disk.Jan 29 2018, 6:33 AM
hashar merged a task: T185821: integration-slave-jessie-1002 out of disk.
hashar added subscribers: saper, RazeSoldier, Jayprakash12345 and 5 others.
MarcoAurelio awarded a token.Jan 29 2018, 9:20 AM
hashar mentioned this in T189365: integration-slave-jessie-XXXX is out of disk space.Mar 14 2018, 8:04 AM
hashar mentioned this in T189411: Jenkins randomly fails tests for AbuseFilter.Mar 16 2018, 8:48 AM
greg merged a task: T192860: Error fetching remote repo 'origin'.Apr 23 2018, 11:52 PM
greg added a subscriber: Huji.
greg added a project: Release-Engineering-Team (Kanban).Apr 23 2018, 11:58 PM
In T179963#3917485, @gerritbot wrote:

Change 405722 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Use shallow clone for phplint jobs

https://gerrit.wikimedia.org/r/405722

This issue keeps happening (well, out of space on the jenkins executers at least) but this patch is still open from right before All Hands :). Add it officially to our (too big of a) short-term-ish backlog.

Krinkle mentioned this in T188958: Zuul Clone frequently fails in php70-phan-docker job.Apr 25 2018, 12:56 AM
greg added a comment.Apr 26 2018, 4:49 PM

And again...

gjg@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ du -sh *
511M	analytics-refinery-release
4.0K	analytics-refinery-release@tmp
422M	analytics-refinery-update-jars
4.0K	analytics-refinery-update-jars@tmp
822M	apps-android-wikipedia-publish
4.6G	apps-android-wikipedia-test
120M	commit-message-validator
236M	composer-package-validate
9.0M	debian-glue
1.8M	debian-glue-non-voting
81M	integration-zuul-layoutdiff
14M	integration-zuul-layoutvalidation-gate
481M	mediawiki-core-code-coverage
2.9G	mediawiki-core-php55lint
410M	mediawiki-core-php70lint
80M	mediawiki-vendor-composer-security
70M	mwext-CirrusSearch-whitespaces
764K	mwgate-composer-validate
1.5G	mwgate-php55lint
24M	mwgate-php56lint
1.5G	mwgate-php70lint
21M	operations-dns-lint
1.6M	operations-dns-tabs
236M	operations-mw-config-php55lint
99M	operations-mw-config-typos
236M	operations-puppet-wmf-style-guide
210M	phabricator-jessie-commits
3.4M	phabricator-jessie-debs
210M	phabricator-jessie-diffs
1.3M	php55lint
52M	php56lint
140M	selenium-Wikibase-chrome
541M	wikimedia-fundraising-civicrm
gjg@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ df -h
Filesystem                          Size  Used Avail Use% Mounted on
udev                                 10M     0   10M   0% /dev
tmpfs                               792M   81M  711M  11% /run
/dev/vda3                            19G   15G  3.1G  83% /
tmpfs                               2.0G     0  2.0G   0% /dev/shm
tmpfs                               5.0M     0  5.0M   0% /run/lock
tmpfs                               2.0G     0  2.0G   0% /sys/fs/cgroup
none                                256M  110M  147M  43% /var/lib/mysql
/dev/mapper/vd-second--local--disk   21G   20G     0 100% /srv
none                                256M     0  256M   0% /srv/home/jenkins-deploy/tmpfs
tmpfs                               396M     0  396M   0% /run/user/2947
tmpfs                               396M     0  396M   0% /run/user/11634
tmpfs                               396M     0  396M   0% /run/user/2890
Krinkle subscribed.Apr 26 2018, 11:40 PM

Could this possibly be the cause of fetch/lock failures I'm seeing?

https://integration.wikimedia.org/ci/job/mwgate-php70lint/1140/console

Building remotely on integration-slave-jessie-1001 (DebianGlue contintLabsSlave DebianJessie) in workspace /srv/jenkins-workspace/workspace/mwgate-php70lint
[..] Fetching changes from the remote Git repository
 > git config remote.origin.url git://contint1001.wikimedia.org/mediawiki/extensions/ReplaceText # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://contint1001.wikimedia.org/mediawiki/extensions/ReplaceText  [..]
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://contint1001.wikimedia.org/mediawiki/extensions/ReplaceText" returned status code 4:
stderr: error: failed to write new configuration file .git/config.lock
Krinkle mentioned this in T192702: Skip Quibble "composer" and "sqlite" jobs on wmf branches.May 2 2018, 5:22 PM
Krinkle renamed this task from mwgate-php55lint workspaces are getting huge to Workspaces for mwgate-php55lint / mwgate-php70lint are getting huge.May 2 2018, 5:29 PM
greg added a comment.May 2 2018, 5:42 PM
gjg@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ du -sh
15G	.
gjg@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ du -sh *
422M	analytics-refinery-update-jars
4.0K	analytics-refinery-update-jars@tmp
902M	apps-android-wikipedia-publish
5.0G	apps-android-wikipedia-test
100M	commit-message-validator
236M	composer-package-validate
1.6M	debian-glue-non-voting
4.0K	fail-archived-repositories
82M	integration-zuul-layoutdiff
11M	integration-zuul-layoutvalidation-gate
452M	mediawiki-core-code-coverage-php7
1.9G	mediawiki-core-php55lint
792M	mediawiki-core-php70lint
79M	mediawiki-vendor-composer-security
54M	mwext-CirrusSearch-whitespaces
776K	mwgate-composer-validate
1.4G	mwgate-php55lint
24M	mwgate-php56lint
2.0G	mwgate-php70lint
12M	operations-dns-lint
1.6M	operations-dns-tabs
211M	operations-mw-config-php55lint
100M	operations-mw-config-typos
238M	operations-puppet-wmf-style-guide
210M	phabricator-jessie-commits
4.1M	phabricator-jessie-debs
1.2M	php55lint
157M	php56lint
147M	selenium-Wikibase-chrome
524M	wikimedia-fundraising-civicrm

rm -rf'd again...

Stashbot added a comment.Jul 14 2018, 3:27 AM

Mentioned in SAL (#wikimedia-releng) [2018-07-14T03:27:01Z] <Krinkle> Clearing various workspaces on integration-slave-jessie-1001 to fix operations-mw-config-php55lint Jenkins builds - T179963

Krinkle added a project: Wikimedia-production-error (ARCHIVED -- Shared Build Failure).Jul 14 2018, 3:28 AM
Krinkle moved this task from Untriaged to In progress on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.Aug 9 2018, 10:22 PM
Krinkle mentioned this in T202457: mediawiki-quibble docker jobs fails due to disk full.Aug 21 2018, 8:58 PM
Stashbot added a comment.Aug 27 2018, 1:47 PM

Mentioned in SAL (#wikimedia-releng) [2018-08-27T13:47:38Z] <hashar> updating phplint jobs to use shallow clone AND wipe the workspace | https://gerrit.wikimedia.org/r/#/c/integration/config/+/405722/ | T179963

gerritbot added a comment.Aug 27 2018, 1:49 PM

Change 405722 merged by jenkins-bot:
[integration/config@master] Use shallow clone for phplint jobs

https://gerrit.wikimedia.org/r/405722

hashar closed this task as Resolved.Aug 27 2018, 9:30 PM
hashar claimed this task.

Should be good now. I simply forgot about this task and its patch https://gerrit.wikimedia.org/r/405722

Liuxinyu970226 unsubscribed.Aug 31 2018, 12:04 AM
Krinkle reopened this task as Open.Aug 31 2018, 3:56 AM
@Legoktm wrote at https://gerrit.wikimedia.org/r/405722

Shallow clones break the git-changed-in-head script, so it thinks that literally every file was changed.

Created a revert of this change as https://gerrit.wikimedia.org/r/#/c/integration/config/+/456515/

Krinkle removed a project: Patch-For-Review.Sep 1 2018, 12:47 AM
hashar removed hashar as the assignee of this task.Sep 24 2018, 4:40 PM
Seb35 subscribed.Sep 24 2018, 5:29 PM

git-changed-in-head works with almost-shallow clones: git clone --depth 2 https://gerrit.wikimedia.org/...

hashar added a comment.Sep 24 2018, 5:43 PM

Yup that works most of the time, but --depth 2 is not sufficient when a chain of patchset is being tested. I guess that is what prompted the revert.

I think I will just phase out those jobs entirely, they come from before we had all MediaWiki extensions and skins normalized to use composer test as an entry point with jakub-onderka/php-parallel-lint.

The git-changed-in-head part is an optimization to make it faster when it was used on huge repositories such as mediawiki/core or Wikibase. Nowadays that is handled directly by Quibble.

The only left use case is to lint php files for untrusted users. And eventually we will drop that use case T192217: Remove the "check" pipeline and Zuul's user-filter.

Krinkle lowered the priority of this task from High to Medium.Mar 12 2019, 6:06 PM
Krinkle moved this task from In progress to Resolved on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.

Moving out of active list of issues causing build failures. Might still be an issue long-term, but we seem to be able to manage it mostly through manual checks and depools.

hashar added a comment.Mar 25 2019, 12:05 PM

I am just going to phase out those jobs starting now. There is no point in keeping them nowadays and they are better replaced by using composer test.

gerritbot added a comment.Mar 25 2019, 12:19 PM

Change 498823 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] qrpedia: rm obsolete php linting jobs

https://gerrit.wikimedia.org/r/498823

gerritbot added a project: Patch-For-Review.Mar 25 2019, 12:19 PM

Change 498824 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] phabricator/*: rm obsolete php linting jobs

https://gerrit.wikimedia.org/r/498824

gerritbot added a comment.Mar 25 2019, 12:23 PM

Change 498823 merged by jenkins-bot:
[integration/config@master] qrpedia: rm obsolete php linting jobs

https://gerrit.wikimedia.org/r/498823

gerritbot added a comment.Mar 25 2019, 12:23 PM

Change 498824 merged by jenkins-bot:
[integration/config@master] phabricator/*: rm obsolete php linting jobs

https://gerrit.wikimedia.org/r/498824

gerritbot added a comment.Mar 25 2019, 12:46 PM

Change 498832 had a related patch set uploaded (by Hashar; owner: Hashar):
[analytics/wmde/scripts@master] build: add php-parallel-lint

https://gerrit.wikimedia.org/r/498832

gerritbot added a comment.Mar 25 2019, 12:55 PM

Change 498832 merged by jenkins-bot:
[analytics/wmde/scripts@master] build: add php-parallel-lint

https://gerrit.wikimedia.org/r/498832

gerritbot added a comment.Mar 25 2019, 12:57 PM

Change 498836 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] labs/toollabs rm php lint jobs

https://gerrit.wikimedia.org/r/498836

gerritbot added a comment.Mar 25 2019, 12:59 PM

Change 498836 merged by jenkins-bot:
[integration/config@master] labs/toollabs rm php lint jobs

https://gerrit.wikimedia.org/r/498836

gerritbot added a comment.Mar 25 2019, 1:01 PM

Change 498838 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] mediawiki/tools/release rm php lint jobs

https://gerrit.wikimedia.org/r/498838

gerritbot added a comment.Mar 25 2019, 1:03 PM

Change 498838 merged by jenkins-bot:
[integration/config@master] mediawiki/tools/release rm php lint jobs

https://gerrit.wikimedia.org/r/498838

gerritbot added a comment.Mar 25 2019, 1:09 PM

Change 498841 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] labs/tools/Luke081515IRCBot switch to composer test

https://gerrit.wikimedia.org/r/498841

gerritbot added a comment.Mar 25 2019, 1:10 PM

Change 498841 merged by jenkins-bot:
[integration/config@master] labs/tools/Luke081515IRCBot switch to composer test

https://gerrit.wikimedia.org/r/498841

hashar mentioned this in rADES5b669dfb14f7: build: add php-parallel-lint.Mar 25 2019, 1:18 PM
gerritbot added a comment.Mar 25 2019, 1:19 PM

Change 498847 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove mediawiki/tools/code-utils

https://gerrit.wikimedia.org/r/498847

gerritbot added a comment.Mar 25 2019, 1:22 PM

Change 498847 merged by jenkins-bot:
[integration/config@master] Remove mediawiki/tools/code-utils

https://gerrit.wikimedia.org/r/498847

gerritbot added a comment.Mar 25 2019, 1:30 PM

Change 498855 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] DonationInterface: drop php 5.6 linting job

https://gerrit.wikimedia.org/r/498855

gerritbot added a comment.Mar 25 2019, 1:32 PM

Change 498855 merged by jenkins-bot:
[integration/config@master] DonationInterface: drop php 5.6 linting job

https://gerrit.wikimedia.org/r/498855

gerritbot added a comment.Mar 25 2019, 1:33 PM

Change 498857 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove PHP lint jobs from MediaWiki repos

https://gerrit.wikimedia.org/r/498857

gerritbot added a comment.Mar 25 2019, 1:36 PM

Change 498857 merged by jenkins-bot:
[integration/config@master] Remove PHP lint jobs from MediaWiki repos

https://gerrit.wikimedia.org/r/498857

gerritbot added a comment.Mar 25 2019, 2:18 PM

Change 498871 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] mediawiki-config: replace legacy php lint jobs

https://gerrit.wikimedia.org/r/498871

gerritbot added a comment.Mar 25 2019, 2:22 PM

Change 498871 merged by jenkins-bot:
[integration/config@master] mediawiki-config: replace legacy php lint jobs

https://gerrit.wikimedia.org/r/498871

gerritbot added a comment.Mar 25 2019, 2:52 PM

Change 498894 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Drop legacy php56lint from civicrm

https://gerrit.wikimedia.org/r/498894

gerritbot added a comment.Mar 25 2019, 3:00 PM

Change 498897 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove php lint templates/macro etc

https://gerrit.wikimedia.org/r/498897

gerritbot added a comment.Mar 25 2019, 3:01 PM

Change 498894 merged by jenkins-bot:
[integration/config@master] Drop legacy php56lint from civicrm

https://gerrit.wikimedia.org/r/498894

hashar closed this task as Resolved.Mar 25 2019, 3:12 PM
hashar claimed this task.

The jobs are gone!

gerritbot added a comment.Mar 25 2019, 3:15 PM

Change 498897 merged by jenkins-bot:
[integration/config@master] Remove php lint templates/macro etc

https://gerrit.wikimedia.org/r/498897

gerritbot added a comment.Apr 26 2019, 10:15 AM

Change 506627 had a related patch set uploaded (by Ladsgroup; owner: Hashar):
[analytics/wmde/scripts@production] build: add php-parallel-lint

https://gerrit.wikimedia.org/r/506627

gerritbot added a comment.Apr 26 2019, 10:15 AM

Change 506627 merged by jenkins-bot:
[analytics/wmde/scripts@production] build: add php-parallel-lint

https://gerrit.wikimedia.org/r/506627

Ladsgroup mentioned this in rADES00ed76fd7475: build: add php-parallel-lint.Apr 26 2019, 10:15 AM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 28 2019, 11:46 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL