Page MenuHomePhabricator

document quibble cache / src reuse for local installation
Closed, ResolvedPublic

Description

5:59:11 <zeljkof> hashar: running in the background while doing other things, because it's slow, will copy/paste output and create a task if it's a bug
16:02:06 <hashar> usually the main sources of speed up are:
16:02:43 <hashar> 1) have a "cache" directory on the host and bind mount it in the docker container: mkdir cache; chmod 777 cache . Then pass to docker run: -v "$(pwd)/cache:/cache"
16:03:20 <hashar> 2) use git bare repositories on the host (see README.md which get them created on the host in a directory named ref). Then bind mount them read only in the container at /srv/git : -v "$(pwd)/ref:/srv/git:ro"
16:03:53 <hashar> zuul-cloner will try to find a git repo there and clone from it. That is a local clone and is fast. Then zuul-cloner refresh from Gerrit
16:04:41 <hashar> 3) have a src directory on the host which is bind mounted in the container. This way once the repositories have been cloned and npm install / composer install have been run, you can skip those steps
16:05:04 <hashar> so that is something like: mkdir src; chmod 777 src; docker run -v "$(pwd)/src:/src"
16:05:08 <hashar> and for the enxt run:
16:05:17 <hashar> docker run -v "$(pwd)/src:/src" --skip-zuul --skip-deps

This is recommended in the readme, but does not work on my machine (Mac):

~/Documents/gerrit/integration/quibble$ install --directory --mode 777 cache
install: illegal option -- -
usage: install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]
               [-o owner] file1 file2
       install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]
               [-o owner] file1 ... fileN directory
       install -d [-v] [-g group] [-m mode] [-o owner] directory ...

I guess it should be replaced with your recommendation from IRC:

mkdir cache; chmod 777 cache

I am having trouble with --skip-deps (T193164#4161300) and --skip-zuul (T193164#4161290).

Related Objects

Event Timeline

hashar created this task.Apr 26 2018, 2:06 PM
hashar added a project: Documentation.
~/Documents/gerrit/integration/quibble$ time docker run -it quibble
...
real	16m40.402s
user	0m0.603s
sys	0m0.700s
zeljkofilipin added a comment.EditedApr 26 2018, 2:22 PM
~/Documents/gerrit/integration/quibble$ mkdir cache; chmod 777 cache

~/Documents/gerrit/integration/quibble$ time docker run -it -v "$(pwd)"/ref:/srv/git:ro -v "$(pwd)"/cache:/cache quibble
...
real	7m38.562s
user	0m0.484s
sys	0m0.560s
zeljkofilipin added a comment.EditedApr 26 2018, 2:24 PM
~/Documents/gerrit/integration/quibble$ mkdir -p ref/mediawiki/skins

~/Documents/gerrit/integration/quibble$ git clone --bare https://gerrit.wikimedia.org/r/mediawiki/core ref/mediawiki/core.git
...

~/Documents/gerrit/integration/quibble$ git clone --bare https://gerrit.wikimedia.org/r/mediawiki/vendor ref/mediawiki/vendor.git
...

~/Documents/gerrit/integration/quibble$ git clone --bare https://gerrit.wikimedia.org/r/mediawiki/skins/Vector ref/mediawiki/skins/Vector.git 
...

~/Documents/gerrit/integration/quibble$ time docker run -it -v "$(pwd)"/ref:/srv/git:ro -v "$(pwd)"/cache:/cache quibble --skip-zuul
DEBUG:quibble.cmd:Running stages: all
INFO:quibble.cmd:Adding mediawiki/vendor
INFO:quibble.cmd:Projects: mediawiki/core, mediawiki/skins/Vector, mediawiki/vendor
WARNING:quibble.cmd:ZUUL_PROJECT not set. Assuming mediawiki/core
INFO:backend.MySQL:Initializing MySQL data directory
INFO:backend.MySQL:Starting MySQL
INFO:backend.MySQL:Waiting for MySQL socket
INFO:backend.MySQL:Creating the wiki database and grant
INFO:backend.MySQL:MySQL is ready
INFO:mw.maintenance.install:php maintenance/install.php --scriptpath= --dbtype=mysql --dbname=wikidb --dbuser=wikiuser --dbpass=secret --dbserver=localhost:/tmp/quibble-mysql-ypf2c58b/socket --with-extensions --pass=testpass TestWiki WikiAdmin
Traceback (most recent call last):
  File "/usr/local/bin/quibble", line 11, in <module>
    load_entry_point('quibble==0.0.0', 'console_scripts', 'quibble')()
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/cmd.py", line 405, in main
    cmd.execute()
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/cmd.py", line 328, in execute
    self.mw_install()
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/cmd.py", line 193, in mw_install
    mwdir=self.mw_install_path
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/mediawiki/maintenance.py", line 34, in install
    p = subprocess.Popen(cmd, cwd=mwdir, env={'LANG': 'C.UTF-8'})
  File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/workspace/src'
INFO:backend.MySQL:Terminating MySQL

real	0m14.236s
user	0m0.073s
sys	0m0.020s
zeljkofilipin added a comment.EditedApr 26 2018, 2:26 PM
~/Documents/gerrit/integration/quibble$ time docker run -it -v "$(pwd)"/ref:/srv/git:ro -v "$(pwd)"/cache:/cache quibble --skip-deps
DEBUG:quibble.cmd:Running stages: all
INFO:quibble.cmd:Adding mediawiki/vendor
INFO:quibble.cmd:Projects: mediawiki/core, mediawiki/skins/Vector, mediawiki/vendor
INFO:zuul.CloneMapper:Workspace path set to: /workspace/src
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper:  mediawiki/core -> /workspace/src
INFO:zuul.CloneMapper:  mediawiki/skins/Vector -> /workspace/src/skins/Vector
INFO:zuul.CloneMapper:  mediawiki/vendor -> /workspace/src/vendor
DEBUG:zuul.CloneMapper:Checking overlap in destination directories...
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 3 repositories
INFO:zuul.Cloner:Creating repo mediawiki/core from cache /srv/git/mediawiki/core.git
INFO:zuul.Cloner:Updating origin remote in repo mediawiki/core to https://gerrit.wikimedia.org/r/p/mediawiki/core
DEBUG:zuul.Repo:Resetting repository /workspace/src
DEBUG:zuul.Repo:Updating repository /workspace/src
INFO:zuul.Cloner:Falling back to branch master
DEBUG:zuul.Repo:Checking out remotes/origin/master
INFO:zuul.Cloner:Prepared mediawiki/core repo with branch master at commit b1b0070dc7557311cc38e7cc99e4231ac4b92940
INFO:zuul.Cloner:Creating repo mediawiki/skins/Vector from cache /srv/git/mediawiki/skins/Vector.git
INFO:zuul.Cloner:Updating origin remote in repo mediawiki/skins/Vector to https://gerrit.wikimedia.org/r/p/mediawiki/skins/Vector
DEBUG:zuul.Repo:Resetting repository /workspace/src/skins/Vector
DEBUG:zuul.Repo:Updating repository /workspace/src/skins/Vector
INFO:zuul.Cloner:Falling back to branch master
DEBUG:zuul.Repo:Checking out remotes/origin/master
INFO:zuul.Cloner:Prepared mediawiki/skins/Vector repo with branch master at commit 317467b7ef70132a6103109e900b4161bd3ff1bb
INFO:zuul.Cloner:Creating repo mediawiki/vendor from cache /srv/git/mediawiki/vendor.git
INFO:zuul.Cloner:Updating origin remote in repo mediawiki/vendor to https://gerrit.wikimedia.org/r/p/mediawiki/vendor
DEBUG:zuul.Repo:Resetting repository /workspace/src/vendor
DEBUG:zuul.Repo:Updating repository /workspace/src/vendor
INFO:zuul.Cloner:Falling back to branch master
DEBUG:zuul.Repo:Checking out remotes/origin/master
INFO:zuul.Cloner:Prepared mediawiki/vendor repo with branch master at commit e405557577643ea9c5f4777acb87a92d1f05e62e
INFO:zuul.Cloner:Prepared all repositories
WARNING:quibble.cmd:ZUUL_PROJECT not set. Assuming mediawiki/core
INFO:backend.MySQL:Initializing MySQL data directory
INFO:backend.MySQL:Starting MySQL
INFO:backend.MySQL:Waiting for MySQL socket
INFO:backend.MySQL:Creating the wiki database and grant
INFO:backend.MySQL:MySQL is ready
INFO:mw.maintenance.install:php maintenance/install.php --scriptpath= --dbtype=mysql --dbname=wikidb --dbuser=wikiuser --dbpass=secret --dbserver=localhost:/tmp/quibble-mysql-5wz2c62y/socket --with-extensions --pass=testpass TestWiki WikiAdmin
PHP 7.0.27-0+deb9u1 is installed.
Found ImageMagick: /usr/bin/convert. Image thumbnailing will be enabled if you enable uploads.
Found the Git version control software: /usr/bin/git.
Using server URL "".
Warning: Your default directory for uploads (/workspace/src/images/) is not checked for vulnerability to arbitrary script execution during the CLI install.
Using the intl PECL extension for Unicode normalization.
The environment has been checked. You can install MediaWiki.
Setting up database
done
Creating tables
done
Creating database user
done
Populating default interwiki table
done
Initializing statistics
done
Generating secret keys
done
Prevent running unneeded updates
done
Creating administrator user account
done
Creating main page with default content
done
MediaWiki has been successfully installed. You can now visit <> to view your wiki. If you have questions, check out our frequently asked questions list: <https://www.mediawiki.org/wiki/Manual:FAQ> or use one of the support forums linked on that page.
Proceeding '00_dev_settings.php'...
Proceeding '10_env_mw_install_path.php'...
Proceeding '10_set_wgWikimediaJenkinsCI.php'...
Proceeding '20_set_wgdjvu.php'...
No syntax errors detected in /workspace/src/LocalSettings.php
INFO:quibble.cmd:Copying /workspace/src/LocalSettings.php to /log/LocalSettings.php
INFO:quibble.cmd:mediawiki/vendor used. Skipping external dependencies
INFO:mw.maintenance.update:php maintenance/update.php --quick --skip-external-dependencies
MediaWiki 1.32.0-alpha Updater

Skipping checking whether external dependencies are up to date, proceed at your own risk
Going to run database updates for wikidb
Depending on the size of your database this may take a while!
Turning off Content Handler DB fields for this part of upgrade.
...have ipb_id field in ipblocks table.
...have ipb_expiry field in ipblocks table.
...already have interwiki table
...indexes seem up to 20031107 standards.
...have rc_type field in recentchanges table.
...index new_name_timestamp already set on recentchanges table.
...have user_real_name field in user table.
...querycache table already exists.
...objectcache table already exists.
...categorylinks table already exists.
...have pagelinks; skipping old links table updates
...il_from OK
...have rc_ip field in recentchanges table.
...index PRIMARY already set on image table.
...have rc_id field in recentchanges table.
...have rc_patrolled field in recentchanges table.
...logging table already exists.
...have user_token field in user table.
...have wl_notificationtimestamp field in watchlist table.
...watchlist talk page rows already present.
...user table does not contain user_emailauthenticationtimestamp field.
...page table already exists.
...have log_params field in logging table.
...logging table has correct log_title encoding.
...have ar_rev_id field in archive table.
...have page_len field in page table.
...revision table does not contain inverse_timestamp field.
...have rev_text_id field in revision table.
...have rev_deleted field in revision table.
...have img_width field in image table.
...have img_metadata field in image table.
...have user_email_token field in user table.
...have ar_text_id field in archive table.
...page_namespace is already a full int (int(11)).
...ar_namespace is already a full int (int(11)).
...rc_namespace is already a full int (int(11)).
...wl_namespace is already a full int (int(11)).
...qc_namespace is already a full int (int(11)).
...log_namespace is already a full int (int(11)).
...have img_media_type field in image table.
...already have pagelinks table.
...image table does not contain img_type field.
...already have unique user_name index.
...user_groups table exists and is in current format.
...have ss_total_pages field in site_stats table.
...user_newtalk table already exists.
...transcache table already exists.
...have iw_trans field in interwiki table.
...wl_notificationtimestamp is already nullable.
...index times already set on logging table.
...have ipb_range_start field in ipblocks table.
...no page_random rows needed to be set
...have user_registration field in user table.
...templatelinks table already exists
...externallinks table already exists.
...job table already exists.
...have ss_images field in site_stats table.
...langlinks table already exists.
...querycache_info table already exists.
...filearchive table already exists.
...have ipb_anon_only field in ipblocks table.
...index rc_ns_usertext already set on recentchanges table.
...index rc_user_text already set on recentchanges table.
...have user_newpass_time field in user table.
...redirect table already exists.
...querycachetwo table already exists.
...have ipb_enable_autoblock field in ipblocks table.
...index pl_namespace on table pagelinks includes field pl_from.
...index tl_namespace on table templatelinks includes field tl_from.
...index il_to on table imagelinks includes field il_from.
...have rc_old_len field in recentchanges table.
...have user_editcount field in user table.
...page_restrictions table already exists.
...have log_id field in logging table.
...have rev_parent_id field in revision table.
...have pr_id field in page_restrictions table.
...have rev_len field in revision table.
...have rc_deleted field in recentchanges table.
...have log_deleted field in logging table.
...have ar_deleted field in archive table.
...have ipb_deleted field in ipblocks table.
...have fa_deleted field in filearchive table.
...have ar_len field in archive table.
...have ipb_block_email field in ipblocks table.
...index cl_sortkey on table categorylinks includes field cl_from.
...have oi_metadata field in oldimage table.
Adding index usertext_timestamp to table archive ...done.
...index img_usertext_timestamp already set on image table.
...index oi_usertext_timestamp already set on oldimage table.
...have ar_page_id field in archive table.
...have img_sha1 field in image table.
...protected_titles table already exists.
...have ipb_by_text field in ipblocks table.
...page_props table already exists.
...updatelog table already exists.
...category table already exists.
Populating category table, printing progress markers. For large databases, you
may want to hit Ctrl-C and do this manually with maintenance/
populateCategory.php.
Category population complete.
Done populating category table.
...have ar_parent_id field in archive table.
...have user_last_timestamp field in user_newtalk table.
Populating rev_parent_id fields, printing progress markers. For large
databases, you may want to hit Ctrl-C and do this manually with
maintenance/populateParentId.php.
Populating rev_parent_id column
...doing rev_id from 1 to 200
rev_parent_id population complete ... 0 rows [0 changed]
...protected_titles table has correct pt_title encoding.
...have ss_active_users field in site_stats table.
...ss_active_users user count set...
...have ipb_allow_usertalk field in ipblocks table.
...change_tag table already exists.
...tag_summary table already exists.
...valid_tag table already exists.
...user_properties table already exists.
...log_search table already exists.
...have log_user_text field in logging table.
Populating log_user_text field, printing progress markers. For large
databases, you may want to hit Ctrl-C and do this manually with
maintenance/populateLogUsertext.php.
Nothing to do.
done.
Populating log_search table, printing progress markers. For large
databases, you may want to hit Ctrl-C and do this manually with
maintenance/populateLogSearch.php.
Nothing to do.
done.
...l10n_cache table already exists.
...index change_tag_rc_tag already set on change_tag table.
...have rd_interwiki field in redirect table.
Converting tc_time from UNIX epoch to MediaWiki timestamp ...done.
Altering all *_mime_minor fields to 100 bytes in size ...done.
...iwlinks table already exists.
...index iwl_prefix_title_from already set on iwlinks table.
...have ul_value field in updatelog table.
...have iw_api field in interwiki table.
...iwl_prefix key doesn't exist.
...have cl_collation field in categorylinks table.
Updating categorylinks (again) ...done.
...module_deps table already exists.
...ar_page_revid key doesn't exist.
...index ar_revid already set on archive table.
...ll_lang is up-to-date.
...user_last_timestamp is already nullable.
...index user_email already set on user table.
...up_property in table user_properties already modified by patch patch-up_property.sql.
...uploadstash table already exists.
...user_former_groups table already exists.
...index type_action already set on logging table.
...have rev_sha1 field in revision table.
...batch conversion of user_options: nothing to migrate. done.
...user table does not contain user_options field.
...have ar_sha1 field in archive table.
...index page_redirect_namespace_len already set on page table.
...have us_chunk_inx field in uploadstash table.
...have job_timestamp field in job table.
...index page_user_timestamp already set on revision table.
...have ipb_parent_block_id field in ipblocks table.
...index ipb_parent_block_id already set on ipblocks table.
...category table does not contain cat_hidden field.
...have rev_content_format field in revision table.
...have rev_content_model field in revision table.
...have ar_content_format field in archive table.
...have ar_content_model field in archive table.
...have page_content_model field in page table.
Content Handler DB fields should be usable now.
...site_stats table does not contain ss_admins field.
...recentchanges table does not contain rc_moved_to_title field.
...sites table already exists.
...have fa_sha1 field in filearchive table.
...have job_token field in job table.
...have job_attempts field in job table.
...have us_props field in uploadstash table.
...ug_group in table user_groups already modified by patch patch-ug_group-length-increase-255.sql.
...ufg_group in table user_former_groups already modified by patch patch-ufg_group-length-increase-255.sql.
...index pp_propname_page already set on page_props table.
...index img_media_mime already set on image table.
...iwl_prefix_title_from index is already non-UNIQUE.
...index iwl_prefix_from_title already set on iwlinks table.
...have ar_id field in archive table.
...have el_id field in externallinks table.
...have rc_source field in recentchanges table.
...index log_user_text_type_time already set on logging table.
...index log_user_text_time already set on logging table.
...have page_links_updated field in page table.
...have user_password_expires field in user table.
...have pp_sortkey field in page_props table.
...recentchanges table does not contain rc_cur_time field.
...index wl_user_notificationtimestamp already set on watchlist table.
...have page_lang field in page table.
...have pl_from_namespace field in pagelinks table.
...have tl_from_namespace field in templatelinks table.
...have il_from_namespace field in imagelinks table.
...img_major_mime in table image already modified by patch patch-img_major_mime-chemical.sql.
...oi_major_mime in table oldimage already modified by patch patch-oi_major_mime-chemical.sql.
...fa_major_mime in table filearchive already modified by patch patch-fa_major_mime-chemical.sql.
...comment fields are up to date...hitcounter doesn't exist.
...site_stats table does not contain ss_total_views field.
...page table does not contain page_counter field.
...msg_resource_links doesn't exist.
...msg_resource doesn't exist.
...bot_passwords table already exists.
...have wl_id field in watchlist table.
...cl_collation key doesn't exist.
...index cl_collation_ext already set on categorylinks table.
...collations up-to-date.
...index rc_name_type_patrolled_timestamp already set on recentchanges table.
...rev_page_id index already non-unique.
...pl_namespace, tl_namespace, il_to indices are already non-UNIQUE.
...have ct_id field in change_tag table.
...have ts_id field in tag_summary table.
Modifying rc_ip field of table recentchanges ...done.
...index usertext_timestamp already set on archive table.
...have el_index_60 field in externallinks table.
...ug_user_group key doesn't exist.
...have ug_expiry field in user_groups table.
...index img_user_timestamp already set on image table.
Modifying img_media_type field of table image ...done.
...ip_changes table already exists.
...index PRIMARY already set on categorylinks table.
...index PRIMARY already set on templatelinks table.
...index PRIMARY already set on pagelinks table.
...index PRIMARY already set on text table.
...index PRIMARY already set on imagelinks table.
...index PRIMARY already set on iwlinks table.
...index PRIMARY already set on langlinks table.
...index PRIMARY already set on log_search table.
...index PRIMARY already set on module_deps table.
...index PRIMARY already set on objectcache table.
...index PRIMARY already set on querycache_info table.
...index PRIMARY already set on site_stats table.
...index PRIMARY already set on transcache table.
...index PRIMARY already set on user_former_groups table.
...index PRIMARY already set on user_properties table.
...comment table already exists.
...have img_description_id field in image table.
...index PRIMARY already set on l10n_cache table.
...bot_passwords.bp_user is already unsigned int.
...change_tag.ct_log_id is already unsigned int.
...change_tag.ct_rev_id is already unsigned int.
...page_restrictions.pr_user is already unsigned int.
...tag_summary.ts_log_id is already unsigned int.
...tag_summary.ts_rev_id is already unsigned int.
...user_newtalk.user_id is already unsigned int.
...user_properties.up_user is already unsigned int.
...slots table already exists.
...have slot_origin field in slots table.
...content table already exists.
...slot_roles table already exists.
...content_models table already exists.
...actor table already exists.
Modifying rev_text_id field of table revision ...done.
Modifying table site_stats ...done.
...index rc_namespace_title_timestamp already set on recentchanges table.
...site_stats is populated...done.
Checking existence of old default messages...done.
Populating rev_len column
...doing rev_id from 1 to 200
Populating ar_len column
...archive table seems to be empty.
rev_len and ar_len population complete [0 revision rows, 0 archive rows].
Populating rev_sha1 column
...doing rev_id from 1 to 200
Populating ar_sha1 column
...archive table seems to be empty.
Populating ar_sha1 column legacy rows
rev_sha1 and ar_sha1 population complete [0 revision rows, 0 archive rows].
Populating img_sha1 field

Done 0 files in 0.0 seconds
Fixing protocol-relative entries in the externallinks table...
Done, 0 rows updated.
Populating fa_sha1 field from fa_storage_key

Done 0 files in 0.0 seconds
Updating *_from_namespace fields in links tables.
...doing page_id from 1 to 200
Adding empty categories with description pages...
Removing empty categories without description pages...
Category cleanup complete.
Populating page_props.pp_sortkey...
Populating page_props.pp_sortkey complete.
Copying IP revisions to ip_changes, from rev_id 0 to rev_id 1
...checking 1 revisions for IP edits that need copying, between rev_ids 0 and 1
Attempted to insert 0 IP revisions, 0 actually done.
Purging caches...done.

Done in 0.3 s.
INFO:quibble.cmd:PHPUnit without Database group
INFO:test.run_phpunit:php tests/phpunit/phpunit.php --debug-tests --exclude-group Broken,ParserFuzz,Stub,Database
PHPUnit not found. Please install it and other dev dependencies by
		running `composer install` in MediaWiki root directory.
Traceback (most recent call last):
  File "/usr/local/bin/quibble", line 11, in <module>
    load_entry_point('quibble==0.0.0', 'console_scripts', 'quibble')()
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/cmd.py", line 405, in main
    cmd.execute()
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/cmd.py", line 343, in execute
    mwdir=self.mw_install_path)
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/test.py", line 131, in run_phpunit_databaseless
    run_phpunit(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/test.py", line 121, in run_phpunit
    subprocess.check_call(cmd, cwd=mwdir, env={'LANG': 'C.UTF-8'})
  File "/usr/lib/python3.5/subprocess.py", line 271, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['php', 'tests/phpunit/phpunit.php', '--debug-tests', '--exclude-group', 'Broken,ParserFuzz,Stub,Database']' returned non-zero exit status 1
INFO:backend.MySQL:Terminating MySQL

real	0m37.059s
user	0m0.080s
sys	0m0.036s
~/Documents/gerrit/integration/quibble$ time docker run -it -v "$(pwd)"/ref:/srv/git:ro -v "$(pwd)"/cache:/cache quibble --run selenium
...
real	3m2.853s
user	0m0.147s
sys	0m0.115s
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)

The git clone properly uses /srv/git so that is a start. The thing is all the docker run above are using a new source tree. So if you --skip-zuul there is no source tree and if you use --skip-deps composer/npm install are not run. Thus the command fails.

With src mounted, you will want to use quibble once to have the repository cloned and the dependencies installed. They will be written to the src volume and kept on the host even after the docker container is deleted.

Then on subsequent run, since src now has the repo checked out and the dependencies, you can skip both zuul and deps: -v "$(pwd)/src":/workspace/src --skip-zuul --skip-deps.

zeljkofilipin added a comment.EditedApr 26 2018, 3:27 PM
~/Documents/gerrit/integration/quibble$ time docker run -it -v "$(pwd)/src:/workspace/src" -v "$(pwd)"/ref:/srv/git:ro -v "$(pwd)"/cache:/cache quibble
DEBUG:quibble.cmd:Running stages: all
INFO:quibble.cmd:Adding mediawiki/vendor
INFO:quibble.cmd:Projects: mediawiki/core, mediawiki/skins/Vector, mediawiki/vendor
INFO:zuul.CloneMapper:Workspace path set to: /workspace/src
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper:  mediawiki/core -> /workspace/src
INFO:zuul.CloneMapper:  mediawiki/skins/Vector -> /workspace/src/skins/Vector
INFO:zuul.CloneMapper:  mediawiki/vendor -> /workspace/src/vendor
DEBUG:zuul.CloneMapper:Checking overlap in destination directories...
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 3 repositories
INFO:zuul.Cloner:Creating repo mediawiki/core from upstream https://gerrit.wikimedia.org/r/p/mediawiki/core
DEBUG:zuul.Repo:Resetting repository /workspace/src
DEBUG:zuul.Repo:Updating repository /workspace/src
Traceback (most recent call last):
  File "/usr/local/bin/quibble", line 11, in <module>
    load_entry_point('quibble==0.0.0', 'console_scripts', 'quibble')()
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/cmd.py", line 405, in main
    cmd.execute()
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/cmd.py", line 300, in execute
    self.clone(projects_to_clone)
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/cmd.py", line 142, in clone
    cache_dir=self.args.git_cache)
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/quibble/zuul.py", line 47, in clone
    return zuul_cloner.execute()
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/zuul/lib/cloner.py", line 75, in execute
    self.prepareRepo(project, dest)
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/zuul/lib/cloner.py", line 169, in prepareRepo
    repo.reset()
  File "/usr/local/lib/python3.5/dist-packages/quibble-0.0.0-py3.5.egg/zuul/merger/merger.py", line 102, in reset
    for ref in origin.refs:
  File "/usr/lib/python3/dist-packages/git/remote.py", line 527, in refs
    out_refs.extend(RemoteReference.list_items(self.repo, remote=self.name))
  File "/usr/lib/python3/dist-packages/git/util.py", line 932, in list_items
    out_list.extend(cls.iter_items(repo, *args, **kwargs))
  File "/usr/lib/python3/dist-packages/git/refs/symbolic.py", line 592, in _iter_items
    for sha, rela_path in cls._iter_packed_refs(repo):  # @UnusedVariable
  File "/usr/lib/python3/dist-packages/git/refs/symbolic.py", line 92, in _iter_packed_refs
    raise TypeError("PackingType of packed-Refs not understood: %r" % line)
TypeError: PackingType of packed-Refs not understood: '# pack-refs with: peeled fully-peeled sorted'

real	0m8.760s
user	0m0.069s
sys	0m0.017s
hashar moved this task from Backlog to Defect on the Quibble board.Apr 27 2018, 9:47 PM
hashar moved this task from Defect to Enhancement on the Quibble board.
hashar moved this task from Enhancement to Defect on the Quibble board.

Change 428628 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Convert doc to Sphinx

https://gerrit.wikimedia.org/r/428628

hashar claimed this task.Apr 30 2018, 10:11 AM

Finally paired that one with @zeljkofilipin this morning. Doc is polished up https://gerrit.wikimedia.org/r/#/c/428628/3..5/README.rst and ultimately will end up being published on https://doc.wikimedia.org/ .

zeljkofilipin triaged this task as Normal priority.Apr 30 2018, 2:14 PM

Change 428628 merged by jenkins-bot:
[integration/quibble@master] Convert doc to Sphinx

https://gerrit.wikimedia.org/r/428628

Change 430795 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Publish doc for integration/quibble

https://gerrit.wikimedia.org/r/430795

Change 430795 merged by jenkins-bot:
[integration/config@master] Publish doc for integration/quibble

https://gerrit.wikimedia.org/r/430795

hashar closed this task as Resolved.May 3 2018, 9:11 PM

And the doc is online: https://doc.wikimedia.org/quibble/

Vvjjkkii renamed this task from document quibble cache / src reuse for local installation to n5daaaaaaa.Jul 1 2018, 1:13 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed hashar as the assignee of this task.
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: gerritbot.
CommunityTechBot renamed this task from n5daaaaaaa to document quibble cache / src reuse for local installation.Jul 2 2018, 3:07 PM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to hashar.
CommunityTechBot lowered the priority of this task from High to Normal.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: gerritbot.

Change 446772 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/docroot@master] doc: add link to Quibble

https://gerrit.wikimedia.org/r/446772

Change 446772 merged by jenkins-bot:
[integration/docroot@master] doc: add link to Quibble

https://gerrit.wikimedia.org/r/446772