Page MenuHomePhabricator

ORES MediaWiki-Vagrant roles should be ported to Stretch
Closed, ResolvedPublic

Details

Related Gerrit Patches:

Event Timeline

awight created this task.Jan 3 2018, 3:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2018, 3:42 PM
awight claimed this task.Jan 3 2018, 7:36 PM
awight added a project: Wikilabels.
awight added a comment.Jan 3 2018, 7:42 PM

Remove /vagrant/srv/ores before attempting the migration.

awight added a comment.Jan 3 2018, 7:54 PM

Small glitch, probably a missing ordering:

1==> default: Info: /Stage[main]/Wikilabels/Virtualenv::Package[wikilabels]/Exec[pip_install_wikilabels_dependencies_in_/vagrant/srv/wikilabels]: Scheduling refresh of Exec[pip_install_wikilabels_editable_in_/vagrant/srv/wikilabels]
2==> default: Notice: /Stage[main]/Wikilabels/Virtualenv::Package[wikilabels]/Exec[pip_install_wikilabels_editable_in_/vagrant/srv/wikilabels]: Triggered 'refresh' from 1 events
3==> default: Notice: /Stage[main]/Wikilabels/Exec[create wikilabels database]/returns: executed successfully
4==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: INFO:wikilabels.utilities.load_schema:Loading schema...
5==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: INFO:wikilabels.database.db:Initializing connection pool.
6==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: Traceback (most recent call last):
7==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/bin/wikilabels", line 11, in <module>
8==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: load_entry_point('wikilabels', 'console_scripts', 'wikilabels')()
9==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/src/wikilabels/wikilabels/wikilabels.py", line 49, in main
10==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: module.main(sys.argv[2:])
11==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/src/wikilabels/wikilabels/utilities/load_schema.py", line 39, in main
12==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: run(db, reload_test_data)
13==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/src/wikilabels/wikilabels/utilities/load_schema.py", line 50, in run
14==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: db.execute(schema_sql)
15==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/src/wikilabels/wikilabels/database/db.py", line 34, in execute
16==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: with self.transaction() as transactor:
17==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
18==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: return next(self.gen)
19==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/src/wikilabels/wikilabels/database/db.py", line 42, in transaction
20==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: self._initialize_pool()
21==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/src/wikilabels/wikilabels/database/db.py", line 31, in _initialize_pool
22==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: *args, cursor_factory=RealDictCursor, **kwargs)
23==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/lib/python3.5/site-packages/psycopg2/pool.py", line 161, in __init__
24==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: self, minconn, maxconn, *args, **kwargs)
25==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/lib/python3.5/site-packages/psycopg2/pool.py", line 58, in __init__
26==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: self._connect()
27==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/lib/python3.5/site-packages/psycopg2/pool.py", line 62, in _connect
28==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: conn = psycopg2.connect(*self._args, **self._kwargs)
29==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: File "/vagrant/srv/wikilabels/lib/python3.5/site-packages/psycopg2/__init__.py", line 130, in connect
30==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
31==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: psycopg2.OperationalError: FATAL: password authentication failed for user "wikilabels"
32==> default: Notice: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: FATAL: password authentication failed for user "wikilabels"
33==> default: Error: echo y | sudo -u www-data /vagrant/srv/wikilabels/bin/wikilabels load_schema --reload-test-data returned 1 instead of one of [0]
34==> default: Error: /Stage[main]/Wikilabels/Exec[initialize wikilabels database]/returns: change from notrun to 0 failed: echo y | sudo -u www-data /vagrant/srv/wikilabels/bin/wikilabels load_schema --reload-test-data returned 1 instead of one of [0]

awight added a comment.Jan 3 2018, 7:56 PM

The ordering looks good, maybe this was a race condition? Goes away with reprovisioning.

awight added a comment.Jan 3 2018, 7:58 PM

Also worked fine when I dropped the wikilabels database and reprovisioned. Ignoring unless it happens again :-/

awight added a comment.Jan 3 2018, 8:07 PM

lol,

Message from syslogd@mediawikivagrant at Jan  3 20:07:27 ...
 kernel:[ 4092.405027] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ffffffffc04b790f

Message from syslogd@mediawikivagrant at Jan  3 20:07:27 ...
 kernel:[ 4092.405027] 

Message from syslogd@mediawikivagrant at Jan  3 20:07:27 ...
 kernel:[ 4092.405486] Kernel Offset: 0x3d800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
awight moved this task from Backlog to In Progress on the MediaWiki-Vagrant board.Jan 3 2018, 8:08 PM

Change 401791 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/vagrant@stretch-migration] Use absolute scope for systemd template variables

https://gerrit.wikimedia.org/r/401791

awight added a comment.Jan 3 2018, 8:16 PM

Another glitch,

1[1cf50762d7ca0390a9bac343] /wiki/Special:RecentChanges InvalidArgumentException from line 58 of /vagrant/mediawiki/extensions/ORES/includes/Storage/SqlModelLookup.php: No model available for [damaging]
2
3Backtrace:
4
5#0 /vagrant/mediawiki/extensions/ORES/includes/ThresholdLookup.php(103): ORES\Storage\SqlModelLookup->getModelVersion(string)
6#1 /vagrant/mediawiki/extensions/ORES/includes/ThresholdLookup.php(94): ORES\ThresholdLookup->fetchThresholdsFromCache(string)
7#2 /vagrant/mediawiki/extensions/ORES/includes/ThresholdLookup.php(84): ORES\ThresholdLookup->fetchThresholds(string, boolean)
8#3 /vagrant/mediawiki/extensions/ORES/includes/Hooks/ChangesListHooksHandler.php(63): ORES\ThresholdLookup->getThresholds(string)
9#4 /vagrant/mediawiki/includes/Hooks.php(177): ORES\Hooks\ChangesListHooksHandler::onChangesListSpecialPageStructuredFilters(SpecialRecentChanges)
10#5 /vagrant/mediawiki/includes/Hooks.php(205): Hooks::callHook(string, array, array, NULL)
11#6 /vagrant/mediawiki/includes/specialpage/ChangesListSpecialPage.php(909): Hooks::run(string, array)
12#7 /vagrant/mediawiki/includes/specials/SpecialRecentchanges.php(186): ChangesListSpecialPage->registerFilters()
13#8 /vagrant/mediawiki/includes/specialpage/ChangesListSpecialPage.php(1050): SpecialRecentChanges->registerFilters()
14#9 /vagrant/mediawiki/includes/specialpage/ChangesListSpecialPage.php(870): ChangesListSpecialPage->setup(NULL)
15#10 /vagrant/mediawiki/includes/specialpage/ChangesListSpecialPage.php(640): ChangesListSpecialPage->getOptions()
16#11 /vagrant/mediawiki/includes/specialpage/ChangesListSpecialPage.php(556): ChangesListSpecialPage->considerActionsForDefaultSavedQuery()
17#12 /vagrant/mediawiki/includes/specials/SpecialRecentchanges.php(168): ChangesListSpecialPage->execute(NULL)
18#13 /vagrant/mediawiki/includes/specialpage/SpecialPage.php(522): SpecialRecentChanges->execute(NULL)
19#14 /vagrant/mediawiki/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run(NULL)
20#15 /vagrant/mediawiki/includes/MediaWiki.php(287): SpecialPageFactory::executePath(Title, RequestContext)
21#16 /vagrant/mediawiki/includes/MediaWiki.php(851): MediaWiki->performRequest()
22#17 /vagrant/mediawiki/includes/MediaWiki.php(523): MediaWiki->main()
23#18 /vagrant/mediawiki/index.php(43): MediaWiki->run()
24#19 /var/www/w/index.php(5): require(string)
25#20 {main}

awight added a comment.Jan 3 2018, 8:30 PM

Similar crash from the PopulateDatabase script,

1vagrant@mediawikivagrant:/vagrant/mediawiki$ php extensions/ORES/maintenance/PopulateDatabase.php Processing 2 revisions
2[4b4fce8ce12a70a21aa31600] [no req] InvalidArgumentException from line 58 of /vagrant/mediawiki/extensions/ORES/includes/Storage/SqlModelLookup.php: No model available for [damaging]
3Backtrace:
4#0 /vagrant/mediawiki/extensions/ORES/includes/Scoring.php(82): ORES\Storage\SqlModelLookup->getModelVersion(string)
5#1 /vagrant/mediawiki/extensions/ORES/includes/Scoring.php(58): ORES\Scoring->checkModelVersion(string, array)
6#2 /vagrant/mediawiki/extensions/ORES/maintenance/PopulateDatabase.php(114): ORES\Scoring->getScores(array)
7#3 /vagrant/mediawiki/extensions/ORES/maintenance/PopulateDatabase.php(89): ORES\PopulateDatabase->processScores(array, ORES\Scoring, ORES\Storage\SqlScoreStorage)
8#4 /vagrant/mediawiki/maintenance/doMaintenance.php(94): ORES\PopulateDatabase->execute()
9#5 /vagrant/mediawiki/extensions/ORES/maintenance/PopulateDatabase.php(126): require_once(string)
10#6 {main}

Change 404521 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/vagrant@master] Use absolute scope for systemd template variables

https://gerrit.wikimedia.org/r/404521

Change 401791 abandoned by Awight:
Use absolute scope for systemd template variables

https://gerrit.wikimedia.org/r/401791

Change 404521 merged by jenkins-bot:
[mediawiki/vagrant@master] Use absolute scope for systemd template variables

https://gerrit.wikimedia.org/r/404521

I'm stuck at a strange error, it seems that the wikilabels database is created but the wikilabels user isn't granted permission. This causes the load_schema function to fail, and no tables are created.

I'm also being absolutely slayed by T184119.

The wikilabels thing was a code issue, a fix is ready for review.

I'm still seeing high CPU usage for both the ores-wsgi and wikilabels services, but we can address that later.

Halfak closed this task as Resolved.Jan 30 2018, 8:31 PM
awight mentioned this in Unknown Object (Phame Post).May 2 2018, 6:42 PM
awight mentioned this in Unknown Object (Phame Post).
awight mentioned this in Unknown Object (Phame Post).