It seems wikitech-static is not syncing again.
Examples:
https://wikitech-static.wikimedia.org/wiki/Portal:Cloud_VPS/Admin
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | ArielGlenn | T204840 wikitech-static: not synced | |||
| Resolved | ArielGlenn | T206013 importDump.php --uploads crashes on wikitech-static |
Event Timeline
Comment Actions
Running root@wikitech-static-ord:~# /usr/local/sbin/import-wikitech.sh by hand to see if it reports something useful.
Comment Actions
This is the result:
root@wikitech-static-ord:~# /usr/local/sbin/import-wikitech.sh --2018-09-19 15:50:35-- https://wikitech.wikimedia.org/dumps/labswiki-20180919.xml.gz Resolving wikitech.wikimedia.org (wikitech.wikimedia.org)... 208.80.153.224 Connecting to wikitech.wikimedia.org (wikitech.wikimedia.org)|208.80.153.224|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10260995 (9.8M) [application/x-gzip] Saving to: ‘/srv/imports/labswiki-20180919.xml.gz’ /srv/imports/labswiki-201 100%[=====================================>] 9.79M 14.5MB/s in 0.7s 2018-09-19 15:50:35 (14.5 MB/s) - ‘/srv/imports/labswiki-20180919.xml.gz’ saved [10260995/10260995] 100 (338.11 pages/sec 338.11 revs/sec) 200 (405.37 pages/sec 405.37 revs/sec) 300 (443.03 pages/sec 443.03 revs/sec) 400 (466.38 pages/sec 466.38 revs/sec) 500 (479.33 pages/sec 479.33 revs/sec) 600 (491.68 pages/sec 491.68 revs/sec) 700 (485.47 pages/sec 485.47 revs/sec) 800 (493.32 pages/sec 493.32 revs/sec) 900 (496.76 pages/sec 496.76 revs/sec) 1000 (487.07 pages/sec 487.07 revs/sec) 1100 (463.10 pages/sec 463.10 revs/sec) 1200 (326.64 pages/sec 326.64 revs/sec) 1300 (137.25 pages/sec 137.25 revs/sec) 1400 (124.51 pages/sec 124.51 revs/sec) 1500 (124.37 pages/sec 124.37 revs/sec) 1600 (111.23 pages/sec 111.23 revs/sec) 1700 (92.04 pages/sec 92.04 revs/sec) 1800 (90.68 pages/sec 90.68 revs/sec) 1900 (77.90 pages/sec 77.90 revs/sec) 2000 (73.49 pages/sec 73.49 revs/sec) 2100 (70.67 pages/sec 70.67 revs/sec) 2200 (71.01 pages/sec 71.01 revs/sec) 2300 (67.58 pages/sec 67.58 revs/sec) 2400 (67.18 pages/sec 67.18 revs/sec) 2500 (69.30 pages/sec 69.30 revs/sec) 2600 (70.17 pages/sec 70.17 revs/sec) 2700 (71.08 pages/sec 71.08 revs/sec) 2800 (73.20 pages/sec 73.20 revs/sec) 2900 (73.78 pages/sec 73.78 revs/sec) 3000 (74.78 pages/sec 74.78 revs/sec) 3100 (76.21 pages/sec 76.21 revs/sec) 3200 (78.00 pages/sec 78.00 revs/sec) 3300 (73.12 pages/sec 73.12 revs/sec) 3400 (72.87 pages/sec 72.87 revs/sec) 3500 (74.04 pages/sec 74.04 revs/sec) 3600 (74.43 pages/sec 74.43 revs/sec) 3700 (75.61 pages/sec 75.61 revs/sec) 3800 (76.62 pages/sec 76.62 revs/sec) 3900 (77.91 pages/sec 77.91 revs/sec) 4000 (79.20 pages/sec 79.20 revs/sec) 4100 (79.79 pages/sec 79.79 revs/sec) 4200 (79.79 pages/sec 79.79 revs/sec) 4300 (79.65 pages/sec 79.65 revs/sec) 4400 (74.13 pages/sec 74.13 revs/sec) 4500 (75.25 pages/sec 75.25 revs/sec) 4600 (76.28 pages/sec 76.28 revs/sec) 4700 (77.41 pages/sec 77.41 revs/sec) 4800 (78.08 pages/sec 78.08 revs/sec) 4900 (78.95 pages/sec 78.95 revs/sec) 5000 (80.09 pages/sec 80.09 revs/sec) 5100 (80.35 pages/sec 80.35 revs/sec) 5200 (81.38 pages/sec 81.38 revs/sec) 5300 (82.08 pages/sec 82.08 revs/sec) 5400 (81.22 pages/sec 81.22 revs/sec) 5500 (81.94 pages/sec 81.94 revs/sec) 5600 (82.95 pages/sec 82.95 revs/sec) 5700 (83.15 pages/sec 83.15 revs/sec) 5800 (84.12 pages/sec 84.12 revs/sec) 5900 (83.54 pages/sec 83.54 revs/sec) 6000 (84.61 pages/sec 84.61 revs/sec) 6100 (85.33 pages/sec 85.33 revs/sec) 6200 (86.32 pages/sec 86.32 revs/sec) 6300 (87.21 pages/sec 87.21 revs/sec) 6400 (88.26 pages/sec 88.26 revs/sec) 6500 (89.06 pages/sec 89.06 revs/sec) 6600 (88.52 pages/sec 88.52 revs/sec) 6700 (89.40 pages/sec 89.40 revs/sec) 6800 (90.25 pages/sec 90.25 revs/sec) 6900 (90.86 pages/sec 90.86 revs/sec) 7000 (91.73 pages/sec 91.73 revs/sec) 7100 (91.51 pages/sec 91.51 revs/sec) 7200 (92.12 pages/sec 92.12 revs/sec) 7300 (92.82 pages/sec 92.82 revs/sec) 7400 (93.70 pages/sec 93.70 revs/sec) 7500 (94.45 pages/sec 94.45 revs/sec) 7600 (95.16 pages/sec 95.16 revs/sec) 7700 (95.90 pages/sec 95.90 revs/sec) 7800 (95.90 pages/sec 95.90 revs/sec) 7900 (89.48 pages/sec 89.48 revs/sec) 8000 (89.91 pages/sec 89.91 revs/sec) 8100 (90.60 pages/sec 90.60 revs/sec) 8200 (91.37 pages/sec 91.37 revs/sec) 8300 (91.08 pages/sec 91.08 revs/sec) 8400 (91.73 pages/sec 91.73 revs/sec) 8500 (92.37 pages/sec 92.37 revs/sec) 8600 (93.00 pages/sec 93.00 revs/sec) 8700 (93.85 pages/sec 93.85 revs/sec) 8800 (94.49 pages/sec 94.49 revs/sec) 8900 (95.02 pages/sec 95.02 revs/sec) 9000 (95.63 pages/sec 95.63 revs/sec) 9100 (96.10 pages/sec 96.10 revs/sec) 9200 (96.93 pages/sec 96.93 revs/sec) 9300 (97.71 pages/sec 97.71 revs/sec) 9400 (98.25 pages/sec 98.25 revs/sec) 9500 (98.76 pages/sec 98.76 revs/sec) 9600 (99.54 pages/sec 99.54 revs/sec) 9700 (98.12 pages/sec 98.12 revs/sec) 9800 (98.35 pages/sec 98.35 revs/sec) 9900 (98.86 pages/sec 98.86 revs/sec) 10000 (99.64 pages/sec 99.64 revs/sec) 10100 (99.90 pages/sec 99.90 revs/sec) 10200 (99.94 pages/sec 99.94 revs/sec) 10300 (100.02 pages/sec 100.02 revs/sec) 10400 (100.73 pages/sec 100.73 revs/sec) 10500 (101.27 pages/sec 101.27 revs/sec) 10600 (97.05 pages/sec 97.05 revs/sec) 10700 (97.69 pages/sec 97.69 revs/sec) 10800 (98.27 pages/sec 98.27 revs/sec) 10900 (98.67 pages/sec 98.67 revs/sec) 11000 (99.30 pages/sec 99.30 revs/sec) 11100 (99.93 pages/sec 99.93 revs/sec) 11200 (100.64 pages/sec 100.64 revs/sec) 11300 (101.31 pages/sec 101.31 revs/sec) 11400 (101.80 pages/sec 101.80 revs/sec) 11500 (102.50 pages/sec 102.50 revs/sec) 11600 (102.96 pages/sec 102.96 revs/sec) 11700 (103.58 pages/sec 103.58 revs/sec) 11800 (103.27 pages/sec 103.27 revs/sec) 11900 (103.90 pages/sec 103.90 revs/sec) 12000 (104.57 pages/sec 104.57 revs/sec) 12100 (104.89 pages/sec 104.89 revs/sec) 12200 (104.96 pages/sec 104.96 revs/sec) 12300 (105.47 pages/sec 105.47 revs/sec) 12400 (105.86 pages/sec 105.86 revs/sec) 12500 (105.84 pages/sec 105.84 revs/sec) 12600 (105.64 pages/sec 105.64 revs/sec) 12700 (106.23 pages/sec 106.23 revs/sec) 12800 (106.69 pages/sec 106.69 revs/sec) 12900 (107.29 pages/sec 107.29 revs/sec) 13000 (107.06 pages/sec 107.06 revs/sec) 13100 (107.25 pages/sec 107.25 revs/sec) 13200 (106.85 pages/sec 106.85 revs/sec) 13300 (106.80 pages/sec 106.80 revs/sec) 13400 (106.72 pages/sec 106.72 revs/sec) 13500 (106.91 pages/sec 106.91 revs/sec) 13600 (107.46 pages/sec 107.46 revs/sec) 13700 (106.93 pages/sec 106.93 revs/sec) 13800 (107.33 pages/sec 107.33 revs/sec) 13900 (107.45 pages/sec 107.45 revs/sec) 14000 (107.15 pages/sec 107.15 revs/sec) 14100 (107.47 pages/sec 107.47 revs/sec) 14200 (107.93 pages/sec 107.93 revs/sec) 14300 (108.50 pages/sec 108.50 revs/sec) 14400 (108.37 pages/sec 108.37 revs/sec) 14500 (103.46 pages/sec 103.46 revs/sec) 14600 (102.65 pages/sec 102.65 revs/sec) 14700 (100.66 pages/sec 100.66 revs/sec) 14800 (100.12 pages/sec 100.12 revs/sec) 14900 (99.93 pages/sec 99.93 revs/sec) 15000 (99.84 pages/sec 99.84 revs/sec) 15100 (99.67 pages/sec 99.67 revs/sec) 15200 (98.29 pages/sec 98.29 revs/sec) 15300 (98.46 pages/sec 98.46 revs/sec) 15400 (97.90 pages/sec 97.90 revs/sec) 15500 (98.34 pages/sec 98.34 revs/sec) 15600 (97.23 pages/sec 97.23 revs/sec) 15700 (95.63 pages/sec 95.63 revs/sec) 15800 (94.97 pages/sec 94.97 revs/sec) 15900 (94.16 pages/sec 94.16 revs/sec) 16000 (94.08 pages/sec 94.08 revs/sec) 16100 (94.06 pages/sec 94.06 revs/sec) 16200 (91.36 pages/sec 91.36 revs/sec) 16300 (91.28 pages/sec 91.28 revs/sec) 16400 (91.46 pages/sec 91.46 revs/sec) 16500 (91.49 pages/sec 91.49 revs/sec) 16600 (90.44 pages/sec 90.44 revs/sec) 16700 (90.65 pages/sec 90.65 revs/sec) 16800 (90.72 pages/sec 90.72 revs/sec) 16900 (89.33 pages/sec 89.33 revs/sec) 17000 (88.97 pages/sec 88.97 revs/sec) 17100 (88.91 pages/sec 88.91 revs/sec) 17200 (88.53 pages/sec 88.53 revs/sec) 17300 (88.80 pages/sec 88.80 revs/sec) 17400 (88.98 pages/sec 88.98 revs/sec) 17500 (85.53 pages/sec 85.53 revs/sec) 17600 (84.81 pages/sec 84.81 revs/sec) 17700 (84.67 pages/sec 84.67 revs/sec) 17800 (84.82 pages/sec 84.82 revs/sec) 17900 (85.09 pages/sec 85.09 revs/sec) 18000 (84.84 pages/sec 84.84 revs/sec) 18100 (84.95 pages/sec 84.95 revs/sec) 18200 (85.05 pages/sec 85.05 revs/sec) 18300 (84.89 pages/sec 84.89 revs/sec) 18400 (84.32 pages/sec 84.32 revs/sec) 18500 (83.64 pages/sec 83.64 revs/sec) 18600 (82.47 pages/sec 82.47 revs/sec) 18700 (81.56 pages/sec 81.56 revs/sec) 18800 (79.30 pages/sec 79.30 revs/sec) 18900 (73.64 pages/sec 73.64 revs/sec) 19000 (66.25 pages/sec 66.25 revs/sec) 19100 (62.21 pages/sec 62.21 revs/sec) 19200 (60.54 pages/sec 60.54 revs/sec) 19300 (58.47 pages/sec 58.47 revs/sec) Done! You might want to run rebuildrecentchanges.php to regenerate RecentChanges, and initSiteStats.php to update page and revision counts Rebuilding $wgRCMaxAge=7776000 seconds (90 days) Clearing recentchanges table for time range... Loading from page and revision tables... Inserting from page and revision tables... Updating links and size differences... Loading from user, page, and logging tables... Flagging bot account edits... Flagging auto-patrolled edits... Removing duplicate revision and logging entries... Deleting feed timestamps. Done. --2018-09-19 15:56:16-- https://wikitech.wikimedia.org/dumps/labswiki-20180919-images.tar.gz Resolving wikitech.wikimedia.org (wikitech.wikimedia.org)... 208.80.153.224 Connecting to wikitech.wikimedia.org (wikitech.wikimedia.org)|208.80.153.224|:443... connected. HTTP request sent, awaiting response... 404 Not Found 2018-09-19 15:56:16 ERROR 404: Not Found. gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now
So, a couple of issues at first glance. Not sure how important they are.
Comment Actions
There is Icinga monitoring for this, example on labweb1001:
But they are all green and not alerting -> "wikitech-static OK - wikitech and wikitech-static in sync (41947 < 200000s)"
So, a couple of issues at first glance. Not sure how important they are.
What issues specifically do you see in that output? It looks like a succesful attempt to dowload the gzipped file.
Comment Actions
The script above is what docs mention https://wikitech.wikimedia.org/wiki/Wikitech-static#Automatic_updates
However, the root cron (crontab -l) is using a different thing:
# m h dom mon dow command 0 4 * * * /wikitech-static/wikitechsync/import-wikitech.sh > /dev/null 2>&1
root@wikitech-static-ord:~# /wikitech-static/wikitechsync/import-wikitech.sh
--2018-09-19 16:05:31-- https://wikitech.wikimedia.org/dumps/labswiki-20180919.xml.gz
Resolving wikitech.wikimedia.org (wikitech.wikimedia.org)... 208.80.153.224
Connecting to wikitech.wikimedia.org (wikitech.wikimedia.org)|208.80.153.224|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10260995 (9.8M) [application/x-gzip]
Saving to: ‘/srv/imports/labswiki-20180919.xml.gz’
/srv/imports/labswiki-20180919.xml.gz 100%[=====================================================================================================================>] 9.79M 15.0MB/s in 0.7s
2018-09-19 16:05:31 (15.0 MB/s) - ‘/srv/imports/labswiki-20180919.xml.gz’ saved [10260995/10260995]
100 (131.51 pages/sec 131.51 revs/sec)
200 (196.36 pages/sec 196.36 revs/sec)
upload: Gerrit_branch_permissions_wrong.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
300 (110.32 pages/sec 110.32 revs/sec)
400 (135.54 pages/sec 135.54 revs/sec)
500 (154.84 pages/sec 154.84 revs/sec)
600 (169.12 pages/sec 169.12 revs/sec)
upload: MariyaMiteva.jpg
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
700 (167.46 pages/sec 167.46 revs/sec)
upload: LabsProjectsInstance.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: LabsProjectsInstance.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: LabsProjectsInstance.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/xmp/XMP.php on line 540
PHP Warning: XMLReader::setParserProperty(): Invalid parser property in /srv/mediawiki/w/includes/libs/xmp/XMP.php on line 549
upload: 20130118-2224-PuTTY_Configuration.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: 20130118-2235-PuTTY_Configuration.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: 20130118-2235-PuTTY_Configuration.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: 20130118-2236-PuTTY_Configuration.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: 20130118-2240-PuTTY_Configuration.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: 20130118-2313-WinSCP_Login.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: 20130118-2312-WinSCP_Login.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: 20130118-2311-WinSCP_Login.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: 20130118-2309-WinSCP_Login.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: 20130119-2158-PuTTY_Configuration.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: 20130119-2156-PuTTY_Configuration.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
800 (97.64 pages/sec 97.64 revs/sec)
upload: Wmflabs-openid.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
upload: Wmflabs-openid.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
900 (101.09 pages/sec 101.09 revs/sec)
1000 (108.29 pages/sec 108.29 revs/sec)
1100 (115.76 pages/sec 115.76 revs/sec)
upload: CURL_extension_installed.png
PHP Warning: XMLReader::open(): Unable to open source data in /srv/mediawiki/w/includes/libs/mime/XmlTypeCheck.php on line 158
[0ac41849c74dbafa20eaa0e8] [no req] Error from line 381 of /srv/mediawiki/w/includes/filerepo/file/OldLocalFile.php: Call to a member function getId() on boolean
Backtrace:
#0 /srv/mediawiki/w/includes/filerepo/file/OldLocalFile.php(342): OldLocalFile->recordOldUpload(string, string, string, string, boolean)
#1 /srv/mediawiki/w/includes/import/WikiRevision.php(794): OldLocalFile->uploadOld(string, string, string, string, boolean)
#2 [internal function]: WikiRevision->importUpload()
#3 /srv/mediawiki/w/includes/libs/rdbms/database/Database.php(2623): call_user_func_array(array, array)
#4 /srv/mediawiki/w/maintenance/importDump.php(211): Wikimedia\Rdbms\Database->deadlockLoop(array)
#5 /srv/mediawiki/w/includes/import/WikiImporter.php(1028): BackupReader->handleUpload(WikiRevision)
#6 /srv/mediawiki/w/includes/import/WikiImporter.php(978): WikiImporter->processUpload(array, array)
#7 /srv/mediawiki/w/includes/import/WikiImporter.php(787): WikiImporter->handleUpload(array)
#8 /srv/mediawiki/w/includes/import/WikiImporter.php(594): WikiImporter->handlePage()
#9 /srv/mediawiki/w/maintenance/importDump.php(329): WikiImporter->doImport()
#10 /srv/mediawiki/w/maintenance/importDump.php(271): BackupReader->importFromHandle(resource)
#11 /srv/mediawiki/w/maintenance/importDump.php(108): BackupReader->importFromFile(string)
#12 /srv/mediawiki/w/maintenance/doMaintenance.php(92): BackupReader->execute()
#13 /srv/mediawiki/w/maintenance/importDump.php(334): require_once(string)
#14 {main}
PHP Fatal error: Uncaught Wikimedia\Rdbms\DBTransactionError: Explicit transaction still active. A caller may have caught an error. in /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php:1200
Stack trace:
#0 [internal function]: Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(Object(Wikimedia\Rdbms\DatabaseMysqli))
#1 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1546): call_user_func_array(Object(Closure), Array)
#2 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1223): Wikimedia\Rdbms\LoadBalancer->forEachOpenMasterConnection(Object(Closure))
#3 [internal function]: Wikimedia\Rdbms\LoadBalancer->approveMasterChanges(Array)
#4 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(183): call_user_func_array(Array, Array)
#5 [internal function]: Wikimedia\Rdbms\LBFactory->Wikimedia\Rdbms\{closure}(Object(Wikimedia\Rdbms\LoadBalancer), 'approveMasterCh...', Array)
#6 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactorySimple.php(149): call_user_func_array in /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php on line 1200
Rebuilding $wgRCMaxAge=7776000 seconds (90 days)
Clearing recentchanges table for time range...
Loading from page and revision tables...
Inserting from page and revision tables...
Updating links and size differences...
Loading from user, page, and logging tables...
Flagging bot account edits...
Flagging auto-patrolled edits...
Removing duplicate revision and logging entries...
Deleting feed timestamps.
Done.
last image dump for wikitech was: never
running command: /usr/bin/php /srv/mediawiki/w/maintenance/getConfiguration.php --wiki=wikitech --group=dump --format=json --regex=wgDBadminuser
running command: /usr/bin/php /srv/mediawiki/w/maintenance/getConfiguration.php --wiki=wikitech --group=dump --format=json --regex=wgDBuser
running command: /usr/bin/php /srv/mediawiki/w/maintenance/getConfiguration.php --wiki=wikitech --group=dump --format=json --regex=wgDBadminpassword
running command: /usr/bin/php /srv/mediawiki/w/maintenance/getConfiguration.php --wiki=wikitech --group=dump --format=json --regex=wgDBpassword
db credentials acquired
base url for image retrieval is https://upload.wikimedia.org/wikipedia/labs
running command /usr/bin/php /srv/mediawiki/w/maintenance/getSlaveServer.php --wiki=wikitech --group=dump
running command /bin/echo "SELECT img_name, img_timestamp from image" | /usr/bin/mysql -h localhost -u wikiuser -psbq3OMQwV8FJ --max_allowed_packet 16M wikitech -r --silent
running command /usr/bin/php /srv/mediawiki/w/maintenance/getSlaveServer.php --wiki=wikitech --group=dump
running command /bin/echo "SELECT il_to from imagelinks" | /usr/bin/mysql -h localhost -u wikiuser -psbq3OMQwV8FJ --max_allowed_packet 16M wikitech -r --silent
processing image list
Cleaning up abandoned file /srv/mediawiki/images/wikitech/lockdir/6lop2j60v8v1baqg2rwzd78dn9zymiu.lock
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/d/d9/20180919160536!MariyaMiteva.jpg
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/0/00/20180316174244!20130118-2311-WinSCP_Login.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/0/05/20180316174244!20130119-2156-PuTTY_Configuration.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/0/04/20180316174244!CURL_extension_installed.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/5/5b/20180316174246!20130118-2224-PuTTY_Configuration.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/6/6b/20180919160533!Gerrit_branch_permissions_wrong.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/e/e0/20130129194235!Wmflabs-openid.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/e/e0/20180316174249!Wmflabs-openid.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/e/e8/20180316174249!20130118-2309-WinSCP_Login.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/b/be/20180316174248!20130118-2240-PuTTY_Configuration.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/b/b8/20180316174248!20130118-2312-WinSCP_Login.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/f/fa/20180316174249!20130118-2236-PuTTY_Configuration.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/4/4c/20180316174246!20130119-2158-PuTTY_Configuration.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/4/48/20180520093845!20130118-2235-PuTTY_Configuration.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/4/48/20180919160538!20130118-2235-PuTTY_Configuration.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/4/41/20130109234925!LabsProjectsInstance.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/4/41/20130829060930!LabsProjectsInstance.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/4/41/20180919160537!LabsProjectsInstance.png
Cleaning up abandoned file /srv/mediawiki/images/wikitech/archive/7/74/20180316174247!20130118-2313-WinSCP_Login.png
mwstore://local-backend/local-public/0/06/Flag_of_Venezuela.svg
mwstore://local-backend/local-public/0/0a/Light_rail.jpg
mwstore://local-backend/local-public/1/11/TGVA_n°341_au_PN_401_bis_à_La_Baule_par_Cramos.JPG
mwstore://local-backend/local-public/1/11/Grant_Metrics_logo.svg
mwstore://local-backend/local-public/1/16/Akbarali_Charankav_Presentation_WikiConference_India_2016.jpg
mwstore://local-backend/local-public/1/17/Screen_Shot_2018-07-20_at_11.52.05_AM.png
mwstore://local-backend/local-public/1/1c/Nikhil_2.jpg
mwstore://local-backend/local-public/2/20/Seal_of_Hebron.tif
mwstore://local-backend/local-public/2/26/Cogdill,_Caitlin_Nov_2013.jpg
mwstore://local-backend/local-public/2/26/MediaWiki_fallback_chains.svg
mwstore://local-backend/local-public/2/2a/Exquisite-kmail3.png
mwstore://local-backend/local-public/2/2e/Webpagereplay_choose_page_in_Grafana.png
mwstore://local-backend/local-public/2/2e/LocationWHAmericas.png
mwstore://local-backend/local-public/2/2e/Gnome-applications-science.svg
mwstore://local-backend/local-public/2/2f/HorizonLaunchInstanceScreenPart2-2018-05-07-14-17.png
mwstore://local-backend/local-public/3/34/Latest_run_and_older_runs_for_WebPageReplay.png
mwstore://local-backend/local-public/3/36/HorizonLaunchInstanceScreenPart4-2018-05-07-14-17.png
mwstore://local-backend/local-public/3/37/HorizonCreateAProxyScreen-2018-05-07-14-22.png
mwstore://local-backend/local-public/3/3a/Grafana-alert-value-mismatch.png
mwstore://local-backend/local-public/3/3f/HorizonLaunchInstanceScreenPart3-2018-05-07-14-17.png
mwstore://local-backend/local-public/4/44/Screen_Shot_2018-07-17_at_4.23.57_PM.png
mwstore://local-backend/local-public/4/45/Wikimedia_Ashburn_DC,_physical_layer.png
mwstore://local-backend/local-public/4/4b/PAWS.svg
mwstore://local-backend/local-public/4/4b/WebPageTest_search_result.png
mwstore://local-backend/local-public/4/4f/Geowiki-screenshot.png
mwstore://local-backend/local-public/5/52/20180524-incident-S8-DB-graph.png
mwstore://local-backend/local-public/5/56/Server_Lifecycle_Statuses.png
mwstore://local-backend/local-public/5/5a/EditHistoryReconstruction.gif
mwstore://local-backend/local-public/5/5c/Screen_Shot_2018-07-17_at_3.42.03_PM.png
mwstore://local-backend/local-public/5/5f/Wikimedia_network_overview.png
mwstore://local-backend/local-public/6/61/Webpagetest_search_form.png
mwstore://local-backend/local-public/6/61/Eqiad1_network_topology.png
mwstore://local-backend/local-public/6/63/Screen_Shot_2018-08-07_at_16.45.54.png
mwstore://local-backend/local-public/6/64/SietecFAST™_Logo.png
mwstore://local-backend/local-public/6/64/Wmcs_dns.pdf
mwstore://local-backend/local-public/6/66/WINSCP_screen_advanced_settings2.png
mwstore://local-backend/local-public/6/68/Nuvola_filesystems_www.png
mwstore://local-backend/local-public/6/6c/Bouncywikilogo.gif
mwstore://local-backend/local-public/7/72/Sync_wiki_versions_showing_when_a_change_is_pushed.png
mwstore://local-backend/local-public/7/78/Example_of_NICs_in_Neutron.png
mwstore://local-backend/local-public/7/7b/Flag_of_Venezuela_(state).svg
mwstore://local-backend/local-public/7/7c/Incident_20180710B_Logstash_clear.png
mwstore://local-backend/local-public/7/7f/RESTBase_Cassandra_cluster_disposition.svg
mwstore://local-backend/local-public/8/81/Toolforge_DB_example.png
mwstore://local-backend/local-public/8/81/OpenStack_at_WMCS.pdf
mwstore://local-backend/local-public/8/84/Prometheus-grafana-annotations.png
mwstore://local-backend/local-public/8/8b/WINSCP_screen_advanced_settings3.png
mwstore://local-backend/local-public/9/91/Angel_Falls_in_the_morning_light.jpg
mwstore://local-backend/local-public/9/99/Slopu.gif
mwstore://local-backend/local-public/9/99/Derick_WikiIndaba.jpg
mwstore://local-backend/local-public/9/9a/Exquisite-kwrite.png
mwstore://local-backend/local-public/9/9b/Containerized_continuous_delivery_2017_concept.png
mwstore://local-backend/local-public/a/a4/India_flag-XL-anim.gif
mwstore://local-backend/local-public/a/a9/Wikimedia_Amsterdam_site,_physical_layer.png
mwstore://local-backend/local-public/a/ae/Star_of_life.svg
mwstore://local-backend/local-public/b/b2/Wikimedia_Amsterdam_DCs,_IP_layer.png
mwstore://local-backend/local-public/b/b6/HorizonPuppetPrefix-2018-05-07-14-13.png
mwstore://local-backend/local-public/b/b9/Nuvola_apps_edu_languages.png
mwstore://local-backend/local-public/b/b9/Wikimedia_Dallas_DCs,_physical_layer.png
mwstore://local-backend/local-public/b/ba/HAR_upload_button.png
mwstore://local-backend/local-public/b/bc/Elephant.jpg
mwstore://local-backend/local-public/c/c5/Dns-discovery_active-active.png
mwstore://local-backend/local-public/c/c7/Venezuela_regions_map.png
mwstore://local-backend/local-public/c/c9/Webpagereplay_folders.png
mwstore://local-backend/local-public/c/ca/Wdqs-components.png
mwstore://local-backend/local-public/c/cb/Exquisite-Modem.png
mwstore://local-backend/local-public/c/cd/Screen_Shot_2018-07-18_at_8.56.23_AM.png
mwstore://local-backend/local-public/d/d3/Statuette_of_dancing_Shiva,_the_Nataraja.jpg
mwstore://local-backend/local-public/d/d4/20180728-eventlogging-throughput-graph.png
mwstore://local-backend/local-public/d/d4/Storm,_Brooke,_March_2018.jpg
mwstore://local-backend/local-public/d/d7/Wikimedia_Amsterdam_DCs_digram_-_L1.png
mwstore://local-backend/local-public/d/d8/Anycast_rec-dns_diagram.png
mwstore://local-backend/local-public/d/dd/Wikivoyage-Logo-v3-icon.svg
mwstore://local-backend/local-public/d/df/Wikimania.png
mwstore://local-backend/local-public/d/df/WebPageTest_trace_log.png
mwstore://local-backend/local-public/e/e5/Screen_Shot_2018-08-13_at_16.08.07.png
mwstore://local-backend/local-public/e/ee/HorizonLaunchInstanceScreenPart1-2018-05-07-14-17.png
mwstore://local-backend/local-public/f/f4/Baustelle.svg
mwstore://local-backend/local-public/f/f4/Dns-discovery_active-passive.png
mwstore://local-backend/local-public/f/f6/Pageviews_Analysis.svg
mwstore://local-backend/local-public/f/f7/Mushroom.svg
mwstore://local-backend/local-public/f/fb/Wikignome_crop.gif
mwstore://local-backend/local-public/f/fc/Wikimedia_San_Francisco_DC,_physical_layer.png
[ ok ] Reloading Apache httpd web server: apache2.Which is full of errors and warnings.
Comment Actions
The latest revision on both pages you linked also looks the same to me:
https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin&oldid=1793552
https://wikitech-static.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin&oldid=296579
both of them say "Revision as of 22:49, 1 June 2018 by BryanDavis "
Comment Actions
Unfortunately, I just synced by hand and proofs are gone. In some pages, there was more than 6 month divergence.
Both @bd808 and @Andrew noticed as well.
My initial guess, is that only part of the content is synced, or the process is stuck at some point which prevents a full sync.
So, the icinga check may not be robust enough (is that checking only the SAL?)
So, a couple of issues at first glance. Not sure how important they are.
What issues specifically do you see in that output? It looks like a succesful attempt to dowload the gzipped file.
If you scroll back in the first paste, you see that image syncing fails because the GZIP is not in the origin (a cron failing in labweb? no idea).
See above.
Also, see second paste.
Comment Actions
@aborrero I see what you mean now. Unfortunately i know nothing about the actual sync there. i just once did the monitoring to check for the timestamps. Yes, it's confusing that docs differ from reality, i think the people to know this are @Andrew @Krinkle (based on history of the docs wiki page) and @Krenair Afair he used to volunteer to handle wikitech-static. Maybe check the lastlog and bash_history on the wt-static machine to see who last touched it.
That being said.. i am not sure i can confirm it's not syncing since the monitoring is green and i see the same revision on both linked pages.
Comment Actions
Yep, sorry for my ignorance, i didn't scroll all the way down in the paste. Gotcha, i see that error now. And makes sense that it just got fixed when you ran it manually.
So, the icinga check may not be robust enough (is that checking only the SAL?)
It's talking to the API like this:
WIKITECH="https://wikitech.wikimedia.org/w/api.php" WIKITECHSTATIC="https://wikitech-static.wikimedia.org/w/api.php" API_QUERY="action=query&titles=Server_Admin_Log&list=recentchanges&format=xml"
It then does a curl $WIKITECH?$API_QUERY and greps the string "timestamp" out of that. Then compares the 2 timestamps.
Comment Actions
(Duplicate of T203206: Wikitech-static not getting all content changes? Not many additional details there though.
But possibly there are clues in the older task from 2017 which has many comments T177450: Not all content is getting replicated to wikitech-static)
Comment Actions
I still don't know why this failed.
Checking the SAL timestamps doesn't seem to be enough to discover sync issues, because for whatever reason the SAL is synced but other content don't. Perhaps https://wikitech-static.wikimedia.org/wiki/Special:RecentChanges is worth checking instead for a more complete overview.
In the sync script /wikitech-static/wikitechsync/import-wikitech.sh there are plenty of warnings and several errors, including backtraces. It worth evaluating with someone with knowledge of mediawiki PHP codebase.
Also, the sync script differs from what's documented:
root@wikitech-static-ord:~# diff -u --color /usr/local/sbin/import-wikitech.sh /wikitech-static/wikitechsync/import-wikitech.sh --- /usr/local/sbin/import-wikitech.sh 2018-03-09 21:43:07.585752064 +0000 +++ /wikitech-static/wikitechsync/import-wikitech.sh 2018-03-20 13:44:51.130398096 +0000 @@ -1,18 +1,21 @@ #!/bin/bash +# +# This should be called from a cron on wikitech-static DATE=$(date '+%Y%m%d') wget https://wikitech.wikimedia.org/dumps/labswiki-${DATE}.xml.gz -O /srv/imports/labswiki-${DATE}.xml.gz -4 cd /srv/mediawiki/w -php maintenance/importDump.php /srv/imports/labswiki-${DATE}.xml.gz +php maintenance/importDump.php --uploads /srv/imports/labswiki-${DATE}.xml.gz php maintenance/rebuildrecentchanges.php php maintenance/rebuildImages.php --missing rm /srv/imports/labswiki-${DATE}.xml.gz -wget https://wikitech.wikimedia.org/dumps/labswiki-${DATE}-images.tar.gz -O /srv/imports/labswiki-${DATE}-images.tar.gz -4 -cd /srv/mediawiki -tar -xzvf /srv/imports/labswiki-${DATE}-images.tar.gz -rm /srv/imports/labswiki-${DATE}-images.tar.gz +/wikitech-static/wikitechsync/get_images.py --wiki wikitech --config /wikitech-static/wikitechsync/dump_images.conf.wikitech-static --verbose +chown -R www-data /srv/mediawiki/images/wikitech +chgrp -R www-data /srv/mediawiki/images/wikitech +php maintenance/rebuildImages.php --missing +service apache2 graceful service memcached restart
This is what I will do for now:
- refresh the icinga script to check Special:RecentChanges instead of the SAL, to try to get actually alerted when wiki content differs
- try to update the docs to better reflect what's actually running for the sync mechanism
Comment Actions
Change 461924 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] icinga: wikitech_static: check Special:RecentChanges instead of the SAL
Comment Actions
Change 461924 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] icinga: wikitech_static: check Special:RecentChanges instead of the SAL
Comment Actions
The other day I created this page in wikitech:
https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/quarry
3 days later, this still doesn't exists:
https://wikitech-static.wikimedia.org/wiki/Portal:Data_Services/Admin/quarry
@Dzahn any idea what could be happening?
Comment Actions
Sorry, i don't have the slightest idea. I was never involved in the syncing of wikis, what i did was exclusively adding the monitoring check if that timestamp is out of date.
Comment Actions
Change 464014 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/wikitech-static@master] import-wikitech.sh: run without --uploads to make sure we get new pages
Comment Actions
Change 464014 merged by Andrew Bogott:
[operations/wikitech-static@master] import-wikitech.sh: run without --uploads to make sure we get new pages
Comment Actions
Notes about many fixes yesterday, from @Andrew or @Vgutierrez or me.
Wikitech-static was updated to 1.33.0-rc.0 yesterday. Some changes needed to be made to config settings to support the change, see https://gerrit.wikimedia.org/r/#/c/operations/wikitech-static/+/515084/
Some extensions were not bundled with the branch and so the usual 'check them out as submodules' trick did not work for updating them; they had to be cloned manually. @Andrew do you want to add a description of what you did?
At this point, imports work, both with and without images, no errors. I had to run
php cleanupUsersWithNoId.php --assign --prefix imported --force
to fix up some old revision table entries that had rev_user_text without the 'imported>' prefix but with rev_user 0, and therefore no entry in the temp actor table.
I also reran
php migrateActors.php after that to force new entries in the temp actor table.
Without the above, some pages were not rendering (no text found, because no rows for rev/user/comment/actor metadata were found with the JOINs the way they are). This included many templates and some MediaWiki: pages.
Image imports were broken by bitrot in /wikitech-static/wikitechsync/get_images.py; it failed on a call out to getConfiguration.php with the option --group-dump, which is a superfluous argument. It used to be that such arguments were silently ignored in maintenance scripts, but this was changed in https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/450097/ The script has been fixed in https://gerrit.wikimedia.org/r/#/c/operations/wikitech-static/+/514965/
The import script failed to restart apache and memcached because it could not find the path to 'service'. This was changed in https://gerrit.wikimedia.org/r/#/c/operations/wikitech-static/+/514967/
Cert renewal was broken; this was fixed by changing authenticator = webroot to for authenticator = standalone in /etc/letsencrypt/renewal/wikitech-static.wikimedia.org.conf which is not yet reflected in gerrit.
The logo was missing; I stole a copy from the other wikitech and dropped it into /srv/mediawiki/images/labswiki.png
Tomorrow my morning I will see if the sync went ok or if new revisions have the rev_user_text issue. (Note that imports are only done of new revisions.)
Search is somehow broken, see: https://wikitech-static.wikimedia.org/w/index.php?search=+Template%3ADeployment+calendar+event&title=Special%3ASearch&go=Go
What else needs to be added here?
Comment Actions
Some extensions were not bundled with the branch and so the usual 'check them out as submodules' trick did not work for
updating them; they had to be cloned manually. @Andrew do you want to add a description of what you did?
Krenair looked at special::versions and rattled off a bunch of extensions that looked out of date; for each I just cd'd into the extension directory and did a 'git checkout master; git fetch; git rebase.' That means they're on a semi-arbitrary checkout point rather than a branch... I'm not sure what the right comprehensive solution for this is.
Comment Actions
Search issues:
root@wikitech-static:/srv/mediawiki/w# fg mysql -u wikiuser -p check table searchindex ; +----------------------+-------+----------+-------------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +----------------------+-------+----------+-------------------------------------------------------------+ | wikitech.searchindex | check | warning | Table is marked as crashed and last repair failed | | wikitech.searchindex | check | warning | 1 client is using or hasn't closed the table properly | | wikitech.searchindex | check | warning | Size of indexfile is: 83738624 Should be: 1024 | | wikitech.searchindex | check | error | Found key at page -1 that points to record outside datafile | | wikitech.searchindex | check | error | Corrupt | +----------------------+-------+----------+-------------------------------------------------------------+ 5 rows in set (0.06 sec)
Comment Actions
MariaDB [wikitech]> repair table searchindex ; +----------------------+--------+----------+-------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +----------------------+--------+----------+-------------------------------------------------------+ | wikitech.searchindex | repair | info | Found block that points outside data file at 85041468 | | wikitech.searchindex | repair | warning | Number of rows changed from 268084 to 271304 | | wikitech.searchindex | repair | status | OK | +----------------------+--------+----------+-------------------------------------------------------+ 3 rows in set (49.76 sec) MariaDB [wikitech]> check table searchindex ; +----------------------+-------+----------+----------+ | Table | Op | Msg_type | Msg_text | +----------------------+-------+----------+----------+ | wikitech.searchindex | check | status | OK | +----------------------+-------+----------+----------+ 1 row in set (2.44 sec)
Search seems to work now. Huh.
Comment Actions
I checked wikitech-static after this morning's import.
- New pages are attributed properly (see https://wikitech-static.wikimedia.org/w/index.php?title=Application_servers/Runbook&action=history ).
- New revisions of existing pages are attributed properly (see https://wikitech-static.wikimedia.org/w/index.php?title=Service_restarts&action=history ).
- recent changes looks correct there (see https://wikitech-static.wikimedia.org/wiki/Special:RecentChanges?hidebots=1&limit=50&days=7&enhanced=1&urlversion=2 ).
- A new image for import made it in successfully (see https://wikitech-static.wikimedia.org/w/index.php?title=File:Allemandou,_Joseph_January_2019.jpg&action=history )
I'm not entirely sure if that last image was completely new, given the upload comment, so I'd like to check again once we have a new image that appears in the new pages log for the day.
I manually rotated /var/log/debug-wikitech.log since it was 1G. That should probably be automated.
There are improvements to the import process that could be done to make things shorter (and easier to track down errors) but that's for another ticket.
Comment Actions
Out of desperation I have uploaded a new image to wikitech. I'll check tomorrow to see if it made it over.
Comment Actions
New image arrived. Proof: https://wikitech.wikimedia.org/wiki/File:%CE%A3%CF%84%CE%B9%CE%B3%CE%BC%CE%B9%CF%8C%CF%84%CF%85%CF%80%CE%BF_%CE%B1%CF%80%CF%8C_2019-06-12_14-45-40.png and https://wikitech-static.wikimedia.org/wiki/File:%CE%A3%CF%84%CE%B9%CE%B3%CE%BC%CE%B9%CF%8C%CF%84%CF%85%CF%80%CE%BF_%CE%B1%CF%80%CF%8C_2019-06-12_14-45-40.png
At this point I think the task can be closed, and new items added to T225621. So doing!
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-07-16T23:23:21Z] <mutante> wikitech-static - testing cert renewal with dry-run option - getting some temp icinga alerts is now expected again because renewal method was changed back from 'apache' to 'standalone' (not by me -> T204840#5243222 i previously did the opposite change in T214640#4907685 to fix it) and that takes down apache during the renewal (T214640)