The migration script: https://github.com/wbstack/migrate/blob/main/putData.sh
Possible steps:
- Download data from internet to developer laptop
- Run script
- Check wiki is present
- Do next wiki
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T303852 [TRACKING] Status of the migration of accounts from WBstack.com to Wikibase.cloud | |||
| Resolved | None | T306513 [TRACKING] Migration of wbstack.com users Batch A - Coordination | |||
| Resolved | Tarrow | T306532 Perform migration of batch A |
Event Timeline
Comment Actions
We hit some problem. We believe it's an edgecase for Wikis with no pages it index in elastic search.
#0 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(210): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError(8, 'Undefined index...', '/var/www/html/v...', 210, Array)
#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(192): Illuminate\\Queue\\Jobs\\Job->failed(Object(RuntimeException))
#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/InteractsWithQueue.php(47): Illuminate\\Queue\\Jobs\\Job->fail(Object(RuntimeException))
#3 /var/www/html/app/Jobs/CirrusSearch/CirrusSearchJob.php(100): App\\Jobs\\Job->fail(Object(RuntimeException))
#4 /var/www/html/app/Jobs/CirrusSearch/QueueSearchIndexBatches.php(57): App\\Jobs\\CirrusSearch\\CirrusSearchJob->validateOrFailRequest(Array, '{\"wbstackQueueS...', '')
#5 /var/www/html/app/Jobs/CirrusSearch/CirrusSearchJob.php(85): App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches->handleResponse('{\"wbstackQueueS...', '')
#6 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\\Jobs\\CirrusSearch\\CirrusSearchJob->handle(Object(App\\Http\\Curl\\CurlRequest))
#7 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#8 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#9 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#10 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(653): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#11 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\\Container\\Container->call(Array)
#12 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#13 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#14 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#15 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/helpers.php(421): Illuminate\\Bus\\Dispatcher->dispatchNow(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches), false)
#16 /var/www/html/vendor/mxl/laravel-job/src/Commands/DispatchNow.php(18): dispatch_now(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#17 /var/www/html/vendor/mxl/laravel-job/src/Commands/Dispatch.php(32): MichaelLedin\\LaravelJob\\Commands\\DispatchNow->dispatch(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#18 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): MichaelLedin\\LaravelJob\\Commands\\Dispatch->handle()
#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#20 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#21 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#22 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(653): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#23 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\\Container\\Container->call(Array)
#24 /var/www/html/vendor/symfony/console/Command/Command.php(298): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#25 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#26 /var/www/html/vendor/symfony/console/Application.php(1005): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#27 /var/www/html/vendor/symfony/console/Application.php(299): Symfony\\Component\\Console\\Application->doRunCommand(Object(MichaelLedin\\LaravelJob\\Commands\\DispatchNow), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#28 /var/www/html/vendor/symfony/console/Application.php(171): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#29 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Application.php(94): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#30 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#31 /var/www/html/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#32 {main}
"}Comment Actions
Migrating user 14. Found problem we also encountered during testing. Two wgReadOnly settings. Manually removed one from data dump
Comment Actions
2 of 3 wikis of no. 5 migrated, third one failed
+ kubectl --context=gke_wikibase-cloud_europe-west3-a_wbaas-3 exec -c mariadb -it sql-mariadb-primary-0 -- sh -c 'mysql --user=REDACTED --password=REDACTED mwdb_wbstack_REDACTED < /tmp/REDACTED.wikibase.cloud-db.sql'
ERROR 1064 (42000) at line 2302: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''{\"compactDiff\":\"{\\\"arrayFormatVersion\\\":1,\\\"labelChanges\\\":[],\\\...' at line 1
command terminated with exit code 1Comment Actions
SQL dump of 3rd wiki was incomplete, running again with complete dump and without "Initial setup" and Logo bits
Comment Actions
migration of no. 5 finished (3 wikis)
but it appears there were some errors during the final munging step, possibly related to http vs https concept URIs
Comment Actions
migration of no. 9 paused
further migration blocked until we increase resources for prod https://phabricator.wikimedia.org/T306688
will continue to do test runs on staging
Comment Actions
test run on staging succeeded - migrations can continue after https://phabricator.wikimedia.org/T306688 is done
Comment Actions
During migration of user no. 4 we encountered some issues with initializing the Elastic Search index. At first the first job of the two timed out, which we could mitigate with a env var (T307474). But after that we continued to see the two jobs failing occasionally.
php artisan job:dispatchNow CirrusSearch\\\\ElasticSearchIndexInit $WIKI_ID php artisan job:dispatchNow CirrusSearch\\\\QueueSearchIndexBatches $WIKI_ID
A brute force approach would be making 100% sure that there is no harm in these failing and running again and just do that, because eventually they succeed. But it would be great to get to the bottom of this because there is probably some lesson in there.
Example stacktraces:
Comment Actions
As we have the assumption that the Elastic Search jobs may run for every wiki we decided to skip them while migrating these wikis: https://phabricator.wikimedia.org/P27428
And run it afterwards
Comment Actions
user no. 4 migrated
wikis that ES jobs still need to run for:
genewikibots.semscape.org ghissue62.wikibase.cloud gw-human-genes.wikibase.cloud isbgenewiki.wikibase.cloud jacana-jacana.semscape.org lexbibclone2.wikibase.cloud lexbibclone3.wikibase.cloud lexbibclone.wikibase.cloud loadrdf.wikibase.cloud lux-saf-beta.wikibase.cloud lux-saf-demo1.wikibase.cloud luxsaf-demo.micel.io luxsafjul011.wikibase.cloud luxsafjul012.wikibase.cloud luxsafjun171.wikibase.cloud luxsafjun301.wikibase.cloud luxsafoption1a.wikibase.cloud luxsafoption1b.wikibase.cloud luxsafoption2.wikibase.cloud maartens-saf.wikibase.cloud newwikibase.wikibase.cloud pawstest.wikibase.cloud pcbeta-1.wikibase.cloud pitangus-sulphuratus.wikibase.cloud proteins.semscape.org saf02.wikibase.cloud saf1a.wikibase.cloud saf1b.wikibase.cloud safcidoc2.wikibase.cloud safdoc.wikibase.cloud safsandbox1.wikibase.cloud safsandbox2.wikibase.cloud safsandbox3.wikibase.cloud safsandbox4.wikibase.cloud safsandbox5.wikibase.cloud saf-wikibase-tutorial-1.wikibase.cloud saf-wikibase-tutorial-2.wikibase.cloud saf-wikibase-tutorial-3.wikibase.cloud saf-wikibase-tutorial-4.wikibase.cloud saf-wikibase-tutorial-5.wikibase.cloud sandbox.wikibase.cloud schema-korf.wikibase.cloud schema-org-7-0.wikibase.cloud semscape.org shexdocs.wikibase.cloud suriname_wildlife.semscape.org swat4hcls.wikibase.cloud symptoms.semscape.org taxonomy.wikibase.cloud tdwg-cv.wikibase.cloud test12.wikibase.cloud test13.wikibase.cloud testbed.wikibase.cloud testsaflux.wikibase.cloud trex1.wikibase.cloud wbdemo3.wikibase.cloud wbjupyter.wikibase.cloud wbpython.wikibase.cloud wbsdemo123456.wikibase.cloud wikidata-lenses.wikibase.cloud wildlife-sme.semscape.org zandbak.wikibase.cloud
Comment Actions
Hi, I am one of the Batch A users (http://lexbib.elex.is, among others). Everything works fine, many thanks!!
Two observations:
- I get 502 bad gateway quite often when submitting edits, have then to retry.
- All concept URI seem to have changed from http: to https: - This is indeed a problem. I'll have to manually update a lot of queries contained somewhere in my wikipages, update my python code, and update URI contained in other resources. Question: Will this change be forever? Or can I still choose between http and https? And, why this change? Wikdiata URI are also http...
Comment Actions
The HTTPS concept URI was covered in T299045 and T301141. T301258 was meant to cover pros and cons, but it was closed as invalid. I suspect they didn't want to get in a situation where they expanded the service using HTTP (which might feasibly result in more people accessing the site using HTTP, which exposes data to the network) and it took a while to fix like in T258590 for Commons or T226453 and T153563 for Wikidata itself (which in the end has not been switched) - or in T303777 for Wikimedia Enterprise where they have actually moved some stuff from HTTPS to HTTP.
I too had to change some queries but I had to do so anyway because the domain part of the concept URI was changed as well (which I highlighted in the WBStack/Wikibase.cloud Telegram chat), and perhaps that factored into their decision to change the scheme at the same time. I suspect relatively few WBStack users were on their own domain, which would not have had this issue if the scheme had not also changed.
Comment Actions
As for 502s, I'd be curious too (though I have not personally done enough edits since migration to experience them) but perhaps it was due to bot indexing and fixed by the response as mentioned in T308340? If it is a resource issue then quota issues mentioned in T305679 may need to be addressed if that has not already been done.
In any case, it might be a good idea to open a separate bug under the Wikibase Cloud project to address it, should they still be happening. And this is also something that should be tracked by logging/metrics - sort of like T302975 but looking at the logs rather than making repetitive accesses itself. This is perhaps a missing ticket under T299508 which does not seem to have anything looking at 502s.
Comment Actions
thanks @GreenReaper for chiming in, all right answers!
@DL2204 what @GreenReaper says is makes sense, to summarize from Wikimedia DE side:
- If you continue to experience issues with the service like those 502s, please report those on phabricator here under Wikibase Cloud tag
- Switch to HTTPS in concept URIs was indeed motivated with ensuring that if/when concept URI is used as an URL that the secure method is used. Apologies for failing to communicate that change in advance.