Page MenuHomePhabricator

Cloud VPS "wikibase-registry" project Stretch deprecation
Closed, ResolvedPublic

Description

The end of life of Debian Stretch is approaching in 2022 and we need to move to Debian Bullseye (or Buster) before that date.

All instances in the wikibase-registry project need to upgrade as soon as possible. Instances not upgraded by 2022-05-01 may be subject to deletion unless prior arrangements for an extended deadline has been approved by the Cloud VPS administration team.

Remaining Debian Stretch instances (live report):

Listed administrators are:

See also:

More info on current project instances is available via openstack browser.

Details

Due Date
Apr 30 2022, 11:59 PM

Event Timeline

StrikerBot triaged this task as Medium priority.Apr 13 2022, 5:00 PM
StrikerBot created this task.
Zabe changed the edit policy from "Custom Policy" to "All Users".Apr 13 2022, 5:31 PM

Flagging this with folks so that it gets prioritized

So, i think it's done but turns out the wdqs-updater has been broken since forever :D

https://openstack-browser.toolforge.org/project/wikibase-registry is cached now but should hopefully update and indicate that the proxies are using the new instance.

Ok the queryservice should now be updated with a fresh dump of the old instance and the queryservice-updater is running again.

How the stuff was moved between the instances.

  1. Create a new instance on https://horizon.wikimedia.org, it now has 20gb less space than the original. however we're currently only using 31% of the disk capacity so should be good.
  2. Install docker.io / docker-compose
  3. tar original files (it contained some changes outside of the git repo and will remain on the new instance), docker volumes on original wikibase-registry
  4. docker save wikibase-registry:1.32-bundle-1 > 1.32-bundle-1.tar to a tar (there is a blog post on how to do this but im lazy https://addshore.com/2019/02/creating-a-dockerfile-for-the-wikibase-registry/)
  5. haul all the tarballs to the new instance
  6. bring up everything docker-compose up and noticed wdqs-updater wasn't updating since a long while
  7. dump the rdf after https://github.com/wmde/wikibase-registry/pull/4 was merged
  8. Create a new empty volume with https://github.com/wmde/wikibase-registry/pull/5
  9. Copy the files from wikibase container to wdqs container
  10. munge the dump and ingest it
bash-4.4# java -cp lib/wikidata-query-tools-*-jar-with-dependencies.jar org.wikidata.query.rdf.tool.Munge --from /tmp/wikibase-registry.ttl --to /tmp/mungeOut/wikidump-%09d.ttl.gz --chunkSize 10000 -w wikibase-registry.wmflabs.org --conceptUri http://wikibase-registry.wmflabs.org
#logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
10:44:56.391 [main] INFO  org.wikidata.query.rdf.tool.Munge - Switching to /tmp/mungeOut/wikidump-000000001.ttl.gz
bash-4.4# ./loadData.sh -n wdq -d /tmp/mungeOut/
Processing wikidump-000000001.ttl.gz
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><meta http-equiv="Content-Type" content="text&#47;html;charset=UTF-8"><title>blazegraph&trade; by SYSTAP</title
></head
><body<p>totalElapsed=291ms, elapsed=223ms, connFlush=0ms, batchResolve=0, whereClause=0ms, deleteClause=0ms, insertClause=0ms</p
><hr><p>COMMIT: totalElapsed=713ms, commitTime=1656499525195, mutationCount=5271</p
></html
>File wikidump-000000002.ttl.gz not found, terminating
  1. switch over WebProxies to the new instance and seems to work

@WMDE-leszek give me the go ahead to delete the old instance and I will try to do that

toan removed toan as the assignee of this task.Jun 29 2022, 11:24 AM

Thanks @toan. I believe it looks all good. Please proceed with retiring the old instance.

Boldly resolving this task.