Page MenuHomePhabricator

toan (Tobias Andersson (WMDE))
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
May 6 2020, 9:37 AM (112 w, 12 h)
Availability
Available
LDAP User
Tobias Andersson
MediaWiki User
Tobias Andersson (WMDE) [ Global Accounts ]

Recent Activity

Today

toan placed T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user up for grabs.
Wed, Jun 29, 3:55 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user from Done to Backlog on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Wed, Jun 29, 3:54 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user.

We just reverted this deployment on production it seems to have caused issues with the queryservice-updater.

Wed, Jun 29, 3:53 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan placed T306105: Cloud VPS "wikibase-registry" project Stretch deprecation up for grabs.
Wed, Jun 29, 11:24 AM · Wikibase.cloud (Launch Migration Kanban (2022)), wmde-team-b-tech, Cloud-VPS (Debian Stretch Deprecation)
toan added a comment to T306105: Cloud VPS "wikibase-registry" project Stretch deprecation.

@WMDE-leszek give me the go ahead to delete the old instance and I will try to do that

Wed, Jun 29, 11:24 AM · Wikibase.cloud (Launch Migration Kanban (2022)), wmde-team-b-tech, Cloud-VPS (Debian Stretch Deprecation)
toan added a comment to T306105: Cloud VPS "wikibase-registry" project Stretch deprecation.

Ok the queryservice should now be updated with a fresh dump of the old instance and the queryservice-updater is running again.

Wed, Jun 29, 11:20 AM · Wikibase.cloud (Launch Migration Kanban (2022)), wmde-team-b-tech, Cloud-VPS (Debian Stretch Deprecation)
toan assigned T311040: Elasticsearch not updating to Rosalie_WMDE.
Wed, Jun 29, 6:51 AM · Wikibase.cloud (Launch Migration Kanban (2022))

Yesterday

toan added a comment to T306105: Cloud VPS "wikibase-registry" project Stretch deprecation.

https://openstack-browser.toolforge.org/project/wikibase-registry is cached now but should hopefully update and indicate that the proxies are using the new instance.

Tue, Jun 28, 4:08 PM · Wikibase.cloud (Launch Migration Kanban (2022)), wmde-team-b-tech, Cloud-VPS (Debian Stretch Deprecation)
toan added a comment to T306105: Cloud VPS "wikibase-registry" project Stretch deprecation.

So, i think it's done but turns out the wdqs-updater has been broken since forever :D

Tue, Jun 28, 4:05 PM · Wikibase.cloud (Launch Migration Kanban (2022)), wmde-team-b-tech, Cloud-VPS (Debian Stretch Deprecation)
toan moved T311040: Elasticsearch not updating from Backlog to Launch Migration Kanban (2022) on the Wikibase.cloud board.
Tue, Jun 28, 1:27 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T306105: Cloud VPS "wikibase-registry" project Stretch deprecation from Backlog to Doing on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Tue, Jun 28, 11:38 AM · Wikibase.cloud (Launch Migration Kanban (2022)), wmde-team-b-tech, Cloud-VPS (Debian Stretch Deprecation)
toan claimed T306105: Cloud VPS "wikibase-registry" project Stretch deprecation.
Tue, Jun 28, 11:18 AM · Wikibase.cloud (Launch Migration Kanban (2022)), wmde-team-b-tech, Cloud-VPS (Debian Stretch Deprecation)
toan moved T311495: Wikibase.cloud instance does not update triple store when after edits in the wiki from Backlog to Launch Migration Kanban (2022) on the Wikibase.cloud board.
Tue, Jun 28, 11:14 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) from Blocked/Stalled to Done on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Tue, Jun 28, 11:12 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan closed T310685: Production warning: Node OOMKilling something as Resolved.
Tue, Jun 28, 11:00 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310685: Production warning: Node OOMKilling something.

We haven't seen any of these errors in a while for the nodes, i'd suggest we close this ticket as resolved or declined.

Tue, Jun 28, 7:58 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310121: Production warning: [RedisBagOStuff] Rejected set() for X due to snapshot lag (late regeneration)..

So, another update. Last 7 days this occurred 24 times in total and has not happened since june 24th.

Tue, Jun 28, 7:54 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T310121: Production warning: [RedisBagOStuff] Rejected set() for X due to snapshot lag (late regeneration)..
Tue, Jun 28, 7:50 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .

Since the pods got more memory last week we've had 10 aborted clients.

Tue, Jun 28, 7:26 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan created T311474: Production error/logspam: [error] Queryservice-ui can't open "/usr/share/nginx/html/custom-config.json".
Tue, Jun 28, 6:49 AM · Wikibase.cloud

Thu, Jun 23

toan added a comment to T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .

Ok, we are definitely seeing this again and this time it seems like normal usage through the api which might indicate something still is broken.

Thu, Jun 23, 4:12 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan placed T310697: [timebox 16hrs] Investigate using Prometheus locally to monitor k8s PV utilisation up for grabs.
Thu, Jun 23, 3:59 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310697: [timebox 16hrs] Investigate using Prometheus locally to monitor k8s PV utilisation.

https://github.com/wmde/wbaas-deploy/pull/446

Thu, Jun 23, 3:50 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310697: [timebox 16hrs] Investigate using Prometheus locally to monitor k8s PV utilisation.

Ok, so I tried out installing this for a while on staging and it indeed seems to be working pretty much out of the box, some dashboards still do not report the metrics i suppose they should but overall seems promising.

Thu, Jun 23, 3:48 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310697: [timebox 16hrs] Investigate using Prometheus locally to monitor k8s PV utilisation.

So starting off i had a wee look at https://github.com/bitnami/charts/tree/master/bitnami/kube-prometheus and realized I'd have to connect all the pieces myself to get the full stack with alertmanager etc.

Thu, Jun 23, 2:25 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) from Review to Blocked/Stalled on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Thu, Jun 23, 11:26 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .

Looking at this together with @Rosalie_WMDE we saw this just happened for a bunch of request, lets atleast figure out what happened there and see if it's some other bot or user.

Thu, Jun 23, 11:26 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .
Thu, Jun 23, 11:25 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan updated subscribers of T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .
Thu, Jun 23, 11:16 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan updated the task description for T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user.
Thu, Jun 23, 9:33 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user.

sweet, seems to work!

Thu, Jun 23, 9:33 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user from Deploy To Production to Product Verification on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Thu, Jun 23, 9:33 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user.
Thu, Jun 23, 9:09 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T307566: Review and test wbstack/api #429 from Review to Done on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Thu, Jun 23, 9:07 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T307566: Review and test wbstack/api #429.

Seems to work, moving to done.

Thu, Jun 23, 9:07 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan updated the task description for T307566: Review and test wbstack/api #429.
Thu, Jun 23, 9:07 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T305429: Review API: "Fix incorrect storage class #391" from Backlog to Done on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Thu, Jun 23, 8:55 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan assigned T305429: Review API: "Fix incorrect storage class #391" to Rosalie_WMDE.
Thu, Jun 23, 8:55 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan created T311205: Upgrade kubernetes from 1.21.
Thu, Jun 23, 8:38 AM · Wikibase.cloud

Wed, Jun 22

toan added a comment to T307566: Review and test wbstack/api #429.

lets cutting a new chart and just change the image value for each environment, it's already there.

Wed, Jun 22, 2:47 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T307566: Review and test wbstack/api #429.
Wed, Jun 22, 2:39 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan placed T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user up for grabs.
Wed, Jun 22, 2:38 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user.

fantastic, it works!

Wed, Jun 22, 2:38 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user from Review to Deploy To Production on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Wed, Jun 22, 2:38 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user.
Wed, Jun 22, 2:34 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T309070: Frequent 502 responses when submitting edits from Up Next to Launch Migration Kanban (2022) on the Wikibase.cloud board.
Wed, Jun 22, 2:07 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310705: Resolve PHP Dependabot PRs from Review to Done on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Wed, Jun 22, 2:05 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310705: Resolve PHP Dependabot PRs.

Sweet, looks like we got them all. thanks!

Wed, Jun 22, 2:05 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T310705: Resolve PHP Dependabot PRs.
Wed, Jun 22, 2:03 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user from Deploy To Staging to Backlog on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Wed, Jun 22, 11:38 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan reassigned T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user from toan to Deniz_WMDE.
Wed, Jun 22, 11:38 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user.

deployed https://github.com/wmde/wbaas-deploy/pull/418 to staging but the email is still exposing the internal mediawiki ip

Wed, Jun 22, 11:37 AM · Wikibase.cloud (Launch Migration Kanban (2022))

Tue, Jun 21

toan moved T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user from Review to Deploy To Staging on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Tue, Jun 21, 3:48 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T311040: Elasticsearch not updating.

Jobs are currently executed on requests, and you can keep an eye on the number of jobs in the queue by checking https://beyond-notability.wikibase.cloud/w/api.php?action=query&meta=siteinfo&siprop=statistics&format=json

Tue, Jun 21, 3:07 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T311040: Elasticsearch not updating.

Hello @Drjwbaker there is currently no dedicated job-runner for wikibase.cloud and there could therefore be a delay in how fast the items are updated/added to elasticsearch

Tue, Jun 21, 3:04 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310333: Production error: api pods keeps getting restarted.

Should we add it for queue & scheduler too?

Tue, Jun 21, 1:12 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310697: [timebox 16hrs] Investigate using Prometheus locally to monitor k8s PV utilisation from Blocked/Stalled to Doing on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Tue, Jun 21, 11:20 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan placed T310333: Production error: api pods keeps getting restarted up for grabs.
Tue, Jun 21, 11:15 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310333: Production error: api pods keeps getting restarted.

https://github.com/wbstack/charts/pull/101

Tue, Jun 21, 11:15 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310333: Production error: api pods keeps getting restarted.

Will do the same change for that pod.

Tue, Jun 21, 10:52 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T310333: Production error: api pods keeps getting restarted.
Tue, Jun 21, 10:52 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan renamed T310333: Production error: api pods keeps getting restarted from Production error: api-web pod keeps getting restarted to Production error: api pods keeps getting restarted.
Tue, Jun 21, 10:52 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310333: Production error: api pods keeps getting restarted from Blocked/Stalled to Doing on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Tue, Jun 21, 10:52 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310697: [timebox 16hrs] Investigate using Prometheus locally to monitor k8s PV utilisation from Doing to Blocked/Stalled on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Tue, Jun 21, 10:52 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310333: Production error: api pods keeps getting restarted.

Ok, i didn't realize it but the backend pod is suffering from the same problems, it's using the default values in the probes.

Tue, Jun 21, 10:51 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310685: Production warning: Node OOMKilling something.

None of the nodes seem to have reports of these events any longer.

Tue, Jun 21, 9:52 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310697: [timebox 16hrs] Investigate using Prometheus locally to monitor k8s PV utilisation from Backlog to Doing on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Tue, Jun 21, 9:52 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T310697: [timebox 16hrs] Investigate using Prometheus locally to monitor k8s PV utilisation.
Tue, Jun 21, 9:52 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan placed T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) up for grabs.
Tue, Jun 21, 9:32 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .

So, after a few days the large amount of aborted connections have stopped with the drop in traffic from the bot.

Tue, Jun 21, 9:21 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user.

new mw tag https://github.com/wbstack/mediawiki/pull/263

Tue, Jun 21, 7:48 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user.
Tue, Jun 21, 7:37 AM · Wikibase.cloud (Launch Migration Kanban (2022))

Fri, Jun 17

toan placed T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user up for grabs.
Fri, Jun 17, 3:23 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T309687: Registering on wikibase.cloud exposes internal IP rather than the registering user.
Fri, Jun 17, 2:35 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310333: Production error: api pods keeps getting restarted.

needs another prod deployment after staging.

Fri, Jun 17, 2:21 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan merged task T308519: api-web pod for production has failing readiness/liveness checks into T310333: Production error: api pods keeps getting restarted.
Fri, Jun 17, 2:19 PM · Wikibase.cloud
toan merged T308519: api-web pod for production has failing readiness/liveness checks into T310333: Production error: api pods keeps getting restarted.
Fri, Jun 17, 2:19 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan placed T310333: Production error: api pods keeps getting restarted up for grabs.
Fri, Jun 17, 2:17 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310333: Production error: api pods keeps getting restarted.

timeout is too short.

Fri, Jun 17, 2:17 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan updated the task description for T310333: Production error: api pods keeps getting restarted.
Fri, Jun 17, 2:17 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) from Done to Blocked/Stalled on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Fri, Jun 17, 1:05 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan updated subscribers of T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .

thanks @Deniz_WMDE for deploying the changes, moving back to blocked/stalled to keep and eye on it.

Fri, Jun 17, 1:05 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .
Fri, Jun 17, 1:04 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310333: Production error: api pods keeps getting restarted from Backlog to Doing on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Fri, Jun 17, 11:12 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T310333: Production error: api pods keeps getting restarted.
Fri, Jun 17, 11:12 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan placed T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) up for grabs.
Fri, Jun 17, 10:25 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .

since we don't have a robots file yet, this has been previously done in the nginx-ingress chart by specifying the user-agent pattern.

Fri, Jun 17, 10:25 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .

Ok, so looking in to these premature closures of the upstream it seems most of these are due a bot.

Fri, Jun 17, 9:53 AM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) from Blocked/Stalled to Doing on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Fri, Jun 17, 7:18 AM · Wikibase.cloud (Launch Migration Kanban (2022))

Thu, Jun 16

toan added a comment to T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .

So, giving up that idea I had a look in trying to correlate the name of the mediawiki database with the actual domain used for the requests I think i found something interesting.

Thu, Jun 16, 8:17 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310597: Production warning: Aborted connection 38950 to db: 'mwdb_wbstack_X' user: 'mwu_X' host: '10.108.4.7' (Got an error reading communication packets) .

The theory being, this is a resource/caching-configuration problem and we were seeing way less of these errors for a few days to a week after the secondary pod was OOMkilled last time.

Thu, Jun 16, 7:43 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310475: Perform Migration of Batch C from Product Verification to Done on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Thu, Jun 16, 2:32 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan placed T310475: Perform Migration of Batch C up for grabs.
Thu, Jun 16, 2:30 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310475: Perform Migration of Batch C from Review to Product Verification on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Thu, Jun 16, 2:30 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan added a comment to T310475: Perform Migration of Batch C.

Batch C looks good to me,

Thu, Jun 16, 2:30 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T310475: Perform Migration of Batch C.
Thu, Jun 16, 1:24 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310786: Change docker-compose setup to not depend on previous builds from Review to Done on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Thu, Jun 16, 1:23 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan placed T310786: Change docker-compose setup to not depend on previous builds up for grabs.
Thu, Jun 16, 1:23 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan claimed T310786: Change docker-compose setup to not depend on previous builds.
Thu, Jun 16, 1:12 PM · Wikibase.cloud (Launch Migration Kanban (2022))
toan moved T310586: Increase Platform Nginx resources to match WBStack from Deploy To Production to Done on the Wikibase.cloud (Launch Migration Kanban (2022)) board.
Thu, Jun 16, 1:12 PM · Wikibase.cloud (Launch Migration Kanban (2022))