Debian Buster is well out of upstream support and all Buster VMs need to be replaced. The corresponding upstream restbase hosts are running Bullseye.
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T327742 Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm | |||
| Resolved | Eevans | T370460 Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) | |||
| Resolved | BUG REPORT | bd808 | T388043 No Puppet resources found on instance deployment-changeprop-1 on project deployment-prep |
Event Timeline
Comment Actions
@Jgiannelos hey! Is deployment-restbase-bullseye (created by you last year) ready to take over the work from restbase04? Other than changing the references to restbase04 in Horizon hiera and LabsServices.php, and in the changeprop Chart (deployment--charts), it should be possible to switch, although the restbase service is not listening to port 7231 on -bullseye - any idea what's wrong?
Comment Actions
Hi, i think this was an attempt to quickly assess how complicated it is to upgrade restbase to a newer version.
From what I see in the logs, restbase errors while trying to connect to cassandra and cassandra fails with
Condition check resulted in distributed storage system for structured data being skipped.
Comment Actions
@Jgiannelos do you have any suggestions about this ticket? Shall I just delete restbase04 and let the chips fall where they may?
Comment Actions
I don't think the bookworm instance is going to be up and running unless somebody who is familiar with cassandra and restbase fixes the errors.
Overall not having RESTBase on beta is not great (other parts of the stack depend on it).
Comment Actions
Hey, after some reports of CI failures caused by summary endpoint on beta I checked and the restbase instance (deployment-restbase-04) is down since 5th of Aug (if I read the logs correctly). As I mentioned above the bullseye node for restbase is not in a working state (it was just a fast WIP to see how much effort is needed for restbase to work with bullseye). Is there a plan to fix the cassandra issues on deployment-restbase-bullseye to bring it in a working state? Should we reconsider turning back on the deprecated VM?
Comment Actions
On 2024-08-05 I sent this email to the releng and sre mailing lists:
Today I am shutting down the following deployment-prep VMs as they are still running Debian Buster and appear to be abandoned. deployment-echostore02 https://phabricator.wikimedia.org/T361383 deployment-eventlog08 https://phabricator.wikimedia.org/T369918 deployment-maps-master01 https://phabricator.wikimedia.org/T361381 deployment-poolcounter06 https://phabricator.wikimedia.org/T370458 deployment-restbase04 https://phabricator.wikimedia.org/T370460 If you would like to revive one or more of them, please comment on the associated ticket so we can track what's happening.
We will need someone to complete the rebuild on Bullseye or Bookworm but I've been unable to find anyone willing to do so. Please do not restart the Buster VM unless there is a clear, timely plan to shut it down again shortly.
Comment Actions
Given that current production works on bullseye (i think ?) the migration is a matter of migrating/rebuilding the stateful part of cassandra. I can help with rebuilding the nodejs related part but I am not very familiar with the rest. Any idea who can help with that ?
Comment Actions
Sorry, I'm pretty much out of ideas at this point. I first announced this deprecation in March and I've spammed the cloud, sre, and releng list quite a lot.
Comment Actions
By way an of an update: I built a new instance —deployment-restbase05— that should be mostly "there". I'm currently stuck on getting the scap deploy to work (I can't seem to get scap to use the beta environment). I'll pick that back up on Monday, and see if I can't finish this.
Comment Actions
Change #1063260 had a related patch set uploaded (by Eevans; author: Eevans):
[mediawiki/services/restbase/deploy@master] Updated beta restbase host
Comment Actions
Change #1063260 merged by Eevans:
[mediawiki/services/restbase/deploy@master] Updated beta restbase host
Comment Actions
Change #1064427 had a related patch set uploaded (by Eevans; author: Eevans):
[operations/software/logstash-logback-encoder@master] beta: deployment-restbase05 as deployment target
Comment Actions
Change #1064427 merged by Eevans:
[operations/software/logstash-logback-encoder@master] beta: deployment-restbase05 as deployment target
Comment Actions
Ok, status update:
With the completion of T372921 restbase is now deployed via scap. It is running, as is Cassandra (complete with roles, schema, etc). Puppet runs clean. However, it does not seem like restbase is working properly:
eevans@deployment-restbase05:~$ check-restbase Traceback (most recent call last): File "/usr/bin/service-checker-swagger", line 11, in <module> load_entry_point('servicechecker==0.2.1', 'console_scripts', 'service-checker-swagger')() File "/usr/lib/python3/dist-packages/servicechecker/swagger.py", line 548, in main checker.run() File "/usr/lib/python3/dist-packages/servicechecker/swagger.py", line 161, in run checks = [{'ep': ep, 'data': data, 'job': gevent.spawn(self._check_endpoint, ep, data)} File "/usr/lib/python3/dist-packages/servicechecker/swagger.py", line 161, in <listcomp> checks = [{'ep': ep, 'data': data, 'job': gevent.spawn(self._check_endpoint, ep, data)} File "/usr/lib/python3/dist-packages/servicechecker/swagger.py", line 122, in get_endpoints for endpoint, data in r['paths'].items(): KeyError: 'paths' eevans@deployment-restbase05:~$
Some sort of configuration error? I'm going to tag in @Jgiannelos at this point (since that's what we'd do for production). Any suggestion @Jgiannelos ?
Comment Actions
Could it be something wrong with cassandra? Although I don't see how the traceback from check-restbase is related:
jgiannelos@deployment-restbase05:~$ curl -s 127.0.0.1:7233/en.wikipedia.beta.wmflabs.org/v1/page/html/Dog | jq .
{
"type": "https://mediawiki.org/wiki/HyperSwitch/errors/query_error",
"title": "Error in Cassandra table storage backend",
"method": "get",
"uri": "/en.wikipedia.beta.wmflabs.org/v1/page/html/Dog"
}Comment Actions
I think I found the issue:
- Service checker is testing against en.wikipedia.org
jgiannelos@deployment-restbase05:/srv/deployment/restbase/deploy/restbase$ cat $(which check-restbase) #!/bin/sh STATSD_HOST="localhost" STATSD_PORT="8125" STATSD_PREFIX="service_checker.restbase.deployment-restbase05" /usr/bin/service-checker-swagger -t 5 172.16.4.101 http://172.16.4.101:7231/en.wikipedia.org/v1
- RESTbase on deployment-prep is not configured for wikipedia prod but for wikipedia beta
- If I run the service checker with the correct path it fails because of the error above
jgiannelos@deployment-restbase05:/srv/deployment/restbase/deploy/restbase$ /usr/bin/service-checker-swagger -t 5 172.16.4.101 http://172.16.4.101:7231/en.wikipedia.beta.wmflabs.org/v1
/en.wikipedia.beta.wmflabs.org/v1/page/title/{title} (Get rev by title from storage) is CRITICAL: Test Get rev by title from storage returned the unexpected status 500 (expecting: 200); /en.wikipedia.beta.wmflabs.org/v1/page/html/{title} (Get html by title from storage) is CRITICAL: Test Get html by title from storage returned the unexpected status 500 (expecting: 200); /en.wikipedia.beta.wmflabs.org/v1/page/mobile-sections/{title} (Get mobile-sections for a test page on enwiki) is CRITICAL: Test Get mobile-sections for a test page on enwiki returned the unexpected status 500 (expecting: 200); /en.wikipedia.beta.wmflabs.org/v1/page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage returned the unexpected status 500 (expecting: 200); /en.wikipedia.beta.wmflabs.org/v1/page/media-list/{title} (Get media-list from storage) is CRITICAL: Test Get media-list from storage returned the unexpected status 500 (expecting: 200); /en.wikipedia.beta.wmflabs.org/v1/page/mobile-html/{title} (Get mobile-html from storage) is CRITICAL: Test Get mobile-html from storage returned the unexpected status 500 (expecting: 200); /en.wikipedia.beta.wmflabs.org/v1/page/mobile-html-offline-resources/{title} (Get offline resource links to accompany page content HTML for test page) is CRITICAL: Test Get offline resource links to accompany page content HTML for test page returned the unexpected status 504 (expecting: 200); /en.wikipedia.beta.wmflabs.org/v1/page/talk/{title} (Get structured talk page for enwiki Salt article) is CRITICAL: Test Get structured talk page for enwiki Salt article returned the unexpected status 500 (expecting: 200); /en.wikipedia.beta.wmflabs.org/v1/media/math/check/{type} (Mathoid - check test formula) is CRITICAL: Test Mathoid - check test formula returned the unexpected status 500 (expecting: 200)Comment Actions
I found this in the logs:
Aug 22 15:16:20 deployment-restbase05 restbase[335631]: {"name":"restbase","hostname":"deployment-restbase05","pid":14,"level":50,"message":"500: query_error","res":{"name":"HTTPError","message":"500: query_error","status":500,"body":{"type":"query_error","title":"Error in Cassandra table storage backend","stack":"Error: restbase-mod-table-cassandra: No schema for others_T_title__revisions3WsaB42Wia1E_eq_KmoYTHe, table: data\n at DB._getRaw (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/restbase-mod-table-cassandra/lib/db.js:261:19)\n at /srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/restbase-mod-table-cassandra/lib/db.js:248:25\n at tryCatcher (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/util.js:16:23)\n at Promise._settlePromiseFromHandler (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:547:31)\n at Promise._settlePromise (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:604:18)\n at Promise._settlePromise0 (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:649:10)\n at Promise._settlePromises (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:729:18)\n at Promise._fulfill (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:673:18)\n at Promise._resolveCallback (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:466:57)\n at Promise._settlePromiseFromHandler (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:559:17)\n at Promise._settlePromise (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:604:18)\n at Promise._settlePromise0 (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:649:10)\n at Promise._settlePromises (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:729:18)\n at Promise._fulfill (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:673:18)\n at Promise._resolveCallback (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:466:57)\n at Promise._settlePromiseFromHandler (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:559:17)","err":{},"req":{"uri":"/en.wikipedia.beta.wmflabs.org/sys/table/title_revisions-ng/","headers":{},"body":"{\"table\":\"title_revisions-ng\",\"attributes\":{\"title\":\"Dog\"},\"limit\":1}"}},"innerBody":"{\"type\":\"query_error\",\"title\":\"Error in Cassandra table storage backend\",\"stack\":\"Error: restbase-mod-table-cassandra: No schema for others_T_title__revisions3WsaB42Wia1E_eq_KmoYTHe, table: data\\n ","internalReq":{"method":"get","headers":{},"query":{},"body":"{\"table\":\"title_revisions-ng\",\"attributes\":{\"title\":\"Dog\"},\"limit\":1}"},"headers":{"content-location":"https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/html/Dog","access-control-allow-origin":"*","access-control-allow-methods":"GET,HEAD","access-control-allow-headers":"accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding","access-control-expose-headers":"etag","x-content-type-options":"nosniff","x-frame-options":"SAMEORIGIN","referrer-policy":"origin-when-cross-origin","x-xss-protection":"1; mode=block","content-security-policy":"default-src 'none'; frame-ancestors 'none'","x-content-security-policy":"default-src 'none'; frame-ancestors 'none'","x-webkit-csp":"default-src 'none'; frame-ancestors 'none'","cache-control":"private, max-age=0, s-maxage=0, must-revalidate","x-request-id":"7c7219d0-6099-11ef-9d3f-ab743dcc319e","server":"deployment-restbase05"}},"stack":"Error: restbase-mod-table-cassandra: No schema for others_T_title__revisions3WsaB42Wia1E_eq_KmoYTHe, table: data\n at DB._getRaw (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/restbase-mod-table-cassandra/lib/db.js:261:19)\n at /srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/restbase-mod-table-cassandra/lib/db.js:248:25\n at tryCatcher (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/util.js:16:23)\n at Promise._settlePromiseFromHandler (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:547:31)\n at Promise._settlePromise (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:604:18)\n at Promise._settlePromise0 (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:649:10)\n at Promise._settlePromises (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:729:18)\n at Promise._fulfill (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:673:18)\n at Promise._resolveCallback (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:466:57)\n at Promise._settlePromiseFromHandler (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:559:17)\n at Promise._settlePromise (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:604:18)\n at Promise._settlePromise0 (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:649:10)\n at Promise._settlePromises (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:729:18)\n at Promise._fulfill (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:673:18)\n at Promise._resolveCallback (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:466:57)\n at Promise._settlePromiseFromHandler (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:559:17)","latency":276,"root_req":{"method":"get","uri":"/en.wikipedia.beta.wmflabs.org/v1/page/html/Dog","headers":{"user-agent":"curl/7.74.0","x-client-ip":"::ffff:127.0.0.1","x-forwarded-for":"::ffff:127.0.0.1","x-request-id":"7c7219d0-6099-11ef-9d3f-ab743dcc319e","x-request-class":"internal"}},"request_id":"7c7219d0-6099-11ef-9d3f-ab743dcc319e","levelPath":"error/request","msg":"500: query_error","time":"2024-08-22T15:16:20.865Z","v":0}Which turned out to be an empty others_T_title__revisions3WsaB42Wia1E_eq_KmoYTHe.meta table. I copied the contents over from production, which seems to have fixed the example request above:
eevans@deployment-restbase05:~$ curl -s 127.0.0.1:7233/en.wikipedia.beta.wmflabs.org/v1/page/html/Dog 2>&1 |head -n 2 <!DOCTYPE html> <html prefix="dc: http://purl.org/dc/terms/ mw: http://mediawiki.org/rdf/" about="https://en.wikipedia.beta.wmflabs.org/wiki/Special:Redirect/revision/619443"><head prefix="mwr: https://en.wikipedia.beta.wmflabs.org/wiki/Special:Redirect/"><meta property="mw:TimeUuid" content="a3f27260-609a-11ef-b9c9-71da46f5e14f"/><meta charset="utf-8"/><meta property="mw:pageId" content="1457"/><meta property="mw:pageNamespace" content="0"/><link rel="dc:replaces" resource="mwr:revision/619435"/><meta property="mw:revisionSHA1" content="43208b242225a93e688392bdc5ee1e3cdace170a"/><meta property="dc:modified" content="2024-04-25T19:00:45.000Z"/><meta property="mw:htmlVersion" content="2.8.0"/><meta property="mw:html:version" content="2.8.0"/><link rel="dc:isVersionOf" href="https://en.wikipedia.beta.wmflabs.org/wiki/Dog"/><base href="https://en.wikipedia.beta.wmflabs.org/wiki/"/><title>Dog</title><meta property="mw:jsConfigVars" content='{"ScribuntoErrors":{"cae5f25a":true},"ScribuntoErrors-cae5f25a":"<p>Script error: The function \"speciesboxName\" does not exist.</p><p>No further details are available.</p>"}'/><meta property="mw:generalModules" content="ext.scribunto.errors|ext.cite.ux-enhancements|mediawiki.page.media|ext.tmh.player"/><meta property="mw:moduleStyles" content="ext.cite.parsoid.styles|ext.cite.styles|ext.tmh.player.styles"/><link rel="stylesheet" href="/w/load.php?lang=en&modules=ext.cite.parsoid.styles%7Cext.cite.styles%7Cext.tmh.player.styles%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.skinning.interface%7Csite.styles&only=styles&skin=vector"/><meta http-equiv="content-language" content="en"/><meta http-equiv="vary" content="Accept"/></head><body id="mwAA" lang="en" class="mw-content-ltr sitedir-ltr ltr mw-body-content parsoid-body mediawiki mw-parser-output" dir="ltr" data-mw-parsoid-version="0.20.0.0-alpha17" data-mw-html-version="2.8.0"><section data-mw-section-id="0" id="mwAQ"><div class="shortdescription nomobile noexcerpt noprint searchaux" style="display:none" about="#mwt1" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"short description","href":"./Template:Short_description"},"params":{"1":{"wt":"Domesticated canid species"}},"i":0}}]}' id="mwAg">Domesticated canid species</div><link rel="mw:PageProp/Category" href="./Category:Articles_with_short_description" about="#mwt1"/><link rel="mw:PageProp/Category" href="./Category:Short_description_is_different_from_Wikidata" about="#mwt1" id="mwAw"/> eevans@deployment-restbase05:~$
@Jgiannelos is there no way to provoke restbase to update the meta tables, or are we going to have to copy over the rest from production too?
Comment Actions
@Jgiannelos is there no way to provoke restbase to update the meta tables, or are we going to have to copy over the rest from production too?
I went ahead and copied them from production...
check-restbase is still provoking some errors though:
Aug 22 15:46:12 deployment-restbase05 restbase[336949]: {"name":"restbase","hostname":"deployment-restbase05","pid":14,"level":50,"message":"getaddrinfo ENOTFOUND deployment-docker-mobileapps01.deployment-prep.eqiad1.wikimedia.cloud","res":{"name":"HTTPError","message":"getaddrinfo ENOTFOUND deployment-docker-mobileapps01.deployment-prep.eqiad1.wikimedia.cloud","status":504,"headers":{"content-type":"application/problem+json","access-control-allow-origin":"*","access-control-allow-methods":"GET,HEAD","access-control-allow-headers":"accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding","access-control-expose-headers":"etag","x-content-type-options":"nosniff","x-frame-options":"SAMEORIGIN","referrer-policy":"origin-when-cross-origin","x-xss-protection":"1; mode=block","content-security-policy":"default-src 'none'; frame-ancestors 'none'","x-content-security-policy":"default-src 'none'; frame-ancestors 'none'","x-webkit-csp":"default-src 'none'; frame-ancestors 'none'","content-location":"https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/mobile-html/User%3ABSitzmann_(WMF)%2FMCS%2FTest%2FFrankenstein","cache-control":"private, max-age=0, s-maxage=0, must-revalidate","x-request-id":"a871ffb0-609d-11ef-a5c0-8fc0e420f039","server":"deployment-restbase05"},"body":{"type":"internal_http_error","detail":"getaddrinfo ENOTFOUND deployment-docker-mobileapps01.deployment-prep.eqiad1.wikimedia.cloud","internalStack":"Error: getaddrinfo ENOTFOUND deployment-docker-mobileapps01.deployment-prep.eqiad1.wikimedia.cloud\n at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)","internalURI":"http://deployment-docker-mobileapps01.deployment-prep.eqiad1.wikimedia.cloud:8888/en.wikipedia.beta.wmflabs.org/v1/page/mobile-html/User%3ABSitzmann_(WMF)%2FMCS%2FTest%2FFrankenstein","internalQuery":"{}","internalErr":"getaddrinfo ENOTFOUND deployment-docker-mobileapps01.deployment-prep.eqiad1.wikimedia.cloud","internalMethod":"get"}},"stack":"HTTPError: getaddrinfo ENOTFOUND deployment-docker-mobileapps01.deployment-prep.eqiad1.wikimedia.cloud\n at /srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/preq/index.js:246:19\n at tryCatcher (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/util.js:16:23)\n at Promise._settlePromiseFromHandler (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:547:31)\n at Promise._settlePromise (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:604:18)\n at Promise._settlePromise0 (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:649:10)\n at Promise._settlePromises (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/promise.js:725:18)\n at _drainQueueStep (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/async.js:93:12)\n at _drainQueue (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/async.js:86:9)\n at Async._drainQueues (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/async.js:102:5)\n at Immediate.Async.drainQueues [as _onImmediate] (/srv/deployment/restbase/deploy-cache/revs/b504108ebe6dc130da330c8b4411bd6f61cb6bf3/node_modules/bluebird/js/release/async.js:15:14)\n at processImmediate (internal/timers.js:461:21)","latency":328,"root_req":{"method":"get","uri":"/en.wikipedia.beta.wmflabs.org/v1/page/mobile-html/User%3ABSitzmann_%28WMF%29%2FMCS%2FTest%2FFrankenstein","headers":{"user-agent":"ServiceChecker-WMF/0.1.2","x-client-ip":"::ffff:172.16.4.101","x-forwarded-for":"::ffff:172.16.4.101","x-request-id":"a871ffb0-609d-11ef-a5c0-8fc0e420f039","x-request-class":"external"}},"request_id":"a871ffb0-609d-11ef-a5c0-8fc0e420f039","levelPath":"error/request","msg":"getaddrinfo ENOTFOUND deployment-docker-mobileapps01.deployment-prep.eqiad1.wikimedia.cloud","time":"2024-08-22T15:46:12.724Z","v":0}@Jgiannelos I guess this is a relatively straightforward configuration deploy (deployment-docker-mobileapps01.deployment-prep.eqiad1.wikimedia.cloud seems to have been superseded by deployment-docker-mobileapps02.deployment-prep.eqiad1.wikimedia.cloud)?
Comment Actions
Change #1065123 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[operations/puppet@production] restbase: Update mobileapps service hostname on deployment-prep
Comment Actions
Change #1065123 merged by Eevans:
[operations/puppet@production] restbase: Update mobileapps service hostname on deployment-prep
Comment Actions
Ok, it seems to be working now.
eevans@deployment-restbase05:~$ /usr/bin/service-checker-swagger -t 5 172.16.4.101 http://172.16.4.101:7231/en.wikipedia.beta.wmflabs.org/v1 All endpoints are healthy eevans@deployment-restbase05:~$
Comment Actions
Change #1065266 had a related patch set uploaded (by Eevans; author: Eevans):
[operations/mediawiki-config@master] Replace deployment-restbase04 w/ deployment-restbase05
Comment Actions
I think we can mark this closed once mw-config has been deployed (see: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1065266).
Comment Actions
Do we have a rough estimate of when https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1065266 will be deployed? I need to mute an alert and would like to know how long to do that for. Thanks in advance.
Comment Actions
Change #1065266 merged by jenkins-bot:
[operations/mediawiki-config@master] Replace deployment-restbase04 w/ deployment-restbase05
Comment Actions
@Jgiannelos https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/summary/Polar_bear is still not resolving. I'm getting "Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes." Should I open a new ticket?
Comment Actions
The same title directly requested from restbase works as expected:
jgiannelos@deployment-restbase05:~$ curl -v -o /dev/null 127.0.0.1:7231/en.wikipedia.beta.wmflabs.org/v1/page/summary/Polar_bear
* Trying 127.0.0.1:7231...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 127.0.0.1 (127.0.0.1) port 7231 (#0)
> GET /en.wikipedia.beta.wmflabs.org/v1/page/summary/Polar_bear HTTP/1.1
> Host: 127.0.0.1:7231
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< content-type: application/json; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/Summary/1.5.0"
< etag: "629130/a2c20830-66ba-11ef-a165-604f84726ac0"
< cache-control: s-maxage=1209600, max-age=300
< content-language: en
< vary: Accept-Encoding
< content-location: https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/summary/Polar_bear
< access-control-allow-origin: *
< access-control-allow-methods: GET,HEAD
< access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
< access-control-expose-headers: etag
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< referrer-policy: origin-when-cross-origin
< x-xss-protection: 1; mode=block
< content-security-policy: default-src 'none'; frame-ancestors 'none'
< x-content-security-policy: default-src 'none'; frame-ancestors 'none'
< x-webkit-csp: default-src 'none'; frame-ancestors 'none'
< x-request-id: ca704400-66ba-11ef-aa18-37ff2cfdf45f
< server: deployment-restbase05
< content-length: 3305
< Date: Fri, 30 Aug 2024 10:29:51 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
{ [3305 bytes data]
100 3305 100 3305 0 0 27773 0 --:--:-- --:--:-- --:--:-- 27773
* Connection #0 to host 127.0.0.1 left intactComment Actions
cloudsearch still returns some references to the old restbase node on deployment-prep, let me see if i can change them.
Comment Actions
Change #1069145 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[operations/deployment-charts@master] changeprop: Update references to latest beta restbase node
Comment Actions
Change #1069148 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[operations/puppet@production] Update references to latest beta restbase node
Comment Actions
Ok it works now:
https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/summary/Polar_bear returns 200
Comment Actions
Change #1069148 merged by Eevans:
[operations/puppet@production] Update references to latest beta restbase node
Comment Actions
https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-daily-beta-Popups/ is passing again! Thanks for getting this updated!
Comment Actions
Change #1069145 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Update references to latest beta restbase node