Page MenuHomePhabricator
Create Task
Maniphest T387631

Are eqiad.mediawiki.job.CampaignEventsFindPotentialInvitees jobs being processed in beta?
Closed, ResolvedPublic
Actions

Description

It seems like jobs are not being enqueued in the job queue in beta. For example, the Special:GenerateInvitationList page uses the job queue, but it is currently not working (i.e., things remain in a "pending" state forever). Logstash showed no trace of clear errors, so I went on deployment-mwlog02 to see the logs, but there's no JobExecutor.log. The last one is from two days ago, but in the meantime things have definitely happen that should have enqueued jobs (for example, edits). Also, the jobqueue is empty:

daimona@deployment-deploy04:~$ mwscript showJobs.php --wiki enwiki
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
https://wikitech.wikimedia.org/wiki/Maintenance_scripts for the new process.
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
0

So, it seems to me that we may not be enqueuing jobs.

Related Objects

Event Timeline

Daimona created this task.Mar 1 2025, 8:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 1 2025, 8:40 PM
Ottomata subscribed.Mar 18 2025, 4:13 PM

I don't know what is wrong, but some details to help you search:

  • To see if a job is being enqueued, you can consume the relevant kafka topic, e.g.
ssh deployment-kafka-main-6.deployment-prep.eqiad1.wikimedia.cloud

# show job topics:
kafka topics --list  | grep mediawiki.job

# consume a job topic, starting from the latest offset / end of the topic
kafkacat -b localhost:9092 -C -t eqiad.mediawiki.job.RecordLintJob -o end
  • Perform an action on MediaWiki in beta that would enqueue a job, ensure that the job event message is consumed by your kafkacat command.

Some discussion in Slack seems to indicate this might have been caused by Cloud DNS flakiness?

bd808 subscribed.EditedMar 18 2025, 4:22 PM
In T387631#10647478, @Ottomata wrote:

Some discussion in Slack seems to indicate this might have been caused by Cloud DNS flakiness?

That speculation was I think mostly from the one debug log event in the last 24 hours (https://beta-logs.wmcloud.org/goto/3b71b4d593faa3cfef918ab8e9e0c5d2) indicating a failure to post to http://deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloud:8292/v1/events caused by a DNS lookup failure. I would expect that if DNS was the root problem here we would see a lot more logging like that.

bd808 added a comment.Mar 18 2025, 4:28 PM

Also, the jobqueue is empty:

I think that test is red herring. The showJobs.php maintenance script does not work the https://wikitech.wikimedia.org/wiki/MediaWiki_JobQueue setup as far as I know. Here's the same check in WMF production:

bd808@mwmaint2002:~$ mwscript showJobs.php --wiki enwiki
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
https://wikitech.wikimedia.org/wiki/Maintenance_scripts for the new process.
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
0
bd808 added a comment.Mar 18 2025, 4:50 PM

Enqueuing a job with eval.php seems to work:

bd808@deployment-mwmaint03:~$ mwscript eval.php enwiki --d 1
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
https://wikitech.wikimedia.org/wiki/Maintenance_scripts for the new process.
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
> MediaWiki\MediaWikiServices::getInstance()->getJobQueueGroup()->push( new NullJob( [] ) );
[debug] [memcached] Wikimedia\ObjectCache\MemcachedPeclBagOStuff::initializeClient: initializing new client instance.
[debug] [memcached] MainWANObjectCache using store Wikimedia\ObjectCache\MemcachedPeclBagOStuff
[debug] [memcached] Wikimedia\ObjectCache\MemcachedPeclBagOStuff::initializeClient: initializing new client instance.
[debug] [memcached] MicroStash using store Wikimedia\ObjectCache\MemcachedPeclBagOStuff
[debug] [localisation] LocalisationCache using store LCStoreCDB
[debug] MediaWiki\Parser\ParserFactory: using default preprocessor
[debug] [EventStreamConfig] Selecting stream configs for target streams: mediawiki.job.null
[debug] [EventStreamConfig] Selecting stream configs for target streams: mediawiki.job.null
[debug] [EventBus] Using event intake service eventgate-main for stream mediawiki.job.null.
[debug] [http] HTTP start: POST http://deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloud:8292/v1/events
[debug] [http] HTTP complete: POST http://deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloud:8292/v1/events code=201 size=0 total=1.914623 connect=0.004705
[debug] [memcached] Wikimedia\ObjectCache\MemcachedPeclBagOStuff debug: set(global:jobqueue:enwiki:hasjobs:2)
[debug] [memcached] Wikimedia\ObjectCache\MemcachedPeclBagOStuff debug: set(global:jobqueue:enwiki:hasjobs:1)
bd808@deployment-kafka-main-6:~$ kafka topics --list  | grep mediawiki.job.null
eqiad.mediawiki.job.null
bd808@deployment-kafka-main-6:~$ kafkacat -b localhost:9092 -C -t eqiad.mediawiki.job.null
{"$schema":"/mediawiki/job/1.0.0","meta":{"uri":"https://placeholder.invalid/wiki/Special:Badtitle","request_id":"69a6430af5475f9ddb13e0df","id":"3884037e-5cc7-488a-b919-5e6eab905076","dt":"2025-03-18T16:44:48Z","domain":"en.wikipedia.beta.wmflabs.org","stream":"mediawiki.job.null"},"database":"enwiki","type":"null","params":{"requestId":"69a6430af5475f9ddb13e0df","lives":1,"usleep":0},"mediawiki_signature":"85fba4b414f39e8997832c692fa3b8d359b94261"}
% Reached end of topic eqiad.mediawiki.job.null [0] at offset 1
Daimona added a comment.Mar 18 2025, 4:54 PM

Looks like the job I was complaining about also gets enqueued:

daimona@deployment-kafka-main-6:~$ kafkacat -b localhost:9092 -C -t eqiad.mediawiki.job.CampaignEventsFindPotentialInvitees -o end
% Reached end of topic eqiad.mediawiki.job.CampaignEventsFindPotentialInvitees [0] at offset 83
{"$schema":"/mediawiki/job/1.0.0","meta":{"uri":"https://placeholder.invalid/wiki/Special:Badtitle","request_id":"Z9mkiG39R8bRNag5GtoqUQAAAFA","id":"05c2092f-bd12-4ce7-8ad4-c75e71918bb2","dt":"2025-03-18T16:51:20Z","domain":"en.wikipedia.beta.wmflabs.org","stream":"mediawiki.job.CampaignEventsFindPotentialInvitees"},"database":"enwiki","type":"CampaignEventsFindPotentialInvitees","params":{"list-id":84,"serialized-worklist":{"enwiki":[[318860,0,"Conflict-title-0.10652077860573383-Iñtërnâtiônàlizætiøn",false],[318859,0,"Conflict-title-0.06786106461025132-Iñtërnâtiônàlizætiøn",false],[318858,0,"Conflict-title-0.33735833858628683-Iñtërnâtiônàlizætiøn",false]]},"requestId":"Z9mkiG39R8bRNag5GtoqUQAAAFA"},"mediawiki_signature":"0a4edd5fa6237a166278925c68d7f277d229b545"}
% Reached end of topic eqiad.mediawiki.job.CampaignEventsFindPotentialInvitees [0] at offset 84

And yet it still shows as processing: https://en.wikipedia.beta.wmflabs.org/wiki/Special:InvitationList/84. So it must be something else.

bd808 renamed this task from Jobs are not being enqueued in beta to Are eqiad.mediawiki.job.CampaignEventsFindPotentialInvitees jobs being processed in beta?.Mar 18 2025, 4:56 PM
bd808 added a comment.Mar 18 2025, 5:07 PM
root@deployment-changeprop-1:~# systemctl status changeprop --no-pager -l
● changeprop.service - Systemd runner for changeprop
     Loaded: loaded (/lib/systemd/system/changeprop.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-03-06 19:03:56 UTC; 1 week 4 days ago
   Main PID: 1065910 (docker)
      Tasks: 9 (limit: 4685)
     Memory: 24.7M
        CPU: 4min 46.632s
     CGroup: /system.slice/changeprop.service
             └─1065910 /usr/bin/docker run --rm=true --env-file /etc/changeprop/env -p 7272:7272 -v changeprop:/etc/changeprop --name changeprop.service docker-registry.wikimedia.org/wikimedia/mediawiki-services-change-propagation:2025-03-06-075118-production /srv/service/server.js -c /etc/changeprop/config.yaml

Mar 18 16:31:10 deployment-changeprop-1 docker-changeprop[1065910]: {"name":"changeprop","hostname":"9389dac532fa","pid":1,"level":"ERROR","message":"Exec error in changeprop","status":404,"event_str":"{\"$schema\":\"/resource_change/1.0.0\",\"meta\":{\"uri\":\"https://wikifunctions.beta.wmflabs.org/wiki/Z20\",\"request_id\":\"b932ff675fa9b7b5975c3ae8\",\"id\":\"4f17fad0-fbb4-461b-ac55-19957b31ffb1\",\"dt\":\"2025-03-18T16:31:10Z\",\"domain\":\"wikifunctions.beta.wmflabs.org\",\"stream\":\"resource_change\"},\"tags\":[\"purge\"]}","stream":"resource_change","error_body_str":"{\"type\":\"https://mediawiki.org/wiki/HyperSwitch/errors/not_found#route\",\"title\":\"Not found.\",\"method\":\"get\",\"uri\":\"/wikifunctions.beta.wmflabs.org/v1/page/html/Z20\",\"internalURI\":\"http://deployment-restbase05.deployment-prep.eqiad1.wikimedia.cloud:7231/wikifunctions.beta.wmflabs.org/v1/page/html/Z20\",\"internalMethod\":\"get\"}","rule_name":"mw_purge","executor":"RuleExecutor","levelPath":"error/exec_error","msg":"Exec error in changeprop","time":"2025-03-18T16:31:10.612Z","v":0}
Mar 18 16:31:10 deployment-changeprop-1 docker-changeprop[1065910]: {"name":"changeprop","hostname":"9389dac532fa","pid":1,"level":"ERROR","message":"Exec error in changeprop","status":404,"event_str":"{\"$schema\":\"/resource_change/1.0.0\",\"meta\":{\"uri\":\"https://wikifunctions.beta.wmflabs.org/wiki/Z86\",\"request_id\":\"b932ff675fa9b7b5975c3ae8\",\"id\":\"d93ef119-cc63-435f-af82-9635f8c644e7\",\"dt\":\"2025-03-18T16:31:10Z\",\"domain\":\"wikifunctions.beta.wmflabs.org\",\"stream\":\"resource_change\"},\"tags\":[\"purge\"]}","stream":"resource_change","error_body_str":"{\"type\":\"https://mediawiki.org/wiki/HyperSwitch/errors/not_found#route\",\"title\":\"Not found.\",\"method\":\"get\",\"uri\":\"/wikifunctions.beta.wmflabs.org/v1/page/html/Z86\",\"internalURI\":\"http://deployment-restbase05.deployment-prep.eqiad1.wikimedia.cloud:7231/wikifunctions.beta.wmflabs.org/v1/page/html/Z86\",\"internalMethod\":\"get\"}","rule_name":"mw_purge","executor":"RuleExecutor","levelPath":"error/exec_error","msg":"Exec error in changeprop","time":"2025-03-18T16:31:10.807Z","v":0}
Mar 18 16:31:11 deployment-changeprop-1 docker-changeprop[1065910]: {"name":"changeprop","hostname":"9389dac532fa","pid":1,"level":"ERROR","message":"Exec error in changeprop","status":404,"event_str":"{\"$schema\":\"/resource_change/1.0.0\",\"meta\":{\"uri\":\"https://wikifunctions.beta.wmflabs.org/wiki/Z8882\",\"request_id\":\"b932ff675fa9b7b5975c3ae8\",\"id\":\"c233938f-cbe6-46e4-8409-af758f766d0a\",\"dt\":\"2025-03-18T16:31:10Z\",\"domain\":\"wikifunctions.beta.wmflabs.org\",\"stream\":\"resource_change\"},\"tags\":[\"purge\"]}","stream":"resource_change","error_body_str":"{\"type\":\"https://mediawiki.org/wiki/HyperSwitch/errors/not_found#route\",\"title\":\"Not found.\",\"method\":\"get\",\"uri\":\"/wikifunctions.beta.wmflabs.org/v1/page/html/Z8882\",\"internalURI\":\"http://deployment-restbase05.deployment-prep.eqiad1.wikimedia.cloud:7231/wikifunctions.beta.wmflabs.org/v1/page/html/Z8882\",\"internalMethod\":\"get\"}","rule_name":"mw_purge","executor":"RuleExecutor","levelPath":"error/exec_error","msg":"Exec error in changeprop","time":"2025-03-18T16:31:11.010Z","v":0}
Mar 18 16:32:26 deployment-changeprop-1 docker-changeprop[1065910]: {"name":"changeprop","hostname":"9389dac532fa","pid":1,"level":"ERROR","message":"Exec error in changeprop","status":504,"event_str":"{\"$schema\":\"/change-prop/retry/1.0.0\",\"meta\":{\"stream\":\"changeprop.retry.resource_change\",\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"domain\":\"en.wikipedia.beta.wmflabs.org\",\"id\":\"bd218400-0415-11f0-9f00-853f2897c76c\",\"dt\":\"2025-03-18T16:26:26.240Z\",\"request_id\":\"bd218401-0415-11f0-9f00-853f2897c76c\"},\"triggered_by\":\"req:Z9medK15qNO_CoD2nr-9XQAAAE4,resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2,changeprop.retry.resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"emitter_id\":\"changeprop#null_edit\",\"retries_left\":1,\"original_event\":{\"$schema\":\"/resource_change/1.0.0\",\"meta\":{\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"request_id\":\"Z9medK15qNO_CoD2nr-9XQAAAE4\",\"id\":\"216029e5-3546-4b0b-87e1-8c3feba28898\",\"dt\":\"2025-03-18T16:25:24Z\",\"domain\":\"en.wikipedia.beta.wmflabs.org\",\"stream\":\"resource_change\"},\"tags\":[\"null_edit\"]},\"error_status\":504}","stream":"changeprop.retry.resource_change","error_body_str":"{\"type\":\"https://mediawiki.org/wiki/HyperSwitch/errors/internal_http_error\",\"method\":\"get\",\"detail\":\"getaddrinfo ENOTFOUND parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud\",\"uri\":\"/en.wikipedia.beta.wmflabs.org/v1/page/html/Page_popups_test_page_2\",\"internalURI\":\"http://deployment-restbase05.deployment-prep.eqiad1.wikimedia.cloud:7231/en.wikipedia.beta.wmflabs.org/v1/page/html/Page_popups_test_page_2\",\"internalMethod\":\"get\"}","rule_name":"null_edit","executor":"RetryExecutor","levelPath":"error/exec_error","msg":"Exec error in changeprop","time":"2025-03-18T16:32:26.570Z","v":0}
Mar 18 16:32:26 deployment-changeprop-1 docker-changeprop[1065910]: {"name":"changeprop","hostname":"9389dac532fa","pid":1,"level":"WARN","message":"Retry count exceeded","event_str":"{\"$schema\":\"/change-prop/retry/1.0.0\",\"meta\":{\"stream\":\"changeprop.retry.resource_change\",\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"domain\":\"en.wikipedia.beta.wmflabs.org\"},\"triggered_by\":\"req:Z9medK15qNO_CoD2nr-9XQAAAE4,resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2,changeprop.retry.resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2,changeprop.retry.resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"emitter_id\":\"changeprop#null_edit\",\"retries_left\":0,\"original_event\":{\"$schema\":\"/resource_change/1.0.0\",\"meta\":{\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"request_id\":\"Z9medK15qNO_CoD2nr-9XQAAAE4\",\"id\":\"216029e5-3546-4b0b-87e1-8c3feba28898\",\"dt\":\"2025-03-18T16:25:24Z\",\"domain\":\"en.wikipedia.beta.wmflabs.org\",\"stream\":\"resource_change\"},\"tags\":[\"null_edit\"]},\"error_status\":504}","status":504,"page":"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2","description":"getaddrinfo ENOTFOUND parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud","stack":"HTTPError: getaddrinfo ENOTFOUND parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud\n    at request.then.query (/srv/service/node_modules/preq/index.js:228:23)\n    at tryCatcher (/srv/service/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/srv/service/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/srv/service/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/srv/service/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/srv/service/node_modules/bluebird/js/release/promise.js:729:18)\n    at _drainQueueStep (/srv/service/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/srv/service/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/srv/service/node_modules/bluebird/js/release/async.js:102:5)\n    at Async.drainQueues [as _onImmediate] (/srv/service/node_modules/bluebird/js/release/async.js:15:14)\n    at process.processImmediate (node:internal/timers:478:21)","stream":"changeprop.retry.resource_change","rule_name":"null_edit","executor":"RetryExecutor","levelPath":"warn/retry_count","msg":"Retry count exceeded","time":"2025-03-18T16:32:26.572Z","v":0}
Mar 18 16:32:26 deployment-changeprop-1 docker-changeprop[1065910]: {"name":"change-propagation","hostname":"9389dac532fa","pid":1,"level":"ERROR","err":{"message":"Local: Maximum application poll interval (max.poll.interval.ms) exceeded","name":"changeprop","stack":"Error: Local: Maximum application poll interval (max.poll.interval.ms) exceeded\n    at Function.createLibrdkafkaError [as create] (/srv/service/node_modules/node-rdkafka/lib/error.js:456:10)\n    at /srv/service/node_modules/node-rdkafka/lib/kafka-consumer.js:468:29","code":-147,"errno":-147,"origin":"kafka","rule_name":"null_edit","executor":"RetryExecutor","levelPath":"error/consumer"},"msg":"Local: Maximum application poll interval (max.poll.interval.ms) exceeded","time":"2025-03-18T16:32:26.588Z","v":0}
Mar 18 16:32:28 deployment-changeprop-1 docker-changeprop[1065910]: {"name":"changeprop","hostname":"9389dac532fa","pid":1,"level":"ERROR","message":"Exec error in changeprop","status":504,"event_str":"{\"$schema\":\"/change-prop/retry/1.0.0\",\"meta\":{\"stream\":\"changeprop.retry.resource_change\",\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"domain\":\"en.wikipedia.beta.wmflabs.org\",\"id\":\"bd218400-0415-11f0-9f00-853f2897c76c\",\"dt\":\"2025-03-18T16:26:26.240Z\",\"request_id\":\"bd218401-0415-11f0-9f00-853f2897c76c\"},\"triggered_by\":\"req:Z9medK15qNO_CoD2nr-9XQAAAE4,resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2,changeprop.retry.resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"emitter_id\":\"changeprop#null_edit\",\"retries_left\":1,\"original_event\":{\"$schema\":\"/resource_change/1.0.0\",\"meta\":{\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"request_id\":\"Z9medK15qNO_CoD2nr-9XQAAAE4\",\"id\":\"216029e5-3546-4b0b-87e1-8c3feba28898\",\"dt\":\"2025-03-18T16:25:24Z\",\"domain\":\"en.wikipedia.beta.wmflabs.org\",\"stream\":\"resource_change\"},\"tags\":[\"null_edit\"]},\"error_status\":504}","stream":"changeprop.retry.resource_change","error_body_str":"{\"type\":\"https://mediawiki.org/wiki/HyperSwitch/errors/internal_http_error\",\"method\":\"get\",\"detail\":\"getaddrinfo ENOTFOUND parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud\",\"uri\":\"/en.wikipedia.beta.wmflabs.org/v1/page/html/Page_popups_test_page_2\",\"internalURI\":\"http://deployment-restbase05.deployment-prep.eqiad1.wikimedia.cloud:7231/en.wikipedia.beta.wmflabs.org/v1/page/html/Page_popups_test_page_2\",\"internalMethod\":\"get\"}","rule_name":"null_edit","executor":"RetryExecutor","levelPath":"error/exec_error","msg":"Exec error in changeprop","time":"2025-03-18T16:32:28.366Z","v":0}
Mar 18 16:32:28 deployment-changeprop-1 docker-changeprop[1065910]: {"name":"changeprop","hostname":"9389dac532fa","pid":1,"level":"WARN","message":"Retry count exceeded","event_str":"{\"$schema\":\"/change-prop/retry/1.0.0\",\"meta\":{\"stream\":\"changeprop.retry.resource_change\",\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"domain\":\"en.wikipedia.beta.wmflabs.org\"},\"triggered_by\":\"req:Z9medK15qNO_CoD2nr-9XQAAAE4,resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2,changeprop.retry.resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2,changeprop.retry.resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"emitter_id\":\"changeprop#null_edit\",\"retries_left\":0,\"original_event\":{\"$schema\":\"/resource_change/1.0.0\",\"meta\":{\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2\",\"request_id\":\"Z9medK15qNO_CoD2nr-9XQAAAE4\",\"id\":\"216029e5-3546-4b0b-87e1-8c3feba28898\",\"dt\":\"2025-03-18T16:25:24Z\",\"domain\":\"en.wikipedia.beta.wmflabs.org\",\"stream\":\"resource_change\"},\"tags\":[\"null_edit\"]},\"error_status\":504}","status":504,"page":"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page_2","description":"getaddrinfo ENOTFOUND parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud","stack":"HTTPError: getaddrinfo ENOTFOUND parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud\n    at request.then.query (/srv/service/node_modules/preq/index.js:228:23)\n    at tryCatcher (/srv/service/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/srv/service/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/srv/service/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/srv/service/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/srv/service/node_modules/bluebird/js/release/promise.js:729:18)\n    at _drainQueueStep (/srv/service/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/srv/service/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/srv/service/node_modules/bluebird/js/release/async.js:102:5)\n    at Async.drainQueues [as _onImmediate] (/srv/service/node_modules/bluebird/js/release/async.js:15:14)\n    at process.processImmediate (node:internal/timers:478:21)","stream":"changeprop.retry.resource_change","rule_name":"null_edit","executor":"RetryExecutor","levelPath":"warn/retry_count","msg":"Retry count exceeded","time":"2025-03-18T16:32:28.368Z","v":0}
Mar 18 16:32:28 deployment-changeprop-1 docker-changeprop[1065910]: {"name":"changeprop","hostname":"9389dac532fa","pid":1,"level":"ERROR","message":"Exec error in changeprop","status":504,"event_str":"{\"$schema\":\"/change-prop/retry/1.0.0\",\"meta\":{\"stream\":\"changeprop.retry.resource_change\",\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page\",\"domain\":\"en.wikipedia.beta.wmflabs.org\",\"id\":\"bdeff790-0415-11f0-9f00-853f2897c76c\",\"dt\":\"2025-03-18T16:26:27.593Z\",\"request_id\":\"bdeff791-0415-11f0-9f00-853f2897c76c\"},\"triggered_by\":\"req:Z9medq15qNO_CoD2nr-9ZQAAAE4,resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page,changeprop.retry.resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page\",\"emitter_id\":\"changeprop#null_edit\",\"retries_left\":1,\"original_event\":{\"$schema\":\"/resource_change/1.0.0\",\"meta\":{\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page\",\"request_id\":\"Z9medq15qNO_CoD2nr-9ZQAAAE4\",\"id\":\"f0b402d7-8fe7-422c-97dc-a665f33b19f9\",\"dt\":\"2025-03-18T16:25:26Z\",\"domain\":\"en.wikipedia.beta.wmflabs.org\",\"stream\":\"resource_change\"},\"tags\":[\"null_edit\"]},\"error_status\":504}","stream":"changeprop.retry.resource_change","error_body_str":"{\"type\":\"https://mediawiki.org/wiki/HyperSwitch/errors/internal_http_error\",\"method\":\"get\",\"detail\":\"getaddrinfo ENOTFOUND parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud\",\"uri\":\"/en.wikipedia.beta.wmflabs.org/v1/page/html/Page_popups_test_page\",\"internalURI\":\"http://deployment-restbase05.deployment-prep.eqiad1.wikimedia.cloud:7231/en.wikipedia.beta.wmflabs.org/v1/page/html/Page_popups_test_page\",\"internalMethod\":\"get\"}","rule_name":"null_edit","executor":"RetryExecutor","levelPath":"error/exec_error","msg":"Exec error in changeprop","time":"2025-03-18T16:32:28.638Z","v":0}
Mar 18 16:32:28 deployment-changeprop-1 docker-changeprop[1065910]: {"name":"changeprop","hostname":"9389dac532fa","pid":1,"level":"WARN","message":"Retry count exceeded","event_str":"{\"$schema\":\"/change-prop/retry/1.0.0\",\"meta\":{\"stream\":\"changeprop.retry.resource_change\",\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page\",\"domain\":\"en.wikipedia.beta.wmflabs.org\"},\"triggered_by\":\"req:Z9medq15qNO_CoD2nr-9ZQAAAE4,resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page,changeprop.retry.resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page,changeprop.retry.resource_change:https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page\",\"emitter_id\":\"changeprop#null_edit\",\"retries_left\":0,\"original_event\":{\"$schema\":\"/resource_change/1.0.0\",\"meta\":{\"uri\":\"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page\",\"request_id\":\"Z9medq15qNO_CoD2nr-9ZQAAAE4\",\"id\":\"f0b402d7-8fe7-422c-97dc-a665f33b19f9\",\"dt\":\"2025-03-18T16:25:26Z\",\"domain\":\"en.wikipedia.beta.wmflabs.org\",\"stream\":\"resource_change\"},\"tags\":[\"null_edit\"]},\"error_status\":504}","status":504,"page":"https://en.wikipedia.beta.wmflabs.org/wiki/Page_popups_test_page","description":"getaddrinfo ENOTFOUND parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud","stack":"HTTPError: getaddrinfo ENOTFOUND parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud\n    at request.then.query (/srv/service/node_modules/preq/index.js:228:23)\n    at tryCatcher (/srv/service/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/srv/service/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/srv/service/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/srv/service/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/srv/service/node_modules/bluebird/js/release/promise.js:729:18)\n    at _drainQueueStep (/srv/service/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/srv/service/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/srv/service/node_modules/bluebird/js/release/async.js:102:5)\n    at Async.drainQueues [as _onImmediate] (/srv/service/node_modules/bluebird/js/release/async.js:15:14)\n    at process.processImmediate (node:internal/timers:478:21)","stream":"changeprop.retry.resource_change","rule_name":"null_edit","executor":"RetryExecutor","levelPath":"warn/retry_count","msg":"Retry count exceeded","time":"2025-03-18T16:32:28.640Z","v":0}
bd808 added a comment.Mar 18 2025, 5:10 PM

And suddenly I remember working on T388043: No Puppet resources found on instance deployment-changeprop-1 on project deployment-prep recently.

Daimona mentioned this in T389239: Bring beta-logs.wmcloud.org to a readable state.Mar 18 2025, 6:05 PM
Stashbot mentioned this in T387276: PHP on Beta cluster fails due to mismatching PCRE.Mar 18 2025, 8:25 PM

Mentioned in SAL (#wikimedia-releng) [2025-03-18T20:25:03Z] <bd808> Rebooting deployment-jobrunner05 because things just seem weird (T387631, T387276)

Daimona closed this task as Resolved.Mar 18 2025, 8:59 PM
Daimona assigned this task to bd808.

The above seems to have done it, thank you! \o/

I was able to generate https://en.wikipedia.beta.wmflabs.org/wiki/Special:InvitationList/85

Restricted Application added a project: User-bd808. · View Herald TranscriptMar 18 2025, 8:59 PM
Daimona added a comment.Mar 18 2025, 9:05 PM

Also, double-checking:

daimona@deployment-mwlog02:~$ wc -l /srv/mw-log/JobExecutor.log 
3027 /srv/mw-log/JobExecutor.log

and these are also logged in beta logstash. So everything seems back to normal.

Daimona mentioned this in T400928: Spike - check if participants are receiving email from organizer and event registration confirmation on beta and prod.Jul 31 2025, 10:54 PM
Maintenance_bot added a project: MediaWiki-API-Platform-Team.Jul 31 2025, 11:30 PM
Daimona mentioned this in T401002: eqiad.mediawiki.job.sendCampaignEmail (and possibly others) seemingly aren't enqueued in beta (kafka split-brain).Aug 1 2025, 1:56 PM
Daimona mentioned this in T422842: Kafka-topics broken in beta: "zookeeper is not a recognized option".Apr 9 2026, 3:12 PM
Daimona mentioned this in T423615: Jobs are not being processed in beta, April 2026 edition.Apr 16 2026, 3:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits