Page MenuHomePhabricator
Create Task
Maniphest T387208

Ensure tls-proxy container is started before launching main container
Closed, ResolvedPublic
Actions

Description

The k8s-controller-sidecar we are using as a stopgap until T386694: Replace k8s-controller-sidecars with built in Sidecar containers on k8s 1.31 does not ensure sidecars are completely started before launching the main container with its payload.

This causes unexpected failure modes where for instance mediawiki tries to reach out to a service through the tls-proxy sidecar but it is not started yet.

We should find a way to ensure proper functionality of at least the envoy tls-proxy before starting mediawiki payloads in mw-cron and mw-script

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
MWScript.php: exit code on mesh, longer timeoutoperations/mediawiki-configmaster+6 -4
php: mwscript bugfixoperations/docker-images/production-imagesmaster+15 -1
Revert^2 "When executing cli scripts, wait for the service mesh"operations/mediawiki-configmaster+64 -1
When executing cli scripts, wait for the service meshoperations/mediawiki-configmaster+54 -1
mwscript: Do not run mesh checks in loopsoperations/puppetproduction+8 -0
mwscript: do not run mesh checks when running in a loopoperations/docker-images/production-imagesmaster+16 -0
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
mwscript: Skip mesh checkrepos/releng/scap!670cgoubertT387208master
Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolveddancyT392693 Drop support for Debian Buster
 ResolvedClement_GoubertT341560 Migrate mwmaint server functionality to mw-on-k8s
 ResolvedClement_GoubertT341555 Implement periodic maintenance scripts for mw-on-k8s
 OpenRLazarusT341553 Allow running one-off scripts manually
 ResolvedRLazarusT348284 Handle sidecar containers in one-off Kubernetes jobs
 ResolvedRLazarusT359127 MW image version for maintenance scripts
 ResolvedRLazarusT367200 mw-script fails to render in CI
 ResolvedRLazarusT368966 Pipe stdin into one-off maintenance scripts on Kubernetes
 ResolvedRLazarusT369142 Show more useful information when mwscript-k8s fails to launch
 DeclinedNoneT369143 Allow cleaning up specific mwscript-k8s runs
 ResolvedRLazarusT369175 mwscript-k8s --attach error: TypeError: 'NoneType' object is not iterable
 ResolvedBUG REPORTRLazarusT369676 `mwscript-k8s --attach` seems to terminate IO after a few seconds without input
 ResolvedRLazarusT376099 --timeout flag for mwscript-k8s
 ResolvedRLazarusT376230 Support bringing text files into the container for one-off maintenance scripts
 ResolvedLucas_Werkmeister_WMDET376604 [PS] Update PropertySuggester update process for mwscript-k8s
 ResolvedArian_BozorgT377986 Q4 2024 update of Property Suggester data
 OpenNoneT376616 MWScript.php doesn't allow wikiless scripts without the .php suffix
 ResolvedRLazarusT376714 Evaluate running a statsd-exporter in the mw-script namespace
 ResolvedRLazarusT376776 mw-scripts SAL integration
 ResolvedRLazarusT377292 Support machine-readable output for mwscript-k8s
 ResolvedRLazarusT378429 Allow members of restricted to run maintenance scripts
 ResolvedRLazarusT378479 Allow using helper scripts inside of mwscript-k8s
 OpenNoneT378754 Persistent logging of mwscript-k8s runs
 OpenNoneT379675 Support output files in mwscript-k8s
 ResolvedRLazarusT380925 Support passing env variables to maintenance scripts in mwscript-k8s
 ResolvedtstarlingT381251 MW maintenance scripts on k8s can't do internal HTTP requests
 DuplicateNoneT382398 Mediawiki maint scripts using service proxied by the tls proxy might fail when running with mwscript-k8s
 InvalidNoneT385818 mwscript-k8s fails when there are parenthesis used in arguments
 DuplicateRLazarusT387127 mwscript-k8s purgeList does not reliably purge cached URLs
 ResolvedJoeT387208 Ensure tls-proxy container is started before launching main container
 ResolvedhasharT387480 Beta update job fails: The service mesh is unavailable, which can lead to unexpected results.
 OpenNoneT390972 Restart CronJobs on failure of the service mesh
 OpenRLazarusT387268 Kubernetes maintenance script status UI
 ResolvedClement_GoubertT389484 Create a mediawiki-cli image
 ResolvedScott_FrenchT389499 Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds
 OpenNoneT394534 Visualise mw-script jobs
ResolvedRLazarusT401737 Custom dblists for mwscript-k8s

Event Timeline

Clement_Goubert created this task.Feb 25 2025, 11:24 AM
Clement_Goubert added a parent task: T341555: Implement periodic maintenance scripts for mw-on-k8s.
gerritbot added a comment.Feb 25 2025, 1:33 PM

Change #1122578 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/mediawiki-config@master] When executing cli scripts, wait for the service mesh

https://gerrit.wikimedia.org/r/1122578

gerritbot added a project: Patch-For-Review.Feb 25 2025, 1:33 PM
gerritbot added a comment.Feb 25 2025, 3:11 PM

Change #1122606 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/docker-images/production-images@master] mwscript: do not run mesh checks when running in a loop

https://gerrit.wikimedia.org/r/1122606

Joe subscribed.Feb 25 2025, 3:31 PM

In the case of pods accepting traffic, our readiness probe should be enough to ensure this.

For scripts, I think the simplest thing is to add a curl request in MWScript.php, which I did now.

I added the ability to skip the mesh check that is useful in foreachwiki (see the patch to production-images attached to the task) and foreachwikiindblist.

What remains to be done:

  • Add the same trick to the puppet versions of the foreachwiki.. scripts
  • [Debatable] add the ability to mwscript-k8s to pass the env variable to skip the check to k8s. This should be done as part of the parent task.
RLazarus merged a task: T387127: mwscript-k8s purgeList does not reliably purge cached URLs.Feb 25 2025, 5:17 PM
Clement_Goubert triaged this task as High priority.Feb 26 2025, 4:03 PM
gerritbot added a comment.Feb 27 2025, 1:57 PM

Change #1123377 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] mwscript: Do not run mesh checks in loops

https://gerrit.wikimedia.org/r/1123377

gerritbot added a comment.Feb 27 2025, 3:37 PM

Change #1122606 merged by Clément Goubert:

[operations/docker-images/production-images@master] mwscript: do not run mesh checks when running in a loop

https://gerrit.wikimedia.org/r/1122606

Stashbot added a comment.Feb 27 2025, 3:37 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-27T15:37:43Z] <claime> Rebuilding php images - T387208

gerritbot added a comment.Feb 27 2025, 3:41 PM

Change #1123377 merged by Clément Goubert:

[operations/puppet@production] mwscript: Do not run mesh checks in loops

https://gerrit.wikimedia.org/r/1123377

gerritbot added a comment.Feb 27 2025, 3:45 PM

Change #1122578 merged by jenkins-bot:

[operations/mediawiki-config@master] When executing cli scripts, wait for the service mesh

https://gerrit.wikimedia.org/r/1122578

Stashbot added a comment.Feb 27 2025, 3:46 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-27T15:46:05Z] <cgoubert@deploy2002> Started scap sync-world: Backport for [[gerrit:1122578|When executing cli scripts, wait for the service mesh (T387208)]]

Clement_Goubert added a comment.Feb 27 2025, 4:11 PM

Had to revert the mediawiki change as scap uses MWScript.php in a few places and this breaks it since there's no mesh on the deployment hosts.
I'll inverse the logic of the patch tomorrow so we explicitly *enable* the mesh check in mw-script and mw-cron, with a possible override, and disable the check in the general case, that way there's no need to modify scap.

We can revisit making the check default when all use-cases except scap have been moved to kubernetes.

Joe added a comment.Feb 27 2025, 4:14 PM
In T387208#10587678, @Clement_Goubert wrote:

Had to revert the mediawiki change as scap uses MWScript.php in a few places and this breaks it since there's no mesh on the deployment hosts.
I'll inverse the logic of the patch tomorrow so we explicitly *enable* the mesh check in mw-script and mw-cron, with a possible override, and disable the check in the general case, that way there's no need to modify scap.

We can revisit making the check default when all use-cases except scap have been moved to kubernetes.

Scap is the exception, not the rule. A lot of maintenance scripts might be broken if the mesh isn't available. This is a common issue on k8s but it's still a very valid check on bare metal.

Maintenance_bot removed a project: Patch-For-Review.Feb 27 2025, 4:30 PM
Clement_Goubert added a subtask: T387480: Beta update job fails: The service mesh is unavailable, which can lead to unexpected results..Feb 28 2025, 11:40 AM
CodeReviewBot added a project: Patch-For-Review.Feb 28 2025, 12:27 PM

cgoubert opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/670

mwscript: Skip mesh check

CodeReviewBot added a comment.Feb 28 2025, 5:33 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/670

mwscript: Skip mesh check

Maintenance_bot removed a project: Patch-For-Review.Feb 28 2025, 6:30 PM
gerritbot added a comment.Mar 3 2025, 12:25 PM

Change #1124051 had a related patch set uploaded (by Clément Goubert; author: Giuseppe Lavagetto):

[operations/mediawiki-config@master] Revert^2 "When executing cli scripts, wait for the service mesh"

https://gerrit.wikimedia.org/r/1124051

gerritbot added a project: Patch-For-Review.Mar 3 2025, 12:25 PM
gerritbot added a comment.Mar 4 2025, 4:47 PM

Change #1124051 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert^2 "When executing cli scripts, wait for the service mesh"

https://gerrit.wikimedia.org/r/1124051

Stashbot added a comment.Mar 4 2025, 4:47 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-04T16:47:35Z] <cgoubert@deploy2002> Started scap sync-world: Backport for [[gerrit:1124051|Revert^2 "When executing cli scripts, wait for the service mesh" (T387208)]]

Stashbot added a comment.Mar 4 2025, 4:50 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-04T16:50:32Z] <cgoubert@deploy2002> cgoubert, oblivian: Backport for [[gerrit:1124051|Revert^2 "When executing cli scripts, wait for the service mesh" (T387208)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Mar 4 2025, 4:58 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-04T16:58:18Z] <cgoubert@deploy2002> Finished scap sync-world: Backport for [[gerrit:1124051|Revert^2 "When executing cli scripts, wait for the service mesh" (T387208)]] (duration: 10m 42s)

Maintenance_bot removed a project: Patch-For-Review.Mar 4 2025, 5:31 PM
RLazarus closed this task as Resolved.Mar 4 2025, 9:37 PM
RLazarus assigned this task to Joe.

There's a nondeterministic element to the original bug obviously, but as far as I can tell from repeated testing on mw-script, this is now working consistently. Thanks!

bking subscribed.Mar 4 2025, 9:47 PM
Clement_Goubert mentioned this in T388538: Migrate discovery-search jobs to mw-cron.Mar 17 2025, 4:54 PM
Clement_Goubert mentioned this in T382398: Mediawiki maint scripts using service proxied by the tls proxy might fail when running with mwscript-k8s.Mar 17 2025, 4:57 PM
dcausse merged a task: T382398: Mediawiki maint scripts using service proxied by the tls proxy might fail when running with mwscript-k8s.Mar 17 2025, 5:04 PM
dcausse added subscribers: dcausse, BTullis, brouberol.
gerritbot added a comment.Apr 9 2025, 8:23 AM

Change #1135379 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/docker-images/production-images@master] php: mwscript bugfix

https://gerrit.wikimedia.org/r/1135379

gerritbot added a project: Patch-For-Review.Apr 9 2025, 8:23 AM
gerritbot added a comment.Apr 9 2025, 8:36 AM

Change #1133935 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/mediawiki-config@master] MWScript.php: exit code on mesh, longer timeout

https://gerrit.wikimedia.org/r/1133935

gerritbot added a comment.Apr 10 2025, 9:39 AM

Change #1135379 merged by Clément Goubert:

[operations/docker-images/production-images@master] php: mwscript bugfix

https://gerrit.wikimedia.org/r/1135379

Stashbot added a comment.Apr 10 2025, 9:40 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-10T09:40:06Z] <claime> Rebuilding php base images to pick up 1135379 - T387208

Stashbot added a comment.Apr 10 2025, 9:44 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-10T09:44:04Z] <cgoubert@deploy1003> Started scap sync-world: Rebuilding mediawiki images to pick up new base images 1135379 - T387208

Stashbot added a comment.Apr 10 2025, 10:19 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-10T10:19:15Z] <cgoubert@deploy1003> sync-world aborted: Rebuilding mediawiki images to pick up new base images 1135379 - T387208 (duration: 35m 23s)

Stashbot added a comment.Apr 10 2025, 10:19 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-10T10:19:35Z] <cgoubert@deploy1003> Started scap sync-world: Rebuilding mediawiki images to pick up new base images 1135379 - T387208

gerritbot added a comment.Apr 10 2025, 10:45 AM

Change #1133935 merged by jenkins-bot:

[operations/mediawiki-config@master] MWScript.php: exit code on mesh, longer timeout

https://gerrit.wikimedia.org/r/1133935

Stashbot added a comment.Apr 10 2025, 10:45 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-10T10:45:56Z] <cgoubert@deploy1003> Started scap sync-world: Backport for [[gerrit:1133935|MWScript.php: exit code on mesh, longer timeout (T390972 T387208)]]

Stashbot mentioned this in T390972: Restart CronJobs on failure of the service mesh.Apr 10 2025, 10:45 AM
Stashbot added a comment.Apr 10 2025, 10:54 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-10T10:54:08Z] <cgoubert@deploy1003> cgoubert: Backport for [[gerrit:1133935|MWScript.php: exit code on mesh, longer timeout (T390972 T387208)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Apr 10 2025, 11:08 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-10T11:08:11Z] <cgoubert@deploy1003> Finished scap sync-world: Backport for [[gerrit:1133935|MWScript.php: exit code on mesh, longer timeout (T390972 T387208)]] (duration: 22m 15s)

Stashbot added a comment.Apr 10 2025, 11:28 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-10T11:28:54Z] <claime> Rebuilding php base images to pick up 1135694 - T387208

Maintenance_bot removed a project: Patch-For-Review.Apr 10 2025, 11:30 AM
Stashbot added a comment.Apr 10 2025, 11:32 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-10T11:32:06Z] <cgoubert@deploy1003> Started scap sync-world: Rebuilding mediawiki images to pick up new base images 1135694 - T387208

Stashbot added a comment.Apr 10 2025, 12:15 PM

Mentioned in SAL (#wikimedia-operations) [2025-04-10T12:15:32Z] <cgoubert@deploy1003> Finished scap sync-world: Rebuilding mediawiki images to pick up new base images 1135694 - T387208 (duration: 44m 51s)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits