User Details
- User Since
- Jun 9 2015, 9:03 AM (407 w, 12 h)
- Availability
- Available
- IRC Nick
- dcausse
- LDAP User
- DCausse
- MediaWiki User
- DCausse (WMF) [ Global Accounts ]
Today
@ItamarWMDE once https://gerrit.wikimedia.org/r/900729 is deployed we should be able to create a grafana query like the one suggested by Joe and adjust the threshold to double check that the query does what we need, I'll ping you once ready :)
Wed, Mar 22
Tue, Mar 21
Mon, Mar 20
Thu, Mar 16
We might have to switch to KafkaSource and this might change the behavior of flink during this kind of scenario.
Hi everyone and sorry to jump into this conversion but just wanted to add a quick note on the usefulness of cross-DC replication as it's something that was not obvious to us when we started to use swift containers in thanos for our search jobs.
If you consider all the dependent services&platforms:
- wikikube
- kafka-main
- MW mw-async* (hopefully a new mw-async-ro should be available at some point)
- thanos*
(* for services that I think have an automatic failover via dns discovery in place)
Thanos being replicated this brings down to only 2 the services/platforms that would cause a downtime to our job if they're down.
Mon, Mar 13
@TJones indeed I think that the query should explicitly add labels.ko to to the filter, it does seem to only add a scoring clause. Pushed a small patch to change how the filter is constructed.
Thanks for the detailed report!
Adding DEFAULTSORT to autocomplete searches is a feature that we can enable on a per-wiki basis, due to the way this tag is used it can't be enabled on every wiki without prior evaluation (see T145427#3515817). This feature was first enabled on mongolian wikipedia a couple weeks ago (see T327878). If this is something the frwiki community would like to experiment with we could enable it.
Thu, Mar 9
WDQS lag issues should be rare now, a node not serving traffic should (as of today) be able to ingest ~8x the throughput that we usually see on wikidata so we should not worry about them.
Using blazegraph_queries_done_total to identify nodes that should be part of the maxlag propagation makes a ton of sense, there might be some adjustments to make in how this metric is produced. It currently also include update queries which might make the threshold a bit hard to adapt esp. on codfw that receives a lot less trafic than eqiad.
If there are no strong reasons against this idea I'm happy to make the necessary adjustments to blazegraph_queries_done_total (or possibly introduce a new metric suited for this purpose).
Tue, Mar 7
can confirm it's fixed by the above patch, saw multiple alerts of this kind being fired during the k8s upgrade. Thanks!
The problem should be resolved, new page moves across namespaces should properly delete the page in the old namespace. Some phantom redirects created because of this bug will stay until the process that cleanups the index fixes these pages (it can take up to 8weeks), if you identify annoying ones please do let us know in this ticket so that we can clean them up manually quicker. Sorry for the inconvenience this has caused.
Mon, Mar 6
Feb 24 2023
Feb 23 2023
Thanks for fixing the issue and dealing with failed sensors! :)
Regarding what we could do to mitigate this in the future I don't think there exists great solution out of the box...
One could set soft_fail=True on their sensors but that means you have to set a meaningful timeout and not sure the benefit would be worth the possible harm in case the data finally appears a couple hours later...
Perhaps the solution to creating these "empty" partitions could be done without relying on canary events?
Feb 22 2023
Are there ways to unblock it? It's causing plenty of hourly jobs to fail on our side.
I think @Ottomata used to take care of this and was not sure what tags to add so please feel free to remove it from here :)
org.wikimedia.analytics.refinery.job.ProduceCanaryEvents seems stuck since yesterday maint operation on the k8s cluster, one thread seems blocked on:
"main" #1 prio=5 os_prio=0 tid=0x00007f301000b000 nid=0x138e runnable [0x00007f3015216000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:464) at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1346) at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:962) at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.wikimedia.eventutilities.core.http.BasicHttpClient.post(BasicHttpClient.java:107) at org.wikimedia.eventutilities.monitoring.CanaryEventProducer.postEvents(CanaryEventProducer.java:286) at org.wikimedia.eventutilities.monitoring.CanaryEventProducer.lambda$postEventsToUris$2(CanaryEventProducer.java:253) at org.wikimedia.eventutilities.monitoring.CanaryEventProducer$$Lambda$52/478182173.apply(Unknown Source) at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) at java.util.stream.Collectors$$Lambda$37/1365790282.accept(Unknown Source) at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1723) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.wikimedia.eventutilities.monitoring.CanaryEventProducer.postEventsToUris(CanaryEventProducer.java:251) at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$.produceCanaryEvents(ProduceCanaryEvents.scala:307) at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$$anonfun$4$$anonfun$apply$1.apply$mcZ$sp(ProduceCanaryEvents.scala:212) at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$$anonfun$4$$anonfun$apply$1.apply(ProduceCanaryEvents.scala:212) at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$$anonfun$4$$anonfun$apply$1.apply(ProduceCanaryEvents.scala:212) at scala.util.Try$.apply(Try.scala:192) at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$$anonfun$4.apply(ProduceCanaryEvents.scala:211) at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$$anonfun$4.apply(ProduceCanaryEvents.scala:210) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$.apply(ProduceCanaryEvents.scala:210) at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$.main(ProduceCanaryEvents.scala:147) at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents.main(ProduceCanaryEvents.scala)
Feb 21 2023
Feb 20 2023
Feb 17 2023
Feb 16 2023
pinging @daniel, in case you have ideas on who could help us to understand and possibly fix this behavior, thanks!
Feb 14 2023
Thanks for all the input! I've updated the task description accordingly.
Sadly I had to change the elasticsearch mapping to allow this and thus this will have to wait for a full re-index after the first patch is merged before enabling the new textbytes keyword.
Feb 13 2023
@chinneeb the change is applied to https://mn.wikipedia.org, typing Мөрдорж does seem to suggest Лувсанжамбын Мөрдорж first now. Please let us know if you see anything unusual.
Feb 10 2023
Feb 9 2023
Q115608572 was restored after applying the fix, items that were not edited after being undeleted will sadly remain absent from WDQS until an edit is made or after a full reload.
Feb 8 2023
Feb 7 2023
The reconcile batch job seems the one to blame, it reports:
23/01/12 03:29:25 INFO ReconcileCollector: Collected 0 inconsistencies from event.rdf_streaming_updater_state_inconsistency/datacenter=eqiad/year=2023/month=1/day=12/hour=0
The inconsistencies were properly detected by the updater:
select * from rdf_streaming_updater_state_inconsistency where year=2023 AND month=01 AND day=12 AND meta.domain="www.wikidata.org" AND item="Q115608572" AND datacenter="codfw";
_schema meta item dt original_ingestion_dt revision_id original_event_info action_type parent_revision_id inconsistency state_revision_id state_status is_wmf_domain normalized_host datacenter year month day hour
NULL {"uri":null,"request_id":null,"id":"5eae0041-d477-4231-a31d-fe60d09b1992","dt":"2023-01-12T00:21:02.243793Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:21:02.243557Z 2023-01-12T00:18:54.174Z 1785108184 {"_schema":null,"meta":{"uri":null,"request_id":"ab00575f-7a12-43cb-9f73-a50bf7e98540","id":"748301dd-194a-40c5-8db1-8bf79f67c797","dt":"2023-01-12T00:18:53Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:18:53Z"} revision-create 1785104250 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"fdfdef74-0249-4c56-adeb-aab5834f8e3a","dt":"2023-01-12T00:25:17.876264Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:25:17.875852Z 2023-01-12T00:22:40.864Z 1809645267 {"_schema":null,"meta":{"uri":null,"request_id":"089f327e-e5f5-4bb4-b5db-dd7163a0c3fc","id":"e0779cda-3936-4705-baa6-a7bb9e21122f","dt":"2023-01-12T00:22:40Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:22:40Z"} revision-create 1785111610 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"eb6219c5-9d08-4148-b046-9b829fb7840e","dt":"2023-01-12T00:21:02.235752Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:21:02.235069Z 2023-01-12T00:18:54.174Z 1785104250 {"_schema":null,"meta":{"uri":null,"request_id":"ab00575f-7a12-43cb-9f73-a50bf7e98540","id":"372a8237-b3cc-43a1-bd53-9d81f8ed3a9c","dt":"2023-01-12T00:18:53Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:18:53Z"} revision-create NULL newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"ca728ff4-7bbe-4d25-9862-56494aac1c2c","dt":"2023-01-12T00:35:24.918556Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:35:24.918252Z 2023-01-12T00:33:23.599Z 1809650380 {"_schema":null,"meta":{"uri":null,"request_id":"708615f4-e67c-4cc0-85fe-0a8dfb73f1fc","id":"e546995d-0120-4b1f-a9d4-f82fb9af88e4","dt":"2023-01-12T00:33:23Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:33:23Z"} revision-create 1809649800 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"197fc474-7793-495b-b34b-b14df361fb4c","dt":"2023-01-12T00:38:51.435003Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:38:51.434637Z 2023-01-12T00:36:45.418Z 1809652184 {"_schema":null,"meta":{"uri":null,"request_id":"8b2822eb-2c55-478e-bbb9-dbd70a2ebadb","id":"3f1a8d11-4dc2-49a3-b536-80a532d11d68","dt":"2023-01-12T00:36:44Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:36:44Z"} revision-create 1809650380 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"a1682210-27d0-43aa-b73a-dab169db6d09","dt":"2023-01-12T00:29:48.706526Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:29:48.706271Z 2023-01-12T00:27:47.489Z 1809647736 {"_schema":null,"meta":{"uri":null,"request_id":"9189d7bf-6a8c-4076-a20e-d6f2b19223a1","id":"db3a7565-4f37-45cd-b8c7-995bb123c23c","dt":"2023-01-12T00:27:47Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:27:47Z"} revision-create 1809646313 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"e3f7a0ec-e592-466a-8cb0-4762db119ab5","dt":"2023-01-12T00:34:46.957749Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:34:46.957575Z 2023-01-12T00:32:02.218Z 1809649627 {"_schema":null,"meta":{"uri":null,"request_id":"8ded7e5c-c8e0-47f3-a798-fb963434d36c","id":"d79048d7-ce32-44b8-ae3e-7112ef988b0a","dt":"2023-01-12T00:32:01Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:32:01Z"} revision-create 1809647736 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"194c0316-111c-4510-82c1-29b6002d88d7","dt":"2023-01-12T00:34:46.959180Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:34:46.959053Z 2023-01-12T00:32:19.613Z 1809649800 {"_schema":null,"meta":{"uri":null,"request_id":"a154604e-a1f3-493f-8747-2ef96e508fbf","id":"89284866-1c87-4fed-9bf3-e3c3cf923997","dt":"2023-01-12T00:32:19Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:32:19Z"} revision-create 1809649627 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"a98d0811-9d1c-45b8-b139-0d39afc06d18","dt":"2023-01-12T00:42:09.402992Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:42:09.402405Z 2023-01-12T00:39:52.536Z 1809653947 {"_schema":null,"meta":{"uri":null,"request_id":"d019b288-3a13-4024-a934-3d4db9b63852","id":"e876202c-be0d-4894-87d0-fab7fc1893a8","dt":"2023-01-12T00:39:52Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:39:52Z"} revision-create 1809653618 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"f0e648e0-59b1-4569-b62e-c111e75cc178","dt":"2023-01-12T00:42:09.406318Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:42:09.406068Z 2023-01-12T00:39:56.528Z 1809653984 {"_schema":null,"meta":{"uri":null,"request_id":"313b92c9-500e-4b1d-8feb-913ed2ce7098","id":"6a746531-76a0-48ff-954f-33c118b6cdc9","dt":"2023-01-12T00:39:56Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:39:56Z"} revision-create 1809653947 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"130d0162-65ba-4bd7-be5d-4828ce278e09","dt":"2023-01-12T00:26:47.884125Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:26:47.883671Z 2023-01-12T00:24:46.864Z 1809646313 {"_schema":null,"meta":{"uri":null,"request_id":"883d110b-61ba-43c1-b510-82a62b5c5cd4","id":"2052e478-bc42-44b3-870a-a276975b96ce","dt":"2023-01-12T00:24:46Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:24:46Z"} revision-create 1809645267 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"79900880-73f1-40cb-9691-e59d1cd1ef8d","dt":"2023-01-12T00:21:02.246341Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:21:02.246120Z 2023-01-12T00:18:54.174Z 1785111610 {"_schema":null,"meta":{"uri":null,"request_id":"ab00575f-7a12-43cb-9f73-a50bf7e98540","id":"f2cee9ac-db27-4393-9978-985b14db0c29","dt":"2023-01-12T00:18:53Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:18:53Z"} revision-create 1785108184 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"242e1923-36a9-4bf1-8104-d9d737427082","dt":"2023-01-12T00:41:42.180087Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:41:42.179762Z 2023-01-12T00:39:36.675Z 1809653618 {"_schema":null,"meta":{"uri":null,"request_id":"8426bbf9-8bae-4d03-8ab7-368c387b4fb3","id":"0fe708d9-21b3-436b-b8e8-59cdfa266abc","dt":"2023-01-12T00:39:36Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:39:36Z"} revision-create 1809653202 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
NULL {"uri":null,"request_id":null,"id":"b1bab8e7-2941-4a50-b719-934d91eb6e0d","dt":"2023-01-12T00:40:42.860127Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"} Q115608572 2023-01-12T00:40:42.859739Z 2023-01-12T00:38:38.662Z 1809653202 {"_schema":null,"meta":{"uri":null,"request_id":"e70af5d6-6fde-4de7-9438-c216d0ec4e11","id":"ff9c5ed9-4a98-4a90-b683-98c588539072","dt":"2023-01-12T00:38:38Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:38:38Z"} revision-create 1809652184 newer_revision_seen 1785111610 DELETED true {"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"} codfw 2023 1 12 0
Time taken: 0.253 seconds, Fetched: 14 row(s)So events are properly flowing in the rdf-streaming-updater.state-inconsistency stream.
Feb 6 2023
Feb 3 2023
Currently the two set of predictions are kept and the user can use one or the other: searching for articletopic:biography vs searching for drafttopic:biography. As far I understood the drafttopic keyword was requested for searching by topic on the Draft (T249341) namespace for which the articletopic model does not work. If the outlink model is not enabled on the Draft namespace it might make to leave this problem out for now and not think too much about how to replace the ORES drafttopic model yet.
Feb 2 2023
I think it's important for the flink-app chart to have a way to pass and/or populate a config_file, or have a dedicated feature for passing the script options, I was messing around with the args of the jobspec but I think it might be hard and messy to apply the various values-release.yaml -> values-$dc.yaml -> values.yaml overrides with a plain yaml array... I don't think also that you can define your own template once you're defining a helmfile.d service.
ParameterTool is nice but @pfischer has found a really nice abstraction on top of it, you define your options using a set of ConfigOption, the nice thing is that they're strongly typed and can be used with ParameterTools#getConfiguration().
ParameterTools can be constructed from argv or a properties file. Note it does seem to only support Map<String, String> so I'd avoid using yaml as the default format but rather use the simpler java property files format, they should be compatible tho in a way that you can easily use something like {{- toYaml .Values.app.config_file.options }} template function to generate it.