In T350009#9379753, @JAllemandou wrote:

As discussed on Slack, I would suggest using dataframe.repartition(X) for the features datasets as data is relatively small and using coalesce impairs job scalability (and the number of files is far too big in comparison to the data size :).

Many thanks for the valuable suggestion! Implemented in this commit with a default value of 4.
Now output files dropped to 1k! 🎉

Also CC @fkaelin .

In T350020#9376060, @jcrespo wrote:

In T350020#9375684, @mfossati wrote:

@jcrespo , would it be possible to use the internal reverse proxy to directly download deleted images via HTTP like here?

No, that won't be possible. Backup access will only be possible through backup access, which is highly restrictive, and it doesn't use swift protocol. I wonder if what you may want is swift production access instead, which won't come with the same limitation as backup access, and that Matthew here, on CC, advised using instead of backups.

Thanks for the heads up about production VS backup. The backup access request was merely based on my understanding that deleted images were stored there. If production access is a better option, then let's definitely opt for it, CC @MatthewVernon.

In any case, we are still blocked on T&C ok

Is there anything we can do from our side to unblock?

@jcrespo , would it be possible to use the internal reverse proxy to directly download deleted images via HTTP like here?

In T350009#9374856, @JAllemandou wrote:

Thanks for the ping @mfossati - 79k files are still quite a lot - would you mind telling me more about the data? (size, partition-scheme etc)?

• The typical size is roughly 60 GB, with 207 folders holding 401 files (coalesce = 400 + _SUCCESS) each.
• no explicit partitioning

Note that the Spark job responsible for this output is the largest and most complex we have among our team's data pipelines, and usually takes 1 day of computation to complete.
The current implementation writes one parquet per wiki, thus resulting in those 207 folders. Modifying this behavior is out of scope, as it would require a lot of work: I had tried a simple solution that writes to a single parquet, causing all sorts of troubles to Spark executors.
Also, further reducing the coalesce value might increase the execution time, which isn't viable neither.

Report

Report

In T348220#9362928, @mfossati wrote:

We're explicitly asking the user to enter a website, although I acknowledge a full URL might not occur often.

With T347750#9332285 we'll actually ask for a link, thus increasing the chance of a URL.

Directly moving to verify on production: we agreed with @Etonkovidova that the latest patch is tiny enough.

In T348220#9360249, @MarkTraceur wrote:

@mfossati this sounds like a lot of work (parsing/regexing through the text, validating the URL(s), displaying an error message) for something that has very little possibility of actually happening, and very little negative impact if it does? Are we seeing issues with this somewhere?

Perhaps I made the task description too broad: what I essentially meant is just a regular expression that matches a website and a warning if none is found.
We're explicitly asking the user to enter a website, although I acknowledge a full URL might not occur often.

Skipping estimation: this ticket can be tackled together with T347558: [S] Coalesce section alignment image suggestions output.

Weekly update

Back to work, continued where I left off.
Sent 3 incremental patches that are now ready for review.

Hey @Ladsgroup , we're using that table in the image suggestions production data pipeline, see this query.
While I'll try to monitor this ticket and mailing lists announcements, it would be great if you could ping me in advance before the breaking change rolls out, so that we can take action.
You could also leave a comment in T350007: [M] Adapt image suggestions to comply with breaking database schema changes if you prefer, thanks!

Closing as invalid: we plan to use actual data to train a machine learning model, as part of T349641: [Investigation EPIC] Machine detection for media with copyright issues in Upload Wizard on Commons.

CC @JAllemandou .

Weekly update

Made good progress, then spent some time debugging why code that handles warning messages for sub-radio buttons was never reached.
Note that acceptance criteria came on Wed 25 and augmented the previously estimated effort in my opinion.

Weekly update

This week's SEAL run went fine. As a result, I checked SLIS output, which looks fine as well. Image suggestions completed, too. Closing!

In T347756#9283719, @Cparle wrote:

When we add more questions we will need to add number to the font of this question. How can we make sure that gets done later?

I thought that we could make this an ordered list in the next iteration? Does that make sense (cc @mfossati @matthiasmullie)
edit: I could make it an ordered list now if you like, but then it'd be numbered even though there's only one item

I was thinking the same 😄

See also https://foundation.wikimedia.org/wiki/Legal:Data_publication_guidelines

Review done. Moving to blocked, pending community feedback integration.

Weekly update

The patch was reviewed by @matthiasmullie, who directly integrated feedback (thanks!).
Pending some work on tests that are currently failing

Weekly update

The SEAL pipeline is currently running in production, waiting for the last DAG task to complete. A couple of quick hotfixes were needed to ensure a proper execution.

Here are my suggestions:

1. match against deletion request opening reasons, closing reasons, and file deletion reasons (AKA edit messages or revision comments), See data lake query to extract file deletion reasons
2. expand all wikilink shorthands, such as COM:DW = COM:DERIV = Commons:Derivative_works. Note that COM prefixes always expand to Commons
3. compile the list by looking at wikilinks and words frequencies, plus word clusters from T340546: [XL] Analysis of deletion requests on Commons. Most frequent are:
   • COM:FOP = COM:PANO = Commons:Freedom_of_panorama
   • COM:SS = COM:SCREENSHOT[S] = Commons:Screenshots
   • COM:ALBUM = Commons:ALBUM

I can confirm Airflow variables are not updated after deployment.
Opened T348963: DagProperties don't automatically update Airflow variables, CC @xcollazo .

Thanks again for your fast reaction @xcollazo , much appreciated!
I can confirm the SEAL DAG is now deployed.

Hey @VirginiaPoundstone , I don't think there's any deadline from our side. However, please note that this was initially raised as one of the causes that put the Hadoop cluster under pressure, CC @JAllemandou .

In T325316#9250803, @xcollazo wrote:

(one is 1.22 GB, the other 1.64)

!!

Yeah I know 😄 , the heavier one is a pre-trained machine learning model and the other one has a lot of machine learning dependencies, despite my slimming efforts.

Maybe we can add them manually to unblock you. Let me try.

Thanks a lot and no worries, not in a hurry.
Would it be viable to bump the memory for future deployments?

Moving away from code review, pending deployment & production monitoring.

@xcollazo , could you please give us a hand? Java is OOMing, perhaps our artifacts are too heavy (one is 1.22 GB, the other 1.64)? Pasting what's happening below:

mfossati@deploy2002:/srv/deployment/airflow-dags/platform_eng$ scap deploy
17:59:22 Started deploy [airflow-dags/platform_eng@520fa55]
17:59:22 Deploying Rev: HEAD = 520fa55c45d057c39c977e7cbe652844e9f66c00
17:59:22 Started deploy [airflow-dags/platform_eng@520fa55]: (no justification provided)
17:59:22
== DEFAULT ==
:* an-airflow1004.eqiad.wmnet
17:59:23 airflow-dags/platform_eng: fetch stage(s): 100% (in-flight: 0; ok: 1; fail: 0; left: 0) |
17:59:24 airflow-dags/platform_eng: config_deploy stage(s): 100% (in-flight: 0; ok: 1; fail: 0; left: 0) |
17:59:49 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'airflow-dags/platform_eng', '-g', 'default', 'promote', '--refresh-config'] (ran as analytics-platform-eng@an-airflow1004.eqiad.wmnet) returned [1]: Could not chdir to home directory /nonexistent: No such file or directory
Registering scripts in directory '/srv/deployment/airflow-dags/platform_eng-cache/revs/520fa55c45d057c39c977e7cbe652844e9f66c00/scap/scripts'
Registering scripts in directory '/srv/deployment/airflow-dags/platform_eng-cache/revs/520fa55c45d057c39c977e7cbe652844e9f66c00/scap/scripts'
Executing check 'artifacts_sync'
Check 'artifacts_sync' failed: hdfsWrite: NewByteArray error:
java.lang.OutOfMemoryError: Java heap space
Traceback (most recent call last):
  File "/usr/lib/airflow/bin/artifact-cache", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/airflow/lib/python3.10/site-packages/workflow_utils/artifact/cli.py", line 32, in main
    artifact.cache_put(force=args['--force'])
  File "/usr/lib/airflow/lib/python3.10/site-packages/workflow_utils/artifact/artifact.py", line 71, in cache_put
    cache.put(self, open_file)
  File "/usr/lib/airflow/lib/python3.10/site-packages/workflow_utils/artifact/cache.py", line 128, in put
    output.write(input_stream.read())
  File "pyarrow/io.pxi", line 359, in pyarrow.lib.NativeFile.write
  File "pyarrow/error.pxi", line 113, in pyarrow.lib.check_status
OSError: [Errno 12] HDFS Write failed. Detail: [errno 12] Cannot allocate memory

All data pipelines succeeded.
The required dataset is now available, row counts of ALIS, SLIS and the delta are all consistent with the previous run.

First complete version & DAG ready for code review.

The upstream dependency is back, so I manually triggered all data pipelines.
Section topics and section alignment suggestions went fine, image suggestions running now.

All merged, closing.

The wmf.wikidata_item_page_link/snapshot=2023-09-18 upstream dependency isn’t available yet.
Corresponding DAG sensors timed out, causing all DAGs to fail.

Note that dumps will be owned by Data Engineering, Data Products team.
Work on Dumps 2.0 is also underway.

Hi there, for unit tests I can certainly suggest chispa, which was initially proposed by @MunizaA .
For integration tests I can't tell.

CC @JAllemandou .

@Urbanecm_WMF , T343844: NEW BUG REPORT fiwiki’s section-level image suggestions aren’t generated in production looks fine now in terms of data.
Here's the section-level image suggestions (SLIS) current count for Vietnamese Wikipedia:

In [1]: isu = spark.read.table('analytics_platform_eng.image_suggestions_suggestions').where('snapshot="2023-09-11"')
In [2]: slis = isu.where(isu.section_index.isNotNull())
In [3]: slis.where(slis.wiki == 'viwiki').count()
Out[3]: 38210

Re the acceptance criterion of this ticket: we have at least 1 SLIS for 293 Wikipedias, not all. Here's the complete list with SLIS raw counts (sorry for the long paste!). You may want to skip Wikipedias with not enough SLIS.

In [4]: slis.select(slis.wiki).distinct().count()
Out[4]: 293
In [5]: slis.groupBy(slis.wiki).count().orderBy('count', ascending=False).show(n=300)

@xcollazo @mforns , actually it's essential to understand the expected behavior of DagProperties: if they don't automatically update Airflow's variables, new deployments won't be sustainable.
Re-opening the ticket: from our side, we can check what happens in the next deployment. but it would be great to get your advice, too.

• Section topics:

In [1]: st = spark.read.parquet('/user/analytics-platform-eng/structured-data/section_topics/2023-09-11')
In [2]: st.where(st.wiki_db == 'fiwiki').count()
Out[2]: 2367759

• Section alignment suggestions:

In [3]: ali = spark.read.parquet('/user/analytics-platform-eng/structured-data/section-alignment-suggestions/suggestions/2023-09-11')
In [4]: ali.where(ali.target_wiki_db == 'fiwiki').count()
Out [4]: 28518

• Section-level image suggestions:

In [5]: isu = spark.read.table('analytics_platform_eng.image_suggestions_suggestions').where('snapshot="2023-09-11"')
In [6]: slis = isu.where(isu.section_index.isNotNull())
In [7]: slis.where(slis.wiki == 'fiwiki').count()
Out[7]: 55348

Awesome! Closing.