Root cause Archiva outage from 2023-09-24
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	xcollazo
	Sep 25 2023, 7:24 PM

Description

While working on testing T346281 in production we hit an Archiva outage:

In T346281#9195395, @xcollazo wrote:
Archiva outage:

Because we use Spark 3.2 for the backfill, we created a custom conda environment. To avoid complicating matters even more, I am using Spark's ivy mechanism to pull one dependency at runtime: iceberg-spark-runtime-3.3_2.12-1.2.1.jar. This is done using our Archiva instance as a proxy. This suddenly started failing with:
sudo -u analytics yarn logs -appOwner analytics -applicationId application_1694521537759_58660
...
:::: ERRORS
	SERVER ERROR: Server Error url=https://archiva.wikimedia.org/repository/mirrored/org/apache/iceberg/iceberg-spark-runtime-3.3_2.12/1.2.1/iceberg-spark-runtime-3.3_2.12-1.2.1.pom

	SERVER ERROR: Server Error url=https://archiva.wikimedia.org/repository/mirrored/org/apache/iceberg/iceberg-spark-runtime-3.3_2.12/1.2.1/iceberg-spark-runtime-3.3_2.12-1.2.1.jar
...
And effectively, when I tried to pull the artifact manually I got:
√ ~ % curl https://archiva.wikimedia.org/repository/mirrored/org/apache/iceberg/iceberg-spark-runtime-3.3_2.12/1.2.1/iceberg-spark-runtime-3.3_2.12-1.2.1.pom
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /repository/mirrored/org/apache/iceberg/iceberg-spark-runtime-3.3_2.12/1.2.1/iceberg-spark-runtime-3.3_2.12-1.2.1.pom. Reason:
<pre>    Server Error</pre></p><h3>Caused by:</h3><pre>java.lang.RuntimeException: /var/cache/archiva/temp2871579391051196831: No space left on device
	at org.apache.archiva.proxy.DefaultRepositoryProxyConnectors.createWorkingDirectory(DefaultRepositoryProxyConnectors.java:1078)
...
I've reported this to the SRE team on Slack. As of this writing, a temporary fix has been applied.

The temporary fix was to rm -rf all temporary folders generated by Archiva.

In this task we should:

Figure out root case for this outage
Fix the root case, or implement an alert so the likelihood of a similar outage diminishes.

Related Objects

Mentioned In: T346281: Figure out if an intermediary table while backfilling is beneficial
Mentioned Here: T346281: Figure out if an intermediary table while backfilling is beneficial

Event Timeline

xcollazo created this task.Sep 25 2023, 7:24 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 25 2023, 7:24 PM

xcollazo mentioned this in T346281: Figure out if an intermediary table while backfilling is beneficial.Sep 26 2023, 8:07 PM

LSobanski renamed this task from Root cause Archiva outage from 2024-09-24 to Root cause Archiva outage from 2023-09-24.Sep 29 2023, 4:16 PM

BTullis moved this task from Incoming to Ready for Work on the Data-Platform-SRE board.Oct 11 2023, 8:26 AM

BTullis triaged this task as High priority.Oct 11 2023, 8:48 AM

Gehel moved this task from Ready for Work to Incidents / Follow up on the Data-Platform-SRE board.Dec 6 2023, 1:16 PM

Stevemunene claimed this task.Dec 19 2023, 4:08 PM

Stevemunene lowered the priority of this task from High to Medium.Dec 19 2023, 5:03 PM

Stevemunene moved this task from Incidents / Follow up to 2023.12.01 - 2023.12.31 on the Data-Platform-SRE board.

Stevemunene edited projects, added Data-Platform-SRE (2023.12.01 - 2023.12.31); removed Data-Platform-SRE.

Gehel edited projects, added Data-Platform-SRE (2024.01.01 - 2024.01.21); removed Data-Platform-SRE (2023.12.01 - 2023.12.31).Dec 31 2023, 9:11 PM

Gehel edited projects, added Data-Platform-SRE; removed Data-Platform-SRE (2024.01.01 - 2024.01.21).Jan 23 2024, 1:55 PM

Gehel moved this task from Incoming to Incidents / Follow up on the Data-Platform-SRE board.

I think that we can resolve this task, given that we know we intend to deprecate Archiva. Feel free to reopen if you believe differently.

Root cause Archiva outage from 2023-09-24Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Root cause Archiva outage from 2023-09-24
Closed, ResolvedPublic
Actions