• Step 1: Figure out test are useful for pyspark jobs
• Step 2: Decide on acceptable parameters for test
• Step 3: Create/script tests on local
• Step 4: Move test to prod- gitlab

XML Schema Validation(Dan is already doing this using IntelliJ):

• If your XML files adhere to a predefined XML schema (XSD), you can validate them against the schema to identify structural differences.
• Any non-conformance with the schema will be flagged as a difference.

Size and Visual Comparison of the XML:

• Open the two XML files in a text editor or XML viewer that supports syntax highlighting for easier readability.
• Manually review the size of the files side by side.

Size and Visual Random Spot Comparison of the tables in HDFS:

• Using Diff or Minus to compare the Hive table (Mediawiki wikitext history) and the iceberg table (wikitext_raw_rc1)
• Manually review the size of same partitions(if exist)

Stream Parsing:

• Comparing/parsing both files in streaming XML Processes

In T343325#9109206, @xcollazo wrote:

In general, I think the document includes great knowledge to have as a newcomer to Dumps 1.0.

From a runbook perspective, I think we should include a listing of common issues and resolutions. As it stands today, there seems to only be one such listings in the document. Granted, as we are all mostly Dumps newbies here, we would need to actually hit those issues in order to solve them and properly document them. So I guess we need time and experience to add more?

7am UTC on Thursday (2023-07-27) works for me.

@ArielGlenn i used my wikimedia email

The data engineering team had a meeting and the conclusion was capture tags based on
*Frequency,
*Ownership,
*Criticality,
*Requires a certain table e ie Webrequest
*Destination of data source ie Iceberg, Hive

• Remove tags that do not meet this Criteria

Documented the following datasets and added wikitech links where applicable

• uniques devices
• banner-activity (druid)
• mobile_apps_session_metrics_by_os

I have documented the following datasets and I am awaiting feedback.

• mediawiki_api_request
• mobile apps session metrics
• mobile apps uniques

In T327255#8542025, @Eevans wrote:

In T327255#8539372, @JEbe-WMF wrote:

[ ... ]

I am not exactly certain. Because I am new, I am not sure what I need and don't but it is stated on my onboarding task to be added the OPS group.

Do feel free to get my manager's approval. cc @odimitrijevic

I went ahead and added you to the wmf group for now. (along with the WMF-NDA Phabricator group). If you do need ops membership, then feel free to re-open this issue (or better yet, @odimitrijevic can do so with the approval).

Welcome aboard!

I would also be requiring access to a Kerberos principal. cc @odimitrijevic @Snwachukwu

In T327255#8539366, @MoritzMuehlenhoff wrote:

In T327255#8538985, @Eevans wrote:

Hi Jennifer,

The specific LDAP group that you want to be added to (optional): WMF and OPS

In order to add you to group ops, I would need your managers approval; Are you sure you need ops? Gauging from the other Data Engineers on your team, I think group wmf might suffice.

Ticket grant Jennifer Ebe LDAP access for Onboarding.