Updates
Refactoring 6 services using AQS

1. Geo-Analytics - Few comments
2. Page-Analytics - Few comments
3. Media-Analytics - Few comments
4. Device-Analytics- Ready to merge
5. Edit-Analytics - Working on it
6. Editor-Analytics - Not started

• Step 1: Figure out test are useful for pyspark jobs
• Step 2: Decide on acceptable parameters for test
• Step 3: Create/script tests on local
• Step 4: Move test to prod- gitlab

XML Schema Validation(Dan is already doing this using IntelliJ):

• If your XML files adhere to a predefined XML schema (XSD), you can validate them against the schema to identify structural differences.
• Any non-conformance with the schema will be flagged as a difference.

Size and Visual Comparison of the XML:

• Open the two XML files in a text editor or XML viewer that supports syntax highlighting for easier readability.
• Manually review the size of the files side by side.

Size and Visual Random Spot Comparison of the tables in HDFS:

• Using Diff or Minus to compare the Hive table (Mediawiki wikitext history) and the iceberg table (wikitext_raw_rc1)
• Manually review the size of same partitions(if exist)

Stream Parsing:

• Comparing/parsing both files in streaming XML Processes