Description
Google Search Console (GSC) data has historically been studied by analysts through Google Cloud Platform BigQuery. This has two ongoing costs:
- BigQuery query cost ad-hoc and scheduled analytical queries run against BQ are expensive.
- GCP storage cost retaining tens of TB of GSC data inside the GCP project incurs a recurring storage bill.
Following T427457, the GSC dataset is now downloaded to HDFS and exposed as a Hive tables in the Data Lake. It is refreshed daily with a small ingestion lag, and can be queried with Spark at no incremental egress or per-query cost.
Now that the data is available on our infrastructure, we want to shift all analytical usage there and lower the GCP bill.
Goals
- Migrate analyst usage: move existing GSC analyses and any scheduled/derived queries from BQ to the Data Lake (Spark / Hive), and confirm result parity.
- Revoke GCP access progressively: decommission the analyst-facing accesses to the GCP project once usage has moved, keeping only what the download pipeline (T427457) needs.
- Remove GCP-resident data: delete the bulk of the GSC data held in GCP once it is safely available and validated in HDFS. This can and should be automated (with the same Airflow dag ?).
Related: T427457