Hypothesis:

If we revisit and revise the sitemap endpoints, we can ensure that they align with our strategic goals and emerging API policies for protecting internal infrastructure and improving API observability.

This hypothesis supports the 2025-26 fiscal year annual plan for product and technology department to deliver on the following objectives and key results:
FY2526 WE5 - Responsible Use of Infrastructure: Developers and reusers access knowledge content in curated pathways, ensuring the sustainability of our infrastructure and responsible content reuse.

Key Result 5.2 - API Evolution: By the end of Q4, 70% of Wikimedia web API endpoints will be supported by common infrastructure.

Why do this?

In August 2025, WMF released[[ https://phabricator.wikimedia.org/T396684 | two new REST API endpoints ]] that enable users to request the sitemap on demand. While originally intended to aid in search engine indexing, a sitemap creates a new entry point and workflow for content reuse in general. While the current implementation solves a specific problem for enabling better search engine indexing, it is slightly at odds with the content reuse strategy that we are actively trying to build through WE5. Specifically, we would like to ensure that as we create new pathways for content consumption, they are aligned with and serving the broader organizational goals of knowing our users and ensuring fair use of infrastructure across all types of content consumers.

Milestones

• [MWP] Introduce access controls on sitemap endpoints
• [Tim S] Limit returned namespaces in default response
• [MWI] Clean up OpenAPI spec generation
• [MWI + Tim S] Create user guide for how we expect users to utilize sitemaps
• [MWI + Tim S] Formally announce endpoint availability
• [SRE?] [Stretch] Actively block more egregious forms of scraping

Detailed PRD available for Foundation Staff.