Page MenuHomePhabricator

Updates to the developer portal site to consolidate policy links and resources for high-volume users
Closed, ResolvedPublic

Description

Finalized Bot Traffic Design Mockups for High-Volume Reusers-

The infrastructure use group is working towards a final draft of new high volume use designs on the developer portal. Draft changes are attached to this ticket. We are requesting help to redraft both the developer portal link on the main page, and the new subdomain for bot-traffic. This will help our group support high-volume reusers and address potential abuse cases. The goal is to have these mockups reviewed by the working group and implemented by March 13, 2025, with the partnerships and SRE teams using this work for reference starting in late March.

Dates:
(1) Review with the requestor target: 2025-03-13
(2) Review with the working group: 2025-03-20 (UPDATED AS OF 3/10/25)
(3) Deploy target (assuming 1,2 have limited comments/revisions) : 2025-03-27 (UPDATED AS OF 3/10/25)

Request details:
(1) Update Developer.wikimedia.org landing page to include a High Volume and Commercial Users option with a description in-line with the style of the rest of the page (open to redraft), linking to:
Developer.wikimedia.org/bot-traffic

(2) Add description at the top of Developer.wikimedia.org/bot-traffic (open to redraft):
“These are the policies that classify a user as high volume, and the resources/options available to those users. Commercial users have historically needed to review policies to ensure they are not harming our infrastructure or causing abuse that leads to access limitations. We highly recommend proactively ensuring your use-case adheres to these policies to avoid any restrictions. If you are not able to adhere to these policies, you can reach out to our tech partnerships and enterprise teams for available options, or click Get Help to start a conversation.”

(3) Provide options on Developer.wikimedia.org/bot-traffic:

Get Help
bot-traffic@wikimedia.org email or form that leads to that email
Terms of Use
https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use
API Policy
https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_API_Usage_Guidelines
Robots Policy
https://wikitech.wikimedia.org/wiki/Robot_policy
Wikimedia Enterprise
https://enterprise.wikimedia.com/
Technical Partnerships
partnerships@wikimedia.org email or form that leads to that email

Reach out to cpetrillo@wikimedia.org with any questions.

Main domain changes.png (854×1 px, 238 KB)

bot-traffic subdomain.png (942×800 px, 115 KB)

Event Timeline

Cpetrillo created this object with visibility "All Users".
Cpetrillo updated the task description. (Show Details)
apaskulin moved this task from Inbox to Content & design on the Wikimedia-Developer-Portal board.

Change #1125555 had a related patch set uploaded (by Alex Paskulin; author: Alex Paskulin):

[wikimedia/developer-portal@main] content: Add resources for high-volume access

https://gerrit.wikimedia.org/r/1125555

[Updated March 13, 2025 following Tech Docs Team feedback]

I've opened a patch with suggested copy and layout that matches the rest of the site. Feedback welcome! Here are a few notes about our recommendations, including two requests. Our intention is to ensure the content is correct and consistent in order to create the best user experience and ease of maintenance going forward.

Organization:

After discussing with the team, we recommend including the bot traffic page within the
"Use content and data" section of the site. Although I appreciate the legal distinction between reusing project content and accessing Wikimedia infrastructure, I don't think that users of the site will be making this distinction. High-volume and commercial users are accessing Wikimedia infrastructure because they want to use project content for something. As such, they are more likely to discover the bot traffic page as part of the use case they identify with (using content).

Looking at the pageview data for the site, the "Use content and data" section is the second most popular section after "Get started". The pageviews for each section decrease as the topics become more specific and as they appear farther down the homepage. Having a highly specific section like "High-volume and commercial access" at the bottom of the homepage will likely be less discoverable and receive less pageviews than nesting it within "Use content and data". The tradeoff is that the URL for the page will be https://developer.wikimedia.org/use-content/bot-traffic/ instead of https://developer.wikimedia.org/bot-traffic/

Another option that is worth considering is using a page on the Foundation Governance Wiki for this content instead of using the Developer Portal. There are two main benefits to this:

  • Foundation Wiki contains official Wikimedia Foundation policies, and as such, it would lend a greater authority to the bot traffic page than it would have on the Developer Portal.
  • Foundation Wiki is also quicker and easier to update than the Developer Portal. Foundation Wiki can be updated with a simple wiki edit, while the Developer Portal needs a task opened, a patch submitted, and a weekly deployment made in order for updates to take effect.

Content:

  • [Request] The Robot policy page currently has a notice stating that the page may be outdated or incorrect. The Dev Portal only links to pages without this type of notice because it can undermine readers' trust. Because of this, I've omitted this link from my patch, but I'm happy to add it whenever the notice is removed, either for this initial deploy or later on.
  • Regarding the two email contacts, unfortunately the Dev Portal doesn't have the capability for forms. We could use direct mailto: links, but this is likely to create spam for these emails. Instead, we recommend creating intermediary wiki pages for each of these contacts that list the email address. This also helps provide more context about who the user is contacting and makes it easier to update these emails if they change.
    • For partnerships@, I found a page on Meta, which I've gone ahead and included in my patch. I've also updated the link text to refer to the "Partnerships team" instead of the "Technical Partnerships team" since that is what is documented on the wiki page. I'm happy to change this once a page for the Technical Partnerships team exists.
    • [Request] I wasn't able to find a page for bot traffic, so I've omitted it from the patch. I'm happy to add it once a wiki page exists for it.
  • I used the standard format for the content page, with descriptions and link text instead of images.
  • I rephrased the page intro to provide more context while being as concise as possible and emphasizing the ask ("We highly recommend proactively ensuring...").

Screenshots:

Screenshot 2025-03-13 at 2.26.54 PM.png (947×1 px, 131 KB)
Screenshot 2025-03-19 at 1.27.43 PM.png (1×1 px, 188 KB)

@Cpetrillo Hi, why is this task access-restricted to all logged-in users instead of being public?

Cpetrillo changed the visibility from "All Users" to "Public (No Login Required)".Mar 10 2025, 3:39 AM

@Aklapper didnt see that I made it users-only. Should be public now

Updating the deploy target to 3-27-25 due to working group feedback and timeline for approvals from folks with conflicts this week.

(Please use a readable date format to avoid misunderstandings - thanks! :)

After some discussion with @Cpetrillo, we've agreed that:

  • Adding a link to the Robot policy page on Wikitech will be managed as a separate request once the outdated banner has been removed.
  • Links to the two emails will use intermediary stub pages on Wikitech (Bot traffic and Technical partnerships; edits welcome!) until more complete pages are created by those groups later this calendar year. This helps with spam prevention, maintainability, and consistency.

I've also shortened some of the text per review feedback. See the diff between patchsets 4 and 4.

Latest screenshots:

Screenshot 2025-03-24 at 11.57.02 AM.png (932×1 px, 127 KB)
Screenshot 2025-03-24 at 11.56.50 AM.png (1×1 px, 175 KB)

Change #1125555 merged by jenkins-bot:

[wikimedia/developer-portal@main] content: Add resources for high-volume access

https://gerrit.wikimedia.org/r/1125555

Change #1131785 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/deployment-charts@master] developer-portal: Bump container to 2025-03-26-234702-production

https://gerrit.wikimedia.org/r/1131785

Change #1131785 merged by jenkins-bot:

[operations/deployment-charts@master] developer-portal: Bump container to 2025-03-26-234702-production

https://gerrit.wikimedia.org/r/1131785

apaskulin added a subscriber: bd808.

https://developer.wikimedia.org/use-content/bot-traffic/ is now live! Feel free to open any follow-up requests with the Wikimedia-Developer-Portal tag.

Thanks everyone for your help and feedback! And thanks to @bd808 for the deploy!