git: Commit history, co-author trailers
A core part of technical contribution is writing code. This will be sourced from the git repos themselves, with a wide net cast on GitHub that can always be expanded.
This is separate to activity on these projects (PRs, MRs, Code Review)
Sub sources:
Gerrit: All repos, unless specifically ignored (some should be, such as forked upstream software)
GitHub: A wide net will be cast, traversing well known organisations and projects
GitLab; All repos

Mediawiki:
Documentation of technical spaces occurs via wikitech and mediawiki.org. We commonly see 3 namespaces used as a technical space on other sites; MediaWiki, Template, and Module. Data will be sourced from a mixture of dumps, replicated databases, and APIs.
Wiki edits on mediawiki.org and wikitech.wikimedia.org.
Other namespace edits on all other Wikimedia sites 8, 9, 10, 11, 828, 829
MediaWiki & Talk
Template & Talk
Module & Talk

Talk page activity would be sourced, as this would not be much more effort, however we may later decide to discard the activity.

irc: Chat presence.
Will be sourced from publicly available IRC logs from bots (wm-bot and stashbot).
This will not be a complete representation of communications.
Where possible this would include bridged Telegram conversations.

sal : System administration logs.
These are indexed in a tool https://sal.toolforge.org/ where we will dump and process the data from

mailinglist: Mailing list threads.
Scraped from the mailing list archives.
The APIs don’t work, so this likely has to come from HTML

Phabricator:
API scrape “transactions” on Phabricator which can be sequentially done and will give us all level of activity across tasks

Gerrit:
Code review, comments and review interactions
Scrape each project separately via the API, then use each project's changes "after:" a certain time to continually top up. From changes we scrape "messages," which gives us all activity.

Gitlab:
This is the most complex because each project must be scraped individually, and each activity type within the project must also be scraped separately. (MRs, Issues, Notes)

Github:
It is set up almost identically to GitLab; however, stricter API limits for data retrieval mean an initial fetch and subsequent synchronization might prove slow.