@kostajh a few of us were also thinking about a session focused on some of the techical aspects of running a LLM on Toolforge/Cloud VPS infrastructure / (hopefully) demoing some LLMs that we'd set up in advance. I haven't submitted the session yet but thoughts on whether to combine efforts here or make separate?

Relevant notes from an earlier community call about this topic: https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2023-2024/Draft/External_Trends/Community_call_notes

Is this open, either on a full-participant, or observer-only basis? There is intense interest in this topic at Wikipedia.

In T333127#8736311, @Isaac wrote:

@kostajh a few of us were also thinking about a session focused on some of the techical aspects of running a LLM on Toolforge/Cloud VPS infrastructure / (hopefully) demoing some LLMs that we'd set up in advance. I haven't submitted the session yet but thoughts on whether to combine efforts here or make separate?

@Isaac I'm sorry I missed your comment! It probably makes sense for you to have your own session, which I see you posted in T333853: [Session] Self-hosting ML models on Cloud Services. T333974: [Session] WikiGPT - Natural Language search results based on Wikipedia knowledge and ChatGPT will also be relevant to folks interested in the session mentioned in this task.

To people subscribed here: are any of you interested to co-facilitate or otherwise help with planning this session? I am envisioning this as an open, but guided, discussion; coming up with a list of prompts (sorry) in advance would help focus the conversation.

@kostajh see below:

@Isaac I'm sorry I missed your comment! It probably makes sense for you to have your own session, which I see you posted in T333853: [Session] Self-hosting ML models on Cloud Services. T333974: [Session] WikiGPT - Natural Language search results based on Wikipedia knowledge and ChatGPT will also be relevant to folks interested in the session mentioned in this task.

Not a problem -- I didn't want to submit too many sessions around this topic but I think these three are nicely distinct (general discussion; demo of possibilities of this tech; demo of what we can currently do on our Cloud Services machines) so not too much overlap in the end.

To people subscribed here: are any of you interested to co-facilitate or otherwise help with planning this session? I am envisioning this as an open, but guided, discussion; coming up with a list of prompts (sorry) in advance would help focus the conversation.

Given that I'm already running a different session, I don't want to steer this one too much, but I love your framing in the task of connecting folks who are a bit more hesitant about this with folks who are a bit more excited. Could be fun and very useful to see what sorts of potential guardrails folks come up with to get the benefits of this new generation of AI while addressing some of the major concerns. Maybe aiming towards the start of a more technical complement to some of the draft policies such as enwiki:Large language models.

Below you may find the link attached that redirects the user to the corresponding Etherpad: https://etherpad.wikimedia.org/p/wmh2023-LLMs%2C_ChatGPT%2C_machine_learning_tools%2C_etc

In T333127#8808800, @kostajh wrote:

To people subscribed here: are any of you interested to co-facilitate or otherwise help with planning this session? I am envisioning this as an open, but guided, discussion; coming up with a list of prompts (sorry) in advance would help focus the conversation.

@kostajh I am happy to co-facilitate - I signed up as session coordinator. Also happy to help planning.

Session Notes:

LLMs, ChatGPT, machine learning tools, etc.

Date & time: Saturday, May 20th at 10:30 am EEST / 7:30 am UTC

Relevant links

• Phabricator task: https://phabricator.wikimedia.org/T333127
• Background reading:
  • ChatGPT: https://en.wikipedia.org/wiki/ChatGPT
  • LLMs: https://en.wikipedia.org/wiki/Large_language_model
• Notes from the community call on AI: https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2023-2024/Draft/External_Trends/Community_call_notes

Participants (~20)

Notes

[Most of activity / discussions / notes happening in-person on notepads]
Breaking into two grous: opportunities and threats

Opportunities:

opportunities for developers, editors, users
themes that came up repeatedly: fact-checking, finding related content to expand content
variety of projects: Wikipedia, Wikidata, Commons

Threats:

three "buckets" of points:
    1. the content itself: copyright infringement, disinformation spamming
    2. what happens to the community?  where does the authority of the tools start and stop?
    3. people no longer coming to our content directly but consuming it elsewhere

        in order to use the tools "safely" we might have to put so many checks in that they become unusable

Kosta:

"opportunities" are all about things we could build; for "threats" there were fewer actionable ideas about how to deal with the issues

Comment about use case for detecting conflicting/false information in articles
Using a tool to check/verify citations
Summarizing text: good idea or bad idea? Summarizing an article might be questionable but summarizing a discussion (e.g., the Talk page) could make it easier to follow and encourage engagement

When prototyping, not thinking about "what is flashiest thing" but "how do we not disrupt existing process"

Laws behind training the models. What's the standpoint of movement and WMF. Is it allowed to train proprietary models on Wikipedia data. Lots of TODOs on political topics.

WP is half articles, then talk pages, community processes, mechanisms, etc. A lot of caveats about using on content text go away when you look at using it on talk page discussions. "Constructive criticism bot" idea. Worries about accuracy/missing points are less urgent in non-content spaces.

• But assumption is that summarizing would not include biases, or innacurately reflect sentiment.

What about for software development / tool development. ChatGPT helps with regex, writing SPARQL. Many opportunities are there.

You have to know what to ask to get a good response. Create a framework for how to ask LLM about e.g. SPARQL queries.

Documentation generation, use it for things that are typically left behind.

• We tried that for Parsoid. It was generous, it generated some correct info, and some incorrect info.

Article hoax problem: creating plausible but factually incorrect.

Is partially accurate documentation better than zero documentation?

We need ability to annotate bot generated content, it would make identification and action on this content much easier.

Project ideas:

• transcribe all podcasts from last year. Runs locally, quite promising. Works well with Dutch. That could apply to Wikimedia system.
• Session tomorrow by Slavina and Isaac, about self-hosted ML on cloud services. Good place to get started.
• C. Scott's idea (didn't catch this)
• Hackathon project for gadget to use chat GPT to summarize article sessions

Opportunities & Threats raised during the brainstorming session

Opportunities:

Generate alt text for images
Turning fuzzy date info into quickstatements for Wikidata (text => structured data)
Summarizing articles and talk pages
Rating and improving readability of content (summarizing/adapting language for different audiences, e.g. children)
Typo detection/correction
Finding conflicting information (internal)
Fact-checking (external)
Summarizing sources to support editors
Summarizing related articles (= related to the current query) or compiling/synthesizing content
Comparing content across languages
Detecting NPOV issues or harrassment etc. and suggesting changes
ChatGPT for help/policy pages to support newcomers
Generating Q&A for content to help improve/customize content
Reformatting content into specific form, e.g. taxons, etc.
Generate SQARQL queries or replica queries from natural language
Image editing to remove copyrighted content
Lexeme connections across languages
Template extensions or adapting code for API changes
Vandalism detection
Improving & detecting documentation gaps
Text to speech
Audio to text

Threats:

Editors creating hoax content:
    WP style, fake citations (citations are hard to verify)
Accidental plagarization:
    licensed content appearing
    copyright law not settled
Bias:
    word choice
    languages that have access
    WP has a lot of smaller language content
Falling readership:
    fewer clicks, readers, editors, donations
Increased barriers to human editors
Difficult to distinguish between LLM and human writing:
    Loss of human curation
    WP is not just a list of facts
Community disappearing
Talk page wars:
    Temp accounts => walls of text
Reputational harm with fake content:
    how much trust do we place?
Transparency in LLM interactions with users:
    Explain how judgment was reached
    Bots with unclear motives/accountability
Authority given to AI tools is unclear
People would assume "WikiGPT" meets same standards as Wikimedia projects
Cycle of fake facts propogating, massively accelerating
Prompt injection attacks
We can't keep up with reader expectations for accessing info

@kostajh: Thanks for participating in the Hackathon! We hope you had a great time.

• If this session / event took place: Please change the task status to resolved via the Add Action... → Change Status dropdown.
  • If there are session notes (e.g. on Etherpad or a wiki page), or if the session was recorded, please make sure these resources are linked from this task.
  • If there are specific follow-up tasks from this session / event: Please create dedicated tasks and add another active project tag to those tasks, so others can find those tasks (as likely nobody in the future will look at the Hackathon workboard when trying to find something they are interested in).
• In this session / event did not take place: Please set the task status to declined.

Thank you,
Phabricator housekeeping service