"Edits done" means edits completed.

We may also want to repeat File:Analysis of Wikipedia Portal Traffic and JavaScript Support.pdf but other page(s)?

@mpopov - Yes, that would be super helpful for the other part of parent task T234695!

@kaldari @JKatzWMF moving to backlog right now because we don't have bandwidth and this looks like it will be an extensive task. Please let me know if it's high priority enough to bump currently planned work - perhaps we can touch base in a meeting?

@kzimmerman - Thanks for looking into it.

@DLynch - Is there any chance this could be done by the engineers on the Editing team without the help of Product Analytics (say in January), or would that be difficult?

Server-side PHP sends EditAttemptStep events, right? And those are limited to action='init' & action='saveSuccess'?

@mpopov Yeah, that's my logic. It does also do saveAttempt and saveFailure server-side. Perhaps checking for saveAttempt would be most inclusive, since that'll always fire.

Is there any chance this could be done by the engineers on the Editing team without the help of Product Analytics (say in January), or would that be difficult?

@kaldari I mean, assuming that I'm not missing anything here, I think it's something we are at least capable of doing -- I couldn't commit without running it by @ppelberg for the timeline. There'd be a bit of "we don't normally do this, quickly educating ourselves on the tools" delay involved. Since I've not done this analysis step before I can't really say how much of a delay that'll be.

To summarize, I think the analysis which makes sense is to look at EditAttemptStep in a chosen time period, filter for editor_interface === 'wikitext', then filter with user_id === 0 and user_id !== 0 for anon / loggedin statistics:

• No-JS: sessions containing init and saveSuccess and not ready
• JS: sessions containing init, ready, and saveSuccess (this will include all VisualEditor sessions inherently)

This wouldn't require adding any further instrumentation or waiting for data to be gathered.

Things that might confuse our results:

• This would be using saveSuccess as a way to limit it to sessions that resulted in successful edits. If either having-JS or not-having-JS makes saving substantially harder (lack of tools / bugs), our numbers would be misleading.
• This would exclude VisualEditor users, depressing the overall JS numbers. This would be easy to compensate for by showing a "number of successful edits from VE" figure in the same time period.
• Bots would probably still be included. Depending on the bot's methodology, it could potentially be classed as JS or no-JS, or bypass this editor entirely and use the API to make its edits.

Thank you for the ping, @DLynch.

To answer the question you posed around our ability to take this on, I asked JK [1] how important the information this task is asking for is and how urgently it is needed. JK said we can expect answers about these things this week.

Also, @kaldari, if there is additional context here, please let us know.

1. It's my understanding this task is intended to serve a higher level decision JK/others are needing to make

@ppelberg - JK says it isn't urgent. Let's circle back on this after the holidays.

In T240697#5757335, @kaldari wrote:

@ppelberg - JK says it isn't urgent. Let's circle back on this after the holidays.

Sounds good. Thanks for following up, @kaldari.

This would be using saveSuccess as a way to limit it to sessions that resulted in successful edits. If either having-JS or not-having-JS makes saving substantially harder (lack of tools / bugs), our numbers would be misleading.

I don't actually think this would be misleading, as we want to find out how many actual edits are made with no-JS (i.e. how many edits would we lose by disabling no-JS editing support).

This would exclude VisualEditor users, depressing the overall JS numbers. This would be easy to compensate for by showing a "number of successful edits from VE" figure in the same time period.

Sounds like a good plan.

Bots would probably still be included. Depending on the bot's methodology, it could potentially be classed as JS or no-JS, or bypass this editor entirely and use the API to make its edits.

@DLynch - What we're specifically looking for is no-JS edits made through any editing interface besides the API or mobile apps (regardless of whether they are by bots or not). Is there a way to exclude API edits from the totals? Basically we just need to justify with actual data whether or not we should continue to maintain a no-JS editor (as part of a broader evaluation of all of our no-JS support). I imagine our no-JS editor is used a fair bit, but we need data rather than speculation. The rationale for having an editing API is separate and doesn't need further justification.

To answer my own question, it looks like we could limit it to cases where editor_interface = wikitext and integration = page (to make sure we exclude app edits) for the no-JS number.

@ppelberg - Well, it's after the holidays, but probably an even worse time to bring this up. Regardless, we need this data to move forward with our no-JS guidelines for engineering. From David's analysis, it sounds like this would be a relatively small task (maybe a few days for one engineer). Is there any chance the Editing team could do this in Q4?

@kaldari Yeah, you beat me to it -- API requests shouldn't do any of this logging, and (assuming that the apps are both doing their logging and tagging it correctly) we can filter it down to just on-page stuff with the filter you suggested.

Do you care about restricting it to just desktop browsers, as well? Or breaking out the platforms in reporting? (I assume that disabling JS on mobile browsers is vanishingly rare, but haven't verified this. Although... I think anyone with JS disabled would have to jump through some hoops to get out of our completely JS-dependent MobileFrontend so there's enough barriers already that we'd not be getting helpful numbers, I suspect.)

@DLynch - Interestingly, disabling JS on the mobile web relegates you to using the core/WikiEditor editor, which always reports as "desktop" (T249944), so breaking it out by platform would probably be pointless.

FYI, if T249945 is completed soon this task may become easier to complete.

After doing a bit more sussing out, I think I've come up with specific steps that could be used to answer each of the 3 questions in the task description:

Snapshot % of user edits done with no JS support:
Within a specific timespan, take:
number of sessions where user_id !== 0, integration === page, actions include init, saveSuccess, and not ready.
Divide by:
number of sessions where user_id !== 0, integration === page, actions include init, ready, and saveSuccess.

Snapshot % of anon edits done with no JS support:
Within a specific timespan, take:
number of sessions where user_id === 0, integration === page, actions include init, saveSuccess, and not ready.
Divide by:
number of sessions where user_id === 0, integration === page, actions include init, ready, and saveSuccess.

Snapshot % of all edits done with no JS support:
Within a specific timespan, take:
number of sessions where integration === page, actions include init, saveSuccess, and not ready.
Divide by:
number of sessions where integration === page, actions include init, ready, and saveSuccess.

Note that I've decided not to worry about which editor the user is using (editor_interface), which should simplify things a bit.

Assignment of this depends on T251464; moving to Needs Investigation

In T240697#6094827, @kaldari wrote:

After doing a bit more sussing out, I think I've come up with specific steps that could be used to answer each of the 3 questions in the task description:

Snapshot % of user edits done with no JS support:
Within a specific timespan, take:
number of sessions where user_id !== 0, integration === page, actions include init, saveSuccess, and not ready.
Divide by:
number of sessions where user_id !== 0, integration === page, actions include init, ready, and saveSuccess.

Snapshot % of anon edits done with no JS support:
Within a specific timespan, take:
number of sessions where user_id === 0, integration === page, actions include init, saveSuccess, and not ready.
Divide by:
number of sessions where user_id === 0, integration === page, actions include init, ready, and saveSuccess.

Snapshot % of all edits done with no JS support:
Within a specific timespan, take:
number of sessions where integration === page, actions include init, saveSuccess, and not ready.
Divide by:
number of sessions where integration === page, actions include init, ready, and saveSuccess.

Note that I've decided not to worry about which editor the user is using (editor_interface), which should simplify things a bit.

@kaldari I tried the above method and have posted the detailed numbers on this Github notebook.

TLDR;
Results : For the year 2020 :

• snapshot % of user edits done with no JS support : 26.74 %
• snapshot % of anon edits done with no JS support : 12.79 %
• snapshot % of all edits done with no JS support : 24.08 %

Results : For the year 2019 :

• snapshot % of user edits done with no JS support : 28.92 %
• snapshot % of anon edits done with no JS support : 13.98 %
• snapshot % of all edits done with no JS support : 25.83 %

The listed percentages on non-JS editors are much higher than expected indicating that a large portion of these users have ad-blockers installed and/or enabled DNT. As a result, we don't think this data is useful in determining the percentage of non-JS users and we would recommend looking at adding instrumentation if more accurate numbers are needed. A breakdown by editor interface helps clarify these numbers and confirms that there is only a small percentage for VisualEditor which might be users with client side event blocking enabled.

Thanks for taking a look @Mayakp.wiki! I'll create a new task for creating new instrumentation that isn't affected by ad blocking.

Since this requires new instrumentation, we'll need the request to go through a product team so engineers can implement it (T263505).

Moving this to upcoming quarter for now. Timing will depend on the implementation of instrumentation in (T263505).

Task description updates

• ADDED the target completion date for this analysis: Friday, 28 May 2021
• REMOVED aggregating edits by country from the ===Acceptance criteria, per the decision we made in T263505#7039567

cc @JKatzWMF

Below is a snapshot of edits made by users without JS enabled across all Wiki Projects based on the new instrumentation added in T263505.
Data is also browseable by project and platform is available in this Superset Dashboard.

Percent of edits completed without JS by user type
Logged-In Users:
Desktop: 6.1%
Mobile: 3.15%

Anon Users:
Desktop: 0.98%
Mobile: 1.57%

All Users:
Desktop: 5.28%
Mobile 2.49%

Data Clarifications and Assumptions:

• Data based on events logged in EditAttemptStep and VisualEditorFeatureUse from 13 May 2021 through 29 May 2021
• Events sampled at 1/16
• The new instrumentation only applies to saved page edits made with WikiEditor. In this analysis, we assume all non-wikieditor edits (editor_interface != 'wikitext') or non-page edits (integration ! ='page')(i.e. edits made with VisualEditor, Discussion Tool or app edits) have JS enabled as these editors require JS to work.
• Mobile and desktop classifications are based on the device family data contained in the usergent field (There is an unresolved bug that currently identifies all WikiEditor edits as platform = desktop. See T249944)
• Only saved events (events that reached event.action = saveSuccess) were reviewed.
• Instrumentation available allows us to identify edits saved with or without JS enabled but does not differentiate between different reasons JS was not available at the time of the edit (i.e. users that intentionally disabled JS for privacy reasons or users that were not able to load JS prior to saving) as indicated in T263505#7036537.

Please let me know if you have any questions or there are any additional breakdowns that would be valuable (For example, if helpful, we could also look at the percent of logged-in users that completed an edit without JS to understand if the data coming above is influenced by few highly active editors).

See notebook for the query used to gather data and additional details.

cc @JKatzWMF @ppelberg

Thanks, @MNeisler!

Would it be possible to make the Superset dashboard also browsable by additional filters for edit namespace and country?

@MNeisler, I have a few questions.

1. Does this include flagged bots?
2. How are scripts get counted? At the English Wikipedia, a large percentage of edits are made via tools such as Twinkle and HotCat that never show an editing window to the editor. Can this method tell the difference between this edit and this edit and this edit?
3. How is the 2003 wikitext editor counted? It has no toolbar at all. It's the only editing environment you can get if you have Javascript disabled, but you can also choose it deliberately, even if you have Javascript enabled.

WikiEditor is the 2010 wikitext editor. AIUI the main editing options, and their actual results, are:

I wonder whether combinations such as "Javascript on, 2003 prefs" or "using a script that never displays an editing window" would be incorrectly counted as "Javascript off" in this analysis.

Thank you for this very useful report, @MNeisler! :)

I'm curious of the number of user making edits in each category. It is imaginable to see a few user making a very significant portion of all edits. If some of them are doing these edits without using Javascript, wouldn't it change the interpretation of the percentages?

Thank you for this! I've added an entry at https://www.mediawiki.org/wiki/No-JavaScript_notes#Metrics (as always, please add/update anything else in that page!)

Thank you for the report! To be clear, this number reflects the fraction of users for whom MediaWiki's frontend code does not run (either because their browser doesn't support Javascript, or they disabled it, or their browser supports an older version of Javascript which we don't target to cut down development costs, and we detect that and don't send them Javascript), not necessarily users who are unable/unwilling to run Javascript in general, right?

Thanks all! See responses to questions below. I'll add some text to the dashboard and report to help clarify these points as well.

Would it be possible to make the Superset dashboard also browsable by additional filters for edit namespace and country?

@dr0ptp4kt Yes, we can further break down the results by namespace but we currently do not have country data available. See T263505#7039567
I'll work with @ppelberg to prioritize any additional breakdowns and follow-up analysis requests following review.

@Whatamidoing-WMF

Does this include flagged bots?

No; any flagged bots were excluded from this analysis.

How are scripts get counted? At the English Wikipedia, a large percentage of edits are made via tools such as Twinkle and HotCat that never show an editing window to the editor. Can this method tell the difference > between this edit and this edit and this edit?

Any edit made with a gadget or other JS-required editing methods (such as VisualEditor) (that is logged as saved in EditAttemptStep) are assumed to have Javascript enabled in this analysis.

The report currently only distinguishes between the following two types of edits:
no JS edit: Any page edit completed with the wikitext editor and saved without JS enabled.
JS edit: (1) Any page edit completed with the wikitext editor and saved with JS enabled (identified by instrumentation added in T263505);
(2) any non-wikitext edit (i.e VisualEditor); or (3) any non-page type edit (i.e., discussiontools). Note: Instrumentation was only added to identify (1) wikitext editor edits saved with JS. The analysis assumes that edits that meet conditions (2) and/or (3) have JS enabled.

How is the 2003 wikitext editor counted? It has no toolbar at all. It's the only editing environment you can get if you have Javascript disabled, but you can also choose it deliberately, even if you have Javascript enabled.

I believe the instrumentation added in T263505 should apply to 2003 wikitext editor edits as well and allows us to distinguish between a 2003 wikitext editor page edit saved with or without JS (So if you are using the 2003 wikitext editor and have Javacscript enabled when you complete your edit, then that would be correctly identified as a JS edit in the report). @DLynch - Can you confirm?

I'm curious of the number of user making edits in each category. It is imaginable to see a few user making a very significant portion of all edits. If some of them are doing these edits without using Javascript, wouldn't it change the interpretation of the percentages?

@Trizek-WMF Yes, I agree. It's possible that a few very active editors are contributing to a significant portion of these edits. We cannot determine the number of anon users but have data available to determine the percent of logged-in users that are making no JS edits on each project. Per my comment above, we'll prioritize any additional breakdowns and follow-up analysis requests following review of the report by all the stakeholders.

To be clear, this number reflects the fraction of users for whom MediaWiki's frontend code does not run (either because their browser doesn't support Javascript, or they disabled it, or their browser supports an older version of Javascript which we don't target to cut down development costs, and we detect that and don't send them Javascript), not necessarily users who are unable/unwilling to run Javascript in general, right?

@Tgr Yes, that's correct. The instrumentation added in T263505 only allows us to distinguish between Wikieditor edits saved with JS or without JS at the time of the edit but does not differentiate between the various possible reasons (as you identified) that MediaWiki's frontend code does not run at the time of the save.

@DLynch are you able to answer the question @MNeisler posed in T240697#7133248?

How is the 2003 wikitext editor counted? It has no toolbar at all. It's the only editing environment you can get if you have Javascript disabled, but you can also choose it deliberately, even if you have Javascript enabled.

I believe the instrumentation added in T263505 should apply to 2003 wikitext editor edits as well and allows us to distinguish between a 2003 wikitext editor page edit saved with or without JS (So if you are using the 2003 wikitext editor and have Javacscript enabled when you complete your edit, then that would be correctly identified as a JS edit in the report). @DLynch - Can you confirm?

@MNeisler Sorry, I missed that question!

If you've disabled the 2010 toolbar by disabling JavaScript, the relevant form fields and JS will still be added to the page by the WikiEditor extension. As such, an edit made in the 2003 environment should still be correctly identified (albeit, you know, as nothing happening because there's no JS to run). However, if you've reached it by disabling the toolbar as a user-preference, the JS won't be added, and so the edit won't be flagged as JS-supported.

You could look for users who have the usebetatoolbar feature set to false, to weed out those whose no-JS result can't be strictly relied upon.

In T240697#7147102, @DLynch wrote:

@MNeisler Sorry, I missed that question!

Thank you for elaborating on this, @DLynch.

@MNeisler and I talked about this today and Megan shared that she included the nuance you've documented in T240697#7147102 to the Superset dashboard's Data Clarifications and Assumptions section; see the list item that reads,

• "Instrumentation does differentiate between different possible reasons JS was not available at the time of the save, such as users that:
  • users that disabled the 2010 Wikieditor toolbar by user preference"