Adding @JFernandez-WMF for vis - if auto-generation of config settings forms is possible, how might this impact design?

The code to do this is a bit messy, and in general it seems like a lot of manual effort to do form generation for every bit of configuration.

It seems that the most verbose parts of the current Growth config code is declaring form fields and processing the submitted data. Is that what is meant here by messy? cc @Tgr

It would be even nicer if we wouldn't have to maintain a lot of form generation code. There are various third-party libraries for generating forms e.g. from JSONSchema...

I agree the current code is hard to maintain, however I'm skeptical about a third-party solution here since MW has an established generation system around HTMLForm. It would be great if we could plug in an improved descriptor system consumed by HTMLForm but all libraries I have found (eg: formr, laravel-form-builder, php-form-builder) are designed to produce a final HTML form, often bound to a specific styling framework (eg bootstrap). It seems all these overlap with what currently HTMLForm does. @Tgr, have you come across any lib that would be a better fit here? Thank you!

similar to how it's done for e.g. Special:ApiSandbox.

I think it would be great to have some degree of client validation like ApiSandbox for user experience but in my opinion just having it on the server is sufficient for an MVP.

In T332849#9387567, @RHo wrote:

Adding @JFernandez-WMF for vis - if auto-generation of config settings forms is possible, how might this impact design?

We're still exploring the alternatives for this but the safest path would be to reuse what's already built-in in MW with HTMLForm. In terms of design that means CC2.0 forms can have the same look and feel than Growth config (which uses the OOUI display format). Codex is also available as a display format but very few "field types" have been implemented so the form falls back to the OOUI version of the field generating a mixed form of OOUI and Codex fields. We can allocate some time to work on implementing missing Codex field components if we want to ship CC2.0 with it.

This is how Special:EditGrowthConfig looks when using Codex display:

What do you think @RHo @JFernandez-WMF? Should we invest in improving the Codex display?

Hi @Sgs - from Product Design team and Design Systems perspective I think it would be a resounding yes to improving for a Codex version. Please do reach out to Design-System-Team if there is support that may be needed from DST (code and design-wise)!

In our sync about a month ago we discussed that DST can explore the effort to improve form generation of Codex components in the new year (which is now!). I can provide an update here when we figure out when that work can be scheduled in. We will also assign engineering and design points of contact on our side for this project.

One question though: it was identified that a client-side only UI would be acceptable for an MVP (with a fallback for no-JS support, but not one where we worry as much about the design). Is that still the case, or are we leaning towards only server-rendering the UI? This affects whether we invest more in exploring improvements to HTMLForm vs. form generation options in Vue.

In T332849#9423282, @Sgs wrote:

It seems that the most verbose parts of the current Growth config code is declaring form fields and processing the submitted data. Is that what is meant here by messy? cc @Tgr

Yeah, it's just special case handling on top of special case handling.

@Tgr, have you come across any lib that would be a better fit here?

json-schema-form-bundle seems to be building a form data structure that could be mapped. But in hindsight I think this task is overkill, it seems like a lot of effort / complexity to solve a relatively easy problem (extensions using community configuration having to provide a form descriptor).

In T332849#9434082, @Tgr wrote:

json-schema-form-bundle seems to be building a form data structure that could be mapped. But in hindsight I think this task is overkill, it seems like a lot of effort / complexity to solve a relatively easy problem (extensions using community configuration having to provide a form descriptor).

Thanks for sharing this. I like the idea of having some standard description mechanism for forms but not sure how an additional set of field types would fit the current HTMLForm typeMappings collection.

I found jsonforms.io approach using an optional UI schema description separated from the data schema quite on point. However I discarded it because of its client side nature. Looking back to the task description I see a client solution could be valid here. The problem if we were to take this path is the incomplete Codex form components collection (textarea, textinput, select, chipinput, searchinput). For instance, in EditGrowthConfig we use the HTMLFormField "namespacesmultiselect", such widget does not exist as a Codex component. Even with OOUI I doubt mediawiki.widgets.NamespacesMultiselectWidget is fully functional in the client.

In T332849#9433177, @CCiufo-WMF wrote:

One question though: it was identified that a client-side only UI would be acceptable for an MVP (with a fallback for no-JS support, but not one where we worry as much about the design). Is that still the case, or are we leaning towards only server-rendering the UI? This affects whether we invest more in exploring improvements to HTMLForm vs. form generation options in Vue.

I think using HTMLForm capabilities (maybe with some improvements) is the safest way at this point because of the specific MW fields (title, namespaces, etc). Building the Special:EditGrowthConfig form in the client with Codex is definitely possible but more costly. It would be great to discuss further with Design-System-Team about this since we're facing the same dilemma. In regards if client-side only is acceptable I'd say yes but maybe @Urbanecm_WMF @KStoller-WMF have a different opinion?

Research notes

One of the first questions that has come up is whether it would be more appropriate to render the form on the server or the client. It clearly depends on the system requirements we want to enforce. In preliminary conversations with @Urbanecm_WMF we did not find a clear advantage of building the form on the server. These are the alternatives we have studied:

Considered options

• HTMLForm (server): same setup as current EditGrowthConfig. Makes use of HTMLForm (OOUI display) and FormSpecialPage. Requires ad-hoc logic for generating a form descriptor that matches HTMLForm descriptors.
• Codex + custom components (client): make use of existing Codex components (TextInput, Checkbox, etc). Requires ad-hoc logic for generating a form descriptor that matches Codex components and also building ad-hoc controls, eg: a wiki page selector
• jsonforms + Codex renderers (client): leverage the form generation logic to jsonforms and create a set of vue renderers using Codex components and styling.

Below some comparison of the pros and cons

Server

Pros

• Support for no-JS experience
• Reduced loading times and minimal payload
• Existing form infrastructure

Cons

• Still requires some JS for client validation
• No SSR or hydratation for Codex
• Integrating a third party tool for form description with HTMLForm seems challenging

Client

Pros

• Opportunity to use Codex and latest DSG
• Opportunity to leverage form generation to a third party tool

Cons

• No existing form building infrastructure
• Increased loading time and payloads

Conclusions

After playing a little more with jsonforms I find the library could be appropriate for externalizing the form generation logic and have a high degree of form customization while using the latest MW UI technology available (Vue/Codex). This path would require a non trivial amount of work to create a minimal set of control and layout renderers but it would save us from writing form generation logic based on JSON schemas. On the other hand if we keep using HTMLForm, the current support of Codex in CodexHTMLForm with css-only components seems not ready for a CommunityConfiguration MVP. A fair amount of work is required to achieve the same UX than OOUI provides at the moment.

Regardless of the rendering technology we pick up, the "uischema" concept (used in jsonforms and other libs) seems on point for the problem we want to solve. It does not need to be necessarily written in JSON but such concept could allow to both to work with HTMLForm and jsonforms (or other form renderers).

In T332849#9466680, @Sgs wrote:

Research notes

One of the first questions that has come up is whether it would be more appropriate to render the form on the server or the client. It clearly depends on the system requirements we want to enforce. In preliminary conversations with @Urbanecm_WMF we did not find a clear advantage of building the form on the server. These are the alternatives we have studied:

Considered options

• HTMLForm (server): same setup as current EditGrowthConfig. Makes use of HTMLForm (OOUI display) and FormSpecialPage. Requires ad-hoc logic for generating a form descriptor that matches HTMLForm descriptors.
• Codex + custom components (client): make use of existing Codex components (TextInput, Checkbox, etc). Requires ad-hoc logic for generating a form descriptor that matches Codex components and also building ad-hoc controls, eg: a wiki page selector
• jsonforms + Codex renderers (client): leverage the form generation logic to jsonforms and create a set of vue renderers using Codex components and styling.

Below some comparison of the pros and cons

Server

Pros

• Support for no-JS experience
• Reduced loading times and minimal payload
• Existing form infrastructure

Cons

• Still requires some JS for client validation
• No SSR or hydratation for Codex
• Integrating a third party tool for form description with HTMLForm seems challenging

Client

Pros

• Opportunity to use Codex and latest DSG
• Opportunity to leverage form generation to a third party tool

Cons

• No existing form building infrastructure
• Increased loading time and payloads

Conclusions

After playing a little more with jsonforms I find the library could be appropriate for externalizing the form generation logic and have a high degree of form customization while using the latest MW UI technology available (Vue/Codex). This path would require a non trivial amount of work to create a minimal set of control and layout renderers but it would save us from writing form generation logic based on JSON schemas. On the other hand if we keep using HTMLForm, the current support of Codex in CodexHTMLForm with css-only components seems not ready for a CommunityConfiguration MVP. A fair amount of work is required to achieve the same UX than OOUI provides at the moment.

Regardless of the rendering technology we pick up, the "uischema" concept (used in jsonforms and other libs) seems on point for the problem we want to solve. It does not need to be necessarily written in JSON but such concept could allow to both to work with HTMLForm and jsonforms (or other form renderers).

If either client side or jsonforms is acceptable despite potential added work, that seems like the way to go. Besides the pros of adopting the new over old (Codex over extending OOUI), perhaps one of the cons "No existing form building infrastructure" could also be reframed as a long term benefit if this work leads to introducing form building infrastructure for use by different Wikimedia products (other teams that may find this useful might be those working on for example campaign/event registration, user surveys, Incident reporting, etc).

On the note about increased loading time and payloads for client, and work to create the minimal controls for jsonforms, maybe this is where @CCiufo-WMF can speak to what support DST can offer?

In T332849#9467147, @RHo wrote:

On the note about increased loading time and payloads for client, and work to create the minimal controls for jsonforms, maybe this is where @CCiufo-WMF can speak to what support DST can offer?

I think we'll be connecting tomorrow on the technical support DST can provide and can report back here after.

If you use a client-side form solution, you need to redo form validation on the backend. Which might not be a big deal for community configuration, since the JSON schema can be used for validation. But as a more generic form tool I think it wouldn't be very appealing.

In T332849#9467501, @Tgr wrote:

If you use a client-side form solution, you need to redo form validation on the backend. Which might not be a big deal for community configuration, since the JSON schema can be used for validation. But as a more generic form tool I think it wouldn't be very appealing.

That's right, and validating against JSON schema with a different tool on each part could trigger issues. Given developers will have control over the data the form sends, would it be fair to consider backend validation errors from form data a functional bug? jsonforms validates against the schema on the client so theoretically data sent through the form would be valid.

The same problem exists if we use a server-side form solution, isn't it? We need to redo form validation in the client. Aside from MW's OOUI version of HTMLForm, I don't think I've seen other form system that have "paired" validation server-client.

The form generation logic in json-schema-form-bundle is pretty simple. Maybe that's just because it's not doing a very good job compared to other similar frameworks, not sure. But writing similar code for HTMLForm wouldn't be hard. We lack the form field classes for some specific types like email, but they aren't hard to add - they are just HTMLTextField subclasses with special validation rules. Alternatively, I think it's fine for a form builder to just refuse to work if it finds something unsupported in the schema - as long as the system is sufficiently configurable, dealing with that situation can just be left to downstream code. For schemas with arrays / objects in them that's probably necessary anyway - you can render some sort of default form by recursing but you probably don't end up with a good user experience that way.

In T332849#9470153, @Tgr wrote:

The form generation logic in json-schema-form-bundle is pretty simple. Maybe that's just because it's not doing a very good job compared to other similar frameworks, not sure. But writing similar code for HTMLForm wouldn't be hard. We lack the form field classes for some specific types like email, but they aren't hard to add - they are just HTMLTextField subclasses with special validation rules. Alternatively, I think it's fine for a form builder to just refuse to work if it finds something unsupported in the schema - as long as the system is sufficiently configurable, dealing with that situation can just be left to downstream code. For schemas with arrays / objects in them that's probably necessary anyway - you can render some sort of default form by recursing but you probably don't end up with a good user experience that way.

Agreed. Thanks for our valuable feedback! I'd like to explore further this idea. Looking forward to gather some feedback from DST on the Codex-HTMLForm approach since the current outcome is a bit poor. Meaning many features the OOUI version support don't seem to work with Codex. eg: required fields are not highlighted when left empty.

After the conversations with DST, their recommendation is to start working on the form builder with a client-only version. That is because of the more complete support for form fields with Codex client components. Working on the CodexHTMLForm to add server render support is still desirable. DST is planning to work on HTMLForm improvements with css-only components but not in the short term. The initial form builder is tracked in T356622.