WebXDC dynamic preview & click listeners (inline apps)

I can offer the same response I’ve already given at the other topic which would have none of the drawbacks mentioned:

I.e., the webxdc app could render a static preview not into a raster image from a canvas, but instead into just a plain HTML DOM, possibly containing CSS and ideally up to a small amount of optional (SVG) images. If the app is implemented correctly (i.e., with progressive enhancement), possibly almost no additional code needs to be written for this use case.

Such a static preview could then be cached in the local database, possibly as a single self-contained HTML file under the blob directory. It could be rendered into a sandboxed (iframe) widget that has JavaScript disabled, hence it would be more lightweight to process than if the fill-500 hack was required every time it scrolled into view.

The simplest alternative to dehydrate such a component would be to require an additional button press to first activate the app (and make it unblurred, restore its colors, hide the play button, etc.) and then allow normal interactions.

A more seamless way would be to overlay a <div> on top to catch clicks, activate it upon clicking and forward the click coordinate through the already mentioned API that the app could dispatch to itself again after loading (i.e., so that normal DOM bubbling mechanisms would take care of finding what what clicked on). Indeed this is less accessible as a user can’t tab to individual buttons in the app, but at least they could inform themselves about it with a screen reader (possible present a dedicated activation button for such users as a substitute?). I would like to note here that I don’t think that we have any accessible Delta Chat clients so far.

And the possibly best solution would be if the static preview HTML was marked up in an elaborate way so that the Delta Chat sandbox could determine which elements can be interacted with (such as by depressing, clicking, swiping and typing text). The sandbox could remove any possible webxdc JavaScript from the static preview and add its own event handlers to such elements.

These event handlers would call back to the sandbox via new postMessage() API to signal which element was interacted with (such as by ID attribute) and in what way. The sandbox would remember this, replace the static preview with the real app, start it, and then invoke a new API on the app to tell it about the interaction that activated it so it could replay it internally. The latter might also be done through the URI anchor in certain cases. However, if we wanted to support interactions other than through a simple click, it would not suffice to take the shortcut of marking up clickable regions with such href links for the former cases or by walking tabindex - we would really need to define data-delta-chat-interaction=click|swipe|keypress attributes for such a purpose.

For some advanced use cases, it could also make sense to allow moving radio buttons, toggling checkboxes and typing into input and textbox while only activating the app itself upon a click (such as on a vote submission button), but also submitting the data already entered on the form above upon loading (i.e., so it could process the given vote).

It’s an open question how to deal with updates. The threat model (and existing susceptibilities) of Delta Chat would allow sending such static HTML state previews to your peers for them to render on their timeline. This means that the poll state or chess game on your timeline could always appear up to date.

If Delta Chat took a turn for more robust protections in general, it could make sense if only the local user could generate a preview for themselves. If a new state update came in, either a clickable banner warning about a possibly outdated preview could be displayed under the preview. As an alternative, if the user has already opened this webxdc in the group previously, Delta Chat could activate the webxdc in the background so that it could process the updates, generate a new static preview and terminate. This would need an additional API and also a newmetadata field on the webxdc to signal that such batch static preview updating is supported.