WebXDC dynamic preview & click listeners (inline apps)

Feature Proposal
, ,
1

Goal

Allow WebXTC apps to work in a very limited way in inline mode, receive click events and react to them. This enables features like polls directly in the chat, custom widgets and more

Proposed additions

This proposal would enable webxdc apps to work inline in a very limited way. It would basically allow an webxdc app to dynamically render the preview image that is shown in the chat whenever certain events happen:

  • comes into view
  • update listener fires (but delayed if not in view)
  • user taps somewhere on the preview image

The following features would have to be added:

  • allow WebXTC app to define (and re-define) the height of its inline area (within certain limits). This is useful i.e. for polls where the list of options can grow
  • allow WebXTC app to receive click event info (coordinates + button) within the preview area
  • allow WebXTC app to decide if it should be opened or not when the preview area was clicked. This allows the app to decide if it should stay in inline mode (i.e. because the click is was just the click on a poll option vs it was the click to add more options which requires the app to open and show an interface that allows to enter the option)

I’m not sure how rendering the preview area should work, options that I see:

  • the JS just renders a bitmap that the client shows. I guess this client implementation would be easiest, but the app side will be a bit more complicated and in general the approach is a little dirty
  • an actual html webview. This would offer more options like animations, but I’m unsure how hard the implementation will be and how big an possible performance impact is. Due to extensibility I still think that this would be a cleaner solution

In both cases, as soon as the app leaves the visible area by a bit, it would be stopped. If possible, we could also impose strict resoure budgets so no heavy animations or calculations can run without the user opening the app.

To improve performance, we could also cache the last rendered state and show it again when the app comes into view again but before it was woken up again.

Advantages

The possibilities would be endless and it would address many different concerns, the prime example being polls 'Inline' realtime polls but many more like some realtime status, game highscores, a preview of a todo list with options that can be ticked immediately, previews of collaborative drawing tools etc.

2

Seems like an extension of Feature suggestion - view apps inline (with some extra details).

TBH the “show image and detect click coordinates” doesn’t sound like good API.

  • Not accessible
  • Not sure if there are any use cases besides polls.
  • Complex for the app itself to implement image rendering and figuring out what exactly was clicked.

Do you have an idea how this can be done in practice?

Would this be just the same app but inline, or something specifically made for the inline mode?

3

I can offer the same response I’ve already given at the other topic which would have none of the drawbacks mentioned:

I.e., the webxdc app could render a static preview not into a raster image from a canvas, but instead into just a plain HTML DOM, possibly containing CSS and ideally up to a small amount of optional (SVG) images. If the app is implemented correctly (i.e., with progressive enhancement), possibly almost no additional code needs to be written for this use case.

Such a static preview could then be cached in the local database, possibly as a single self-contained HTML file under the blob directory. It could be rendered into a sandboxed (iframe) widget that has JavaScript disabled, hence it would be more lightweight to process than if the fill-500 hack was required every time it scrolled into view.

The simplest alternative to dehydrate such a component would be to require an additional button press to first activate the app (and make it unblurred, restore its colors, hide the play button, etc.) and then allow normal interactions.

A more seamless way would be to overlay a <div> on top to catch clicks, activate it upon clicking and forward the click coordinate through the already mentioned API that the app could dispatch to itself again after loading (i.e., so that normal DOM bubbling mechanisms would take care of finding what what clicked on). Indeed this is less accessible as a user can’t tab to individual buttons in the app, but at least they could inform themselves about it with a screen reader (possible present a dedicated activation button for such users as a substitute?). I would like to note here that I don’t think that we have any accessible Delta Chat clients so far.

And the possibly best solution would be if the static preview HTML was marked up in an elaborate way so that the Delta Chat sandbox could determine which elements can be interacted with (such as by depressing, clicking, swiping and typing text). The sandbox could remove any possible webxdc JavaScript from the static preview and add its own event handlers to such elements.

These event handlers would call back to the sandbox via new postMessage() API to signal which element was interacted with (such as by ID attribute) and in what way. The sandbox would remember this, replace the static preview with the real app, start it, and then invoke a new API on the app to tell it about the interaction that activated it so it could replay it internally. The latter might also be done through the URI anchor in certain cases. However, if we wanted to support interactions other than through a simple click, it would not suffice to take the shortcut of marking up clickable regions with such href links for the former cases or by walking tabindex - we would really need to define data-delta-chat-interaction=click|swipe|keypress attributes for such a purpose.

For some advanced use cases, it could also make sense to allow moving radio buttons, toggling checkboxes and typing into input and textbox while only activating the app itself upon a click (such as on a vote submission button), but also submitting the data already entered on the form above upon loading (i.e., so it could process the given vote).

It’s an open question how to deal with updates. The threat model (and existing susceptibilities) of Delta Chat would allow sending such static HTML state previews to your peers for them to render on their timeline. This means that the poll state or chess game on your timeline could always appear up to date.

If Delta Chat took a turn for more robust protections in general, it could make sense if only the local user could generate a preview for themselves. If a new state update came in, either a clickable banner warning about a possibly outdated preview could be displayed under the preview. As an alternative, if the user has already opened this webxdc in the group previously, Delta Chat could activate the webxdc in the background so that it could process the updates, generate a new static preview and terminate. This would need an additional API and also a newmetadata field on the webxdc to signal that such batch static preview updating is supported.

4

Thank you for your replies. The accessibility problems etc. are true, while writing everything I’ve already noticed that rendering an image could be problematic, that’s why I have added the html option. The more I think about it the more I think that this would be the better way. I would also go away from click events with pixel coordinates and instead have something like anchors that have a data attribute that is given to the apps javascript (I would deliberately not do this with click handlers to emphasize that this is not a normal js click event interaction).

Regarding usecases I honestly see many besides polls. Almost any app could benefit by being able to display more information instead of just showing the banner image in the chat which basically does not carry any information. For simple games, the whole game could happen inline (Chess, card games, …). Bots could show a weather widget. Games could show a list of players currently in the game. Games could show highscores. Games could show a “your turn” info. The possibilities are basically endless.

Regarding the threat model, I think it’s a valid point: We should only show the inline data after the user has started the app, until then I would just show the normal app icon. Actually, this behaviour comes automatially because my model would work like that that the app deliberately has to call an api to set something else than the icon. This api call can only happen by running the app anyway.

Basically both answered by ian. Regarding the last question: The apps code would have the option to somehow set the data (i.e. by calling an api with a DOM element) that should be shown.

I think it’s easiest to imagine with a bit of conceptual code on how the API would look like from an app perspective. I will try to write something together in the coming days

5

I could think of use cases, but not specifically ones that you have listed. For many use cases, a huge amount of UX redesign would be needed.

If a group chat where people are having conversations also includes chess and a card game attached, it would be a kludge if the player had to open the group and manually scroll back to the “right” position where the game is every time to be able to make their next move. People rather click on the docked app icon next to the subject (on desktop) or select the app from the app drawer panel. Clicking either one will already open the game full screen. Making a tiny icon in the former case interactive would not make sense, and making the still small thumbnail in the latter interactive would still not feel right.

If you wanted to “pin” webxdc messages (subject to further development) so you would not need to scroll back manually, its interface would probably just be a duplicate for what is already available (either the subject xdc icons or the existing app drawer).

If a group has many apps, loading each and every one of them every time you switch to that group or as you scroll the timeline is very resource consuming. Each sandbox consumes quite a bit of memory and a huge CPU load and delay on startup on most platforms.

Even if you only loaded ones you have already interacted with in the past, you probably don’t want the chat to lag due to a concluded poll or finished game that you would not want to open again anyway.

There are quite a lot of other questions you need to answer before you proceed with this.

I mentioned my personal use cases in the other topics for this personally: one would be replacing huge throw-away raster videos with programmable animation that others can realistically fork & improve later. The other is for controlling native webxdc. Additionally, I would value such a feature for previewing unfamiliar webxdc before loading, such as reading through the poll options to decide whether I need to open it or not, but that only works with strict isolation that I have outlined.