"Upgrading" a WebXDC app [migrating state while keeping pinned apps unchanged]

Mini Apps
1

Has there been much thought about how users might be able to “upgrade” a WebXDC app? Many of the apps are ephemeral (such as polls) and so it doesn’t make sense, the app appears in the conversation and then goes past in scroll and if you want to use it again you attach it again.

But some apps, like shared notes or calendar, are more “sticky” in nature and it would make sense when a new “version” of the xdc file comes out to be able to replace it while keeping all state updates so far the same (of course this assumes the xdc is designed to be backwards-compatible with older versions of its own state updates…)

What do people think about this?

6 Likes
2

We thought about this, but ultimately postponed it because there are so many open questions.
Here two of them:

A: You can’t really expect all xdc to be perfectly backward compatible, so you would either need some backup or downgrade solution.

B: in a p2p scenario you can not be sure that everyone already uses the upgraded version, let alone even received it.

and some solution ideas:

  1. A could be worked around with sending a new message/instance with all the old updates, but the new xdc version, this way the old state would not be lost immediately.
  2. Exporting and importing state/backup between xdc instances could allow a first version of upgradable apps.

Also webxdc apps are currently pinned on a single message, but we could also imagine that you could pin an app to a chat or have it in a way where it is more persistent and new members automatically get it and it’s previous state.
But even then problem B remains, and unless your app is forward and backward compatible (by ignoring unknown properties for example) there really is a need for a solution to problem B.
Also there is the question how this would play into future status update discarding / condensing/aggregating into a single state (I think some call this “checkpointing”).

3

Here’s a user-land workaround (that may not work for some apps, e.g. those who do webxdc.sendUpdate immediately on first open). I’ll demonstrate it in the console (so you probably need Delta Chat Desktop), but export/import can be done inside the app as well, say, with clipboard.

  1. Send the new version of the app to the chat (don’t do anything with it yet).

  2. Open the original app, then open the console for it. Paste this:

    const updates = [];
await webxdc.setUpdateListener(update => {
    updates.push(update);
});
JSON.stringify(updates);

  3. Right-click the resulting string, “Copy string contents”.

  4. Open the new version of the app, and open the console. Paste this:

    const updates = REPLACE_ME;
for (const update of updates) {
    webxdc.sendUpdate(update, 'Import');
}

  5. Replace REPLACE_ME with the coped string and press Enter.

Now the state of the new version should be the same as the state of the old one.

2 Likes
4

A&B: What is the difference to an ordinary Android/iPhone app? You can use an app without Internet for one year and after going online the data must be compatible, too. Sure, it would be a great feature (for all OSes) to visit each app in each old possible version, but again, its not a specific p2p problem.

It would be great to be able to create “upgradable” apps by creating a symmetric key pair and only the owner of this key is able to create new versions of this app. Maybe the (constant) public key and a version specific signature could be part of the manifest for such an app? This would be less chaotic and I think the other OSes are doing it the same way.
(And if you dont have the key, just start with version 1.0 with a new key)

Why this would be nice:
Even if you update some fonts or the user interface and leave the data code untouched, you have to upload a complete new app. For games it is really annoying to loose the highscore every time the game gets some updates. And If you can ex/import it, cheating is too easy :grinning:

1 Like
5

I just wanted to open a new thread for this but I realized there is already a thread for it.

@Simon The webxdc app developers need to care about compatibility not DeltaChat. If they are afraid a new version might break things with old data, they simply send a new instance instead. Like there is a new entry in manifest.toml which says this app version is compatible with versions 1.0 and 2.0 but not 3.0. Or even we could simply force semantic versioning.

Overall, I understand this shouldn’t be taken lightly. But if we do it, it’s worth the effort.

2 Likes
6

We could peek at how web extensions do it. I’m not saying this is the best API I’ve seen, but it works. They have the runtime.onInstalled event emitter, which can fire an event when the extension gets upgraded to a new version. This is usually where you run migrations.

And, of course, extensions utilize signatures as well. Although TBH I’m not sure if there is a big need on signatures in the context of webxdc.

2 Likes
7

I just had this happen; shopping list app and my wife’s XMPP client dumped the webxdc shopping tab. The download had aged out up at the server, so boom–everybody has to install the same app again, and I have to do an export and import. Clunky! I’d be just as happy if each BOM had a UUID, and messages are shared with all apps sharing that UUID. When you break compatibility, you generate a new UUID.

1 Like
8

This problem seems to be different from upgrading and easier to solve, because in this case it would be fine to “upgrade” to the same version of the app.

Which XMPP client it is? The app is not cached forever on the client and the client expects to be able to download the app again when you scroll back to it?

In Delta Chat there is no such problem of the messages disappearing because they are only downloaded once and stored forever. There is however another problem with the app not being available to those who joined tho chat later and only becomes available if the original sender of the app clicks on “resend” it. Maybe XMPP can have it implemented the same way, allowing someone who still has the app to refresh the file on the server by uploading it again and sending as the new message with the same message ID.

9

I just wanted to join this conversation because I see it as a big obstacle for further adaption. I have a ton of app ideas but the upgrade issue bothers me quite a bit. I think for apps that are more hen just toys being able to upgrade is a necessity. Quick example: I have a group with some friends where we are trying to get more healthy and lose weight, so a weight tracking app would be really nice. If people are willing to use it they will definitely come up with new feature ideas that we want to implement.

Without migration I would have to stick to import/export tricks. While this works this might confuse people if different people would use different versions of the app we would basically end up with a fork some people might enter their data into an outdated version of the app while others don’t. So updates are kind of completely impossible in this scenario if one want to prevent confusion.

Even worse: if someone accidentally sends the identical .xdc file twice one ends up with a fork in a road. This can happen very easily “hey, where is the link to the app again?” “No worries, I’ll just send it again, here you go!”: bang, new instance of the app with different data with no way to distinguish them or merge them. Not great.

I think the migration issue would be smaller in praxis then it seems at first glance for various reasons:

  • if migrations only happens if the app provides some kind of identifier in the manifest file then migration is completely opt-in: you don’t want to handle migrations? Easy, there is nothing to do for you
  • updates don’t necessarily need to change the API and even if it’s oftentimes possible to make them additive so older version simply won’t see some of the features
  • app developers are usually used to these migration headaches because they happen all the time in app development, both for
  • in praxis if there is a newer version of an app and most users probably will switch quickly.
  • the app developer can simply send an “appversion” property with every update. when a version of the app receives a version is can’t handle it can show an error message to the user that they should use the latest version instead

I think it makes a lot of sense to add the option to update apps. Apps that don’t want to handle the burden don’t have to while everybody who is willing to walk the extra mile has the option.

2 Likes
10

I don’t feel that any of your concerns are legitimate:

  • All webxdc sent to the chat that you have access to are available under the Apps & Media tab. If you send it twice, it will appear twice and others will notice the problem.
  • You can remove the old one by clicking Delete Message → For Everyone.
  • You can click Resend on the webxdc message if a new joiner is asking for it, and it will not be duplicated for anyone in this case.
  • The Group Editor Bot for example supports pinning arbitrary messages (including your own webxdc) that it sends to each new joiner automatically (again, it does it by clicking Resend).
  • Some better webxdc already support an option to backup (export) and restore (import) its user data at a high level format to a message you can store at a safe place (or apply yourself during migration). I have upgraded many webxdc in the past this way. If the ones you use lacks this feature, feel free to open a feature request for the author (or submit patches yourself).
2 Likes
11

One corner case that you do need to watch out for is that only the original sender is allowed to Resend a previous message (so far). This should not cause a problem in real life, because you ought to remove inactive (dead, unavailable) members from your group chat.

If you do happen to have a webxdc in your group chat that members rely on sent in by a dead member, it’s best if you export the user data, click Clone chat, send in the new version of the webxdc to the new chat and then remove all members from the old chat so that others can’t accidentally use the unmaintained version of the webxdc.

1 Like
12

What about 1)

  1. You export the old Webxdc data
  2. Delete old Webxdc so that nobody can add data anymore
  3. You add the new Webxdc
  4. You import data
  5. Someone asks you where his data is (It was inserted between 1 and 2)
  6. You say “It’s at a better place now“ :grin::sob:

What about 2)

  1. You have 10 people in your exercise group
  2. Everyone has a Webxdc shortcut on their homescreen, which is great for sticking users to delta/arcane chat :partying_face:
  3. You do the updating, including the deletion of the old Webxdc
  4. Each of the 10 users loose their shortcut (for the 10th time :expressionless_face::expressionless_face::expressionless_face::expressionless_face::expressionless_face::expressionless_face::expressionless_face::expressionless_face::expressionless_face::expressionless_face:)
13

1/5: I suggest you announce the exact time of your planned upgrade 24 hours in advance as a maintenance time window within the same group. People can be understanding. :hugs:

2/2: Never used that feature, but sounds interesting. I guess you can ask group members to add the homescreen shortcut again after the upgrade.

14

How can I open “the console for a webxdc app”?

15

That’s disabled by default as a security measure against exfiltration.
The toggle can be found it advanced settings.