Backup service concept

Some context: For the past month I have been using Delta Chat extensively while traveling abroad with my family. This is our first experience with Delta Chat and I am very impressed. We are using it across multiple device OSs (Android, iOS iPad and iPhone, Linux) and highly dynamic networks. We have used it to call each other, video chat, share live locations to rendezvous, and share photos and important files during our activities. As someone who has been experimenting with and contributing to FOSS decentralized messaging platforms for 15+ years, this praise comes from significant experience with this ecosystem. I’ll lavish more praise in another post :grin:.

Two fundamental challenges for serverless apps like Delta Chat are seamless transition between user devices and disaster recovery. Consider the following scenario:

Alice creates a Delta Chat account on her phone using a single random relay after being invited by her nerdy partner. She doesn’t change any defaults settings or think about anything but getting through the steps so she can send and receive messages. After two months she has added her tablet as a second device and has several important group chats and direct messages with friends and family. She relies on it daily and expects her data to be available. She even installed Delta Chat on her kid’s iPad so they can chit chat when she’s at work without worrying about privacy policies and terms of service.

She has never considered actively enabling backups because all of the rest of her apps are automatically backed up to iCloud (or whatever Apple/Google/EvilCorp does; she does not particularly care). Even if she did think about making a manual backup of her Delta Chat data, a file export would be an alien concept because she does not think in terms of “files” on her mobile devices: She has “apps”, and any data she has lives “in” those apps, transferrable only via a Share button (if one exists and then only to the apps listed).

One day, she loses both devices when her luggage is lost irrevocably by the airlines. After paying lots of money to replace her phone, she logs in with her Apple ID and all her apps are magically restored along with their data. Except for Delta Chat, which doesn’t even remember who she is. She is distraught when she discovers that data is as lost as her luggage. She never uses Delta Chat again.

My idea is to add an optional service to relays that stores backups of messages for users who activate the feature. The messages are already e2ee, so the only trust involved would be in the integrity and availability of the backups when needed. In addition to the messages, the user identity itself would be part of the backup, recoverable by knowing the server address and the key generated upon initialization known only to the user. Backup depth, storage limits, and other configuration options would need to be designed.

This could be an opportunity to ethically monetize the relay network and grow the ecosystem, because obviously backups require storage and that costs money. There would be a financial incentive for people to run relays that host backups, and the competition would keep prices low, especially since there is no vendor lock-in effect to distort the market. Supporting S3 natively as the default storage backend would be best to make it as turn-key to deploy and as cheap to operate as possible. Restic could be considered as the core component of the backup engine.

Another possibility would be to implement peer-based backups, where a user requests that certain contacts serve as backup hosts with the option (even encouragement) to reciprocate. Messages would then be stored on their selected contact’s devices, encrypted of course so that again, the trust is only in future availability. This scheme is technically more challenging to implement because the information needed to restore data must be included in the decryption key that the user would be required to store in their password manager; there would be no server to remember for them. Clearly this is more fragile in terms of recovery due to potentially intermittent connectivity to the storage peers, but possibly more robust in terms of actual storage, because the data could be sharded and stored using an erasure encoding across the target contact devices. Lots of fun technical details to puzzle over here.

Mostly I’m having fun brainstorming at the moment, and I am interested to hear the perspective of Delta Chat experts and learn more about the application in the process.

I think users are already too reliant on servers, and the last thing we would need is yet another commercial server hoarding a lifetime of data & metadata.

I would enjoy seeing the mentioned reciprocal scheme for peer to peer message backup based on secret sharing. However, note that even if such a system were to be implemented, instead of Delta Chat implementing it separately, it would be desirable for all apps on your device to use it, but then it would need to be running a daemon for each and keep syncing with subsets of peers for each. This would be quite inefficient.

Here are answers to some of your questions:

I’m afraid implementing a backup service that is specific to just the Delta Chat app is a dead end. A more generic service would be better. WhatsApp suggests you to store backups on Google Drive, Signal stores backups as a file locally.

But yep, brainstorming is fun.

Interesting topic so I’d like to add my 5cts to it :smiling_face_with_sunglasses:

I totally see the scenario @andrew.czmwkpwim described to apply to most of the users who are non-nerds.

My solution would be to notify people which are using a single device only directly after the “onboarding process” (and maybe from time to time like Telegram does¹) to add at least a second device to have a “profile backup”. Only downside would be that people should be made aware of the fact that they have to use the device regularly otherwise some of the messages will be lost in case they need the “fallback” for a “restore”. But the profile would always be there, which is what @andrew.czmwkpwim mentioned to be most important). And this kind of solutions has the necessitiy that people own at least two independent devices which can run Delta Chat (which might not be the case for especially people with little money).

As every device acts on its own and therefore already is a “full backup” I would consider this the most convenient solution. So the only necessary change would be some kind of “backup reminder” regarding Multi-Device. What do you think?

Âą Example of Telegrams notifications (some might call them nag screens :wink:, but they work):

I want to believe that P2P style backups are feasible (and maybe they are for basic usage), but all it takes is not being able to recover that one 5 year old video you cherish to get completely soured on the whole thing.

Signal sells full account backups for $2 per month. A hard cap at 100 GB. This is totally possible with chatmail. You just make a custom client and offer the recovery system as something completely orthogonal to the chatmail transport relays. Use blinded tokens to pay for backup so you’re not connecting users to their data. And you can even improve on Signal’s design by making their 64 character recovery key a shamir’s secret sharing scheme among select contacts.

Two things I haven’t seen discussed anywhere. Does the backup need to carry the PGP key and chatmail credentials too? Otherwise you restore your whole history and you’re still a stranger to all your contacts. And chatmail deletes accounts after ~90 days idle, so if it takes four months to replace a phone the address is gone regardless of how good the backup was.

And yes, this could be a generic S3 shape, so it wouldn’t have to be “chatmail only”.

I also see the problem stated here, but I have a slightly different suggestion on how to solve this:

DeltaChat is based on mailservers and a server configured for that usecase would already be a solution to the problem. The only problem is that most users are probably starting with one of the default relay servers and until they switch to a mailserver with configured persistence, their messages would be lost in such a scenario.

So why not introduce a “sync to server” feature in the client that makes sure that a given mailserver (possibly even multiple) has exactly the message state that the local client has by using IMAP features to create that state on the server? As far as I read (but I don’t know the details) multiple relays are already supported. So if we’d also support multiple persistent mailservers and would allow the client to sync messages to them, we’d have a solution to the problem, right?

This also adresses another point that I have: Currently I’m using Whatsapp and Telegram. In Whatsapp I hate to send any photos because if I want to keep them (and I do - I just like keeping a message history) they will forever take space on my phone. In Telegram it is extremely convenient because media can just stay on the server and when I scroll through the chat it gets downloaded again and when the cache gets evicted I get the storage space back without having to delete anything. From a UX perspective this is fantastic with the only (big) problem being that Telegram has access to all the messages and everything depends on Telegram.

With the ability to sync the local chat to one or many mailservers DeltaChat could be the best of both worlds, E2E Encryption and UX in terms of storage space on the phone and backups. One more advantage that I can see: adding a second device could be possible without the devices having to be on the same network: We’d just have to transmit the key (and mailservers + possibly username/password combo) from the first device to the second (i.e. by QR code or passphrase) and then the second device could download all messages from the server instead which might be much more convenient for users, especially if they have many messages

What I’m unsure about is if imap has all the features necessary to do the syncing but I think it’s possible to make it happen.

I think this would fit perfectly into the existing concept of chatmail and would be much easier to implement then peer to peer syncing features.

Happy to hear your opinions :slight_smile: