The problem: on my fastest machine, syncing a device that hasn’t been online already takes more than a minute per day.
The impact: This means if a device hasn’t been online for a week, which at least for me happens regularly, it’s unusable for 7+ minutes because sync goes from oldest to newest, so anything I write is at a completely wrong position in the history (since it gets inserted wherever the sync has gotten) and I can’t read answers anyway for 7+ minutes until I get to the current time responses.
The ugly: On my slowest device, syncing a day takes up to 15 minutes.
I’m not sure how you would address this. Perhaps one solution might be to sync newest to oldest, and to keep checking for new stuff during sync.
Signal desktop “solves” this by showing a splash screen while syncing.
I think the bottleneck is SQLite doing fsync after each transaction, so it should be possible to speedup by grouping more SQL statements into transactions and doing all writes in the same place, but it is not as easy as it sounds and requires large refactoring.
yeah, it can probably be improved and there are also ideas around that. esp. when using slow server, slow network and/or slow devices.
however, using a fast chatmail server, syncing some 100 messages per day after being offline for some days is usually fine on most devices [1]. that said, i do not see a “fundamental roadblock” here, the issue more queues in with lots of other things
My apologies, but my reasoning was if I need to wait 10-30 minutes to be able to use the app, I thought for many users that’d be somewhat of a fundamental issue.
Hello, this is quite the problem for me too, as I have a laptop that I only use during travels and so I don’t synchronize it very often, and it can take a very long time to sync (10+ minutes!), time during which it gets very hot.
That means there’s a lot of computing happening, which would indicate the SQLite waiting for the storage to sync is not the culprit. Moreover, the large majority of the computing doesn’t happen in deltachat-rpc-server but in the deltachat-desktop processus, which would indicate that the GUI is slowing everything down.
While synchronizing, the GUI updates quickly with every message. I tried minimizing the GUI but it still consumed a lot of CPU.
I’ve got a similar on my Android tablet, which I don’t use often either. It’s even slower to sync than the laptop because it’s a little old.
Anyways, I would rather have a wait screen if it means the synchronization is faster. I cannot use Deltachat while it’s synchronizing anyways.
Interesting that the Signal splash screen is mentioned here … a few years ago, Signal sync for devices that had been offline was drastically slower, and this was a subject of frequent complaints. Yes there was a splash screen, but they also did a ton of work to speed up the sync process, it’s probably a couple orders of magnitude faster than it used to be, no exaggeration.
I made a backup of my Deltachat data folder from one month ago, and repeatedly tried to synchronize from this backup using different programs. There was about 4000 messages to sync from my own account, so with multiple discussions and groups, with media and stuff. This ran on my Macbook Air 2014 running Debian while powered from the mains (Intel Core i5 4260U, 4GB RAM, 128GB SSD), no swapping occured.
software
user interface
storage
time (secs)
deltachat-repl
tmpfs
128
deltachat-repl
SSD
136
arcanechat-tui
no discussion selected
tmpfs
234
arcanechat-tui
no discussion selected
SSD
243
deltachat-desktop
window closed, quietest discussion selected
SSD
254
deltachat-desktop
window closed, busiest discussion selected
SSD
335
deltachat-desktop
window open, quietest discussion selected
SSD
370
deltachat-desktop
window open, busiest discussion selected
SSD
486
So yeah deltachat-repl is about twice as fast, at least. If one wants to optimize deltachat-desktop sync, one can select the quietest discussion and close the window to let it sync in background.
I’m not sure if I can use deltachat-repl to sync an account created with deltachat-desktop, probably deltachat-desktop needs to process messages to update its internal data that is independent from the core’s own internal data.
Anyhow I think this data above suggests that a specialized synchronization splash screen could substantially improve the synchronization time since it seems to be linked to the state of the UI.
EDIT: OMG 2014 not 2024. It’s an old laptop not a newer one. Those times would be way lower on a recent computer. Fortunately I did list the specs.
It is fine to synchronize accounts without an UI, UIs don’t depend on processing incoming messages except for displaying notifications and deciding when to refresh the chat. Running deltachat-repl is a bit risky because it does not check for the accounts.toml lockfile and if you run deltachat-repl next to deltachat-desktop on the same database, you will receive duplicated messages, but if you are sure that Delta Chat is not running while you run deltachat-repl, it is fine.
This slowdown is likely caused by the UI reloading chatlist every time chatlist order changes and similar database access.
I have experienced long delays when chatting; sometimes I am typing blind, because the display is not displaying my sentences until I have finished writing them. Is thhis the same issue? It happens in chats with long histories and recent messages.
Yours doesn’t sound like the same issue. My text input is updating fast when I type. The sync problem is merely when you haven’t turned on your device for a while, and then all the old history needs to catch up.
