Technical background of DC:
Blob files are only stored in database when exporting a backup. While importing these are deleted again and vacuum is done afterwards (This is a problem if storage is short at device!).
With an installed DC no blobs are stored in database. Files are stored in blob directory as files!
The vacuum process in the moment is only started after importing a backup file. While normal operation of DC this is still not available for a user (May come in future versions).
My experience up to now shows that most storage space is occupied by blob files outside of the database. In fact, vacuum could help to reduce occupied space of the database itself, but blob files are feeling to me more important to delete them in case of a “delete process” in DC. I don’t know if this is done by DC in current version.
First I have to discourage anyone to use the feature of Android to integrate the SD-card into internal storage. As experience tells me that construction will fail sooner or later & you’re struggeling to recover the encrypted data if you don’t have a backup.
Secondly I don’t get why nearly all apps don’t allow the user to move it’s database to the SD-card. All the time I get access to other peoples devices I see they are nearly out of internal storage yet their external SD-card has tons of space left. Why not warn the user about security concerns and then let him decide for himself?
Manyverse, a client app for the decentralised social network Scuttlebutt, has to deal with potentially very big local storage demand and already supports putting it’s database on the external SD-card.
So I’d really appreciate Delta Chat giving the option to store data on the external SD-card!
DC already has external storage - IMAP server(that can hold practically unlimited amount of imformation), so IMHO it isn’t necessary to keep all emails locally on device. As I already mentioned here, DC-mobile should keep defined number of messages locally on device base on (future?) preference 100/250/500/all as it done in K9. If user need old messages, one can pull it from IMAP in the same way as it works in K9 - by pulling down message list down.
While text won’t take a lot of space and can be kept in sqlite database locally, it isn’t a case for blobs that takes a lot of space and early or later everyone will step on this issue. IMO DC mobile apps should keep locally on device only cached blobs base on some settings - “how long to keep blobs” locally: last 100/250/500/ALL messages and allow to pull old blobs from IMAP if needed when user scrolling to old messages.
some people have only 100MB of space in the email server and want to keep the messages in their phone, anyway not sure if there are someone interested in keeping messages older than X months
Judging by my XMPP client usage it’d keep all messages on the phone. I often search through my XMPP database on the phone for messages several years old.
All businesses we working with, requires us to keep absolutely all emails on the servers. If DC would be used in such environments it will choke at some point for sure.
we are not talking about force the user to delete messages, just to give the option to do so, you have the need to keep the messages but we have the need to delete them, you will not need to do nothing the default could be to keep them forever
I still can’t get it, why not to use K9 model of keeping messages that proved by time.
It really simple, - keep on device the only number of last messages user wants, for example 100/250/500/all and keep rest of messages on the IMAP server that can be pulled on demand if needed.
Messages that deleted on device should be deleted also from a server. Simple.
This way, those who has limited space on IMAP server can manage what to keep and what can be deleted. In the same time those who has “normal” or huge amount of space on the IMAP server, they can keep messages without killing mobile device by filling all its space (which is very easy to achieve with attachments) and be able in the same time get old messages if needed.
DC isn’t snapchat to exchange with kitten once and forget about it, but a communication program that utilized email protocol that assumed to keep message forever if it needed.
I already has two correspondents(a lawyers who exchanged a lot with attached documents) who stopped using DC because of this issue and early or later every1 will step on that if it wouldn’t be resolved.
AlexJ I was talking about to keep them forever in the server by default, but let people with small server storage opt-out, maybe you have almost infinite storage, but some people have only 100MB
actually I think that would be a nice (optional) feature for Delta Chat
According to the title of this topic, - the issue is that DC keeps locally, on the phone ALL messages including blobs and fill out device’s space.
I’m talking about majority of users who has more than 100Mb on IMAP server and proposed solution how it can be resolved, - the same way as it done in K9.
All phones I know allows for example to set preference - how many text messages to keep, the same option need to be implemented in DC to avoid putting smartphone to the knee.
Actually it should be more flexible to satisfy people like you who has limited space on the server and those who can at least use gmail or their “friends”.
I see it as an options:
Keep locally on device only last: 100/250/500/All cached messages and allow to pull older messages from IMAP server on demand
Automatically delete messages that older than: month/year/none(keep all)
If option #1 is set to “all”, then unhide option: Delete old messages if space occupied by DC exceed 0.1/1/2/5/10/9000Gb
I think it will satisfy all DC users, those who has plenty of space on IMAP server and those who has limited amount of space and what is most important phones won’t be overfilled by DC in all cases.
Properly solving the message history and storage usage (devices/server of any sizes) for all use-cases, while keeping it to a minimal set of user-visible options, isn’t trivial, but shouldn’t be too hard either.
IIRC the required user-facing items for the full solution are only three:
Additional message states: starred (never deleted, manual selection) and archived (usually remain only on device, automatic expiration). (The other (recent) messages get synced with the server and across devices. NOTE: The “archived messages” are different from the old “archived chats” which are only hidden/suspended from the chat list. The “archived messages” stay visible in the chat histories, are not synced with the server (and thus other clients) anymore, and default to remain only on the device.)
Max. number of messages kept on server, and thus synced accross devices (per chat), default value 250? (The client with the smallest setting usually determines the amount of messages kept on the server.)
Default true: delete exceeding messages from server
Disabling this suboption
i) in conjuction with reducing max. messages kept on server/synced to 0, configures a device (with enough storage) to keep all incoming messages , while other devices may still sync more and even delete a part of them before they expire on the server.
ii) in conjunction with a smaller number kept on server and synced, and possibly a small archive limit, configures a device to sync less messages than kept on server, i.e. for low-storage devices or huge-storage servers.
Local (archiving of) messages. Keep messages that expired on server on local device? Default enabled/disabled? (enabled is safe if auto-delete-device is implemented)
Suboptions:
Default disabled: Limit number of (synced + archived) messages (per chat), to limit the number of messages kept on the device after they expire from the synced messages.
Alternative option that will also remove low-traffic chats: Default disabled: Limit age of (synced + archived) messages (per chat)
Required background mechanisms:
auto-delete-server
always guaranteeing enough free (quota) space on the server for continuous operation (and only deleting the last (synced-new) messages of every chat if no other option remains.)
limits the max. number of messages kept on the server
auto-delete-device
guarantees free space on the device (maybe auto-deleting from the synced rather than archived? (and first reducing the the local email inbox list?), or offering to delete/backup archived chats if necessary)
optionally limits the local history if configured
A full current design:
(It has some more details like considering the latest messages of each chat (state synced-new) more important, to prevent the auto-archive and -delete from completely deleting low-traffic chats etc.)
In current 0.200.0 there is the housekeeping function at backup export which deletes unused files.
this works well (imho)
beside of that there should be an automatic or manual function to do this job separately from backup.
One detail of the above background mechanisms would actually make deltachat work without requiring manual adaption of the default number of max. messages kept on the server/device (out of the box) also with small mailboxes/storage:
when the free space falls below say 6% or 110MB, notify the user that DC will soon (below 5% or 100MB) have to start automatically deleting the oldest chat messages on the server/device
for sure, there are usecases for external memory, but it seems still be a comparable low impact for lots of work and maintainance, so resources are spent better elsewhere. esp. as one can increase the memory on system-level with SD cards on many OS (probably on more OS than 5 years ago): Get started with an SD card - Android Help (see “internal storage”)
i have redmi 9 nad tried the workaround to enable sd card as internal but after a week i guess there is no garbage collector and even when i deleted all apps tht where only internal i couldnt install any app and i got constantly a warning that my internal space is always full, had to reset to factory and just accept that i have only 32gb internal…
