Reduce space on internal storage, disk full

files belonging to deleted chats currently stay in the blob-folder currently, the plan is to have a garbage collection that checks for unreferenced files from time to time, however, it’s just not done, it’s strongly related to https://github.com/deltachat/deltachat-core/issues/244.

if messages are deleted, however, the belonging files are already deleted if they are no longer referenced.

nb: the same file can be shared by different messages, so a 20 mb file that is forwarded takes up only one time space.

in general, there is also the idea that the blob directory is no longer used at all and all blobs are stored directly in the database. this would (a) make housekeeping easier and (b) make at-rest encryption easier (https://github.com/deltachat/deltachat-core/issues/194).

i’ve just filed a separte issue for housekeeping: https://github.com/deltachat/deltachat-core/issues/484 - if anyone has experiences with sqlite and larger database, please speak out loud here or in the issue :slight_smile:

I don’t think it would be a good idea. When you deleting record from SQLite database, it won’t reduce size of database file. SQLite just marking deleted space as empty for future “inserts”. To reduce actual file size of database, it would be required to run periodically “vacuum” procedure that will drain uselessly battery. Database should keep only references(path/file_name/modified date/may_be_hash) to blob files that should be handled by operation system, such as attachments/images/etc.

P.S.
I think that functionality that exists in K9mail to set number of recent messages that kept offline on device - is a way to resolve this issue. If one need to read older messages, he/she can use search that can run on the server side and also may explicitly pull old messages by pulling screen down as it works in another mobile clients

P.P.S
BTW, SQLite won’t accept single blob file if it’s bigger than 1Gb and it’s hard-coded limit.

Technical background of DC:
Blob files are only stored in database when exporting a backup. While importing these are deleted again and vacuum is done afterwards (This is a problem if storage is short at device!).

With an installed DC no blobs are stored in database. Files are stored in blob directory as files!
The vacuum process in the moment is only started after importing a backup file. While normal operation of DC this is still not available for a user (May come in future versions).

My experience up to now shows that most storage space is occupied by blob files outside of the database. In fact, vacuum could help to reduce occupied space of the database itself, but blob files are feeling to me more important to delete them in case of a “delete process” in DC. I don’t know if this is done by DC in current version.

My experience is still based at v0.20.0 (!)

@r10s - I hope this is correct ?

just a small note: the upcoming delta chat version will delete blobs belonging to deleted chats :slight_smile:

2 Likes

First I have to discourage anyone to use the feature of Android to integrate the SD-card into internal storage. As experience tells me that construction will fail sooner or later & you’re struggeling to recover the encrypted data if you don’t have a backup.

Secondly I don’t get why nearly all apps don’t allow the user to move it’s database to the SD-card. All the time I get access to other peoples devices I see they are nearly out of internal storage yet their external SD-card has tons of space left. Why not warn the user about security concerns and then let him decide for himself?
Manyverse, a client app for the decentralised social network Scuttlebutt, has to deal with potentially very big local storage demand and already supports putting it’s database on the external SD-card.

So I’d really appreciate Delta Chat giving the option to store data on the external SD-card!

2 Likes

DC already has external storage - IMAP server(that can hold practically unlimited amount of imformation), so IMHO it isn’t necessary to keep all emails locally on device. As I already mentioned here, DC-mobile should keep defined number of messages locally on device base on (future?) preference 100/250/500/all as it done in K9. If user need old messages, one can pull it from IMAP in the same way as it works in K9 - by pulling down message list down.

While text won’t take a lot of space and can be kept in sqlite database locally, it isn’t a case for blobs that takes a lot of space and early or later everyone will step on this issue. IMO DC mobile apps should keep locally on device only cached blobs base on some settings - “how long to keep blobs” locally: last 100/250/500/ALL messages and allow to pull old blobs from IMAP if needed when user scrolling to old messages.

some people have only 100MB of space in the email server and want to keep the messages in their phone, anyway not sure if there are someone interested in keeping messages older than X months

Judging by my XMPP client usage it’d keep all messages on the phone. I often search through my XMPP database on the phone for messages several years old.

dude live the present!!!
(hahaha just joking)

1 Like

All businesses we working with, requires us to keep absolutely all emails on the servers. If DC would be used in such environments it will choke at some point for sure.

1 Like

we are not talking about force the user to delete messages, just to give the option to do so, you have the need to keep the messages but we have the need to delete them, you will not need to do nothing the default could be to keep them forever :slight_smile:

1 Like

And fill all device’s space at some point, right?

I still can’t get it, why not to use K9 model of keeping messages that proved by time.
It really simple, - keep on device the only number of last messages user wants, for example 100/250/500/all and keep rest of messages on the IMAP server that can be pulled on demand if needed.
Messages that deleted on device should be deleted also from a server. Simple.
This way, those who has limited space on IMAP server can manage what to keep and what can be deleted. In the same time those who has “normal” or huge amount of space on the IMAP server, they can keep messages without killing mobile device by filling all its space (which is very easy to achieve with attachments) and be able in the same time get old messages if needed.

DC isn’t snapchat to exchange with kitten once and forget about it, but a communication program that utilized email protocol that assumed to keep message forever if it needed.
I already has two correspondents(a lawyers who exchanged a lot with attached documents) who stopped using DC because of this issue and early or later every1 will step on that if it wouldn’t be resolved.

1 Like

AlexJ I was talking about to keep them forever in the server by default, but let people with small server storage opt-out, maybe you have almost infinite storage, but some people have only 100MB :stuck_out_tongue:

actually I think that would be a nice (optional) feature for Delta Chat

According to the title of this topic, - the issue is that DC keeps locally, on the phone ALL messages including blobs and fill out device’s space.
I’m talking about majority of users who has more than 100Mb on IMAP server and proposed solution how it can be resolved, - the same way as it done in K9.

All phones I know allows for example to set preference - how many text messages to keep, the same option need to be implemented in DC to avoid putting smartphone to the knee.

Actually it should be more flexible to satisfy people like you who has limited space on the server and those who can at least use gmail or their “friends”.

I see it as an options:

  1. Keep locally on device only last: 100/250/500/All cached messages and allow to pull older messages from IMAP server on demand
  2. Automatically delete messages that older than: month/year/none(keep all)
  3. If option #1 is set to “all”, then unhide option: Delete old messages if space occupied by DC exceed 0.1/1/2/5/10/9000Gb

I think it will satisfy all DC users, those who has plenty of space on IMAP server and those who has limited amount of space and what is most important phones won’t be overfilled by DC in all cases.

3 Likes

Properly solving the message history and storage usage (devices/server of any sizes) for all use-cases, while keeping it to a minimal set of user-visible options, isn’t trivial, but shouldn’t be too hard either.

IIRC the required user-facing items for the full solution are only three:

    1. Additional message states: starred (never deleted, manual selection) and archived (usually remain only on device, automatic expiration). (The other (recent) messages get synced with the server and across devices. NOTE: The “archived messages” are different from the old “archived chats” which are only hidden/suspended from the chat list. The “archived messages” stay visible in the chat histories, are not synced with the server (and thus other clients) anymore, and default to remain only on the device.)
    1. Max. number of messages kept on server, and thus synced accross devices (per chat), default value 250? (The client with the smallest setting usually determines the amount of messages kept on the server.)
      • Default true: delete exceeding messages from server
        Disabling this suboption
        i) in conjuction with reducing max. messages kept on server/synced to 0, configures a device (with enough storage) to keep all incoming messages , while other devices may still sync more and even delete a part of them before they expire on the server.
        ii) in conjunction with a smaller number kept on server and synced, and possibly a small archive limit, configures a device to sync less messages than kept on server, i.e. for low-storage devices or huge-storage servers.
    1. Local (archiving of) messages. Keep messages that expired on server on local device? Default enabled/disabled? (enabled is safe if auto-delete-device is implemented)
      Suboptions:
      • Default disabled: Limit number of (synced + archived) messages (per chat), to limit the number of messages kept on the device after they expire from the synced messages.

      • Alternative option that will also remove low-traffic chats: Default disabled: Limit age of (synced + archived) messages (per chat)

Required background mechanisms:

  • auto-delete-server
    • always guaranteeing enough free (quota) space on the server for continuous operation (and only deleting the last (synced-new) messages of every chat if no other option remains.)
    • limits the max. number of messages kept on the server
  • auto-delete-device
    • guarantees free space on the device (maybe auto-deleting from the synced rather than archived? (and first reducing the the local email inbox list?), or offering to delete/backup archived chats if necessary)
    • optionally limits the local history if configured

A full current design:
(It has some more details like considering the latest messages of each chat (state synced-new) more important, to prevent the auto-archive and -delete from completely deleting low-traffic chats etc.)

I optimized the config options above.

In current 0.200.0 there is the housekeeping function at backup export which deletes unused files.
this works well (imho)
beside of that there should be an automatic or manual function to do this job separately from backup.

Would it make more sense not to use a per chat value here, but the total messages on the server?

1 Like

One detail of the above background mechanisms would actually make deltachat work without requiring manual adaption of the default number of max. messages kept on the server/device (out of the box) also with small mailboxes/storage:

when the free space falls below say 6% or 110MB, notify the user that DC will soon (below 5% or 100MB) have to start automatically deleting the oldest chat messages on the server/device

2 Likes