IMAP strategy to chat within the email ecosystem

1

This publicly editable “wiki” post exists to flesh out the programming (coding) features that allow reliable, non-disturbing chat communication through IMAP email server accounts that are shared by multiple chat and email clients (parallel MUAs), while using as few battery as possible.

The features can be implemented in stages.
Corresponding (old) issue: https://github.com/deltachat/deltachat-core/issues/117

Requirements fulfilled by this design

Spam protection for chat messages.

Syncing of chat messages between multiple devices.

Allow multiple chat and email clients sharing the same email (IMAP) account at the same time (parallel MUAs).

Working even with IMAP servers that do not provide any of the optionally supported features (backwards compatibility):

  • IMAP-idle/push notifications (PUSH)
  • notifications for multiple folders using IMAP-idle/push or notify (2xPUSH)
  • server based message filtering (SRV_FILTER)
  • configration interface for MUAs to manage server based message filtering (SIEVE)

Basic approaches

Determine the “email-approved” contacts (to which the user has actively sent an email,
and to whom those have send emails) by checking non-exempt folders (aside the watched INBOX) for updates only once-a-day (in the early morning). And only (ask to) open incoming email messages automatically in new chats for these known contacts, if the message body text is short (likely no email).

By default, all clients sync all (INBOX) message updates seen on the server (i.e. additions and deletes) locally, and “upload” all local changes (i.e. deletes) to the server. To allow syncing multiple clients, there are some additional conditions.

  1. (also see AltMarkMove scheme below) To allow syncing multiple clients if only one (the INBOX) folder can be watched, before really locally deleting a message that was deleted from INBOX, wait a second and check if the message has not only been moved to the email chat subfolder by another client (means chat message has been displayed or email message has gotten a chat answer).
  2. To still allow immediate propagation of a deletion in the email chat folder to all other clients, move deleted messages (IDs) back to the INBOX before deleting the message.
  3. have an option (https://github.com/deltachat/deltachat-core/issues/164#issuecomment-417868402) for clients to mark individual messages as “archived message” kept locally (i.e. do not delete locally if deleted on server, and ask about also deleting on server if the “archived” message is still present on the server https://github.com/deltachat/deltachat-core/issues/120#issuecomment-417267057).

Make use of server features like 2xPUSH if supported and when available, but otherwise fall back to the alternative behavior that works as best as possible.

At least in data saving mode, only fetch the message list, getting bodyless (subject only) messages, and only download messages with very small text bodies (message size), or if the subject starts with Chat: or the symbolic -[]. (If the message bodies can not be downloaded without downloading attachments, Email-chat compliant applications should always send attachments in separate messages, to allow downloading pictures etc. only if on wifi, or upon request.)

Goals

  • Deltachat (DC) messages should cause a notification in DC as soon as possible.

    (Possible with PUSH and 2xPUSH.)

  • Multiple DC instances on the same account should sync instantly, and work together nicely.

    (Possible with 2xPUSH, or an alternative marking+moving (AltMarkMove) scheme for chat messages.)

  • DC and other MUAs on the same account should work nicely together.

    (Requires supporting standard locations and status flags.)

  • Incoming DC chat messages should by default not cause any notification in classic MUAs like K9, Thunderbird etc.

    (Requires SRV_FILTER.)

Limitations

If no server based filtering (SRV_FILTER) can be configured (automatically, by an admin, or the user), DCs email strategy (alone) can not prevent in all cases that incoming chat messages may trigger disturbing (and duplicate) notifications in classic email clients as well. (It requires the user to either configure all classic email clients to ignore the chat messages, or more simply, just to disable all notifications in the classic email clients and to configure deltachat to also emit notifications for all classic new emails in the INBOX folder.)

Principles

The alternative marking+moving (AltMarkMove) scheme allows all the communication to work with a single-folder (INBOX) push-connection.

  • Server filtering or the chat client immediately marks new chat messages as “seen”, to reduce disturbing classic email clients, while they remain in the INBOX to remain accessible in all MUAs together with regular emails.
    (To fully avoid double notifications for incoming chat messages in all cases (parallel install of a IMAP-push MUA and deltachat), it may still be necessary to disable notifications for new emails in the INBOX folder by the MUA, and enable it in deltachat.)
  • Chat clients notify about and present these “seen” chat messages as new, nevertheless.)
  • Only after the chat messages have been shown to the user, are they moved into the email chat folder.

If watching of two folders is supported, the regular email compliant semantics can be used:

  • Server filtering or the chat client moves chat messages into the chat folder ASAP, to avoid notifications in email clients.
  • If a message is in the DC Folder it means the message was identified as chat message.
  • Chat messages which are marked SEEN have been shown to user with any MUA (including DC).

Items to implement in different areas

setup (first start after installation)

  • [x] If the folder DeltaChat does not exist, neither in the root nor under INBOX<separator>, try creating the folder in the root folder. If creation fails try prepending INBOX<separator>. (https://github.com/deltachat/deltachat-core/issues/47)

  • [ ] Allow the user to select the desired Email-Chat approval mode, which at the same time, also serves to switch to the proper use-case settings https://github.com/deltachat/deltachat-core/wiki/Use-Cases. (It may just be a link “Select Chat approval mode” in the setup wizard that opens an optional modal:)

Chat Approval Mode

Usually, only short email messages from your email-approved contacts, and those in 
your address book, are allowed to appear as new incoming chats. 

Delta Chat can check your emails on the server to automatically detect 
your email-approved contacts (i.e. to which you've sent an email yourself, 
and to whom you've seen those send emails to).

(Link: Edit list of folders to check and ignore)

   * Disable email-approved contacts. No checking of other folders on the server.
     (Whitelisting -- Strictly manual addition of chat partners.)

   * Check email folders to find new email-approved contacts.
     (Default -- Your email usage automatically approves contacts.)

   * **ALL** incoming emails shown in chats. No checking of other email folders.
     (Blacklisting -- You may only block individual contacts, and see spam.)

(The second option from above enables MsgSCAN described on this page. For further configuration consequences of the user’s selection see https://github.com/deltachat/deltachat-core/wiki/Use-Cases.)

  • [ ] Have a custom-imap-folder-ignore-list (for the user to let DC ignore e.g. folders that contain messages forwarded from a secondary address, concerning external topics (support etc.), or in server specific spam folders etc.), to allow preventing that addresses from there are added to the email-approved contacts. https://github.com/deltachat/deltachat-android/issues/239

  • [ ] (This setting will NOT be necessary anymore after implementing the (improved) archived message property (see overview at Reduce space on internal storage, disk full). Then, the recent history can get synced by default, while the archived messages would form independent local histories by default.) Allow the user to select whether to maintain a local chat message archive (message storage authority). (May just be a link to a “Local Chat Archive” modal:)

Local Chat History
Should Delta Chat maintain an independent local history for chat messages?

* No  -- (Default) Chat messages deleted on the server get also deleted on
         the local device. Allows to sync the history between devices.

* Yes -- Chat messages deleted on the server remain on the local device
         until deleted separately. Allows keeping a longer local history.

(re-)inits (at startup if network is available & on Connection state changes)

  • [ ] Determine the server’s capabilities to enable any corresponding deltachat support:

    • PUSH (disabled/false/true)
    • 2xPUSH (disabled,false,IDLE,NOTIFY)
    • SIEVE (disabled/false/true)

  • [ ] ToDo first (to quickly solve “email and chat working independently” by using server filters): If 2xPUSH is not supported (initially always), set AltMarkMove to true.

  • [ ] (For SRV_FILTER support.) Determine “Email-chat server filter rules”.
    Formulate chat filter conditions for:

    With AltMarkMove, let the filter actions only flag chat messages as “seen” (read/presented). (Prevents notifications popping up in classic MUAs about every chat line and receipt.)

    Otherwise, let the server filters move (deliver) chat messages directly into the chat folder.

    Let all chat message rules stop further processing, if they match.

  • [ ] (For SIEVE support.) If SIEVE is available in the server

    • Ensure the current “Email-chat server filter rules” block is inserted at the beginning of the active filter on the IMAP server (if necessary, move block to beginning).

  • [ ] (For SRV_FILTER support.) Otherwise (no SIEVE), if not done before, inform the user/admin, to manually configure the “Email-chat server filter rules” as the first filters, and stopping further processing if they match. The filters are required to prevent non-chat email clients like thunderbird or K-9 from getting flooded with chat messages and return receipts and to avoid that users get notified twice, not only by delta chat but also by the email client. (Also make this filtering info and the required rules available under advanced settings.)

  • [ ] If MsgSCAN is true and the OS is not in data saving mode, and no scan has been done before, do a full scan of all (not ignored) folders ONCE (see polling and scanning below), to build the initial list of email-approved contacts (part of “known contacts”). This needs to be a full scan (i.e. set last id to zero).

  • [ ] re-init push subscriptions:

push subscriptions

  • [ ] If 2xPUSH is not false or disabled, watch both the INBOX and the DeltaChat folder using the determined method.

  • [ ] Otherwise, and if PUSH is true,…

    • [x] …watch only the INBOX (Watching a single INBOX folder is supported by most servers, is most efficient, and still allows multiple clients to properly track current message states by using AltMarkMove)

message handling

  • [ ] Exempt the chat message processing if the header Chat-Version equals disabled. (Allows preventing DC clients to move the message into the chat folder (again, after using the…)

  • [ ] (May be avoidable, if sorting works well and long emails are only shown as icons in the chats.) …option to move long email replies that evolved from a chat back to the inbox, flagged as not yet seen. (Second general feature in https://github.com/deltachat/deltachat-core/wiki/Use-Cases)

  • [ ] ToDo first (to quickly solve “email and chat working independently” by using server filters): If a new chat message is detected in INBOX: If AltMarkMove is true, immediately mark chat messages as “seen”, …

    • [ ] … else (AltMarkMove is NOT true), immediately move the chat message into the chat folder. (Both actions are done as soon as possible, just in case that there are no server filters available, to minimize the issue that other MUAs notify upon finding the message in the INBOX.)

  • [ ] ToDo first (to quickly solve “email and chat working independently” by using server filters): With AltMarkMove, consider all chat messages in INBOX* as unread/not shown yet (i.e. notify and display as new, even if the “seen” flag is set), and consider only those messages as read that have already been moved into the chats folder. (Allows for notifications to work for new chat messages without disturbing email clients by configuring server filters. https://github.com/deltachat/deltachat-android/issues/353)

    • [ ] When chat clients see chat messages get deleted from INBOX, they may assume the message has been shown
    • [ ] To also allow for cross client deletions (obey server storage authority), wait a second and check whether a deleted chat message in INBOX has reappeared in the chat folder, or has been deleted for good.
    • [ ] Emails (non-chat messages) that are deleted on the server (with a classic email client) should always get deleted from deltachat’s “(…) New (incoming) Emails” (contact requests) as well. No separate local email archive!

  • [x] Notify the user about the new message.

  • [x] ? If a new message is shown to the user, set the “seen” flag (fallback in case it hasn’t been set yet, e.g. a manually copied message) and move the message into the chat folder (in case it hasn’t yet). (This legacy function should continue to catch messages that may have been manually copied / modified in the INBOX or chat folder, or when server filtering is missing or failed.)

  • [x] Adapt the always-BCC-to-self scheme https://github.com/deltachat/deltachat-core/issues/370. (Instead of: When sending messages AND AltMarkMove is true, upload sent chat messages into INBOX with the “seen” flag set. (It maybe needed to first upload to chat folder, set flag, and move into INBOX, to avoid the flag to be ever unset?) This enables other chat clients to pick up messages sent at other clients quickly through the push mechanism. Then concurrently move sent messages into the chat folder (again) after a 15 seconds timeout. And for garbage collection, also whenever a sent chat message with a timestamp more than 30 seconds ago is found. )

MsgSCAN (polling and scanning for old and new email-approved contacts)

  • [ ] [Even if all email messages are delivered into the watched INBOX (no server filters), it is still needed to check other folders to find all email-approved contacts. Otherwise, deltachat could never pick up addresses from older emails and would miss messages that have been moved into subfolders by other clients, e.g. during loss of connection, or have been uploaded directly (e.g. into the folder “Sent” or the corresponding contacts’ subfolder). It’s therefore needed to at least check important not-watched folders for changes frequently, and others at least occasionally. (Even in lack of IMAP-Push, the directory listing-only fetching should be a rather cheap query, especially after a large, static archive folder has been listed once, because it is only necessary to check if the last known state has changed.]

    Only while app is in foreground,

    establish an additional watch for the few important existing hardcoded folders (i.e. “Sent”). And do a poll to catch up with changes since the last check. If there are changes, look for new email-approved contacts in the new messages’ header data, e.g. to get classic-MUA-sent messages.

    And only if

    • MsgSCAN == true,

    • the device is not in data saving mode (e.g. free wifi instead of cellular network),

    • it is after 4AM

    • and no check has been done yet after 4AM today,
      (doing one early run in the day)

      then also check (poll) all other, not-ignored folders for changes since the last check. If there are changes, look for new email-approved contacts in the new messages’ header data.

      To allow disabling polling, the custom-folders-to-ignore string (https://github.com/deltachat/deltachat-android/issues/239) may support “*”, to disable checking of any custom subfolders (those not-hardcoded, i.e. don’t disable checking the “Sent” folder).

    (Or, with 2xPUSH == NOTIFY, maybe better create watches for the set of hardcoded (or all?) folders instead of polling, to pick up new email-approved contacts instantly and more efficiently?)

termination (network disconnection events)

  • [ ] If the network is lost or turned off, close any stalled network connection threads and stop all re-connection and scanning attempts, to save battery
  • [ ] (and related) On the sending side, delay sending jobs deterministically to ensure sending messages as soon as possible, but not to often (see https://github.com/deltachat/deltachat-android-ii/issues/130).
1 Like
2

Please make this a wiki page too.

1 Like