Bug / feature request: making sure the message are displied in the correct order even if some take longer to arrive

Feature Proposal
1

i have been testing out delta chat for two days… i really like the concept because i always tried to convince other to use GPG for email, but it generally turned out to much to ask. and delta chat solve this problem… And i love the idea of using email for messaging…
but unfortunately within one day of use i have to conclude that its not really “usable” jet (for my use case). And i specially signed up here to write about this “bug”.

background

im using the latest version on android and my chat partner also… we live half a world apart, with my chat partner living i a very strict government controlled country.

We started testing all the general features like sending pics, videos, audio recording… And everything seemed to work find… but after testing and having fun, finding that everything works, we really started to use it for conversations, that’s where problems start.

the problem

each message is sent by email, but not all arrive in a timely manner. about 70% of the messages arrive on time (keeping the normal flow of conversation) about 1% don’t arrive (but give no warring or any thing, we only found out by comparing screen shots) and the rest of the messages come in random times sometimes with a 5 min delay sometimes with an delay of up to an hour or so…

but the biggest problem isn’t even the delays, but that the messages are displayed in order off arriving, which makes the contestation look like a random mess. we really kept trying but when our conversations starting to end in small fights on what was and wasn’t said we finally switched back to our old messenger.

Expected behavior

obviously the expected behaviour would be that even if messages arrive late, they still get placed on the right spot in the conversations! so that when you read back the chat, it all at least makes sense.

How to implement?

of course this is up to the makers, but my guess would be to introduce the concept of a “conversation message counter” i guess that if their has been radio silents between 2 chatters for more then lets say 3 hours, the conversation counter can be resets (if reset is needed)… then when you start to chat, each text does +1 on the counter. and every messages sent knows its own count. so when my client receives message 3 and 5 but 4 has not arrived yet, it can already place a orange place holder text cloud that says that it still waiting on a delayed message. it would be great if the app knew where in the conversation to place the missing text, so if ever message didn’t only contain its own count but also the time stamps of the last 3 messages. it should in general be able to keep in tact the natural flow of the conversation, and exactly know where to place each text even if it didn’t arrive jet… and then their could also be a timeout period, like if a orange message still didn’t arrive after 1 hour it becomes red… or maybe even create a mechanism that waits 5 minutes and then negotiates with the senders client to automaticity resent the delate message using the count as message count or maybe their can be a longer hexdec message ID and use that instead of the message count. and then of course if the message was replace by resending, and then later the old messages still arrives, the message gets discarded. but the time it arrived late could be logged… by keeping statistics on delayed messages on how much to late they arrived, the users can know be informed of their “bad connection”…

closing thoughts

anyway, the current lack of “conversation flow fidelity”, makes it only useful for sending a few messages, but surly not for chatting… which is really a petty, because it’s otherwise a really nice messenger… but their are ways to fix it. so i hope this gets some priority, because this pertains the the general usability of the app…
let me know if you find this a reasonable request, or if see an other solution to solve this problem…

2

I have been using delta chat for several months now and I must say that I have never encountered this kind of problem. In my experience messages are usually sent and received almost instantaneously, at the most with a delay of a few seconds, not certainly with a 5 minute delay or even an hour. I think that more than a problem of delta chat itself may be a problem of the mail provider, but I’m not sure.

3

thank you for your response on my first thread here… i would like to say: yes & no… maybe i did not explain the problem clear enough, so i like to add some extra info here…

i understand, not everyone will have these problems… but i like to not focus on the result (whether or not emails come in a timely manner for most), but on the underlying mechanism used…
this is a chat app build on-top of existing email infrastructure. and want to use it in a way for which it was not intended. so its nice to be creative and find new way of reusing what we already have. but of course need to predict and prevent the problems that might accrue from it.

my bug report / feature request, is not about complaining that it doesn’t work for me, but that i notice that the app doesn’t play nice when there are delivery delays (i mean resulting in partially unintelligible conversations). and that i think their is a solution to solve this problem. the crux of the matter is that we know that there is no way to guaranty everyone’s email will always come in a timely chronological order.

the basic truth is that 99% of the people in the world don’t run/own/control their own mail servers (and that is probably a good thing). but everywhere in the world server owners/maintainers & governmental firewalls might have set up routing rules for all email passing through… some might start adding delays if you sent more then an x amount of mails per 1, 10 or 60 minutes. some governments (and maybe even some companies who make their money from doing something with the information they gather from scanning your emails). are actively trying to discourage usage of encryption, but not by completely blocking it, but by trying to find ways to make it less usable. some providers might have very stringed spam-filters that might cause some problems. (the problem of drop emails can also be fixed with the same solution)

to summarize it, it would be better to not just assume that the email of everyone will always just arrive in a timely manner. especial sins this app likes to be a goto for people who live in the kind of country that interfears with email sending. then allowing them to break the app’s functionality by adding delays for encrypted emails shouldn’t be something we would want, if its possible to prevent it.

i understand also that as long as the app is build onto of email infrastructure their will be no way to give a technical guarantee on timely arrival of emails… but what it can do is make sure then when it happens it doesn’t result in miscommunication between the users, because both parties are clueless on what the original conversation flow was…
receiving the texts (sometimes) in a non-chronological order, is not a big problem as long as the app is smart enough to keep the flow of the conversation in tact…

1 Like
4

There is already a Date header on each message which contains the timestamp of the message set by the sender. This is enough to order the messages correctly. These are the dates that are displayed in the corner of message bubbles, so you can reconstruct the correct order with them. The reason Delta Chat does not do this for you is because of the rule that messages are always added below the last read message. This ensures you will notice them when you open the chat and start reading from the last read message. To solve your problem it is enough to make this rule less strict, i.e. allow to add new messages above the last read messages, but not way too much above: only reorder within 5 minutes, for example.

Detecting lost messages and resending them is a different problem. It might be that no new headers and counters are needed to do it too, because all messages contain a References header which refer to the last Message-ID seen by the sender in this chat. If you didn’t see this Message-ID, it means you have not received some message and Delta Chat could report it after some timeout.

5

@link2xt: thank you for your explanation…

i was thinking these 2 thinks where so similar that there might be one solution to fix both problems. but now you explain that it are 2 different problems, then i like to focus on my original problem.

but before doing so, i have one curiosity about the “reference header Message-ID”… in my estimation that “detecting (all) lost messages” problem could only be solved with the message-ID is if the message-ID had some sort of +1 count in them… and if the clients keep an internal record of the ID sent out so (during that session)… so that if the receiving client “sees” that the ID in the reference header doesn’t match with the last sent out ID, then it can use both IDs to calculate how many, and which messages where missing… with out that info requesting the resent of a particular message wouldn’t be possible i guess.
so my question on that subject is, is it true the there is some sort of synchronicity that allows the clients to calculate how many messages where missed if any? and does the clients keep internal record on sent message ID together with sent timestamp… so that in case of resending the lost messages, it can include the original timestamp?

so now back to my main problem of displaying the correct flow of the conversation… so i see now that their is a conflict of interest between keeping clear track of new(unseen) messages and displaying the messages where they fit in the conversation… so if the current behaviour (displaying base on moment of receiving) stays the default behaviour, at least their should be an option or button to allow rearranging the conversation into the correct order of sending…
maybe their can me a mix where all new messages are displayed at the button, but if needed, by clicking a flow-sync button, you can, after reading it and not able to figure out where in the conversation that message fits, the text rearrange its self based on sent order, but then new messages still arrive from at the button… that way only people who and when having this problem can click the button as much as they like, but others might not have to touch it. (so i guess that should be a relatively easy fix, that wouldn’t demand drastic changes to implement it?)

i didn’t think of the timestamp, but ya that might work… the problem though is that to display the messages in the correct order of the conversation flow, relying on time means both parties need to have their clocks synced. otherwise you still and up with a textual mess. so that will introduce a whole set of new problems, like how to check, and sync both parties their clock. (but ya, if their could be a reliable way to confirm both clocks are in sync that would be nice)

that’s also where the idea, of some sort of count together with the timestamp of the last 3 sent messages, came from… but this was mostly based on wanting to be able to place the text bubbles in the correct spot, even if they didn’t arrive jet. but maybe their are other ways to solve that problem. btw, i choice for 3 here, because i noticed that not all emails arrive late, usually at the least ever 4th or so messages i get does actually comes “instantly”. so this way as long as not more the 3 messages are delayed, the app will still be able to place the text bubbles place-holders with correct time.

  • does the clients already does some sort of check to confirm the system time matches that of a time server or something like that? or would it be possible to implement that?
  • is it possible to compute the message-IDs of multiple lost/delayed messages base on reference-ID and the clients last know sent message-ID? (is their some sort of count in the ID or are it random generated strings)
  • does the reference header also include a timestamp or is it only the messages-ID?
  • any change of getting such a “flow-sync button” based on a computable message-ID and timestamps?
6

Most operating systems today synchronize clocks over NTP. I don’t remember having to synchronize the clock manually anytime recently.

7

true, on most, but its an option that can be turned on and off… in older versions of CyanogenMod i had to manually install an NTP sync app, on later versions you could turn it on, and now its on by default but you can turn it off… but i recently installed CalyxOS and if i remember correctly it was not enabled by default (but not 100% sure)
but people could have been playing around with their setting and forgot to turn it back on… some people might be so paranoid they after a fresh install they close anything that makes contact with online stuff… and then later don’t realize the consequence when installing delta chat.

my main point is, that when wanting to create a smooth user experience, its not good to just assume that all external factors are just working properly… so if their will be a new feature that allows to reorder the messages base on them sent time stamps, then it would be good to at the very least, inform users if their clock is more out of sync then an x amount of seconds with a ntp… or maybe it can check if the ntp auto-sync option is enabled in the OS, and asking the user to enable it…