Rich message formatting [render HTML email inline]

Feature Proposal
1

I’ve seen the threads "Markdown" support in chat [bold, italics, underlined, strikethrough, inline code, labeled links, delimited links] and Basic Markdown support when writing messages [blockquote, bold, italics] , but I think Markdown is not the right way to go (see below).

With Telegram’s recent push for richly formatted messages ( Rich Text Editor, Communities, Ephemeral Messages in Groups, 350 Million GIFs ), I think things will move more and more towards richer formatting options. DeltaChat messages being emails means it is, in my opinion, in a uniquely good position for great formatting. While it should not support arbitrary HTML, I think HTML should be utilized to allow formatting of messages. We can then extend which tags and formatting options are actually allowed bit by bit, while simply printing not-allowed tags as text, which gives us a kind of fallback.

Proposal

Messages sent as text/html support a subset of HTML. I’ve taken inspiration from how Telegram is doing it, because I think it’s always good to try finding a common way to do things if there’s no reason against it. On the other hand, I think in the beginning we should focus on a small subset, for example:

  • <ul><li></li></ul>
  • <h1>, <h2>, <h3>
  • <b>bold text</b>, <strong>bold text</strong>
  • <i>italic text</i>, <em>italic text</em>
  • <u>underlined text</u>
  • <s>strikethrough text</s>, <strike>strikethrough text</strike>, <del>strikethrough text</del>
  • <code>inline fixed-width code</code>
  • <blockquote>

Future options

In the future this could be extended by a selection of additional tags, e.g.:

  • spoilers
  • tables
  • math/LaTeX
  • links
  • possibly images
  • buttons — I think this one would be interesting, as it could be a clean solution for bot buttons. We could allow adding a maximum of, let’s say, 5 buttons to a message, each with a data-event attribute that defines which data is sent back when the button is pressed. But this is something that needs to be discussed separately.

Why no Markdown

I think, as stated by others in the threads linked above, Markdown is not a good candidate as a transport format, because it is arbitrary and complicated (some Markdown standards even allow inline HTML). We’d also need a clear way of knowing when something should be parsed as Markdown and when it should not — there’s nothing more annoying than actual text being misinterpreted as Markdown. So in my opinion, Markdown should, if at all, be a feature of the sending client, which interprets the Markdown and converts it to proper HTML.

Sure, if we say we only support the very basic Markdown features like bold and italic, Markdown might be enough, but in my opinion this is not future-proof and opens the door to a lot of backwards-compatibility issues later down the road.

Feedback & contributing

I’d love to hear your opinions. Since I’m new to DeltaChat, I’d also like to ask how decisions for features like this are made and how contributing works. If I find the time, I’d be open to attempting a first implementation — assuming AI-assisted work is allowed. I’m a developer, so I won’t throw slop at you, but my experience is more on the backend side and I don’t have the time to familiarize myself with the whole project manually.

AI-Disclosure: Written by hand, in the end used Claude Opus to correct some of my English mistakes.

2

Related:

3

I have written a simple Proof of Concept which can be seen here: GitHub - JoshuaDietz/deltachat-desktop at html-messages · GitHub Note that the code is NOT yet ready for review, I just wanted to get some feedback on the feature itself first. For the first version I’ve decided to go with a simple approach: When someone enters one of the supported tags, an icon appears where he can toggle between the tags being interpreted as html (new behaviour) or as normal text (old behaviour). Unsupported tags are always shown as text.

image
image868×850 80.6 KB

image
image434×548 27.2 KB

image
image1998×378 91.7 KB

Feedback would be appreciated!

Since this is the first feature I’m working on in this project, I wanted to ask: Is this something that is likely to have a change of being merged? Can I continue work on it? What is missing? Do I have to follow a process?

In the future I can also imagine doing some more iterations on it, the ideal solution would probably be a WYSIWYG Feature. Markdown input support (converted to html before send) could also be interesting. But before that, I’d like to have the foundation merged and it would also require some more discussions on how to make this UX-wise.

4

Interpreting HTML tags within text is an unconventional approach.

What was on my mind is that Delta Chat could have a setting for sending out messages as an HTML MIME part if it contains formatting. Composing it with WYSIWYG or a Markdown widget would be reasonable.

Delta Chat should also support rendering of received HTML messages inline, possibly by framing it with a scrollable widget if it is too long. If we are afraid of classic HTML messages looking bad like this, we could only show messages with Chat-Version this way.

5

I very briefly looked at the code. I agree that setting HTML and using getMessageHtml (get_message_html on the Rust side) is roughly the way rich formatting should be in email-based messenger. Internally it should be text/html MIME parts. See also e.g. comments on Render markdown in messages by darichey · Pull Request #6109 · deltachat/deltachat-desktop · GitHub, also there was an attempt to write a single parser at GitHub - deltachat/message-parser: Parsing of Links, Email adresses, simple text formatting (markdown subset), user mentions, hashtags and more in DeltaChat messages. · GitHub that only made it to desktop and was removed eventually, and ArcaneChat has another incompatible markdown support.

Largest problem I see with implementing it is compatibility between clients. If you only implement it on desktop, then you send HTML and mobile clients will show “show full message” link just to look at some bold/italic text. Could actually be fine if users don’t use it all the time.

Overall the approach of making it possible to set HTML on the sender and recognizing if HTML is simple/safe enough to be displayed inline on the receiver looks good.

For the UI, I am not an expert, but got the impression e.g from Render markdown in messages by darichey · Pull Request #6109 · deltachat/deltachat-desktop · GitHub that we likely want basic “select, see inline markup popup, click on a button to mark selected text as bold/italic” controls first rather than expert controls to format posts with headings and code blocks. Lightweight markup maybe later. Advanced stuff like tables likely never, and definitely not users typing HTML or BBCode manually, it is only ok for developers. When in doubt copy Signal (it has only inline markup), WhatsApp, Telegram, Discord in this order, with the basic features first.

Desktop has some history of having “experimental features”, so possibly if desktop developers agree. I also think we want this feature eventually because other messengers have it, but it is not on any roadmap yet and implementing it as non-experimental will likely require some coordination so everyone agrees to implement it on all platforms at once. Otherwise some users will frequently send marked up messages and some users (e.g. iOS or Ubuntu Touch) will see “Show full message”.

Here is an example of inline markup controls in Signal desktop:

image
image260×130 4.88 KB

6

Wouldn’t it be better to use Markdown for formatted messages? It’s simpler, and the text is better readable if a client doesn’t support it. In my opinion, it wouldn’t make sense to support Markdown fully—only the most important features.

7

I think this is addressed in the first post and in my post above already.

As for the text being readable in clients that don’t support HTML, if you send “foo bar baz” in the UI, on the wire a multipart/alternative message is sent with a text part that can say foo **bar** baz and HTML part saying foo <b>bar</b> baz, but there is no need to interpret the text part and know that it is markdown as a receiver.

8

I would like to share the cautionary tale of the Markdown Monster:

This document is supposed to demonstrate just how bloated and riddled with edge cases Markdown is. It can distinguish 24 different Markdown implementations. Unfortunately, all the Rust implementations of Markdown were broken, so they are not included.

9

(FTR I am not sure if HTML instead of Markdown is the way to go)

I don’t know if my personal agreement is enough, but I am alright with experiments.

Though the question would be what’s the point of this experiment / feature. We know it’s possible to render HTML (safely).
Making users manually write markup doesn’t sound good.
Is it only about adding a toggle to render HTML messages as text?

10

Note that before the removal of the Markdown experiment we used to have the code to render those elements, you may just reuse that.

11

Thank you all for your feedback. Trying to address everything:

Since I got carried away a little on that one, first a TL;DR; of my main point:

I think we need to approach this in two steps:

  1. Standardize which tags to support, how they will look like when being rendered and establish support for them in as many clients as possible. In this phase the feature is deliberately hidden, i.e. by only being usable if you enter html into the input - but I’m competely open to other approaches like first having to be enabled in the settings. The point is that the rendering side has to be there first before everyone uses the feature. This is the point of my current proof of concept.
  2. A while later, start to roll out an easy-to-use editor that makes these features accessible. Probably WYSIWYG (I really like the signal-example that @link2xt posted) but we could also get with a markdown input (but no markdown transmission, arguments see below). At this point, most clients already support rendering these messages and for the ones that do not, we don’t loose too much.

Here’s my detailed answer:

I’m unsure what you mean by that. To my understanding of what you mean, this is exactly what I’m doing here: If a message arrives as an html message, we render it as html with the deliberate limitation to only support some tags and not the full feature set of html. I think this is the way to go because in the end this should primarily be a chat app that works well on all kinds of devices, especially on phones and not a generic mail client that supports every html feature. If we would go the full html route, this would come with all kinds of problems, i.e. broken layouts, the necessity to allow a zoom feature, complications with other common features (i.e. how would one implement themes if messages can style themselves), text that can be hidden but can come along when copying a message etc etc. That’s why I think limiting it to a subset that all clients can render in a clearly defined way while still sticking with the standard that we use html like emails already do is the way to go.

I think always sending as html when it contains formatting is not the right approach, containing formatting should not be an automatism for sending it as html. But I think with the introduction of WYSIWYG this question will solve itself because by using the WYSIWYG features, the user already communicated his desire to send something with formatting clearly, so this would just be a temporary problem anyway. More on WYSIWYG below.

For the next one it makes sense to batch a few together:

My point of having no UI for now and no WYSIWYG is exactly the compatibility one: We need to FIRST have a clear and standard way how the transmitted data will look like and how it is being rendered. This standard should find its way into all clients BEFORE we start rolling out WYSIWYG features, that make the feature easily accessible. So in the beginning, the feature would only be available for people who already know that it is there and how to use it (i.e. by typing tags into the input) while clients have time to catch up. This will address the compatibility concern. Note that I am also willing to help with other clients than desktop - but I’m not experienced with mobile development.

So to sum it up: The point of the feature is to first establish a standard base-level (meaning a few allowed html tags and how they will be presented visually) of compatibility in as many clients as possible.

On this one my opinion is pretty strong tbh: No. Markdown is a terrible format for transporting formatted emails. AFAIK it is not standard as an email transport format. The Markdown-Monster post from ian is part of the reason why I think so. The other part is extensibility. Allowing markdown automatically means restricting the project to the limited set of features that are possible with markdown. While I do think that these features will be plenty for now, I also think that they will not be sufficient in the future. The trend of allowing more and more formatting in messages is pretty clear and in times of bots it will continue. Telegram recently implemented tables, image galleries and more (see my first post). I don’t think that this has to be the goal for now, but shutting the door on the option would be a big mistake in my opinion.

Another problem that I have with markdown is that you can not use it “partly”. When your message opted into markdown, you can suddenly not start a line with a hyphen anymore because it WILL be interpreted as a bullet point. It happens so often that stuff it randomly interpreted as markdown when it is none, it’s annoying.

What I am not opposing at all is to at some point allow markdown to be an input format where the client converts the markdown to a standard set of html before sending. But clients interpreting incoming text that looks like markdown as markdown would be arbitrary and has many options to confuse & annoy users in my opinion.

While I think backward-compatibility is important, I also think that settling on an inferior format to have a slightly better experience in outdated clients is not the right way to go. Sure, unrendered markdown looks better than unrendered html. But especially with the gradual rollout approach described above, the problem should not be too common while even if it occurs the message will still arrive and somewhat be readable. As a contrast I just remembered how Telegram is doing it: They constantly roll out new message types. When your client receives an unknown message, it does not even attempt to render it but instead just displays a message saying that the client is outdated and that it cannot show the message which is a pretty breaking change. So showing a few html tags is imho a compromise worth taking.

While I also don’t see the necessity right now I think it’s important to have a foundation that allows it in the future. I think that features like advanced formatting are especially beneficial for niche projects like delta. I still remember how Telegram got common and how I learned about it: in University. Everyone was using it for their homework because it allowed sending even big files in a super convenient way. Then as people used it more, they extended it to their own networks. I see a similar path for delta: Having features (beyond the security/privacy/decentralization ones - too many people do not care :confused: ) that make delta more attractive for a certain group of users - i.e. being able to have discussions that (at some point in the future) include nicely formatted formulas, tables and more. Then when this group of people is used to Delta and likes it, they will invite more and more of their social networks to also try it out. I hope it doesn’t sound too much like I’m just a dreamer but I would really like to have a world in which people would use something open like Delta for their personal communication instead of relying on cooperations that can change the rules at any time. And for that to happen I think we do need the flexibility to introduce the features that almost no one has (obviously without bloating the whole project)

Thanks for taking the time to read this novel :blush:

12

I think the most advanced implementation of rich message formatting today for a messaging app can be claimed by Matrix, so it might be considered a kind of upper bound of what would be worthwhile to support on the long term:

suggested set of HTML tags to permit, denying the use and rendering of anything else, is: del, h1, h2, h3, h4, h5, h6, blockquote, p, a, ul, ol, sup, sub, li, b, i, u, strong, em, s, code, hr, br, div, table, thead, tbody, tr, th, td, caption, pre, span, img, details, summary.

Tag Permitted Attributes
span data-mx-bg-color, data-mx-color, data-mx-spoiler (see spoiler messages), data-mx-maths (see mathematical messages)
a target, href (provided the value is not relative and has a scheme matching one of: https, http, ftp, mailto, magnet)
img width, height, alt, title, src (provided it is a Matrix Content (mxc://) URI)
ol start
code class (only classes which start with language- for syntax highlighting)
div data-mx-maths (see mathematical messages)

Some of the details would need to be adapted, such as by extending the supported schemes. We can ignore the MathML-related attributes for now. I’m not sure whether attribute-based spoilers are a good idea and I would just accept summary & details for this use case (could also be substituted with a checkbox & label given sufficiently advanced CSS selectors, but no CSS is supported so far). To ease rendering, I’d probably disallow arbitrary nesting, though.

Embedding images attached as MIME parts by ID in the href could make sense similarly to how it is done in HTML email (and again, this is why it makes more sense to just use text/html with MIME multipart in the first place).

It seems odd how they have chosen to throw away the legacy standard of <font color that was supported by previous versions of the specs or a CSS-based alternative (even if specified inline in the style attribute).

Additionally, something not even Matrix supports, but I find useful would be to distinguish between ASCII art that must be shown with a pan & zoom widget and source code that must be wrapped so it doesn’t scroll horizontally (with a smart parser, even up to keeping the indent level). As a crude approximation, I usually add white-space:pre-wrap CSS to the latter.

13

Nice, I did not know that Matrix had such sophisticated fromatting.

Regaring nesting: I would not completely disregard it, i.e. bold within a list item should be allowed and possibly even nesting list items is an acceptable usecase. But I think when going with a subset of matrix has (at least in the beginning) nesting will not be a problem anyway as there is not much to nest..
I also like the img and a features.

While I’m unsure if we should standardize a bigger set of supported tags in the beginning it might make sense to at least have decided on a bigger set from the start on because getting all clients synced again and again as features are being added will probably be more work than being able to communicate a bigger set of supported tags in one go (and then clients can still decide to i.e. not implement all tags in the beginning and possibly warn the user instead that the message contains markup that the client can not fully render).

14

Sure, combining simple unstructured inline markup is easy. However, rendering becomes error prone between structural elements, less accessible and less responsive if you support lists nested in lists, lists nested in tables, tables nested in lists nested in tables, etc.

gemini’s gemtext lacks support for nested lists for a similar reason.

15

WYSIWYG Editor Proof of concept finished

I finally found the time to build a proof of concept for the WYSIWYG editor how I imagine it. In the comparison you can see the changes excluding the html message changes from before. As before - this code is NOT ready for review yet, but the idea is :slight_smile:

Comparison: Comparing html-messages...html-wysiwyg-editor · JoshuaDietz/deltachat-desktop · GitHub

Features:

Headings, bold, italic, strikethrough, underline, lists, codeblocks, quotes, Fullscreen editing mode, Markdown mode.

Since not everyone has the time to clone my branch and run it himself, here’s a little video where I’ve tried to record all features:

I’ve tried to build the UI in such a way that it makes sense for mobile as well.

As I said before, I think the right approach is to first finalize and merge the rendering part and only when clients had some time to catch up, add the WYSIWYG editor.

You can also see in the recording that there still seems to be a little bug when I try to exit code mode when I type `Esc`, so it’s not perfect yet and there’s definitely still work to do - especially in code cleanup.

As always: happy to hear your opinions!

16

I’m not sure if you’re implying that you want to merge this into the desktop repo? I doubt that we want to maintain 5,000 LoC WYSIWYG editor. There is plenty of existing implementations.

17

As I said, for now this is just a proof of concept. If we reuse an existing editor doesn’t matter to me, my goal is to have a easy to use feature with great UX that can actually compete with other messengers. But in general I think the burden of maintaining an own solution that is tailored to the own needs is nowadays often less work then bending a general solution to the own needs. Especially when you want to have the same feature on multiple platforms. So while I do not think that this code should be merged (definitely not in the current condition), I do think that probably we’re better off with a custom solution.