New Provider DB

some thoughts about the new data structure for the provider db I brainstormed together with @compl4xx

  • Message about limits (shown in the device chat)
    • maximum message size
  • 2 types of preperation steps to show users: pre-login and post-login
    • pre-login examples: Enabling IMAP/SMTP, getting an ‘app password’ for DeltaChat to use
    • post-login examples: change default inbox at the provider settings to prevent webde from moving to unknown folder, ?spam settings?
  • name
  • domains
  • credentials: emailPass or OAuth2 or AppPassword?
  • status: (most recent?)
    • OK, PREP, BROKEN
    • date
  • web-only
    • optional extra markdown file for tutorials on the web with embed images/videos? (maybe rather link to a webpage in device chat)
    • registration
      • identification requirements (phonenumber, passport, postal adress)
      • bussiness model: ad financed, paid, community funded, donations, government funded
    • benchmark results
      • average message transmission speed?
    • comments for results like on protondb (currated, moderated)
    • popularity (how often is it searched for in google? searchstat.com)

In the long run

  • Values for the core to use:
    • supports burner accounts api data/method
    • ?rate limiting (maybe show a counter how many message you can still send per day (sms like))
    • Connection data
      • own data - like force disable TLS for cuba.
      • Autoconfigure data from Tunderbirds autoconfigure db
5 Likes

nice overview. i think the connection data (port, server etc.) can be added in the short term already.

2 Likes

related github issues:
provider-db#42 provider-db#44 provider-db#45

1 Like

I’d add the database entry “maximum message size”.

i have to clarify my post from above: while i still think, it is a nice overview of possibilities, i think it is actually a bit too much we can really take care for. i think, esp. in the first time, we have to maintain the database, at least for the bigger providers, ourself.

i would reduce the database strictly to what we really need in the first step in the app.

also, i would add dedicated, separate fields only as really needed - classifying everything is a table sounds nice and correct, however, in practice, all these nice classifications often do not work out and you need even more fields.

so, my suggestion is: separate fields only for things actually proven to be needed by the apps, the rest can go to a freetext field.

the database yml-format:

---                                                                    
name: Example Provider
state: [OK | PREPARATION | BROKEN]
domains:
- example.org
servers:
  - type: imap
    socket: SSL
    hostname: imap.example.org
    port: 123
    username: defaults to email address
  - type: imap
    socket: STARTTLS
    hostname: imap.example.org
    port: 456
    username: defaults to email address
  - type: smtp
    socket: SSL
    hostname: smtp.example.org
    port: 789
    username: defaults to email address
before_login_hint: 
    optional text that is be displayed before the user logs in
    typically in combination with state=PREPARATION or state=BROKEN
after_login_hint: optional,
    text that is added to the device chat after login
date: [optional, YYYY-MM]
website: [optional, website of the provider]
---                                                                             
[optional markdown that descripes the preperation steps, this gets displayed on the website]

PREPARATION means steps needed to be done by the user that cannot be done automatically.

all fields but name, state and domains are optional. however, in practice, either at least servers or before_login_hint will be specified to make some sense.

minimal json-format example that gets to the ui:

{
  "state": "PREPARATION",
  "before_login_hint": "please create an app-specific password"
}

the other fields are needed by the core only, eg. servers is used to login, after_login_hint adds a message to the device chat. no need for the ui to care about this.

1 Like

Server should be optional in the yaml, as it will overwrite thunderbirds data.
I like the seperation of date from the state as this makes the data structure more flat.
The optional fields were ~ in my pr, because thats what the serde yaml parser understands easily.

Also iIwould not make the date optional, its useful for the ui to know how long ago the last check was to hint the user to look at the website instead if its older than x days, though that would require us to update the status regularly, wouldn’t it?

yip, the server section can be optional. also as eg. for outlook we would just say “broken” - or for maybe gmx the server-settings work out of the box (so the defaults probed as now) but we want to say “prepare”.

wrt date: eg. the thunderbird database has no date, so, at least in the core, there will be many cases without a date. also, i do not think that this is super-useful, esp. as it is easily badly maintained. maybe we should just show the date from git on the website and that’s it. but this is nothing really important currently.

Is website meant to point to the provider-db-page for this provider? If not, that field lacks from the example.

Nope, website just refers to the providers homepage.
The data structure saved in the fontmatter yaml is not exactly what the provider crate will expose, see https://github.com/deltachat/provider-db/blob/47beeb1db20db28479706fd34ac7ca27937cb95a/src/provider.rs#L23-L32
for what is actually saved and exposed to the ui (keep in mind that this file is from my “old” saturday-morning pr, where you guys only revied the readme as far as I can tell.)

To document the latest developments:

We just agreed in a call to use this structure for now - @r10s will start implementing the UI on Android, so we have a reference.

(if I understood it correctly^^)

1 Like

Cool, the UI is implemented in Android :slight_smile:

The Desktop implementation status is here: Make use of the new provider-db · Issue #1408 · deltachat/deltachat-desktop · GitHub

I don’t found provider Vodafone (vodafone.de, vodafonemail.de).

Have you checked if the IMAP and SMTP settings for vodafone.de/vodafonemail.de work fine in Delta Chat? If they do, they can be applied of course.

I haven’t tried it yet. Is a PREPARE possible necessary ?
vodafonemail

It would be most helpful if you could check out the Vodafone settings by entering them manually. If Delta Chat works successfully using these settings, they can be implemented in the database. Please let us know about it, thank you.

I have now tested with vodafonemail.de. It worked right away. You can use that in provider-db.

1 Like

Great, thanks for reporting, I have filed an issue here: https://github.com/deltachat/provider-db/issues/129
Edit: Vodafone.de has been added to the Delta Chat provider database. Thanks!