Many useful WebXDC apps (especially music, media, weather, news, etc.) would benefit from the ability to load resources from specific trusted domains while maintaining WebXDC’s strong security guarantees.
My Idea:
Add an optional allowed-domains in manifest.toml:
… allowed-domains = ["api.spotify.com", ... ]
When a user opens a WebXDC that declares network access:
Show a clear security warning before starting the app, similar to Android app permissions.
For example: “This app wants to connect to: api.spotify[.]com. Only continue if you trust it.”
I think in certain cases, it would also be useful if each user could also specify additional origins to allow either per instance, or per app (correlated by file hash). Think of an RSS feed reader xdc that could aggregate entries from the list of websites each user wants to follow.
A workaround that works pretty well, which you can do today already:
Use a bot to get internet access. Your webxdc mini app can use the status update API or the real-time API for sending requests that the bot then answers.
The bot is a normal program, it can give you internet access or access to whatever you want, from public APIs, smart home sensors, news feeds and everything else you can do with programming.
Having the bot in the group also makes it clear that you opted in that it can read from and write to the chat. Offer your users to self-host for extra privacy.
That’s said some form of heavily limited internet access could also be useful for big apps and games that need to load extra resources and assets, which are larger than what fits inside an email. (Though I personally would be more interested in some way to send huge (100mb-3gb) attachments and thus also large apps since mini apps are attachments. To not add dependence on servers that could go down or in theory leak/steal data.)
I’d like to hear legit use cases for this. Because I think most of the time if one needs access to some website then what you’re after is not a WebXDC app but a regular website or a browser extension.
Regarding the technical part of this proposal: this would require an amount of non-trivial code, because we’d have to go from “disable internet by all means” to “enable internet, but in a very specific way”.
One should be aware that this would bypass the usual security stuff that browsers have around HTTP requests, namely Access-Control-Allow-Origin, cookie-related stuff, maybe more.
I just remembered one myself, thanks to @ian’s comment: the “maps” feature in Delta Chat is precisely this: a WebXDC app that uses this special “fetch” API provided by Core to fetch map tiles:
But I personally dislike that. There is a concept of offline maps. A few hundred MB can cover a country region. The “maps” app should just tell users where to download the map data and how to import it. Then no internet access would be required.
One drawback for suggesting as a workaround to visit the original website is if it would incur a huge cost, such as if the webpage and its assets took a lot of time and network bandwidth to load up and could track & fingerprint the user quite a bit.
In contrast, it would take orders of magnitudes less bandwidth if a webxdc integration had access to the precise data resource (either from a separate XML, JSON, RSS, Atom or protobuf file or parsed out from the shell HTML) without all the naggers.
Another possible advantage compared to visiting the webpage would be if automation was involved, such as a (system-) webxdc that kept checking the given API endpoint (or HTML) for changes according to a schedule so the user would not have to and notified the user of events immediately (e.g., by notification or by synthesizing a new message from such an event).
If the target scenario involved on-demand fetching and aggregating information from multiple sites (even if it’s just a few or up to a dozen), it would still be vastly superior if it could be automated than if the user was forced to tediously open each bookmark and visually connect the dots.
One distinct disadvantage of suggesting to proxy such traffic through a central bot (other than affordability) is that certain service API will then start blocking you due to the volume and pattern of traffic coming through that single IP as multiple users are sharing the same bot.
Chat apps in the 90s and 00s had plugins/scripts similar to webxdc and quite a lot made use of network access. I can’t find a good overview of this, but you could find features such as
sending a rich preview along with your link (from basically any site you have already visited after confirmation)
following feeds,
weather forecasts,
search content on various portals (the web, package managers, multimedia memes, Wikipedia, book, movie & music metadata),
compute with Wolfram Alpha,
integrate with VCS & ticketing,
link shortening, snippet creation,
meeting & eventing,
monitoring your sites for uptime,
evaluate code snippets in various languages in a VM sandbox
start an interactive telnet session to a BBS (this is actually seeing a resurgence now)!
In my eyes you are describing a bot. This is similar to a “send later” feature, it needs to be always active to be complete/awesome.
Then the bot needs efficient code architecture and caching instead of hammering sites with the same requests over and over again.
Also, bots can live anywhere, like even on an old android phone or laptop that you just keep plugged into the wall. It’s not like you need port-forwarding, static IPs, domains or a hoster.
So “affordability” is not really an issue if you really want it, “convenience” might be, and you could sell this convenience to others by running a hosting service for bots.
Maybe an android app that makes hosting bots simpler would make sense, like what if instead of using the Termux | The main termux site and help pages. terminal you could just install and app and press a button to host a bot.
I also thought about this a lot already, it’s not as easy/cheap as it sounds. Even if you skip technical implementation complexity, there is still the question on who would audit or review those rule - users can not judge if a cryptic regexp filter rule is safe or not (even I could not).
So auditing the app’s code to make sure it doesn’t craft some parameter to exfiltrate data and does not have XSS vulnerabilities would be needed and also easier than having a perfect filter rule. This brings us to the idea of signed approved apps that may get an exception to the no-internet rule, but those also bring their own challenges.
At this point I wonder a bit why does it need to be a webxdc mini app vs a traditional installable PWA?
Like is there a need for internet data and also group collaboration at the same time? What would be use cases for this concretely?
A scheduled send later feature could still serve 99% of its utility if it could only send out the message while one of my clients are online. Sometimes it’s as simple as “don’t reveal how long I have been up yesterday” or “don’t wake up the recipient if you know they forgot to mute their phone while you figured something out in the evening and instead just send it sometimes during daytime at the earliest convenience”. It might even be possible to form mutual sending circles of encrypted & signed messages with friends, but I think we have already discussed that in another issue.
In the criticism of the proxy scheme, I did not imply that 100 users would be polling the exact same endpoints. For example, each user would open a different page on Wikipedia, and no amount of caching will solve this given the long tail of requests (and possibly freshness requirements of users).
A large number of individuals in our circles had been running mobile phones and tablets from the wall 24/7 for various use cases. Experience shows that they require a lot of manual attention in comparison to a real server that could consolidate dozens of them. Their batteries will go bad, sometimes explode, but often wear down to a level where finishing boot is not possible after a disconnection. Battery elimination is sometimes possible depending on model by purchasing parts and soldering, but that can also increase power consumption and reduce reliability further due to more noise & interference. Wireless networking is not as stable as ethernet and many models can’t reconnect in a robust, reliable manner, and USB-ethernet dongles cost space & money.
The board, OS and runtime itself is not that stable, crashing or closing your payload from time to time, definitely needing monitoring and someone around to keep touching it or power cycling it in a timely manner. Lost sysop time per use case in unit period of time is vastly more than managing servers of equivalent capacity, especially if deploying one device per household according to your recommendation. In comparison, architectures built around clients providing best-effort delay-tolerant services in a fail-through redundant manner would carry none of these drawbacks.
There could be multiple tiers of whitelist, each with a different confirmation dialog (and possibly a client-wide option):
a static list of origins in the manifest,
URL regexp in the manifest,
runtime constructed URL regexp (probably not needed as it becomes nearly unrestricted at that point)
individual rules that each user of the webxdc may add for themselves manually,
allow any URL of any origin, but ask the user every single time before sending the request and provide details for review (URL, method unless GET, any additional headers) - useful for generating rich previews
An installable PWA is still scoped to its origin, meaning you can’t create a mashup app using data from multiple websites this way unless all APIs you would need already exist and are forgiving for CORS.
Quite a few concrete use cases exist - one for implementing each already implemented bot to be able to run within DC as a system xdc as described in a separate issue. And then the newer proposed features enumerated above and in the other issue.
I wasn’t suggesting complex regex rules at all. I proposed a simple list of domains in the manifest, which would be clearly displayed to the user.
Secondly, the user can make their own informed decision. When the app is opened, they would see a clear warning that this app wants to connect to which domains.
It’s the same principle as mobile app permissions; the user doesn’t need to be a security expert. They can simply choose not to use the app if they don’t trust the listed domains.
It doesn’t have to be that internet access and group collaboration are the absolute core of WebXDC apps. Many WebXDC apps work fine completely offline. However, optional limited internet access would still unlock significantly better features for a wide range of apps, even if the core experience remains offline-first.
Using a bot is a practical workaround, but it adds real hassle.
Someone has to host and maintain the bot. For every app that needs internet access, both developers and users face extra setup and complexity. Just for some simple API requests that could be done client-side.