1 # Apas: ActivityPub via email
3 By Oliver Lowe <[o@olowe.co](mailto:o@olowe.co)> ([otl@apubtest2.srcbeat.com](https://apas.srcbeat.com/otl/actor.json))
5 > Every program attempts to expand until it can read mail.
6 > Those programs which cannot so expand are replaced by ones which can.
8 —[Zawinski's Law of Software Envelopment]
10 **[Source code]** | **[GoDoc]**
12 [Source code]: https://git.olowe.co/apub
13 [GoDoc]: https://godoc.org/olowe.co/apub
17 The most popular systems on the [ActivityPub]-speaking [Fediverse] are imitations of non-federated platforms.
18 [Mastodon] and [Misskey] are [Twitter] clones;
19 [Lemmy] is a [Reddit] clone.
21 the protocol connecting all these systems together –
22 is often said to be similar to email,
23 which involves exchanging messages.
24 In the case of at least Mastodon and Lemmy,
25 ActivityPub was implemented after the bulk of each sofware was designed.
26 Message exchange – federation by ActivityPub – is arguably a second-class citizen for these
27 traditional [CRUD] web applications backed by SQL databases and fronted by web browser UIs.
29 Apas is an experiment in exposing ActivityPub in a familiar and popular interface: email.
30 Its primary goal is to clarify how ActivityPub and the Fediverse work for the broader community.
31 A number of secondary goals are detailed later.
33 [CRUD]: https://en.wikipedia.org/wiki/Create,_read,_update_and_delete
34 [upas]: http://doc.cat-v.org/bell_labs/upas_mail_system/
37 As a fan of [Plan 9] and a weirdo who likes to fiddle with network protocols for fun,
38 I was disappointed with what using Mastodon, Lemmy et al. felt like.
40 What excites me is *communication!*
41 Exchanging messages *between* people, systems, and places we can't think of yet!
42 It's what makes receiving even just a single email from a random person such a viscerally distinct experience from
43 thousands reading your post you uploaded somewhere.
46 Implementing a subset of the Mastodon and Lemmy HTTP APIs in a couple of languages was relatively straightforward.
47 After writing some small clients and tooling
48 it felt like I was just dealing with platforms,
49 not a federated universe.
50 The pattern was familiar for many software developers:
53 * that gets written to a database,
54 * you get an ID back, indicating success.
56 But the whole federation bit is obscured.
57 You hope that others can see that post... somehow...?
58 The platform thinking is evident in the language we see around these systems:
59 "I saw this on Lemmy", or "this is trending on Mastodon", or "find me on Akkoma".
60 Nobody says "find me on email" or "someone sent this on email".
62 Interoperability efforts fall flat when expertise in one system does not translate to another.
63 Moderation and tooling discussions are artificially limited to a particular system.
64 Should a plugin for Friendica filtering posts containing racist language only work with Friendica when all the systems work together?
65 Should it even be a plugin tied to one particular system in a particular programming language at all?
67 Finally, interoperability and portability is the "killer feature" of ActivityPub systems and any significant software system.
68 We know software developers can write standalone Twitter clones day-in, day-out.
69 But no amount of funding to Instagram or any other incumbent commercial platform
70 will ever make it available for
71 shitty government systems,
78 automatic vacuum cleaners
81 Writing better software systems often means communicating better.
82 That means understanding ActivityPub better.
86 **Apas** is mostly inspired by the [upas] email system available with [Plan 9];
87 a collection of small programs operate on files and streams,
88 relaying messages out to the Internet or
89 delivering to mailboxes on the filesystem.
90 (But it's so much more limited and poorly designed than upas that I was hesitant to even write this bit!)
92 An important difference from existing Fediverse software is that
93 **apas** only represents Activities as messages.
94 There are no application-specific data structures like
95 posts, toots, comments, or pages.
96 Messages can be files in a filesystem,
97 which simplifies implementation significantly.
101 **Apas** marshals ActivityPub objects into [RFC5322] messages and vice-versa.
103 The [Note Activity] is probably the most recognisable object exchanged by ActivityPub servers.
104 They are represented as comments by Lemmy, and posts (toots?) by Mastodon.
105 For instance, imagine a reply from Alex to Bowie talking about motorcycle tyres.
106 It's passed around the Fediverse as JSON-encoded data like this:
110 "id": "https://apub.example.com/alex/12345678",
111 "attributedTo": "https://apub.example.com/alex"
112 "to": "https://apas.test.example/bowie/87654321",
113 "cc": "https://apas.test.example/bowie/followers",
114 "inReplyTo": "https://apas.test.example/bowie/87654321",
115 "name": "Thoughts on 50/50 tyres"
116 "content": "But what if you don't know when you want to ride off-road?",
119 For **apas** this is equivalent to the mail message:
121 From: "Alex " <alex@apub.example.com>
122 To: "Bowie" <bowie@apas.test.example>
123 CC: "Bowie (followers)" <bowie+followers@apas.test.example>
124 Message-ID: <https://apub.example.com/alex/12345678>
125 In-Reply-To: <https://apas.test.example/bowie/87654321>
126 Subject: Thoughts on 50/50 tyres
128 But what if you don't know when you want to ride off-road?
130 Unlike other Fediverse software,
131 the message to be distributed is written and read by people; not just machines.
132 For developers, administrators, and advanced users, seeing data like
133 this builds familiarity with the behaviour between systems,
134 and facilitates communication.
135 We go from "why can I see my toot on Kbin but not on Pleroma?" to
136 "why didn't your Pleroma server receive my message?"
137 which is a much easier question to answer; it's what the systems are actually doing.
139 If there was only one thing to take away from **apas**, it's that
140 familiarity with data over an API is hugely helpful for
141 troubleshooting. Especially when typical bug reports consist of URLs
142 to irrelevant web apps (or even just screenshots!) trying to explain
143 what was sent versus what was received. That's before we we even
144 address what could be in a database, itself requiring its own query
145 language and tightly-controlled administrative access to read.
147 [Note Activity]: https://www.w3.org/TR/activitystreams-vocabulary/#dfn-note
151 Presenting posts, comments, notes, etc. as a mail message immediately
152 clarifies a big source of confusion with existing systems:
153 why isn't my post showing up?
154 It becomes easier to reason about this when it is obvious where is a message is sent.
155 An email lists recipients explicitly.
156 When replying to a Kbin comment via Mastodon,
157 it takes knowledge of how each system is implemented to know who the
158 recipients are, if any.
160 Regular email clients
161 (or even any old text editor!)
162 provide an interactive way to test other AP systems.
163 For instance, we can easily test how the message is received if we address the recipient in the `CC` field instead of `To`,
164 or if we list the same recipient 20 times in both fields.
165 **Apas** could report deliverability errors either:
168 * as a bounced message
170 At the moment, error messages are returned immediately.
171 This has provided a pleasant enough testing experience
172 that makes learning ActivityPub an interactive process,
173 directly from any mail client, especially compared with the
174 usual drudgery of sifting through logs of big web applications.
176 Sending is involves two programs each playing its own role:
178 1. Asubmission program accepts messages from authorised users, and
179 2. A mailer handles sending messages to the Internet.
183 #### 2.2.1. Submission
185 Messages are submitted to a server running `apsubmit`.
186 `apsubmit` is a SMTP server.
187 It listens for SMTP connections,
188 authenticates the session,
189 then passes the received message to the mailer `apsend`.
191 SMTP is a widely implemented protocol.
193 existing mail clients,
195 and systems that I don't even know exist,
196 to publish to the Fediverse.
199 it has been fine using [mutt] via SSH on a Linux server,
200 [Sylpheed] on my OpenBSD laptop,
201 [MailMate] on a shared iMac,
202 and the built-in Mail app on my iPhone for replies.
203 I'll leave others to come up with more ideas;
204 keep in mind weather stations, printers, video records can usually
205 send email but definitely cannot speak ActivityPub!
208 `apsubmit` takes advantage of the `RCPT` stage of the SMTP transaction.
209 It verifies that listed recipients exist and have inboxes we can target.
210 This is in contrast with e.g. Mastodon,
211 which will always accept creating the following post:
213 @john@nowhere.invalid @deleteduser@example.org what do you think?
215 There's several possible error conditions here. For `john`, perhaps:
217 * their server is down and messages are undeliverable,
218 * their Actor is misconfigured and is missing an `inbox` endpoint,
219 * the address is totally invalid
221 For `deleteduser`, perhaps the account no longer exists.
222 Mastodon never notifies of any delivery errors.
223 We could ask the server administrators to trawl through the server's logs for us,
224 or ask `johnny` and `deleteduser` out-of-band if they got our message.
225 Accounting for some common errors at submission time obviates that extra work.
229 Sending messages is performed by a command-line utility called `apsend`.
230 `apsend` reads a message from standard input and disposes of it based on the recipients.
231 If the above message from Alex to Bowie was in a file called "note.eml",
232 we could send it with the following shell command:
236 In general, `apsend` is not intended to be executed directly.
237 Instead, a frontend like an email client (sending via SMTP)
238 or a tool like Plan 9's [marshal(1)]
239 should be used which take care of adding entries to and formatting the
242 [marshal(1)]: https://9p.io/magic/man2html?man=marshal§=1
246 Core to **apas** is handling ActivityPub objects as files in a filesystem.
247 This reveals there are many different ways to retrieve Activitity from the Fediverse
248 beyond the typical process of servers sending Activity to an Actor's inbox.
249 **Apas** of course supports this (see 2.3.2),
250 but it's worth mentioning other techniques to show how flexible
251 working with the Fediverse can be.
252 It may also help clarify discussions on user privacy.
256 This was the first implementation of receiving ActivityPub objects for **apas**.
257 The command `apget` fetches the Activity at a URL, then writes it to the standard output.
258 Throughout testing, I ran the tool in shell scripts like the below to deliver messages to my inbox:
260 apget https://apub.example.com/alex/12345678 | apsend otl
262 Little shell scripts can fetch a series of posts:
264 for i in `seq 12345671 12345678`
266 apget -m https://apub.example.com/alex/$i
269 Obviously this is inefficient compared to other methods,
270 but we're not Google.
271 Handy ad-hoc testing.
273 #### 2.3.2 Targeting the ActivityPub inbox
275 This is the typical ActivityPub process.
276 For example, someone could mention us in a Mastodon post:
278 @bowie@apas.test.example hope apas is going OK!
280 Which results in the Mastodon server sending Activity to bowie's Actor inbox.
282 `apserve` provides a typical HTTP server for a minimal ActivityPub service.
283 It is responsible for:
285 * receiving Activity over HTTP (ActivityPub inbox)
286 * serving users' sent Activity for other servers to fetch (ActivityPub outbox)
287 * serving each user's Actor
288 * resolving WebFinger lookups
292 Delivery is not handled by `apserve`.
293 Instead, `apserve` converts Activities to mail messages,
294 and passes them on to `apsend` for delivery.
298 [Follows] can be sent using `apsend`.
299 Because Follows are not represented clearly as mail,
300 the Follow needs to be written as JSON directly.
301 For example, for user bowie to follow alex:
304 "@context": "https://www.w3.org/ns/activitystreams"
305 "actor": "https://apas.example.org/bowie",
307 "object": "https://apub.example.com/alex"
310 then piped to `apsend`:
312 apsend -j alex@apub.example.com < follow.json
314 Wrapped up in X-line shell script named `apfollow`,
315 following and unfollowing is equivalent to running the commands:
317 apfollow alex@apub.example.com
318 apfollow -u alex@apub.example.com
320 #### 2.3.3 RSS/Atom feeds
322 Many ActivityPub servers also make content available via [web feeds].
323 This could be an efficient way to fetch content using a battle-tested format
324 from resource-constrained servers.
326 One possible tool is something that manages reading new entries in a feed.
327 For each entry, it extracts the ActivityPub object ID from the
328 `<guid>` or `<link>` tag for RSS and Atom respectively.
331 <title>Atom-Powered Robots Run Amok</title>
332 <link href="https://apub.example.com/alex/12345678"/>
333 <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
334 <updated>2023-12-13T18:30:02Z</updated>
335 <summary>hello, world!</summary>
338 #### 2.3.4 Fediverse software HTTP APIs
340 Many existing systems provide a HTTP API which provides a convenient
341 way of finding content based on some application-specific logic e.g.
342 a group, full-text search, or time created.
344 An early **apas** prototype was really just a Python script which
345 synchronised my [GoToSocial] (like Mastodon) timeline with a directory on disk.
348 for status in timeline():
349 note = apget(status.source_id)
350 with open(status.id) as f:
355 As mentioned already,
356 core to **apas** is handling ActivityPub objects
358 files in a filesystem.
359 It's not meant to be the most performant system (not to say that it's slow),
360 but it lets us develop an understanding of the ActivityPub protocol
361 and focus on the data over APIs via quick prototyping.
363 [web feeds]: https://www.rfc-editor.org/rfc/rfc4287
364 [Follows]: https://www.w3.org/TR/activitystreams-vocabulary/#dfn-follow
365 [GoToSocial]: https://gotosocial.org
367 #### 2.x TODO Filtering, spam?
370 - small portable programs instead of plugins to growing systems
374 Messages are stored in the [Maildir] format; one message per file.
375 This is not an important part of the system.
376 Maildir is used only because of the easy implementation for `apsend`;
377 it just neads to create create files.
379 How messages are presented to users –
380 no matter how they are stored –
381 is a job for which software has been written for decades already.
383 Here are some that are being used or
389 During development, being able to just run cat(1) on a file to debug Content-Type encoding bug
390 was a breath of fresh air when compared to what is more common in web development.
392 That is, running a unit test which queries a relational database running in a container in a virtual machine hopefully all configured correctly, then marshalling that into the application's unique data structure, to ActivityPub, then finally JSON-encoded (half-joking).
394 #### 2.4.2 Existing solutions
396 **Maildir**. Some clients can interact with Maildir directly, like [mutt].
398 **IMAP**. The obvious and most popular method for accessing mailboxes over the network. Dovecot works well. IMAP is very widely supported by mail clients.
401 Throughout this document I've referred to "mail messages".
402 But the so-called "Internet Message Format" described in RFC 5322 is also used by
403 [Usenet] via a protocol known as [NNTP].
404 The protocol is a simple line-based text protocol
405 with many open-source libraries available.
406 Serving Fediverse messages from a filesystem over NNTP would be a fun project.
407 Similar to how the Linux Kernel Mailing list is available over NNTP at `nntp.lore.kernel.org`.
409 **Mailing list archive web interfaces**.
410 Finally yet another opportunity to give those old Perl scripts another lease of life.
413 [upas] is the system I studied to implement **apas**.
414 Messages could be relayed from `apsend` to `upas/send`,
415 or the Maildir could be converted to mdir(6).
416 Then we would have a *real* filesystem interface over 9P.
417 Another project for another time.
419 [Maildir]: https://en.wikipedia.org/wiki/Maildir
420 [Usenet]: https://en.wikipedia.org/wiki/Usenet
421 [NNTP]: https://datatracker.ietf.org/doc/html/rfc3977
423 ## 3. Workarounds & limitations
425 The mapping between Activity objects, mail messages,
426 ActivityPub HTTP methods, and SMTP transactions
427 has a number of limitations.
428 **Apas** uses some workarounds internally to fill some gaps.
430 The [Mention] Activity,
431 used by Mastodon for notifications,
432 is implemented by reading the To field of submitted messages.
433 Recipients in `To` are added as Mentions.
434 For example, the message:
436 To: "Oliver Lowe" <otl@hachyderm.io>
440 results in an entry in `tags` in an ActivityPub Note:
447 "href": "https://hachyderm.io/users/otl",
448 "name": "@otl@hachyderm.io"
452 There is not an easy way to address an Actor's followers using the `acct:` mail address syntax.
453 `apas` understands a non-standard syntax using "plus addressing".
454 For example to address the followers of user@example.com
455 the address user+followers@example.com may be used.
456 These followers addresses cannot be resolved by WebFinger.
458 Likes and Dislikes are silently dropped by `apserve`.
459 The reader can decide whether this is a workaround, feature, or bug.
461 Accept and Rejects from Follow requests can be received via ActivityPub
462 and delivered as mail but for notifications only.
463 The reverse does not work;
464 **apas** cannot read a Follow request from a mail message.
466 To simplifly delivery to local mailboxes,
467 Actors served by `apserve` have no shared inbox/outbox.
468 Fortunately shared inbox endpoints are inteded as a performance opimitisation for
469 servers hosting many Actors, which is beyond the scope of **apas**.
471 [Zawinski's Law of Software Envelopment]: https://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski's_Law
472 [NetNewsWire]: https://netnewswire.com
473 [mutt]: http://www.mutt.org
474 [MailMate]: https://freron.com
475 [Sylpheed]: https://sylpheed.sraoss.jp/en/
477 [ActivityPub]: https://en.wikipedia.org/wiki/ActivityPub
478 [Misskey]: https://misskey-hub.net
479 [Twitter]: https://en.wikipedia.org/wiki/Twitter
480 [Reddit]: https://en.wikipedia.org/wiki/Reddit
482 [Mention]: https://www.w3.org/TR/activitystreams-vocabulary/#microsyntaxes
484 [2023 Reddit API controversy]: https://en.wikipedia.org/wiki/2023_Reddit_API_controversy
485 [3rd Party Twitter Apps]: https://www.theverge.com/2023/1/22/23564460/twitter-third-party-apps-history-contributions
486 [Lemmy]: https://join-lemmy.org
487 [Mastodon]: https://joinmastodon.org
488 [RFC5322]: https://www.rfc-editor.org/rfc/rfc5322
489 [Plan 9]: http://9p.io/plan9/
