4 <meta name="viewport" content="width=device-width, initial-scale=1">
5 <title>Apas: bla bla bla</title>
6 <link rel="stylesheet" href="style.css">
11 <h1 id="apas-activitypub-via-email">Apas: ActivityPub via email</h1>
12 <p>By Oliver Lowe <<a href="mailto:o@olowe.co">o@olowe.co</a>> (<a href="https://apas.srcbeat.com/otl/actor.json">otl@apubtest2.srcbeat.com</a>)</p>
14 <p>Every program attempts to expand until it can read mail.
15 Those programs which cannot so expand are replaced by ones which can.</p>
17 <p>—<a href="https://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski's_Law">Zawinski’s Law of Software Envelopment</a></p>
18 <p><strong><a href="https://git.olowe.co/apub">Source code</a></strong> | <strong><a href="https://godoc.org/olowe.co/apub">GoDoc</a></strong></p>
20 <p>The most popular systems on the <a href="https://en.wikipedia.org/wiki/ActivityPub">ActivityPub</a>-speaking [Fediverse] are imitations of non-federated platforms.
21 <a href="https://joinmastodon.org">Mastodon</a> and <a href="https://misskey-hub.net">Misskey</a> are <a href="https://en.wikipedia.org/wiki/Twitter">Twitter</a> clones;
22 <a href="https://join-lemmy.org">Lemmy</a> is a <a href="https://en.wikipedia.org/wiki/Reddit">Reddit</a> clone.
24 the protocol connecting all these systems together –
25 is often said to be similar to email,
26 which involves exchanging messages.
27 In the case of at least Mastodon and Lemmy,
28 ActivityPub was implemented after the bulk of each sofware was designed.
29 Message exchange – federation by ActivityPub – is arguably a second-class citizen for these
30 traditional <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD</a> web applications backed by SQL databases and fronted by web browser UIs.</p>
31 <p>Apas is an experiment in exposing ActivityPub in a familiar and popular interface: email.
32 Its primary goal is to clarify how ActivityPub and the Fediverse work for the broader community.
33 A number of secondary goals are detailed later.</p>
34 <h2 id="1.-motivation">1. Motivation</h2>
35 <p>As a fan of <a href="http://9p.io/plan9/">Plan 9</a> and a weirdo who likes to fiddle with network protocols for fun,
36 I was disappointed with what using Mastodon, Lemmy et al. felt like.</p>
37 <p>What excites me is <em>communication!</em>
38 Exchanging messages <em>between</em> people, systems, and places we can’t think of yet!
39 It’s what makes receiving even just a single email from a random person such a viscerally distinct experience from
40 thousands reading your post you uploaded somewhere.
41 We’re communicating!</p>
42 <p>Implementing a subset of the Mastodon and Lemmy HTTP APIs in a couple of languages was relatively straightforward.
43 After writing some small clients and tooling
44 it felt like I was just dealing with platforms,
45 not a federated universe.
46 The pattern was familiar for many software developers:</p>
48 <li>You create a post,</li>
49 <li>that gets written to a database,</li>
50 <li>you get an ID back, indicating success.</li>
52 <p>But the whole federation bit is obscured.
53 You hope that others can see that post… somehow…?
54 The platform thinking is evident in the language we see around these systems:
55 “I saw this on Lemmy”, or “this is trending on Mastodon”, or “find me on Akkoma”.
56 Nobody says “find me on email” or “someone sent this on email”.</p>
57 <p>Interoperability efforts fall flat when expertise in one system does not translate to another.
58 Moderation and tooling discussions are artificially limited to a particular system.
59 Should a plugin for Friendica filtering posts containing racist language only work with Friendica when all the systems work together?
60 Should it even be a plugin tied to one particular system in a particular programming language at all?</p>
61 <p>Finally, interoperability and portability is the “killer feature” of ActivityPub systems and any significant software system.
62 We know software developers can write standalone Twitter clones day-in, day-out.
63 But no amount of funding to Instagram or any other incumbent commercial platform
64 will ever make it available for
65 shitty government systems,
72 automatic vacuum cleaners
73 <em>all at once</em>.</p>
74 <p>Writing better software systems often means communicating better.
75 That means understanding ActivityPub better.</p>
76 <h2 id="2.-overview">2. Overview</h2>
77 <p><strong>Apas</strong> is mostly inspired by the <a href="http://doc.cat-v.org/bell_labs/upas_mail_system/">upas</a> email system available with <a href="http://9p.io/plan9/">Plan 9</a>;
78 a collection of small programs operate on files and streams,
79 relaying messages out to the Internet or
80 delivering to mailboxes on the filesystem.
81 (But it’s so much more limited and poorly designed than upas that I was hesitant to even write this bit!)</p>
82 <p>An important difference from existing Fediverse software is that
83 <strong>apas</strong> only represents Activities as messages.
84 There are no application-specific data structures like
85 posts, toots, comments, or pages.
86 Messages can be files in a filesystem,
87 which simplifies implementation significantly.</p>
88 <h3 id="2.1-messages">2.1 Messages</h3>
89 <p><strong>Apas</strong> marshals ActivityPub objects into <a href="https://www.rfc-editor.org/rfc/rfc5322">RFC5322</a> messages and vice-versa.</p>
90 <p>The <a href="https://www.w3.org/TR/activitystreams-vocabulary/#dfn-note">Note Activity</a> is probably the most recognisable object exchanged by ActivityPub servers.
91 They are represented as comments by Lemmy, and posts (toots?) by Mastodon.
92 For instance, imagine a reply from Alex to Bowie talking about motorcycle tyres.
93 It’s passed around the Fediverse as JSON-encoded data like this:</p>
96 "id": "https://apub.example.com/alex/12345678",
97 "attributedTo": "https://apub.example.com/alex"
98 "to": "https://apas.test.example/bowie/87654321",
99 "cc": "https://apas.test.example/bowie/followers",
100 "inReplyTo": "https://apas.test.example/bowie/87654321",
101 "name": "Thoughts on 50/50 tyres"
102 "content": "But what if you don't know when you want to ride off-road?",
105 <p>For <strong>apas</strong> this is equivalent to the mail message:</p>
106 <pre><code>From: "Alex " <alex@apub.example.com>
107 To: "Bowie" <bowie@apas.test.example>
108 CC: "Bowie (followers)" <bowie+followers@apas.test.example>
109 Message-ID: <https://apub.example.com/alex/12345678>
110 In-Reply-To: <https://apas.test.example/bowie/87654321>
111 Subject: Thoughts on 50/50 tyres
113 But what if you don't know when you want to ride off-road?
115 <p>Unlike other Fediverse software,
116 the message to be distributed is written and read by people; not just machines.
117 For developers, administrators, and advanced users, seeing data like
118 this builds familiarity with the behaviour between systems,
119 and facilitates communication.
120 We go from “why can I see my toot on Kbin but not on Pleroma?” to
121 “why didn’t your Pleroma server receive my message?”
122 which is a much easier question to answer; it’s what the systems are actually doing.</p>
123 <p>If there was only one thing to take away from <strong>apas</strong>, it’s that
124 familiarity with data over an API is hugely helpful for
125 troubleshooting. Especially when typical bug reports consist of URLs
126 to irrelevant web apps (or even just screenshots!) trying to explain
127 what was sent versus what was received. That’s before we we even
128 address what could be in a database, itself requiring its own query
129 language and tightly-controlled administrative access to read.</p>
130 <h3 id="2.2.-sending">2.2. Sending</h3>
131 <p>Presenting posts, comments, notes, etc. as a mail message immediately
132 clarifies a big source of confusion with existing systems:
133 why isn’t my post showing up?
134 It becomes easier to reason about this when it is obvious where is a message is sent.
135 An email lists recipients explicitly.
136 When replying to a Kbin comment via Mastodon,
137 it takes knowledge of how each system is implemented to know who the
138 recipients are, if any.</p>
139 <p>Regular email clients
140 (or even any old text editor!)
141 provide an interactive way to test other AP systems.
142 For instance, we can easily test how the message is received if we address the recipient in the <code>CC</code> field instead of <code>To</code>,
143 or if we list the same recipient 20 times in both fields.
144 <strong>Apas</strong> could report deliverability errors either:</p>
146 <li>immediately, or</li>
147 <li>as a bounced message</li>
149 <p>At the moment, error messages are returned immediately.
150 This has provided a pleasant enough testing experience
151 that makes learning ActivityPub an interactive process,
152 directly from any mail client, especially compared with the
153 usual drudgery of sifting through logs of big web applications.</p>
154 <p>Sending is involves two programs each playing its own role:</p>
156 <li>Asubmission program accepts messages from authorised users, and</li>
157 <li>A mailer handles sending messages to the Internet.</li>
159 <p><img src="send.png" alt="" /></p>
160 <h4 id="2.2.1.-submission">2.2.1. Submission</h4>
161 <p>Messages are submitted to a server running <code>apsubmit</code>.
162 <code>apsubmit</code> is a SMTP server.
163 It listens for SMTP connections,
164 authenticates the session,
165 then passes the received message to the mailer <code>apsend</code>.</p>
166 <p>SMTP is a widely implemented protocol.
167 <code>apsubmit</code> enables
168 existing mail clients,
170 and systems that I don’t even know exist,
171 to publish to the Fediverse.</p>
173 it has been fine using <a href="http://www.mutt.org">mutt</a> via SSH on a Linux server,
174 <a href="https://sylpheed.sraoss.jp/en/">Sylpheed</a> on my OpenBSD laptop,
175 <a href="https://freron.com">MailMate</a> on a shared iMac,
176 and the built-in Mail app on my iPhone for replies.
177 I’ll leave others to come up with more ideas;
178 keep in mind weather stations, printers, video records can usually
179 send email but definitely cannot speak ActivityPub!</p>
180 <p>For fast feedback,
181 <code>apsubmit</code> takes advantage of the <code>RCPT</code> stage of the SMTP transaction.
182 It verifies that listed recipients exist and have inboxes we can target.
183 This is in contrast with e.g. Mastodon,
184 which will always accept creating the following post:</p>
185 <pre><code>@john@nowhere.invalid @deleteduser@example.org what do you think?
187 <p>There’s several possible error conditions here. For <code>john</code>, perhaps:</p>
189 <li>their server is down and messages are undeliverable,</li>
190 <li>their Actor is misconfigured and is missing an <code>inbox</code> endpoint,</li>
191 <li>the address is totally invalid</li>
193 <p>For <code>deleteduser</code>, perhaps the account no longer exists.
194 Mastodon never notifies of any delivery errors.
195 We could ask the server administrators to trawl through the server’s logs for us,
196 or ask <code>johnny</code> and <code>deleteduser</code> out-of-band if they got our message.
197 Accounting for some common errors at submission time obviates that extra work.</p>
198 <h4 id="2.2.2-mailer">2.2.2 Mailer</h4>
199 <p>Sending messages is performed by a command-line utility called <code>apsend</code>.
200 <code>apsend</code> reads a message from standard input and disposes of it based on the recipients.
201 If the above message from Alex to Bowie was in a file called “note.eml”,
202 we could send it with the following shell command:</p>
203 <pre><code>apsend -t < file.eml
205 <p>In general, <code>apsend</code> is not intended to be executed directly.
206 Instead, a frontend like an email client (sending via SMTP)
207 or a tool like Plan 9’s <a href="https://9p.io/magic/man2html?man=marshal&sect=1">marshal(1)</a>
208 should be used which take care of adding entries to and formatting the
209 header correctly.</p>
210 <h3 id="2.3-receiving">2.3 Receiving</h3>
211 <p>Core to <strong>apas</strong> is handling ActivityPub objects as files in a filesystem.
212 This reveals there are many different ways to retrieve Activitity from the Fediverse
213 beyond the typical process of servers sending Activity to an Actor’s inbox.
214 <strong>Apas</strong> of course supports this (see 2.3.2),
215 but it’s worth mentioning other techniques to show how flexible
216 working with the Fediverse can be.
217 It may also help clarify discussions on user privacy.</p>
218 <h4 id="2.3.1-direct">2.3.1 Direct</h4>
219 <p>This was the first implementation of receiving ActivityPub objects for <strong>apas</strong>.
220 The command <code>apget</code> fetches the Activity at a URL, then writes it to the standard output.
221 Throughout testing, I ran the tool in shell scripts like the below to deliver messages to my inbox:</p>
222 <pre><code>apget https://apub.example.com/alex/12345678 | apsend otl
224 <p>Little shell scripts can fetch a series of posts:</p>
225 <pre><code>for i in `seq 12345671 12345678`
227 apget -m https://apub.example.com/alex/$i
230 <p>Obviously this is inefficient compared to other methods,
231 but we’re not Google.
232 Handy ad-hoc testing.</p>
233 <h4 id="2.3.2-targeting-the-activitypub-inbox">2.3.2 Targeting the ActivityPub inbox</h4>
234 <p>This is the typical ActivityPub process.
235 For example, someone could mention us in a Mastodon post:</p>
236 <pre><code>@bowie@apas.test.example hope apas is going OK!
238 <p>Which results in the Mastodon server sending Activity to bowie’s Actor inbox.
239 In <strong>apas</strong>,
240 <code>apserve</code> provides a typical HTTP server for a minimal ActivityPub service.
241 It is responsible for:</p>
243 <li>receiving Activity over HTTP (ActivityPub inbox)</li>
244 <li>serving users’ sent Activity for other servers to fetch (ActivityPub outbox)</li>
245 <li>serving each user’s Actor</li>
246 <li>resolving WebFinger lookups</li>
248 <p><img src="receive.png" alt="" /></p>
249 <p>Delivery is not handled by <code>apserve</code>.
250 Instead, <code>apserve</code> converts Activities to mail messages,
251 and passes them on to <code>apsend</code> for delivery.</p>
252 <h4 id="2.3.2-following">2.3.2 Following</h4>
253 <p><a href="https://www.w3.org/TR/activitystreams-vocabulary/#dfn-follow">Follows</a> can be sent using <code>apsend</code>.
254 Because Follows are not represented clearly as mail,
255 the Follow needs to be written as JSON directly.
256 For example, for user bowie to follow alex:</p>
258 "@context": "https://www.w3.org/ns/activitystreams"
259 "actor": "https://apas.example.org/bowie",
261 "object": "https://apub.example.com/alex"
264 <p>then piped to <code>apsend</code>:</p>
265 <pre><code>apsend -j alex@apub.example.com < follow.json
267 <p>Wrapped up in X-line shell script named <code>apfollow</code>,
268 following and unfollowing is equivalent to running the commands:</p>
269 <pre><code>apfollow alex@apub.example.com
270 apfollow -u alex@apub.example.com
272 <h4 id="2.3.3-rssatom-feeds">2.3.3 RSS/Atom feeds</h4>
273 <p>Many ActivityPub servers also make content available via <a href="https://www.rfc-editor.org/rfc/rfc4287">web feeds</a>.
274 This could be an efficient way to fetch content using a battle-tested format
275 from resource-constrained servers.</p>
276 <p>One possible tool is something that manages reading new entries in a feed.
277 For each entry, it extracts the ActivityPub object ID from the
278 <code><guid></code> or <code><link></code> tag for RSS and Atom respectively.</p>
279 <pre><code><entry>
280 <title>Atom-Powered Robots Run Amok</title>
281 <link href="https://apub.example.com/alex/12345678"/>
282 <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
283 <updated>2023-12-13T18:30:02Z</updated>
284 <summary>hello, world!</summary>
287 <h4 id="2.3.4-fediverse-software-http-apis">2.3.4 Fediverse software HTTP APIs</h4>
288 <p>Many existing systems provide a HTTP API which provides a convenient
289 way of finding content based on some application-specific logic e.g.
290 a group, full-text search, or time created.</p>
291 <p>An early <strong>apas</strong> prototype was really just a Python script which
292 synchronised my <a href="like%20Mastodon">GoToSocial</a> timeline with a directory on disk.
294 <pre><code>for status in timeline():
295 note = apget(status.source_id)
296 with open(status.id) as f:
299 <h4 id="2.3.5-takeaway">2.3.5 Takeaway</h4>
300 <p>As mentioned already,
301 core to <strong>apas</strong> is handling ActivityPub objects
303 files in a filesystem.
304 It’s not meant to be the most performant system (not to say that it’s slow),
305 but it lets us develop an understanding of the ActivityPub protocol
306 and focus on the data over APIs via quick prototyping.</p>
307 <h4 id="2.x-todo-filtering-spam">2.x TODO Filtering, spam?</h4>
309 <li>text streams</li>
310 <li>small portable programs instead of plugins to growing systems</li>
312 <h3 id="2.4-reading">2.4 Reading</h3>
313 <p>Messages are stored in the <a href="https://en.wikipedia.org/wiki/Maildir">Maildir</a> format; one message per file.
314 This is not an important part of the system.
315 Maildir is used only because of the easy implementation for <code>apsend</code>;
316 it just neads to create create files.</p>
317 <p>How messages are presented to users –
318 no matter how they are stored –
319 is a job for which software has been written for decades already.</p>
320 <p>Here are some that are being used or</p>
321 <h4 id="2.4.1-">2.4.1 <code>read()</code></h4>
323 <p>During development, being able to just run cat(1) on a file to debug Content-Type encoding bug
324 was a breath of fresh air when compared to what is more common in web development.</p>
325 <p>That is, running a unit test which queries a relational database running in a container in a virtual machine hopefully all configured correctly, then marshalling that into the application’s unique data structure, to ActivityPub, then finally JSON-encoded (half-joking).</p>
326 <h4 id="2.4.2-existing-solutions">2.4.2 Existing solutions</h4>
327 <p><strong>Maildir</strong>. Some clients can interact with Maildir directly, like <a href="http://www.mutt.org">mutt</a>.</p>
328 <p><strong>IMAP</strong>. The obvious and most popular method for accessing mailboxes over the network. Dovecot works well. IMAP is very widely supported by mail clients.</p>
329 <p><strong>NNTP/Usenet</strong>.
330 Throughout this document I’ve referred to “mail messages”.
331 But the so-called “Internet Message Format” described in RFC 5322 is also used by
332 <a href="https://en.wikipedia.org/wiki/Usenet">Usenet</a> via a protocol known as <a href="https://datatracker.ietf.org/doc/html/rfc3977">NNTP</a>.
333 The protocol is a simple line-based text protocol
334 with many open-source libraries available.
335 Serving Fediverse messages from a filesystem over NNTP would be a fun project.
336 Similar to how the Linux Kernel Mailing list is available over NNTP at <code>nntp.lore.kernel.org</code>.</p>
337 <p><strong>Mailing list archive web interfaces</strong>.
338 Finally yet another opportunity to give those old Perl scripts another lease of life.</p>
339 <p><strong>upasfs(4)</strong>.
340 <a href="http://doc.cat-v.org/bell_labs/upas_mail_system/">upas</a> is the system I studied to implement <strong>apas</strong>.
341 Messages could be relayed from <code>apsend</code> to <code>upas/send</code>,
342 or the Maildir could be converted to mdir(6).
343 Then we would have a <em>real</em> filesystem interface over 9P.
344 Another project for another time.</p>
345 <h2 id="3.-workarounds-limitations">3. Workarounds & limitations</h2>
346 <p>The mapping between Activity objects, mail messages,
347 ActivityPub HTTP methods, and SMTP transactions
348 has a number of limitations.
349 <strong>Apas</strong> uses some workarounds internally to fill some gaps.</p>
350 <p>The <a href="https://www.w3.org/TR/activitystreams-vocabulary/#microsyntaxes">Mention</a> Activity,
351 used by Mastodon for notifications,
352 is implemented by reading the To field of submitted messages.
353 Recipients in <code>To</code> are added as Mentions.
354 For example, the message:</p>
355 <pre><code>To: "Oliver Lowe" <otl@hachyderm.io>
359 <p>results in an entry in <code>tags</code> in an ActivityPub Note:</p>
365 "href": "https://hachyderm.io/users/otl",
366 "name": "@otl@hachyderm.io"
370 <p>There is not an easy way to address an Actor’s followers using the <code>acct:</code> mail address syntax.
371 <code>apas</code> understands a non-standard syntax using “plus addressing”.
372 For example to address the followers of <a href="mailto:user@example.com">user@example.com</a>
373 the address <a href="mailto:user+followers@example.com">user+followers@example.com</a> may be used.
374 These followers addresses cannot be resolved by WebFinger.</p>
375 <p>Likes and Dislikes are silently dropped by <code>apserve</code>.
376 The reader can decide whether this is a workaround, feature, or bug.</p>
377 <p>Accept and Rejects from Follow requests can be received via ActivityPub
378 and delivered as mail but for notifications only.
379 The reverse does not work;
380 <strong>apas</strong> cannot read a Follow request from a mail message.</p>
381 <p>To simplifly delivery to local mailboxes,
382 Actors served by <code>apserve</code> have no shared inbox/outbox.
383 Fortunately shared inbox endpoints are inteded as a performance opimitisation for
384 servers hosting many Actors, which is beyond the scope of <strong>apas</strong>.</p>
