Welcome to Stubmail

To Do

• done. Convert this document to a Wiki

See Also

• the wiki

Contents

Concepts

For more philosophy, please see:

• Meng Weng Wong's RSS/Email white paper
• Nathan Cheng's HTMP post on CircleID

Design Rules

If this were a commercial product, this section would be titled Marketing Requirements / Business Rules.

UDP stub notifications are allowed to be unreliable. In the worst case, if they get lost, the onus is on the receiver to poll.

The retrieval server always encrypts the message index before dumping it. The message index is encrypted to the PGP public key of the receiver. This makes it okay for anyone to request a message index, because only the actual receiver will be able to decrypt it. Anyone can request a message index. This may one day be superseded by SSL client side certificates.

The receiver's addressbook is the final authority on what mail the receiver reads. If a sender is whitelisted in the addressbook, mail will be read. If a sender is not in the addressbook, there is no guarantee the receiver will choose to poll it.

Every stub sender received by the stub listener MUST be conveyed to the receiver during a stublist notification download. This satisfies free-speech considerations: the receiver ISP who runs the stub listener SHOULD NOT perform spam filtering on the stublist. This does not mean that every actual stub goes to the receiver; the stublist MAY BE uniq'ed by sender, so that multiple stubs from one sender are reduced to a single stub in the stublist.

In the reference implementation, primary identifiers are 32-bit PGP key IDs. All participants, both senders and receivers, must have a PGP key ID if they want to receive mail. That said, the system makes it really easy for people to automatically create PGP keys.

Mapping from an email address to a sender-store URL is outside the scope of this architecture. There are a number of out-of-band ways to perform this mapping. They include:

• The X-Stubmail header, which says: "this message is also available at the following URL."
• Mapping from an email address to a YADIS-style identity URL; that identity URL then maps to a PGP public key.
• Mapping from an email address directly to a PGP public key via a keyserver; the spoofing aspects to this problem are addressed by the PGP web-of-trust architecture.

Download

Developer-only subversion access to the source repository is at svn://blizzard.pobox.com/stubmail

Public subversion access to source repository WILL BE at http://code.stubmail.com/svn/

prerequisite: PGP / GPG

We recommend that you install the latest version of GPG. Create a keypair for yourself.

XXX: Later, when you know what you're doing, you can add a uid whose comment field is set to your sender-side URL. Receivers will automatically poll you for mail by appending their PGP key ID to this URL.

prerequisite: Apache + Perl

The reference implementation requires a number of Perl modules. Fortunately, they are all in core.

The reference implementation requires a web server with CGI capability. We recommend Apache.

If you're casting about for a server to install all this on, we recommend Debian Linux.

Quickstart Guide

Installation Instructions

Install the CGIs:

• post_manager
• retrieval server
• stublist server

Install the daemons:

• stublistener

Create one of these configuration directories:

• /etc/stubmail
• /usr/local/etc/stubmail
• $HOME/.stubmail

Populate your chosen configuration directory with:

• fileroot: a file containing a path for where messages are stored, e.g. /usr/share/stubmail

Create the fileroot:

• e.g., /usr/share/stubmail

Create an addressbook:

• $HOME/.stubmail-addressbook (feel free to copy the addressbook from examples/)

Release Milestones

These milestones describe individual components.

research prototype: in development, still being banged into shape; responsible developer is trying to get it internally consistent and working against his own codebase.

design: checked in; active developers are experimenting with it, arguing about design direction and implications for overall architecture, and getting the code to interoperate against other components.

development: other components depend on this code; it is accepted as part of the product architecture, and bugs are being found and patched.

testing: test suite has been written for component.

documentation: technical specification for the component describes it in enough detail to be independently reimplementable.

alpha release: component is used by primary developers, is documented, passes tests. Early adopters and friends of the project are invited to install and use it.

beta release: component is part of the stable release for the general public.

Release Status

Perl Library

During design phase, we code directly into the relevant component; when that code hits development, we move it to the library.

Workflow: Sending Messages

smtp2stub proxy

Input: RFC2822 message over RFC2821 submission on port 25 or 587. Message may be plaintext or PGP encrypted.

Actions: submits message to the post_manager.

Actions: If the receiver is not stub-capable, send message using SMTP, with the message index URL in the headers.

Actions: If the receiver is stub-capable, does not send message using SMTP.

post_manager

Input: well-formed HTTP submission

Returns:

• receiver_accepts_stubs: boolean: does receiver accept stubs?
• send_smtp_copy: boolean: should the calling program send an SMTP copy of the message?
• retrieval_url: url_string: message index URL representing sender-receiver tuple.
• disk_write: enum: success / failure, indicating whether post_manager was able to write the message to disk. if failure, calling program should indicate that message delivery was unsuccessful and tell the client to try again later.

Actions: always writes message to disk. returns in disk_write.

Actions: if receiver accepts stubs, sends stub notification. returns receiver_accepts_stubs.

Crypto: if receiver has no key, and message is not marked public, spontaneously generates PGP keypair, encrypts the message, and emails the receiver the private and public keypair.

Implementation note: In the reference implementation, when writing an outbound message file, the post manager touches all parent directories up to the /S/ directory. This makes it easier for the retrieval server to support the since parameter.

scribble to disk

Workflow: Stub Notification

Stub Capability Discovery

This discovery protocol is in design phase.

Problem: Given a receiver email address, is that receiver stub-capable?

Possible Answers:

• No, the receiver doesn't know anything about stubs.
• Yes, the receiver is stub-capable, but would like an SMTP copy just to be on the safe side.
• Yes, the receiver is stub-capable, and has no need for an SMTP copy.

Considerations: One end-user at a domain may have a stub-capable mail reader, but another end-user may not.

Discovery Protocol:

1. Send an SRV query packet for _stub._udp.domain.
2. If no SRV responses are received, the receiver domain is not stub-capable.
3. If one or more SRV responses are received, interpret them as follows:
4. If one or more SRV responses are received, and the class fields are set to value 1, then the domain is stub-capable, but no localpart precision is available; the value of the records are stub notification servers. Send stub notifications to those servers directly according to priority.
5. If exactly one SRV response is received, and the class field is set to value 3, the receiver domain wishes to specify localpart precision. The value of the record is a macro string. %{l} means "localpart". Construct another SRV query packet.
6. Send another SRV query according to the macro provided.
7. If no SRV responses are received, the receiver is not stub-capable.
8. If one or more SRV responses are received, the receiver is stub capable.
9. XXX: If all the class fields in all the SRV responses contain a value less than 512, the receiver declines an SMTP copy.
10. XXX: If any of the class fields in the SRV responses contain a value greater than 512, the receiver wishes to receive an SMTP copy.

Proposal: meng proposes 20060420 that the bits of the class field should be interpreted as follows:

1. The domain is stub-capable.
2. The domain wishes to specify localpart granularity.
3. This stublist server usually sends UDP ACKs. This serves as a hint to the stub sender: if this bit is set to 0, the stub sender may send zero or one stubs.

Stub Sender

Stub Listener

Stublist Dumper

The Stublist Dumper operates at the receiver end. The Retriever Client asks it for stubs received since the last time that Retriever polled for a stublist.

In the reference implementation, it is a CGI.

Workflow: Retrieving Messages

Stublist and Message Retriever Client

The Retriever Client operates at the receiver end. It connects to one or more receiver-side Stublist Servers and polls for stubs. Informed by the addressbook, by the stubs received, and by the time elapsed since last it ran, it connects to zero or more sender-side Retrieval Servers and polls for sent messages.

The reference implementation offers two Retriever Clients.

http2mbox queries stublist servers, queries Retrieval Servers using the Simple Format Protocol, sucks down new mail, and outputs an RFC2822 mbox to STDOUT. It is suitable for Unix-style mail retrieval. It reads and writes to ~/.stubmail-addressbook.

We also offer a set of patches to Thunderbird. Our patched Thunderbird queries Retrieval Servers using the ATOM/XML protocol and displays messages directly in the Thunderbird GUI. Instead of using ~/.stubmail-addressbook, this approach subscribes to multiple senders directly using Thunderbird's native RSS code.

Retrieval Server

The Retrieval Server operates at the sender end.

The Retrieval Server represents the sender's outbox via the HTTP Message Retrieval API. It offers sent messages to the retriving receiver.

In the reference implementation, it is a CGI.

Atom Message View

APIs

We generally follow the spirit of REST, adapted to our needs.

HTTP Message Path Conventions

In general, messages from a sender S to a receiver R are available under the tree /mailroot/S/R/.

Note that /S/R/ trees are encrypted. This means that all fetches against /S/R/ and its subtrees are encrypted against the receiver R's public key.

Public messages are available under the special tree /mailroot/S/public/. The string "public" is literal. The /public/ tree is unencrypted.

The reference implementation stores messages on disk under /mailroot/S/R/YYYY/MM/DD.

In the reference implementation, S and R are long-form PGP public key IDs.

20060420-16:37:17 mengwong@newyears:~% gpg --keyid-format=short --list-keys mengwong
pub   1024D/5A83DCB4 1998-06-16
uid                  Meng Weng Wong 
sub   2048g/192B0C49 1998-06-16

20060420-16:37:20 mengwong@newyears:~% gpg --keyid-format=long --list-keys mengwong 
pub   1024D/66E62E505A83DCB4 1998-06-16
uid                          Meng Weng Wong 
sub   2048g/2C0CBC52192B0C49 1998-06-16

20060420-16:37:28 mengwong@newyears:~% gpg --with-colons --list-keys mengwongtru::1:1144625328:1260045786:3:1:5
pub:u:1024:17:66E62E505A83DCB4:1998-06-16:::u:Meng Weng Wong ::scaESCA:
sub:u:2048:16:2C0CBC52192B0C49:1998-06-16::::::e:

The /S/R/ convention is not a hard requirement; a Stubmail sender server may construct any URL it wishes, as long as that URL represents a unique sender/receiver pair. So a Stubmail sender server may construct a URL containing an opaque string meaningful only to itself.

HTTP Message Retrieval API

Clients can pass a since parameter to request that message indexes exclude data from before a certain epoch time.

Clients can pass a format parameter to request a specific format for the message index.

Data Formats

HTTP Message Index Format

The message index contains four types of elements:

• Subdirectories
• Message Files
• Attachment Files
• Comments

If the message index arrives in the form of application/pgp-encrypted content, you will need to decrypt it before you can parse it.

The simple format used by the reference implementation looks like this:

XXX

Note that the HTTP Message Index API offers the format parameter which clients may use to request the message index in an alternative format, such as ATOM or RSS.

Addressbook

Stub Notifications

Stublist Format

Native Message Payload Format

Atom Format

HTTP Message Submission

The post_manager expects the following parameters to arrive in a POST:

• sender key ID
• sender email address
• privacy field: one of "public" or "private"
• receiver key ID list
• receiver email address list
• receiver email address list
• arbitrary data blob

When a submission arrives at the post_manager, it SHOULD already be encrypted by the submitter. If it is not encrypted, and the message is not marked public, the post_manager MUST encrypt it, opportunistically if necessary.

Cryptography Notes

Archiving Compliance

Some regulated environments require that all outbound messages be stored. In such an environment, senders MUST encrypt all outbound messages against the sending server's public key in addition to the receivers' public keys. Sender servers MAY reject messages that are not encrypted to their public keys.

Prior Art

IM2000. Weemail. HTMP. Others listed at http://mengwong.com/rssemail/

Project History

Primary Participants

Community and Mailing Lists

Wishlist

If we can somehow convert a PGP key into a self-signed openSSL keypair usable as a client-side certificate, the message index server can just dump straight Apache directory indexes over https rather than doing all that work with automatically encrypting /S/R/ directories against R's public key.