IMAP, JMAP and the future of open email standards
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 561 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/44339 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
EmailCommunications protocolAddress spaceServer (computing)EncryptionClient (computing)Extension (kinesiology)SynchronizationLatent heatWebDAVInternetworkingGroup theoryBoundary value problemSoftwareConstraint (mathematics)Basis <Mathematik>Suite (music)OracleUniqueness quantificationElectronic mailing listNetwork topologyLevel (video gaming)Open sourceRight angleData managementNumberMultiplication signLattice (order)User interfaceServer (computing)PlanningSingle-precision floating-point formatFirewall (computing)AuthenticationClient (computing)Musical ensembleEmailUniqueness quantificationExtension (kinesiology)Electronic mailing listBoundary value problemData Encryption StandardConstraint (mathematics)SoftwareBasis <Mathematik>Suite (music)Open setSynchronizationPoint (geometry)Communications protocoloutputAreaInternetworkingBitSolid geometrySystem callEncryptionLatent heatIdentifiabilityAnalogyExpected valueProper mapGroup theoryData structureBuildingSet (mathematics)Integrated development environmentSlide ruleSoftware developerCode refactoringSoftware testingPresentation of a groupComputer animation
07:14
Network topologyExtension (kinesiology)EmailDirected setServer (computing)Open sourceLevel (video gaming)Single-precision floating-point formatStapeldateiComplete metric spaceDependent and independent variablesEndliche ModelltheorieReduction of orderNumberRoundness (object)Query languageMusical ensembleImplementationOpen sourceAuthenticationInterface (computing)Client (computing)View (database)Server (computing)WindowMultiplication signSet (mathematics)Closed setGoodness of fitObject (grammar)Projective planeElectronic mailing listData managementState of matterWebsitePosition operatorCommunications protocolSynchronizationEmailLatent heatArithmetic progressionMobile appMessage passingOrder (biology)TrailString (computer science)Product (business)Extension (kinesiology)Physical systemWeb browserAdditionEntire functionAliasingDomain nameConnected spaceData typeLattice (order)Office suiteFile formatHTTP cookieCase moddingToken ringIntrusion detection systemLink (knot theory)Category of beingDirected graphFlow separationCore dumpReal numberProxy serverKnowledge baseSoftware developerRight angleComputer virusPoint (geometry)Absolute valueStreaming mediaBoolean algebraComputer animation
14:23
Musical ensembleComputer animation
15:05
Point cloudComputer animation
Transcript: English(auto-generated)
00:05
standards. I'm going to talk about Jmap and IMAP and then hopefully there'll be plenty of time for questions at the end. I've just flown in from Australia so it's the middle of the night for me. If I fall asleep you know why. First of all, the disclaimer. I'm not talking about end-to-end encryption. I didn't
00:21
know Vincent was going to be speaking directly before me. All I'm talking about is client-to-server email protocols. There's plenty to talk about there. We're not addressing server-to-server, we're not addressing end-to-end encryption at all today. Alright, this is what we have now. Not many people using POP3 anymore. Open standards, it's pretty much IMAP plus
00:42
extensions, including vendor specific extensions, and CalDAV, LDAP, a bunch of other standards for things that are not email. Microsoft have three protocols of their very own. Gmail have their own proprietary protocol. There's a bunch of custom protocols for accessing email because IMAP's kind
01:01
of miserable to use in today's world. So along came JMap. Fastmail built our own protocol because we needed to for our web interface and we looked at it and said we could evolve this to be something that could be an open standard. We built it based on IMAP, on CuriSync, on the DAV sync collection.
01:23
Basically my experience of building the Cyrus IMAP servers internal data structures gave us the basic plan for what to do with JMap. Our key goal was a single protocol for everything so that there's not that you can send email but not save your draft to the server or you can read your email but
01:40
you can't send because there's firewall issues or authentication issues. For a company that does tech support for our users we see that a lot. Alright, we then took our protocol that we built 2010-2011, we simplified it to make something that could be an open standard and we'd always wanted to have an open standard. We're strong supporters of open source and we're even
02:02
stronger supporters of open standards and we published it ourselves. When I came here four years ago I was trying to sell the idea of use ourselves published standard but the big companies in particular said we'd love to do this, it looks really good, management won't support it unless it has an RFC number. So we decided to take it to the ITF and create an RFC
02:23
number. ITF for those of you who don't know is the Internet Engineering Task Force. They've been around meeting now for 34 years, three times a year, 104 meetings next month. They're the custodians of standards for the Internet and just a quick show of hands who's heard of the ITF in this
02:42
room? A bit over half of you, excellent. ITF is a fantastic resource for anyone who's building a new protocol. We've been doing email for 20 years at Fastmail but the people that we met at ITF have that depth of experience in building protocols and in a more diverse set of environments than we work
03:02
with. So it's a fantastic resource for anyone and the work we had through the ITF made J-Map both simpler and more powerful at the same time which is awesome. This is kind of the key slide that I wanted to bring to this audience as mainly developers. The best things happen at the boundary between freedom and constraints. Standards are constraints. They mean that York software
03:25
can communicate with other software. There's something that lasts longer than the next refactor or the next point release. Standards documents once they're written are set in stone and they will last 20-30 years hopefully if they're well written. And of course they're a great basis for your test suite because they are a solid boundary. Great standards take a lot longer than
03:44
software to write and so that's why. It's been four years and I'm basically plugging the same thing that I did four years ago except a lot closer to being locked into stone. It's actually at last call at ITF now. Now of course along with J-Map which is the glorious future we have IMAP right now
04:04
and soon after I joined ITF two years ago I got asked to also co-chair another working group called Extra. Extra is extending IMAP and we have done we had ten documents that we were working on two years ago. Eight of them are now standards. One of them is in the last stages and the final one is
04:24
still underway. We've been incredibly prolific. We've had a lot of people working from it's a small group but a lot of companies and a lot of people who really know what they're working on what they're doing working on stuff. So it's making IMAP more useful as well. The OpenExchange dovcot people will be
04:42
talking tomorrow in the RTC working group working room about something new they're working on as well so IMAP is still very much alive. Come along and see that as well. Of course I'm very proud of my own spec. This is something I've been wanting for ten years. It's unique identifiers for emails that stay with them when they move between folders which is such a
05:04
small thing but it means that if you rename a folder in one client you don't need to redownload all the email just to make sure you have the same emails if another client sees it. So really happy with that and of course the one standard that's still underway IMAP4rev2 which is going to take the
05:21
base standard plus 30 extensions that we currently have and hopefully give us a raised floor just to use the analogy we heard before that everyone can expect proper compliant IMAP servers support that and you can go to your vendor and say why aren't you supporting Rev2? Rev2 is where it's at and then hopefully IMAP will become better. Of course I'm kind of sabotaging
05:45
JMap which is what I want by making IMAP better but I think they're both going to be really useful. You can get involved. At the IETF there's obviously these two groups working. I'm working on another thing as well which is calendar extensions at IETF and if you have an idea for anything that you
06:05
think should be more of a standard and less of an immediate point release it's a great place to workshop your idea it's a great place to bring your input even if you don't have something you want to do yourself if you have expertise in an area IETF would love to have you come along. If you come to
06:22
the meetings that obviously costs money it costs money to run them and so there's a charge for that but you can participate for free in the mailing lists and everyone who's a remote participant is considered just as valuable as the people who are there. So do get involved in that. This is a bit of a sidetrack from JMap but standards are really important. So here's
06:41
where we're at. We have IMAP extensions. There have been seven or eight of them implemented and released by a lot of people just in the last couple of years as we get things that individual vendors had produced but there was no where to take them now as standards that everyone can do. Next to IETF is coming up in a month. The first two days the Saturday Sunday are a hackathon so if you
07:03
want to work on any email standard I'll be there hacking away on something JMap related. I'm sure there'll be other people come and come and sit around with us and have a hackathon and then the meeting during the week is where we'll actually talk about the standards work. I'm off to Calconnect which is another standards body next week to talk about calendaring. We're meeting in
07:22
Zurich at the Google offices there and we'll be talking about extensions to calendaring standards. One of the things we're working on there is a JSON calendar format that we can then add to JMap calendars and of course JMap.io is the site that we've been using to keep track of the work on the standards itself. You can play with JMap right now.
07:43
Sirus IMAP server has very complete support. We run that at Fastmail for our production systems and of course if you sign up for a Fastmail trial account you can play with JMap in the browser. There's no time in a 15 minute talk to show you the protocol dumps over the wire but you can open up your browser's protocol dumps and you can see exactly what's happening with our
08:03
client web client talking to a real JMap server. Sadly the proxy is out of date. I had hoped to get it up to date this week but I'm going to be hacking on that a lot next week during Calconnect and then a few days after that and staying here in Europe with one of our other developers and working solidly on getting some of the Perl stuff up to date as well.
08:26
And that's all I've got for my talk. Thank you very much. Lots of links here. Any questions? Right in the middle. I'm going to run to you with the microphone so the people in the live stream can hear as well what the question is. Can you raise your
08:43
hand? All right. This works. So I noticed that you based this on CuraSync so I of course wonder what in particular GMAP is doing to reduce latency on the protocol? Yep. So for a start we decided to
09:05
work with external push capabilities rather than requiring a connected system so you can do out of bound push that then just triggers get an update. JMap works with the concept of states. So rather than a mod seek which
09:21
you then have to know the old mod seek and get a new mod seek and the numbers have to be incrementing. It's just an opaque state and that came more from the sync token in CalDAV and CardDAV but the idea is you say from this state tell me what's changed, what's been added, what's been removed either to the entire pool of messages or of an object of
09:41
particular data type or you can say for a query if you have a listing of messages in a particular order with the same query string and this old query state if there's a new state for the query tell me what's been added at which position in the list and what's been removed from the list and so that can quite efficiently update either a view on a particular window or a set of objects. Yes. So the
10:18
question here was did we do anything to merge things like
10:21
status and list of subscribe folders? GMAP supports batching so you can send multiple requests in a single batch of requests and it will send you back responses for all of them so that's to reduce the number of round trips. It also has a concept of back references so if you're getting a list of objects or a list of object IDs you can then say also fetch these properties for these objects
10:42
with a back reference and it will send you a response for the same objects that were in that initial query. Thank you. Hi. Is the GMAP implementation on the FastMail server up to the latest spec? Because I was trying fastmail.com slash well known or slash GMAP and did get a 404?
11:03
You need to be at the authentication side of things. It's not following the exact spec so you need to have a FastMail account and use the FastMail interface to get the correct cookies. You can't just use GMAP authentication as is. One of the things that came out of IOTF was that authentication is hard. Not only hard to do but hard
11:24
to get past all the other groups that are also working on HTTP authentication and don't want a separate system. So you'll need to have FastMail specific cookies in order to use it but after that yes it is the FastMail interface is the entire GMAP spec plus our own
11:40
additional objects for things like managing aliases, managing domains, managing your internal account details. Thank you. We have another question here. Hello. Still a work in progress but are there any client projects who've got in there early to have a play with GMAP? Yes there are.
12:00
There's a list on the GMAP.io website of projects that we know about but there are probably also other projects that aren't willing to publish where they're up to just yet. I know there are a couple of clients and servers that are not yet up to date with the most recent spec and they're waiting just until the spec's completely finalised and published. So they're not chasing a moving target
12:20
they've implemented a point in time six months or a year ago and there are certainly both closed source and open source client projects. We have an open source client of our own as well that we're publishing. It's a fairly basic client as well as our complete closed source client. Since you've been working on the IMAP spec is there any work done
12:41
to support labels like in Gmail instead of just folders? In IMAP there's the XGM labels which Gmail do. There isn't something exactly the same for IMAP generally. You can use keywords if you want to do labelling that way but it doesn't fit really well with the model.
13:02
GMAP supports the idea that a message can be in multiple mailboxes at the same time and that's the way that you implement labels there. So labels work quite well in GMAP. They're not really easy to tack into IMAP without basically replicating what Google did and people don't feel confident standardising that.
13:21
We have time for one more question. If you want to continue the discussion I'm sure you will be available after the talk as well. Yes, absolutely. Right at the back. I see a hand over there. This is good for me in the workout. Excellent. I see Cyrus Eiben upon the list. What about Dovecot?
13:43
Good question. Dovecot say they're interested. I see some smiles in the middle of the room here. Dovecot say they're interested in implementing GMAP but I have not heard where they're up to yet. Do you want to?
14:00
It's on the roadmap. The response is as a true project manager I can't tell you it's on the roadmap. It'll be done by Christmas. We don't know which year. Okay, thank you very much. Give it a warm up. Thank you everybody.
14:41
Okay, since we have a lot of people coming and leaving the room may I ask you to go as much to the middle as possible once you sit down. Make sure there's no empty seats next to you just so we have a seat for everyone that wants to enter.
Recommendations
Series of 9 media