Reading Email with Emacs

I've been reading and sending email with GNU Emacs since about 1991. In that time I have used at least five different emacs email clients: MH-E, VM, Mew, gnus, and now mu4e. I have also adapted to many different email server setups, and I have helped many of my current and especially my former colleagues do the same.

Some years ago now our departmental email server was shut down and our email moved to a Microsoft Exchange server. The Exchange IMAP server (used for reading email) in conjunction with gnus was so slow that it was unusable, so I switched to a combination of the mu4e emacs client with a local maildir store kept in sync with the server by offlineimap. I was serving as department chair at the time and the super fast search provided by mu was really nice to have with all the email that comes with that position.

A few years ago the university decided to completely outsource its email services to Microsoft, so I configured offlineimap to retrieve my email from outlook.office365.com. This posed no problems until recently, when Microsoft deprecated the “Basic Auth” protocol for checking email in favor of “Modern Auth” (oath2) and I was once again forced lose a long weekend adapting to a new email setup.

I admit that I am trying to do all this while minimizing the number of things that I actually have to understand about how any of these protocols work. I spent way too much of my earlier life setting up departmental email servers and clients before finally swearing off ever doing that again, and I have spent a great deal of time in the intervening years adapting to changing email environments and helping colleagues to do the same. At this point in my life I am completely uninterested in how any of this works beyond what is absolutely required for me to use it — I just want to continue to read and write work email in the same development environment (emacs) that I use for everything else.

There are two parts to this problem: retrieving and syncing email with Microsoft's IMAP server, which I have been doing with offlineimap; and sending email messages via Microsoft's SMTP (Symmetric Mail Transport Protocol) server, which I have been doing for many years with emacs's built-in sendmail code. There do exist emacs packages named oauth2, auth-source-xoauth2, and oauth2-request, but I didn't get very far with any of them and they wouldn't help me on the IMAP side, because I use an external program (offlineimap) for that interaction (and having become used to mu4e I'm not going back to gnus).

So, I needed (1) to find another way to send email that works with oauth2, and (2) to either configure offlineimap to work with oauth2 or find another program to handle interactions with the IMAP server. For sending email messages, the most commonly used sendmail replacement seems to be msmtp, so I planned to use that, assuming that I could configure it to use oauth2. (If you don't know, the actual sending of messages is traditionally handled separately from reading and managing messages and folders.)

BTW, to retrieve/sync and send email via Microsofts servers with oauth2, one must have a registered "client app". There is a mechanism for registering such an app with Microsoft Azure (whatever that is), but this failed for me because my institution does not allow me (or anyone?) to register a new app. This means that I must do what everyone else apparently does, which is to use the client ID and secret for Mozilla's Thunderbird email client (there's actually nothing secret about these and you will see them below).

Early on in my search for a solution, I tried M365-IMAP (python) to handle the authentication. I was able to make it work with offlineimap, but the authorization token or tokens have to be updated periodically (every hour or so?). I also don't want to be storing them in a clear text file, so the barebones usage outlined there won't work for me. Still, at a minimum, this is a good way to retrieve some actual authentication tokens to experiment with.

I also tried mailctl and I got it to work with msmtp but not with offlineimap. I tried another method, but the initial authentication step (which uses a browser) wouldn't work for me because all the graphics (buttons) for my institutions login page didn't show up, leaving me unable to complete the 2-factor authentication. (I think I was trying either email-oauth2-proxy or oauth-helper-office-365 when I ran into this problem.)

I finally came across Larie Tratt's pizauth and this actually seems to work for me and takes care of any worries about storing any authentication details on disk. However, I didn't manage to get it working with offlineimap (I'm sure that's on me), so I decided to switch over to isync/mbsync, which I was already aware of as a faster alternative to offlineimap. The rest of this describes the process of getting this working on Ubuntu 22.04 (LTS), a.k.a., "Ubuntu Jammy".

First we need a Rust compiler. For this, just go to the Getting Started page for rust and follow the instructions to use the Rustup tool to install Rust. This will install a personal copy of rust in your home directory (as opposed to a system-wide installation). Then add the following lines to your ~/.profile (or wherever you keep your shell startup stuff):

source ~/.cargo/env

You may need to reload your .profile (or your .bashrc) or just log out and log back in again to get the rust compiler on your path.

Next download pizauth either from Laurie Tratt's web pages or from the Github repository and untar/unzip it if necessary. I was able to compile the program with make but sudo make install failed because sudo doesn't have the Rust compiler on it's path. To fix this, I edited the Makefile and changed these lines

install:
	cargo build --release
	install -d ${PREFIX}/bin
	...

to

install:
	# cargo build --release
	install -d ${PREFIX}/bin
        ...

This works becuase the cargo build --release step is already done when you run make (without sudo) to compile the program, so it can safely be left out of the installation step. (For reference, cargo is the Rust package manager.)

So again, after this edit, just run the following commands to compile pizauth and install it in /usr/local.

  cd pizauth-0.1.0
  make
  sudo make install

I next created the file ~/.config/pizauth.conf, whose contents are given below. The client_id and client_secret are actually for thunderbird, but this works. For your own use you may want to change "UF" to something else (e.g., "work") and you will have to replace my email address with your own on the login_hint line.

account "UF" {
    auth_uri = "https://login.microsoftonline.com/common/oauth2/v2.0/authorize";
    token_uri = "https://login.microsoftonline.com/common/oauth2/v2.0/token";
    client_id = "08162f7c-0fd2-4200-a84a-f25a4db0b584";
    client_secret = "TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82";
    scopes = [
      "https://outlook.office365.com/IMAP.AccessAsUser.All",
      "https://outlook.office365.com/SMTP.Send",
      "offline_access"
    ];
    // You don't have to specify login_hint, but it does make
    // authentication a little easier.
    login_hint = "presnell@ufl.edu";
}

Next run

  pizauth server

You shouldn't see any output from this. Assuming that all has gone well to here, run (again, change UF to whatever you used for the account name in your pizauth.conf file.

  pizauth show UF

The first time you do this a URL will be output. Copy this URL and open it in your browser and do whatever is needed to authenticate with your institution (if your browser is already authenticated, you may not have to do anything but open the URL). To see if this worked, run pizauth show UF again and you should see the access token obtained.

This one is easy to install:

  sudo apt install msmtp

Next create the file ~/.msmtprc with the following contents, which again will need to be modified to match your situation.

  ## UF email via office365 (Microsoft)
  account UF
  auth    xoauth2
  host    smtp.office365.com
  protocol smtp
  port    587
  tls     on
  tls_starttls on
  from    presnell@ufl.edu
  user    presnell@ufl.edu
  passwordeval pizauth show UF

At this stage you can test that sending email is working by running something like:

  printf "Subject: Test\n\nHello there me." | msmtp -a UF your_personal_account@gmail.com

and checking that the mail is received at your personal account.

An error similar to this:

sh: 1: pizauth: Permission denied
msmtp: cannot read output of 'pizauth show UF'

probably indicates that your OS (Ubuntu for me) has put some overly restrictive apparmor conditions on msmtp. You can fix this by disabling apparmor for msmtp like this:

  sudo ln -s /etc/apparmor.d/usr.bin.msmtp /etc/apparmor.d/disable/
  sudo apparmor_parser -R /etc/apparmor.d/usr.bin.msmtp

Again, installion of the isync package is routine.

  sudo apt install isync

To use mbsync with oauth2, you also need to install an xoauth2 plugin for SASL (Simple Authentication and Security Layer). There are several available, and I first tried the ppa on Cononical's Launchpad, but it did not work for me. The most commonly used plugin seems to be this one (I think that it is used in Arch linux and in Free BSD). To install it, I followed the instructions in the Install Cyrus SASL OAuth2 section of this page. As prerequisites, you need to install that libsasl2-dev package and if you want to do the last step (verifying that XOATH2 is known to SASL probably isn't necessary), you will also need to install the sasl2-bin package to get the saslpluginviewer command:

  sudo apt install libsasl2-dev sasl2-bin

After this just follow the instructions mentioned above, i.e.,

  # Clone the Cyrus SASL OAuth2 sources.
  git clone https://github.com/moriyoshi/cyrus-sasl-xoauth2.git

  # Configure and make.
  cd cyrus-sasl-xoauth2
  ./autogen.sh
  ./configure

  # SASL2 libraries on Ubuntu are in /usr/lib/x86_64-linux-gnu/; modify the Makefile accordingly
  sed -i 's%pkglibdir = ${CYRUS_SASL_PREFIX}/lib/sasl2%pkglibdir = ${CYRUS_SASL_PREFIX}/lib/x86_64-linux-gnu/sasl2%' Makefile

  make
  sudo make install

  # Verify XOAUTH2 is known to SASL.
  saslpluginviewer | grep XOAUTH2

Finally, I created a ~/.mbsyncrc file with the contents below. By default Outlook creates and syncs a bunch of folders that I don't use. I basically keep everything in INBOX and Archive and use mu to find things quickly, so it's easiest for me to just list the folders that I want in the Patterns line. If you create a lot of folders for your email, then it might be easier to say what you don't want (there are examples online of how to do this), or just use * and download everything.

Also, for initial testing, the RECOMMENDATIONS section of the mbsync man page recommends leaving Expunge None (the default) to make sure you don't lose mail if anything goes awry.

Finally, I set Create Near so that folders (mailboxes) would be created on the local side when I initially sync things up, but I don't really want to create any new folders on either end. You might prefer Create Both, which I assume is a lot more common.

Note that a "Channel" in mbsync combines a Far with a Near store; in this case the UF channel combines UFRemote and UFLocal.

  IMAPAccount UF
  Host outlook.office365.com
  User presnell@ufl.edu
  Port 993
  SSLType IMAPS
  SSLVersions TLSv1.1 TLSv1.2
  AuthMechs XOAUTH2
  PassCmd "pizauth show UF"

  IMAPStore UFRemote
  Account UF

  MailDirStore UFLocal
  Path ~/Maildir/UF/
  Inbox ~/Maildir/UF/INBOX
  Subfolders Verbatim

  Channel UF
  Far :UFRemote:
  Near :UFLocal:
  SyncState *
  ## I only want these 5 folders:
  Patterns "INBOX" "Archive" "Drafts" "Sent Items" "Deleted Items"
  ## You can sync EVERYTHING by using this line instead:
  ## Patterns *
  Create Near
  Expunge Both

I also had to manually create the ~Maildir/UF directory. You probably don't want that to be readable by anyone but you either, so set the permissions accordingly.

  mkdir -p ~/Maildir/UF
  chmod -R go-rwx ~/Maildir

Finally, test by running the following command. It will take a while the first time to download all your email.

  mbsync UF

Socket error: secure read from outlook.office365.com (52.96.29.82:993): error:0A000126:SSL routines::unexpected eof while reading

I'm going to install the latest version of mu and mu4e from github. If you're happy with an older version, you can install the Ubuntu package maildir-utils with apt.

It might be a good idea to clear out any existing mu and mu4e installations if you have used them before, e.g., with sudo apt purge maildir-utils. I had installed them manually from source, so I did something like this:

  sudo rm /usr/local/bin/mu
  sudo rm -r /usr/local/share/emacs/site-lisp/mu4e
  sudo rm -r /usr/local/share/man/man*/mu-*
  sudo rm /usr/local/share/man1/mu.1
  sudo rm /usr/local/share/man1/mug.1
  sudo rm -r /usr/local/share/doc/mu

Make sure these prerequisite libraries are installed. Add emacs to the list of packages if you don't already have it (I use emacs-snapshot from a ppa so I'm not including that here).

  sudo apt install libgmime-3.0-dev libxapian-dev

Next install the meson build system.

  sudo apt install meson ninja-build

Now we can download the most recent release of mu (currently 1.8.10) and install as below. It is possible that there are prerequisite packages that need to be installed with apt, but if so, I seem to have all of them already installed from previous installations of mu.

  cd ~/Download
  tar -xf mu-1.8.10.tar.xz
  cd mu-1.8.10
  meson build && ninja -C build
  sudo ninja -C build install   # By default this will install into /usr/local

To index your mail, just run:

mu init --maildir=~/Maildir --my-address=presnell@ufl.edu --my-address=presnell@stat.ufl.edu
mu info  # Checking everything is correct
mu index

• It looks like this person had a similar ordeal

  My comments for Laurie Tratt's blog post on pizauth:

Works for me! Thanks so much for this. I lost access to my work email last week when MS turned off basic authentication. After I had struggled to arrive at an acceptable (no tokens stored in unencrypted files) and complete (IMAP and SMTP both working smoothly) solution with any of the other options I came across, pizauth saved the day. I had been using offlineimap + mu + mu4e and emacs's built-in sendmail stuff. I ended up with pizauth + mbsync + msmtp + mu + mu4e, but it's working, so I'm happy.

A few comments:

1. I have rust installed via rustup (so in my home directory) and on my linux box (Ubuntu 22.04) I couldn't get sudo to use it for "sudo make install", so I commented the "cargo build –release" step out of the install recipe in the Makefile. With that change "make; sudo make install" worked without a hitch.

2. My institution does not allow me to register a new application with Azure, so I'm just using the client id and secret for thunderbird, which I assume is what most people are doing.

3. Offlineimap will work with OAuth2 (I verified this by generating access and renew tokens and manually entering them into my .offlineimaprc file). I think that it should work with pizauth (by calling pizauth in a manner similar to this (https://wiki.archlinux.org/title/OfflineIMAP#Configuring_OAuth2_and_getting_access_tokens_via_mailctl) or this (https://www.macs.hw.ac.uk/~rs46/posts/2022-01-11-mu4e-oauth.html) but somehow I failed to get it to work. FWIW, I thought that offlineimap needed both an access token and a renewal token, and assuming that both would eventually expire if not used, I don't see how to get the refresh token from pizauth.

   Today I noticed that offlineimap also has built-in functionality for renewing OAuth2 tokens (https://gist.github.com/piyueh/a2d65e095ea675a2c715ad42b7b61d10). I didn't realize this, so I may have misunderstood some of the configuration advice that I encountered elsewhere. I certainly would have continued with offlineimap if I had managed to get everything working properly, but mbsync does seem to be a bit faster, so there's that I guess.