imap: Add empty-partial-as-quoted to imap_client_workarounds - #317
Open
0nelight wants to merge 1 commit into
Open
imap: Add empty-partial-as-quoted to imap_client_workarounds#3170nelight wants to merge 1 commit into
0nelight wants to merge 1 commit into
Conversation
A FETCH of a body section range that lies past the end of that section is
answered with an empty string, as RFC 9051 6.4.5 requires. Dovecot writes
it as a zero-length literal, "{0}\r\n": get_prefix() emits the header for
data that is streamed out afterwards, and a literal is the only string
form that can carry arbitrary octets, so one format string serves every
size.
The framing parser in swift-nio-imap mishandles exactly the zero-length
case. It emits the header line as a completed frame and enters
.insideLiteral(remaining: 0); the next pass leaves that state again
without consuming a byte, so frameLength is still 0, and parseFrame()
returns .incomplete. parseFrames() stops at the first .incomplete, so the
")\r\n" already sitting in the client's own buffer -- and every tagged
reply behind it -- is never parsed. Only fresh network bytes re-drive the
parser, and after a fully answered FETCH none arrive.
iOS Mail (ID name com.apple.email.maild) is built on a fork of that
library. The effect is a stall of about 90 seconds after a completed,
pipelined FETCH, ended only by the client's own watchdog. It reproduces
with any message whose fetched section is much smaller than RFC822.SIZE,
because the client sizes its 384 KiB slice plan from the latter and every
slice past the end of the section then gets an empty string.
"" and {0} are the same value -- the ABNF has string = quoted / literal --
so this changes the encoding of the reply, not the reply. Dovecot already
prefers "" for an empty string everywhere else: imap_append_nstring()
cannot produce a literal for one, and imap_append_string_for_humans() has
an explicit "" case. get_prefix() was the exception.
The workaround is not restricted to partial fetches. The defect is in the
framing of any zero-length literal, so a plain FETCH of an empty section
is affected just as much.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mailing list thread: https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/message/DMYA4ABWVAZDOMNOL55L7B2UEDAQSL4V/
The symptom
iOS Mail stalls for ~90 seconds after a fully answered pipelined
FETCH,then drops the connection, reconnects, and shows the message instantly. The
server logs look like an idle client going away:
From a decrypted capture of one such session — ten pipelined commands, all ten
answered and tagged within 417 ms, then silence:
The message is 6,928,200 bytes but section
[1]is only 297,588. The clientsizes its 384 KiB slice plan from
RFC822.SIZE, so nine slices land past theend of the section and are answered with an empty string, as RFC 9051 6.4.5
requires. Dovecot writes that as a zero-length literal,
{0}\r\n.Root cause, and it is not Dovecot
The IMAP client in iOS 26.6.1 (build 23G83) is a fork of swift-nio-imap. The
legacy ObjC stack (
MFIMAPConnection) is still present inMessage.framework,but it is not what serves these connections: the Swift stack's
Tag(connectionIdentifier: UInt8, commandCounter: UInt32)produces exactly thetag shapes in the capture above (
P65, andAN196/EX33in others). ItsFramingParsermishandles the zero-length literal:frameLengthis reset to 0,state becomes
.insideLiteral(remaining: 0)bytesAvailable >= remainingis trivially true forremaining == 0, soframeLength += 0leaves it at 0 and the state returnsto
.normalTraversalparseFrame(),if (0 < frameLength)fails, and it returns.incomplete(byteCountNeeded: 2)parseFrames()stops at the first.incompleteThe
)\r\nthat is already sitting in the client's own buffer — and everytagged
OKbehind it — is never parsed. Nothing re-drives the parser exceptfresh network bytes, and after a completed FETCH none arrive. The client waits
for two bytes it already has, until its own watchdog fires at ~90 s.
This is a client defect. Dovecot's reply is correct and this PR does not
suggest otherwise. The workaround is for operators who cannot wait for the
client to be fixed, which is what
imap_client_workaroundshas always been for.Upstream
The client-side defect is now filed in the open-source project the
iOS stack forks from, with a runnable reproducer rather than a description:
FramingParserfed* 1 FETCH (BODY[1]<9> {0}\r\n)\r\nA1 OK Fetch completed\r\nin a single bufferyields
COMPLETE(header),INCOMPLETE(2), and leaves 26 bytes - the)andthe tagged
OK- unparsed. The same input with{1}frames all of it.pass after. Full suite: 989 tests.
That gives this workaround a defined end of life: once the fix ships in iOS, it
is dead weight and can be removed.
The change
Behind a new
imap_client_workaroundsvalue,empty-partial-as-quoted,get_prefix()writes an empty string as""instead of{0}. The ABNF hasstring = quoted / literal, so this changes the encoding of the reply, not thereply.
Arguably it is also a consistency fix. Dovecot already prefers
""for an emptystring everywhere else:
imap_append_nstring()cannot produce a literal for one(its character loop never runs), and
imap_append_string_for_humans()has anexplicit
size == remove_count -> str_append(dest, "\"\"")case.get_prefix()was the exception — understandably so, since it writes the headerfor data that is streamed out afterwards, and a literal is the only string form
that can carry arbitrary octets. Zero is the one size where the two forms are
interchangeable.
The workaround is deliberately not restricted to partial fetches: the defect
is in the framing of any zero-length literal, so a plain
FETCHof an emptysection is affected too. That does make the setting name narrower than the
behaviour — happy to rename it to
empty-string-as-quotedif you prefer.Testing
Over a full session in the shape iOS Mail sends (
ID,COMPRESS DEFLATE,SELECT,RFC822.SIZE/BODYSTRUCTURE, then ten pipelinedBODY.PEEK[1]<offset.393216>) against a 6.9 MB message whose section[1]is285 KB:
Also checked: a plain
FETCHof an empty MIME part (covered), a run withCOMPRESS=DEFLATEnegotiated (unaffected), and a control run of the samepackage with the workaround off (unchanged, nine zero-length literals, no
crashes).
Running on my production server since 2026-09-08. On the wire:
Applies cleanly to 2.4.4, 2.4.5 and
main.Prior art
tb-lsub-flags(1c3e6a4, 2011) has the same shape: enum bit inimap-settings.h, list entry inimap-settings.c, behaviour change in thecommand file. The fourth file it touched,
doc/example-config/conf.d/20-imap.conf,no longer exists after 241e4b3.