Connection continuity and recovery

[rdmark]

Connection continuity and recovery: design and plan

Scope

Server shutdown, unreliable reconnects, active open files, locked files, and Reconnect UAM support are parts of one connection-continuity design. The design must restore a usable AFP session without treating an interrupted operation as if it had definitely succeeded.

This plan consolidates the recovery problem statement tracked in issue #319, together with shutdown handling and Reconnect UAM support.

The close-session teardown fix preserves FUSE-owned volume and open-fork state when a server closes a session. It does not make existing AFP fork IDs valid after a reconnect; those IDs are session-specific. Reopening active forks and replaying operations therefore need explicit recovery rules.

transport fails or server announces shutdown
        ↓
preserve local state; fail operations with an unknown outcome
        ↓
verify server identity and resume or authenticate
        ↓
reopen volumes and invalidate per-session caches
        ↓
restore eligible open forks and persistent locks
        ↓
resume ordinary operations

Current behavior

• Attention packets identify shutdown and crash notifications, but currently disconnect immediately even when a countdown is supplied.
• AFPATTN_NORECONNECT is defined but ignored.
• Transport loss fails pending requests with EIO, resets desktop references, and leaves the server eligible for reconnect on a later request.
• Reconnect verifies the server signature, starts a new DSI session, logs in, resumes its session token where supported, and reopens volumes.
• Open FUSE handles retain stale AFP fork IDs after reconnect.
• Directory-ID cache entries currently survive transport loss even though the volume session has changed.
• The current byte-range locks are acquired for individual reads and writes, then released. There is no persistent FUSE lock journal to replay.
• DHX and DHX2 do not retain the UAM-specific key material required by the Reconnect UAM.

Architecture

1. Recovery coordinator

Give each server an explicit lifecycle and a single recovery owner:

• CONNECTED
• QUIESCING — shutdown notice received, optionally with a deadline
• DISCONNECTED
• RECOVERING
• RECONNECT_BLOCKED — server requested no automatic reconnect
• SUSPENDED — user-requested suspension
• FAILED — recovery needs user intervention

The coordinator records a reason such as EOF, I/O error, protocol error, close-session, shutdown, crash, identity mismatch, or authentication failure. It serializes recovery so callers cannot race to create multiple sessions.

2. Session and volume recovery

The existing signature verification, DSI open, login, session token, andvolume reopen logic is the base recovery path. Recovery must then clear all per-session state, including desktop references and directory-ID caches.

Authentication should be represented as a resume strategy:

1. session-token resume;
2. Reconnect UAM, with volatile UAM-specific secret material; or
3. fresh login using a valid credential source.

Do not silently fall back to fresh authentication after an identity mismatch or authentication failure.

3. Open-fork recovery journal

The open-fork list must retain enough information to reopen a fork without using a stale directory ID:

• canonical AFP path;
• data or resource fork type;
• AFP access and deny modes;
• FUSE open flags and recovery generation; and
• state: live, restoring, stale, or failed.

After the volume reopens, resolve the pathname again, issue FPOpenFork, and atomically replace the fork ID. A deleted, renamed, inaccessible, or conflicting fork becomes stale and returns ESTALE or EIO; it must never be silently redirected.

4. Operation and locking rules

An in-flight request that loses transport has an unknown outcome. Until replay cache behavior is modeled and tested, do not replay writes, creates, renames, deletes, metadata changes, or flushes. Return EIO for the interrupted request, then allow later operations on successfully recovered handles.

Operation-scoped byte-range locks need no replay because they are released at the end of each read or write. If persistent POSIX/FUSE locks are introduced, add a separate lock journal keyed by handle/path, owner, range, and lock type; reacquire only after fork restoration and report conflicts to the caller.

Decisions to make

Decision	Options	Recommended default
Shutdown-notice timing	Disconnect immediately; remain connected until actual close	Remain connected, report the countdown, and disconnect on close/EOF.
AFPATTN_NORECONNECT	Ignore; prohibit all reconnects; block automatic reconnects only	Block automatic reconnects; let explicit afpc fs resume override it.
In-flight requests	Retry automatically; fail	Fail with EIO. Retrying writes requires proven replay-cache semantics and request classification.
Open FUSE handles after reconnect	Transparently reopen; retain but fail; invalidate	Reopen eligible handles; return ESTALE/EIO when a handle cannot be restored.
Directory-ID cache	Preserve; invalidate per volume	Invalidate on transport loss and volume reopen.
User-visible state	Log only; retain state for status	Retain disconnect reason, shutdown deadline, and reconnect policy and expose them in status.

Recommended implementation phases

1. Add recovery state, disconnect reason, shutdown deadline, automatic-reconnect inhibition, status output, and fault-injection tests.
2. Honor AFPATTN_NORECONNECT; report a shutdown countdown without tearing down the connection prematurely. Invalidate directory-ID caches on actual transport loss.
3. Convert the open-fork list into a recovery journal and mark handles stale until automatic fork restoration is available.
4. Restore active forks after session and volume recovery, with correct synchronization against FUSE close and concurrent I/O.
5. Add a resume-strategy abstraction and implement Reconnect UAM material storage for the UAMs that support it.
6. Add persistent-lock recovery only if persistent FUSE locking is in scope.
7. Add replay-cache support only after write outcome and retry semantics are specified and tested.

First decision required

Choose the countdown policy: should a shutdown attention notice make the mount unavailable immediately, or should the client continue serving requests until the server actually closes the connection? The recommended policy is to remain connected and report the countdown, then transition to recovery on actual close or transport failure.