added true async in zfs filesystem by adding a thin async zpl proxy

[tiehexue]

Motivation and Context

Long story short, this PR is another way to add true async to zfs. There are a lot discussion in #18684 , refer to it for more background.

Description

Now we have three PR for true async IO in zfs, #18684 is the deepest one which relies on zio_nowait, #10377 is in the middle which try to build async dmu, now this PR is on the top. It just adds a new zpl_iter_read_async to replace zpl_iter_read in zpl_file_operations as .read_iter.

Two module parameter added: zfs_async_dio_enabled and zfs_async_read_task_depth. The first one is a switch to enable or disable async, once enabled, the second one defined how many threads will be created as async context. The first one could be merge to the second one if necessary.

If it is a async IO from libaio/io_uring and also direct IO, async path is active, the request is submit to the taskq, and all code path are same with sync one, except ki_complete must be called. So from the data integrity view, it is same as sync path.

Most of code changes comes from calling zfs_setup_direct which pins user user pages before submit to taskq. Other code are self-explained.

Honestly, I do not like this way. It is like a fake. However, the other two ways have much more problems. The deepest one have to make zpl, vfs, dmu code into two parts, ones before zio_nowait(root zio), the other called in callbacks, which not easy to ensure data integrity, even for read. The middle one is not possible too, dmu is already complex enough. So this PR is the only way I think possible.

And again, as I learned from the community, using fio as an example, numjobs=16 iodepth=1 in sync VS. numjobs=1 iodepth=16 in async, the IOPS are comparable. And in the newly added test, I run a numjobs*iodepth matrix, got following nubmers (A 8-core ubuntu26 vm with zfs_async_read_task_depth 32):

01:22:32.02 NOTE: O_DIRECT randread IOPS (fio libaio, direct=1, bs=128K):
01:22:32.02 NOTE:   numjobs  iodepth  sync     async
01:22:32.02 NOTE:   1      1      4806     2985
01:22:32.02 NOTE:   1      8      4494     7924
01:22:32.02 NOTE:   1      32      4599     18442
01:22:32.02 NOTE:   1      64      4910     23255
01:22:32.02 NOTE:   32      1      38375     24269
01:22:32.02 NOTE:   32      8      32224     30910
01:22:32.02 NOTE:   32      32      32838     30959
01:22:32.02 NOTE:   32      64      33702     32752
01:22:32.02 NOTE:   64      1      34440     26928
01:22:32.02 NOTE:   64      8      31403     30195
01:22:32.02 NOTE:   64      32      34309     32607
01:22:32.02 NOTE:   64      64      36468     33229

This result said to me "why do you want async, just add more threads in application"?

Anyway, I put this PR here, because I believed "true async" should be valuable, and later I may try async write in another commit or PR.

And one more thing, this PR does bring "overhead" for async infrastructure, but the IOPS improvement nearly same as #18684 which just using zio_nowait. I also tried a unbounded lockless per-worker stack in https://github.com/tiehexue/zfs/tree/async_zpl_lockless_stack , I did not see distinct performance up which I hope to see. All test are run in vm with loop devices.

How Has This Been Tested?

Test locally and in personal fork CI

Types of Changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[AntonHPE]

@tiehexue

And again, as I learned from the community, using fio as an example, numjobs=16 iodepth=1 in sync VS. numjobs=1 iodepth=16 in async, the IOPS are comparable.

Not exactly. You are measured "randread IOPS" in 128K size, too big for random... OLTP workloads usually operates 4-16K sizes. With 128K you are measured more bandwidth than IOPS.

Here is single PCIe 5.0 NVMe SSD (CM7-V 3.2TB), formatted 4096 bytes for data + 64 bytes for metadata, and encrypted by dm-crypt with LUKS2/AEAD with --integrity-inline option.

[root@qdevice ~]# fio --name=test --rw=randread --bs=8k --filename=/dev/mapper/crypt_nvme1 --direct=1 --numjobs=16 --iodepth=1 --exitall --group_reporting --ioengine=psync --runtime=60 --time_based
test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=psync, iodepth=1
...
fio-3.42-116-gc76c
Starting 16 processes
Jobs: 16 (f=16): [r(16)][100.0%][r=1750MiB/s][r=224k IOPS][eta 00m:00s]
test: (groupid=0, jobs=16): err= 0: pid=5650: Fri Aug 7 21:37:49 2026
read: IOPS=224k, BW=1748MiB/s (1833MB/s)(102GiB/60001msec)
clat (usec): min=21, max=717, avg=71.22, stdev=16.65
lat (usec): min=21, max=717, avg=71.32, stdev=16.65
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 53],
| 30.00th=[ 59], 40.00th=[ 75], 50.00th=[ 76], 60.00th=[ 76],
| 70.00th=[ 77], 80.00th=[ 79], 90.00th=[ 82], 95.00th=[ 99],
| 99.00th=[ 127], 99.50th=[ 133], 99.90th=[ 157], 99.95th=[ 167],
| 99.99th=[ 186]
bw ( MiB/s): min= 1726, max= 1762, per=100.00%, avg=1747.99, stdev= 0.48, samples=1904
iops : min=220949, max=225618, avg=223742.29, stdev=62.03, samples=1904
lat (usec) : 50=1.66%, 100=93.51%, 250=4.83%, 500=0.01%, 750=0.01%
cpu : usr=0.79%, sys=2.55%, ctx=13422684, majf=0, minf=48290
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=13422672,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0.00ns, window=0.00ns, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=1748MiB/s (1833MB/s), 1748MiB/s-1748MiB/s (1833MB/s-1833MB/s), io=102GiB (110GB), run=60001-60001msec
[root@qdevice ~]#
[root@qdevice ~]#
[root@qdevice ~]#
[root@qdevice ~]# fio --name=test --rw=randread --bs=8k --filename=/dev/mapper/crypt_nvme1 --direct=1 --numjobs=1 --iodepth=16 --exitall --group_reporting --ioengine=libaio --runtime=60 --time_based
test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=16
fio-3.42-116-gc76c
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=2417MiB/s][r=309k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=5742: Fri Aug 7 21:40:04 2026
read: IOPS=312k, BW=2439MiB/s (2557MB/s)(143GiB/60001msec)
slat (nsec): min=852, max=1053.9k, avg=1529.11, stdev=878.82
clat (usec): min=7, max=1237, avg=49.49, stdev=25.80
lat (usec): min=16, max=1239, avg=51.02, stdev=25.75
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 21], 10.00th=[ 22], 20.00th=[ 25],
| 30.00th=[ 29], 40.00th=[ 32], 50.00th=[ 38], 60.00th=[ 56],
| 70.00th=[ 69], 80.00th=[ 79], 90.00th=[ 88], 95.00th=[ 93],
| 99.00th=[ 99], 99.50th=[ 101], 99.90th=[ 106], 99.95th=[ 112],
| 99.99th=[ 139]
bw ( MiB/s): min= 2321, max= 2504, per=100.00%, avg=2438.59, stdev=44.63, samples=120
iops : min=297136, max=320600, avg=312139.00, stdev=5712.55, samples=120
lat (usec) : 10=0.01%, 20=4.24%, 50=51.70%, 100=43.38%, 250=0.68%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%
cpu : usr=12.37%, sys=39.90%, ctx=2521002, majf=0, minf=60682
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=18728340,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0.00ns, window=0.00ns, percentile=100.00%, depth=16

[mjc]

Reusing zfs_read_impl() eliminates most of the "forgot one obscure read semantic" bugs. Awesome.

In general I like this a lot more, but there's still a lot to do.

• zpl_async_read_queue() calls zfs_setup_direct() before dispatch. that ends up pinning the whole userspace DIO buffer. after, it enqueues the op onto a global taskq, that taskq has a fixed number (default 4) of workers, and that queue is allowed to grow unbounded. so that goes from active workers × request size = pinned memory to queued requests × request size = pinned memory. this must be fixed. bounding the taskq after the pages are pinned isn’t enough; admission needs to happen before the pin.

• the taskq is global and head-of-line blocking. should probably be per-pool.

• zpl_async_read_task calls zpl_file_accessed even when _impl failed.

• the module param looks writable (0644) but it is only called by zpl_async_read_init, so runtime changes don't seem to do anything.

• the benchmark above uses zfs_async_read_task_depth=32, but the code defaults to 4. so we’re not seeing performance for the proposed default. I would benchmark the default too, and probably sweep task depth against actual hardware rather than picking it from the VM result.

• zfs_async_dio_enabled should default off

• pre-running zfs_setup_direct means zfs_read_impl skips it whenever UIO_DIRECT is on, so in those cases the setup_direct stuff gets frozen at queue time. this doesn't follow the "same code path for shared things" idea we were discussing. probably prudent to audit everything in zfs_setup_direct and see if all of it is fine with an arbitrarily long queue.

• teardown mechanism is much better than before the cross-thread rrmlock is gone.

• copying iov_iter by value and hanging onto the AIO request until ki_complete is much better than the previous code.

The taskq proxy also puts another concurrency scheduler above ZIO. zfs_async_read_task_depth now caps how many requests can actually enter the normal read path at once. That may explain some of the gap you’re seeing: with depth=64 and task depth=32, half the requests can be waiting for a worker rather than being visible to ZIO, while increasing task depth adds scheduler/context-switch overhead. I’d profile this before concluding sync threads and async are equivalent.

The unbounded queue/global taskq issues are correctness and isolation problems.

the extra taskq handoff and worker concurrency cap are the parts I am guessing affect the benchmarks.

[tiehexue]

@mjc Thanks for your quick review, I did a force push for further review, keep only one-commit so far.

Reusing zfs_read_impl() eliminates most of the "forgot one obscure read semantic" bugs. Awesome.

In general I like this a lot more, but there's still a lot to do.

zpl_async_read_queue() calls zfs_setup_direct() before dispatch. that ends up pinning the whole userspace DIO buffer. after, it enqueues the op onto a global taskq, that taskq has a fixed number (default 4) of workers, and that queue is allowed to grow unbounded. so that goes from active workers × request size = pinned memory to queued requests × request size = pinned memory. this must be fixed. bounding the taskq after the pages are pinned isn’t enough; admission needs to happen before the pin.
--- Currently, I add zfs_async_read_max_inflight as a gate which limits the pinned memory.

the taskq is global and head-of-line blocking. should probably be per-pool.
--- this is done by per-pool

zpl_async_read_task calls zpl_file_accessed even when _impl failed.
--- this is fixed.

the module param looks writable (0644) but it is only called by zpl_async_read_init, so runtime changes don't seem to do anything.
--- changed to 0444

the benchmark above uses zfs_async_read_task_depth=32, but the code defaults to 4. so we’re not seeing performance for the proposed default. I would benchmark the default too, and probably sweep task depth against actual hardware rather than picking it from the VM result.
--- this is changed to 4 for CI which is 2-vcpu, now leave it to 32.

zfs_async_dio_enabled should default off
--- this is for regression testing in CI.

pre-running zfs_setup_direct means zfs_read_impl skips it whenever UIO_DIRECT is on, so in those cases the setup_direct stuff gets frozen at queue time. this doesn't follow the "same code path for shared things" idea we were discussing. probably prudent to audit everything in zfs_setup_direct and see if all of it is fine with an arbitrarily long queue.
--- this is done by called zfs_dio_pin_pages (newly added), even without set UIO_DIRECT. However, some code is added to do unpin.

The taskq proxy also puts another concurrency scheduler above ZIO. zfs_async_read_task_depth now caps how many requests can actually enter the normal read path at once. That may explain some of the gap you’re seeing: with depth=64 and task depth=32, half the requests can be waiting for a worker rather than being visible to ZIO, while increasing task depth adds scheduler/context-switch overhead. I’d profile this before concluding sync threads and async are equivalent.
--- Yes, this is a fundamental point.

the extra taskq handoff and worker concurrency cap are the parts I am guessing affect the benchmarks.
--- Unless go back to zio_nowait, we need extra taskq.

[tiehexue]

@AntonHPE Thanks for your following up. I had created packages include almalinux10 at https://github.com/tiehexue/zfs/releases/tag/w2.5.0 . Let me know if the package can be downloaded and tested in your real hardware.

There are three new module parameters:

1. zfs_async_dio_enabled = 1;
   If set to 0, no more async. It is a runtime value.

2. zfs_async_read_task_depth = 32;
   How many threads are created per each zfs pool. This is read only, set when module loading. I am not sure which value will be better. The threads are dynamic, recycled when idle.

3. zfs_async_read_max_inflight = 256;
   If inflights greater than zfs_async_read_max_inflight, it will go to sync path. This can be set a larger value, and is a runtime value.

Below is the latest result in my local vm (8-core, ZTS test pool, raidz on 3 file-backed loop devices). As you pointed out, set record size to 4, the numbers looks good. You may test with more configuration on numjobs, iodepth, recordsize for both sync and async. Just ask AI for a script to automate it :-)

10:17:26.48 NOTE: O_DIRECT randread IOPS (fio libaio, direct=1, bs=4K):
10:17:26.48 NOTE:   numjobs  iodepth  sync     async
10:17:26.48 NOTE:   1      1      3997     2590
10:17:26.48 NOTE:   1      8      3720     10381
10:17:26.48 NOTE:   1      32      4036     37832
10:17:26.48 NOTE:   1      64      4902     45770
10:17:26.48 NOTE:   32      1      63417     52118
10:17:26.48 NOTE:   32      8      60809     63827
10:17:26.48 NOTE:   32      32      62200     54644
10:17:26.48 NOTE:   32      64      60072     63324

[AntonHPE]

@tiehexue

Let me know if the package can be downloaded and tested in your real hardware.

I downloaded the package, but I will be able to run tests on Wednesday and beyond.

I'm currently on

[root@memverge3 repo]# cat /etc/*release
NAME="Rocky Linux"
VERSION="10.2 (Red Quartz)"
RELEASE_TYPE="stable"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="10.2"
PLATFORM_ID="platform:el10"
PRETTY_NAME="Rocky Linux 10.2 (Red Quartz)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:10::baseos"
HOME_URL="https://rockylinux.org/"
VENDOR_NAME="RESF"
VENDOR_URL="https://resf.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2035-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-10"
ROCKY_SUPPORT_PRODUCT_VERSION="10.2"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="10.2"
Rocky Linux release 10.2 (Red Quartz)
Rocky Linux release 10.2 (Red Quartz)
Rocky Linux release 10.2 (Red Quartz)
[root@memverge3 repo]#
[root@memverge3 repo]# uname -r
6.12.0-211.44.1.el10_2.x86_64
[root@memverge3 repo]#

What type of installation should I select, DKMS (epel-testing/10.2/x86_64/)

or

KMOD (epel-testing/10.2/kmod/x86_64/) ?

[mjc]

so the taskq is no longer global, great - but I meant pool as in SPA/storage resource domain, not one taskq per zfsvfs - zfsvfs is per-dataset I think?

zfs_async_read_max_inflight makes sense as an admission gate. it’s per-request, though, and requests can have radically different sizes, so the actual pinned-memory bound is roughly 256 * max accepted request size. probably worth sanity-checking that default.

the pin-only split seems not right:

submission calls zfs_dio_pin_pages(&aio->uio, UIO_READ); which pins but also clears UIO_DIRECT intentionally -> the worker calls read_impl -> calls zfs_setup_direct which re-runs the eligibility, and can reuse the pre-pinned pages if DIO is still eligible.

However, if DIO is not eligible by the time execution is ready, bc cached data appeared, mmap state changed or something similar, then zfs_setup_direct might leave UIO_DIRECT cleared, and then normal read path can continue from a taskq thread using the saved userspace iterator.

I think it should decide or fallback before handing execution to a context that cannot safely perform ordinary userspace copies - but really I'm out of my depth there. it could instead find a way to guarantee that completion accesses the pinned pages rather than user VA. I have no idea which approach is correct or if there's an obvious one I'm missing.

sync atime behavior seems to accidentally have been changed - it now stamps only when a range lock was obtained and positive bytes were read.

it would still be good to default-off the tunable and have the benchmark turn it on.

[tiehexue]

@tiehexue

Let me know if the package can be downloaded and tested in your real hardware.

I downloaded the package, but I will be able to run tests on Wednesday and beyond.

I'm currently on

[root@memverge3 repo]# cat /etc/*release NAME="Rocky Linux" VERSION="10.2 (Red Quartz)" RELEASE_TYPE="stable" ID="rocky" ID_LIKE="rhel centos fedora" VERSION_ID="10.2" PLATFORM_ID="platform:el10" PRETTY_NAME="Rocky Linux 10.2 (Red Quartz)" ANSI_COLOR="0;32" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:rocky:rocky:10::baseos" HOME_URL="https://rockylinux.org/" VENDOR_NAME="RESF" VENDOR_URL="https://resf.org/" BUG_REPORT_URL="https://bugs.rockylinux.org/" SUPPORT_END="2035-05-31" ROCKY_SUPPORT_PRODUCT="Rocky-Linux-10" ROCKY_SUPPORT_PRODUCT_VERSION="10.2" REDHAT_SUPPORT_PRODUCT="Rocky Linux" REDHAT_SUPPORT_PRODUCT_VERSION="10.2" Rocky Linux release 10.2 (Red Quartz) Rocky Linux release 10.2 (Red Quartz) Rocky Linux release 10.2 (Red Quartz) [root@memverge3 repo]# [root@memverge3 repo]# uname -r 6.12.0-211.44.1.el10_2.x86_64 [root@memverge3 repo]#

What type of installation should I select, DKMS (epel-testing/10.2/x86_64/)

or

KMOD (epel-testing/10.2/kmod/x86_64/) ?

Is it almalinux 10, you test before? You can download, just before you test it. I may update the package for coming changes.

[tiehexue]

@mjc Thanks for review again. The per-pool, atime, defualt off should be fixed.

The zfs_async_read_max_inflight I set to default 1024, even larger than 256, it is for benchmark, so 32 numjobs and 32 iodepth does not fallback to sync. Let me make another commit once we have a better value, or, make both zfs_async_read_max_inflight and zfsvfs->z_async_dio_inflight as data length.

For the pin page issue, call zfs_setup_direct before submission, there still a chance that fallback to sync in the thread. That is why I just do the pin only with a new function.

"guarantee that completion accesses the pinned pages rather than user VA", I thought this too. Yes, we already make room for the read. If fallback to sync, despite other potential error, the pin is a waste. Let me go on this way, and if good, make another commit.

[tiehexue]

@mjc two new commit for zfs_async_read_max_inflight and pin page issue.

Now zfs_async_read_max_inflight is bytes total and checked per dataset. This could be "prompted" per pool, or a property of pool or dataset.

For pinned page, now if dio is declined, the pinned page is still used, so no user VA touched in async context.

[tiehexue]

@AntonHPE checkout latest build before you test at https://github.com/tiehexue/zfs/releases .

Now, zfs_async_read_max_inflight is default to 512M, as total inflight request size. An IO is fallback to sync path if inflight is bigger than zfs_async_read_max_inflight. This valus should be large enough for benchmark.

[mkhllr]

This is my first comment here, so briefly: I worked through the crash-class bugs
on #18684 and I have #18844 open against the same read path, so I have been
building this branch and running libaio against it. Measurements below are at
d6918a687, two commits past the head @mjc reviewed.

The pinned-page approach came out of @mjc's 08-08 review and the implementation
is @tiehexue's. I am reporting measurements against it. Three results: the
pinned-page copy fixes a real failure, the eligibility decision is still being
taken on freed memory, and the teardown drain deadlocks in a way that is my
fault as much as anyone's.

Rig: Linux 6.8.0-137 guest, --enable-debug, pool on a single virtio disk,
loaded srcversion checked against the tree before every run
(F85E87B380D3BC946E8D8DA for the stock build of zfs-2.4.99-919_gd6918a687;
the patched build in the second section reports its own). Conditions are
alternated per trial rather than run in blocks. Happy to hand over the scripts.

The pinned-page copy fixes the non-page-aligned EOF tail

I had not raised this one before, and it is already fixed by 009665d65, so
this is just confirmation that the fix does what it says. Before that commit, a
request reaching a file whose size is not a multiple of the page size returned
EFAULT with the bytes at and past the last page-aligned boundary never
written: zfs_read_impl() clears UIO_DIRECT for the dio_remaining_resid
tail, and copy_to_iter() has no user mm on the taskq thread.

Ten reps per condition per row, async alternated with sync:

file size	segments	request	tail	async=1	sync control
40000	1 (PREAD)	40960	3136	10/10 correct	10/10
40000	10 (PREADV)	40960	3136	10/10 correct	10/10
12000	4 (PREADV)	16384	3808	10/10 correct	10/10
4196	2 (PREADV)	8192	100	10/10 correct	10/10

The PREAD row submits one contiguous buffer, so it builds an ITER_UBUF with
no iovec array in play; that is how it stays independent of the iterator problem
in the next section. The 40000-byte rows previously failed with the first wrong
byte at exactly 36864. The debug counter added in 009665d65 advanced on all ten
reps of every row, so the tail demonstrably took the pinned-page branch.

The eligibility decision is still taken on freed memory

The EFAULT is gone. The decline that produced it is not. 25 trials per
condition, 16 KiB page-aligned file, four page-sized page-aligned segments,
IOCB_CMD_PREADV:

condition	correct	EFAULT	EINVAL	served via the pinned-page ARC copy
zfs_async_dio_enabled=1, zfs_dio_strict=0	25	0	0	13 of 25
zfs_async_dio_enabled=0, zfs_dio_strict=0	25	0	0	0 of 25
zfs_async_dio_enabled=1, zfs_dio_strict=1	11	0	14	0 of 25

The middle row is the control: the identical iovec, evaluated in the submitting
thread, is page-aligned every time. The third row says which check decides
otherwise, because zfs_dio_strict gates exactly one branch in the whole tree:

	if (!zfs_uio_page_aligned(uio) ||
	    !zfs_uio_aligned(uio, PAGE_SIZE)) {
		/*
		 * Misaligned requests can be executed through the ARC as
		 * uncached I/O.  But if O_DIRECT was set by user and we
		 * were set to be strict, then it is a failure.
		 */
		if ((*ioflagp & O_DIRECT) && zfs_dio_strict)
			error = SET_ERROR(EINVAL);
		goto out;
	}

zfs_uio_aligned() only reads uio_loffset and uio_resid, so the check
getting a different answer on the worker is zfs_uio_page_aligned(), which for
UIO_ITER is iov_iter_alignment(uio->uio_iter). For ITER_IOVEC that walks
the backing array, and in v6.8 fs/aio.c owns that array only for the duration
of aio_read():

	struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs;
	...
	ret = aio_setup_rw(ITER_DEST, iocb, &iovec, vectored, compat, &iter);
	if (ret < 0)
		return ret;
	ret = rw_verify_area(READ, file, &req->ki_pos, iov_iter_count(&iter));
	if (!ret)
		aio_rw_done(req, call_read_iter(file, req, &iter));
	kfree(iovec);

The single-segment shape is the control for that: ITER_UBUF keeps the user
address in the iterator and leaves no array behind, and 25 trials of it gave 0
declines and 0 EINVAL in all three conditions. I measured one and four
segments here; where exactly the boundary falls comes from reading
__import_iovec() rather than from a segment scan at this head. nr_segs == 1
goes to __import_iovec_ubuf(), two through UIO_FASTIOV (8) use aio_read()'s
stack inline_vecs[] with *iovp set to NULL so the kfree() is a no-op, and
above eight is a kmalloc()ed block that the kfree() releases. Both of the
multi-segment cases are gone by the time the worker looks.

The comment in zpl_async_read_queue() states the assumption this rests on:

	 * The iov_iter passed by libaio lives on the io_submit() stack; copy
	 * it by value and point the uio at our copy.  The backing iovec array
	 * is kept valid by the aio/io_uring request until ki_complete().

For libaio the second sentence does not hold, which is what the numbers are
showing.

Two consequences, both separate from the corruption that is now fixed. With
zfs_dio_strict=1, a valid page-aligned O_DIRECT read failed with EINVAL on
14 of 25 trials. With the default zfs_dio_strict=0, 13 of 25 were silently
served through the ARC, which defeats the O_DIRECT request without telling the
caller. The rate is not stable, as you would expect from a read of released
memory: a 16 KiB four-segment row in a different run on the same build declined
3 times in 10.

To check that this really is the cause rather than something correlated with it,
I removed only that walk and re-ran the identical measurement. The pre-pinned
shortcut currently sits below the alignment check, so the walk happens before it
is reached. Alignment was established against a live iterator at submission time
and cannot change afterwards, which is the same reasoning
zpl_async_read_queue() already gives for hoisting the check there, so it can be
skipped once the pages are pinned:

	if (uio->uio_dio.pages == NULL &&
	    (!zfs_uio_page_aligned(uio) ||
	    !zfs_uio_aligned(uio, PAGE_SIZE))) {

That one change takes all three conditions to clean, 25 trials each:

condition	correct	EFAULT	EINVAL	served via the pinned-page ARC copy
zfs_async_dio_enabled=1, zfs_dio_strict=0	25	0	0	0 of 25
zfs_async_dio_enabled=0, zfs_dio_strict=0	25	0	0	0 of 25
zfs_async_dio_enabled=1, zfs_dio_strict=1	25	0	0	0 of 25

13 of 25 and 14 of 25 both go to 0 of 25. On the two async rows the Direct I/O
path is now taken on every trial: the counter fires on any non-Direct delivery
once the pages are pinned, and a 16 KiB page-aligned file has no tail, so zero
counted copies alongside 25 correct reads leaves Direct I/O as the only route.

Nothing else changes. uio_dio.pages is NULL at this point for synchronous reads
and for the write path, since both of those pin inside zfs_setup_direct()
rather than before it, so the guard reduces to the existing check for them.
zpl_async_read_queue() is the only caller that pins early. The control row
shows the synchronous read side of that; the write path is by inspection, as I
did not exercise an O_DIRECT write here.

I am offering this as evidence about the cause rather than as a patch, though the
direct and async groups do pass with it in place, all 22 tests on the same
rig, which is not the same thing as a full ZTS run. You may well prefer to
restructure rather than add a guard. zn_has_cached_data() and the
zfs_dio_enabled / os_direct tests are genuinely mutable and should still be
re-evaluated on the worker. Declining there is safe now, which is what
009665d65 bought.

The teardown drain deadlocks, and the broken precondition is mine

The z_async_dio_inflight drain counter is a design I suggested on #18684, and I
gave it a precondition when I did:

completions release the hold as their last touch of the zfsvfs;
zfsvfs_teardown() drains the counter right after taking the teardown lock as
writer. At that point no new async I/O can be submitted, and completions never
take the teardown lock, so the wait cannot deadlock.

On the async read path both halves of that are now false. zpl_async_read_task()
calls zpl_file_accessed() before zpl_async_read_rele(), so the hold is not
the last touch, and the atime update reaches zfs_dirty_inode(), which takes the
teardown lock for read. A zfs rollback against a dataset with a read in flight
wedges both tasks permanently:

--- pid 263662 (zfs) state=D ---
[<0>] cv_wait_common+0x185/0x2f0 [spl]
[<0>] __cv_wait+0x15/0x30 [spl]
[<0>] zfsvfs_teardown+0xdb/0x540 [zfs]
[<0>] zfs_suspend_fs+0x10/0x20 [zfs]
[<0>] zfs_ioc_rollback+0x11f/0x1d0 [zfs]
[<0>] zfsdev_ioctl_common+0x572/0xad0 [zfs]
[<0>] zfsdev_ioctl+0x57/0xf0 [zfs]
[<0>] __x64_sys_ioctl+0xa3/0xf0
[<0>] x64_sys_call+0x12a3/0x25a0
[<0>] do_syscall_64+0x87/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
--- pid 263609 (aiodio) state=S ---
[<0>] read_events+0x205/0x210
[<0>] do_io_getevents+0x76/0x100
[<0>] __x64_sys_io_pgetevents+0xd6/0x190
[<0>] x64_sys_call+0x22bc/0x25a0
[<0>] do_syscall_64+0x87/0x180
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
--- taskq pid 263614 (zpl_async_read_) ---
[<0>] cv_wait_common+0x185/0x2f0 [spl]
[<0>] __cv_wait+0x15/0x30 [spl]
[<0>] rrw_enter_read_impl+0x76/0x290 [zfs]
[<0>] rrm_enter_read+0x4f/0x80 [zfs]
[<0>] zfs_dirty_inode+0x111/0x4b0 [zfs]
[<0>] zpl_dirty_inode+0x2c/0x90 [zfs]
[<0>] __mark_inode_dirty+0x5e/0x3b0
[<0>] generic_update_time+0x50/0x60
[<0>] touch_atime+0xf6/0x120
[<0>] zpl_async_read_task+0x11e/0x170 [zfs]
[<0>] taskq_thread+0x375/0x8e0 [spl]
[<0>] kthread+0xf2/0x120
[<0>] ret_from_fork+0x47/0x70
[<0>] ret_from_fork_asm+0x1b/0x30

The middle stanza is the submitting process, parked in io_getevents() waiting
for a completion that will never arrive. Neither task returned, and the hung-task
detector kept escalating past 151 seconds for both of them. I have run this once.
It armed on the first attempt, with a 20 s zinject -E delay holding the read in
flight, and the run recorded both that the read was still in flight when the
rollback started and that the injected delay fired, so the overlap was genuine.
The dataset had atime=on relatime=off to make the update due on every read;
under the default relatime it is due far less often.

I do not think reordering the two calls is enough on its own, which is why I am
not sending a patch. Releasing the hold first would let teardown proceed while
the task still touches the zfsvfs, and the atime write-back wants the teardown
lock wherever it runs. Draining before taking the write lock, or keeping the
atime update out of the completion context, both look plausible from here and
both are your call rather than mine.

One thing worth flagging about how reachable this is. The atime update only runs
after a successful read, so before 009665d65 the multi-segment reads that
failed never got that far. Now that they succeed they do reach it, so the
pinned-page fix widens the window for this cycle.

Why the direct and async groups pass on this build

I ran both groups against the stock head on the same rig: all 22 tests pass, 28
of 28 result lines counting setup and cleanup, each with a timed result line
rather than a skip. That includes all three async tests and
dio_unaligned_filesize. So the current coverage catches neither of the first
two findings. async_read_003_pos does run here, and the counter it needs is
present in CI too, since the QEMU test workflow builds with --enable-debug
(zfs-qemu.yml).

The reason looks to be request shape. The fio read invocations in async.kshlib
pass a single --filename with no vectored option, which is consistent with
single-buffer requests, and single-buffer is the shape that did not decline in
any of my trials. Rather than assert what fio submits internally I probed it.
Under zfs_dio_strict=1, async_read_001_pos's read invocation reproduced flag
for flag returns rc=0 with the pinned-copy counter unmoved across 64 MiB of
Direct I/O reads, while a four-segment request against the same build in the same
window returns EINVAL. The control says strict mode was armed, so the reading is that
fio's requests never reach that branch. I probed the libaio invocation; the other
engines rest on reading the test sources rather than on measurement.

Two cheap additions would close it, if you think it is worth covering. A vectored
async Direct I/O read, IOCB_CMD_PREADV with two or more page-aligned segments,
exercises the shape that fails. And an assertion in the opposite direction from
the one async_read_003_pos makes: for a request that should stay eligible,
zfs_async_read_pinned_copies must not move. The counter is a good instrument
for that, since it distinguishes "served by Direct I/O" from "quietly served by
the ARC", which is otherwise invisible from userspace.

The teardown cycle is not covered either. Both groups do export the pool,
async_reimport_pool() in async.kshlib and dio_unaligned_filesize, but in
each case after the I/O has finished, so nothing exercises a teardown that
overlaps an async read still in flight.

Two smaller things

You already said the in-flight accounting is per dataset. Two follow-ons. The
block comment above zpl_async_read_hold() still describes it as per pool
("EOPNOTSUPP if the pool is at its in-flight limit") while the counter it tests
is zfsvfs->z_async_dio_inflight, and with the default at 512 MiB the pinned
ceiling is 512 MiB times the number of mounted datasets in the pool, which seems
worth folding into the bound @mjc asked about.

The pinned-page copy is not limited to the async path. It triggers on
rw == UIO_READ && uio->uio_dio.pages != NULL && !(uio->uio_extflg & UIO_DIRECT),
which a plain synchronous Direct I/O read also satisfies for its EOF tail. My
sync control rows went through the new branch on all ten reps of every row. The
commit message says "make sync path use pinned page directly", which covers the
fallback inside the worker; what is not called out is that an ordinary
synchronous O_DIRECT read now reaches its tail through the pinned pages too. It
looks intentional and I did not find a case where it is wrong.

Overlap

My #18844 adds a flag on the file handle after a benign Direct I/O verify failure
and hangs off zpl_iter_read(). Queued async reads never reach that function, so
whichever of the two lands first, the other needs a look. For the record only.

[tiehexue]

@mkhllr thanks! I am not surprised you are here :-)

Give me some time to consolidate your comments. Would you like to provide a patch, especially test cases which bring bugs?