Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions pkgs/code_assets/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
## 1.3.0

- Validate that a bundled dynamic library was built for the target
architecture by reading its ELF, Mach-O (including fat), or PE header. A
recognized library built for the wrong architecture or in the wrong format
for the target OS is a validation error; unrecognized files only produce a
warning on the new optional `logger` of `CodeAssetExtension`,
`validateCodeAssetBuildOutput`, and `validateCodeAssetLinkOutput`, with a
pointer to file an issue.

## 1.2.1

- Avoid throwing a null-assertion error/exception when `input.config.code` is
Expand Down
10 changes: 8 additions & 2 deletions pkgs/code_assets/lib/src/code_assets/extension.dart
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
// BSD-style license that can be found in the LICENSE file.

import 'package:hooks/hooks.dart';
import 'package:logging/logging.dart';

import 'architecture.dart';
import 'c_compiler_config.dart';
Expand Down Expand Up @@ -55,6 +56,10 @@ final class CodeAssetExtension extends ProtocolExtension {
/// runtime libraries and memory allocators across the FFI boundary.
final Sanitizer? sanitizer;

/// Receives warnings from validations which are skipped rather than failed,
/// such as an unrecognized native library header.
final Logger? logger;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather have the logger from hooks_runner be used instead of a logger here in the constructor, but adding a logger argument to the validateXxx methods would be a breaking change.

See https://github.com/flutter/flutter/blob/e69e703dce29fda392f8024109c1961306cb365a/packages/flutter_tools/lib/src/isolated/native_assets/targets.dart#L237. We don't want to have to construct the same logger there as we did in https://github.com/flutter/flutter/blob/e69e703dce29fda392f8024109c1961306cb365a/packages/flutter_tools/lib/src/isolated/native_assets/native_assets.dart#L561

Maybe we can add the logger to the ValidationContext?

WDYT?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm wait the code assets extension creates the validation context, so it also doesn't have access to the logger.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed it's breaking even as an optional named param, since an external ProtocolExtension subclass that doesn't also declare it would fail invalid_override, so I won't touch those signatures. The additive way to your ValidationContext idea: add a mutable Logger? logger to ProtocolExtension, have NativeAssetsBuildRunner set e.logger before the override call in its existing for (final e in extensions) loop, and CodeAssetExtension forwards it into the context it builds. That covers your follow-up: the extension still creates the context, but now has the runner's logger to put in it, so Flutter never builds a second one. I'll leave the existing constructor logger as a fallback so the change stays additive.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction on the mechanism I proposed. Keeping CodeAssetExtension's existing final Logger? logger alongside a new logger on ProtocolExtension does not work: the subclass field shadows the base getter, so an e.logger assignment from the runner lands on the base field while every read dispatches to the subclass getter, a silent no-op (confirmed with a quick repro). The workable shape is a single storage: put Logger? logger; on ProtocolExtension, drop the field from CodeAssetExtension, and set this.logger in its constructor body so existing callers still pass it. If you would rather the runner not mutate a caller-owned field, a void setupValidationLogger(Logger logger) {} default hook does the same job.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a method to ProtocolExtension like setupValidationLogger is then not breaking because we made it a base class.

Then we could also consider something along the lines of:

base class ProtocolExtension {
  // ..
  late Logger _logger;

  Logger get logger => _logger;

  /// Sets up context by the hooks_runner that uses the protocol extensions.
  ///
  /// Guaranteed to be run before any other methods are called on this object.
  void setupLogger(Logger logger) {
    this._logger = logger;
  }
}

We might get passed the same protocol extension object in multiple build/link methods in one run. So we do end up setting the logger more than once potentially. So the field can't be final.

I think technically this is still a breaking change, because package:hooks_runner might resolve to an older version while package:code_assets might resolve to a newer version, causing the setupLogger never to be called. However, I don't think this will happen in Dart or Flutter because all packages are rolled at the same time.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added setupLogger on ProtocolExtension, a non-final late field with a logger getter, since the same extension object can be reused across a run. The hooks_runner calls it before any other method in both build and link. CodeAssetExtension's own logger field is removed so nothing shadows the base getter.


/// Constructs a [CodeAssetExtension].
CodeAssetExtension({
required this.targetArchitecture,
Expand All @@ -65,6 +70,7 @@ final class CodeAssetExtension extends ProtocolExtension {
this.iOS,
this.macOS,
this.sanitizer,
this.logger,
});

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to make the LibraryHeaderRecognizer pluggable. We're adding support for custom architectures and OSes in:

If we make it pluggable here, we can support multiple third-party recognizers (provided the flutter commandline tool re-exposes the pluggability).

Maybe we can structure the code in a way that we have MachOLibraryHeaderRecognizer, ElfLibraryHeaderRecognizer, PortableExecutableLibraryHeaderRecognizer.

And then

interface LibraryHeaderRecognizer {
  Future<LibraryHeader?> tryRecognize(Uri file);
}

final class LibraryHeader {
  final List<Architecture> architectures;
  final OS? os;
  // (for arm64 we might need to add an abi version field later as well here)

  // we can add a nullable field here later without it being a breaking change.
  LibraryHeader({required this.architectures, this.os});
}

As this API is part of the public API in order to make it pluggable/extensible, we should only use a Architecture and OS and skip exposing the BinaryFormat I think.

Hm, this design doesn't enable ${expectedFormat.displayName}. in the error message though. If you have a bright idea of how to make it pluggable and have the expected format supported let me know. I think it's hard because we need to both support adding new OSes as well as new architectures. So you don't know up front which LibraryHeaderRecognizer is supposed to give you an answer. Unless you start adding something like bool canReadFor({OS os, Architecture architecture}).

Maybe the API should be

interface LibraryHeaderRecognizer {
  Future<List<String>> validateLibrary({Uri file, OS os, Architecture architecture});
}

Or it should simply be a list of functions

CodeAssetExtension({
  libraryValidators List<Future<List<String>> Function({Uri file, OS os, Architecture architecture, Logger logger})>
});

And then if you recognize the magic bytes, and they are wrong, you can give an error. If no recognizer gives errors, then that's good.
Buuut, then we can't give an error on that no recognizer understood the file. Which we could do with the first design.

If you have a better idea, let me know!

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A validator API seems like the cleaner fit. I would keep it specific to native
libraries and avoid publishing a header model:

final class NativeLibraryValidationContext {
  final Uri file;
  final CodeConfig config;

  const NativeLibraryValidationContext({
    required this.file,
    required this.config,
  });
}

abstract interface class NativeLibraryValidator {
  Future<NativeLibraryValidation> validate(
    NativeLibraryValidationContext context,
  );
}

sealed class NativeLibraryValidation {
  const NativeLibraryValidation._();

  const factory NativeLibraryValidation.notRecognized() = _NotRecognized;
  factory NativeLibraryValidation.inconclusive(String reason) = _Inconclusive;
  const factory NativeLibraryValidation.matched() = _Matched;
  factory NativeLibraryValidation.rejected(Iterable<String> errors) = _Rejected;
}

CodeAssetExtension would take an
Iterable<NativeLibraryValidator> additionalLibraryValidators = const []
and snapshot it. These are trusted extensions supplied by the invoker, not by
the package that produced the hook output. As with the current check, they
would only run for existing DynamicLoadingBundled files.
Flutter would still need a separate re-exposure point before third-party
validators could be supplied through the flutter tool.

The built-ins and all additional validators would run in registration order.
The results would be combined as follows:

  1. Any rejected result wins, with all rejection errors collected.
  2. Otherwise, a matched result passes the check.
  3. Otherwise, inconclusive reasons produce one warning.
  4. If every result is notRecognized, we keep the current generic warning.

matched means that the validator is authoritative for every dimension in
the current validation contract, not just one header field. For this PR that
contract is the expected container family for the target OS and the target
architecture family. It is not a claim of complete OS or ABI compatibility.
A future arm64e subtype or ABI check would be a contract revision and would
need an explicit capability or version gate so that an older validator cannot
silently keep returning matched.

A validator should return notRecognized outside its domain and
inconclusive when it recognizes the file but cannot cover the whole current
contract. It may return matched or rejected only within the scope it fully
understands. The factories would reject an empty reason or error list and
would snapshot returned errors.

Each validator invocation would be wrapped independently. A synchronous or
asynchronous exception becomes a validation error for that validator, and the
remaining validators still run. It is not treated as notRecognized and does
not escape the hook runner's Result path.

A multi-architecture Mach-O is separate from that aggregation. The current
protocol invokes a hook once per target architecture and leaves
universal-binary assembly to the invoker, so a Mach-O containing more than one
architecture should be rejected as a protocol invariant before a custom
matched result can affect the outcome.

Validators would return diagnostics rather than log directly. The extension
can log once after aggregation, when it knows whether an inconclusive result
was resolved by another validator. Dart and Flutter currently construct the
extension without their runner logger, so those call sites would also need to
be wired before warnings are visible there.

This keeps BinaryFormat private. It also avoids treating ELF, Mach-O, or PE
as a complete OS/ABI compatibility claim. Does this contract match what you
had in mind?

I also found that a cache hit returns before _validate. I will route cached
outputs through the same validation as fresh outputs. A cached output read or
parse failure, or a semantic-validation failure, will invalidate both
output.json and the matching dependencies.dependencies_hash_file.json, so
the next invocation reruns the hook instead of reusing a poisoned cache. I
will cover that sequence and the requested real wrong-target hooks_runner
case with regression tests.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented this as NativeLibraryValidator in 8a67fb2. The extension snapshots invoker-supplied validators, runs the built-in and every additional validator in registration order, and accumulates rejections and validator failures. If any validator rejects or throws, those errors are reported and inconclusive warnings are suppressed. Otherwise, a match suppresses inconclusive warnings. The header model and BinaryFormat remain private.

7560bd4 makes multi-architecture Mach-O rejection independent of target OS and validator aggregation, so an additional match cannot override it. 136e3f2 also rejects a structurally valid fat header declaring more than 32 slices without reading or allocating the slice table. For a custom target architecture, the built-in validator remains inconclusive, preserving the future extension path for #3413.

23f70e0 routes cache hits through the same validation as fresh output. 933b5ef invalidates malformed or non-object cached JSON, and 136e3f2 does the same for an unknown cached status, for both build and link hooks. Read, parse, and semantic-validation failures remove both cache files, so the next invocation reruns the hook. The real wrong-architecture integration test builds an opposite-architecture dylib through native_toolchain_c.

Dart 3.14.0-33.0.dev passes dart tool/ci.dart --all, and Dart 3.11.0 passes dart tool/ci.dart --all --no-apitool --no-format locally.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, I like this design!

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep the recognizers internal for this PR. The built-in path's value is the X binary, but <os> expects Y message, which needs the OS-to-expected-format map the dispatcher owns. A tryRecognize(file) recognizer can't reconstruct it, and once recognition is fully pluggable the dispatcher can't know which recognizer should have answered, so it can't name the expected format. Your validateLibrary({file, os, architecture}) shape does get the target OS and could emit that message, but as you noted it loses the "no recognizer understood the file" error. Both are real trade-offs I'd rather resolve against a concrete caller: #3413 gives one, and the per-format split makes exposing the extension point then a small non-breaking add. So I lean lock-down now and revisit with #3413, or reserve the public shape now if you'd prefer. WDYT?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah locking it down and not exposing it in the CodeAssetsExtension for now SGTM.

It's nice that we already set the code up in a mostly extensible way 👍

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Locked it down. The public export and the CodeAssetExtension constructor parameter are both gone, along with the now-dead runtime injection plumbing. The per-format interface stays, so adding a validator once a concrete caller like #3413 arrives is a small non-breaking change.


@override
Expand Down Expand Up @@ -103,13 +109,13 @@ final class CodeAssetExtension extends ProtocolExtension {
Future<ValidationErrors> validateBuildOutput(
BuildInput input,
BuildOutput output,
) => validateCodeAssetBuildOutput(input, output);
) => validateCodeAssetBuildOutput(input, output, logger: logger);

@override
Future<ValidationErrors> validateLinkOutput(
LinkInput input,
LinkOutput output,
) => validateCodeAssetLinkOutput(input, output);
) => validateCodeAssetLinkOutput(input, output, logger: logger);

@override
Future<ValidationErrors> validateApplicationAssets(
Expand Down
203 changes: 203 additions & 0 deletions pkgs/code_assets/lib/src/code_assets/native_library_headers.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
// Copyright (c) 2026, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

import 'dart:io';
import 'dart:typed_data';

import 'architecture.dart';

/// The binary format of a native library file.
enum BinaryFormat {
/// The Executable and Linkable Format used on Linux, Android, and Fuchsia.
elf('ELF'),

/// The Mach object format used on macOS and iOS.
machO('Mach-O'),

/// The Portable Executable format used on Windows.
pe('PE');

const BinaryFormat(this.displayName);

/// The name of this format in messages.
final String displayName;
}

/// The result of reading a native library's header.
sealed class NativeLibraryHeader {
const NativeLibraryHeader();
}

/// A header whose binary format was recognized.
final class RecognizedHeader extends NativeLibraryHeader {
/// Constructs a [RecognizedHeader].
const RecognizedHeader(this.format, this.architectures, this.machineValues);

/// The binary format of the file.
final BinaryFormat format;

/// The architectures the file was built for, aligned with [machineValues].
///
/// A `null` entry means the format was recognized but its machine value is
/// not one of the [Architecture]s. Fat Mach-O files contain one entry per
/// slice; the other formats always contain exactly one.
final List<Architecture?> architectures;

/// The raw machine values from the header, for diagnostics.
final List<int> machineValues;
}

/// A file whose binary format was not recognized.
final class UnrecognizedHeader extends NativeLibraryHeader {
/// Constructs an [UnrecognizedHeader].
const UnrecognizedHeader();
}

/// Reads just enough of [file] to identify its binary format and the
/// architectures it was built for.
///
/// Never throws for file contents: anything that cannot be identified,
/// including short and empty files, is an [UnrecognizedHeader].
NativeLibraryHeader readNativeLibraryHeader(File file) {
final RandomAccessFile raf;
try {
raf = file.openSync();
} on FileSystemException {
return const UnrecognizedHeader();
}
try {
final length = raf.lengthSync();
final head = raf.readSync(64);
if (head.length >= 20 && _hasElfMagic(head)) return _readElf(head);
if (head.length >= 8) {
final bigEndianMagic = _u32(head, 0, littleEndian: false);
if (bigEndianMagic == 0xcafebabe || bigEndianMagic == 0xcafebabf) {
return _readFatMachO(raf, head, is64: bigEndianMagic == 0xcafebabf);
}
final thin = _readThinMachO(head);
if (thin != null) return thin;
}
if (head.length >= 0x40 && head[0] == 0x4d && head[1] == 0x5a) {
return _readPe(raf, head, length);
}
return const UnrecognizedHeader();
} on FileSystemException {
return const UnrecognizedHeader();
} finally {
raf.closeSync();
}
}

bool _hasElfMagic(Uint8List head) =>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for all of the magic numbers everywhere in this code, if we have sources lets add links to the sources in doc comments.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added source links for these constants in 8a67fb2 and updated the System V, fat-header, and machine-header links in 7560bd4.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do. Each format's magic numbers become private static consts in its own file with a doc-comment link to the format spec (the ELF e_machine table, Apple's loader.h/fat.h/machine.h, and the Microsoft PE spec).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the source links for the magic numbers in doc comments; they moved into MachOValidator with the split.

head[0] == 0x7f && head[1] == 0x45 && head[2] == 0x4c && head[3] == 0x46;

NativeLibraryHeader _readElf(Uint8List head) {
final elfClass = head[4];
final elfData = head[5];
if ((elfClass != 1 && elfClass != 2) || (elfData != 1 && elfData != 2)) {
return const UnrecognizedHeader();
}
final machine = _u16(head, 18, littleEndian: elfData == 1);
final architecture = switch (machine) {
3 => Architecture.ia32,
40 => Architecture.arm,
62 => Architecture.x64,
183 => Architecture.arm64,
243 => elfClass == 1 ? Architecture.riscv32 : Architecture.riscv64,
_ => null,
};
return RecognizedHeader(BinaryFormat.elf, [architecture], [machine]);
}

Architecture? _machOArchitecture(int cpuType) => switch (cpuType) {
0x00000007 => Architecture.ia32,
0x01000007 => Architecture.x64,
0x0000000c => Architecture.arm,
0x0100000c => Architecture.arm64,
_ => null,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please leave a comment what you'd expect for arm64e with a TODO referring to:

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the arm64e note with a TODO pointing to #3379 in 8a67fb2.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do. I'll add a TODO in the Mach-O parser referencing #3379: arm64e uses the same CPU_TYPE_ARM64 and identifies its ABI in the cpusubtype, so until we read that subtype we treat it as arm64 rather than claiming to validate its ABI.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the arm64e comment with a TODO referencing #3379; it moved into the per-format Mach-O file with the split.

};

NativeLibraryHeader? _readThinMachO(Uint8List head) {
final littleEndianMagic = _u32(head, 0, littleEndian: true);
final bool littleEndian;
if (littleEndianMagic == 0xfeedface || littleEndianMagic == 0xfeedfacf) {
littleEndian = true;
} else if (littleEndianMagic == 0xcefaedfe ||
littleEndianMagic == 0xcffaedfe) {
littleEndian = false;
} else {
return null;
}
final cpuType = _u32(head, 4, littleEndian: littleEndian);
return RecognizedHeader(
BinaryFormat.machO,
[_machOArchitecture(cpuType)],
[cpuType],
);
}

NativeLibraryHeader _readFatMachO(
RandomAccessFile raf,
Uint8List head, {
required bool is64,
}) {
// The fat header and arch entries are always big-endian. The number of
// slices is tiny in practice; the bound also rejects Java class files,
// which share the 0xcafebabe magic but carry a version number here.
final sliceCount = _u32(head, 4, littleEndian: false);
if (sliceCount == 0 || sliceCount > 32) return const UnrecognizedHeader();
final entrySize = is64 ? 32 : 20;
raf.setPositionSync(8);
final entries = raf.readSync(sliceCount * entrySize);
if (entries.length < sliceCount * entrySize) {
return const UnrecognizedHeader();
}
final machineValues = [
for (var slice = 0; slice < sliceCount; slice++)
_u32(entries, slice * entrySize, littleEndian: false),
];
return RecognizedHeader(BinaryFormat.machO, [
for (final cpuType in machineValues) _machOArchitecture(cpuType),
], machineValues);
}

NativeLibraryHeader _readPe(RandomAccessFile raf, Uint8List head, int length) {
final peOffset = _u32(head, 0x3c, littleEndian: true);
if (peOffset + 6 > length) return const UnrecognizedHeader();
raf.setPositionSync(peOffset);
final signature = raf.readSync(6);
if (signature.length < 6 ||
signature[0] != 0x50 ||
signature[1] != 0x45 ||
signature[2] != 0 ||
signature[3] != 0) {
return const UnrecognizedHeader();
}
final machine = _u16(signature, 4, littleEndian: true);
final architecture = switch (machine) {
0x014c => Architecture.ia32,
0x01c4 => Architecture.arm,
0x5064 => Architecture.riscv64,
0x8664 => Architecture.x64,
0xaa64 => Architecture.arm64,
_ => null,
};
return RecognizedHeader(BinaryFormat.pe, [architecture], [machine]);
}

int _u16(Uint8List bytes, int offset, {required bool littleEndian}) =>
littleEndian
? bytes[offset] | (bytes[offset + 1] << 8)
: (bytes[offset] << 8) | bytes[offset + 1];

int _u32(Uint8List bytes, int offset, {required bool littleEndian}) =>
littleEndian
? bytes[offset] |
(bytes[offset + 1] << 8) |
(bytes[offset + 2] << 16) |
(bytes[offset + 3] << 24)
: (bytes[offset] << 24) |
(bytes[offset + 1] << 16) |
(bytes[offset + 2] << 8) |
bytes[offset + 3];
Loading
Loading