[5/10] libs/libc/elf: Load an FDPIC object. - #19942
Draft
casaroli wants to merge 8 commits into
Draft
Conversation
github-actions
Bot
added
Arch: arm
❌ Cross-repo dependency could not be appliedThe Build report says the declared dependency PR(s) could not be applied, so CI did not run against the combined code:
Reason: cherry-pick failed (if your PR has merge commits, rebase instead) CI run: https://github.com/apache/nuttx/actions/runs/32671745418 |
|
CONFIG_BINFMT_CONSTRUCTORS has never had any effect on a relocatable module loaded through exec(). elf_loadbinary() records .preinit_array and .init_array in binp->mod, and nothing reads them back: $ git grep -n "initarr" binfmt/ sched/ binfmt/elf.c:256: binp->mod.initarr = loadinfo.initarr; binfmt/elf.c:264: binp->mod.initarr = loadinfo.initarr; A C++ module's global objects are therefore left as .bss and its constructors are dropped without a word. The task that runs the module then reads a global that no constructor ever wrote. Call the arrays at the end of the load, which is where libelf_insert() has always called them for a module that arrives through dlopen(). It is the last thing the load does, so a global is initialized before the module's main() can see it, and nothing that can still fail runs after a constructor has. Three cases are left alone, because something else already serves them or this task cannot. A fully linked executable carries crt0, which calls the same array from _sctors to _ectors -- libs/libc/elf/gnu-elf.ld puts .init_array and .ctors there -- on the task that runs the module and in its own address environment. So ET_EXEC is skipped. A build with an address environment is skipped for the same reason: the environment of the module is not selected here, so this task cannot reach the array at all. Such a build loads an executable, thus crt0 covers it. A module that reaches its globals through a PIC base register is skipped because the loading task carries its own base, not the module's, and a constructor would address the wrong data. Installing the base for the length of one call needs a function descriptor, which is what FDPIC adds later in this series. The arrays cannot run on the task that will run the module. exec_module() had a hook for that, exec_ctors() through nxtask_starthook(), and it went with CONFIG_SCHED_STARTHOOK. Destructors already run on this path: elf_unloadbinary() calls libelf_uninit(), which walks .fini_array. So the object is destroyed without ever having been constructed, and that asymmetry is the bug. A module with no constructors is unaffected, as is any configuration with CONFIG_BINFMT_CONSTRUCTORS disabled. Built for mps3-an547:picostest with CONFIG_BINFMT_CONSTRUCTORS enabled, which the configuration does not set by default. Runtime evidence on hardware follows. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
apps/examples/fdpicxip is in apps master and its modules/Makefile reads FDPICDIR = $(NUTTX_DIR)/tools/fdpic MODULE_MK = $(FDPICDIR)/nuttx-fdpic.mk EMBED = $(FDPICDIR)/fdpic-embed.py which is not in this tree, so the committed module blobs cannot be regenerated. This adds what that Makefile names. nuttx-fdpic.mk builds a module: compile with the stock arm-none-eabi GCC, which emits correct FDPIC objects for C and C++, then assemble and link with an arm-uclinuxfdpiceabi binutils, which is the only piece a distribution does not carry. build-binutils.sh builds that, and takes about a minute. nuttx-exports.sh turns a built firmware's exec_symtab.c into a symbol list, fdpic-verify.sh checks a module's imports against it, and fdpic-embed.py turns a module into a C header for an image that carries one inside it. The tools are host side and nothing in the NuttX build calls them. The loader that runs what they build is proposed separately, so a module built here has nothing to load it yet, and the page says so. Documentation/components/tools/fdpic.rst describes them, under Host Tools, where the tools index picks it up by glob. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
The ET_DYN path computes run-time addresses from link-time ones in five places, each open-coding the arithmetic, and two of them disagree about how: libelf_relocatedyn() adds textalloc to a relocation's r_offset in one branch and subtracts datasec before adding datastart in the next, while the value translation a few lines further down picks between those two forms with an explicit test on datasec. Collect that into libelf_addr(), which makes the test once: an address below the data segment's link-time base belongs to text, anything at or above it to data. This changes nothing today. libelf_elfsize() sets segpad = datasec - (text_vaddr + textsize) and libelf_load() then places datastart = textalloc + textsize + segpad so datastart - datasec is textalloc, and the data branch reduces to textalloc + vaddr -- exactly what the text branch returns, and exactly what adding a single load bias did before. The two forms are the same arithmetic written twice. They stop being the same once text and data are placed independently, which is what an FDPIC object requires: its two PT_LOAD segments are relocated separately so that the read-only one can be mapped in place on the media while only the writable one is copied. Having the translation in one function is what makes that possible without auditing every open-coded expression again. Built for mps3-an547:picostest, which is CONFIG_ELF with CONFIG_PIC, and boots identically to the same configuration without this change. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
The commits that follow teach the ELF loader to load an FDPIC object. This puts the option they hang off and the definitions they share in one place first, so each of them builds on its own. CONFIG_FDPIC depends on ARCH_HAVE_ELF_FDPIC, which an architecture selects when it has a PIC base register and the FDPIC relocations. Only armv7-m and armv8-m select it today, and it defaults off, so nothing changes for anyone who does not ask for it. include/nuttx/fdpic.h holds what both sides of the loader need: the two word function descriptor an FDPIC module passes instead of a code address, the test for whether the caller is such a module, and the call sequence that enters one with its own data base. All of it is behind CONFIG_FDPIC, thus the header is empty without it and a file may include it unconditionally. The call sequence itself is architecture specific, so arch/arm/include/arch.h supplies it as up_fdpic_invoke(), beside the other PIC base register macros. up_setpicbase() cannot serve here: the register has to hold the module's base for exactly one call and then go back, and nothing in C tells the compiler the register is live across that call, so the save, the install, the branch and the restore have to be one sequence. Built for mps3-an547:bl and mps3-an547:picostest, with CONFIG_FDPIC off, which is every configuration in the tree. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
An ET_DYN object is loaded into one allocation with its data behind its text, because its data references sit at a fixed distance from the code that makes them. An FDPIC object does not work that way: it reaches its data through a base register, so the two segments can be placed wherever suits, and the point of the format is that the read-only one is left on the media and executed there while only the writable one is copied. One copy of the text then serves every instance. So libelf_load() grows a second case. The object announces itself in the OS/ABI byte, which is noted once in libelf_loadhdrs() rather than re-derived; e_flags cannot be used for this, as an FDPIC object's are an unremarkable EABI version and testing them would reject every valid module. Text is taken from the media address plus the segment's own file offset -- the same arithmetic the ET_REL path already does with sh_offset -- and libelf_loadfile() does not read it. If the filesystem cannot show its media, the loader copies the text to RAM instead. The module then loses the shared text and the flash saving, but it runs. Obtaining that address needs two mechanisms, and they are not interchangeable. A compacting filesystem can move a file's blocks, so it hands out an address only with a pin that holds them still and expects the pin back; xipfs is the one in tree. A filesystem whose layout never changes has nothing to hold and answers FIOC_XIPBASE with a bare address; romfs and tmpfs are those. libelf_xipacquire() asks for the pin first, because a filesystem that needs one is not safe without it, and libelf_unload() gives it back. The loader asks for a pin only if it can hold one, or the pin would stay for ever. The pin is thus not specific to FDPIC. Any module that executes in place from a compacting filesystem takes one, and gives it back at unload. mmap() is not used, though both filesystems implement it. The mapping would be recorded against whichever task called the loader, while the release happens when the module's own task exits, which is a different group -- so the pin would outlive the module and the extent would never become movable again. Unloading has to change with placement: the existing path frees only textalloc because ET_DYN had a single allocation, which would leak an FDPIC object's data and free media the filesystem only lent us. Nothing here runs for a non-FDPIC object; every branch is behind the flag and the single-allocation path is untouched. Built and booted mps3-an547:picostest, which is CONFIG_ELF with CONFIG_PIC, with no change in behaviour. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
libelf_relocatedyn() reads the handful of DT_* tags it needs to walk the relocation tables and ignores the rest. Three more matter now. DT_PLTGOT is where the object's data base lives. An FDPIC module runs with that in the PIC base register, and every function descriptor built for it names the same base as the one its callee should run with, so without it there is nothing to put in a descriptor's second word. The DT_*_ARRAY tags are the constructor and destructor tables. These are already found through the section headers a few lines further down, and that path is kept, but the dynamic tags are the authoritative copy and an object is not obliged to carry section headers at all. Both paths now translate through libelf_addr(), so they agree on the answer rather than depending on which ran last. The tag values themselves were missing from include/elf.h and are added. Sizing the descriptor pool has to happen here rather than later. R_ARM_FUNCDESC asks the loader to manufacture a descriptor and hand back its address, which means the space must exist by the time the relocation is applied, and by then the segment has been placed. So libelf_elfsize() reserves it behind the writable data, bounded by the relocation count -- one relocation cannot ask for more than one descriptor. That bound has slack in it, but a descriptor is two words and modules are small, which is cheaper than walking every relocation twice to get an exact count. Nothing here runs for a non-FDPIC object. Built and booted mps3-an547:picostest with no change in behaviour. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
Running one for the first time turned up two holes in the ET_DYN path. Neither shows up in a build. An undefined symbol is resolved with libelf_findglobal(), which searches only the table of globally registered symbols. The export table that exec() hands its caller went no further than the ET_REL path, so an ET_DYN module could not import anything the caller supplied. Invisible while such modules resolved everything internally; an FDPIC module imports its libc, and every import failed with "Unable to resolve addr of ext ref printf" although the caller had passed a table containing printf. The export table is now threaded into libelf_relocatedyn() and consulted when the global table has no answer, leaving the existing lookup order intact. A relocation naming a symbol defined inside the object was dropped silently. The code handles a relocation with no symbol, and one against an undefined symbol, but a defined symbol fell through both. That was harmless while every dynamic relocation arriving here had symbol index zero, which is the case for R_ARM_RELATIVE. FDPIC brings the first ones that do not: a pointer to a static function is emitted against the *section* symbol, so the value is the section base and the offset within it -- including the Thumb bit -- is carried as the addend. Deriving a value from the word being patched, as the no-symbol case does, would translate that addend as though it were an address. Confirmed against a real module: .text at 0x23c plus an addend of 0x95 gives 0x2d1, which is the function with its Thumb bit. Also stop libelf_symname() reporting a nameless symbol as an error. A section symbol has no name, and libelf_findsymbol() walks the whole table looking for optional entries such as nx_stacksize, so it meets these routinely and checks for -ESRCH itself. At error level it printed ten or more lines per module load and buried the diagnostics that matter. Built and run on lm3s6965-ek with the examples/elf ROMFS. The ET_REL test modules load as before, and an FDPIC module now loads, relocates, resolves printf and puts from the table exec() supplied, and calls through a function descriptor of its own. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
A module that dlopen()s a library gets back function addresses from dlsym() and calls them. Under FDPIC a bare code address is not enough: the callee needs its own data base as well, so what dlsym() returns has to be a function descriptor. The exported symbol table carries no type information -- symtab_s is a name and a value, and its own comment says typing would have to be added to support anything but function pointers -- so by the time dlsym() is asked there is no way to tell a function from an object. libelf_insertsymtab() is the last point that can: st_info is still in hand there. So an FDPIC object's exported functions are published as the address of a descriptor carved from the module's pool, and dlopen(), dlsym() and the module registry need no knowledge of FDPIC at all. The pool is sized for the dynamic symbol table as well as the relocations, since both can draw from it. That leaves the symbol values themselves, which were wrong for any ET_DYN object. libelf_loadsymtab() adds the symbol's section address to its value, which is right for ET_REL, where the section address is where the section was actually placed and the value is relative to it. In a shared object both are already full link-time addresses, so adding them counts the section twice. It needs translating onto wherever the object was placed instead. Library data is shared between everything that dlopen()s it, because the registry holds one instance per name. Giving each user its own copy would mean teaching the registry about instances, which is a much larger change to shared code; an executable loaded through exec() already gets its own data, since that path loads a fresh copy each time. Built and run on lm3s6965-ek with the examples/elf ROMFS; the FDPIC module continues to load, relocate and call through its own descriptors. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
casaroli
force-pushed
the
fdpic-loader-core
branch
from
August 25, 2026 09:27
4b54f93 to
ae1aa73
Compare
❌ Cross-repo dependency could not be appliedThe Build report says the declared dependency PR(s) could not be applied, so CI did not run against the combined code:
Reason: cherry-pick failed (if your PR has merge commits, rebase instead) CI run: https://github.com/apache/nuttx/actions/runs/32832058242 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
depends-on: [/pull/19938 /pull/19939 /pull/19940 /pull/19941]
Summary
Fifth of ten PRs that #19673 is being split into. This is the loader side: enough for the ELF loader to place an FDPIC object and bind it. The ARM relocations are
[6/10], the callback entry points[7/10], theexec()path[8/10],DT_NEEDED[9/10], documentation and a board configuration[10/10].An FDPIC object places its two
PT_LOADsegments independently, so its read-only segment runs where the filesystem already holds it and only the writable segment is copied to RAM, once per running instance. Several instances of one module therefore share one copy of the text. A filesystem that cannot show its media gets the text copied to RAM instead, so the module still runs but shares nothing.The first commit adds
CONFIG_FDPIC,ARCH_HAVE_ELF_FDPICandinclude/nuttx/fdpic.h, so the four that follow each build on their own.The read-only segment needs the filesystem to hold its blocks still. A compacting filesystem such as XIPFS gives its media address together with a pin, which the loader holds through a file reference, because the unload runs on a different task from the load.
Impact
CONFIG_FDPICdefaults off and no in-tree configuration sets it, so nothing changes for anyone yet. With it off the four loader commits compile to what they were.The four PRs this depends on are not merged, so nine commits show here until they are.
Testing
mps3-an547:blandmps3-an547:picostestbuild withCONFIG_FDPICoff, which is every configuration in the tree.tools/checkpatch.shpasses on each of the five commits.mps3-an547:blis the one to keep an eye on: it setsCONFIG_ARCH_USE_SEPARATED_SECTION, andCONFIG_LIBC_ELFpulls inARCH_USE_TEXT_HEAP, which selects the three argumentup_textheap_memalign(). #19673 calls the two argument form there and fails to build; that is fixed in the commit that adds the call.Runtime evidence follows with
[6/10], since an FDPIC module cannot be relocated until the ARM relocations land. The full stack has been run on QEMUmps2-an500and on a Pimoroni Pico Plus 2: 131/131, 34/34 and 7/7, with the logs in #19673.