Debug mode captures no stack trace at all under 64-bit

[TetzkatLipHoka]

Under 64-bit, FastMM_LoadDebugSupportLibrary binds a stack tracing routine that cannot work there, so debug blocks are recorded with no stack trace at all. A leak report prints

This block was allocated on ..., and the stack trace (return addresses) at the time was:

The block is currently used for an object of class: Unknown

with nothing between the two lines. The same program built for 32-bit resolves the trace normally.

Cause

FastMM_LoadDebugSupportLibrary binds GetRawStackTrace on every target:

if (@FastMM_GetStackTrace = @FastMM_NoOpGetStackTrace)
  and Assigned(DebugLibrary_GetRawStackTrace) then
begin
  FastMM_GetStackTrace := DebugLibrary_GetRawStackTrace;
end;

That routine walks the RBP frame chain, and the x64 ABI does not maintain one, so the loop finds nothing and returns an empty buffer.

The support library already has a routine for this case - GetFrameBasedStackTrace, which under 64-bit is implemented on RtlCaptureStackBackTrace, with the comment "We use the Windows API to do frame based stack tracing under 64-bit". It is exported. It is simply never bound.

Measurement

Calling both entry points directly out of the library, bypassing FastMM, with a 20 entry buffer and ASkipFrames zero:

Target	GetRawStackTrace	GetFrameBasedStackTrace
Win32	12 of 20 frames	-
Win64	0 of 20	8 of 20

What fixes it

Preferring the frame based routine under 64-bit, keeping the raw one as the fallback for a support library that does not export it, and leaving 32-bit alone - the raw tracer works there and yields the deeper traces it exists for. With that, a Win64 leak report comes back as

00007FF6BECDD396 [FastMM5.pas][FastMM5][FastMM_DebugGetMem_GetDebugBlock][8255]
00007FF6BECC5834 [System][System][@GetMem]
00007FF6BECF6EBE [LeakCheck.dpr][LeakCheck][AllocateSomethingAndLeakIt][23]

Both binding paths need it, the GetProcAddress one and the statically bound one.

A detail worth mentioning: with ASkipFrames at zero the first captured frame is GetFrameBasedStackTrace itself, so one of the entries is spent on the tracer. Harmless, but it means the effective depth is one less than requested on 64-bit.

No pull request, per your note in #101. It is on win64-frame-based-stack-trace in my fork if you want to look; the test suite passes 18 of 18 runs with it and 32-bit reports are unchanged line for line. Happy to open a PR if you ask for one.

[TetzkatLipHoka]

Correction to the cause I gave above - I had the mechanism wrong, and the right one is more interesting.

It is not that the raw tracer cannot follow the frame chain under 64-bit. The chain is walked fine. The addresses are then thrown away by the filter:

if (LReturnAddress > $ffff) {$if SizeOf(Pointer) = 8}and (LReturnAddress <= $ffffffff){$IFEND} then

Every return address above 4 GB is discarded, and that ceiling is not arbitrary - it follows from MemoryPageAccessMap: array[0..1024 * 1024 - 1], one entry per 4 KB page, which covers exactly 4 GB and nothing above it.

Delphi's default image base under 64-bit is $140000000, so an ordinary build sits above the ceiling and every frame is rejected. Rebuilding the same probe with -K00400000 puts the image below 4 GB, and the raw tracer returns 10 of 20 frames instead of 0. Nothing else changed.

That also answers whether this ever worked: it does work, for images low enough in the address space. It is a limit that Delphi's default image base happens to sit above, not a regression.

{$STACKFRAMES ON} makes no difference either way - I tested that as well, in case the frame chain was the issue after all.

The conclusion is unchanged and the reason for it is now stronger: GetFrameBasedStackTrace is built on RtlCaptureStackBackTrace, which needs no page map and therefore has no such ceiling. Widening the map is not a realistic alternative on a 64-bit address space.

One more thing that fits here: DebugLibrary_GetFrameBasedStackTrace is declared in the interface section, so an application can bind it itself, which is presumably how the trailing entry problem in #93 came to light. What is missing is only the automatic choice under 64-bit.

[ajax16384]

GetRawStackTrace might be fixed for Win + 64bit by following steps:

1. completely removing "LReturnAddress <= $ffffffff" check
2. replacing MemoryPageAccessMap with "handwritten" hashtable resides in preallocated global var.

Some estimation for 128K hash table (which might be sufficient for most 64bit ASLR scenario):

MemoryPageAccessMap direct index - ~2-4 CPU cycles
Hashtable (no collisions) - ~10-20 CPU cycles
Hashtable (with collisions) - ~50-100 CPU cycles
VirtualQuery - ~1000-10000 CPU cycles

[pleriche]

Thanks for the report and suggestions. I've been meaning to address this for a while now, but just haven't gotten round to it yet. It is high up on my to-do list.