Qualify diagnostic functions as pragma(printf) by nordlow · Pull Request #11638 · dlang/dmd

nordlow · 2020-08-28T09:49:49Z

Will improve type-checking of parameters used in C-printf-style diagnostics formatting.

dlang-bot · 2020-08-28T09:49:51Z

Thanks for your pull request and interest in making D better, @nordlow! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

My PR is fully covered with tests (you can see the coverage diff by visiting the details link of the codecov check)
My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
I have provided a detailed rationale explaining my changes
New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub run digger -- build "master + dmd#11638"

MoonlightSentinel · 2020-08-28T09:55:32Z

This requires building with -ignore for older host compilers ( < ~~2.091.1 (?)~~ 2.092.0).

MoonlightSentinel · 2020-08-28T10:18:47Z

Maybe do something like this:

diff --git a/src/build.d b/src/build.d
index 184b29313..76c6ec9f5 100755
--- a/src/build.d
+++ b/src/build.d
@@ -1007,6 +1007,11 @@ void processEnvironment()
     else
         env["HOST_DMD_KIND"] = "gdc";

+    // Determine version:
+    const verStart = hostDMDVersion.countUntil('v') + 1;
+    const versionNum = hostDMDVersion[verStart .. verStart + "X.XXX.X".length];
+    env["HOST_DMD_VERSION"] = versionNum;
+
     env["DMD_PATH"] = env["G"].buildPath("dmd").exeName;
     env.getDefault("DETAB", "detab");
     env.getDefault("TOLF", "tolf");
@@ -1023,6 +1028,10 @@ void processEnvironment()
     if (env["HOST_DMD_KIND"] != "gdc")
         dflags ~= ["-dip25"]; // gdmd doesn't support -dip25

+    // Ignore
+    if (versionNum < "2.092.0")
+        dflags ~= "-ignore";
+
     // TODO: add support for dObjc
     auto dObjc = false;
     version(OSX) version(X86_64)

EDIT: For dmd as host compiler, others don't work as well. See #11639 for an alternative solution.

nordlow · 2020-08-29T05:23:32Z

This requires building with -ignore for older host compilers ( < ~~2.091.1 (?)~~ 2.092.0).

Thanks. I'll try your patch.

nordlow · 2020-08-29T16:05:35Z

Does the error

src/dmd/errors.d(45): Error: pragma `printf` is missing a terminating `;`

mean that -ignore didn't have any effect in this case?

MoonlightSentinel · 2020-08-30T10:54:37Z

Found that -ignore did not work for unknown pragmas with bodys until 2.089.1 (fixed by this PR).

Guess we would have to implement another workaround for older host compilers or wait until they become recent enough...

nordlow · 2020-08-30T22:15:39Z

Doing version-branch each definition of course for now until minimum compiler version is bumped.

There's much error checking to gain here so lets see what people think about this brute-force version-branching.

Is this the dmd version where pragma(printf) was introduced?:
https://dlang.org/changelog/2.092.0.html

Update: We could create wrapper functions that have different versions depending on __VERSION__...

MoonlightSentinel · 2020-08-31T12:12:50Z

Is this the dmd version where pragma(printf) was introduced?:
https://dlang.org/changelog/2.092.0.html

Think so.

Update: We could create wrapper functions that have different versions depending on __VERSION__...

Seems reasonable.

nordlow · 2020-08-31T12:53:32Z

@MoonlightSentinel: What do you make of the error https://auto-tester.puremagic.com/show-run.ghtml?projectid=1&runid=4161908&isPull=true? I'm clueless.

MoonlightSentinel · 2020-08-31T13:22:09Z

Running locally using Ubuntu-WSL (built with -checkaction=context):

core.exception.AssertError@src/dmd/lexer.d(2730): "escape hex sequence has 1 hex digits instead of 2" != "escape hex sequence has 1 hex digits instead of -703763272"       
----------------
??:? _d_assert_msg [0x7fa00f693bf2]
src/dmd/lexer.d:2730 nothrow @nogc bool dmd.lexer.__unittest_L2716_C1().expectDiagnosticHandler(ref const(dmd.globals.Loc), dmd.console.Color, const(char)*, const(char)*, core.internal.vararg.sysv_x64.__va_list_tag*, const(char)*, const(char)*) [0x7fa00f5b1d2f]
src/dmd/errors.d:338 nothrow void dmd.errors.verrorPrint(ref const(dmd.globals.Loc), dmd.console.Color, const(char)*, const(char)*, core.internal.vararg.sysv_x64.__va_list_tag*, const(char)*, const(char)*) [0x7fa00f5a1161]
src/dmd/errors.d:421 _Z6verrorRK3LocPKcP13__va_list_tagS3_S3_S3_ [0x7fa00f5a15c6]
src/dmd/errors.d:59 _Z6_errorRK3LocPKcz [0x7fa00f5a0309]
src/dmd/errors.d:53 _Z5errorRK3LocPKcz [0x7fa00f5a023e]
src/dmd/lexer.d:1230 nothrow dchar dmd.lexer.Lexer.escapeSequence(ref const(dmd.globals.Loc), ref const(char)*) [0x7fa00f5ae7a8]
src/dmd/lexer.d:2742 nothrow void dmd.lexer.__unittest_L2716_C1().test(immutable(char)[], immutable(char)[], dchar, uint) [0x7fa00f5b1da6]
src/dmd/lexer.d:2754 nothrow void dmd.lexer.__unittest_L2716_C1() [0x7fa00f5b17d6]

Win64 passes the test, so it looks like a parameter corruption.

nordlow · 2020-08-31T15:17:58Z

So I have to duplicate the creation of the va_list aps, then.

nordlow · 2020-08-31T17:07:14Z

Ready to go, @MoonlightSentinel?

MoonlightSentinel · 2020-08-31T17:36:38Z

LGTM except that this overload is missing:

dmd/src/dmd/errors.d

Lines 62 to 75 in ea93b5c

    
           /** 
        
            * Same as above, but allows Loc() literals to be passed. 
        
            * Params: 
        
            *      loc    = location of error 
        
            *      format = printf-style format specification 
        
            *      ...    = printf-style variadic arguments 
        
            */ 
        
           extern (D) void error(Loc loc, const(char)* format, ...) 
        
           { 
        
               va_list ap; 
        
               va_start(ap, format); 
        
               verror(loc, format, ap); 
        
               va_end(ap); 
        
           }

This overload cannot be marked with pragma(printf) as it's extern(D)so you would have to mark it as extern(C++). But I think you can simply remove it because it's only used once:

dmd/src/dmd/dinifile.d

Line 346 in ace2aee

    
           error(Loc(filename.xarraydup.ptr, lineNum, 0), "Use `NAME=value` syntax, not `%s`", pn);

nordlow · 2020-08-31T17:43:19Z

LGTM except that this overload is missing:

The call in dinifile.d requires an r-value Loc. Shall I convert the call to use an l-value or add r-value overloads for

extern (C++) void error(const Loc loc, const(char)* format, ...)

or, perhaps, even better, remove the extern(D)-overload and convert all arguments taking Loc to be passed as in value? Loc is 16 bytes wide and will be passed in registers, AFAICT. If we later add more members to Loc the new rules for in in future versions of the host dmd compiler will handle this correctly.

MoonlightSentinel · 2020-08-31T17:48:19Z

Rewrite it to store a local Loc for now, we can look into optimizations later.

-error(Loc(filename.xarraydup.ptr, lineNum, 0), "Use `NAME=value` syntax, not `%s`", pn); 
+Loc(filename.xarraydup.ptr, lineNum, 0)
+error(loc, "Use `NAME=value` syntax, not `%s`", pn);

nordlow · 2020-08-31T17:49:07Z

Rewrite it to store a local Loc for now, we can look into optimizations later.

Using in Loc loc builds ok.

You mean do the transition from const ref to in at larger scale?

MoonlightSentinel · 2020-08-31T17:51:58Z

Yes, let's stick with the current default (const ref) and discuss optimizations in a dedicated PR.

nordlow · 2020-08-31T18:38:29Z

Yes, let's stick with the current default (const ref) and discuss optimizations in a dedicated PR.

Done.

nordlow · 2020-08-31T19:15:45Z

Why does druntime fail to build in buildkite/dmd? I can't see any error message.

Only a warning:

Warning: struct HasNonConstToHash has method toHash, however it cannot be called with const(HasNonConstToHash) this.

MoonlightSentinel · 2020-08-31T19:24:22Z

Seems like a spurious OOM, restarted the job manually.

Testing printf
--
  | ./generated/linux/debug/64/printf
  | /var/lib/buildkite-agent/builds/ci-agent-a5eb829d-1249-4ef6-a546-0eb660ed8d77-5/dlang/dmd/dmd/generated/linux/release/64/dmd -debug=LOGGING -m64 -fPIC -w -I../../src -I../../import -Isrc -defaultlib= -debuglib= -dip1000 -L-lpthread -L-lm -L/var/lib/buildkite-agent/builds/ci-agent-a5eb829d-1249-4ef6-a546-0eb660ed8d77-5/dlang/dmd/druntime/generated/linux/release/64/libdruntime.a -O -release -unittest -version=CoreUnittest -main -ofgenerated/linux/release/64/logging ../../src/gc/impl/conservative/gc.d ../../src/rt/lifetime.d
  | src/core/exception.d(647): [unittest] Memory allocation failed
  | core.exception.OutOfMemoryError@src/core/exception.d(647): Memory allocation failed
  | ----------------
  | 1/1 modules FAILED unittests
  | Makefile:15: recipe for target 'generated/linux/debug/64/printf.done' failed
  | make[2]: *** [generated/linux/debug/64/printf.done] Error 1

nordlow · 2020-08-31T19:25:12Z

Seems like a spurious OOM, restarted the job manually.

Is dmd taking too much memory?

MoonlightSentinel · 2020-08-31T21:22:23Z

Is dmd taking too much memory?

The OOM occurred in the printf executable (but I guess Buildkite just got too many jobs at once there)

nordlow · 2020-08-31T21:36:43Z

The OOM occurred in the printf executable (but I guess Buildkite just got too many jobs at once there)

Is there no guarantee for the minimum amount of memory available?

wilzbach · 2020-08-31T21:39:43Z

The OOM occurred in the printf executable (but I guess Buildkite just got too many jobs at once there)

Is there no guarantee for the minimum amount of memory available?

Nope, we run the agents ourselves and to save resources we run multiple agents on one machine (and they share the memory). It usually works out well though ;-)

nordlow · 2020-09-01T13:01:35Z

What's up with this failure?

MoonlightSentinel · 2020-09-01T14:31:22Z

What's up with this failure?

No idea but it looks environmental (std.parallelism wasn't changed in the last few days AFAICT)

nordlow · 2020-09-01T14:56:43Z

No idea but it looks environmental (std.parallelism wasn't changed in the last few days AFAICT)

Ok, thanks. Shall I do a forced push again?

MoonlightSentinel · 2020-09-01T14:57:25Z

No, I've restarted it manually

nordlow · 2020-09-01T14:57:45Z

No, I've restarted it manually

Thanks.

nordlow · 2020-09-01T14:58:28Z

No, I've restarted it manually

Can you restart parts of the autotester or only the whole?

I presume I don't have the permissions to restart individual jobs manually, right?

MoonlightSentinel · 2020-09-01T15:03:48Z

Can you restart parts of the autotester or only the whole?

Individual jobs (causing less workload than force-pushing)

I presume I don't have the permissions to restart individual jobs manually, right?

~~No, the auto-tester requires a linked account with mege rights IIRC. (But that situation should improve once we retire the auto-tester inf favour of other CI's, see e.g. #11653).~~

See below

wilzbach · 2020-09-01T15:08:46Z

No, the auto-tester requires a linked account with mege rights IIRC. (But that situation should improve once we retire the auto-tester inf favour of other CI's, see e.g. #11653).

Actually, anyone whose PRs can get tested on the auto-tester can also login there and deprecate the builds. Not that it would be important as the builds are constantly restarted. Once "auto-merge" is applied the PR branch will get higher priority and restart the build until it is green.

wilzbach · 2020-09-02T02:28:37Z

+    {
+        _vdeprecationSupplemental(loc, format, ap);
+    }
+extern (C++) void _vdeprecationSupplemental(const ref Loc loc, const(char)* format, va_list ap)


Should have been private and extern(D)

MoonlightSentinel mentioned this pull request Aug 28, 2020

[RFC] build.d: Use JSON output to detect host compiler #11639

Merged

nordlow force-pushed the use-pragma-printf branch 2 times, most recently from 3480fe4 to 028681c Compare August 29, 2020 06:11

nordlow force-pushed the use-pragma-printf branch 5 times, most recently from 8097edc to 3390539 Compare August 31, 2020 11:27

MoonlightSentinel reviewed Aug 31, 2020

View reviewed changes

Comment thread src/dmd/errors.d Outdated

MoonlightSentinel reviewed Aug 31, 2020

View reviewed changes

Comment thread src/dmd/errors.d Outdated

nordlow force-pushed the use-pragma-printf branch from 3390539 to ea93b5c Compare August 31, 2020 15:23

nordlow force-pushed the use-pragma-printf branch from ea93b5c to 8378abd Compare August 31, 2020 18:36

MoonlightSentinel approved these changes Aug 31, 2020

View reviewed changes

Qualify diagnostic functions as pragma(printf)

5b10609

nordlow force-pushed the use-pragma-printf branch from 8378abd to 5b10609 Compare September 1, 2020 14:02

MoonlightSentinel added the Merge:auto-merge label Sep 1, 2020

dlang-bot merged commit b25fcdf into dlang:master Sep 1, 2020

nordlow deleted the use-pragma-printf branch September 1, 2020 20:51

wilzbach reviewed Sep 2, 2020

View reviewed changes

Uh oh!

Conversation

nordlow commented Aug 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dlang-bot commented Aug 28, 2020

Bugzilla references

Testing this PR locally

Uh oh!

MoonlightSentinel commented Aug 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MoonlightSentinel commented Aug 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nordlow commented Aug 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nordlow commented Aug 29, 2020

Uh oh!

MoonlightSentinel commented Aug 30, 2020

Uh oh!

nordlow commented Aug 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MoonlightSentinel commented Aug 31, 2020

Uh oh!

nordlow commented Aug 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MoonlightSentinel commented Aug 31, 2020

Uh oh!

Uh oh!

Uh oh!

nordlow commented Aug 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nordlow commented Aug 31, 2020

Uh oh!

MoonlightSentinel commented Aug 31, 2020

Uh oh!

nordlow commented Aug 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MoonlightSentinel commented Aug 31, 2020

Uh oh!

nordlow commented Aug 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MoonlightSentinel commented Aug 31, 2020

Uh oh!

nordlow commented Aug 31, 2020

Uh oh!

nordlow commented Aug 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MoonlightSentinel commented Aug 31, 2020

Uh oh!

nordlow commented Aug 31, 2020

Uh oh!

MoonlightSentinel commented Aug 31, 2020

Uh oh!

nordlow commented Aug 31, 2020

Uh oh!

wilzbach commented Aug 31, 2020

Uh oh!

nordlow commented Sep 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MoonlightSentinel commented Sep 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nordlow commented Sep 1, 2020

Uh oh!

MoonlightSentinel commented Sep 1, 2020

Uh oh!

nordlow commented Sep 1, 2020

Uh oh!

nordlow commented Sep 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nordlow commented Aug 28, 2020 •

edited

Loading

MoonlightSentinel commented Aug 28, 2020 •

edited

Loading

MoonlightSentinel commented Aug 28, 2020 •

edited

Loading

nordlow commented Aug 29, 2020 •

edited

Loading

nordlow commented Aug 30, 2020 •

edited

Loading

nordlow commented Aug 31, 2020 •

edited

Loading

nordlow commented Aug 31, 2020 •

edited

Loading

nordlow commented Aug 31, 2020 •

edited

Loading

nordlow commented Aug 31, 2020 •

edited

Loading

nordlow commented Aug 31, 2020 •

edited

Loading

nordlow commented Sep 1, 2020 •

edited

Loading

MoonlightSentinel commented Sep 1, 2020 •

edited

Loading

nordlow commented Sep 1, 2020 •

edited

Loading

MoonlightSentinel commented Sep 1, 2020 •

edited

Loading