fix: validate NCCL_TESTS_SPLIT env vars and guard division by zero - #379
Open
randomizedcoder wants to merge 2 commits into
Open
fix: validate NCCL_TESTS_SPLIT env vars and guard division by zero#379randomizedcoder wants to merge 2 commits into
randomizedcoder wants to merge 2 commits into
Conversation
…ases)
Table-driven tests for two classes of unsafe env var handling in the
NCCL_TESTS_SPLIT / NCCL_TESTS_SPLIT_MASK code path:
- strtoul without endptr: demonstrates strtoul("xyz", NULL, 16)
silently returns 0, indistinguishable from valid "0" (CWE-807)
- division by zero: fork-based test proves proc % 0 delivers SIGFPE,
crashing any MPI process that sets NCCL_TESTS_SPLIT=MOD0 (CWE-369)
- parseInt 0b-prefix bug: parseInt("0bxyz") falsely succeeds with
num=0 due to endptr comparison against wrong pointer
Source-verification test intentionally FAILS at this commit to
demonstrate the unsafe patterns exist before the fix.
Signed-off-by: dave.seddon.ca@gmail.com
Signed-off-by: randomizedcoder dave.seddon.ca@gmail.com <dave.seddon.ca@gmail.com>
…by zero strtoul(splitMaskEnv, NULL, 16) silently returns 0 on garbage input (CWE-807). proc % color and proc / color crash with SIGFPE when color == 0 from NCCL_TESTS_SPLIT=MOD0 or DIV0 (CWE-369). Fix SPLIT_MASK: add endptr + errno validation with warning on invalid input (same pattern as NCCL_TESTS_DEVICE from prior commit). Fix MOD/DIV: guard color == 0 with warning and safe fallback (color remains 0, equivalent to no split). Fix parseInt: add errno check for overflow, fix endptr comparison for 0b binary prefix path (compared against wrong pointer). Signed-off-by: dave.seddon.ca@gmail.com Signed-off-by: randomizedcoder dave.seddon.ca@gmail.com <dave.seddon.ca@gmail.com>
Author
|
Ok, these are the top 3 for this repo |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Static analysis findings
flawfinder (Level 3, CWE-807) flagged unvalidated
getenv()input passed directly tostrtoul()without error checking. Manual review of the same code path revealed a division-by-zero crash (CWE-369) and an endptr comparison bug in theparseInthelper function.strtoul(splitMaskEnv, NULL, 16)— no endptr, no errno checkcommon.cu:1262proc % colorwith no zero guardcommon.cu:1278proc / colorwith no zero guardcommon.cu:1283parseIntendptr compared against wrong pointer after0bprefixcommon.cu:1237parseIntmissingerrnocheck forstrtouloverflowcommon.cu:1232-1235All findings are in
src/common.cuinside the#ifdef MPI_SUPPORTblock that handlesNCCL_TESTS_SPLIT_MASKandNCCL_TESTS_SPLITenvironment variables for MPI communicator splitting.Why these matter
strtoulwithout endptr silently returns 0 on garbage input. The current code does:strtoul("xyz", NULL, 16)returns 0 — indistinguishable fromstrtoul("0", NULL, 16). ForNCCL_TESTS_SPLIT_MASK=xyz, all MPI ranks silently get masked to color 0 (no split) instead of receiving an error.Division by zero crashes the MPI process with SIGFPE. The
NCCL_TESTS_SPLITenv var supportsMODandDIVoperations:If a user sets
NCCL_TESTS_SPLIT=MOD0orNCCL_TESTS_SPLIT=DIV0,parseIntparses "0" as a valid value, then the integer division by zero deliversSIGFPEand kills the process. The fork-based adversarial test (test_divzero_sigfpe_exploit) proves this crash.The
parseInthelper has a subtle endptr bug that creates another crash path. For input"0bxyz",parseIntmatches the"0b"prefix and callsstrtoul("xyz", &p, 2). Since"xyz"has no valid binary digits,strtoulconsumes nothing and setsp = "xyz". But the error check comparesp == s(pointing to"0bxyz"), notp == s+2(pointing to"xyz"). Sincep != s,parseIntfalsely returns true with*num = 0. This feeds into the MOD/DIV path and causes the same SIGFPE crash:NCCL_TESTS_SPLIT=MOD0bxyzcrashes.Git history
All affected code traces to a single commit:
a89cf07This commit added the entire
NCCL_TESTS_SPLIT/NCCL_TESTS_SPLIT_MASKfeature in one pass, including theparseInthelper and theAND/OR/MOD/DIVdispatch logic. Thestrtoul(..., NULL, 16)shortcut and the missing division-by-zero guards are the kind of oversights that happen when the focus is on getting the feature working for valid inputs. TheparseIntendptr bug is a subtle interaction between the0b-prefix path and the single comparison point — easy to miss in review.This is the only commit that has ever touched this code. The SPLIT feature is relatively new (Jan 2025) and hasn't been modified since, so these bugs have been present since introduction. The
parseInthelper was written specifically for this feature and isn't used anywhere else.Changes
Commit 1: Tests (TDD — intentionally FAIL before fix)
tests/c/test_split_safety.c— 7 test cases:test_source_verified— greps source for unsafe patterns (intentionally FAILS)test_strtoul_no_endptr_exploit— provesstrtoul("xyz", NULL, 16)indistinguishable from"0"test_strtoul_overflow_exploit— proves overflow undetectable without errnotest_divzero_sigfpe_exploit— fork-provesproc % 0andproc / 0deliver SIGFPEtest_parseInt_0b_endptr_exploit— provesparseInt("0bxyz")falsely succeeds with num=0test_strtoul_hex_validation— table-driven: 9 cases for safe hex parsingtest_division_guard— table-driven: guard pattern with color==0 and color>0Commit 2: Fixes
parseInt: introducestartpointer so endptr comparison works for both0band non-prefix paths; adderrno = 0+ERANGEcheck for overflowSPLIT_MASK: replace rawstrtoul(..., NULL, 16)with endptr + errno validation, warn on invalid inputMOD/DIV: addcolor == 0guard with warning, fallback to color=0 (no split)Test plan
make -C tests/c test— 7/7 pass after fixtest_divzero_sigfpe_exploitfork-proves SIGFPE crash is realtest_strtoul_no_endptr_exploitproves strtoul silent failuretest_parseInt_0b_endptr_exploitproves parseInt endptr bugFiles changed
src/common.cu— parseInt fix (errno + 0b endptr), SPLIT_MASK strtoul validation, MOD/DIV guardstests/c/test_split_safety.c— 7 test cases (new file)tests/c/Makefile— test build infrastructuretests/Makefile— parent test dispatcher🤖 Generated with Claude Code