Skip to content
Open
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
e91f30a
feat(cli): send telemetry from a detached subprocess
Jul 28, 2026
0c31c3d
test(cli): cover the detached telemetry sender
Jul 28, 2026
b710b27
chore(cli): address telemetry sender review feedback (accurate trace,…
Jul 29, 2026
8d41011
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Jul 29, 2026
adfa346
fix(cli): give detached telemetry sender a realistic network timeout …
Jul 29, 2026
94d0305
fix(cli): enforce endpoint TLS identity on the proxied telemetry path…
Jul 29, 2026
0ebb2c4
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Aug 1, 2026
147ad84
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Aug 3, 2026
6977aaf
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Aug 5, 2026
f76604c
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Aug 13, 2026
5a8df75
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Aug 13, 2026
ccd9f1e
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Aug 18, 2026
c971336
refactor(cli): bundle the telemetry sender and forward the CA path, n…
sanjanaravikumar-az Aug 19, 2026
3fc8889
feat(cli): make fire-and-forget telemetry delivery observable
sanjanaravikumar-az Aug 19, 2026
d7b09e3
test(cli): assert telemetry actually arrives, not that we said we sen…
sanjanaravikumar-az Aug 19, 2026
6ac52d1
docs(cli): trim the telemetry comments and document the debug variable
sanjanaravikumar-az Aug 19, 2026
ea02d99
refactor(cli): phase 5 review cleanup for the detached telemetry sender
sanjanaravikumar-az Aug 20, 2026
be0542f
fix(cli): phase 6 review fixes for the detached telemetry sender
sanjanaravikumar-az Aug 20, 2026
cca1889
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Aug 21, 2026
c47a1b7
fix(cli): make the SOCKS unit tests and telemetry integ tests CI-safe
sanjanaravikumar-az Aug 21, 2026
fb1b70d
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Aug 24, 2026
83c98f6
test(cli): assert SOCKS proxy destination + add socks5 fail-closed case
Aug 27, 2026
ba1a1f0
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Aug 27, 2026
671cfa1
refactor(cli): address review comments on the detached telemetry sender
Aug 31, 2026
178861f
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Aug 31, 2026
bc1a7ae
test(cli): assert telemetry behaviour, not trace lines; allow detache…
Aug 31, 2026
868f0b2
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Sep 2, 2026
0729a09
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Sep 2, 2026
681ee6a
test(cli): remove expect(output) log-line proxy assertions per review…
Sep 2, 2026
ea383e4
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Sep 3, 2026
badb873
Merge branch 'main' into sanjrkmr/telemetry-subprocess
sanjanaravikumar-az Sep 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ integTest(
withDefaultFixture(async (fixture) => {
const output = await fixture.cdk(['cli-telemetry', '--disable'], { verboseLevel: 3 });

// Check the trace that telemetry was not executed successfully
expect(output).not.toContain('Telemetry Sent Successfully');
// Check the trace that telemetry was never handed to a sender
expect(output).not.toContain('Telemetry dispatched');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But now our tests won't ensure telemetry is actually being sent - asserting on a dispatch is not enough. We need a way for the sender to communicate back to the test that the telemetry endpoint responded with 200.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What we probably need is to stand up an HTTP server, set that as the telemetry endpoint, then assert on what gets sent to that endpoint.

Or in this case, we need to assert that after X seconds, we still didn't get any data POSTed to that endpoint.

And that holds for all tests, it will be a better one than asserting on the log line.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, these now talk to a real endpoint (mockttp, via a shared disposable helper) and assert on what actually shows up. it waits out a quiet period and asserts nothing was POSTed; the positive tests assert the batch lands. took out the log-line asserts.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And yet here and below I still see assertions on expect(output). Let's get rid of those.

Here is how to write tests (ideally):

One conceptual behavior is asserted per test

Any assertion is either:

  • The behavior we are testing, which we should only be testing once; OR
  • A proxy for the behavior we are testing (for example, sending to the endpoint is hard to test so we test for a log line instead)

In this case, "sending to the endpoint", or "asserting on the data that gets sent" is NOT hard to test, so we don't need to test for a proxy.

Is "this log line appears in the output" the behavior we are testing? If we did want to write a test to say "the log contains X when telemetry is sent", then we would have written a single test to assert exactly that. So there is no reason for this to appear in 10 different tests.

This is either copy/paste detritus, or tests are asserting too much at once. In either case, we should get rid of those asserts.

I know you didn't make this mess, but you did touch it and we always leave the camping grounds cleaner than we found them 😉 .


// Check the trace that endpoint telemetry was never connected
expect(output).toContain('Endpoint Telemetry NOT connected');
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ integTest(
verboseLevel: 3, // trace mode
});

// Check the trace that telemetry was executed successfully
expect(deployOutput).toContain('Telemetry Sent Successfully');
// Check the trace that telemetry was handed to the detached sender that delivers it
expect(deployOutput).toContain('Telemetry dispatched');

const json = fs.readJSONSync(telemetryFile);
expect(json).toEqual([
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ integTest(
modEnv: { DYNAMIC_LAMBDA_PROPERTY_VALUE: 'updated' },
});

// Check the trace that telemetry was executed successfully
expect(deployOutput).toContain('Telemetry Sent Successfully');
// Check the trace that telemetry was handed to the detached sender that delivers it
expect(deployOutput).toContain('Telemetry dispatched');

const json = fs.readJSONSync(telemetryFile);
expect(json).toEqual(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ integTest(
}, // trace mode
);

// Check the trace that telemetry was executed successfully
expect(synthOutput).toContain('Telemetry Sent Successfully');
// Check the trace that telemetry was handed to the detached sender that delivers it
expect(synthOutput).toContain('Telemetry dispatched');

const json = fs.readJSONSync(telemetryFile);
expect(json).toEqual([
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ integTest(

expect(output).toContain('This is an error');

// Check the trace that telemetry was executed successfully despite error in synth
expect(output).toContain('Telemetry Sent Successfully');
// Check the trace that telemetry was handed to the detached sender despite the error in synth
expect(output).toContain('Telemetry dispatched');

const json = fs.readJSONSync(telemetryFile);
expect(json).toEqual([
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ integTest(
{ verboseLevel: 3 }, // trace mode
);

// Check the trace that telemetry was executed successfully
expect(synthOutput).toContain('Telemetry Sent Successfully');
// Check the trace that telemetry was handed to the detached sender that delivers it
expect(synthOutput).toContain('Telemetry dispatched');

const json = fs.readJSONSync(telemetryFile);
expect(json).toEqual([
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import * as net from 'node:net';
import type { AddressInfo } from 'node:net';
import { integTest, withDefaultFixture } from '../../lib';

/**
* Telemetry is delivered by a detached child process, so the CLI must not wait for the POST.
*
* The endpoint here is a black hole: a TCP listener that accepts the connection and then never
* writes a byte, so anything talking to it hangs until its own timeout. Before the sender was
* detached, the flush at the end of the invocation blocked on exactly that, which is why this
* asserts on wall-clock time rather than on output.
*/
integTest(
'cdk synth does not wait for the telemetry endpoint',
withDefaultFixture(async (fixture) => {
const sockets: net.Socket[] = [];
const blackHole = net.createServer((socket) => {
// Accept and hold. Never respond, never close.
sockets.push(socket);
});
await new Promise<void>((ok) => blackHole.listen(0, '127.0.0.1', ok));
const port = (blackHole.address() as AddressInfo).port;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put all of this in a helper, not in the test itself. That way, we can reuse a server component for other integ tests.

For fun and games, make it implement I[Async]Disposable and use using (although a try/finally on a "normal" object with a dispose method will do in a pinch)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

server lives in one shared disposable helper (dispose/using). nothing inline in the test.


try {
// Baseline: the same synth with telemetry switched off entirely.
const disabledStart = Date.now();
await fixture.cdkSynth({
options: [fixture.fullStackName('test-1')],
modEnv: { CDK_DISABLE_CLI_TELEMETRY: 'true' },
});
const disabledMs = Date.now() - disabledStart;

// The same synth, with telemetry pointed at the black hole.
const blackHoleStart = Date.now();
await fixture.cdkSynth({
options: [fixture.fullStackName('test-1')],
modEnv: { TELEMETRY_ENDPOINT: `https://127.0.0.1:${port}/metrics` },
});
const blackHoleMs = Date.now() - blackHoleStart;

const overhead = blackHoleMs - disabledMs;
fixture.log(`synth with telemetry disabled: ${disabledMs}ms, pointed at a black hole: ${blackHoleMs}ms (overhead ${overhead}ms)`);

// The detached sender is what hangs on the black hole, not us. The headroom is generous
// because CI machines are noisy; what this rules out is the CLI blocking on the request
// timeout, which shows up as whole seconds.
expect(overhead).toBeLessThan(2000);
Comment thread
sanjanaravikumar-az marked this conversation as resolved.
Outdated
} finally {
for (const socket of sockets) {
socket.destroy();
}
await new Promise<void>((ok) => blackHole.close(() => ok()));
}
}),
);
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
import * as https from 'node:https';
import type { AddressInfo } from 'node:net';
import * as mockttp from 'mockttp';
import { integTest, withDefaultFixture } from '../../lib';
import { startProxyServer } from '../../lib/proxy';

/**
* Telemetry has to keep working for users behind a corporate proxy.
*
* This matters more than it looks. The POST is made by a detached child process that has no access

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matters more than it looks

???????????

Please get the clanker-speech out of my sight.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes changed

* to the parent's `proxy-agent` instance -- it only has Node built-ins -- so it re-implements HTTP
* CONNECT tunnelling and has to be handed the proxy URL and CA bundle explicitly. This test proves
* that hand-off end to end against the same TLS-terminating proxy the other proxy tests use, whose
* certificate is signed by a throwaway CA that is not in any system trust store.
*
* `TELEMETRY_ENDPOINT` is pointed at a local server rather than the real one, so the test neither
* needs egress to production nor posts real telemetry from CI. What is under test is the CLI ->
* proxy hop: that the child opened a CONNECT tunnel and completed a TLS handshake against a
* certificate it could only have verified using the forwarded CA. The proxy -> endpoint hop is
* deliberately out of scope (the proxy will not trust the local server's self-signed certificate,
* which does not matter -- the proxy records the decrypted request either way).
*/
integTest(
'telemetry is delivered through a configured proxy',
withDefaultFixture(async (fixture) => {
// Stand-in for the telemetry endpoint. Never actually serves a response to the proxy; it only
// needs to occupy a port so the CONNECT target is real.
const { key, cert } = await mockttp.generateCACertificate();
const endpointServer = https.createServer({ key, cert }, (_req, res) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same -- get the details of this server out of the test. Create a disposable class or object to represent the running server.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it, now its the same shared disposable helper and test just spins it up.

res.writeHead(200, { 'content-type': 'application/json' });
res.end('{"ok":true}');
});
await new Promise<void>((ok) => endpointServer.listen(0, '127.0.0.1', ok));
const endpointPort = (endpointServer.address() as AddressInfo).port;
const telemetryEndpoint = `https://localhost:${endpointPort}/metrics`;

const proxyServer = await startProxyServer();
try {
const output = await fixture.cdkSynth({
options: [
fixture.fullStackName('test-1'),
'--proxy', proxyServer.url,
'--ca-bundle-path', proxyServer.certPath,
],
modEnv: {
CDK_HOME: fixture.integTestDir,
TELEMETRY_ENDPOINT: telemetryEndpoint,
},
verboseLevel: 3, // trace
});

// The parent reports the hand-off, not the delivery.
expect(output).toContain('Telemetry dispatched');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we care about what it logs? I don't.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed it so now it only asserts on what reaches the endpoint now.


// Delivery happens after the CLI exits, so poll rather than asserting immediately.
const telemetryRequest = await waitFor(
async () => {
const requests = await proxyServer.getSeenRequests();
return requests.find((req) => req.url.includes(`localhost:${endpointPort}`));
},
30_000,
);

expect(telemetryRequest).toBeDefined();
expect(telemetryRequest!.method).toBe('POST');

// The proxy terminates TLS, so we can read the decrypted body and confirm the child sent a
// well-formed batch (and therefore that both the proxy URL and the CA made it across).
const body = JSON.parse(telemetryRequest!.body.buffer.toString('utf-8'));
expect(Array.isArray(body.events)).toBe(true);
expect(body.events.length).toBeGreaterThan(0);
expect(body.events[0]).toEqual(expect.objectContaining({
identifiers: expect.objectContaining({ sessionId: expect.anything() }),
}));
} finally {
await proxyServer.stop();
await new Promise<void>((ok) => endpointServer.close(() => ok()));
}
}),
);

/**
* Poll `fn` until it returns something truthy, or give up after `timeoutMs`.
*/
async function waitFor<A>(fn: () => Promise<A | undefined>, timeoutMs: number): Promise<A | undefined> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Surely this helper function exists somewhere already? Find it or put this in a shared place.

const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
const result = await fn();
if (result) {
return result;
}
await new Promise((ok) => setTimeout(ok, 500));
}
return undefined;
}
14 changes: 14 additions & 0 deletions packages/aws-cdk/bin/cdk
Original file line number Diff line number Diff line change
@@ -1,4 +1,18 @@
#!/usr/bin/env node
// Publish our own location so the CLI can respawn us as a detached telemetry sender.
// This is the only place that knows it reliably; process.argv[1] may be a .bin symlink,
// the `cdk` alias package's wrapper, or an embedding script.
process.env.CDK_CLI_BIN_PATH = __filename;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah but even if it is a symlink, we are able to launch it (because we are apparently launching it!).

What's the problem?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, is this CLI now communicating with itself via environment variables?

This could have been:

require("../lib/cli/telemetry/sender").main(__filename);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, changing it back


// That detached sender is this same script with a flag. Dispatch before requiring the CLI,
// whose bundle costs ~600ms to load and which the sender does not need.
if (process.env.CDK_TELEMETRY_SENDER === '1') {
require("../lib/cli/telemetry/sender").main();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe call that file sender-bundle.js as well, just for clarity.

// Relies on the CommonJS module wrapper (modules are functions); would be a SyntaxError if this
// file ever became native ESM.
return;
}

// source maps must be enabled before importing files
process.setSourceMapsEnabled(true);
const { cli } = require("../lib");
Expand Down
7 changes: 4 additions & 3 deletions packages/aws-cdk/lib/cli/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -116,13 +116,14 @@ export async function exec(args: string[], synthesizer?: Synthesizer): Promise<n
});

// Always create and use ProxyAgent to support configuration via env vars
const proxyAgent = await new ProxyAgentProvider(ioHelper).create({
proxyAddress: configuration.settings.get(['proxy']),
const proxyUrl: string | undefined = configuration.settings.get(['proxy']);
const { agent: proxyAgent, caCert } = await new ProxyAgentProvider(ioHelper).create({
proxyAddress: proxyUrl,
caBundlePath: configuration.settings.get(['caBundlePath']),
});

try {
await ioHost.startTelemetry(argv, configuration.context, proxyAgent);
await ioHost.startTelemetry(argv, configuration.context, { proxyUrl, caCert });
} catch (e: any) {
await ioHost.asIoHelper().defaults.trace(`Telemetry instantiation failed: ${e.message}`);
}
Expand Down
31 changes: 28 additions & 3 deletions packages/aws-cdk/lib/cli/io-host/cli-io-host.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import type { Agent } from 'node:https';
import * as util from 'node:util';
import { RequireApproval } from '@aws-cdk/cloud-assembly-schema';
import { ToolkitError } from '@aws-cdk/toolkit-lib';
Expand All @@ -9,6 +8,7 @@ import type { IoHelper, ActivityPrinterProps, IActivityPrinter, IoMessageMaker,
import { asIoHelper, IO, isMessageRelevantForLevel, CurrentActivityPrinter, HistoryActivityPrinter } from '../../../lib/api-private';
import type { Context } from '../../api/context';
import { StackActivityProgress } from '../../commands/deploy';
import { cliBinPath } from '../telemetry/cli-bin-path';
import { canCollectTelemetry } from '../telemetry/collect-telemetry';
import { cdkCliErrorName } from '../telemetry/error';
import type { EventResult } from '../telemetry/messages';
Expand Down Expand Up @@ -37,6 +37,29 @@ type CliAction =
| 'cli-telemetry'
| 'none';

/**
* How telemetry should reach the network.
*
* The endpoint sink does not make the request itself -- it hands off to a detached child process
* that only has Node built-ins available. That child cannot be given an `Agent`, so the proxy and
* certificate configuration have to travel as plain data instead.
*/
export interface TelemetryNetworkOptions {
/**
* Proxy configured via `--proxy` or the `proxy` setting.
*
* @default - the sender resolves it from the environment
*/
readonly proxyUrl?: string;

/**
* Contents of the CA bundle configured via `--ca-bundle-path` or `AWS_CA_BUNDLE`.
*
* @default - only the system trust store
*/
readonly caCert?: string;
}

export interface CliIoHostProps {
/**
* The initial Toolkit action the hosts starts with.
Expand Down Expand Up @@ -376,7 +399,7 @@ export class CliIoHost implements IIoHost, ObservableIoHost {
this.routeStackActivityToPrinter();
}

public async startTelemetry(args: any, context: Context, proxyAgent?: Agent) {
public async startTelemetry(args: any, context: Context, network: TelemetryNetworkOptions = {}) {
// eslint-disable-next-line @typescript-eslint/no-require-imports
const config = require('../cli-type-registry.json');
const validCommands = Object.keys(config.commands);
Expand Down Expand Up @@ -407,8 +430,10 @@ export class CliIoHost implements IIoHost, ObservableIoHost {
try {
sinks.push(new EndpointTelemetrySink({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not a new sink type? Why not a SubprocessTelemetrySink ?

Seems more accurate.

ioHost: this,
agent: proxyAgent,
endpoint: telemetryEndpoint,
binCdkPath: cliBinPath(),
proxyUrl: network.proxyUrl,
caCert: network.caCert,
}));
await this.asIoHelper().defaults.trace('Endpoint Telemetry connected');
} catch (e: any) {
Expand Down
35 changes: 30 additions & 5 deletions packages/aws-cdk/lib/cli/proxy-agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,24 +21,49 @@ interface ProxyAgentOptions {
readonly caBundlePath?: string;
}

/**
* The proxy configuration resolved for this invocation.
*/
export interface ResolvedProxyAgent {
/**
* The agent to pass to anything making HTTPS requests in this process.
*/
readonly agent: ProxyAgent;

/**
* Contents of the resolved CA bundle, if one was configured.
*
* Exposed because the detached telemetry sender cannot use `agent` -- it runs in another process
* and only has Node built-ins -- so it needs the certificate itself.
*
* @default - no CA bundle was configured
*/
readonly caCert?: string;
}

export class ProxyAgentProvider {
private readonly ioHelper: IoHelper;

public constructor(ioHelper: IoHelper) {
this.ioHelper = ioHelper;
}

public async create(options: ProxyAgentOptions) {
public async create(options: ProxyAgentOptions): Promise<ResolvedProxyAgent> {
// Force it to use the proxy provided through the command line.
// Otherwise, let the ProxyAgent auto-detect the proxy using environment variables.
const getProxyForUrl = options.proxyAddress != null
? () => Promise.resolve(options.proxyAddress!)
: undefined;

return new ProxyAgent({
ca: await this.tryGetCACert(options.caBundlePath),
getProxyForUrl,
});
const caCert = await this.tryGetCACert(options.caBundlePath);

return {
agent: new ProxyAgent({
ca: caCert,
getProxyForUrl,
}),
caCert,
};
}

private async tryGetCACert(bundlePath?: string) {
Expand Down
Loading
Loading