How the remote instrumenter works
The CloudFormation stack#
Deploying the template creates a Lambda function named datadog-remote-instrumenter that runs on ARM64 with the Node.js 24.x runtime, reserved concurrency of 1, a 900-second timeout, and 512 MB of memory. It carries the Datadog Extension layer and a Remote Instrumentation layer, and it uses an S3 bucket to persist state about what it has already instrumented.
The stack's top-level resources are: LambdaFunction, LambdaExecutionRole, InstrumenterCodeSigningConfig, CloudTrail, S3Bucket, S3BucketPolicy, LambdaEventRule, LambdaEventRulePermission, LambdaSchedulerRole, LambdaInvocationScheduler, and a CloudFormationLifeCycle custom resource that runs setup and teardown logic on stack create, update, and delete.
Two things trigger the instrumenter to run:
- An EventBridge rule watches CloudTrail for Lambda management API calls —
CreateFunction,UpdateFunctionConfiguration,TagResource, andUntagResource— so a newly created or newly tagged function gets picked up quickly. Events produced by the instrumenter's own calls are filtered out to avoid a feedback loop. - An EventBridge Scheduler rule invokes the instrumenter every 5 minutes regardless of Lambda API activity, as a backstop that catches configuration changes and retries functions that failed on a previous pass.
The IAM role attached to the instrumenter is scoped narrowly to the Lambda and S3 operations it needs, and the stack supports an optional code signing configuration for accounts that require it.
The source code (src/)#
The Lambda's entry point is handler.ts. It branches on five kinds of event, based on the source:
| Event | What happens |
|---|---|
| CloudFormation stack create | Fetches remote config, enumerates all Lambda functions, tags them, and instruments the matching ones. Always responds SUCCESS to CloudFormation, even if individual functions failed, so the stack doesn't roll back over an unrelated function's error. |
| CloudFormation stack delete | Empties the S3 state bucket and un-instruments every function the stack had touched, so deleting the stack cleanly removes Datadog layers again. |
| CloudFormation stack update | Deletes the stored config hash so the next scheduled run treats the configuration as changed and re-evaluates every function. |
| Lambda management event (from the EventBridge rule) | Instruments or re-evaluates the one function named in the event. |
| Scheduled invocation (every 5 minutes) | Compares a hash of the current remote configuration against the hash stored in S3. If it changed, re-instruments all functions and updates the hash. If it's unchanged but some functions previously failed, retries only those. |
Supporting modules referenced by the handler: aws-resources.ts (Lambda, S3, CloudTrail, and tagging clients), functions.ts (enumerating and tagging functions), instrument.ts (applying or removing Datadog layers and environment variables on a function), config.ts (fetching and caching remote configuration), consts.ts (runtime-to-layer mappings and status constants), apply-state.ts and error-storage.ts (tracking what succeeded, failed, or was skipped, and persisting errors for retry), tag.ts (tag matching logic), lambda-event.ts (parsing CloudTrail-sourced events), metrics.ts and logger.ts (telemetry about the instrumenter itself), and sleep.ts (retry backoff).
Remote configuration#
The instrumenter's targeting rules — which functions to instrument and how — are delivered as remote configuration from Datadog, not baked into the CloudFormation parameters. Per config.ts, the instrumenter polls a local Datadog Agent Remote Configuration endpoint, caches the result briefly to avoid excessive calls, and retries a couple of times if it detects a change but the new configuration hasn't propagated yet. This is what lets you change which functions are targeted from the Datadog UI without redeploying the CloudFormation stack.
Continue to Configuration Reference for the deployable parameters, or Function Targeting for how the matching rules themselves work.