Please reach out before starting work on any major code changes. This will ensure we avoid duplicating work, or that your code can't be merged due to a rapidly changing base. If you would like support for a module that is not listed, contact support to share a request.
Changes should be incremental and understandable. As much as possible, large-scale efforts should be broken up into many PRs over time for better reviewability. If a feature would require more changes to be "complete" it's fine to land partial changes if they are not wired up to anything yet, so long as tests are included which at least prove those parts work in isolation.
There are great benefits to taking a measured and iterative approach to improvement. When working on code in fewer places there is far less risk of running into merge conflicts or incompatibilities with other systems. Keeping contributions small makes them easy to review which makes that much quicker to land. Additionally, keeping things small and iterative makes it easier for other teams to review and understand what the code does.
Sometimes code can be self-documenting, but often it can't. That is especially true to someone reviewing code they haven't worked on. Be conscious of writing code in a self-describing way and leave comments anywhere that self-description fails. This goes a long way towards making even complex code coherent to one not already familiar with it.
Try to write code in a way the describes the intent when read. For example, verbs can be used for function and method names to communicate that they are used to do some specific action. In doing so it becomes clear when referenced by name elsewhere that it is a function and what the function is meant to do. If a function can not be described with a simple verb it's probably too complex or does too many things.
Very dense code is hard to read. It helps to make use of empty lines to separate logical groupings of statements. Long lines should be split up into multiple lines to make them more readable. Complex objects or arrays should generally be split over several lines. Sometimes it's a good idea to assign a variable only to immediately use it in a call as it can be more descriptive than just using the expression in place. It's not always clear what an argument is for if it doesn't visibly have a name somehow. Remember, lines are free, our time is not.
Large refactors should generally be avoided in favour of iterative approaches. For example, rather than rewriting how every plugin works, one might make a special-case plugin that works a bit different for their particular use-case. If several dozen files need to change to add a feature we've probably done something wrong.
Sometimes new patterns or new ideas emerge which would be a substantial improvement over the existing state. It can be tempting to want to go all-in on a new way to do something, but the code churn can be hard to manage. It's best to introduce such new things incrementally and advocate for their adoption gradually through the rest of the codebase. As old systems are gradually phased out, the infrastructure which supports them can be deleted or relegated to lazy-loading only if and when that specific part of the system needs to be used.
It's very difficult to know if a change is valid unless there are tests to prove it. As an extension of that, it's also difficult to know the use of that code is valid if the way it is integrated is not properly tested. For this reason we generally favour integration tests over unit tests. If an API is expected to be used in different places or in different ways then it should generally include unit tests too for each unique scenario, however great care should be taken to ensure unit tests are actually testing the logic and not just testing the mocks. It's a very common mistake to write a unit test that abstracts away the actual use of the interface so much that it doesn't actually test how that interface works in real-world scenarios. Remember to test how it handles failures, how it operates under heavy load, and how it impacts usability of what its purpose is.
Observability products tend to have quite a bit of their behavior running in app code hot paths. It's important we extensively benchmark anything we expect to have heavy use to ensure it performs well and we don't cause any significant regressions through future changes. Measuring once at the time of writing is insufficient--a graph with just one data point is not going to tell you much of anything.
We always backport changes from master to older versions to avoid release lines drifting apart and to prevent merge conflicts. We should be diligent about keeping breaking changes to a minimum and ensuring we don't use language or runtime features which are too new. This way we can generally be confident that a change can be backported.
Breaking changes must be guarded by version checks so they can land in master and be safely backported to older versions. Check the major version of the dd-trace package using the version.js module in the root of the project:
const { DD_MAJOR } = require('./version')
if (DD_MAJOR >= 6) {
// New behavior for v6+
} else {
// Old behavior for v5 and earlier
}To reduce the surface area of a breaking change, the breaking aspects could be placed behind a flag which is disabled by default or isolated to a function. In the next major the change would then be just to change the default of the flag or to start or stop calling the isolated function. By isolating the breaking logic it also becomes easier to delete later when it's no longer relevant on any release line.
Currently we do not have CI to test PRs for mergeability to past release lines, but we intend to expand our CI to include that in the future. For the time being, it's recommended when developing locally to try to cherry-pick your changes onto the previous vN.x branches to see if the tests pass there too.
This library follows the semantic versioning standard, but there are some subtleties left under-specified so this section is meant to clarify exactly how we interpret the meaning of semver. Additionally, it exists to communicate that we also use semver labels on all PRs to indicate which type of release the change should land in. Outside contributions should be evaluated and a semver label selected by the relevant team.
If the change is a bug or security fix, it should be labelled as semver-patch. These changes should generally not alter existing behavior in any way other than to correct the specific issue.
Any addition of new functionality should be labelled as semver-minor and should not change any existing behavior either in how any existing API works or in changing the contents or value of any existing data being reported except in purely additive cases where all existing data retains its prior state. Such changes may include new configuration options which when used will change behavior, or may include the addition of new data being captured such as a new instrumentation, but should not impact the current operating design of any existing features.
In the event that some existing functionality does need to change, as much as possible the non-breaking aspects of that change should be made in a semver-minor PR and the actually breaking aspects should be done via a follow-up PR with only the specific aspects which are breaking. Remember to always consider backportability.
When writing major changes we use a series of labels in the form of dont-land-on-vN.x where N is the major release line which a PR should not land in. Every PR marked as semver-major should include these tags. These tags allow our branch-diff tooling to work smoothly as we can exclude PRs not intended for the release line we're preparing a release proposal for. The semver-major labels on their own are not sufficient as they don't encode any indication of from which releases they are a major change.
For outside contributions we will have the relevant team add these labels when they review and determine when they plan to release it.
We follow an all-green policy which means that for any PR to be merged all tests must be passing. If a test is flaky or failing consistently the owner of that test should make it a priority to fix that test and unblock other teams from landing changes. For outside contributors there are currently several tests which will always fail as full CI permission is required. For these PRs our current process is for the relevant team to copy the PR and resubmit it to run tests as a user with full CI permission.
Eventually we plan to look into putting these permission-required tests behind a label which team members can add to their PRs at creation to run the full CI and can add to outside contributor PRs to trigger the CI from their own user credentials. If the label is not present there will be another action which checks the label is present. Rather than showing a bunch of confusing failures to new contributors it would just show a single job failure which indicates an additional label is required, and we can name it in a way that makes it clear that it's not the responsibility of the outside contributor to add it. Something like approve-full-ci is one possible choice there.
Always search the codebase first before creating new code to avoid duplicates. Check for existing utilities, helpers, or patterns that solve similar problems. Reuse existing code when possible rather than reinventing solutions.
All commits in a pull request must be signed. We require commit signing to ensure the authenticity and integrity of contributions to the project.
Datadog employees: We recommend using the sign-pull-request tool for easy signing of commits.
You can also sign your commits manually using one of the following methods:
If you have already created commits without signing them, you can sign them retroactively by using an interactive rebase:
$ git rebase --exec 'git commit --amend --no-edit -n -S' -i <base-branch>Since this project supports multiple Node.js versions, using a version manager such as nvm is recommended. If you're unsure which version of Node.js to use, just use the latest version, which should always work.
We use yarn 1.x for its workspace functionality, so make sure to install that as well. The easiest way to install yarn 1.x with with npm:
$ npm install -g yarnTo install dependencies once you have Node and yarn installed, run this in the project directory:
$ yarnUse kebab-case for file names (e.g., my-module.js, not myModule.js).
Organize imports in the following order (each group separated by an empty line):
- Node.js core modules first (sorted alphabetically) - always prefix with
node: - Third-party modules (sorted alphabetically)
- Internal imports (sorted by path proximity first - closest first - then alphabetically)
Example:
const fs = require('node:fs')
const path = require('node:path')
const express = require('express')
const lodash = require('lodash')
const { myConf } = require('./config')
const { foo } = require('./helper')
const log = require('../log')Follow the ECMAScript standard and Node.js APIs supported by Node.js 18.0.0. Never use features or APIs only supported in newer versions unless explicitly required. If newer APIs are needed, guard them with version checks using the version.js module located in the root of the project:
const { NODE_MAJOR } = require('./version')
if (NODE_MAJOR >= 20) {
// Use Node.js 20+ API
}This tracer runs in application hot paths, so performance is critical:
- Avoid
async/awaitand promises in production code (they add overhead). Use callbacks or synchronous patterns instead. Async/await is acceptable in test files and worker threads. - Minimize memory allocations in frequently-called code paths
- Prefer imperative loops over functional array methods (
for-of,for,whileinstead ofmap(),forEach(),filter()) to avoid closure overhead and intermediate arrays - Avoid creating unnecessary objects, closures, or arrays
- Reuse objects and buffers where possible
Example of preferred loop style:
// ❌ Avoid - creates closure overhead
items.forEach(item => {
process(item)
})
// ✅ Prefer - no closure overhead
for (const item of items) {
process(item)
}
// ❌ Avoid - loops over data multiple times, creates intermediate array, adds closure overhead
const result = items
.filter(item => item.active)
.map(item => item.value)
// ✅ Prefer - single loop, no intermediate array
const result = []
for (const item of items) {
if (item.active) {
result.push(item.value)
}
}The tracer should never crash user applications. Catch errors and log them with log.error() or log.warn() as appropriate. Resume normal operation if possible, or disable the plugin/subsystem if not.
Use the log module (located at packages/dd-trace/src/log/index.js) for all logging:
const log = require('../log')
log.debug('Debug message with value: %s', someValue)
log.info('Info message')
log.warn('Warning with data: %o', objectValue)
log.error('Error reading file %s', filepath, err)Important: Use printf-style formatting (%s, %d, %o) instead of template strings to avoid unnecessary string concatenation when logging is disabled.
For expensive computations in the log message itself, use a callback function:
// Callback is only executed if debug logging is enabled
log.debug(() => `Processed data: ${expensive.computation()}`)When logging errors, pass the error object as the last argument after the format string:
log.error('Error processing request', err)
// or with additional context:
log.error('Error reading file %s', filename, err)To enable debug logging when running tests or applications:
# Run application with debug logging
DD_TRACE_DEBUG=true node your-app.js
# Run a specific test suite with debug logging
DD_TRACE_DEBUG=true yarn test:debuggerDocument all APIs with TypeScript-compatible JSDoc to ensure proper types without using TypeScript. This enables type checking and IDE autocompletion while maintaining the JavaScript codebase. Use TypeScript type syntax in JSDoc annotations (e.g., @param {string}, @returns {Promise<void>}).
Never use the type any - be specific.
To add a new configuration option:
- Add the default value in
packages/dd-trace/src/config/defaults.js - Map the environment variable in
packages/dd-trace/src/config/index.js(add to destructuring in#applyEnvironment()method) - Add TypeScript definitions in
index.d.ts - Add to telemetry name mapping (if applicable) in
packages/dd-trace/src/telemetry/telemetry.js - Update supported configurations in
packages/dd-trace/src/config/supported-configurations.json - Document the option in
docs/API.md(for non-internal/experimental options) - Add tests in
packages/dd-trace/test/config/index.spec.js
Naming Convention: Size/time-based config options should have unit suffixes (e.g., timeoutMs, maxBytes, intervalSeconds).
Plugins are modular code components in packages/datadog-plugin-*/ directories that integrate with specific third-party libraries and frameworks. They subscribe to diagnostic channels to receive instrumentation events and handle APM tracing logic, feature-specific logic, and more.
To create a new plugin for a third-party package, follow these steps:
mkdir -p packages/datadog-plugin-<plugin-name>/src- Copy an
index.jsfile from another plugin to use as a starting point:cp packages/datadog-plugin-kafkajs/src/index.js packages/datadog-plugin-<plugin-name>/src - Edit index.js as appropriate for your plugin
mkdir -p packages/datadog-plugin-<plugin-name>/test- Create an packages/datadog-plugin-/test/index.spec.js file and add the necessary tests. See other plugin tests for inspiration to file structure.
- Edit index.spec.js as appropriate for your new plugin
- Add entries to the following files for your new plugin:
packages/dd-trace/src/plugins/index.jsindex.d.tsdocs/test.tsdocs/API.md.github/workflows/apm-integrations.yml(see Adding a Plugin Test to CI)
The pg-native package requires pg_config to be in your $PATH to be able to install.
Please refer to the "Install" section of the pg-native documentation for how to ensure your environment is configured correctly.
When developing, it's often faster to run individual test files rather than entire test suites.
To target specific tests, use the --grep flag with mocha to match test names:
yarn test:debugger --grep "test name pattern"
yarn test:appsec --grep "specific test"This project uses mocha for testing.
Use the Node.js core assert library for assertions in tests. Import from node:assert/strict to ensure all assertions use strict equality without type coercion:
const assert = require('node:assert/strict')
assert.equal(actual, expected)
assert.deepEqual(actualObject, expectedObject)For asserting that an object contains certain properties (deeply), use assertObjectContains from integration-tests/helpers/index.js:
const { assertObjectContains } = require('../helpers')
// Assert an object contains specific properties (actual object can have more)
assertObjectContains(response, {
status: 200,
body: { user: { name: 'Alice' } }
})This helper performs partial deep equality checking and provides better error messages than individual assertions.
Never rely on actual time passing in unit tests. Tests that use setTimeout(), setInterval(), or Date.now() should use sinon's fake timers to mock time. This makes tests:
- Run instantly instead of waiting for real time
- Deterministic and reliable (no timing-related flakiness)
- Easier to reason about
Example:
const sinon = require('sinon')
describe('my test', () => {
let clock
beforeEach(() => {
clock = sinon.useFakeTimers()
})
afterEach(() => {
clock.restore()
})
it('should handle timeout', () => {
let called = false
setTimeout(() => { called = true }, 1000)
clock.tick(1000) // Instantly advance time by 1 second
assert.equal(called, true)
})
})Use clock.tick(ms) to advance time, clock.restore() to restore real timers, and clock.reset() to reset fake time to 0.
Coverage is measured with nyc. To check coverage for your changes, use the :ci variant of the test scripts:
# Run tests with coverage for specific components
yarn test:debugger:ci
yarn test:appsec:ci
yarn test:llmobs:sdk:ci
yarn test:lambda:ciCoverage Philosophy: Given the nature of this library (instrumenting third-party code, hooking into runtime internals), unit tests can become overly complex when everything needs to be mocked. Integration tests that run in sandboxes don't count towards nyc's coverage metrics, so coverage numbers may look low even when code is well-tested. Don't add redundant unit tests solely to improve coverage numbers.
Before running plugin tests, the supporting docker containers need to be running. You can attempt to start all of them using docker-compose, but that's a drain on your system, and not all the images will even run at all on AMD64 devices.
Note The
aerospike,couchbase,grpcandoracledbinstrumentations rely on native modules that do not compile on ARM64 devices (for example M1/M2 Mac)
- their tests cannot be run locally on these devices.
Instead, you can follow this procedure for the plugin you want to run tests for:
- Check the CI config in
.github/workflows/*.ymlto see what the appropriate values for theSERVICESandPLUGINSenvironment variables are for the plugin you're trying to test (noting that not all plugins requireSERVICES). For example, for theamqplibplugin, theSERVICESvalue israbbitmq, and thePLUGINSvalue isamqplib. - Run the appropriate docker-compose command to start the required services. For example, for the
amqplibplugin, you would run:docker compose up -d rabbitmq. - Run
yarn services, with the environment variables set above. This will install any versions of the library to be tested against into theversionsdirectory, and check that the appropriate services are running prior to running the test. - Now, you can run
yarn test:pluginswith the environment variables set above to run the tests for the plugin you're interested in.
To wrap that all up into a simple few lines of shell commands, here is all of the above, for the amqplib plugin:
# These are exported for simplicity, but you can also just set them inline.
export SERVICES="rabbitmq" # retrieved from .github/workflows/apm-integrations.yml
export PLUGINS="amqplib" # retrieved from .github/workflows/apm-integrations.yml
docker compose up -d $SERVICES
yarn services
yarn test:plugins # This one actually runs the tests. Can be run many times.You can also run the tests for multiple plugins at once by separating them with a pipe (|) delimiter. For example, to run the tests for the amqplib and bluebird plugins:
PLUGINS="amqplib|bluebird" yarn test:pluginsThere are several types of unit tests, for various types of components. The following commands may be useful:
# Tracer core tests (i.e. testing `packages/dd-trace`)
$ yarn test:trace:core
# "Core" library tests (i.e. testing `packages/datadog-core`
$ yarn test:core
# Instrumentations tests (i.e. testing `packages/datadog-instrumentations`
$ yarn test:instrumentationsSeveral other components have test commands as well. See package.json for
details.
When running integration tests, some packages are installed from npm into temporary sandboxes.
If running locally without an internet connection,
it's possible to use the environment variable OFFLINE=true to make yarn use the --prefer-offline flag,
which will use the local yarn cache instead of fetching packages from npm.
The plugin tests run on pull requests in Github Actions. Each plugin test suite has its own Github job, so adding a new suite to CI
requires adding a new job to the Github Actions config. The file containing these configs is .github/workflows/apm-integrations.yml.
You can copypaste and modify an existing plugin job configuration in this file to create a new job config.
We use ESLint to make sure that new code conforms to our coding standards.
To run the linter, use:
$ yarn lintThis also checks that the LICENSE-3rdparty.csv file is up-to-date, and checks
dependencies for vulnerabilities.
Our microbenchmarks live in benchmark/sirun. Each directory in there
corresponds to a specific benchmark test and its variants, which are used to
track regressions and improvements over time.
In addition to those, when two or more approaches must be compared, please write
a benchmark in the benchmark/index.js module so that we can keep track of the
most efficient algorithm. To run your benchmark, use:
$ yarn bench