Observability

Overview

Whenever changes occur in your ngrok account or when traffic transits through your endpoints, an event is fired. You may subscribe to these events and in some cases, filter them to those relevant to you and publish them to any number of destinations.

ngrok's event system was designed for three primary use cases:

• Sending logs of your ngrok traffic to external services such as Datadog and Amazon CloudWatch Logs
• Sending audit logs of ngrok configuration changes to your SIEM
• Enabling you to programmatically respond to events on your ngrok account

There are over 40 event types published via ngrok at the time of writing. They break down into two important classes of event:

• Traffic Events: When traffic transits through your endpoints like processing an HTTP request or TCP connection
• Audit Events: Changes to your account like create/update/delete of Domains, API Keys, IP Policies, etc.

Traffic Inspector

Traffic Events are also captured in ngrok's Traffic Inspector, which provides an easy, built-in way to debug and test traffic flowing through your ngrok account.

Concepts

ngrok's event logging system is composed of three simple primitives. You create an Event Subscription to subscribe to a set of Event Sources and publish those events to one or more Event Destinations.

• Event Subscriptions: Define which Event Sources to capture and which Event Destinations to publish to.
• Event Sources: The ngrok event types to capture and optionally a set of fields to record and a boolean filter expression.
• Event Destinations: Where events are sent, e.g. Datadog, CloudWatch, Kinesis.

Quickstart

You can create your first Event Subscription on the Events page of your ngrok Dashboard to begin capturing and publishing events.

We also publish guides to get started with each of ngrok's Event Destinations:

• AWS CloudWatch Logs
• AWS Firehose
• AWS Kinesis
• Azure Logs Ingestion
• Datadog Logs

Event Subscriptions

Event subscriptions define which Event Sources to capture and which destinations to publish to. If you're familiar with other event systems, they may call this a listener, a hook, a probe or a tap.

When you create an Event Subscription, you select:

• One or more Event Sources to subscribe to
• One or more Event Destinations to publish the subscribed events to

Event Subscriptions can be created on the ngrok Dashboard or via the Event Subscription API Resource.

Event Sources

Event Sources choose which events an Event Subscription captures. Other event systems may call this a selector.

The ngrok Event Sources reference documents the full list of Event Sources and their fields.

Versioning and Payload Stability

Event Sources include a version number in their name. Each Event Source ends with a version string after a dot, e.g. api_key_created.v0.

New fields may be added to the payloads of an Event Source without a version change. Any other change to the event representation will lead to the introduction of a new version of the Event Source.

Audit Events

ngrok emits Audit Events when changes are made to your account like create/update/delete of Domains, API Keys, IP Policies, etc.

Traffic Events

When traffic transits through your endpoints, ngrok emits Traffic Events like processing an HTTP request or TCP connection.

Traffic Events support the ability to select exactly which fields you'd like to include in the captured event. This provides you with control over how much data is sent to the destinations.

Traffic Events also support the ability to be filtered. As part of the Event Source definition you can specify a CEL expression which is evaluated against each event to determine if it should be captured. See the Filters section for further detail.

Filters

You may specify a filter on Traffic Events since the velocity of these events can be quite high. Filters are a boolean expression defined in Google's Common Expression Language (CEL). Filters are evaluated on each event as it is published to determine whether it should be sent to the Event Subscription's destinations.

Traffic Events are often high cardinality. We support filters because you may only interested in logging a subset of traffic (e.g. specific client IPs). Filters also enable you to control spend on downstream destinations where you incur costs based on the quantity of data processed and stored.

The event object is made available to the CEL filter expression as the ev variable. ev corresponds to the object field of the captured event. Use standard JSONPath syntax to access the fields of the event you want to use in your filters.

Filter expressions have the following limitations:

• Only some fields of an event are available in the ev object. The Event Source's reference documentation will specify whether a field may be evaluated in a filter expression.
• Event metadata fields like principal or event_timestamp are not available

Filter Examples

Filter for connections from a particular hostname

ev.conn.server_name.matches("ngrok-docs-examples\\.ngrok\\.dev")

Filter for connections to an endpoint that didn't use HTTPS

ev.conn.server_name.matches(".*-your-org\\.ngrok\\.dev") &&
ev.conn.server_port == 80

Filter for connections to a hostname that exclude traffic from a client IP

ev.conn.client_ip != "2601:0:8200:0:4cd7:fd52:0:7823" &&
ev.conn.server_name == "ngrok-docs-examples.ngrok.dev"

Event Destinations

An Event Destination encapsulates the configuration to publish events to other systems. Other event systems may call this a sink or output.

Event Destinations are typically third-party logging aggregators. The following destinations are currently supported:

• Amazon CloudWatch
• Amazon Firehose
• Amazon Kinesis
• AWS S3 (via Kinesis Firehose)
• Azure Logs Ingestion
• Datadog Logs

Each destination requires provider-specific configuration. If you create a destination in the ngrok dashboard, you'll be prompted to send a test event to verify the integration.

note

When configuring AWS destinations you'll be prompted to optionally download a small helper script which will automatically configure the appropriate IAM objects necessary for integration. You may also set these values up via the AWS Console or tools like Terraform or Pulumi.

Amazon S3 is not a directly supported destination. Instead, configure Amazon Firehose to deliver events into an S3 bucket.

Event Payloads

Events are serialized as JSON when they are published to a destination.

All events include the following fields:

Name	Description	Example
account_id	unique identifier for the account, always prefixed with ac_	ac_2OtNvAlhso10Gx6s7eupzX3F98q
event_id	unique identifier for this event, always prefixed with ev_	ev_1vPlyBW3OR44bpPphS4HIZyajDD
event_type	identifies the object, action, and version of the event	ip_policy_created.v0
event_timestamp	timestamp of when the event fired in RFC 3339 format	2021-07-16T21:44:37Z
object	the event object	See examples below
principal	an object of the principal who actioned this event, null for traffic events	See example below

The object property of the event is distinct for each Event Source and contain a JSON object with additional information about the event.

For Audit Events, the object representation is identical to its API resource at the time of capture.

Traffic Events define their own object representation because they have no corresponding API resource definition.

Principal Object

The principal object in every event describe the user or bot user responsible for initiating the event. Principal is defined for all Audit Events and it is null for Traffic Events (since they do not include a credential).

Name	Description	Example
id	unique identifier for the principal, either a user or bot	usr_2OtNv9qH5Nk4NuNeszZ39gBxZ4H
subject	human readable unique identifier for the principal, either a user email or a bot user name	foo@example.com
source	where the principal initiated the event, either 'Dashboard' or 'API'	API
credential	id and uri resource for the credential used for authentication, null if source is 'Dashboard'	{"id": "ak_2Oxt94wYsBTLwFUoMZcJRvJTaub","uri": "https://api.ngrok.com/api_keys/ak_2Oxt94wYsBTLwFUoMZcJRvJTaub"}

Example Payloads

ip_policy_created.v0 event

{
	"event_id": "ev_25X2AsJ5xpvuOParTYUQWe12XKo",
	"event_type": "ip_policy_created.v0",
	"event_timestamp": "2022-02-23T23:29:29Z",
	"account_id": "ac_2OtNvAlhso10Gx6s7eupzX3F98q",
	"principal": {
		"id": "usr_2OtNv9qH5Nk4NuNeszZ39gBxZ4H",
		"subject": "foo@example.com",
		"source": "API",
		"credential": {
			"id": "ak_2Oxt94wYsBTLwFUoMZcJRvJTaub",
			"uri": "https://api.ngrok.com/api_keys/ak_2Oxt94wYsBTLwFUoMZcJRvJTaub"
		}
	},
	"object": {
		"id": "ipp_25X2Ao39z73FlVQKZ1iReMPe6Qv",
		"uri": "https://api.ngrok.com/ip_policies/ipp_25X2Ao39z73FlVQKZ1iReMPe6Qv",
		"created_at": "2022-02-23T23:29:29Z",
		"description": "Home network IP",
		"metadata": "",
		"action": "allow"
	}
}

http_request_complete.v0 event

{
	"event_id": "ev_25X3yFS6TDkig1KDJWIc4nnJO0c",
	"event_type": "http_request_complete.v0",
	"event_timestamp": "2022-02-23T23:44:16Z",
	"account_id": "ac_2OtNvAlhso10Gx6s7eupzX3F98q",
	"object": {
		"conn": {
			"client_ip": "2601:0:8200:9e:4cd7:0:c97f:7823",
			"server_name": "ngrok-docs-example.ngrok.app",
			"server_port": ""
		},
		"http": {
			"request": {
				"first_byte_ts": null,
				"last_byte_ts": null,
				"method": "GET",
				"url": {
					"path": "/docs/obs"
				},
				"version": "HTTP/2.0"
			},
			"response": {
				"body_length": 13079,
				"first_byte_ts": "2022-02-23T23:44:16.732791273Z",
				"last_byte_ts": "2022-02-23T23:44:16.737257209Z",
				"status_code": 200
			}
		}
	}
}

tcp_connection_closed.v0 event

{
	"event_id": "ev_25X4osod1q306srserDeFyghTC4",
	"event_type": "tcp_connection_closed.v0",
	"event_timestamp": "2022-02-23T23:51:14Z",
	"account_id": "ac_2OtNvAlhso10Gx6s7eupzX3F98q",
	"object": {
		"conn": {
			"bytes_in": 3437,
			"bytes_out": 90256,
			"client_ip": "2601:0:8200:9e:4cd7:0:c97f:7823",
			"end_ts": "2022-02-23T23:51:14.005372199Z",
			"server_name": "ngrok-docs-example.ngrok.app",
			"server_port": "",
			"start_ts": "2022-02-23T23:44:16.528374173Z"
		}
	}
}

Pricing

Events are available to all ngrok users with a free tier that includes the transmission of up to 10,000 events per month.

The Traffic Inspector is included with all ngrok accounts with a retention period of 3 days. An additional 90 days can be purchased as an add-on to any production plan.