SSM Port Forwarding: Testing a Private-Subnet Service from Your Laptop Without Opening Any Inbound Path

TL;DR

• Situation: a FastAPI service runs on ECS in private subnets, and the environment allows no inbound access at all.
• Approach: SSM Port Forwarding, which connects localhost:8888 on the laptop to port 8000, where uvicorn listens inside the container.
• How it works: the laptop and the container each dial out to the SSM service, and SSM relays traffic between them. IAM decides who can open the tunnel.

1. The service is deployed, but you can’t reach it

Our system runs in the AWS environment of a financial-industry client, and that environment allows no public entry point. The backend is a FastAPI service on ECS behind an internal ALB, and every workload sits in a private subnet. There’s no internet-facing load balancer, no bastion host, and no VPN.

The trouble starts at testing time: the service is deployed, but requests from a laptop never arrive.

The internal ALB’s DNS name resolves from the public internet, but it resolves to private IPs (all names and IPs in this post are illustrative):

1
2

dig +short internal-my-alb-1234567890.us-east-1.elb.amazonaws.com
# → 10.0.1.37, 10.0.3.47

flowchart LR
    DEV[Developer laptop]
    ALB[internal ALB<br>10.0.x.x private IP]
    SVC[ECS service<br>private subnet]
    DEV -.->|private IP not routable from internet<br>curl timeout| ALB
    ALB --> SVC

Those two 10.0.x.x addresses are only routable inside the private network. Once a packet leaves the laptop, there is no route into the VPC, so curl just times out.

The instinct is to check whether the Security Group allows 443. But a Security Group only evaluates a packet once it reaches the target, and here the packet never even enters the VPC.

No rule is misconfigured; there is simply no path from outside to inside. To test from a laptop, you first need some channel that reaches the private subnet.

2. Five options

There are five common approaches. They differ in how they open an entry point, what you have to maintain, and how broad the access they grant is.

Option A: Make the entry point internet-facing

The most direct fix is to switch the test environment’s ALB to internet-facing and restrict source IPs with a Security Group.

flowchart LR
    DEV[Developer laptop] -->|443 inbound| PALB[internet-facing ALB<br>public IP + SG allowlist]
    PALB --> SVC[Private-subnet service]

It needs no extra tooling, and the laptop really can reach the service. But a closed environment now has a permanent public entry point, and even with an allowlist, someone has to maintain it and get it through security review. Our environment doesn’t allow any public entry point, so this option is out.

Option B: A bastion host with an SSH tunnel

If you can’t expose the ALB, the next common approach is to put a jump host in between. The laptop connects to the bastion over SSH and uses ssh -L to bring the remote service’s port back to the local machine.

flowchart LR
    DEV[Developer laptop] -->|SSH 22 inbound| BAS[Bastion EC2<br>public IP]
    BAS -->|forwards traffic| SVC[Private-subnet service]

It’s mature and easy to understand, but you now have another machine to patch and monitor, a set of SSH keys to issue, store, and revoke, and a long-lived target on the internet with port 22 open. That’s too heavy for occasionally testing one API from a laptop.

Option C: Client VPN

Client VPN solves the problem more completely. Once connected, the laptop behaves as if it were inside the VPC and can reach any internal resource it’s authorized for.

flowchart LR
    DEV[Developer laptop] -->|VPN tunnel| VPNEP[Client VPN Endpoint]
    VPNEP -->|laptop treated as inside VPC| SVC[Private-subnet service]

The cost is setting up an endpoint, certificates, and routes, and access is usually granted to a whole network range. I only wanted to reach one port on one container, so it’s far more than I need.

Option D: NAT Gateway, which solves the opposite direction

NAT Gateway comes up a lot in these discussions, but the arrows show that it solves a different problem.

flowchart LR
    SVC[Private-subnet service] -->|inside to outside: OK| NAT[NAT Gateway]
    NAT --> NET[Internet]
    NET -.->|outside to inside: still blocked| SVC

A NAT Gateway lets services in a private subnet initiate connections outward, for example to pull images, call AWS APIs, or ship logs. It doesn’t let a laptop on the internet connect back in.

It does point at something useful, though:

Being able to get out doesn’t mean being able to get in. Private networks usually allow outbound traffic while tightly restricting inbound.

Options A, B, and C all open a path from outside to inside, while D only provides one from inside to outside. So flip the idea around: if both the laptop and the container connect outward and a third party joins the two connections, you never have to open inbound.

Option E: SSM Port Forwarding

AWS Systems Manager (SSM) is AWS’s service for managing machines. Its Session Manager feature lets you connect from a laptop into a host or a container by relaying through AWS, as long as the target is running the SSM Agent, a small program from AWS that talks to SSM. Port Forwarding is one type of connection Session Manager can open: it links a port on your laptop to a port on the target.

In our case, localhost:8888 on the laptop connects to localhost:8000 inside the container, where uvicorn listens. A local program only has to call localhost:8888, and the request travels through the tunnel into the container.

The tunnel doesn’t give the laptop a direct path into the VPC. Instead, the session-manager-plugin on the laptop and the SSM Agent in the container each dial out to the SSM service, and SSM joins the two.

flowchart LR
    subgraph LT[Developer laptop]
        PLG[session-manager-plugin<br>listening on localhost:8888]
    end
    subgraph AWSX[AWS]
        SSM[SSM Service<br>relay]
    end
    subgraph VPC[ECS task in private subnet]
        AG[SSM Agent]
        APP[Service :8000]
    end
    PLG -->|① outbound TLS| SSM
    AG -->|② outbound TLS| SSM
    AG -->|③ connects to its own 8000| APP

That’s why it needs no public IP and no new inbound rule, and why the channel disappears as soon as the session ends.

The five options side by side

3. Why SSM Port Forwarding?

All I needed was temporary access to one port on one container, without opening inbound, and SSM Port Forwarding is exactly that size. If you need long-term access to a whole internal network, choose Client VPN; if you’re serving production traffic, an internet-facing ALB is the proper entry point.

Under these constraints, it has four advantages:

1. It doesn’t change the network’s entry points: everything is outbound, and the VPC gets no public path.
2. There’s no host to run: no bastion to maintain for the tunnel, and no SSH keys to manage.
3. Access follows identity: IAM decides who can open a tunnel, and every StartSession leaves a CloudTrail event.
4. It’s gone when you’re done: once the session closes, the local port no longer leads to the container.

It has costs too:

• Traffic goes through the SSM relay, so it’s not suited to load testing or moving large amounts of data.
• Sessions don’t persist: they disconnect after sitting idle for a while, and you have to reopen them.
• The tunnel depends on the Agent in the container. While a task is being replaced, or if the Agent is unhealthy, the connection may fail.
• Traffic goes straight into the container without passing through the ALB. It can validate the application, but not the production entry path (see section 7).

For ad hoc testing and debugging, these limits are acceptable.

4. How can it connect without inbound access?

You can think of SSM Port Forwarding as “two outgoing phone calls and one switchboard.”

Two people who can’t call each other directly both call the same switchboard. The switchboard never has to call either of them; it just patches the two calls together, and the two can talk.

sequenceDiagram
    participant CLI as Laptop: aws cli + plugin
    participant SSM as SSM Service
    participant AG as SSM Agent in container
    participant APP as Service :8000
    Note over AG,SSM: When the task starts, the Agent has already dialed out<br>and opened the control channel (TLS 443)
    CLI->>SSM: StartSession API (HTTPS, IAM-signed)
    SSM-->>CLI: session ID + WebSocket URL + one-time token
    CLI->>SSM: Dials out to open WebSocket data channel
    SSM->>AG: Notifies new session over control channel
    AG->>SSM: Dials out to open its own WebSocket data channel
    Note over CLI,AG: SSM relays application-layer bytes<br>between the two WebSockets
    CLI->>SSM: HTTP bytes from curl
    SSM->>AG: Relays
    AG->>APP: Opens local TCP connection and writes to 8000
    APP-->>CLI: Response returns the same way

The whole process has four stages:

1. The Agent is already on the line. When the task starts, the SSM Agent opens a long-lived TLS 443 connection to SSM, called the control channel. It only receives notices such as “a new session is starting” and carries no actual data.
2. The operator requests a session. A session is one connection from the laptop to the container, from the moment it opens until it closes. The operator runs aws ssm start-session to call the StartSession API and names an SSM document. A document is a configuration AWS has already written that tells the Agent what this session should do, such as open a shell or forward a port. Once IAM approves the request, the CLI receives a WebSocket URL and a one-time token. WebSocket is a long-lived, two-way connection built on HTTPS (see appendix A2), and the token is only valid for this session.
3. Both ends open data channels. The plugin and the Agent each dial out to SSM and open a WebSocket, called a data channel, which carries only this session’s data. To a firewall, both sides look like ordinary outbound 443 connections.
4. SSM starts relaying. When the plugin receives local data, it sends it through SSM to the Agent. The Agent then opens a local TCP connection and writes the data to port 8000 in the container.

The control channel lives as long as the Agent does, while each data channel is created and torn down with its session. That’s why the Agent has to reach RUNNING first, meaning the control channel is up, before an operator can open a session.

SSM moves application-layer bytes, and each leg of the path has its own TCP connection, so the application’s access log usually shows 127.0.0.1 as the source rather than the laptop’s IP (see appendix A3).

And because it only moves bytes, HTTP, SSE, and any other protocol that runs over TCP can use this tunnel.

5. Prerequisites

For Port Forwarding to connect, the container needs an SSM Agent, but ECS doesn’t put one there by default, so you rely on a feature called ECS Exec. ECS Exec was built to let a person run commands inside a container from their laptop (aws ecs execute-command). When it’s enabled, ECS places the SSM Agent into the container at task startup, and Port Forwarding borrows that same Agent.

There are five things to set up: the ECS service, the task role, the container’s outbound path, the operator’s IAM identity, and the local plugin. The task role and the operator’s IAM identity are permission settings. Even though nothing opens inbound, that doesn’t mean anyone can open a tunnel, because the checkpoint has moved from the network to IAM. The task role decides whether the container can accept a session, and the operator’s permissions decide who can open one.

Enable ECS Exec

The ECS service must have enableExecuteCommand turned on. When a task starts, ECS mounts the SSM Agent binary into the container and reports its status under the name ExecuteCommandAgent (ECS calls platform-managed programs like this “managed agents”).

The setting only applies to newly started tasks, so after you enable enableExecuteCommand you have to redeploy; existing tasks won’t get an Agent.

Let the task role open control and data channels

The task role is the IAM identity a container uses while it runs, and any program inside the container uses it to call AWS APIs. The Agent runs inside the container too, so the permissions go on the task role, not the execution role (the identity ECS uses before the task starts, to pull images and write logs; see appendix A5 for a comparison). The task role needs these four actions:

1
2
3
4
5
6
7
8
9
10

{
  "Effect": "Allow",
  "Action": [
    "ssmmessages:CreateControlChannel",
    "ssmmessages:CreateDataChannel",
    "ssmmessages:OpenControlChannel",
    "ssmmessages:OpenDataChannel"
  ],
  "Resource": "*"
}

The four actions map to the two kinds of channel from section 4, and each channel takes two steps, register and connect:

Without these permissions, the Agent can’t reach SSM, and ExecuteCommandAgent never reaches RUNNING.

Let the container reach SSM

The task role grants permission, but the connection also needs a network path: the Agent must be able to reach SSM Messages, the part of SSM that handles control and data channels.

The container can reach AWS services through a NAT Gateway, or you can create a com.amazonaws.<region>.ssmmessages endpoint in the VPC.

A VPC endpoint provides a private entry point inside the VPC, so the container can reach SSM Messages without going over the public internet. For ECS Exec, you only need the ssmmessages endpoint (plus kms if you encrypt with your own KMS key). The ssm, ec2messages, and other endpoints that the Systems Manager docs list for EC2 aren’t needed for ECS.

Allow the operator to start sessions

The operator is the person opening the tunnel, and their IAM identity needs ssm:StartSession. AWS’s example policy also lets operators manage their own sessions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

[
  {
    "Effect": "Allow",
    "Action": "ssm:StartSession",
    "Resource": [
      "arn:aws:ecs:<region>:<account>:task/<cluster>/*",
      "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSessionToRemoteHost"
    ]
  },
  {
    "Effect": "Allow",
    "Action": [
      "ssm:TerminateSession",
      "ssm:ResumeSession",
      "ssmmessages:OpenDataChannel"
    ],
    "Resource": "arn:aws:ssm:*:*:session/${aws:userid}-*"
  }
]

The StartSession resource lists two things: the ECS task ARN limits which tasks the operator can connect to, and the document ARN limits them to Port Forwarding. Without permission, the API returns AccessDenied immediately, so there’s no WebSocket URL or token, and no data channel can be opened.

The policy lists a document ARN because the document determines what a session can do. A document is a configuration AWS has already written that tells the Agent whether this session should open a shell or forward a port. If the policy doesn’t restrict the document, an operator who runs aws ssm start-session without naming one gets the default, SSM-SessionManagerRunShell, which opens a shell straight into the container. That shell isn’t recorded, because the logging feature that saves commands and output to CloudWatch Logs or S3 is only wired into aws ecs execute-command, so afterward nobody can tell what the operator did inside the container. For that reason, AWS’s ECS Exec documentation recommends denying ssm:StartSession on ECS tasks outright, and adds that these sessions also count against the session quota.

This post restricts the document instead of denying the action entirely. That way, operators can open Port Forwarding but not a shell, and Port Forwarding carries HTTP requests, so there were never any commands to record. CloudTrail, the AWS service that records API calls, still logs the operator, time, source IP, target, and document for every StartSession, ResumeSession, and TerminateSession; appendix A4 lists which fields to look at.

Install the local plugin

Finally, install the session-manager-plugin locally. The AWS CLI calls the StartSession API, but it’s this plugin that actually listens on the local port and forwards the data.

If you’re not sure which of these conditions is missing, AWS’s amazon-ecs-exec-checker can check your ECS Exec setup.

6. Opening the tunnel

The command that opens the tunnel is aws ssm start-session, and it needs two pieces of information: which container to connect to (the target), and what this session should do (the document).

Start with the target. SSM identifies an ECS container with a string in a fixed format:

1

ecs:<cluster>_<task-id>_<container-runtime-id>

The string has three parts: the cluster the container is in, the ID of its task, and the container’s own runtime ID. It has to go down to the container level because one task can run several containers at once, for example a sidecar (a helper container) that collects logs next to the main application. You already know the cluster name, and you can find the task ID with aws ecs list-tasks; for the runtime ID, you name the container and look it up with aws ecs describe-tasks.

Next, the document. For ECS targets, the AWS docs explicitly list AWS-StartPortForwardingSessionToRemoteHost, so that’s what we use here. This document takes three parameters: host is the machine the Agent connects to, and localhost means the container itself; portNumber is the port on that machine, and since our image runs uvicorn on 8000 by default, it’s 8000; localPortNumber is the port the plugin listens on locally, which here is 8888.

The commands below put the two together: they find a running task, confirm the Agent has started, look up the runtime ID to build the target, and finally open the session, mapping the container’s 8000 to the laptop’s 8888 (the cluster, service, and container names are illustrative):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

CLUSTER=my-cluster
SERVICE=my-service
CONTAINER=app

# 1. Find a RUNNING task
TASK_ARN=$(aws ecs list-tasks --cluster "$CLUSTER" --service-name "$SERVICE" \
  --desired-status RUNNING --query 'taskArns[0]' --output text)
TASK_ID=${TASK_ARN##*/}

# 2. Confirm the target container's ExecuteCommandAgent is running
aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$TASK_ARN" \
  --query "tasks[0].containers[?name=='$CONTAINER'].managedAgents[?name=='ExecuteCommandAgent'].lastStatus" \
  --output text
# → RUNNING

# 3. Get the target container's runtime ID
RUNTIME_ID=$(aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$TASK_ARN" \
  --query "tasks[0].containers[?name=='$CONTAINER'].runtimeId" --output text)

# 4. Start the session; this process must keep running
aws ssm start-session \
  --target "ecs:${CLUSTER}_${TASK_ID}_${RUNTIME_ID}" \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["localhost"],"portNumber":["8000"],"localPortNumber":["8888"]}'

If it works, this terminal stays in the foreground. As long as the process keeps running, another terminal can send requests through the local port:

1

curl -i http://localhost:8888/health

The two localhosts here refer to different machines. localhost:8888 is on the laptop, where the plugin listens; host=localhost in the document is the container itself, where the application listens on 8000.

Because this path goes straight into the container and skips the ALB’s TLS termination (the ALB is what normally handles HTTPS encryption and decryption), the test URL is usually http://, not https://.

Common snags

• The Agent status is empty. The most common cause is enabling enableExecuteCommand without redeploying the task, since only new tasks start the managed agent. If it’s still empty after a redeploy, check versions: Fargate needs platform version 1.4.0 or later, and the EC2 launch type needs ECS agent 1.50.2 or later.
• No runtime ID comes back. If the task has a sidecar, CONTAINER must point to the right container. The container name isn’t necessarily the same as the ECS service name.

RemoteHost can reach more than localhost

The same approach works for hosts outside the container. An RDS database (AWS’s managed database) in a private subnet is just as unreachable from the laptop, but the container can reach it. The host in AWS-StartPortForwardingSessionToRemoteHost doesn’t have to be localhost; set it to the RDS address and the Agent connects to RDS instead, turning the same ECS task into a relay to the database.

The other document, AWS-StartPortForwardingSession, ends the tunnel at the SSM target itself. In practice it also works directly against ECS targets, but that combination isn’t documented in the official ECS docs:

1
2
3
4

aws ssm start-session \
  --target "ecs:${CLUSTER}_${TASK_ID}_${RUNTIME_ID}" \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["8000"],"localPortNumber":["8888"]}'

7. A working tunnel doesn’t mean the production path works

A tunnel test only covers the layers the traffic actually passes through.

SSM Port Forwarding sends traffic straight into the container, so it can validate application logic, API responses, and the container port. Production requests also go through DNS, TLS, the ALB listener, and the target group, and none of those layers are on the tunnel’s path.

These problems can still show up on the production path even when tunnel tests pass completely:

• An expired certificate or a mismatched domain makes the TLS handshake fail.
• The security policy rejects a particular TLS version or cipher.
• A listener rule is misconfigured, or the application handles X-Forwarded-* headers differently.
• The ALB idle timeout cuts off streaming connections that go quiet for too long.
• A target group health check marks a healthy service as unhealthy, and production requests get 502s.
• Load-balancing behavior and ALB access logs on the production path go unverified.

For example, SSE can keep streaming through the tunnel but get cut off by the idle timeout when it goes through the ALB.

8. The same shape elsewhere: reverse tunnels

This approach isn’t unique to AWS. It’s generally called a reverse tunnel, or more broadly the broker pattern.

What these have in common is that an internal agent connects out to a relay service first. External commands or data then flow back in over that existing connection, so no internal endpoint ever has to accept inbound traffic.

Other examples with the same shape:

• A GitHub Actions self-hosted runner pulls jobs from GitHub instead of letting GitHub connect into the build environment.
• Cloudflare Tunnel and ngrok run a local agent that connects out to a relay, which then sends public traffic back to the local machine.
• AWS IoT devices behind a home NAT open a long-lived MQTT connection and stay attached to a broker.
• Remote desktop software has two computers, each behind NAT, connect out to a relay.
• Tailscale uses DERP to help establish connections, and keeps relaying through it when a direct connection isn’t possible.

SSM itself offers several features over the same pipeline. aws ecs execute-command carries terminal input and output (that is, a shell); aws ssm start-session --target i-xxxx opens a session to an EC2 instance; and send-command runs commands on machines that SSM manages.

Appendix

A1. How does a timeout differ from connection refused?

• A timeout usually means the packet got no response. The route may not exist, a firewall may have dropped the packet, or the destination may be unreachable.
• connection refused means the packet reached the host, but nothing is listening on the target port, so the operating system sent back an RST.

For example, while the tunnel is open, the only thing listening on the laptop is localhost:8888. If you call the laptop’s localhost:8000 instead, and nothing local uses that port, you’ll usually get connection refused.

A2. What is the WebSocket “upgrade”?

A WebSocket starts as an ordinary HTTPS request with an Upgrade: websocket header. Once the server replies 101 Switching Protocols, the same TCP connection switches from HTTP’s request-response pattern to a persistent two-way channel.

Because the connection starts out as standard TLS 443 traffic, a firewall still sees it as an ordinary outbound HTTPS connection.

A3. Bytes are relayed, not connections

Think of the application data as a letter and the TCP header as its envelope.

• The letter is the HTTP bytes curl sends, and those contents are relayed all the way to the application.
• An envelope is only valid for one leg of the trip. curl ↔ plugin, plugin ↔ SSM, Agent ↔ SSM, and Agent ↔ application are all separate connections.
• At the end of each leg, the old envelope is thrown away, and the letter gets a new one for the next leg.

So the application sees a new local TCP connection opened by the Agent, and the source is usually 127.0.0.1. If the application only listens on IPv6, it may see ::1 instead.

A4. Which CloudTrail fields should you check?

In CloudTrail Event history, filter on Event name = StartSession. The most useful fields in each event are:

• userIdentity.arn: which IAM identity started the session.
• eventTime and sourceIPAddress: when it started and from which IP.
• requestParameters.target: the connection target, in the format ecs:<cluster>_<task-id>_<runtime-id>.
• requestParameters.documentName: whether it used Port Forwarding, RemoteHost, or another document.

You can also look up TerminateSession and ResumeSession to see when, and by whom, a session was closed or resumed.

A5. Task role vs. execution role

• taskRoleArn is the identity the container uses while it runs. Both the application and the SSM Agent use this role when they call AWS APIs.
• executionRoleArn is the identity the ECS platform uses before the task starts, for example to pull images, create log streams, or read values from Secrets Manager and inject them into the container.

Think of the execution role as the landlord’s permissions for setting up the room, and the task role as the tenant’s identity after moving in. The SSM Agent runs inside the container, so it uses the task role.