​
Introduction

AI developers using macOS or Windows can seamlessly connect to over 30 types of Salad GPU nodes via VS Code, enabling real-time testing and debugging in live environments — before moving to production. This gives them full access to GPU resources, environment variables, IMDS endpoints, and more. Developers can validate VRAM usage, network behavior, and external resource access; fine-tune application parameters such as local queue lengths, batch sizes, and thread counts; and catch rare, real-world edge cases that static testing often misses. There are several ways to enable VS Code remote development on SaladCloud. Tailscale can provide secure, direct network access between your developer laptop and Salad nodes, supporting both TCP and UDP-based applications. Using the Remote - SSH extension in VS Code Desktop, you can connect to a Salad node over its Tailscale IP, which launches a VS Code Server on the node. This allows you to easily open remote folders or clone GitHub repositories to begin development and testing directly on the node. This solution is simple to deploy, scales greatly, and is highly recommended. For more details on integrating Tailscale with workloads on SaladCloud, please refer to this guide. You can also leverage VS Code Remote - Tunnels, powered by Microsoft Dev Tunnels, which supports two integration modes with SaladCloud — automatic and interactive — each tailored to different use cases. Both modes are covered in this guide. Notes: Since Salad nodes are interruptible and may shut down unexpectedly, it’s crucial to commit your changes regularly to your repositories via VS Code or back them up to the cloud using the appropriate SDKs or APIs.

​
Solution Comparison and Use Cases

Remote - SSHAutomatic Mode
Remote - Tunnels		Interactive Mode
Remote - Tunnels
Container Image ModificationsJoin the Salad node to the tailnet with a generated auth key.Launch the VS Code Server with an existing tunnel (tunnel ID and access token).Non-intrusive
Supported Clients and ApplicationsYour developer laptop must be added to the tailnet, supporting all TCP and UDP-based applications.VS Code Desktop and a browser from anywhere over the Internet.VS Code Desktop and a browser from anywhere over the Internet.
Cost-Effectiveness and ScalabilityNo additional cost.
A free Tailscale personal account supports up to 100 devices, with no limits on speed or data transfer.		No additional cost.
Each account (Github or Microsoft) supports up to 10 active tunnels for simultaneous access to 10 Salad nodes, with each tunnel offering speeds up to 20 MB/s and a total monthly data transfer limit of 5 GB per account.		No additional cost.
Each account (Github or Microsoft) supports up to 10 active tunnels for simultaneous access to 10 Salad nodes, with each tunnel offering speeds up to 20 MB/s and a total monthly data transfer limit of 5 GB per account.
User Experience (after node reallocation or container group restarts)Retrieve the Tailscale IP of the selected instance, then connect to it.Refresh clients to reconnect automatically.Open the terminal of the selected instance in the SaladCloud Portal and run scripts manually. And then connect to it.
Suggested Use CaseNeed access to all Salad nodes of a workload for testing or debugging with VS Code.Develop applications over Salad GPU nodes with VS Code for a while.
This setup only supports single-replica container groups on SaladCloud.		Occasionally access any Salad node for testing or debugging with VS Code.

​
Microsoft Dev Tunnels

Dev Tunnels enable secure, authenticated connections over the Internet, allowing developers to expose local services — such as applications running on Salad nodes — to remote clients, including browsers and custom applications. Both clients and servers can remain behind firewalls and NATs; they only need to initiate outbound connections to Azure, which handles secure relaying, identity and management services. Each server requires a dedicated tunnel, which can be accessed by multiple clients. While tunnels are primarily for TCP connections, UDP-based protocols are also supported by Dev Tunnels. Currently in public preview and free to use, Dev Tunnels require either a GitHub or Microsoft account for access. Each account supports up to 10 tunnels, with each tunnel offering speeds of up to 20 MB/s and a total monthly data transfer limit of 5.0 GB per account. Please check the link for more details. Below is an illustration of how applications leverage Dev Tunnels on SaladCloud: Follow this link to install the Dev Tunnels CLI on any machine and familiarize yourself with its features.
Dev Tunnels CLI
# Sign in interactively using a Github account and a device code
ubuntu@wsl-nuc:~$ devtunnel user login --github --use-device-code-auth
Browse to https://github.com/login/device and enter the code: EAD8-CC93
Logged in as xxxxxx using GitHub.

# Open a browser, navigate to the provided URL, enter the code, authorize access, and then return to continue.

# Show the current usage and limits in this account
ubuntu@wsl-nuc:~$ devtunnel limits
Name                 Current            Limit              Resets After             Description
BandwidthPerUser     9.8 MB (0%)        5.0 GB             2025-05-16 5:21 PM       User Bandwidth Consumption
More information on limits: https://aka.ms/devtunnels/limits

​
VS Code Remote - Tunnels

VS Code Remote - Tunnels is built on Microsoft Dev Tunnels to simplify remote development. It allows developers to connect to remote VS Code Servers (such as those running on SaladCloud) using the Remote - Tunnels extension in VS Code Desktop or a browser, without requiring direct network access between the local and remote environments. To enable access, the remote environment must launch the VS Code Server in tunnel mode — either by creating a new tunnel or using an existing one to expose it. This can be done by running the VS Code Server CLI (ideal for headless environments like Salad nodes) or by enabling the Remote Tunnel Access from an existing VS Code Desktop installation. Once the tunnel is active, developers can connect to the remote VS Code Server through it. Similar to Dev Tunnels, this solution requires a GitHub or Microsoft account, and all Dev Tunnels specifications and limitations still apply. For instance, each Salad node must have its own dedicated tunnel, but multiple clients — using either VS Code Desktop or a browser — can connect to the same node through that tunnel. There are two ways to use this solution on SaladCloud:
  • Automatic Mode – This mode supports only single-replica container group deployments on SaladCloud, and the container image must be modified to launch the VS Code Server using an existing tunnel. You can manually create a tunnel and generate an access token using the Dev Tunnels CLI, which requires interactive browser-based authentication and must be done outside of SaladCloud. Both the tunnel ID and access token can then be passed as environment variables to a single-replica container group on SaladCloud. Once the container starts — or after a node reallocation — the VS Code Server automatically starts and connects using the provided tunnel and token, without requiring any manual intervention at runtime.
  • Interactive Mode – Non-intrusive, requiring no changes to the container image. You can access any running Salad node through the terminal in the SaladCloud Portal, then manually install and launch the VS Code Server in tunnel mode. This process requires interactive browser-based authentication, which can be completed on your developer laptop.

​
Automatic Mode

​
Prepare Your Container Image

Here is an example application that demonstrates how to enable the automatic mode for container images running on SaladCloud. The Dockerfile creates a containerized environment by using a base image, then installing essential utilities and required dependencies including the VS Code Server CLI. It also copies the required Python code and startup script into the image.
Dockerfile
# Base image selection
FROM docker.io/python:3.12.3
#FROM docker.io/pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime

# Install essential utilities
RUN apt-get update && apt-get install -y curl net-tools iputils-ping

# Upgrade pip and install Python packages
RUN pip install --upgrade pip
RUN pip install flask python-dotenv

# Install VS Code Server CLI
RUN curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' -o vscode_cli.tar.gz && \
    tar -xf vscode_cli.tar.gz && \
    mv code /usr/local/bin/code && \
    rm vscode_cli.tar.gz

# Set working directory and copy application files
WORKDIR /app
COPY hello.py /app
COPY start.sh /app
RUN chmod +x /app/start.sh

# Default startup command
CMD ["./start.sh"]
# the pre-built image: docker.io/saladtechnologies/vscode-remote-tunnels:0.0.1-vscode-remote-tunnels:0.0.1-automatic
When the image runs on a Salad node, the startup.sh script launches the VS Code Server using an existing tunnel.
start.sh
#!/bin/bash

echo -e "\nLaunch the VS Code Server using an existing tunnel..."
# SALAD_CONTAINER_GROUP_ID as the tunnel name
code tunnel --accept-server-license-terms --name $SALAD_CONTAINER_GROUP_ID --tunnel-id $TUNNEL_ID --host-token $ACCESS_TOKEN
# SALAD_MACHINE_ID as the tunnel name
#code tunnel --accept-server-license-terms --name $SALAD_MACHINE_ID --tunnel-id $TUNNEL_ID --host-token $ACCESS_TOKEN

# You may also launch the VS Code Server in the background, then run your original command — such as python hello.py
# code tunnel --accept-server-license-terms --name $SALAD_CONTAINER_GROUP_ID --tunnel-id $TUNNEL_ID --host-token $ACCESS_TOKEN &
# python hello.py

# The containers running on SaladCloud must have a continuously running process; and if the process completes, SaladCloud will automatically reallocate the instances to rerun the image.
# Sometimes, you can keep the container running with a simple command like this:
# sleep infinity
The --name parameter assigns a user-friendly name for the tunnel (uniquely associated with the Salad node), which is used when connecting from a browser: 
https://vscode.dev/tunnel/name_of_tunnel
SALAD_MACHINE_ID is a unique identifier for each Salad node and is easily viewable in the SaladCloud Portal. SALAD_CONTAINER_GROUP_ID is available through the SaladCloud APIs or SDKs, and can also be retrieved by running env inside the terminal of a container instance or checking the container logs in the SaladCloud Portal. Currently, all instances (or replicas) within a container group share the same hostname, which is set to the value of SALAD_CONTAINER_GROUP_ID. You can use either SALAD_MACHINE_ID or SALAD_CONTAINER_GROUP_ID as the value for the --name parameter.
  • With SALAD_MACHINE_ID, you can easily connect to a specific node from clients, as the ID is readily accessible from the SaladCloud Portal.
  • Using SALAD_CONTAINER_GROUP_ID (as demonstrated in this example) helps avoid client-side reconfiguration — such as refreshing clients to reconnect automatically — after node reallocation or container group restarts, since the value remains consistent across instances launched at different time within the same container group.
Note: Each node requires its own dedicated tunnel, so this image must be run using a single-replica container group with a unique tunnel. If multiple instances share the same tunnel, they will conflict with each other — resulting in unstable behavior or failure to connect.

​
Create a Tunnel and Generate Access Tokens

Let’s create a tunnel with an access token using the Dev Tunnels CLI:
Dev Tunnels CLI
# Create a tunnel
ubuntu@wsl-nuc:~$ devtunnel create
Tunnel ID             : joyful-field-ct6cnqj.usw3
Description           :
Labels                :
Access control        : {}
Host connections      : 0
Client connections    : 0
Current upload rate   : 0 MB/s (limit: 20 MB/s)
Current download rate : 0 MB/s (limit: 20 MB/s)
Tunnel Expiration     : 30 days

ubuntu@wsl-nuc:~$ devtunnel list
Found 1 tunnel.
Tunnel ID                           Host Connections     Labels                    Ports                Expiration                Description
joyful-field-ct6cnqj.usw3           0                                              0                    30 days

# Generate an access token for the tunnel
ubuntu@wsl-nuc:~$ devtunnel token joyful-field-ct6cnqj.usw3 --scope manage --scope host
Token tunnel ID: joyful-field-ct6cnqj.usw3
Token scopes: manage host
Token lifetime: 1.00:00:01
Token expiration: 2025-04-18 22:02:07 UTC
Token: eyJhbGciOiJFUzI1NiIsImtpZCI6IkM3NDYxNEM5OTE0NjUwNzI2REI1RUZBM0M1OTBDQzdGNjJFOUI4QzQiLCJ0eXAiOiJKV1QifQ.eyJjbHVzdGVySWQiOiJ1c3czIiwidHVubmVsSWQiOiJqb3lmdWwtZmllbGQtY3Q2Y25xaiIsInNjcCI6Im1hbmFnZSBob3N0IiwiZXhwIjoxNzQ1MDEzNzI3LCJpc3MiOiJodHRwczovL3R1bm5lbHMuYXBpLnZpc3VhbHN0dWRpby5jb20vIiwibmJmIjoxNzQ0OTI2NDI3fQ.z-tv08Co7DBZvBjBtSXRExx5hef73vSApsgp3DQBrdDRTGvm4-yYBqanL-XCCEwcOjfE3v5acTfhCnexnPHXrg
The default tunnel expiration time is 30 days, while the token expiration time is limited to 24 hours. Once a token expires, it can no longer be used to authenticate a new VS Code Server session. However, any existing session established before the expiration will typically remain valid.

​
Deploy the Container Group

Now your can create a container group with the following parameters using the SaladCloud Portal. For deployment using SaladCloud APIs/SDKs, please refer to this link. 
Image Source: docker.io/saladtechnologies/vscode-remote-tunnels:0.0.1-vscode-remote-tunnels:0.0.1-automatic
Resource: 4 vCPUs, 4GB Memory
Replica Count: Must be 1 (single-replica)
Environment Variable: TUNNEL_ID, ACCESS_TOKEN

​
Connect to the Salad Node

Once the instance is running, you can retrieve the SALAD_CONTAINER_GROUP_ID and connect to it using both VS Code Desktop and a browser. 
https://vscode.dev/tunnel/SALAD_CONTAINER_GROUP_ID
Start the Python Flask server in the browser: Use curl to test the application locally in the VS Code Desktop:

​
Perform a Check

Let’s check the tunnel using the Dev Tunnels CLI:
Dev Tunnels CLI
ubuntu@wsl-nuc:~$ devtunnel list
Found 1 tunnel.

Tunnel ID                           Host Connections     Labels                    Ports                Expiration                Description
joyful-field-ct6cnqj.usw3           1                    3cd9c60e-0a98-4b96-8...   2                    30 days

# 1 host connection and 2 client connections
# The --name paramater (SALAD_CONTAINER_GROUP_ID) is saved as a label
ubuntu@wsl-nuc:~$ devtunnel show joyful-field-ct6cnqj.usw3
Tunnel ID             : joyful-field-ct6cnqj.usw3
Description           :
Labels                : 3cd9c60e-0a98-4b96-8147-ecc2261cb45b,protocolv4,vscode-server-launcher,_flag4
Access control        : {}
Host connections      : 1
Client connections    : 2
Current upload rate   : 148 bytes/s (limit: 20 MB/s)
Current download rate : 132 bytes/s (limit: 20 MB/s)
Upload total          : 5458 KB
Download total        : 26 MB
Ports                 : 2
  8888  auto  https://gtnn333j-8888.usw3.devtunnels.ms/
  31545 auto  https://gtnn333j-31545.usw3.devtunnels.ms/
Tunnel Expiration     : 30 days

​
Workflow Summary

The automatic mode may fit scenarios where you’re developing applications across various Salad GPU nodes over an extended period. The following summarizes the workflow:
StepActionPerformed On
0Modify the image to support the automatic mode, and create a 30-day tunnel using the Dev Tunnels API.Developer Laptop
1Generate an access token each day using the Dev Tunnels API.Developer Laptop
2Run the single-replica container group with the tunnel ID and the updated token.
When a node goes offline unexpectedly, SaladCloud automatically allocates a new one.
Stop the container group when it’s not in use.		SaladCloud
3Connect using VS Code Desktop or a browser.
Commit code changes regularly to the repositories.
After node reallocation or container group restart, refresh clients to reconnect automatically.		Developer Laptop

​
Interactive Mode

​
Launch the VS Code Server Interactively

To test or debug code in any running instance, open its terminal in the SaladCloud Portal. From there, you can manually install the VS Code Server CLI — or optionally include it in your container image during the build process — and start the VS Code Server in tunnel mode interactively.
VS Code Server CLI
# Sign in interactively using a Github account and a device code
# Even though the command is part of the VS Code Server CLI, it specifically authenticates with the Dev Tunnels service

root@564eafc5-c872-414b-b4f6-a6032163a31f:~$ code tunnel user login --provider github
To grant access to the server, please log into https://github.com/login/device and use code 8223-8B7E

# Open a browser on your developer laptop, navigate to the provided URL, enter the code, authorize access, and then return to continue.

# Start the VS Code Server in tunnel mode to automatically provision a tunnel, launch the server, and expose it through that tunnel.

root@564eafc5-c872-414b-b4f6-a6032163a31f:~$ nohup code tunnel --accept-server-license-terms --name 001 &> output.log &
[1] 24

# You can now safely close the terminal
# Connect using VS Code Desktop or a browser (https://vscode.dev/tunnel/001)

root@564eafc5-c872-414b-b4f6-a6032163a31f:~$ code tunnel status
{"tunnel":{"name":"001","started_at":"2025-04-16T21:30:06.255541221Z","tunnel":"Connected","last_connected_at":"2025-04-16T21:30:13.282022913Z","last_disconnected_at":null,"last_fail_reason":null},"service_installed":false}

root@564eafc5-c872-414b-b4f6-a6032163a31f:~$ tail -f output.log

[2025-04-16 21:29:55] info Creating tunnel with the name: 001
Open this link in your browser https://vscode.dev/tunnel/001

[2025-04-16 21:30:35] info [tunnels::connections::relay_tunnel_host] Opened new client on channel 2
[2025-04-16 21:30:35] info [russh::server] wrote id
[2025-04-16 21:30:36] info [russh::server] read other id
[2025-04-16 21:30:36] info [russh::server] session is running
[2025-04-16 21:30:42] info [rpc.0] Checking /root/.vscode/cli/servers/Stable-17baf841131aa23349f217ca7c570c76ee87b957/log.txt and /root/.vscode/cli/servers/Stable-17baf841131aa23349f217ca7c570c76ee87b957/pid.txt for a running server...
[2025-04-16 21:30:42] info [rpc.0] Starting server...
[2025-04-16 21:30:42] info [rpc.0] Server started

​
Perform a Check

The VS Code Server CLI interacts with the Dev Tunnels service under the hood for authentication and tunnel provisioning. You can check the tunnels created by the VS Code Server CLI using the Dev Tunnels CLI:
Dev Tunnels CLI
# The --name paramater (001) is saved as a label
ubuntu@wsl-nuc:~$ devtunnel list
Found 1 tunnels.

Tunnel ID                           Host Connections     Labels                    Ports                Expiration                Description
new-hill-7rpsj2g.uks1               1                    001,protocolv4,vscode...  0                    30 days
The VS Code Server CLI doesn’t always provision a new tunnel. If you relaunch the server on the same Salad node after a restart, it typically reuses the existing tunnel, even when a new name is provided. When more than 10 tunnels are created, the older and inactive tunnels would be automatically deleted by Dev Tunnels.

​
Workflow Summary

If you only need occasional access to any Salad node for testing or debugging, this mode is ideal. The following summarizes the workflow:
StepActionPerformed On
1Open the terminal of a running instance in the SaladCloud Portal, and install the VS Code Server CLI.SaladCloud
2Run the VS Code Server CLI to log in to the Dev Tunnels service to receive the URL and an authentication code. CLI Example:
code tunnel user login –provider github		SaladCloud
3Open a browser, log in using the URL, enter the code and authorize access.Developer Laptop
4Run the VS Code Server CLI to launch a VS Code Server in tunnel model, and you can safely close the terminal after this step. CLI Example:
nohup code tunnel —accept-server-license-terms —name 001 &> output.log &
tail -f output.txt		SaladCloud
5Connect using VS Code Desktop or a browser.Developer Laptop