My Container Group Can’t Pull My Image
If your container group is unable to pull your container image, try the following steps:- Verify the image url is correct. Check for typos in the image name and registry.
- If using a private registry, ensure that the credentials provided are valid and have the necessary permissions, including inspecting the manifest and downloading all of the layers. Ensure there are no leading or training whitespace characters in your credentials.
- If using a private registry, ensure your account level supports an adequate rate limit for the number of container groups you are deploying. This is especially important if you are using Docker Hub, and deploying many single-replica container groups.
- Ensure the image is in Docker/OCI format and built for AMD64 architecture. Salad does not support ARM64 images.
- Ensure the image size is <= 35GB (compressed). Larger images will fail to pull.
- If using a custom registry, ensure that the registry is publicly accessible from the internet.
My Container Group Loops Between Downloading and Allocating
If you see your container group repeatedly transitioning between “Downloading” and “Allocating”, this almost always indicates that the container is failing immediately when it starts. In this case, Salad will attempt to restart the container on the same machine to minimize downtime, but after repeated failures will reallocate the workload to a different node. To confirm this is what is happening, you can check the System Events tab in the portal, which will show things like when an instance started downloading, and exit codes.
- If you see repeated exit codes of
0
, this usually indicates that the container is exiting immediately after startup. Some base images (likeubuntu
) will do this by default. To keep them running, you can add a command likesleep infinity
to your container configuration. Containers, unlike VMs, require a long-running process to stay alive. - If you see exit codes like
137
, this usually indicates that the container is running out of memory. Try increasing the memory allocation for your container group. - If you see other exit codes, these are usually application-specific. Check your application logs to determine the
cause of the failure.
- If your application is exiting non-0 with no logs being captured, this usually indicates that the application is
failing before it can write to stdout or stderr. To diagnose, try running the container locally to see if it fails
immediately, and if so, why. If it works locally, but not on SaladCloud, try changing the
command
field in your container group configuration tosleep infinity
, which will keep the container up. While it is up, you can use the web terminal to try manually running the application command to see why it is failing.
- If your application is exiting non-0 with no logs being captured, this usually indicates that the application is
failing before it can write to stdout or stderr. To diagnose, try running the container locally to see if it fails
immediately, and if so, why. If it works locally, but not on SaladCloud, try changing the
My Container Group Is Stuck Downloading / Stuck at Downloading 99%
If a node has been in the “Downloading” state for a long time (more than 2 minutes per GB of image size), you may have landed on a node with below-average network performance. This can happen because Salad nodes often run on residential internet connections, where other household devices may share the bandwidth. Many of these nodes will eventually become healthy, but if high network performance is essential to your application, try reallocating the node. It will be placed on a different machine, with different network conditions. Note that the progress bar is a conservative estimate — actual download time may vary.My Container Group Is Running But I Get 503 Errors With The Container Gateway
- Ensure your container is listening on the correct port. The port you specify in the Salad configuration must match the port your application is listening on inside the container.
- Ensure your container supports IPv6. Salad
Gateway uses IPv6 to route traffic to your container. Enabling IPv6 is typically as easy as setting your host to
::
instead of0.0.0.0
. - Ensure your readiness probe does not pass until the server is able to accept traffic. If the readiness probe passes before the server is ready, Salad Gateway will route traffic to the container, but the container may not be able to handle it yet, resulting in 503 errors.
- Ensure at least one node is running and “ready” in the Salad portal. If all nodes are down, or not ready, the gateway will return 503 errors.
My Container Group Is Running But I Get 403 Errors With The Container Gateway
- If your container group has Authentication enabled, ensure that your Salad API Key is valid, and included in the
Salad-Api-Key
header of your requests. - If your container group does not have Authentication enabled but you are getting 403 errors, this issue can occur in rare cases due to an internal delay creating the necessary route tables. This usually resolves itself within a few minutes. If it does not, please contact support. In this scenario, duplicating the container group will almost always resolve the issue, but please contact support anyway, so we can track the issue and continue to improve.
My Container Group Is Running But I Can’t Access It With The Web Terminal
- Wait a few seconds after the terminal page loads, as it sometimes takes a moment to establish a connection.
- Refresh the page if it does not connect after a few seconds.
- You may also gain terminal access to nodes using tools like tailscale ssh, or even installing the VS Code server or JupyterLab in your container image.