Fixing 'Access Denied' to Private Registry in Hybrid Cloud Docker Builds
Quick Fix Summary
TL;DRRun `docker login <registry-url>` with correct credentials.
Docker cannot authenticate with the specified private container registry due to invalid, expired, or missing credentials, or network/configuration issues preventing the authentication request.
Diagnosis & Causes
Recovery Steps
Step 1: Verify Credentials and Basic Connectivity
Check your current Docker login session and test basic connectivity to the registry.
# Check currently logged-in registries
cat ~/.docker/config.json | grep auth || echo "No auth entries found"
# Test login (will prompt for credentials if needed)
docker login <your-registry-url>Step 2: Inspect and Correct the Docker Configuration
Examine the Docker daemon configuration for registry mirrors or proxies that might be interfering, and ensure credentials are stored correctly.
# Check for registry mirrors in daemon config
sudo cat /etc/docker/daemon.json 2>/dev/null || echo "No daemon.json found"
# List all credential helpers
docker-credential-<helper> list || echo "No helper configured"
# For Amazon ECR, ensure you have the latest auth token
aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.comStep 3: Validate Network and Firewall Rules
Confirm the build host can reach the registry's authentication service (often on a different port/domain than the image pull).
# Test network reachability to common registry auth endpoints
nc -zv <registry-domain> 443
curl -I https://<registry-domain>/v2/
# If behind a corporate proxy, configure Docker daemon to use it
# Add to /etc/systemd/system/docker.service.d/http-proxy.conf:
[Service]
Environment="HTTP_PROXY=http://proxy:port/" "HTTPS_PROXY=http://proxy:port/"
sudo systemctl daemon-reload
sudo systemctl restart dockerStep 4: Check Image Tag and Use Full Registry Path
Ensure the image reference in your Dockerfile `FROM` statement or build command includes the full registry URL.
# Incorrect (may default to Docker Hub)
FROM my-private-image:tag
# Correct
FROM <registry-url>/my-private-image:tag
# Build with explicit tagging
docker build -t <registry-url>/my-app:latest .Step 5: Verify Service Account or Instance Permissions (Cloud)
In hybrid/cloud builds (e.g., on a VM), ensure the machine's IAM role or service account has permissions to pull from the registry.
# For GCP GCR/AR on a GCE instance:
gcloud auth list
# For AWS ECR on an EC2 instance:
aws sts get-caller-identity
# Attach necessary IAM policy if permissions are missingStep 6: Manually Create or Update Docker Config
If automated login fails, manually encode and place credentials in the Docker config file.
# Create base64 auth string (format: username:password)
echo -n 'username:password' | base64
# Edit or create ~/.docker/config.json
cat > ~/.docker/config.json << EOF
{
"auths": {
"<registry-url>": {
"auth": "<your-base64-string>"
}
}
}
EOFArchitect's Pro Tip
"This often happens when CI/CD agents or cloud VMs use short-lived credentials (like OIDC tokens or IAM roles) that expire. For Jenkins, GitLab Runners, or GitHub Actions, ensure the credential refresh is part of the job's pre-build step, not just the pipeline setup."
Frequently Asked Questions
I'm logged in but still get 'access denied'. What's wrong?
Your credentials in `~/.docker/config.json` might be for a different registry URL (e.g., using `https://index.docker.io/v1/` instead of `https://registry.hub.docker.com`). Docker treats these as separate entries. Run `docker logout` and then `docker login` again with the exact URL used in your image tag.
How do I troubleshoot this in Kubernetes (e.g., ImagePullBackOff)?
The error originates at the node level. SSH into the node and run the Docker commands from this guide. Ensure the node's `~/.docker/config.json` is correct, or that a `imagePullSecrets` Kubernetes secret is correctly created and referenced in your Pod spec.