Root Cause Analysis: Why Nginx Fails with 'Invalid Chunked Response' from Upstream
Quick Fix Summary
TL;DRDisable chunked transfer encoding with 'proxy_http_version 1.0' or buffer the entire response.
Nginx's HTTP/1.1 proxy module expects properly formatted chunked encoding from upstream servers. When upstream sends malformed chunk size headers or premature termination, Nginx rejects the response as invalid. This is a protocol violation that breaks the client connection.
Diagnosis & Causes
Recovery Steps
Step 1: Force HTTP/1.0 to Disable Chunked Encoding
HTTP/1.0 doesn't support chunked transfer encoding. This forces simple response buffering.
location /api/ {
proxy_pass http://backend;
proxy_http_version 1.0;
proxy_set_header Connection "";}Step 2: Enable Full Response Buffering
Buffer the entire upstream response before sending to client. This handles malformed chunking internally.
location /api/ {
proxy_pass http://backend;
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
proxy_busy_buffers_size 8k;}Step 3: Debug Upstream Response with tcpdump
Capture raw HTTP traffic to inspect malformed chunked encoding from upstream.
sudo tcpdump -i any -A -s 0 'port 8080 and host backend_ip'
# Look for chunk size lines like:
# "Transfer-Encoding: chunked"
# "\r\n" (empty line)
# "1a" (hex chunk size)
# "{data}"
# "0\r\n\r\n" (termination)Step 4: Validate Upstream Application Code
Check application frameworks for proper chunked encoding implementation.
# Python Flask example - Ensure no manual Content-Length with chunked
from flask import Response
def stream_data():
def generate():
yield b"Hello "
yield b"World"
return Response(generate(), content_type='text/plain')
# NOT: Response(data, headers={'Content-Length': '10'})Step 5: Configure Upstream Timeouts and Retries
Prevent race conditions from upstream timeouts during chunked transfer.
location /api/ {
proxy_pass http://backend;
proxy_read_timeout 300s;
proxy_connect_timeout 75s;
proxy_send_timeout 300s;
proxy_next_upstream error timeout invalid_header http_500;}Step 6: Patch Nginx with Custom Lua Handler
Use OpenResty's Lua module to sanitize malformed chunked responses.
location /api/ {
proxy_pass http://backend;
header_filter_by_lua_block {
ngx.header["Transfer-Encoding"] = nil
ngx.header["Content-Length"] = #ngx.arg[1]
}
body_filter_by_lua_block {
ngx.arg[1] = ngx.arg[1]:gsub("\\r\\n[0-9a-fA-F]+\\r\\n", "")
}}Architect's Pro Tip
"The error often surfaces during zero-downtime deployments when old and new application versions simultaneously proxy traffic with different HTTP semantics."
Frequently Asked Questions
Why does Nginx reject chunked encoding but other proxies don't?
Nginx strictly follows RFC 7230 Section 4.1 for chunked encoding validation. Some proxies are more lenient, accepting malformed responses that eventually cause client-side issues.
Can this error cause data corruption or just connection drops?
Primarily connection drops (502 Bad Gateway). However, if buffering is partial, clients may receive truncated responses leading to logical data corruption.
Is this more common with specific programming languages or frameworks?
Yes. Older Java servlet containers, misconfigured Python ASGI servers, and custom C++ HTTP implementations frequently produce invalid chunked boundaries during streaming responses.