This article is about how to upload files to Compute Engine fronted by a Google HTTP Load Balancer and how to diagnose and correct file upload timeouts. The result is a better understanding of how to configure the Google Cloud HTTP Load Balancer.
The default timeout for Google HTTPS Load Balancers is 30 seconds for the backend to respond before considering it a failed request. For normal HTTP GET requests, 30 seconds should be plenty of time. If you are using databases where queries can take a long time, factor that into your timeout setting. You need to balance the timeout with security concerns. The longer the timeout, the more connections can be opened and kept open during a denial of service attack. The Slowloris attack is an example of exhausting a server’s resources (link).
For file uploads, 30 seconds is often too short of a timeout. A 10 MByte upload at 1 Mbit per second will take 80 seconds. A 30-second timeout will cause these types of upload to constantly fail.
Timeouts and Retries
Google HTTP Load Balancers have two distinct types of timeouts:
- A configurable response timeout, which represents the amount of time the load balancer will wait for your backend to return a complete response. It is not an idle (keepalive) timeout. This timeout is configurable by modifying the timeout setting for your backend service. The default value is 30 seconds. Consider increasing this timeout under these circumstances:
- If you expect a backend to take longer to return HTTP responses, or
- If the connection is upgraded to a WebSocket.
- A TCP session timeout, whose value is fixed at 10 minutes (600 seconds). This session timeout is sometimes called a keepalive or idle timeout, and its value is not configurable by modifying your backend service. You must configure the web server software used by your backends so that its keepalive timeout is longer than 600 seconds to prevent connections from being closed prematurely by the backend. This timeout does not apply to WebSockets.
Long-Lived Connections (TCP Session Timeout)
If your backend instances initiate or accept long-lived connections with an external host, you should adjust the TCP keep-alive settings. Refer to this document on Communicating between your instances and the Internet.
HTTP Load Balancing Logging and Monitoring
In Stackdriver logs, select “Cloud HTTP Load Balancer -> All forwarding_rule_name.
The load balancer will log errors in Stackdriver. Look for the following “statusDetails”:
- backend_timeout – The backend timed out while generating a response. HTTP Error 502.
- backend_connection_closed_before_data_sent_to_client – The backend unexpectedly closed its connection to the load balancer before the response was proxied to the client. This can happen if the load balancer is sending traffic to another entity, such as a third party load balancer running on a VM instance, that has a TCP timeout that is shorter than the HTTP(S) load balancer’s 10 minute (600 second) timeout. Manually setting the TCP timeout (keepalive) on the target service to greater than 600 seconds may fix this problem. HTTP Error 502.
Do some calculations on the size of the uploads required or measuring the response time for your database queries. Create a test environment and load the server with requests, uploads, downloads, etc – whatever you need to support. Add a multiplier such as 20%, 100%, or whatever fits your goals. Consider security issues such as DoS. For uploads always cleanse file names. Always filter and cleanse database queries. Prohibit file uploads above a specified size.
- Go to the Google Cloud Console.
- From the “hamburger” menu (top left) select “Network services” -> “Load balancing”.
- Click on your load balancer.
- Click on “Backend configuration”
- Click the “pencil” edit icon.
- Click the “pencil” edit icon next to “Timeout”.
- Change the timeout value to your desired value.
- Click Update.
- Repeat for each backend.
- Click Update in the main Load balancer screen.
- Google Cloud Load Balancing Backend Services
- Nginx keepalive_timeout is not compatible with Google HTTP Load Balancers
- HTTP(S) Load Balancing Logging and Monitoring
I design software for enterprise-class systems and data centers. My background is 30+ years in storage (SCSI, FC, iSCSI, disk arrays, imaging) virtualization. 20+ years in identity, security, and forensics.
For the past 14+ years, I have been working in the cloud (AWS, Azure, Google, Alibaba, IBM, Oracle) designing hybrid and multi-cloud software solutions. I am an MVP/GDE with several.