503 caused by health probe
I was working with a container app deployed on Azure recently. This container app provides a REST endpoint that allows users to upload files for processing.
A few days ago, uploading larger files started failing repeatedly. These files weren't particularly large either. One that constantly failed to upload was only 4 MB.
I tried uploading this file both via curl and through the web UI. Both attempts failed with a 503 status code. That ruled out a CORS issue, which would have resulted in a different status code and would not have caused curl to fail.
Interestingly, we didn't find these POST requests in the logs. This suggested the requests never reached the container app.
By inspecting the configation of this and other container apps deployed on Azure, I noticed a health probe setting for this app.
It turned out Azure was checking whether the service was alive every 10 seconds. While the app became temporarily unresponsive during the upload and processing, the health probe likely timed out.
Azure would have interpreted this as a sign that the container was down, and either removed it from the load balancer or tried to restarted it. Either way, the request was abruptly terminated, resulting in the 503 error.