Varnish 503ing on ~1/100 POSTs

Ronan Mullally ronan at iol.ie
Tue Apr 5 23:00:44 CEST 2011


Hej Kristian,

On Tue, 5 Apr 2011, Kristian Grønfeldt Sørensen wrote:

> That indicates to me that the hack that was implemented to fix
> http://www.varnish-cache.org/trac/ticket/749 is not doing what it was
> supposed to do. The earlier varnishlog snippet from your original post
> includes a restart, which I assume is the restart added by the fix for
> #749 - unless you are doing a manual restart in your VCL. It seems that
> the backend connection that you get when the restart is done is also
> closed before Varnish sends the request.

Correct.  I've since implemented a manual restart to make a few extra
requests and the varnishlog for a typical incident is below.  Note that
this occured with Keepalive *disabled* on the apache backends
(and sess_timeout=5 on varnish).


-Ronan

   46 ReqStart     c 1.2.3.4 2902 1066443305
   46 RxRequest    c POST
   46 RxURL        c /ajax.php?do=verifyusername
   46 RxProtocol   c HTTP/1.1
   46 RxHeader     c Accept: */*
   46 RxHeader     c Accept-Language: ko
   46 RxHeader     c Referer: http://www.redcafe.net/register.php?do=register
   46 RxHeader     c x-requested-with: XMLHttpRequest
   46 RxHeader     c Content-Type: application/x-www-form-urlencoded; charset=UTF-8
   46 RxHeader     c Accept-Encoding: gzip, deflate
   46 RxHeader     c User-Agent: Mozilla/4.0 (compatible; ....
   46 RxHeader     c Host: www.redcafe.net
   46 RxHeader     c Content-Length: 48
   46 RxHeader     c Connection: Keep-Alive
   46 RxHeader     c Cache-Control: no-cache
   46 RxHeader     c Cookie: bbsessionhash=....
   46 VCL_call     c recv
   46 VCL_return   c pass
   46 VCL_call     c hash
   46 VCL_return   c hash
   46 VCL_call     c pass
   46 VCL_return   c pass
   48 BackendOpen  b redcafe2 193.27.1.46 62143 193.27.1.45 80
   46 Backend      c 48 redcafe redcafe2
   48 TxRequest    b POST
   48 TxURL        b /ajax.php?do=verifyusername
   48 TxProtocol   b HTTP/1.1
   48 TxHeader     b Accept: */*
   48 TxHeader     b Accept-Language: ko
   48 TxHeader     b Referer: http://www.redcafe.net/register.php?do=register
   48 TxHeader     b x-requested-with: XMLHttpRequest
   48 TxHeader     b Content-Type: application/x-www-form-urlencoded; charset=UTF-8
   48 TxHeader     b User-Agent: Mozilla/4.0 (....
   48 TxHeader     b Host: www.redcafe.net
   48 TxHeader     b Content-Length: 48
   48 TxHeader     b Cache-Control: no-cache
   48 TxHeader     b Cookie: bbsessionhash=....
   48 TxHeader     b Accept-Encoding: gzip
   48 TxHeader     b X-Forwarded-For: 1.2.3.4
   48 TxHeader     b X-Varnish: 1066443305
 * 46 FetchError   c backend write error: 104 (Connection reset by peer)
   48 BackendClose b redcafe2
   46 VCL_call     c error
   46 VCL_return   c restart
   46 VCL_call     c recv
   46 VCL_return   c pass
   46 VCL_call     c hash
   46 VCL_return   c hash
   46 VCL_call     c pass
   46 VCL_return   c pass
   48 BackendOpen  b redcafe2 193.27.1.46 62147 193.27.1.45 80
   46 Backend      c 48 redcafe redcafe2
   48 TxRequest    b POST
   48 TxURL        b /ajax.php?do=verifyusername
   48 TxProtocol   b HTTP/1.1
   48 TxHeader     b Accept: */*
   48 TxHeader     b Accept-Language: ko
   48 TxHeader     b Referer: http://www.redcafe.net/register.php?do=register
   48 TxHeader     b x-requested-with: XMLHttpRequest
   48 TxHeader     b Content-Type: application/x-www-form-urlencoded; charset=UTF-8
   48 TxHeader     b User-Agent: Mozilla/4.0 (...
   48 TxHeader     b Host: www.redcafe.net
   48 TxHeader     b Content-Length: 48
   48 TxHeader     b Cache-Control: no-cache
   48 TxHeader     b Cookie: bbsessionhash=...
   48 TxHeader     b X-Forwarded-For: 1.2.3.4
   48 TxHeader     b Accept-Encoding: gzip
   48 TxHeader     b X-Varnish: 1066443305
 * 46 FetchError   c backend write error: 0 (Success)
   48 BackendClose b redcafe2
   46 VCL_call     c error
   46 VCL_return   c restart
   46 VCL_call     c recv
   46 VCL_return   c pass
   46 VCL_call     c hash
   46 VCL_return   c hash
   46 VCL_call     c pass
   46 VCL_return   c pass
   48 BackendOpen  b redcafe1 193.27.1.46 24825 193.27.1.44 80
   46 Backend      c 48 redcafe redcafe1
   48 TxRequest    b POST
   48 TxURL        b /ajax.php?do=verifyusername
   48 TxProtocol   b HTTP/1.1
   48 TxHeader     b Accept: */*
   48 TxHeader     b Accept-Language: ko
   48 TxHeader     b Referer: http://www.redcafe.net/register.php?do=register
   48 TxHeader     b x-requested-with: XMLHttpRequest
   48 TxHeader     b Content-Type: application/x-www-form-urlencoded; charset=UTF-8
   48 TxHeader     b User-Agent: Mozilla/4.0 (...
   48 TxHeader     b Host: www.redcafe.net
   48 TxHeader     b Content-Length: 48
   48 TxHeader     b Cache-Control: no-cache
   48 TxHeader     b Cookie: bbsessionhash=...
   48 TxHeader     b X-Forwarded-For: 1.2.3.4
   48 TxHeader     b Accept-Encoding: gzip
   48 TxHeader     b X-Varnish: 1066443305
 * 46 FetchError   c backend write error: 0 (Success)
   48 BackendClose b redcafe1
   46 VCL_call     c error
   46 VCL_return   c restart
   46 VCL_call     c recv
   46 VCL_return   c pass
   46 VCL_call     c hash
   46 VCL_return   c hash
   46 VCL_call     c pass
   46 VCL_return   c pass
   48 BackendOpen  b redcafe2 193.27.1.46 62149 193.27.1.45 80
   46 Backend      c 48 redcafe redcafe2
   48 TxRequest    b POST
   48 TxURL        b /ajax.php?do=verifyusername
   48 TxProtocol   b HTTP/1.1
   48 TxHeader     b Accept: */*
   48 TxHeader     b Accept-Language: ko
   48 TxHeader     b Referer: http://www.redcafe.net/register.php?do=register
   48 TxHeader     b x-requested-with: XMLHttpRequest
   48 TxHeader     b Content-Type: application/x-www-form-urlencoded; charset=UTF-8
   48 TxHeader     b User-Agent: Mozilla/4.0 (...
   48 TxHeader     b Host: www.redcafe.net
   48 TxHeader     b Content-Length: 48
   48 TxHeader     b Cache-Control: no-cache
   48 TxHeader     b Cookie: bbsessionhash=...
   48 TxHeader     b X-Forwarded-For: 1.2.3.4
   48 TxHeader     b Accept-Encoding: gzip
   48 TxHeader     b X-Varnish: 1066443305
 * 46 FetchError   c backend write error: 0 (Success)
   48 BackendClose b redcafe2
   46 SessionClose c remote closed
   46 VCL_call     c error
   46 VCL_return   c restart
   46 VCL_call     c recv
   46 VCL_return   c pass
   46 VCL_call     c error
   46 VCL_return   c restart
   46 VCL_call     c deliver
   46 VCL_return   c deliver
   46 TxProtocol   c HTTP/1.1
   46 TxStatus     c 503
   46 TxResponse   c Service Unavailable
   46 TxHeader     c Server: Varnish
   46 TxHeader     c Retry-After: 0
   46 TxHeader     c Date: Tue, 05 Apr 2011 17:20:00 GMT
   46 TxHeader     c X-Varnish: 1066443305
   46 TxHeader     c Age: 0
   46 TxHeader     c Via: 1.1 varnish
   46 TxHeader     c Connection: close
   46 Debug        c "Write error, retval = -1, len = 174, errno = Broken pipe"
   -1 Length       - 0
   46 ReqEnd       c 1066443305 1302024000.284072638 1302024000.408756971 0.000043154 0.124658108 0.000026226
   46 StatSess     c 1.2.3.4 2902 0 1 1 0 4 0 174 0


-Ronan


More information about the varnish-misc mailing list