only 503s after a few hours (trunk-4384)

[Fernando Papa]

Hello,

After a few hours, Varnish trunk-4384 started replying with 503.
New work-load, first day of use.  Previously used Varnish only with internal 
traffic (eg no crazy headers, just carefully crafted requests).

Facts:

* Xeon 2.4 GHz (quad-core, 512KBx4 cache) server.  CentOS 5.4.
* Worked fine for ~9 hours, sending to backend 150+ K requests plus Varnish 
pings.
* Backend is an application server.  Payload is <1 KB plus minimal headers.
* I can still access backend bypassing Varnish.
* Varnish continues pinging backend, and receiving a timely 200 OK every 
time (backend logs).
* req.backend.healthy = true.  Tested with a codepath along the lines 
(req.request == "GET") && (req.url == "/") && (req.backend.healthy), that 
does an 'error 200 "OK"'.
* varnishlog does not output anything at all.  Not even the backend pings.
* Tested every path within vcl_recv.  Exiting with "error" works, but 
"lookup" results in a 503.
* Very low CPU usage.  Sorry, didn't check memory usage before restarting 
it.

CFLAGS='-O2 -fomit-frame-pointer -march=pentium4 -mfpmath=sse' 
 ./configure --prefix=$HOME/local --disable-largefile

$HOME/local/sbin/varnishd -a *:80 -f $HOME/local/etc/varnish/xxx.vcl -p 
obj_workspace=4096 -s malloc,100M -p thread_pools=`grep -c 'model name' 
/proc/cpuinfo`

backend fd1
{
   .host="localhost";
   .port="85";
   .connect_timeout = 1s;
   .first_byte_timeout = 20s;
   .between_bytes_timeout =2s;
   .probe = {
      .request =
         "GET /xxx HTTP/1.1"
         "Host: yyy"
         "Connection: close";
      .timeout = 0.5 s;
      .window = 8;
      .threshold = 3;
   }
}

director fdcluster round-robin
{
   { .backend = fd1; }
}

Restarted Varnish, now it works again.

Any idea?  Anything wrong in the command line?

Thanks,
Fernando.