only 503s after a few hours (trunk-4384)

[Fernando Papa]

Hello,

>>5 Debug        - "Too many open files when accept(2)ing. Sleeping."
>><EOF>
>
> That's your trouble:  The kernel does not allow varnish to open
> as many filedescriptors as it needs.
>
> Use whatever means your kernel has to change this number, possibly
> ulimit(1) or similar.


That certainly improves uptime.
I believe it is leaking descriptors.  It can't possible use so many with the 
kind of traffic (~7 req/s) it's receiving.

I restarted it yesterday with 128K descriptors.

# ps -AF | grep varn
root      4726     1  0 21419   928   2 Dec09 ?        00:00:00 varnishd -a 
*:80 -f frontdoor.vcl -p obj_workspace 4096 -s malloc,100M -p thread_pools 4
nobody    4727  4726  0 140497 153232 3 Dec09 ?        00:06:48 varnishd -a 
*:80 -f frontdoor.vcl -p obj_workspace 4096 -s malloc,100M -p thread_pools 4

# ls -l /proc/4727/fd | wc -l
1253

It's currently using 1253 descriptors.

# ls -lat /proc/4727/fd | grep "socket" | head -n 1
lrwx------ 1 nobody nobody 64 Dec 10 11:40 1244 -> socket:[17728046]

# ls -lat /proc/4727/fd | grep "socket" | tail -n 5
lrwx------ 1 nobody nobody 64 Dec  9 15:27 23 -> socket:[16402456]
lrwx------ 1 nobody nobody 64 Dec  9 15:27 24 -> socket:[16261656]
lrwx------ 1 nobody nobody 64 Dec  9 15:27 25 -> socket:[16411051]
lrwx------ 1 nobody nobody 64 Dec  9 15:27 4 -> socket:[16216281]
lrwx------ 1 nobody nobody 64 Dec  9 15:27 5 -> socket:[16216282]

# ls -lat /proc/4727/fd | grep "15:" | wc -l
76

It has 76 sockets that are 20 hours old.  Listening and monitoring sockets 
might explain a few of these.  [[ NOTE: Re-checked after an hour and it 
still has 76 ]]

# lsof -l -R -V -Tqs | grep 16216282
varnishd 4727 4726 99 5u IPv4 16216282 TCP *:http (LISTEN QR=0 QS=0)

OK.  16216281 also listens, but is IPv6.

# lsof -l -R -V -Tqs | grep 16411051
varnishd   4727  4726       99   25u     sock        0,5 
16411051 can't identify protocol

# grep 16411051 /proc/net/*
<EOF>

# netstat --verbose --numeric --extend --all --notrim | grep 16411051
<EOF>