Varnish crashing when system starts to swap

[DHF]

Calle Korjus wrote:
> This is our startup command:
>
> /opt/varnish/sbin/varnishd -a :80 -p lru_interval 3600 -f /opt/varnish/conf/default.vcl -T 127.0.0.1:6082 -t 3600 -w 128,1000,60 -u varnish -g varnish -s file,/srv/varnish/varnish_storage.bin,30G -P /var/run/varnish.pid
>
> Varnish looks fine until it's had abour 1,5 million requests, then we can see the kswapd0 and kswapd1 start working and load average rises to about 200 and the machine gets totally unresponsive. Top shows a lot of cpu beeing spent on i/o waits and varnish child process restarts sometimes. In best case the process restarts and the server starts behaving within 5 minutes but sometimes varnish dies completely. One thing we have noticed is that the reserved memory for varnish keeps rising and when it crashes it is usually around 14G.
>   
I would try lowering the storage file size to within your total system
ram, subtracting some memory for buffers and cache and apache.  See if
it still spirals into swap hell.  You could also try setting rlimits for
the varnish user, though I don't know if settings in
/etc/security/limits.conf apply to privilege dropped processes.

> The varnish storage file is running on the same physical disk as the system and the swap, could that be the problem? Should varnish really allocate so much memory so that the system starts to swap to disk?
>   
I think what is happening is that your hit ratio is low  and your
storage size is quite large, so varnish has enough objects marked as hot
that its trying to hold them all in memory?  I don't know for sure, I
could be way off.  I think if you restrict the storage size there will
be increased disk activity as you churn the cache, but you won't be
churning swap space as well, and you shouldn't exhaust the virtual
memory of the system.  I'd have to test that though.

--Dave