abnormally high load?

[Kristian Lyngstol]

On Wed, Aug 12, 2009 at 09:59:53AM -0600, Jeremy Hinegardner wrote:
> I'm trying to figure out if this is a normal situation or not.  We have a
> varnish instance in front of 12 tokyo tyrant instances with some inline C 
> in the VCL to determine which backend to talk to.

How does your VCL look?

> Sporadically during the day the load on the dedicated varnish machine will spike
> up into the 150-200 range and the response times from varnish drop by an order
> of magnitude.  This is an 8CPU Centos 5.3 machine with 8GB of RAM
> and the varnish commandline is:
> 
>     varnishd -a :6081 \
>              -T localhost:6082 \
>              -f /etc/varnish/default.vcl \
>              -w 8,512 \
>              -u varnish -g varnish \
>              -h classic,1000003 \
>              -s file,/var/lib/varnish/varnish_storage.bin,4G"

I can instantly tell you that you have too few threads.

Generally, you want to set your thread_pool_min to something reasonably
high compared to your normal client load. You can also set thread_pools to
8, as you have 8 CPU cores (though I doubt that's the issue). Starting with
something like -w 200,2000. This should give you a reasonable amount of
threads. Idle threads are virtually cost-free, while creating new threads
is expensive (given the premise that you'll be doing it when you are most
bussy).

Secondly, you should watch your varnishstat and check your syslog. You are
looking for overflowed work requests and dropped worked requests in
varnishstat, and you are looking for assert-errors in your syslog. You
should also monitor the backend_connection_failures counter, which should
always be low.

> I could use any help on attempting to figure out why we get these spikes and
> ways to mitigate them.   Anecdotally, these spikes appear to happen when the
> cache hit ratio is low.

What sort of cache hit ratio are we talking about? And what sort of request
rate (in requests/second)?

> I've read through many of the past threads on tuning varnish for high load and
> attempted many of the configurations, yet none have really.
> 
> The client activity we are dealing with is, on the high end, around 400 client
> programs hitting varnish with an average rate of say 30 requests / sec each.

This number sounds a bit high. 400 clients with 30 req/s will lead to 12000
req/s. Is this really what varnishstat is telling you too? In that case,
your cache hit rate will be crucial.

It could also be useful to see the entire output of varnishstat after
Varnish have been running for a while, to get a general idea of how varnish
is behaving in your setup.

-- 
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
Url : http://projects.linpro.no/pipermail/varnish-misc/attachments/20090812/e0477bdc/attachment.pgp