abnormally high load?
Jeremy Hinegardner
jeremy at hinegardner.org
Wed Aug 12 20:08:46 CEST 2009
On Wed, Aug 12, 2009 at 05:41:52PM +0100, Rob S wrote:
> Jeremy Hinegardner wrote:
>> Hi all,
>>
>> I'm trying to figure out if this is a normal situation or not. We have a
>> varnish instance in front of 12 tokyo tyrant instances with some inline C
>> in the VCL to determine which backend to talk to.
>>
>>
>
> If you restart varnish during one of these spikes, does it instantly
> disappear? I've seen this happen (though only spiking to about 12), and
> this is when Varnish has munched through far more memory than we've
> allocated it. This problem is one I've been looking into with Ken
> Brownfield, and touches on
> http://projects.linpro.no/pipermail/varnish-misc/2009-April/002743.html and
> http://projects.linpro.no/pipermail/varnish-misc/2009-June/002840.html
>
> Do any of these tie up with your experience?
Possibly, the correlation I can see with those instances is this section of our
VCL
sub vcl_recv {
...
} else if ( req.request == "PUT" || req.request == "PURGE" ) {
purge( "req.url == " req.url );
if ( req.request == "PUT" ) {
pass;
} else {
error 200 "PURGE Success";
}
}
...
}
We do a consistent stream of PUT operations, its probably 10-15% of all
our operations. So our ban list would get farily large I'm guessing?
I'm not seeing evidence of a memory leak, and the pmap of the process does show
4G in the varnish_storage.bin mapping.
I've attached the output of 'varnishstat -1' if that helps. This is after I've
diverted some traffic around varnish because of the load.
If this purge() is the culprit, then I should make this change?
sub vcl_recv {
...
} else if ( req.request == "PUT" || req.request == "PURGE" ) {
lookup;
}
...
}
sub vcl_hit {
if ( req.request == "PUT" || req.request == "PURGE" ) {
set obj.ttl = 0s;
if ( req.request == "PURGE" ) {
error 200 "PURGE Success";
}
pass;
}
}
sub vcl_miss {
if ( req.request == "PUT" ) {
pass;
}
if ( req.request == "PURGE") {
error 404 "Not in cache.";
}
}
enjoy,
-jeremy
--
========================================================================
Jeremy Hinegardner jeremy at hinegardner.org
-------------- next part --------------
uptime 69385 . Child uptime
client_conn 29347846 422.97 Client connections accepted
client_req 29342148 422.89 Client requests received
cache_hit 15094822 217.55 Cache hits
cache_hitpass 0 0.00 Cache hits for pass
cache_miss 17210495 248.04 Cache misses
backend_conn 23663670 341.05 Backend connections success
backend_unhealthy 0 0.00 Backend connections not attempted
backend_busy 0 0.00 Backend connections too many
backend_fail 0 0.00 Backend connections failures
backend_reuse 23663248 341.04 Backend connections reuses
backend_recycle 23663599 341.05 Backend connections recycles
backend_unused 0 0.00 Backend connections unused
n_srcaddr 4 . N struct srcaddr
n_srcaddr_act 1 . N active struct srcaddr
n_sess_mem 537 . N struct sess_mem
n_sess 3752 . N struct sess
n_object 314256 . N struct object
n_objecthead 314769 . N struct objecthead
n_smf 647386 . N struct smf
n_smf_frag 22063 . N small free smf
n_smf_large 4 . N large free smf
n_vbe_conn 377 . N struct vbe_conn
n_bereq 175 . N struct bereq
n_wrk 45 . N worker threads
n_wrk_create 2659 0.04 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 397529 5.73 N worker threads limited
n_wrk_queue 0 0.00 N queued work requests
n_wrk_overflow 121342 1.75 N overflowed work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 12 . N backends
n_expired 270565 . N expired objects
n_lru_nuked 4241197 . N LRU nuked objects
n_lru_saved 0 . N LRU saved objects
n_lru_moved 15005184 . N LRU moved objects
n_deathrow 0 . N objects on deathrow
losthdr 0 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 29343483 422.91 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 29347845 422.97 Total Sessions
s_req 29346604 422.95 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 6482264 93.42 Total pass
s_fetch 10818028 155.91 Total fetch
s_hdrbytes 5839972531 84167.65 Total header bytes
s_bodybytes 26762607898 385711.72 Total body bytes
sess_closed 29347707 422.97 Session Closed
sess_pipeline 0 0.00 Session Pipeline
sess_readahead 0 0.00 Session Read Ahead
sess_linger 0 0.00 Session Linger
sess_herd 1 0.00 Session herd
shm_records 1586983473 22872.14 SHM records
shm_writes 170510686 2457.46 SHM writes
shm_flushes 27352 0.39 SHM flushes due to overflow
shm_cont 1049297 15.12 SHM MTX contention
shm_cycles 446 0.01 SHM cycles through buffer
sm_nreq 55032175 793.14 allocator requests
sm_nobj 625319 . outstanding allocations
sm_balloc 3887177728 . bytes allocated
sm_bfree 407789568 . bytes free
sma_nreq 0 0.00 SMA allocator requests
sma_nobj 0 . SMA outstanding allocations
sma_nbytes 0 . SMA outstanding bytes
sma_balloc 0 . SMA bytes allocated
sma_bfree 0 . SMA bytes free
sms_nreq 3432449 49.47 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 18446744073709543408 . SMS outstanding bytes
sms_balloc 1505666175 . SMS bytes allocated
sms_bfree 1505674383 . SMS bytes freed
backend_req 23663973 341.05 Backend requests made
n_vcl 3 0.00 N vcl total
n_vcl_avail 3 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_purge 490689 . N total active purges
n_purge_add 5083838 73.27 N new purges added
n_purge_retire 4593149 66.20 N old purges deleted
n_purge_obj_test 15296959 220.46 N objects tested
n_purge_re_test 249073265875 3589727.84 N regexps tested against
n_purge_dups 0 0.00 N duplicate purges removed
hcb_nolock 0 0.00 HCB Lookups without lock
hcb_lock 0 0.00 HCB Lookups with lock
hcb_insert 0 0.00 HCB Inserts
esi_parse 0 0.00 Objects ESI parsed (unlock)
esi_errors 0 0.00 ESI parse errors (unlock)
More information about the varnish-misc
mailing list