Varnish virtual memory usage

Ken Brownfield kb+varnish at slide.com
Thu Nov 5 02:16:05 CET 2009


Ah, sorry, missed that the command-line was in there.

Given 1G of cache, a large sess_workspace and shm_workspace buffers,  
and the number of threads, the math adds up correctly.  Do you  
definitely need those large buffers?

Your memory footprint will simply increase with thread count; reducing  
active simultaneous connections, reducing the stack size, and reducing  
the large sess_workspace are the only ways I know of for you to  
control the memory.  I'm really not seeing a leak or malfunction,  
IMHO.  The reason behind your high/growing worker count is worth  
investigating (lower send_timeout? slow/disconnecting clients? strace  
the threads to see what they're doing?)

Minor thing: overflow_max is a percentage, so 10000 is probably ignored?
-- 
Ken

On Nov 4, 2009, at 4:47 PM, Henry Paulissen wrote:

> See the pmap.txt attachment.
> The startup command is in the beginning of the file.
>
>
> /usr/local/varnish/sbin/varnishd -P /var/run/xxx.pid -a 0.0.0.0:xxx -f
> /usr/local/varnish/etc/varnish/xxx.xxx.xxx.vcl -T 0.0.0.0:xxx -s  
> malloc,1G
> -i xxx -n /usr/local/varnish/var/varnish/xxx -p obj_workspace 8192 -p
> sess_workspace 262144 -p listen_depth 8192 -p lru_interval 60 -p
> sess_timeout 10 -p shm_workspace 32768 -p ping_interval 2 -p  
> thread_pools 4
> -p thread_pool_min 50 -p thread_pool_max 4000 -p esi_syntax 1 -p
> overflow_max 10000
>
>
> -----Oorspronkelijk bericht-----
> Van: Ken Brownfield [mailto:kb+varnish at slide.com]
> Verzonden: donderdag 5 november 2009 1:42
> Aan: Henry Paulissen
> Onderwerp: Re: Varnish virtual memory usage
>
> Is your -s set at 1.5GB?  What's your varnishd command line?
>
> I'm not sure if you realize that thread_pool does not control the
> number of threads, only the number of pools (and mutexes).  I think
> thread_pool_max is what you're looking for?
> -- 
> Ken
>
> On Nov 4, 2009, at 4:37 PM, Henry Paulissen wrote:
>
>> Running varnishd now for abount 30 minutes with a thread_pool of 4.
>>
>> =
>> =
>> =
>> =
>> =
>> =
>> = 
>> =====================================================================
>> ========================
>> =
>> =
>> =
>> =
>> =
>> =
>> = 
>> =====================================================================
>> ========================
>> uptime                   2637          .   Child uptime
>> client_conn            316759       120.12 Client connections  
>> accepted
>> client_drop                 0         0.00 Connection dropped, no  
>> sess
>> client_req             316738       120.11 Client requests received
>> cache_hit               32477        12.32 Cache hits
>> cache_hitpass               0         0.00 Cache hits for pass
>> cache_miss              93703        35.53 Cache misses
>> backend_conn           261033        98.99 Backend conn. success
>> backend_unhealthy            0         0.00 Backend conn. not
>> attempted
>> backend_busy                0         0.00 Backend conn. too many
>> backend_fail                0         0.00 Backend conn. failures
>> backend_reuse           23305         8.84 Backend conn. reuses
>> backend_toolate           528         0.20 Backend conn. was closed
>> backend_recycle         23833         9.04 Backend conn. recycles
>> backend_unused              0         0.00 Backend conn. unused
>> fetch_head                  0         0.00 Fetch head
>> fetch_length           280973       106.55 Fetch with Length
>> fetch_chunked            1801         0.68 Fetch chunked
>> fetch_eof                   0         0.00 Fetch EOF
>> fetch_bad                   0         0.00 Fetch had bad headers
>> fetch_close              1329         0.50 Fetch wanted close
>> fetch_oldhttp               0         0.00 Fetch pre HTTP/1.1 closed
>> fetch_zero                  0         0.00 Fetch zero len
>> fetch_failed                0         0.00 Fetch failed
>> n_sess_mem                284          .   N struct sess_mem
>> n_sess                     35          .   N struct sess
>> n_object                90560          .   N struct object
>> n_vampireobject             0          .   N unresurrected objects
>> n_objectcore            90616          .   N struct objectcore
>> n_objecthead            25146          .   N struct objecthead
>> n_smf                       0          .   N struct smf
>> n_smf_frag                  0          .   N small free smf
>> n_smf_large                 0          .   N large free smf
>> n_vbe_conn                 10          .   N struct vbe_conn
>> n_wrk                     200          .   N worker threads
>> n_wrk_create              248         0.09 N worker threads created
>> n_wrk_failed                0         0.00 N worker threads not
>> created
>> n_wrk_max              100988        38.30 N worker threads limited
>> n_wrk_queue                 0         0.00 N queued work requests
>> n_wrk_overflow            630         0.24 N overflowed work requests
>> n_wrk_drop                  0         0.00 N dropped work requests
>> n_backend                   5          .   N backends
>> n_expired                1027          .   N expired objects
>> n_lru_nuked              2108          .   N LRU nuked objects
>> n_lru_saved                 0          .   N LRU saved objects
>> n_lru_moved             12558          .   N LRU moved objects
>> n_deathrow                  0          .   N objects on deathrow
>> losthdr                     5         0.00 HTTP header overflows
>> n_objsendfile               0         0.00 Objects sent with sendfile
>> n_objwrite             315222       119.54 Objects sent with write
>> n_objoverflow               0         0.00 Objects overflowing
>> workspace
>> s_sess                 316740       120.11 Total Sessions
>> s_req                  316738       120.11 Total Requests
>> s_pipe                      0         0.00 Total pipe
>> s_pass                 190664        72.30 Total pass
>> s_fetch                284103       107.74 Total fetch
>> s_hdrbytes          114236150     43320.50 Total header bytes
>> s_bodybytes         355198316    134697.88 Total body bytes
>> sess_closed            316740       120.11 Session Closed
>> sess_pipeline               0         0.00 Session Pipeline
>> sess_readahead              0         0.00 Session Read Ahead
>> sess_linger                 0         0.00 Session Linger
>> sess_herd                  33         0.01 Session herd
>> shm_records          27534992     10441.79 SHM records
>> shm_writes            1555265       589.79 SHM writes
>> shm_flushes                 0         0.00 SHM flushes due to  
>> overflow
>> shm_cont                 1689         0.64 SHM MTX contention
>> shm_cycles                 12         0.00 SHM cycles through buffer
>> sm_nreq                     0         0.00 allocator requests
>> sm_nobj                     0          .   outstanding allocations
>> sm_balloc                   0          .   bytes allocated
>> sm_bfree                    0          .   bytes free
>> sma_nreq               379783       144.02 SMA allocator requests
>> sma_nobj               181121          .   SMA outstanding  
>> allocations
>> sma_nbytes         1073735584          .   SMA outstanding bytes
>> sma_balloc         1488895305          .   SMA bytes allocated
>> sma_bfree           415159721          .   SMA bytes free
>> sms_nreq                  268         0.10 SMS allocator requests
>> sms_nobj                    0          .   SMS outstanding  
>> allocations
>> sms_nbytes                  0          .   SMS outstanding bytes
>> sms_balloc             156684          .   SMS bytes allocated
>> sms_bfree              156684          .   SMS bytes freed
>> backend_req            284202       107.77 Backend requests made
>> n_vcl                       1         0.00 N vcl total
>> n_vcl_avail                 1         0.00 N vcl available
>> n_vcl_discard               0         0.00 N vcl discarded
>> n_purge                     1          .   N total active purges
>> n_purge_add                 1         0.00 N new purges added
>> n_purge_retire              0         0.00 N old purges deleted
>> n_purge_obj_test            0         0.00 N objects tested
>> n_purge_re_test             0         0.00 N regexps tested against
>> n_purge_dups                0         0.00 N duplicate purges removed
>> hcb_nolock                  0         0.00 HCB Lookups without lock
>> hcb_lock                    0         0.00 HCB Lookups with lock
>> hcb_insert                  0         0.00 HCB Inserts
>> esi_parse                   0         0.00 Objects ESI parsed  
>> (unlock)
>> esi_errors                  0         0.00 ESI parse errors (unlock)
>> =
>> =
>> =
>> =
>> =
>> =
>> = 
>> =====================================================================
>> ========================
>> =
>> =
>> =
>> =
>> =
>> =
>> = 
>> =====================================================================
>> ========================
>>
>> As you can see I have now 200 worker threads.
>> Still its using 1.8G and is still increasing (~1 to 5 mb/s)
>>
>>
>> -----Oorspronkelijk bericht-----
>> Van: Ken Brownfield [mailto:kb+varnish at slide.com]
>> Verzonden: donderdag 5 november 2009 1:18
>> Aan: Henry Paulissen
>> CC: varnish-misc at projects.linpro.no
>> Onderwerp: Re: Varnish virtual memory usage
>>
>> Hmm, well the memory adds up to a 1.5G -s option (can you confirm  
>> what
>> you use with -s?) and memory required to run the number of threads
>> you're running.  Unless your -s is drastically smaller than 1.5GB,  
>> the
>> pmap you sent is of a normal, non-leaking process.
>>
>> Ken
>>
>> On Nov 4, 2009, at 3:48 PM, Henry Paulissen wrote:
>>
>>> Our load balancer transforms all connections from keep-alive to
>>> close.
>>> So keep-alive connections aren’t the issue here.
>>>
>>> Also, if I limit the thread count I still see the same behavior.
>>>
>>> -----Oorspronkelijk bericht-----
>>> Van: Ken Brownfield [mailto:kb at slide.com]
>>> Verzonden: donderdag 5 november 2009 0:31
>>> Aan: Henry Paulissen
>>> CC: varnish-misc at projects.linpro.no
>>> Onderwerp: Re: Varnish virtual memory usage
>>>
>>> Looks like varnish is allocating ~1.5GB of RAM for pure cache (which
>>> may roughly match your "-s file" option) but 1,610 threads with your
>>> 1MB stack limit will use 1.7GB of RAM.  Pmap is reporting the
>>> footprint of this instance as roughly 3.6GB, and I'm assuming top/ps
>>> agree with that number.
>>>
>>> Unless your "-s file" option is significantly less than 1-1.5GB, the
>>> sheer thread count explains your memory usage: maybe using a
>>> stacksize
>>> of 512K or 256K could help, and/or disable keepalives on the client
>>> side?
>>>
>>> Also, if you happen to be using a load balancer, TCP Buffering
>>> (NetScaler) or Proxy Buffering? (BigIP) or the like can drastically
>>> reduce the thread count (and they can handle the persistent
>>> keepalives
>>> as well).
>>>
>>> But IMHO, an event-based (for example) handler for "idle" or "slow"
>>> threads is probably the next important feature, just below
>>> persistence.  Without something like TCP buffering, the memory
>>> available for actual caching is dwarfed by the thread stacksize  
>>> alloc
>>> overhead.
>>>
>>> Ken
>>>
>>> On Nov 4, 2009, at 3:18 PM, Henry Paulissen wrote:
>>>
>>>> I attached the memory dump.
>>>>
>>>> Child processes count gives me 1610 processes (on this instance).
>>>> Currently the server isn’t so busy (~175 requests / sec).
>>>>
>>>> Varnishstat -1:
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> = 
>>>> ===================================================================
>>>> ======
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> = 
>>>> ===================================================================
>>>> ======
>>>> uptime                   3090          .   Child uptime
>>>> client_conn            435325       140.88 Client connections
>>>> accepted
>>>> client_drop                 0         0.00 Connection dropped, no
>>>> sess
>>>> client_req             435294       140.87 Client requests received
>>>> cache_hit               45740        14.80 Cache hits
>>>> cache_hitpass               0         0.00 Cache hits for pass
>>>> cache_miss             126445        40.92 Cache misses
>>>> backend_conn           355277       114.98 Backend conn. success
>>>> backend_unhealthy            0         0.00 Backend conn. not
>>>> attempted
>>>> backend_busy                0         0.00 Backend conn. too many
>>>> backend_fail                0         0.00 Backend conn. failures
>>>> backend_reuse           34331        11.11 Backend conn. reuses
>>>> backend_toolate           690         0.22 Backend conn. was closed
>>>> backend_recycle         35021        11.33 Backend conn. recycles
>>>> backend_unused              0         0.00 Backend conn. unused
>>>> fetch_head                  0         0.00 Fetch head
>>>> fetch_length           384525       124.44 Fetch with Length
>>>> fetch_chunked            2441         0.79 Fetch chunked
>>>> fetch_eof                   0         0.00 Fetch EOF
>>>> fetch_bad                   0         0.00 Fetch had bad headers
>>>> fetch_close              2028         0.66 Fetch wanted close
>>>> fetch_oldhttp               0         0.00 Fetch pre HTTP/1.1  
>>>> closed
>>>> fetch_zero                  0         0.00 Fetch zero len
>>>> fetch_failed                0         0.00 Fetch failed
>>>> n_sess_mem                989          .   N struct sess_mem
>>>> n_sess                     94          .   N struct sess
>>>> n_object                89296          .   N struct object
>>>> n_vampireobject             0          .   N unresurrected objects
>>>> n_objectcore            89640          .   N struct objectcore
>>>> n_objecthead            25379          .   N struct objecthead
>>>> n_smf                       0          .   N struct smf
>>>> n_smf_frag                  0          .   N small free smf
>>>> n_smf_large                 0          .   N large free smf
>>>> n_vbe_conn                 26          .   N struct vbe_conn
>>>> n_wrk                    1600          .   N worker threads
>>>> n_wrk_create             1600         0.52 N worker threads created
>>>> n_wrk_failed                0         0.00 N worker threads not
>>>> created
>>>> n_wrk_max                1274         0.41 N worker threads limited
>>>> n_wrk_queue                 0         0.00 N queued work requests
>>>> n_wrk_overflow           1342         0.43 N overflowed work
>>>> requests
>>>> n_wrk_drop                  0         0.00 N dropped work requests
>>>> n_backend                   5          .   N backends
>>>> n_expired                1393          .   N expired objects
>>>> n_lru_nuked             35678          .   N LRU nuked objects
>>>> n_lru_saved                 0          .   N LRU saved objects
>>>> n_lru_moved             20020          .   N LRU moved objects
>>>> n_deathrow                  0          .   N objects on deathrow
>>>> losthdr                    11         0.00 HTTP header overflows
>>>> n_objsendfile               0         0.00 Objects sent with
>>>> sendfile
>>>> n_objwrite             433558       140.31 Objects sent with write
>>>> n_objoverflow               0         0.00 Objects overflowing
>>>> workspace
>>>> s_sess                 435298       140.87 Total Sessions
>>>> s_req                  435294       140.87 Total Requests
>>>> s_pipe                      0         0.00 Total pipe
>>>> s_pass                 263190        85.17 Total pass
>>>> s_fetch                388994       125.89 Total fetch
>>>> s_hdrbytes          157405143     50940.18 Total header bytes
>>>> s_bodybytes         533077018    172516.83 Total body bytes
>>>> sess_closed            435291       140.87 Session Closed
>>>> sess_pipeline               0         0.00 Session Pipeline
>>>> sess_readahead              0         0.00 Session Read Ahead
>>>> sess_linger                 0         0.00 Session Linger
>>>> sess_herd                  69         0.02 Session herd
>>>> shm_records          37936743     12277.26 SHM records
>>>> shm_writes            2141029       692.89 SHM writes
>>>> shm_flushes                 0         0.00 SHM flushes due to
>>>> overflow
>>>> shm_cont                 3956         1.28 SHM MTX contention
>>>> shm_cycles                 16         0.01 SHM cycles through  
>>>> buffer
>>>> sm_nreq                     0         0.00 allocator requests
>>>> sm_nobj                     0          .   outstanding allocations
>>>> sm_balloc                   0          .   bytes allocated
>>>> sm_bfree                    0          .   bytes free
>>>> sma_nreq               550879       178.28 SMA allocator requests
>>>> sma_nobj               178590          .   SMA outstanding
>>>> allocations
>>>> sma_nbytes         1073690180          .   SMA outstanding bytes
>>>> sma_balloc         2066782844          .   SMA bytes allocated
>>>> sma_bfree           993092664          .   SMA bytes free
>>>> sms_nreq                  649         0.21 SMS allocator requests
>>>> sms_nobj                    0          .   SMS outstanding
>>>> allocations
>>>> sms_nbytes                  0          .   SMS outstanding bytes
>>>> sms_balloc             378848          .   SMS bytes allocated
>>>> sms_bfree              378848          .   SMS bytes freed
>>>> backend_req            389342       126.00 Backend requests made
>>>> n_vcl                       1         0.00 N vcl total
>>>> n_vcl_avail                 1         0.00 N vcl available
>>>> n_vcl_discard               0         0.00 N vcl discarded
>>>> n_purge                     1          .   N total active purges
>>>> n_purge_add                 1         0.00 N new purges added
>>>> n_purge_retire              0         0.00 N old purges deleted
>>>> n_purge_obj_test            0         0.00 N objects tested
>>>> n_purge_re_test             0         0.00 N regexps tested against
>>>> n_purge_dups                0         0.00 N duplicate purges
>>>> removed
>>>> hcb_nolock                  0         0.00 HCB Lookups without lock
>>>> hcb_lock                    0         0.00 HCB Lookups with lock
>>>> hcb_insert                  0         0.00 HCB Inserts
>>>> esi_parse                   0         0.00 Objects ESI parsed
>>>> (unlock)
>>>> esi_errors                  0         0.00 ESI parse errors  
>>>> (unlock)
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> = 
>>>> ===================================================================
>>>> ======
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> = 
>>>> ===================================================================
>>>> ======
>>>>
>>>>
>>>>
>>>> -----Oorspronkelijk bericht-----
>>>> Van: Ken Brownfield [mailto:kb at slide.com]
>>>> Verzonden: donderdag 5 november 2009 0:01
>>>> Aan: Henry Paulissen
>>>> CC: Rogério Schneider
>>>> Onderwerp: Re: Varnish virtual memory usage
>>>>
>>>> Curious: For a heavily leaked varnish instance, can you run "pmap  
>>>> -x
>>>> PID" on the parent PID and child PID, and record how many threads
>>>> are
>>>> active (something like 'ps -efT | grep varnish | wc -l')?  Might
>>>> help
>>>> isolate the RAM usage.
>>>>
>>>> Sorry if you have done this already; didn't find it in my email
>>>> archive.
>>>>
>>>> Ken
>>>>
>>>> On Nov 4, 2009, at 2:53 PM, Henry Paulissen wrote:
>>>>
>>>>> No, varnishd still usages way more than allowed.
>>>>> The only solutions I found at the moment are:
>>>>>
>>>>> Run on x64 linux and restart varnish every 4 hours (crontab).
>>>>> Run on x32 linux (all is working as expected but you cant allocate
>>>>> more as
>>>>> 4G each instance).
>>>>>
>>>>>
>>>>> I hope linpro will find this issue and address it.
>>>>>
>>>>>
>>>>>
>>>>> Again @ linpro: if you need a machine (with live traffic) to run
>>>>> some tests,
>>>>> please contact me.
>>>>> We have multiple machines in high availability, so testing and
>>>>> rebooting a
>>>>> instance wouldn’t hurt us.	
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> -----Oorspronkelijk bericht-----
>>>>> Van: Rogério Schneider [mailto:stockrt at gmail.com]
>>>>> Verzonden: woensdag 4 november 2009 22:04
>>>>> Aan: Henry Paulissen
>>>>> CC: Scott Wilson; varnish-misc at projects.linpro.no
>>>>> Onderwerp: Re: Varnish virtual memory usage
>>>>>
>>>>> On Thu, Oct 22, 2009 at 6:04 AM, Henry Paulissen
>>>>> <h.paulissen at qbell.nl>
>>>>> wrote:
>>>>>> I will report back.
>>>>>
>>>>> Did this solve the problem?
>>>>>
>>>>> Removing this?
>>>>>
>>>>>>>   if (req.http.Cache-Control == "no-cache" || req.http.Pragma ==
>>>>> "no-cache") {
>>>>>>>           purge_url(req.url);
>>>>>>>   }
>>>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>> Att,
>>>>> -- 
>>>>> Rogério Schneider
>>>>>
>>>>> MSN: stockrt at hotmail.com
>>>>> GTalk: stockrt at gmail.com
>>>>> Skype: stockrt
>>>>> http://stockrt.github.com
>>>>>
>>>>> _______________________________________________
>>>>> varnish-misc mailing list
>>>>> varnish-misc at projects.linpro.no
>>>>> http://projects.linpro.no/mailman/listinfo/varnish-misc
>>>> <pmap.txt>
>>>
>> <pmap.txt>
>




More information about the varnish-misc mailing list