Varnish, long lived cache and purge on change

[Rob S]

phk and other deep Varnish developers,

Do you think it'd ever be viable to have a sort of process that goes 
through the tail of the purge queue and applies the purges then deletes 
them from the queue?  If so, how much work would it be to implement?  
There are a fair number of us who would really appreciate something like 
this, and I'm sure would make a contribution if someone was to implement 
something.

Thanks,


Rob

Karl Pietri wrote:
> Hey Ken, =)
>     Yeah this is what i was afraid of.  I think we have a work around 
> by normalizing the hash key to a few select things we want to support 
> and on change setting the ttl of those objects to 0.  This would avoid 
> using the url.purge.  All of our urls in this case are pretty, and not 
> images.
>
> Thanks for the great info and sorry about the 4th thread on the 
> subject, i did not search thoroughly enough in the archives. 
>
> -Karl
>
> On Tue, Aug 18, 2009 at 4:34 PM, Ken Brownfield <kb+varnish at slide.com 
> <mailto:kb%2Bvarnish at slide.com>> wrote:
>
>     Hey Karl. :-)
>
>     The implementation of purge in Varnish is really a queue of
>     refcounted ban objects.  Every image hit is compared to the ban
>     list to see if the object in cache should be reloaded from a backend.
>
>     If you have purge_dups off, /every/ request to Varnish will regex
>     against every single ban in the list.  If you have purge_dups on,
>     it will at least not compare against duplicate bans.
>
>     However, a ban that has been created will stay around until
>     /every/ object that was in the cache at the time of that ban has
>     been re-requested, dupe or no.  If you have lots of content,
>     especially content that may not be accessed very often, the ban
>     list can become enormous.  Even with purge_dups, duplicate ban
>     entries remain in memory.  And the bans are only freed from RAM
>     when their refcount hits 0 /AND/ they're at the very tail end of
>     the ban queue.
>
>     Because of the implementation, there's no clear way around this
>     AFAICT.
>
>     You can get a list of bans with the "purge.list" management
>     command, but if it's more than ~2400 long you'll need to use
>     netcat to get the list.  Also, purged dups will NOT show up in
>     this list, even though they're sitting on RAM.  I have a trivial
>     patch that will make dups show up in purge.list if you'd like to
>     get an idea of how many bans you have.
>
>     The implementation is actually really clever, IMHO, especially
>     with regard to how it avoids locks, and there's really no other
>     scalable way to implement a regex purge that I've been able to
>     dream up.
>
>     The only memory-reducing option within the existing implementation
>     is to actually delete/free duplicate bans from the list, and to
>     delete/free bans when an object hit causes the associated ban's
>     refcount to hit 0.  However, this requires all access to the ban
>     list to be locked, which is likely a significant performance hit.
>      I've written this patch, and it works, but I haven't put
>     significant load on it.
>
>     I'm not sure if Varnish supports non-regex/non-wildcard purges?
>      This would at least not have to go through the ban system,  but
>     obviously it doesn't work for arbitrary path purges.
>
>     We version our static content, which avoids cache thrash and this
>     purge side-effect.  This is very easy if you have a central
>     URL-generation system in code (templates, ajax, etc), but probably
>     more problematic in situations where the URL needs to be "pretty".
>
>     Ken
>
>     On Aug 18, 2009, at 4:06 PM, Karl Pietri wrote:
>
>>     Hello everyone,
>>         Recently we decided that our primary page that everyone views
>>     doesn't really change all that often.  In fact it changes very
>>     rarely except for the stats counters (views, downloads, etc).  So
>>     we decided that we wanted to store everything in varnish for a
>>     super long time (and tell the client its not cacheable or
>>     cacheable for a very short amount of time), flush the page from
>>     varnish when it truly changes and have a very fast ajax call to
>>     update the stats.  This worked great for about 2 days.   Then we
>>     ran out of ram and varnish started causing a ton of swap activity
>>     and it increased the response times of everything on the site to
>>     unusable.
>>
>>     After poking about i seem to have found the culprit.  When you
>>     use url.purge it seems to keep a record of that and check every
>>     object as it is fetched to see if it was purged or not.  To test
>>     this i set a script to purge a lot of stuff and got the same
>>     problem to happen.
>>
>>
>>     from varnishstat -1
>>
>>      n_purge                236369          .   N total active purges
>>     n_purge_add            236388         2.31 N new purges added
>>     n_purge_retire             19         0.00 N old purges deleted
>>     n_purge_obj_test      1651452        16.12 N objects tested
>>     n_purge_re_test    5052057513     49316.27 N regexps tested against
>>     n_purge_dups                0         0.00 N duplicate purges removed
>>
>>     each uptick is when i add 100k new purge records.  you can see
>>     what will happen soon.
>>
>>     http://wrenchies.net/~karl/Automator/ScreenShots/20090818160328.png
>>     <http://wrenchies.net/%7Ekarl/Automator/ScreenShots/20090818160328.png>
>>
>>     We really want to take advantage of this style of essentially
>>     having static html served by varnish and flush it out when it
>>     changes.
>>
>>     Does any one have advice on how to do this?
>>
>>     Originally we had implemented this using the vlc to set the ttl
>>     to 0 but with all the combinations of accept-encoding that are
>>     possible we were getting many things not being purged from the cache.
>>
>>     Another thought would be to refetch the page on change instead of
>>     purging it but that has the same problem with accept-encoding.
>>
>>     after 36 hours between the 2 machines we have collected 1.3M
>>     objects in the cache and have not even come close to running out
>>     of space.  We would actually like to increase our ttl for the
>>     cached objects even longer.
>>
>>     I hope someone can help me out here.
>>
>>     -Karl Pietri
>>     _______________________________________________
>>     varnish-misc mailing list
>>     varnish-misc at projects.linpro.no
>>     <mailto:varnish-misc at projects.linpro.no>
>>     http://projects.linpro.no/mailman/listinfo/varnish-misc
>
>