From riccardo.brunetti at host.it Tue Jul 11 11:13:01 2023 From: riccardo.brunetti at host.it (Riccardo Brunetti) Date: Tue, 11 Jul 2023 11:13:01 +0000 Subject: Varnish for multiple magento sites Message-ID: <1d56f84d-0e23-b8a6-faed-064dec06f3e0@host.it> Hello. I'm new to varnish and I have a question concerning how to manage multiple sites using the same varnish cache frontend. More specifically, I need to setup a single varnish cache server for two different Magento2 sites. Looking around I found that it is possible to manage different backends using something like: > if (req.http.host == "somesite") { > > ??????? set req.backend_hint = somebackend; > > ??? } Now, I have two different Magento2 sites and, using the above expression, I can handle the two different backends. The problem is that I can't understand how to handle the PURGE/BAN of the two independently. As far as I understand from the .vcl file that Magento2 itself produces there is nothing inside the "purge" section that specifies which resources must be purged. It seems to me that is site A performs a purge, than also the cache of site B resources will be cleaned. Can you help me with this or point me to some example or tutorials? Thanks a lot Riccardo? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Tue Jul 11 15:09:05 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Tue, 11 Jul 2023 08:09:05 -0700 Subject: Varnish for multiple magento sites In-Reply-To: <1d56f84d-0e23-b8a6-faed-064dec06f3e0@host.it> References: <1d56f84d-0e23-b8a6-faed-064dec06f3e0@host.it> Message-ID: Hi Ricardo, Having your VCL (even anonymized) would help here, otherwise debugging is pretty hard. For the moment, I'm going to assume you are using a variation of https://github.com/magento/magento2/blob/13e54e1b28a5d590ab885bd4df9f58877b549052/app/code/Magento/PageCache/etc/varnish6.vcl and deal in generalities. The way that vcl invalidates content is through bans: https://github.com/magento/magento2/blob/13e54e1b28a5d590ab885bd4df9f58877b549052/app/code/Magento/PageCache/etc/varnish6.vcl#L30-L47 which doesn't need the host header, it's just uses unique tags pushed by the backend in response headers. If it was using the actual purge mechanism, then modifying the host should be sufficient because purge acts on the object found in the cache (and if you can get a hit, you can get purged). Here's a good primer on invalidation: https://docs.varnish-software.com/tutorials/cache-invalidation/ Kind regards, -- Guillaume Quintard On Tue, Jul 11, 2023 at 4:14?AM Riccardo Brunetti wrote: > Hello. > I'm new to varnish and I have a question concerning how to manage multiple > sites using the same varnish cache frontend. > > More specifically, I need to setup a single varnish cache server for two > different Magento2 sites. > > Looking around I found that it is possible to manage different backends > using something like: > > if (req.http.host == "somesite") { > set req.backend_hint = somebackend; > } > > Now, I have two different Magento2 sites and, using the above expression, > I can handle the two different backends. > The problem is that I can't understand how to handle the PURGE/BAN of the > two independently. > > As far as I understand from the .vcl file that Magento2 itself produces > there is nothing inside the "purge" section that specifies which resources > must be purged. > It seems to me that is site A performs a purge, than also the cache of > site B resources will be cleaned. > > Can you help me with this or point me to some example or tutorials? > > Thanks a lot > Riccardo > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From riccardo.brunetti at host.it Wed Jul 12 11:15:08 2023 From: riccardo.brunetti at host.it (Riccardo Brunetti) Date: Wed, 12 Jul 2023 11:15:08 +0000 Subject: Varnish for multiple magento sites In-Reply-To: References: <1d56f84d-0e23-b8a6-faed-064dec06f3e0@host.it> Message-ID: Hello Guillaume. Thanks for your answer. The VCL is actually almost identical to that you mentioned in the link (I'm attaching it without references to names and IP anyway) What somehow worries me is that, if I understand, the ban is performed according to some "X-Magento-Tags". Now, if I look at the output of varnishlog and search for "*Tags*", what I get is: 1) while navigating the site: > .... > > -?? RespHeader???? X-Magento-Tags: NAVIGATIONPRO_MENU_2 > > -?? RespUnset????? X-Magento-Tags: NAVIGATIONPRO_MENU_2 > > -?? RespHeader???? X-Magento-Tags: store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_bottom_strenghts,cms_b > > -?? RespUnset????? X-Magento-Tags: store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_bottom_strenghts,cms_b > > -?? BerespHeader?? X-Magento-Tags: cat_c_595,cat_c_p_595,store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_ > > -?? BerespHeader?? X-Magento-Tags: NAVIGATIONPRO_MENU_2 > > -?? RespHeader???? X-Magento-Tags: NAVIGATIONPRO_MENU_2 > > -?? RespUnset????? X-Magento-Tags: NAVIGATIONPRO_MENU_2 > > -?? RespHeader???? X-Magento-Tags: cat_c_595,cat_c_p_595,store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_ > > -?? RespUnset????? X-Magento-Tags: cat_c_595,cat_c_p_595,store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_ > > ..... 2) when performing a purge (php bin/magento c:f): > ... > > -?? ReqHeader????? X-Magento-Tags-Pattern: .* > > ... In both cases I can't see any specific reference to that particular site. Thanks again. Riccardo 11/07/2023, 17:09 Guillaume Quintard ha scritto: > Hi Ricardo, > > > > Having your VCL (even anonymized) would help here, otherwise debugging is pretty hard. For the moment, I'm going to assume you are using a variation of https://github.com/magento/magento2/blob/13e54e1b28a5d590ab885bd4df9f58877b549052/app/code/Magento/PageCache/etc/varnish6.vcl and deal in generalities. > > > > The way that vcl invalidates content is through bans: https://github.com/magento/magento2/blob/13e54e1b28a5d590ab885bd4df9f58877b549052/app/code/Magento/PageCache/etc/varnish6.vcl#L30-L47 which doesn't need the host header, it's just uses unique tags pushed by the backend in response headers. > > If it was using the actual purge mechanism, then modifying the host should be sufficient because purge acts on the object found in the cache (and if you can get a hit, you can get purged). > > > > Here's a good primer on invalidation: https://docs.varnish-software.com/tutorials/cache-invalidation/ > > > > Kind regards, > > > > > > > > -- > > Guillaume Quintard > > > > > > On Tue, Jul 11, 2023 at 4:14?AM Riccardo Brunetti wrote: > > > Hello. > > > > I'm new to varnish and I have a question concerning how to manage multiple sites using the same varnish cache frontend. > > > > > > > > More specifically, I need to setup a single varnish cache server for two different Magento2 sites. > > > > > > > > Looking around I found that it is possible to manage different backends using something like: > > > > > > > > > if (req.http.host == "somesite") { > > > > > > ??????? set req.backend_hint = somebackend; > > > > > > ??? } > > > > Now, I have two different Magento2 sites and, using the above expression, I can handle the two different backends. > > > > The problem is that I can't understand how to handle the PURGE/BAN of the two independently. > > > > > > > > As far as I understand from the .vcl file that Magento2 itself produces there is nothing inside the "purge" section that specifies which resources must be purged. > > > > It seems to me that is site A performs a purge, than also the cache of site B resources will be cleaned. > > > > > > > > Can you help me with this or point me to some example or tutorials? > > > > > > > > Thanks a lot > > > > Riccardo? > > > > _______________________________________________ > > > > varnish-misc mailing list > > > > varnish-misc at varnish-cache.org > > > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Magento.vcl Type: application/octet-stream Size: 8260 bytes Desc: not available URL: From guillaume.quintard at gmail.com Wed Jul 12 15:27:34 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Wed, 12 Jul 2023 08:27:34 -0700 Subject: Varnish for multiple magento sites In-Reply-To: References: <1d56f84d-0e23-b8a6-faed-064dec06f3e0@host.it> Message-ID: Hi Riccardo, You are right, I assumed that the tags were uuids of some sorts and welp, they're not. The best way would be to ask magento to fix it by including a hash of the host or something in each tag, but that's probably not going to happen any time soon. The next best way is to teach Varnish to be a bit more selective when banning. This is done in three steps. First, we are going to mark the backend response with the host it comes from (maybe Magento2 already does it in some form, in which case you can use that header instead): # add this to the beginning of vcl_backend_response sub vcl_backend_response { set beresp.http.x-host = bereq.http.host; ... Then, we change the ban() calls to only apply to the responses with the right x-host headers: if (req.http.X-Magento-Tags-Pattern) { ban("obj.http.x-host == + req.http.host + " && obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern); } if (req.http.X-Pool) { ban("obj.http.x-host == + req.http.host + "&& obj.http.X-Pool ~ " + req.http.X-Pool); } Lastly, we don't need the x-host header to be exposed to the client, so we strip it at the beginning of vcl_deliver: sub vcl_deliver { unset resp.http.x-host; ... disclaimer: This test is completely untested, and it's early in the morning, so you probably shouldn't trust me too much and you should test this before throwing it in prod. Technically, you *could* use req.http.host directly, but you don't want to because of the ban-lurker and its performance implications. Hope this helps. -- Guillaume Quintard On Wed, Jul 12, 2023 at 4:15?AM Riccardo Brunetti wrote: > Hello Guillaume. > Thanks for your answer. > The VCL is actually almost identical to that you mentioned in the link > (I'm attaching it without references to names and IP anyway) > > What somehow worries me is that, if I understand, the ban is performed > according to some "X-Magento-Tags". > Now, if I look at the output of varnishlog and search for "*Tags*", what I > get is: > > 1) while navigating the site: > > .... > - RespHeader X-Magento-Tags: NAVIGATIONPRO_MENU_2 > - RespUnset X-Magento-Tags: NAVIGATIONPRO_MENU_2 > - RespHeader X-Magento-Tags: > store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_bottom_strenghts,cms_b > - RespUnset X-Magento-Tags: > store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_bottom_strenghts,cms_b > - BerespHeader X-Magento-Tags: > cat_c_595,cat_c_p_595,store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_ > - BerespHeader X-Magento-Tags: NAVIGATIONPRO_MENU_2 > - RespHeader X-Magento-Tags: NAVIGATIONPRO_MENU_2 > - RespUnset X-Magento-Tags: NAVIGATIONPRO_MENU_2 > - RespHeader X-Magento-Tags: > cat_c_595,cat_c_p_595,store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_ > - RespUnset X-Magento-Tags: > cat_c_595,cat_c_p_595,store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_ > ..... > > 2) when performing a purge (php bin/magento c:f): > > ... > - ReqHeader X-Magento-Tags-Pattern: .* > ... > > In both cases I can't see any specific reference to that particular site. > > Thanks again. > Riccardo > > 11/07/2023, 17:09 Guillaume Quintard ha scritto: > > Hi Ricardo, > > Having your VCL (even anonymized) would help here, otherwise debugging is > pretty hard. For the moment, I'm going to assume you are using a variation > of > https://github.com/magento/magento2/blob/13e54e1b28a5d590ab885bd4df9f58877b549052/app/code/Magento/PageCache/etc/varnish6.vcl > and deal in generalities. > > The way that vcl invalidates content is through bans: > https://github.com/magento/magento2/blob/13e54e1b28a5d590ab885bd4df9f58877b549052/app/code/Magento/PageCache/etc/varnish6.vcl#L30-L47 > which doesn't need the host header, it's just uses unique tags pushed by > the backend in response headers. > If it was using the actual purge mechanism, then modifying the host should > be sufficient because purge acts on the object found in the cache (and if > you can get a hit, you can get purged). > > Here's a good primer on invalidation: > https://docs.varnish-software.com/tutorials/cache-invalidation/ > > Kind regards, > > > > -- > Guillaume Quintard > > > On Tue, Jul 11, 2023 at 4:14?AM Riccardo Brunetti < > riccardo.brunetti at host.it> wrote: > >> Hello. >> I'm new to varnish and I have a question concerning how to manage >> multiple sites using the same varnish cache frontend. >> >> More specifically, I need to setup a single varnish cache server for two >> different Magento2 sites. >> >> Looking around I found that it is possible to manage different backends >> using something like: >> >> if (req.http.host == "somesite") { >> set req.backend_hint = somebackend; >> } >> >> Now, I have two different Magento2 sites and, using the above expression, >> I can handle the two different backends. >> The problem is that I can't understand how to handle the PURGE/BAN of the >> two independently. >> >> As far as I understand from the .vcl file that Magento2 itself produces >> there is nothing inside the "purge" section that specifies which resources >> must be purged. >> It seems to me that is site A performs a purge, than also the cache of >> site B resources will be cleaned. >> >> Can you help me with this or point me to some example or tutorials? >> >> Thanks a lot >> Riccardo >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From riccardo.brunetti at host.it Thu Jul 13 06:23:14 2023 From: riccardo.brunetti at host.it (Riccardo Brunetti) Date: Thu, 13 Jul 2023 06:23:14 +0000 Subject: Varnish for multiple magento sites In-Reply-To: References: <1d56f84d-0e23-b8a6-faed-064dec06f3e0@host.it> <8dc51eb0-2145-11ee-a31a-8155b750d4de> Message-ID: <8dd0eb80-a0f7-2069-7cd4-ffedb52caf44@host.it> Hi?Guillaume, I'l give it a try. Thank you very much Riccardo 12/07/2023, 17:27 Guillaume Quintard ha scritto: > Hi Riccardo, > > > > You are right, I assumed that the tags were uuids of some sorts and welp, they're not. > > > > The best way would be to ask magento to fix it by including a hash of the host or something in each tag, but that's probably not going to happen any time soon. > > > > The next best way is to teach Varnish to be a bit more selective when banning. This is done in three steps. > > > > First, we are going to mark the backend response with the host it comes from (maybe Magento2 already does it in some form, in which case you can use that header instead): > > > > # add this to the beginning of vcl_backend_response > > sub vcl_backend_response { > > ??? set beresp.http.x-host = bereq.http.host; > > ??? ... > > > > Then, we change the ban() calls to only apply to the responses with the right x-host headers: > > > > ? ? ? ? if (req.http.X-Magento-Tags-Pattern) { > ? ? ? ? ? ban("obj.http.x-host ==?+ req.http.host + " && obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern); > ? ? ? ? } > ? ? ? ? if (req.http.X-Pool) { > ? ? ? ? ? ban("obj.http.x-host ==?+ req.http.host + "&& obj.http.X-Pool ~ " + req.http.X-Pool); > ? ? ? ? } > > > Lastly, we don't need the x-host header to be exposed to the client, so we strip it at the beginning of vcl_deliver: > > > > sub vcl_deliver { > > ??? unset resp.http.x-host; > > ??? ... > > > > disclaimer: This test is completely untested, and it's early in the morning, so you probably shouldn't trust me too much and you should test this before throwing it in prod. > > > > Technically, you *could* use req.http.host directly, but you don't want to because of the ban-lurker and its performance implications. > > > > Hope this helps. > > > > -- > > Guillaume Quintard > > > > > > On Wed, Jul 12, 2023 at 4:15?AM Riccardo Brunetti wrote: > > > Hello Guillaume. > > > > Thanks for your answer. > > > > The VCL is actually almost identical to that you mentioned in the link (I'm attaching it without references to names and IP anyway) > > > > > > > > What somehow worries me is that, if I understand, the ban is performed according to some "X-Magento-Tags". > > > > Now, if I look at the output of varnishlog and search for "*Tags*", what I get is: > > > > > > > > 1) while navigating the site: > > > > > .... > > > > > > -?? RespHeader???? X-Magento-Tags: NAVIGATIONPRO_MENU_2 > > > > > > -?? RespUnset????? X-Magento-Tags: NAVIGATIONPRO_MENU_2 > > > > > > -?? RespHeader???? X-Magento-Tags: store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_bottom_strenghts,cms_b > > > > > > -?? RespUnset????? X-Magento-Tags: store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_bottom_strenghts,cms_b > > > > > > -?? BerespHeader?? X-Magento-Tags: cat_c_595,cat_c_p_595,store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_ > > > > > > -?? BerespHeader?? X-Magento-Tags: NAVIGATIONPRO_MENU_2 > > > > > > -?? RespHeader???? X-Magento-Tags: NAVIGATIONPRO_MENU_2 > > > > > > -?? RespUnset????? X-Magento-Tags: NAVIGATIONPRO_MENU_2 > > > > > > -?? RespHeader???? X-Magento-Tags: cat_c_595,cat_c_p_595,store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_ > > > > > > -?? RespUnset????? X-Magento-Tags: cat_c_595,cat_c_p_595,store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_footer_payments,cms_b_header_block_custom_links,cms_b_main_bottom_newsletter,cms_b_main_ > > > > > > ..... > > > > 2) when performing a purge (php bin/magento c:f): > > > > > ... > > > > > > -?? ReqHeader????? X-Magento-Tags-Pattern: .* > > > > > > ... > > > > In both cases I can't see any specific reference to that particular site. > > > > > > > > Thanks again. > > > > Riccardo > > > > > > > > 11/07/2023, 17:09 Guillaume Quintard ha scritto: > > > > > Hi Ricardo, > > > > > > > > > > > > Having your VCL (even anonymized) would help here, otherwise debugging is pretty hard. For the moment, I'm going to assume you are using a variation of https://github.com/magento/magento2/blob/13e54e1b28a5d590ab885bd4df9f58877b549052/app/code/Magento/PageCache/etc/varnish6.vcl and deal in generalities. > > > > > > > > > > > > The way that vcl invalidates content is through bans: https://github.com/magento/magento2/blob/13e54e1b28a5d590ab885bd4df9f58877b549052/app/code/Magento/PageCache/etc/varnish6.vcl#L30-L47 which doesn't need the host header, it's just uses unique tags pushed by the backend in response headers. > > > > > > If it was using the actual purge mechanism, then modifying the host should be sufficient because purge acts on the object found in the cache (and if you can get a hit, you can get purged). > > > > > > > > > > > > Here's a good primer on invalidation: https://docs.varnish-software.com/tutorials/cache-invalidation/ > > > > > > > > > > > > Kind regards, > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Guillaume Quintard > > > > > > > > > > > > > > > > > > On Tue, Jul 11, 2023 at 4:14?AM Riccardo Brunetti wrote: > > > > > > > Hello. > > > > > > > > I'm new to varnish and I have a question concerning how to manage multiple sites using the same varnish cache frontend. > > > > > > > > > > > > > > > > More specifically, I need to setup a single varnish cache server for two different Magento2 sites. > > > > > > > > > > > > > > > > Looking around I found that it is possible to manage different backends using something like: > > > > > > > > > > > > > > > > > if (req.http.host == "somesite") { > > > > > > > > > > ??????? set req.backend_hint = somebackend; > > > > > > > > > > ??? } > > > > > > > > Now, I have two different Magento2 sites and, using the above expression, I can handle the two different backends. > > > > > > > > The problem is that I can't understand how to handle the PURGE/BAN of the two independently. > > > > > > > > > > > > > > > > As far as I understand from the .vcl file that Magento2 itself produces there is nothing inside the "purge" section that specifies which resources must be purged. > > > > > > > > It seems to me that is site A performs a purge, than also the cache of site B resources will be cleaned. > > > > > > > > > > > > > > > > Can you help me with this or point me to some example or tutorials? > > > > > > > > > > > > > > > > Thanks a lot > > > > > > > > Riccardo? > > > > > > > > _______________________________________________ > > > > > > > > varnish-misc mailing list > > > > > > > > varnish-misc at varnish-cache.org > > > > > > > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From luca.gervasi at gmail.com Thu Jul 13 09:44:53 2023 From: luca.gervasi at gmail.com (Luca Gervasi) Date: Thu, 13 Jul 2023 11:44:53 +0200 Subject: varnishlog record length Message-ID: Hi, i'm helping to fix some issues with cookies (number, length...) thus I need to dump all the cookies in every request (tied with the host header). I tried with varnishlog but the actual value is limited (in our configuration) to ~ 700 bytes by the vsl_reclen parameter, I suppose. Reading the documentation, that value ranges between 16b and 4084b, which is lower than what I need to dump the full monster cookie (~ 24kb). At the moment I'm using tcpdump in the proxy port, but it is really impractical as I have to grep/awk and group the requests...and I'm not really proud about its manutenablity. Is there a way to dump, for every request that contains the cookie header, such header plus others? (in my case: host)? I was thinking about tapping into varnish functions using eBPF. What do you think? Do you have something already working for my case that I could tamper with? Thanks Luca -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Jul 13 13:06:45 2023 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 13 Jul 2023 13:06:45 +0000 Subject: varnishlog record length In-Reply-To: References: Message-ID: On Thu, Jul 13, 2023 at 9:46?AM Luca Gervasi wrote: > > Hi, > i'm helping to fix some issues with cookies (number, length...) thus I need to dump all the cookies in every request (tied with the host header). I tried with varnishlog but the actual value is limited (in our configuration) to ~ 700 bytes by the vsl_reclen parameter, I suppose. Reading the documentation, that value ranges between 16b and 4084b, which is lower than what I need to dump the full monster cookie (~ 24kb). This limit is actually dynamic and better reflected in the reference manual for vsl_buffer and vsl_reclen: https://varnish-cache.org/docs/7.3/reference/varnishd.html#vsl-buffer https://varnish-cache.org/docs/7.3/reference/varnishd.html#vsl-reclen If you want to increase reclen to 24kB, it first needs to fit inside a VSL buffer. In your case I would recommend bumping vsl_buffer to at least 64kB, and add 64kB to workspace_client and workspace_backend too (the VSL buffer lives on these workspaces). > At the moment I'm using tcpdump in the proxy port, but it is really impractical as I have to grep/awk and group the requests...and I'm not really proud about its manutenablity. > > Is there a way to dump, for every request that contains the cookie header, such header plus others? (in my case: host)? https://varnish-cache.org/docs/7.3/reference/vsl-query.html > I was thinking about tapping into varnish functions using eBPF. What do you think? Do you have something already working for my case that I could tamper with? Try bumping vsl_buffer and vsl_reclen first, eBPF is probably way overkill. Cheers, Dridi From mark.slater at mail.com Fri Jul 14 14:28:16 2023 From: mark.slater at mail.com (Mark Slater) Date: Fri, 14 Jul 2023 15:28:16 +0100 Subject: Conditional requests for cached 404 responses In-Reply-To: References: Message-ID: Hi, I'm running Varnish in front of a back end that has to do some work to determine whether a request should receive a 404 response. However, it can cheaply determine whether a previous 404 is still valid. I see Varnish issuing conditional requests for cached 200 responses, but I haven't managed to achieve the same for cached 404 responses. Here's my sample VCL: vcl 4.1; backend default { .host = "localhost"; .port = "8081"; } sub vcl_backend_response { set beresp.keep = 5m; } I'm testing with canned responses on port 8081 For the working 200 case, I return: HTTP/1.1 200 OK cache-control: max-age=5 etag: "foo" content-length: 13 connection: close Hello, World! When I make requests to Varnish, I see, as expected, a first request to the back end, followed by five seconds of nothing to the back end, because Varnish is responding with its cached copy, followed by a conditional request to the back end: GET / HTTP/1.1 Host: localhost:8080 User-Agent: curl/7.68.0 Accept: */* X-Forwarded-For: 127.0.0.1 Accept-Encoding: gzip X-Varnish: 3 GET / HTTP/1.1 Host: localhost:8080 User-Agent: curl/7.68.0 Accept: */* X-Forwarded-For: 127.0.0.1 Accept-Encoding: gzip If-None-Match: "foo" X-Varnish: 32773 For the failing 404 case, my canned back end responds: HTTP/1.1 404 Not Found cache-control: max-age=5 etag: "foo" content-length: 13 connection: close Hello, World! Now when I make requests to Varnish, I get a cached response for five seconds as before, but when the response goes stale, rather than issuing a conditional request to revalidate it, Varnish is issuing unconditional requests: GET / HTTP/1.1 Host: localhost:8080 User-Agent: curl/7.68.0 Accept: */* X-Forwarded-For: 127.0.0.1 Accept-Encoding: gzip X-Varnish: 3 GET / HTTP/1.1 Host: localhost:8080 User-Agent: curl/7.68.0 Accept: */* X-Forwarded-For: 127.0.0.1 Accept-Encoding: gzip X-Varnish: 32771 Is that something I can adjust with configuration? If it's relevant, I'm running: Debug: Version: varnish-6.2.1 revision 9f8588e4ab785244e06c3446fe09bf9db5dd8753 Debug: Platform: Linux,5.4.0-153-generic,x86_64,-jnone,-sdefault,-sdefault,-hcritbit Incidentally, 200 responses with content-length 0 also seem to exhibit this behaviour. Thanks in advance, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Sat Jul 15 05:07:34 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Fri, 14 Jul 2023 22:07:34 -0700 Subject: Conditional requests for cached 404 responses In-Reply-To: References: Message-ID: Hi Mark, You are correct: https://github.com/varnishcache/varnish-cache/blob/varnish-7.3.0/bin/varnishd/cache/cache_fetch.c#L699-L703 We only set the OF_IMSCAND flag (that we use to say that we can conditional download) if: - the object is not a Hit-For-Miss (HFM) - if the status is 200 - we either have a convincing Last-modified, or an Etag header You can also test it with this VTC: varnishtest "conditional requests" server s1 { rxreq txresp -status 200 -hdr "ETag: 1234" -hdr "Last-Modified: Wed, 21 Oct 2015 07:28:00 GMT" -body "dad" rxreq expect req.http.if-none-match == "1234" expect req.http.if-modified-since == "Wed, 21 Oct 2015 07:28:00 GMT" txresp } -start varnish v1 -vcl+backend { sub vcl_backend_response { set beresp.ttl = 0.1s; set beresp.grace = 0s; set beresp.keep = 1y; return (deliver); } } -start client c1 { txreq rxresp delay 0.2 txreq rxresp } -run Change the 200 to a 404 and the test will now fail. I quickly skimmed the HTTP spec and see no reason for us to actually check the status, but I'm sure somebody closer to the code will pop up to provide some light on the topic. Cheers, -- Guillaume Quintard On Fri, Jul 14, 2023 at 7:30?AM Mark Slater wrote: > Hi, > > I'm running Varnish in front of a back end that has to do some work to > determine whether a request should receive a 404 response. However, it can > cheaply determine whether a previous 404 is still valid. > > I see Varnish issuing conditional requests for cached 200 responses, but I > haven't managed to achieve the same for cached 404 responses. > > Here's my sample VCL: > > > vcl 4.1; > > backend default { > .host = "localhost"; > .port = "8081"; > } > > sub vcl_backend_response { > set beresp.keep = 5m; > } > > > I'm testing with canned responses on port 8081 For the working 200 case, > I return: > > > HTTP/1.1 200 OK > cache-control: max-age=5 > etag: "foo" > content-length: 13 > connection: close > > Hello, World! > > > When I make requests to Varnish, I see, as expected, a first request to > the back end, followed by five seconds of nothing to the back end, because > Varnish is responding with its cached copy, followed by a conditional > request to the back end: > > > GET / HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.68.0 > Accept: */* > X-Forwarded-For: 127.0.0.1 > Accept-Encoding: gzip > X-Varnish: 3 > > GET / HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.68.0 > Accept: */* > X-Forwarded-For: 127.0.0.1 > Accept-Encoding: gzip > If-None-Match: "foo" > X-Varnish: 32773 > > > For the failing 404 case, my canned back end responds: > > > HTTP/1.1 404 Not Found > cache-control: max-age=5 > etag: "foo" > content-length: 13 > connection: close > > Hello, World! > > > Now when I make requests to Varnish, I get a cached response for five > seconds as before, but when the response goes stale, rather than issuing a > conditional request to revalidate it, Varnish is issuing unconditional > requests: > > > GET / HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.68.0 > Accept: */* > X-Forwarded-For: 127.0.0.1 > Accept-Encoding: gzip > X-Varnish: 3 > > GET / HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.68.0 > Accept: */* > X-Forwarded-For: 127.0.0.1 > Accept-Encoding: gzip > X-Varnish: 32771 > > > Is that something I can adjust with configuration? If it's relevant, I'm > running: > > Debug: Version: varnish-6.2.1 revision > 9f8588e4ab785244e06c3446fe09bf9db5dd8753 > Debug: Platform: > Linux,5.4.0-153-generic,x86_64,-jnone,-sdefault,-sdefault,-hcritbit > > Incidentally, 200 responses with content-length 0 also seem to exhibit > this behaviour. > > Thanks in advance, > > Mark > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Sat Jul 15 09:30:14 2023 From: dridi at varni.sh (Dridi Boukelmoune) Date: Sat, 15 Jul 2023 09:30:14 +0000 Subject: Conditional requests for cached 404 responses In-Reply-To: References: Message-ID: On Sat, Jul 15, 2023 at 5:09?AM Guillaume Quintard wrote: > > Hi Mark, > > You are correct: https://github.com/varnishcache/varnish-cache/blob/varnish-7.3.0/bin/varnishd/cache/cache_fetch.c#L699-L703 > > We only set the OF_IMSCAND flag (that we use to say that we can conditional download) if: > - the object is not a Hit-For-Miss (HFM) > - if the status is 200 > - we either have a convincing Last-modified, or an Etag header > > You can also test it with this VTC: > varnishtest "conditional requests" > > server s1 { > rxreq > txresp -status 200 -hdr "ETag: 1234" -hdr "Last-Modified: Wed, 21 Oct 2015 07:28:00 GMT" -body "dad" > > rxreq > expect req.http.if-none-match == "1234" > expect req.http.if-modified-since == "Wed, 21 Oct 2015 07:28:00 GMT" > txresp > } -start > > varnish v1 -vcl+backend { > sub vcl_backend_response { > set beresp.ttl = 0.1s; > set beresp.grace = 0s; > set beresp.keep = 1y; > return (deliver); > } > } -start > > client c1 { > txreq > rxresp > > delay 0.2 > > txreq > rxresp > } -run > > Change the 200 to a 404 and the test will now fail. > > I quickly skimmed the HTTP spec and see no reason for us to actually check the status, but I'm sure somebody closer to the code will pop up to provide some light on the topic. I was writing a very similar test case but haven't spent time on the RFC but their is also the concern of not breaking existing setups. Similarly to how how we handle request cookies with extra caution, we could imagine something like this in the built-in VCL for vcl_backend_request: sub bereq_revalidate { if (bereq.uncacheable || obj_stale.status == 200) { return; } unset bereq.http.If-None-Match; unset bereq.http.If-Modified-Since; } Then enabling revalidation for 404 is only a matter of adding this: sub bereq_revalidate { if (obj_stale.status == 404) { return; } } We briefly discussed accessing the stale object during last VDD and extensively discussed other aspects of (re)validation. https://github.com/varnishcache/varnish-cache/wiki/VDD23Q1#compliance Right now the workaround would be to store the actual beresp.status in a header when you wish to enable revalidation, change it to 200 and restore the original in vcl_deliver. Dridi From guillaume.quintard at gmail.com Sun Jul 16 00:11:36 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Sat, 15 Jul 2023 17:11:36 -0700 Subject: Making vmods more approachable. Message-ID: TL;DR: here's a bunch of examples building vmods from source so you can get better at it: https://github.com/varnish/docker-varnish/tree/master/vmod-examples Varnish is great, stable and very versatile, however part of that versatility comes from vmods, and those tend to not be packaged by distributions. Some are, of course, but they are few and far between, mostly relying on the judgement (for selection) and good will (for actual packaging) of a few packagers. And this obviously limits access to that versatility, hurting Varnish's reach. I do package a couple of vmods for a couple of distributions, but the benefits are fairly limited because fragmentation sucks. So, truly, the best way to distribute vmods is through source code. Sadly, it still seems like people are afraid of compiling C code (can't blame them, it's a scary new experience), and efforts like https://github.com/varnish/toolbox/tree/master/install-vmod and https://github.com/xcir/vmod-packager to help install/package do make things a bit easier, but we are not there yet. With that context in mind, I've created a new vmod-examples/ directory in https://github.com/varnish/docker-varnish/ which shows how to build 14 vmods (for now) on a docker container. Hopefully, we can reach a few more people, and reassure them, showing that compiling vmods is easy and viable. We do this by: - putting the examples in a central, fairly visible location - using docker as a base, since the containers are easy to build/trash, I hope it'll encourage people to experiment more - providing the actual instructions to build for a specific distributions, rather that pointing at libraries that your system may or may not package, and possibly under a weird, badly discoverable name - targeting the latest Varnish, guaranteeing users will at least have something that compiles on the first try without losing time tracking the right branch/commit/tarball. Hopefully, we can grow that list and make more vmods accessible to more users. If you want more vmods in there, please let me know, either here or in the discord channel, I'll be happy to push more of them as long as they are maintained (or at least compile) and that there's no clear better alternative. Lastly, I really need to thank the maintainers of the the featured vmods, without them there wouldn;t be anything to showcase: - UPLEX (https://code.uplex.de/uplex-varnish and https://gitlab.com/uplex) - Carlos Abalde (https://github.com/carlosabalde) - Shohei Tanaka (https://github.com/xcir/) - otto-de (https://github.com/otto-de) - varnishcache-friends (https://github.com/varnishcache-friends) That's it for me, as usual, comments, questions and PRs are more than welcome! -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Sun Jul 16 23:21:05 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Sun, 16 Jul 2023 16:21:05 -0700 Subject: IRC #varnish room is now invite-only? Message-ID: Hi team, Somebody popped up this morning on the discord channel saying that the #varnish room on IRC is invite-only, I just checked and it does seem to be the case. Is that on purpose? Cheers, -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From karim.ayari at univ-lyon1.fr Mon Jul 17 06:40:46 2023 From: karim.ayari at univ-lyon1.fr (AYARI KARIM) Date: Mon, 17 Jul 2023 06:40:46 +0000 Subject: IRC #varnish room is now invite-only? In-Reply-To: References: Message-ID: Hi Guillaume, i tried and i confirmed that the channel is on Invite only Karim ________________________________ De : varnish-misc de la part de Guillaume Quintard Envoy? : lundi 17 juillet 2023 01:21 ? : varnish-misc at varnish-cache.org Objet : IRC #varnish room is now invite-only? Hi team, Somebody popped up this morning on the discord channel saying that the #varnish room on IRC is invite-only, I just checked and it does seem to be the case. Is that on purpose? Cheers, -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: S?lection_031.png Type: image/png Size: 9608 bytes Desc: S?lection_031.png URL: From phk at phk.freebsd.dk Mon Jul 17 07:09:52 2023 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 17 Jul 2023 07:09:52 +0000 Subject: IRC #varnish room is now invite-only? In-Reply-To: References: Message-ID: <202307170709.36H79qsw010286@critter.freebsd.dk> -------- Guillaume Quintard writes: > Hi team, > > Somebody popped up this morning on the discord channel saying that the > #varnish room on IRC is invite-only, I just checked and it does seem to be > the case. > > Is that on purpose? Yes, the "irc.supernets.org" script-kiddies had their annual superbowl stupidity and spammed every unprotected IRC channel they could find in the entire world with advertisements of their mental deficiencies. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From mark.slater at mail.com Fri Jul 21 15:08:46 2023 From: mark.slater at mail.com (Mark Slater) Date: Fri, 21 Jul 2023 16:08:46 +0100 Subject: Conditional requests for cached 404 responses In-Reply-To: References: Message-ID: Thanks for this. I changed my sample VCL to be the following, and it seems to work as intended: vcl 4.1; import std; backend default { .host = "localhost"; .port = "8081"; } sub vcl_backend_response { set beresp.keep = 5m; set beresp.http.actual-status = beresp.status; set beresp.status = 200; } sub vcl_deliver { set resp.status = std.integer(resp.http.actual-status, 0); unset resp.http.actual-status; } However, on further investigation, I'm not sure it's a good idea. Here's the commit that introduced the current behaviour: https://github.com/varnishcache/varnish-cache/commit/e99b5cfd886ec38a7f883e23ba516063cf4c16f8. The commit changes it from attempting conditional download of any cached object, to only those with status 200. I read through the RFC, and it appears it mandates this behaviour: https://datatracker.ietf.org/doc/html/rfc7232#section-4.1, specifically: > The 304 (Not Modified) status code indicates that a conditional GET or HEAD request has been received and would have resulted in a 200 (OK) response if it were not for the fact that the condition evaluated to false. I.e. a conditional request that would have resulted in a 404 *cannot* respond with 304, so the cached 404 cannot be refreshed. This seems a bit of a shame, but I can't claim to know the RFC well enough to know if there's a strong reason for it to be this way. Regards, Mark On Sat, 15 Jul 2023 at 10:30, Dridi Boukelmoune wrote: > On Sat, Jul 15, 2023 at 5:09?AM Guillaume Quintard > wrote: > > > > Hi Mark, > > > > You are correct: > https://github.com/varnishcache/varnish-cache/blob/varnish-7.3.0/bin/varnishd/cache/cache_fetch.c#L699-L703 > > > > We only set the OF_IMSCAND flag (that we use to say that we can > conditional download) if: > > - the object is not a Hit-For-Miss (HFM) > > - if the status is 200 > > - we either have a convincing Last-modified, or an Etag header > > > > You can also test it with this VTC: > > varnishtest "conditional requests" > > > > server s1 { > > rxreq > > txresp -status 200 -hdr "ETag: 1234" -hdr "Last-Modified: Wed, > 21 Oct 2015 07:28:00 GMT" -body "dad" > > > > rxreq > > expect req.http.if-none-match == "1234" > > expect req.http.if-modified-since == "Wed, 21 Oct 2015 07:28:00 > GMT" > > txresp > > } -start > > > > varnish v1 -vcl+backend { > > sub vcl_backend_response { > > set beresp.ttl = 0.1s; > > set beresp.grace = 0s; > > set beresp.keep = 1y; > > return (deliver); > > } > > } -start > > > > client c1 { > > txreq > > rxresp > > > > delay 0.2 > > > > txreq > > rxresp > > } -run > > > > Change the 200 to a 404 and the test will now fail. > > > > I quickly skimmed the HTTP spec and see no reason for us to actually > check the status, but I'm sure somebody closer to the code will pop up to > provide some light on the topic. > > I was writing a very similar test case but haven't spent time on the > RFC but their is also the concern of not breaking existing setups. > > Similarly to how how we handle request cookies with extra caution, we > could imagine something like this in the built-in VCL for > vcl_backend_request: > > sub bereq_revalidate { > if (bereq.uncacheable || obj_stale.status == 200) { > return; > } > unset bereq.http.If-None-Match; > unset bereq.http.If-Modified-Since; > } > > Then enabling revalidation for 404 is only a matter of adding this: > > sub bereq_revalidate { > if (obj_stale.status == 404) { > return; > } > } > > We briefly discussed accessing the stale object during last VDD and > extensively discussed other aspects of (re)validation. > > https://github.com/varnishcache/varnish-cache/wiki/VDD23Q1#compliance > > Right now the workaround would be to store the actual beresp.status in > a header when you wish to enable revalidation, change it to 200 and > restore the original in vcl_deliver. > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: