varnish-misc Digest, Vol 68, Issue 9
Graham Lyons
graham.lyons at bbc.co.uk
Mon Nov 7 10:06:01 CET 2011
We did some experimentation with saint mode before deciding not to turn it on because we found that we'd be opening ourselves up to exactly this issue.
Once the saint mode threshold is reached for a backend then that backend won't be contacted for anything at all and Varnish will return a 503. It can't return the stale content if it's not in the cache.
On 5 Nov 2011, at 11:00, varnish-misc-request at varnish-cache.org wrote:
> Send varnish-misc mailing list submissions to
> varnish-misc at varnish-cache.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> or, via email, send a message with subject or body 'help' to
> varnish-misc-request at varnish-cache.org
>
> You can reach the person managing the list at
> varnish-misc-owner at varnish-cache.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of varnish-misc digest..."
>
>
> Today's Topics:
>
> 1. Re: Intermittent 503's from Varnish (Lei Wang)
> 2. Re: Intermittent 503's from Varnish (AD)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 4 Nov 2011 13:26:50 -0700
> From: Lei Wang <leiwang at rhapsody.com>
> To: "Gresens, August" <AGresens at scholastic.com>
> Cc: varnish-misc at varnish-cache.org
> Subject: Re: Intermittent 503's from Varnish
> Message-ID:
> <CAD_kORcxrk3JrTANuP8_FGXKt6rWmhHcxQXnPA2wOLWjcEt0Ww at mail.gmail.com>
> Content-Type: text/plain; charset="windows-1252"
>
> Hi August,
>
> Thank for the info. I will definitely try that today.
>
> Have a nice day.
>
> Lei
>
> On Fri, Nov 4, 2011 at 12:48 PM, Gresens, August <AGresens at scholastic.com>wrote:
>
>> We were able to resolve this. Our Varnish instance is pointing a load
>> balanced pool of backend servers (a vip on a hardware load balancer), so
>> to Varnish there is only one backend. We had a situation in which certain
>> images were returning 500 errors from the backend. Because saint mode was
>> enabled, it invalidated the backend (the only back end) after a certain
>> number of failures within a particular window. We disabled saint mode and
>> this fixed this issue. Best, August****
>>
>> ** **
>> ------------------------------
>>
>> *From:* Lei Wang [mailto:leiwang at rhapsody.com]
>> *Sent:* Friday, November 04, 2011 3:40 PM
>> *To:* Rob S
>> *Cc:* Gresens, August; varnish-misc at varnish-cache.org
>> *Subject:* Re: Intermittent 503's from Varnish****
>>
>> ** **
>>
>> I had the same problem. Please help!
>>
>> /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f
>> /etc/varnish/varnish80.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u
>> varnish -g varnish -i varnish81 -S /etc/varnish/secret -p
>> thread_pool_add_delay 2 -p thread_pools 1 -p thread_pool_min 800 -p
>> thread_pool_max 2000 -p session_linger 50 -p sess_workspace 4096 -n
>> varnish80 -s malloc,6.5G
>>
>> configuration related to this problem:
>> backend eadrm {
>> .host = "drm-ea.xxxxxxxx.com";
>> .port = "80";
>> .connect_timeout = 40s;
>> .first_byte_timeout = 80s;
>> .between_bytes_timeout = 40s;
>> }
>>
>> } elsif (req.http.host ~ "drm-ea") {
>> unset beresp.http.set-cookie;
>> set beresp.ttl = 86400s;
>> log "Adjusting EA-DRM TTL to 86400 seconds";
>>
>> Thanks,
>>
>> Lei****
>>
>> On Fri, Nov 4, 2011 at 11:52 AM, Rob S <rtshilston at gmail.com> wrote:****
>>
>> Can you send the varnishlog for one of these failing requests? Is
>> this a POST or a GET request? Are you using default VCL, or can you
>> send the VCL you are using?
>>
>> Rob****
>>
>> On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August <AGresens at scholastic.com>
>> wrote:
>>> Hello
>>>
>>>
>>>
>>> We?re having a very nasty production issue. We?re getting intermittent
>> 503
>>> from Varnish in front of a healthy backend.
>>>
>>>
>>>
>>> This seems to happen on cache-miss, but we?re not seeing requests make
>> it to
>>> the backend. Varnish immediately returns the 503. After the first 503,
>> the
>>> page can usually be loaded by requesting it a few more times. This seems
>> to
>>> indicate that there is some condition within Varnish that prevents it
>> from
>>> hitting the backend that clears up quickly.
>>>
>>>
>>>
>>> Has anyone seen this issue before? Is there anything we can try?
>>>
>>>
>>>
>>> This is our startup config:
>>>
>>>
>>>
>>> DAEMON_OPTS="-a :80 \
>>>
>>> -T localhost:6082 \
>>>
>>> -f /etc/varnish/default.vcl \
>>>
>>> -S /etc/varnish/secret \
>>>
>>> -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \
>>>
>>> -p http_headers=128 \
>>>
>>> -p sess_workspace=262144 \
>>>
>>> -p connect_timeout=60 \
>>>
>>> -p max_restarts=6 \
>>>
>>> -p log_hashstring=on \
>>>
>>> -p cli_timeout=60 \
>>>
>>> -t 0"
>>>
>>>
>>>
>>> We are on version 2.0.5
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> August
>>>
>>>
>>>
>>>
>>> ****
>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc****
>>
>> ** **
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20111104/48abd99e/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 4 Nov 2011 16:33:51 -0400
> From: AD <straightflush at gmail.com>
> To: Lei Wang <leiwang at rhapsody.com>
> Cc: varnish-misc at varnish-cache.org
> Subject: Re: Intermittent 503's from Varnish
> Message-ID:
> <CANZWjw4jviRFXhLftWu2FwBcpPKZ2yBi1Cwrb4xQCHHTbRVF4w at mail.gmail.com>
> Content-Type: text/plain; charset="windows-1252"
>
> Shouldnt saint mode return a 200 OK (stale cache item) if the backend fails
> and you are within the "Saint Window" for that object? I dont think
> turning off saint mode is going to correct this issue.
>
> On Fri, Nov 4, 2011 at 4:26 PM, Lei Wang <leiwang at rhapsody.com> wrote:
>
>> Hi August,
>>
>> Thank for the info. I will definitely try that today.
>>
>> Have a nice day.
>>
>> Lei
>>
>>
>> On Fri, Nov 4, 2011 at 12:48 PM, Gresens, August <AGresens at scholastic.com>wrote:
>>
>>> We were able to resolve this. Our Varnish instance is pointing a load
>>> balanced pool of backend servers (a vip on a hardware load balancer), so
>>> to Varnish there is only one backend. We had a situation in which certain
>>> images were returning 500 errors from the backend. Because saint mode was
>>> enabled, it invalidated the backend (the only back end) after a certain
>>> number of failures within a particular window. We disabled saint mode and
>>> this fixed this issue. Best, August****
>>>
>>> ** **
>>> ------------------------------
>>>
>>> *From:* Lei Wang [mailto:leiwang at rhapsody.com]
>>> *Sent:* Friday, November 04, 2011 3:40 PM
>>> *To:* Rob S
>>> *Cc:* Gresens, August; varnish-misc at varnish-cache.org
>>> *Subject:* Re: Intermittent 503's from Varnish****
>>>
>>> ** **
>>>
>>> I had the same problem. Please help!
>>>
>>> /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f
>>> /etc/varnish/varnish80.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u
>>> varnish -g varnish -i varnish81 -S /etc/varnish/secret -p
>>> thread_pool_add_delay 2 -p thread_pools 1 -p thread_pool_min 800 -p
>>> thread_pool_max 2000 -p session_linger 50 -p sess_workspace 4096 -n
>>> varnish80 -s malloc,6.5G
>>>
>>> configuration related to this problem:
>>> backend eadrm {
>>> .host = "drm-ea.xxxxxxxx.com";
>>> .port = "80";
>>> .connect_timeout = 40s;
>>> .first_byte_timeout = 80s;
>>> .between_bytes_timeout = 40s;
>>> }
>>>
>>> } elsif (req.http.host ~ "drm-ea") {
>>> unset beresp.http.set-cookie;
>>> set beresp.ttl = 86400s;
>>> log "Adjusting EA-DRM TTL to 86400 seconds";
>>>
>>> Thanks,
>>>
>>> Lei****
>>>
>>> On Fri, Nov 4, 2011 at 11:52 AM, Rob S <rtshilston at gmail.com> wrote:****
>>>
>>> Can you send the varnishlog for one of these failing requests? Is
>>> this a POST or a GET request? Are you using default VCL, or can you
>>> send the VCL you are using?
>>>
>>> Rob****
>>>
>>> On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August <AGresens at scholastic.com>
>>> wrote:
>>>> Hello
>>>>
>>>>
>>>>
>>>> We?re having a very nasty production issue. We?re getting intermittent
>>> 503
>>>> from Varnish in front of a healthy backend.
>>>>
>>>>
>>>>
>>>> This seems to happen on cache-miss, but we?re not seeing requests make
>>> it to
>>>> the backend. Varnish immediately returns the 503. After the first 503,
>>> the
>>>> page can usually be loaded by requesting it a few more times. This
>>> seems to
>>>> indicate that there is some condition within Varnish that prevents it
>>> from
>>>> hitting the backend that clears up quickly.
>>>>
>>>>
>>>>
>>>> Has anyone seen this issue before? Is there anything we can try?
>>>>
>>>>
>>>>
>>>> This is our startup config:
>>>>
>>>>
>>>>
>>>> DAEMON_OPTS="-a :80 \
>>>>
>>>> -T localhost:6082 \
>>>>
>>>> -f /etc/varnish/default.vcl \
>>>>
>>>> -S /etc/varnish/secret \
>>>>
>>>> -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \
>>>>
>>>> -p http_headers=128 \
>>>>
>>>> -p sess_workspace=262144 \
>>>>
>>>> -p connect_timeout=60 \
>>>>
>>>> -p max_restarts=6 \
>>>>
>>>> -p log_hashstring=on \
>>>>
>>>> -p cli_timeout=60 \
>>>>
>>>> -t 0"
>>>>
>>>>
>>>>
>>>> We are on version 2.0.5
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>> August
>>>>
>>>>
>>>>
>>>>
>>>> ****
>>>
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc at varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc****
>>>
>>> ** **
>>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20111104/8fe01fe8/attachment-0001.html>
>
> ------------------------------
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> End of varnish-misc Digest, Vol 68, Issue 9
> *******************************************
http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
More information about the varnish-misc
mailing list