varnish and mp4 files
Guillaume Quintard
guillaume.quintard at gmail.com
Fri Oct 27 16:24:44 UTC 2023
Hi,
Check the varnishncsa logs to see if something fishy stands out, or even
just the backend logs.
Notably, try
varnishncsa -q 'not VCL_call eq PIPE'
That will give you all the transactions that go through vcl_pipe{} (more
log tips here: https://docs.varnish-software.com/tutorials/vsl-query/)
If that doesn't work we can try to filter big files and whatnot.
--
Guillaume Quintard
On Thu, Oct 26, 2023 at 11:47 PM AYARI KARIM <karim.ayari at univ-lyon1.fr>
wrote:
> Hello,
>
>
> I'm coming back to this subject because after a moment of calm my problem
> of memory leak and OOM killer with videos is back :(
>
> we have videos hosted on Moodle behind varnish which are themselves behind
> haproxy.
>
>
> version varnish
> 6.2.1-2ubuntu0.2 amd64
>
>
> this expression seems to work
>
>
> *if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {*
> * std.log("ispiped:true");*
> * return (pipe);*
> *}*
>
>
> mp4 files are piped :
>
>
>
>
>
>
> *- VCL_call RECV - VCL_Log ispiped:true -
> VCL_return pipe - VCL_call HASH - VCL_return lookup*
>
> first graph is memory usage (total of 16GB), the second one requests
> fetched, we can see 2 restarts after OOM killing at 16h25 and 18h20
>
>
>
>
> I don't see any other file extension to add in the expression
>
>
> thank you for your help
>
>
> ---
> Karim Ayari
>
> ------------------------------
> *De :* varnish-misc <varnish-misc-bounces+karim.ayari=
> univ-lyon1.fr at varnish-cache.org> de la part de Karim Ayari <
> karim.ayari at univ-lyon1.fr>
> *Envoyé :* mardi 21 février 2023 09:05
> *À :* Guillaume Quintard
> *Cc :* varnish-misc at varnish-cache.org
> *Objet :* Re: varnish and mp4 files
>
>
> sorry I thought I noted it in my previous email.
> here are the good lines for using pipe :
>
>
> *if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {*
> * std.log("ispiped:true");*
> * return (pipe);*
> *}*
>
> thank you!
>
> Karim.
>
>
>
> Le 20/02/2023 à 21:50, Guillaume Quintard a écrit :
>
> Looks like Rainer replied directly to you and not to the list, would you
> mind sharing/highlighting the fix for people having the same issue?
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari <karim.ayari at univ-lyon1.fr>
> wrote:
>
>> thank you both for your replies. I forgot the most important thing: the
>> varnish server has 16 GB of ram and the cache is 1 GB.
>>
>> the cache never seems full
>>
>>
>> the command used : */usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T
>> localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s
>> malloc,1g* *-p http_max_hdr=96*
>>
>> I had tried using pipe it didn't work, but Rainer's lines works fine.
>>
>> this solution should suffice because the videos will soon have to be
>> hosted on our video platform.
>>
>>
>> my vcl (file found on github) :
>>
>>
>>
>> *.... *
>>
>> *sub vcl_recv {*
>>
>> * # Keep client IP*
>> * if (req.restarts == 0) {*
>> * if (req.http.x-forwarded-for) {*
>> * set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ",
>> " + client.ip;*
>> * } else {*
>> * unset req.http.X-Forwarded-For;*
>> * set req.http.X-Forwarded-For = client.ip;*
>> * }*
>> * }*
>>
>> * if (req.http.X-Real-IP) {*
>> * set req.http.X-Forwarded-For = req.http.X-Real-IP;*
>> * } else {*
>> * set req.http.X-Forwarded-For = client.ip;*
>> * }*
>>
>> * # Only deal with "normal" types*
>> * if (req.method != "GET" &&*
>> * req.method != "HEAD" &&*
>> * req.method != "PUT" &&*
>> * req.method != "POST" &&*
>> * req.method != "TRACE" &&*
>> * req.method != "OPTIONS" &&*
>> * req.method != "PATCH" &&*
>> * req.method != "DELETE") {*
>> * /* Non-RFC2616 or CONNECT which is weird. */*
>> * /*Why send the packet upstream, while the visitor is using a
>> non-valid HTTP method? */*
>> * return (synth(404, "Non-valid HTTP method!"));*
>> * }*
>>
>> * # Varnish don't mess with healthchecks*
>> * if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
>> "^/healthcheck.php") {*
>> * return (pass);*
>> * }*
>> * # Pipe requests to backup.php straight to backend - prevents problem
>> with progress bar long polling 503 problem*
>> * # This is here because backup.php is POSTing to itself - Filter
>> before !GET&&!HEAD*
>> * if (req.url ~ "^/backup/backup.php")*
>> * {*
>> * return (pipe);*
>> * }*
>>
>> * # Varnish only deals with GET and HEAD by default. If request method
>> is not GET or HEAD, pass request to backend*
>> * if (req.method != "GET" && req.method != "HEAD") {*
>> * return (pass);*
>> * }*
>>
>> * if (req.http.Cookie) {*
>> * # Remove any Google Analytics based cookies*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(;
>> )?", "");*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(;
>> )?", "");*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(;
>> )?", "");*
>> * set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");*
>> * if (req.http.Cookie ~ "^\s*$") {*
>> * unset req.http.Cookie;*
>> * }*
>> * }*
>>
>> * ### Rules for Moodle ###*
>>
>> * # Perform lookup for selected assets that we know are static but
>> Moodle still needs a Cookie*
>> * if( req.url ~ "^/theme/.+\.?" ||*
>> * req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||*
>> * req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||*
>> * req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$"
>> ||*
>> * req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"*
>> * )*
>> * {*
>> * # Set internal temporary header, based on which we will do
>> things in vcl_backend_response*
>> * set req.http.X-Long-TTL = "86400";*
>> * return (hash);*
>> * }*
>> * # Requests containing "Cookie" or "Authorization" headers will not
>> be cached*
>> * if (req.http.Authorization || req.http.Cookie) {*
>> * return (pass);*
>> * }*
>> * # Almost everything in Moodle correctly serves Cache-Control
>> headers, if*
>> * # needed, which varnish will honor, but there are some which don't.
>> Rather*
>> * # than explicitly finding them all and listing them here we just
>> fail safe*
>> * # and don't cache unknown urls that get this far.*
>> * return (pass);*
>>
>> *}*
>>
>> *sub vcl_backend_response {*
>> * # Set backend name*
>> * set beresp.http.X-Backend = beresp.backend.name
>> <http://beresp.backend.name>;*
>>
>> * if (beresp.http.Cache-Control && bereq.http.X-Long-TTL && beresp.ttl
>> < std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
>> !beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL
>> header*
>> * set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
>> beresp.http.Pragma;*
>> * set beresp.http.X-Orig-Cache-Control =
>> beresp.http.Cache-Control;*
>> * set beresp.http.Cache-Control = "public,
>> max-age="+bereq.http.X-Long-TTL+", no-transform";*
>> * set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);*
>> * unset bereq.http.X-Long-TTL;*
>> * }*
>> * else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
>> !beresp.http.WWW-Authenticate ) {*
>> * set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
>> beresp.http.Pragma;*
>> * set beresp.http.Cache-Control = "public,
>> max-age="+bereq.http.X-Long-TTL+", no-transform";*
>> * set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);*
>> * unset bereq.http.X-Long-TTL;*
>> * }*
>> * else { # Don't touch headers if max-age > defined in X-Long-TTL
>> header*
>> * unset bereq.http.X-Long-TTL;*
>> * }*
>> * # Here we set X-Trace header, prepending it to X-Trace header
>> received from backend. Useful for troubleshooting*
>> * if (beresp.http.x-trace && !beresp.was_304) {*
>> * set beresp.http.X-Trace = regsub(server.identity,
>> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
>> <http://beresp.backend.name>,
>> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;*
>> * }*
>> * else {*
>> * set beresp.http.X-Trace = regsub(server.identity,
>> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
>> <http://beresp.backend.name>,
>> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");*
>> * }*
>> *}*
>>
>> *sub vcl_deliver {*
>>
>> *# Revert back to original Cache-Control header before delivery to client*
>> * if (resp.http.X-Orig-Cache-Control)*
>> * {*
>> * set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;*
>> * unset resp.http.X-Orig-Cache-Control;*
>> * }*
>> * # Revert back to original Pragma header before delivery to client*
>> * if (resp.http.X-Orig-Pragma)*
>> * {*
>> * set resp.http.Pragma = resp.http.X-Orig-Pragma;*
>> * unset resp.http.X-Orig-Pragma;*
>> * }*
>>
>> * if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and
>> the number of hits, disable when not needed*
>> * set resp.http.X-Cache = "HIT";*
>> * } else {*
>> * set resp.http.X-Cache = "MISS";*
>> * }*
>> * set resp.http.X-Cache-Hits = obj.hits;*
>>
>> *# If desired "Via: 1.1 Varnish-v4" response header can be removed from
>> response*
>> * unset resp.http.Via;*
>> * unset resp.http.Server;*
>>
>> * return (deliver);*
>> *}*
>>
>> *sub vcl_backend_error {*
>> * # More comprehensive varnish error page. Display time, instance
>> hostname, host header, url for easier troubleshooting.*
>> * set beresp.http.Content-Type = "text/html; charset=utf-8";*
>> * set beresp.http.Retry-After = "5";*
>> * synthetic( {"*
>> * <!DOCTYPE html>*
>> * <html>*
>> * <head>*
>> * <title>"} + beresp.status + " " + beresp.reason + {"</title>*
>> * </head>*
>> * <body>*
>> * <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>*
>> * <p>"} + beresp.reason + {"</p>*
>> * <h3>Guru Meditation:</h3>*
>> * <p>Time: "} + now + {"</p>*
>> * <p>Node: "} + server.hostname + {"</p>*
>> * <p>Host: "} + bereq.http.host + {"</p>*
>> * <p>URL: "} + bereq.url + {"</p>*
>> * <p>XID: "} + bereq.xid + {"</p>*
>> * <hr>*
>> * <p>Varnish cache server*
>> * </body>*
>> * </html>*
>> * "} );*
>> * return (deliver);*
>> *}*
>>
>> *sub vcl_synth {*
>> * #Redirect using '301 - Permanent Redirect', permanent redirect*
>> * if (resp.status == 851) { *
>> * set resp.http.Location = req.http.x-redir;*
>> * set resp.http.X-Varnish-Redirect = true;*
>> * set resp.status = 301;*
>> * return (deliver);*
>> * }*
>> * #Redirect using '302 - Found', temporary redirect*
>> * if (resp.status == 852) { *
>> * set resp.http.Location = req.http.x-redir;*
>> * set resp.http.X-Varnish-Redirect = true;*
>> * set resp.status = 302;*
>> * return (deliver);*
>> * }*
>> * #Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests,
>> dont change method on redirected requests*
>> * if (resp.status == 857) { *
>> * set resp.http.Location = req.http.x-redir;*
>> * set resp.http.X-Varnish-Redirect = true;*
>> * set resp.status = 307;*
>> * return (deliver);*
>> * }*
>> * #Respond with 403 - Forbidden*
>> * if (resp.status == 863) {*
>> * set resp.http.X-Varnish-Error = true;*
>> * set resp.status = 403;*
>> * return (deliver);*
>> * }*
>> *}*
>>
>> *sub vcl_purge {*
>> * if (req.method != "PURGE") {*
>> * set req.http.X-Purge = "Yes";*
>> * return (restart);*
>> * }*
>> *}*
>>
>> ...
>>
>>
>> Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :
>>
>> Hello Karim,
>>
>> You VCL would be useful to debug this (as well as the command line you
>> are running Varnish with), but it sounds like Varnish is using the
>> Transient storage (
>> https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
>> to store the file, and as the storage isn't bounded, it explodes.
>> We can fix this in a couple of ways, from storing the file in the regular
>> cache storage, to using pipe, to waiting a few days for
>> https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
>> to be released.
>>
>> Question is: should that file be cached?
>>
>> Cheers,
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari <karim.ayari at univ-lyon1.fr>
>> wrote:
>>
>>> Hi!
>>>
>>> I am currently experiencing a memory load problem with video playback.
>>>
>>> here is the infrastructure :
>>>
>>> client --> haproxy --> varnish --> moodle workers (x5)
>>>
>>> a teacher uploaded a 400MB video to Moodle, when we start playing the
>>> video with browser player, Varnish consumes all the memory until it
>>> runs out and oom killer to kill varnishd. i have no configuration for
>>> mp4 files in my vcl file, so by default they are not hidden (?). I
>>> can't find a solution :(
>>>
>>> I can give my vcl file if necessary.
>>>
>>> (I am a beginner on varnish :))
>>>
>>> thank you for your support.
>>>
>>> Karim
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20231027/81bd8d08/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: psCUnmMoUHIDeDdC.png
Type: image/png
Size: 49126 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20231027/81bd8d08/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dKCCKWp0GtXF0le5.png
Type: image/png
Size: 42447 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20231027/81bd8d08/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 48105 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20231027/81bd8d08/attachment-0005.png>
More information about the varnish-misc
mailing list