From karim.ayari at univ-lyon1.fr Thu Dec 14 15:10:00 2023 From: karim.ayari at univ-lyon1.fr (AYARI KARIM) Date: Thu, 14 Dec 2023 15:10:00 +0000 Subject: varnish and mp4 files In-Reply-To: References: <8a22d5e0-bfcb-abbe-6c48-11a956b1d364@univ-lyon1.fr> , Message-ID: <320cf261999c477ca3498be582dfcdb1@univ-lyon1.fr> Hi, Sorry for my late response, but I found another video extension that must have been a problem. It seems ok for 1 month, fingers crossed ;) thank you Guillaume. Karim Ayari ________________________________ De : Guillaume Quintard Envoy? : vendredi 27 octobre 2023 18:24 ? : AYARI KARIM Cc : varnish-misc at varnish-cache.org Objet : Re: varnish and mp4 files Hi, Check the varnishncsa logs to see if something fishy stands out, or even just the backend logs. Notably, try varnishncsa -q 'not VCL_call eq PIPE' That will give you all the transactions that go through vcl_pipe{} (more log tips here: https://docs.varnish-software.com/tutorials/vsl-query/) If that doesn't work we can try to filter big files and whatnot. -- Guillaume Quintard On Thu, Oct 26, 2023 at 11:47?PM AYARI KARIM > wrote: Hello, I'm coming back to this subject because after a moment of calm my problem of memory leak and OOM killer with videos is back :( we have videos hosted on Moodle behind varnish which are themselves behind haproxy. version varnish 6.2.1-2ubuntu0.2 amd64 this expression seems to work if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") { std.log("ispiped:true"); return (pipe); } mp4 files are piped : - VCL_call RECV - VCL_Log ispiped:true - VCL_return pipe - VCL_call HASH - VCL_return lookup first graph is memory usage (total of 16GB), the second one requests fetched, we can see 2 restarts after OOM killing at 16h25 and 18h20 [cid:18b71ec5db3f456b1e53] I don't see any other file extension to add in the expression thank you for your help --- Karim Ayari ________________________________ De : varnish-misc > de la part de Karim Ayari > Envoy? : mardi 21 f?vrier 2023 09:05 ? : Guillaume Quintard Cc : varnish-misc at varnish-cache.org Objet : Re: varnish and mp4 files sorry I thought I noted it in my previous email. here are the good lines for using pipe : if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") { std.log("ispiped:true"); return (pipe); } thank you! Karim. Le 20/02/2023 ? 21:50, Guillaume Quintard a ?crit : Looks like Rainer replied directly to you and not to the list, would you mind sharing/highlighting the fix for people having the same issue? Cheers, -- Guillaume Quintard On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari > wrote: thank you both for your replies. I forgot the most important thing: the varnish server has 16 GB of ram and the cache is 1 GB. the cache never seems full [cid:18b71ec5db3af62603d1][cid:18b71ec5db3c3ba5d52] the command used : /usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s malloc,1g -p http_max_hdr=96 I had tried using pipe it didn't work, but Rainer's lines works fine. this solution should suffice because the videos will soon have to be hosted on our video platform. my vcl (file found on github) : .... sub vcl_recv { # Keep client IP if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { unset req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; } } if (req.http.X-Real-IP) { set req.http.X-Forwarded-For = req.http.X-Real-IP; } else { set req.http.X-Forwarded-For = client.ip; } # Only deal with "normal" types if (req.method != "GET" && req.method != "HEAD" && req.method != "PUT" && req.method != "POST" && req.method != "TRACE" && req.method != "OPTIONS" && req.method != "PATCH" && req.method != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ /*Why send the packet upstream, while the visitor is using a non-valid HTTP method? */ return (synth(404, "Non-valid HTTP method!")); } # Varnish don't mess with healthchecks if (req.url ~ "^/admin/tool/heartbeat" || req.url ~ "^/healthcheck.php") { return (pass); } # Pipe requests to backup.php straight to backend - prevents problem with progress bar long polling 503 problem # This is here because backup.php is POSTing to itself - Filter before !GET&&!HEAD if (req.url ~ "^/backup/backup.php") { return (pipe); } # Varnish only deals with GET and HEAD by default. If request method is not GET or HEAD, pass request to backend if (req.method != "GET" && req.method != "HEAD") { return (pass); } if (req.http.Cookie) { # Remove any Google Analytics based cookies set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", ""); if (req.http.Cookie ~ "^\s*$") { unset req.http.Cookie; } } ### Rules for Moodle ### # Perform lookup for selected assets that we know are static but Moodle still needs a Cookie if( req.url ~ "^/theme/.+\.?" || req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" || req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" || req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" || req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$" ) { # Set internal temporary header, based on which we will do things in vcl_backend_response set req.http.X-Long-TTL = "86400"; return (hash); } # Requests containing "Cookie" or "Authorization" headers will not be cached if (req.http.Authorization || req.http.Cookie) { return (pass); } # Almost everything in Moodle correctly serves Cache-Control headers, if # needed, which varnish will honor, but there are some which don't. Rather # than explicitly finding them all and listing them here we just fail safe # and don't cache unknown urls that get this far. return (pass); } sub vcl_backend_response { # Set backend name set beresp.http.X-Backend = beresp.backend.name; if (beresp.http.Cache-Control && bereq.http.X-Long-TTL && beresp.ttl < std.duration(bereq.http.X-Long-TTL + "s", 1s) && !beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL header set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset beresp.http.Pragma; set beresp.http.X-Orig-Cache-Control = beresp.http.Cache-Control; set beresp.http.Cache-Control = "public, max-age="+bereq.http.X-Long-TTL+", no-transform"; set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s); unset bereq.http.X-Long-TTL; } else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL && !beresp.http.WWW-Authenticate ) { set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset beresp.http.Pragma; set beresp.http.Cache-Control = "public, max-age="+bereq.http.X-Long-TTL+", no-transform"; set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s); unset bereq.http.X-Long-TTL; } else { # Don't touch headers if max-age > defined in X-Long-TTL header unset bereq.http.X-Long-TTL; } # Here we set X-Trace header, prepending it to X-Trace header received from backend. Useful for troubleshooting if (beresp.http.x-trace && !beresp.was_304) { set beresp.http.X-Trace = regsub(server.identity, "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name, "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace; } else { set beresp.http.X-Trace = regsub(server.identity, "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name, "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)"); } } sub vcl_deliver { # Revert back to original Cache-Control header before delivery to client if (resp.http.X-Orig-Cache-Control) { set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control; unset resp.http.X-Orig-Cache-Control; } # Revert back to original Pragma header before delivery to client if (resp.http.X-Orig-Pragma) { set resp.http.Pragma = resp.http.X-Orig-Pragma; unset resp.http.X-Orig-Pragma; } if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } set resp.http.X-Cache-Hits = obj.hits; # If desired "Via: 1.1 Varnish-v4" response header can be removed from response unset resp.http.Via; unset resp.http.Server; return (deliver); } sub vcl_backend_error { # More comprehensive varnish error page. Display time, instance hostname, host header, url for easier troubleshooting. set beresp.http.Content-Type = "text/html; charset=utf-8"; set beresp.http.Retry-After = "5"; synthetic( {" "} + beresp.status + " " + beresp.reason + {"

Error "} + beresp.status + " " + beresp.reason + {"

"} + beresp.reason + {"

Guru Meditation:

Time: "} + now + {"

Node: "} + server.hostname + {"

Host: "} + bereq.http.host + {"

URL: "} + bereq.url + {"

XID: "} + bereq.xid + {"

Varnish cache server "} ); return (deliver); } sub vcl_synth { #Redirect using '301 - Permanent Redirect', permanent redirect if (resp.status == 851) { set resp.http.Location = req.http.x-redir; set resp.http.X-Varnish-Redirect = true; set resp.status = 301; return (deliver); } #Redirect using '302 - Found', temporary redirect if (resp.status == 852) { set resp.http.Location = req.http.x-redir; set resp.http.X-Varnish-Redirect = true; set resp.status = 302; return (deliver); } #Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests, dont change method on redirected requests if (resp.status == 857) { set resp.http.Location = req.http.x-redir; set resp.http.X-Varnish-Redirect = true; set resp.status = 307; return (deliver); } #Respond with 403 - Forbidden if (resp.status == 863) { set resp.http.X-Varnish-Error = true; set resp.status = 403; return (deliver); } } sub vcl_purge { if (req.method != "PURGE") { set req.http.X-Purge = "Yes"; return (restart); } } ... Le 20/02/2023 ? 18:55, Guillaume Quintard a ?crit : Hello Karim, You VCL would be useful to debug this (as well as the command line you are running Varnish with), but it sounds like Varnish is using the Transient storage (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage) to store the file, and as the storage isn't bounded, it explodes. We can fix this in a couple of ways, from storing the file in the regular cache storage, to using pipe, to waiting a few days for https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643 to be released. Question is: should that file be cached? Cheers, -- Guillaume Quintard On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari > wrote: Hi! I am currently experiencing a memory load problem with video playback. here is the infrastructure : client --> haproxy --> varnish --> moodle workers (x5) a teacher uploaded a 400MB video to Moodle, when we start playing the video with browser player, Varnish consumes all the memory until it runs out and oom killer to kill varnishd. i have no configuration for mp4 files in my vcl file, so by default they are not hidden (?). I can't find a solution :( I can give my vcl file if necessary. (I am a beginner on varnish :)) thank you for your support. Karim _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: psCUnmMoUHIDeDdC.png Type: image/png Size: 49126 bytes Desc: psCUnmMoUHIDeDdC.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dKCCKWp0GtXF0le5.png Type: image/png Size: 42447 bytes Desc: dKCCKWp0GtXF0le5.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 48105 bytes Desc: pastedImage.png URL: