varnish and mp4 files
AYARI KARIM
karim.ayari at univ-lyon1.fr
Fri Oct 27 06:42:56 UTC 2023
Hello,
I'm coming back to this subject because after a moment of calm my problem of memory leak and OOM killer with videos is back :(
we have videos hosted on Moodle behind varnish which are themselves behind haproxy.
version varnish 6.2.1-2ubuntu0.2 amd64
this expression seems to work
if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {
std.log("ispiped:true");
return (pipe);
}
mp4 files are piped :
- VCL_call RECV
- VCL_Log ispiped:true
- VCL_return pipe
- VCL_call HASH
- VCL_return lookup
first graph is memory usage (total of 16GB), the second one requests fetched, we can see 2 restarts after OOM killing at 16h25 and 18h20
[cid:404902a3-6553-4f34-8b67-7b5b9853b3c3]
I don't see any other file extension to add in the expression
thank you for your help
---
Karim Ayari
________________________________
De : varnish-misc <varnish-misc-bounces+karim.ayari=univ-lyon1.fr at varnish-cache.org> de la part de Karim Ayari <karim.ayari at univ-lyon1.fr>
Envoyé : mardi 21 février 2023 09:05
À : Guillaume Quintard
Cc : varnish-misc at varnish-cache.org
Objet : Re: varnish and mp4 files
sorry I thought I noted it in my previous email.
here are the good lines for using pipe :
if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {
std.log("ispiped:true");
return (pipe);
}
thank you!
Karim.
Le 20/02/2023 à 21:50, Guillaume Quintard a écrit :
Looks like Rainer replied directly to you and not to the list, would you mind sharing/highlighting the fix for people having the same issue?
Cheers,
--
Guillaume Quintard
On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari <karim.ayari at univ-lyon1.fr<mailto:karim.ayari at univ-lyon1.fr>> wrote:
thank you both for your replies. I forgot the most important thing: the varnish server has 16 GB of ram and the cache is 1 GB.
the cache never seems full
[cid:part1.hll3XDne.buey2s0m at univ-lyon1.fr][cid:part2.3Xb0PaGW.FM1miA0J at univ-lyon1.fr]
the command used : /usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s malloc,1g -p http_max_hdr=96
I had tried using pipe it didn't work, but Rainer's lines works fine.
this solution should suffice because the videos will soon have to be hosted on our video platform.
my vcl (file found on github) :
....
sub vcl_recv {
# Keep client IP
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
} else {
unset req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
}
}
if (req.http.X-Real-IP) {
set req.http.X-Forwarded-For = req.http.X-Real-IP;
} else {
set req.http.X-Forwarded-For = client.ip;
}
# Only deal with "normal" types
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "PATCH" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
/*Why send the packet upstream, while the visitor is using a non-valid HTTP method? */
return (synth(404, "Non-valid HTTP method!"));
}
# Varnish don't mess with healthchecks
if (req.url ~ "^/admin/tool/heartbeat" || req.url ~ "^/healthcheck.php") {
return (pass);
}
# Pipe requests to backup.php straight to backend - prevents problem with progress bar long polling 503 problem
# This is here because backup.php is POSTing to itself - Filter before !GET&&!HEAD
if (req.url ~ "^/backup/backup.php")
{
return (pipe);
}
# Varnish only deals with GET and HEAD by default. If request method is not GET or HEAD, pass request to backend
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
if (req.http.Cookie) {
# Remove any Google Analytics based cookies
set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");
set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");
if (req.http.Cookie ~ "^\s*$") {
unset req.http.Cookie;
}
}
### Rules for Moodle ###
# Perform lookup for selected assets that we know are static but Moodle still needs a Cookie
if( req.url ~ "^/theme/.+\.?" ||
req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||
req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||
req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||
req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"
)
{
# Set internal temporary header, based on which we will do things in vcl_backend_response
set req.http.X-Long-TTL = "86400";
return (hash);
}
# Requests containing "Cookie" or "Authorization" headers will not be cached
if (req.http.Authorization || req.http.Cookie) {
return (pass);
}
# Almost everything in Moodle correctly serves Cache-Control headers, if
# needed, which varnish will honor, but there are some which don't. Rather
# than explicitly finding them all and listing them here we just fail safe
# and don't cache unknown urls that get this far.
return (pass);
}
sub vcl_backend_response {
# Set backend name
set beresp.http.X-Backend = beresp.backend.name<http://beresp.backend.name>;
if (beresp.http.Cache-Control && bereq.http.X-Long-TTL && beresp.ttl < std.duration(bereq.http.X-Long-TTL + "s", 1s) && !beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL header
set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset beresp.http.Pragma;
set beresp.http.X-Orig-Cache-Control = beresp.http.Cache-Control;
set beresp.http.Cache-Control = "public, max-age="+bereq.http.X-Long-TTL+", no-transform";
set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);
unset bereq.http.X-Long-TTL;
}
else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL && !beresp.http.WWW-Authenticate ) {
set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset beresp.http.Pragma;
set beresp.http.Cache-Control = "public, max-age="+bereq.http.X-Long-TTL+", no-transform";
set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);
unset bereq.http.X-Long-TTL;
}
else { # Don't touch headers if max-age > defined in X-Long-TTL header
unset bereq.http.X-Long-TTL;
}
# Here we set X-Trace header, prepending it to X-Trace header received from backend. Useful for troubleshooting
if (beresp.http.x-trace && !beresp.was_304) {
set beresp.http.X-Trace = regsub(server.identity, "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name<http://beresp.backend.name>, "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;
}
else {
set beresp.http.X-Trace = regsub(server.identity, "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name<http://beresp.backend.name>, "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");
}
}
sub vcl_deliver {
# Revert back to original Cache-Control header before delivery to client
if (resp.http.X-Orig-Cache-Control)
{
set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;
unset resp.http.X-Orig-Cache-Control;
}
# Revert back to original Pragma header before delivery to client
if (resp.http.X-Orig-Pragma)
{
set resp.http.Pragma = resp.http.X-Orig-Pragma;
unset resp.http.X-Orig-Pragma;
}
if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
set resp.http.X-Cache-Hits = obj.hits;
# If desired "Via: 1.1 Varnish-v4" response header can be removed from response
unset resp.http.Via;
unset resp.http.Server;
return (deliver);
}
sub vcl_backend_error {
# More comprehensive varnish error page. Display time, instance hostname, host header, url for easier troubleshooting.
set beresp.http.Content-Type = "text/html; charset=utf-8";
set beresp.http.Retry-After = "5";
synthetic( {"
<!DOCTYPE html>
<html>
<head>
<title>"} + beresp.status + " " + beresp.reason + {"</title>
</head>
<body>
<h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
<p>"} + beresp.reason + {"</p>
<h3>Guru Meditation:</h3>
<p>Time: "} + now + {"</p>
<p>Node: "} + server.hostname + {"</p>
<p>Host: "} + bereq.http.host + {"</p>
<p>URL: "} + bereq.url + {"</p>
<p>XID: "} + bereq.xid + {"</p>
<hr>
<p>Varnish cache server
</body>
</html>
"} );
return (deliver);
}
sub vcl_synth {
#Redirect using '301 - Permanent Redirect', permanent redirect
if (resp.status == 851) {
set resp.http.Location = req.http.x-redir;
set resp.http.X-Varnish-Redirect = true;
set resp.status = 301;
return (deliver);
}
#Redirect using '302 - Found', temporary redirect
if (resp.status == 852) {
set resp.http.Location = req.http.x-redir;
set resp.http.X-Varnish-Redirect = true;
set resp.status = 302;
return (deliver);
}
#Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests, dont change method on redirected requests
if (resp.status == 857) {
set resp.http.Location = req.http.x-redir;
set resp.http.X-Varnish-Redirect = true;
set resp.status = 307;
return (deliver);
}
#Respond with 403 - Forbidden
if (resp.status == 863) {
set resp.http.X-Varnish-Error = true;
set resp.status = 403;
return (deliver);
}
}
sub vcl_purge {
if (req.method != "PURGE") {
set req.http.X-Purge = "Yes";
return (restart);
}
}
...
Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :
Hello Karim,
You VCL would be useful to debug this (as well as the command line you are running Varnish with), but it sounds like Varnish is using the Transient storage (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage) to store the file, and as the storage isn't bounded, it explodes.
We can fix this in a couple of ways, from storing the file in the regular cache storage, to using pipe, to waiting a few days for https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643 to be released.
Question is: should that file be cached?
Cheers,
--
Guillaume Quintard
On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari <karim.ayari at univ-lyon1.fr<mailto:karim.ayari at univ-lyon1.fr>> wrote:
Hi!
I am currently experiencing a memory load problem with video playback.
here is the infrastructure :
client --> haproxy --> varnish --> moodle workers (x5)
a teacher uploaded a 400MB video to Moodle, when we start playing the video with browser player, Varnish consumes all the memory until it runs out and oom killer to kill varnishd. i have no configuration for mp4 files in my vcl file, so by default they are not hidden (?). I can't find a solution :(
I can give my vcl file if necessary.
(I am a beginner on varnish :))
thank you for your support.
Karim
_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20231027/25b7757f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: psCUnmMoUHIDeDdC.png
Type: image/png
Size: 49126 bytes
Desc: psCUnmMoUHIDeDdC.png
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20231027/25b7757f/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dKCCKWp0GtXF0le5.png
Type: image/png
Size: 42447 bytes
Desc: dKCCKWp0GtXF0le5.png
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20231027/25b7757f/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 48105 bytes
Desc: pastedImage.png
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20231027/25b7757f/attachment-0005.png>
More information about the varnish-misc
mailing list