From dridi at varni.sh Mon Nov 6 13:40:49 2023 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 6 Nov 2023 13:40:49 +0000 Subject: [master] 325faac2c hpack: Remove redundant/incorrect header validation In-Reply-To: <20230926130206.C7D1A1118B9@lists.varnish-cache.org> References: <20230926130206.C7D1A1118B9@lists.varnish-cache.org> Message-ID: On Tue, Sep 26, 2023 at 1:02?PM Dridi Boukelmoune wrote: > > > commit 325faac2cdfb2080f2ecccd340da49b1c41f7fe5 > Author: Walid Boudebouda > Date: Fri Sep 8 17:13:19 2023 +0200 > > hpack: Remove redundant/incorrect header validation > > Control characters will be caught by vct_ishdrval() anyways, but this > condition would also reject allowed obs-text non-ASCII characters. > > Signed-off-by: Dridi Boukelmoune I'm responsible for the incorrect statement above. The macro that will catch invalid characters for header field names is vct_istchar() actually. Dridi > diff --git a/bin/varnishd/http2/cache_http2_hpack.c b/bin/varnishd/http2/cache_http2_hpack.c > index 92befad53..b5451eb22 100644 > --- a/bin/varnishd/http2/cache_http2_hpack.c > +++ b/bin/varnishd/http2/cache_http2_hpack.c > @@ -75,12 +75,6 @@ h2h_checkhdr(const struct http *hp, const char *b, size_t namelen, size_t len) > break; > /* FALL_THROUGH */ > case FLD_NAME: > - if (*p <= 0x20 || *p >= 0x7f) { > - VSLb(hp->vsl, SLT_BogoHeader, > - "Illegal field header name (control): %.*s", > - (int)(len > 20 ? 20 : len), b); > - return (H2SE_PROTOCOL_ERROR); > - } > if (isupper(*p)) { > VSLb(hp->vsl, SLT_BogoHeader, > "Illegal field header name (upper-case): %.*s", > _______________________________________________ > varnish-commit mailing list > varnish-commit at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit From dridi.boukelmoune at gmail.com Mon Nov 13 08:24:08 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 13 Nov 2023 08:24:08 +0000 (UTC) Subject: [master] 34c87dd2c changes.rst: minor language tweak Message-ID: <20231113082408.EC5321113E3@lists.varnish-cache.org> commit 34c87dd2ce90b7b12e49551834d9c6fa00b4c59a Author: Dag Haavi Finstad Date: Mon Nov 6 14:13:50 2023 +0100 changes.rst: minor language tweak diff --git a/doc/changes.rst b/doc/changes.rst index 11737f68a..786803a7c 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -53,10 +53,10 @@ Varnish Cache NEXT (2024-03-15) In particular, this feature is used to reduce resource consumption of HTTP/2 "rapid reset" attacks (see below). - Note, in particular, that *req_reset* events may lead to client - tasks for which no VCL is called ever. Presumably, this is thus the - first time that valid `vcl(7)` client transactions may not contain - any ``VCL_call`` records. + Note that *req_reset* events may lead to client tasks for which no + VCL is called ever. Presumably, this is thus the first time that + valid `vcl(7)` client transactions may not contain any ``VCL_call`` + records. * Added mitigation options and visibility for HTTP/2 "rapid reset" attacks (CVE-2023-44487_, 3996_, 3997_, 3998_, 3999_). From dridi.boukelmoune at gmail.com Mon Nov 13 10:20:07 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 13 Nov 2023 10:20:07 +0000 (UTC) Subject: [7.3] bb28f2ac3 vtc_varnish: Log h2 frames Message-ID: <20231113102007.E333111514A@lists.varnish-cache.org> commit bb28f2ac35f45b002a924dc5d7493ba3b91c8b1f Author: Dridi Boukelmoune Date: Thu Jun 15 19:45:53 2023 +0200 vtc_varnish: Log h2 frames diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c index f763fcce7..ca596f190 100644 --- a/bin/varnishtest/vtc_varnish.c +++ b/bin/varnishtest/vtc_varnish.c @@ -417,7 +417,7 @@ varnish_launch(struct varnish *v) VSB_cat(vsb, " -p syslog_cli_traffic=off"); VSB_cat(vsb, " -p thread_pool_min=10"); VSB_cat(vsb, " -p debug=+vtc_mode"); - VSB_cat(vsb, " -p vsl_mask=+Debug"); + VSB_cat(vsb, " -p vsl_mask=+Debug,+H2RxHdr,+H2RxBody"); VSB_cat(vsb, " -p h2_initial_window_size=1m"); VSB_cat(vsb, " -p h2_rx_window_low_water=64k"); if (!v->has_a_arg) { From dridi.boukelmoune at gmail.com Mon Nov 13 10:20:08 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 13 Nov 2023 10:20:08 +0000 (UTC) Subject: [7.3] 0e75bd8ee changes.rst: Add new heading Message-ID: <20231113102008.0756011514C@lists.varnish-cache.org> commit 0e75bd8ee18c4f042f14e357c38827979cc59d04 Author: Dag Haavi Finstad Date: Tue Nov 7 10:39:19 2023 +0100 changes.rst: Add new heading diff --git a/doc/changes.rst b/doc/changes.rst index 08e53512e..3f6397a8d 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -30,6 +30,10 @@ directory, also available in HTML format at http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. +================================ +Varnish Cache 7.3.1 (2023-11-13) +================================ + ================================ Varnish Cache 7.3.0 (2023-03-15) From dridi.boukelmoune at gmail.com Mon Nov 13 10:20:08 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 13 Nov 2023 10:20:08 +0000 (UTC) Subject: [7.3] 66562c8c6 Changelog TLC Message-ID: <20231113102008.2BBA8115150@lists.varnish-cache.org> commit 66562c8c68cb5467df52c2c2d438ab033be8d1bf Author: Nils Goroll Date: Wed Oct 18 14:24:56 2023 +0200 Changelog TLC Conflicts: doc/changes.rst diff --git a/doc/changes.rst b/doc/changes.rst index 3f6397a8d..bcf680fd8 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -30,10 +30,101 @@ directory, also available in HTML format at http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. + ================================ Varnish Cache 7.3.1 (2023-11-13) ================================ +* Added mitigation options and visibility for HTTP/2 "rapid reset" + attacks (CVE-2023-44487_, 3996_, 3997_, 3998_, 3999_). + + Global rate limit controls have been added as parameters, which can + be overridden per HTTP/2 session from VCL using the new vmod ``h2``: + + * The ``h2_rapid_reset`` parameter and ``h2.rapid_reset()`` function + define a threshold duration for an ``RST_STREAM`` to be classified + as "rapid": If an ``RST_STREAM`` frame is parsed sooner than this + duration after a ``HEADERS`` frame, it is accounted against the + rate limit described below. + + The default is one second. + + * The ``h2_rapid_reset_limit`` parameter and + ``h2.rapid_reset_limit()`` function define how many "rapid" resets + may be received during the time span defined by the + ``h2_rapid_reset_period`` parameter / ``h2.rapid_reset_period()`` + function before the HTTP/2 connection is forcibly closed with a + ``GOAWAY`` and all ongoing VCL client tasks of the connection are + aborted. + + The defaults are 100 and 60 seconds, corresponding to an allowance + of 100 "rapid" resets per minute. + + * The ``h2.rapid_reset_budget()`` function can be used to query the + number of currently allowed "rapid" resets. + + * Sessions closed due to rapid reset rate limiting are reported as + ``SessClose RAPID_RESET`` in `vsl(7)` and accounted to + ``main.sc_rapid_reset`` in `vsc` as visible through + ``varnishstat(1)``. + +* A race condition has been fixed in the backend probe code which + could trigger a panic with dynamic backends (dyn100_). + +* A bug has been fixed in the ESI code which would prevent use of + internal status codes >1000 as their modulus 1000 value (3958_). + +* Fixed a performance issue in http/2 upload processing (3930_). + +* A regression introduced with Varnish Cache 7.3.0 was fixed: On + HTTP/2 connections, URLs starting with ``//`` no longer trigger a + protocol error (3911_). + +* A bug has been fixed where ``unset bereq.body`` had no effect when + used with a cached body (3914_). + +* Fixed a compatibility issue in the abstract socket support (3908_). + +* Improved HPACK header validation. + +* HTTP/2 header field validation is now more strict with respect to + allowed characters (3952_). + +* The ``vcl_req_reset`` feature (controllable through the ``feature`` + parameter, see `varnishd(1)`) has been added and enabled by default + to terminate client side VCL processing early when the client is + gone. + + *req_reset* events trigger a VCL failure and are reported to + `vsl(7)` as ``Timestamp: Reset`` and accounted to ``main.req_reset`` + in `vsc` as visible through ``varnishstat(1)``. + + In particular, this feature is used to reduce resource consumption + of HTTP/2 "rapid reset" attacks (see below). + + Note, in particular, that *req_reset* events may lead to client + tasks for which no VCL is called ever. Presumably, this is thus the + first time that valid `vcl(7)` client transactions may not contain + any ``VCL_call`` records. + +* The ``cli_limit`` parameter default has been increased from 48KB to + 64KB. + + +.. _CVE-2023-44487: https://nvd.nist.gov/vuln/detail/CVE-2023-44487 + +.. _3908: https://github.com/varnishcache/varnish-cache/issues/3908 +.. _3911: https://github.com/varnishcache/varnish-cache/issues/3911 +.. _3914: https://github.com/varnishcache/varnish-cache/issues/3914 +.. _3930: https://github.com/varnishcache/varnish-cache/issues/3930 +.. _3952: https://github.com/varnishcache/varnish-cache/issues/3952 +.. _3958: https://github.com/varnishcache/varnish-cache/issues/3958 +.. _3996: https://github.com/varnishcache/varnish-cache/pull/3996 +.. _3997: https://github.com/varnishcache/varnish-cache/pull/3997 +.. _3998: https://github.com/varnishcache/varnish-cache/pull/3998 +.. _3999: https://github.com/varnishcache/varnish-cache/pull/3999 + +.. _dyn100: https://github.com/nigoroll/libvmod-dynamic/issues/100 ================================ Varnish Cache 7.3.0 (2023-03-15) From dridi.boukelmoune at gmail.com Mon Nov 13 10:20:08 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 13 Nov 2023 10:20:08 +0000 (UTC) Subject: [7.3] 1d95c7f15 changes.rst: minor language tweak Message-ID: <20231113102008.4AC80115154@lists.varnish-cache.org> commit 1d95c7f15941efb7e7fd4123075a5663fcdaca21 Author: Dag Haavi Finstad Date: Mon Nov 6 14:13:50 2023 +0100 changes.rst: minor language tweak Conflicts: doc/changes.rst diff --git a/doc/changes.rst b/doc/changes.rst index bcf680fd8..d33618592 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -102,10 +102,10 @@ Varnish Cache 7.3.1 (2023-11-13) In particular, this feature is used to reduce resource consumption of HTTP/2 "rapid reset" attacks (see below). - Note, in particular, that *req_reset* events may lead to client - tasks for which no VCL is called ever. Presumably, this is thus the - first time that valid `vcl(7)` client transactions may not contain - any ``VCL_call`` records. + Note that *req_reset* events may lead to client tasks for which no + VCL is called ever. Presumably, this is thus the first time that + valid `vcl(7)` client transactions may not contain any ``VCL_call`` + records. * The ``cli_limit`` parameter default has been increased from 48KB to 64KB. From dridi.boukelmoune at gmail.com Mon Nov 13 10:20:08 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 13 Nov 2023 10:20:08 +0000 (UTC) Subject: [7.3] f529d82f8 Prepare for 7.3.1 Message-ID: <20231113102008.6A0F8115158@lists.varnish-cache.org> commit f529d82f892cd4537b8272a18bac6c970be9deb6 Author: Dag Haavi Finstad Date: Tue Nov 7 13:38:00 2023 +0100 Prepare for 7.3.1 diff --git a/configure.ac b/configure.ac index 1dc434839..d86cd4652 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ AC_PREREQ(2.69) AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2023 Varnish Software]) AC_REVISION([$Id$]) -AC_INIT([Varnish], [7.3.0], [varnish-dev at varnish-cache.org]) +AC_INIT([Varnish], [7.3.1], [varnish-dev at varnish-cache.org]) AC_CONFIG_SRCDIR(include/miniobj.h) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_MACRO_DIR([m4]) From dridi.boukelmoune at gmail.com Tue Nov 14 08:22:07 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 14 Nov 2023 08:22:07 +0000 (UTC) Subject: [6.0] a8b11a96a Remove stretch as its EOL Message-ID: <20231114082208.031C011F25E@lists.varnish-cache.org> commit a8b11a96abed2f8b3150905ac6cdb0b1330f3447 Author: Simon Stridsberg Date: Mon Nov 13 21:42:40 2023 +0100 Remove stretch as its EOL diff --git a/.circleci/config.yml b/.circleci/config.yml index 0214e77c8..97e0ac01a 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -436,7 +436,6 @@ workflows: - ubuntu:bionic - ubuntu:focal - ubuntu:jammy - - debian:stretch - debian:buster - debian:bullseye - debian:bookworm From dridi.boukelmoune at gmail.com Tue Nov 21 06:03:09 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:03:09 +0000 (UTC) Subject: [master] 63e4f71ef param: Clear least significant bits with none Message-ID: <20231121060309.69D9D10F7CC@lists.varnish-cache.org> commit 63e4f71efb9392407863f87ca4a89127d252ecb0 Author: Dridi Boukelmoune Date: Tue Nov 7 07:21:03 2023 +0100 param: Clear least significant bits with none The most significant bits of the least significant octet would be omitted when the number of bits is not a multiple of eight. diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index 6990f1b79..0c5301b83 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -607,6 +607,13 @@ bit(uint8_t *p, unsigned no, enum bit_do act) return (*p & b); } +static inline void +bit_clear(uint8_t *p, unsigned l) +{ + + memset(p, 0, (l + 7) >> 3); +} + /*-------------------------------------------------------------------- */ @@ -665,14 +672,14 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, if (arg != NULL && !strcmp(arg, "default") && strcmp(par->def, "none")) { - memset(p, 0, l >> 3); + bit_clear(p, l); return (tweak_generic_bits(vsb, par, par->def, p, l, tags, desc, sign)); } if (arg != NULL && arg != JSON_FMT) { if (sign == '+' && !strcmp(arg, "none")) - memset(p, 0, l >> 3); + bit_clear(p, l); else return (bit_tweak(vsb, p, l, arg, tags, desc, sign)); } else { diff --git a/bin/varnishtest/tests/c00054.vtc b/bin/varnishtest/tests/c00054.vtc index 420386b89..6730a6baa 100644 --- a/bin/varnishtest/tests/c00054.vtc +++ b/bin/varnishtest/tests/c00054.vtc @@ -1,13 +1,5 @@ varnishtest "bitmap params masking" - -server s1 { - rxreq - txresp -} -start - -varnish v1 -vcl+backend {} -start - varnish v1 -cliok "param.show vsl_mask" varnish v1 -cliok "param.set vsl_mask -VCL_trace" varnish v1 -cliok "param.show vsl_mask" @@ -15,6 +7,9 @@ varnish v1 -cliok "param.set vsl_mask -WorkThread,-TTL" varnish v1 -cliok "param.show vsl_mask" varnish v1 -cliok "param.set vsl_mask +WorkThread,+TTL,+Hash" varnish v1 -cliok "param.show vsl_mask" + +varnish v1 -cliexpect {"value": "none"} "param.set -j feature none" + varnish v1 -clierr 106 "param.set vsl_mask FooBar" varnish v1 -clierr 106 "param.set vsl_mask -FooBar" varnish v1 -clierr 106 {param.set vsl_mask \"} @@ -24,9 +19,3 @@ varnish v1 -cliok "param.show debug" varnish v1 -cliok "param.show feature" varnish v1 -cliok "param.set feature +short_panic" varnish v1 -cliok "param.show feature" - - -client c1 { - txreq - rxresp -} -run From dridi.boukelmoune at gmail.com Tue Nov 21 06:03:09 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:03:09 +0000 (UTC) Subject: [master] 70977c91c param: New "all" counterpart to "none" for vsl_mask Message-ID: <20231121060309.8913410F7D0@lists.varnish-cache.org> commit 70977c91c47117809bbc8ce89bdb12e083b756a8 Author: Dridi Boukelmoune Date: Tue Nov 7 07:25:08 2023 +0100 param: New "all" counterpart to "none" for vsl_mask diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index 0c5301b83..bfd0f2bee 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -680,6 +680,8 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, if (arg != NULL && arg != JSON_FMT) { if (sign == '+' && !strcmp(arg, "none")) bit_clear(p, l); + else if (sign == '-' && !strcmp(arg, "all")) + bit_clear(p, l); else return (bit_tweak(vsb, p, l, arg, tags, desc, sign)); } else { @@ -693,7 +695,7 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, } } if (*s == '\0') - VSB_cat(vsb, sign == '+' ? "none" : "(all enabled)"); + VSB_cat(vsb, sign == '+' ? "none" : "all"); if (arg == JSON_FMT) VSB_putc(vsb, '"'); } diff --git a/bin/varnishtest/tests/c00054.vtc b/bin/varnishtest/tests/c00054.vtc index 6730a6baa..63aef00a1 100644 --- a/bin/varnishtest/tests/c00054.vtc +++ b/bin/varnishtest/tests/c00054.vtc @@ -9,6 +9,7 @@ varnish v1 -cliok "param.set vsl_mask +WorkThread,+TTL,+Hash" varnish v1 -cliok "param.show vsl_mask" varnish v1 -cliexpect {"value": "none"} "param.set -j feature none" +varnish v1 -cliexpect {"value": "all"} "param.set -j vsl_mask all" varnish v1 -clierr 106 "param.set vsl_mask FooBar" varnish v1 -clierr 106 "param.set vsl_mask -FooBar" diff --git a/include/tbl/params.h b/include/tbl/params.h index 073e5be3b..de16b146e 100644 --- a/include/tbl/params.h +++ b/include/tbl/params.h @@ -1881,6 +1881,7 @@ PARAM_BITS( "-WorkThread", /* descr */ "Mask individual VSL messages from being logged.\n" + "\tall\tEnable all tags\n" "\tdefault\tSet default value\n" "\nUse +/- prefix in front of VSL tag name to unmask/mask " "individual VSL messages.") From dridi.boukelmoune at gmail.com Tue Nov 21 06:03:09 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:03:09 +0000 (UTC) Subject: [master] edd71dd57 param: Set all and none as bit parameters values Message-ID: <20231121060309.A293610F7D4@lists.varnish-cache.org> commit edd71dd5796f69be9193130ee79b113906e4a57d Author: Dridi Boukelmoune Date: Tue Nov 7 07:34:58 2023 +0100 param: Set all and none as bit parameters values As opposed to special values. This enables setting "absolute" values atomically: param.set foo none,+bar,+baz diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index bfd0f2bee..0a8e1e595 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -634,6 +634,14 @@ bit_tweak(struct vsb *vsb, uint8_t *p, unsigned l, const char *arg, } for (i = 1; av[i] != NULL; i++) { s = av[i]; + if (sign == '+' && !strcmp(s, "none")) { + bit_clear(p, l); + continue; + } + if (sign == '-' && !strcmp(s, "all")) { + bit_clear(p, l); + continue; + } if (*s != '-' && *s != '+') { VSB_printf(vsb, "Missing '+' or '-' (%s)\n", s); VAV_Free(av); @@ -678,12 +686,7 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, } if (arg != NULL && arg != JSON_FMT) { - if (sign == '+' && !strcmp(arg, "none")) - bit_clear(p, l); - else if (sign == '-' && !strcmp(arg, "all")) - bit_clear(p, l); - else - return (bit_tweak(vsb, p, l, arg, tags, desc, sign)); + return (bit_tweak(vsb, p, l, arg, tags, desc, sign)); } else { if (arg == JSON_FMT) VSB_putc(vsb, '"'); From dridi.boukelmoune at gmail.com Tue Nov 21 06:03:09 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:03:09 +0000 (UTC) Subject: [master] 03fe53475 param: Remove unncessary else branch Message-ID: <20231121060309.BE2CB10F7DA@lists.varnish-cache.org> commit 03fe53475d85a375e8f6f4dee6ad46d030300415 Author: Dridi Boukelmoune Date: Tue Nov 7 07:49:30 2023 +0100 param: Remove unncessary else branch Better diff with the --ignore-all-space option. diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index 0a8e1e595..f1ac4d4b5 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -685,23 +685,22 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, desc, sign)); } - if (arg != NULL && arg != JSON_FMT) { + if (arg != NULL && arg != JSON_FMT) return (bit_tweak(vsb, p, l, arg, tags, desc, sign)); - } else { - if (arg == JSON_FMT) - VSB_putc(vsb, '"'); - s = ""; - for (j = 0; j < l; j++) { - if (bit(p, j, BTST)) { - VSB_printf(vsb, "%s%c%s", s, sign, tags[j]); - s = ","; - } + + if (arg == JSON_FMT) + VSB_putc(vsb, '"'); + s = ""; + for (j = 0; j < l; j++) { + if (bit(p, j, BTST)) { + VSB_printf(vsb, "%s%c%s", s, sign, tags[j]); + s = ","; } - if (*s == '\0') - VSB_cat(vsb, sign == '+' ? "none" : "all"); - if (arg == JSON_FMT) - VSB_putc(vsb, '"'); } + if (*s == '\0') + VSB_cat(vsb, sign == '+' ? "none" : "all"); + if (arg == JSON_FMT) + VSB_putc(vsb, '"'); return (0); } From dridi.boukelmoune at gmail.com Tue Nov 21 06:03:09 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:03:09 +0000 (UTC) Subject: [master] bdc32bea8 param: Show bits parameters as absolute values Message-ID: <20231121060309.D87F610F7E6@lists.varnish-cache.org> commit bdc32bea807ff8eb954a3783a5e62ae9396b410b Author: Dridi Boukelmoune Date: Tue Nov 7 07:58:19 2023 +0100 param: Show bits parameters as absolute values diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index f1ac4d4b5..82a1e192a 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -675,7 +675,6 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, uint8_t *p, unsigned l, const char * const *tags, const char *desc, char sign) { - const char *s; unsigned j; if (arg != NULL && !strcmp(arg, "default") && @@ -690,15 +689,11 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, if (arg == JSON_FMT) VSB_putc(vsb, '"'); - s = ""; + VSB_cat(vsb, sign == '+' ? "none" : "all"); for (j = 0; j < l; j++) { - if (bit(p, j, BTST)) { - VSB_printf(vsb, "%s%c%s", s, sign, tags[j]); - s = ","; - } + if (bit(p, j, BTST)) + VSB_printf(vsb, ",%c%s", sign, tags[j]); } - if (*s == '\0') - VSB_cat(vsb, sign == '+' ? "none" : "all"); if (arg == JSON_FMT) VSB_putc(vsb, '"'); return (0); From dridi.boukelmoune at gmail.com Tue Nov 21 06:03:09 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:03:09 +0000 (UTC) Subject: [master] aa2fc004b param: Give all bits parameters absolute defaults Message-ID: <20231121060310.0195410F7EC@lists.varnish-cache.org> commit aa2fc004b4014ced061bf8e5c6942b22c48c48ea Author: Dridi Boukelmoune Date: Tue Nov 7 09:06:01 2023 +0100 param: Give all bits parameters absolute defaults This is primarily for self-documentation purposes, but also to simplify the bit tweak. diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index 82a1e192a..6ec97c01a 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -677,9 +677,7 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, { unsigned j; - if (arg != NULL && !strcmp(arg, "default") && - strcmp(par->def, "none")) { - bit_clear(p, l); + if (arg != NULL && !strcmp(arg, "default")) { return (tweak_generic_bits(vsb, par, par->def, p, l, tags, desc, sign)); } diff --git a/include/tbl/params.h b/include/tbl/params.h index de16b146e..36923172a 100644 --- a/include/tbl/params.h +++ b/include/tbl/params.h @@ -1829,6 +1829,7 @@ PARAM_BITS( /* name */ feature, /* fld */ feature_bits, /* def */ + "none," "+validate_headers," "+vcl_req_reset", /* descr */ @@ -1847,6 +1848,7 @@ PARAM_BITS( /* name */ vcc_feature, /* fld */ vcc_feature_bits, /* def */ + "none," "+err_unref," "+unsafe_path", /* descr */ @@ -1865,6 +1867,7 @@ PARAM_BITS( /* name */ vsl_mask, /* fld */ vsl_mask, /* def */ + "all," "-Debug," "-ExpKill," "-H2RxBody," From dridi.boukelmoune at gmail.com Tue Nov 21 06:03:10 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:03:10 +0000 (UTC) Subject: [master] e8f8afc51 cli: Teach param.reset -j to output param.show Message-ID: <20231121060310.30E5010F7F4@lists.varnish-cache.org> commit e8f8afc51f669c46b542cf8b9764b23afbb3ad7b Author: Dridi Boukelmoune Date: Tue Nov 7 09:16:56 2023 +0100 cli: Teach param.reset -j to output param.show diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index 1cfd789ba..d305a1006 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -713,7 +713,7 @@ mcf_wash_param(struct cli *cli, struct parspec *pp, enum mcf_which_e which, static struct cli_proto cli_params[] = { { CLICMD_PARAM_SHOW, "", mcf_param_show, mcf_param_show_json }, { CLICMD_PARAM_SET, "", mcf_param_set, mcf_param_set_json }, - { CLICMD_PARAM_RESET, "", mcf_param_reset }, + { CLICMD_PARAM_RESET, "", mcf_param_reset, mcf_param_set_json }, { NULL } }; diff --git a/bin/varnishtest/tests/c00054.vtc b/bin/varnishtest/tests/c00054.vtc index 63aef00a1..0ee1c2674 100644 --- a/bin/varnishtest/tests/c00054.vtc +++ b/bin/varnishtest/tests/c00054.vtc @@ -10,6 +10,7 @@ varnish v1 -cliok "param.show vsl_mask" varnish v1 -cliexpect {"value": "none"} "param.set -j feature none" varnish v1 -cliexpect {"value": "all"} "param.set -j vsl_mask all" +varnish v1 -cliexpect {"value": "all(,-\w+)+"} "param.reset -j vsl_mask" varnish v1 -clierr 106 "param.set vsl_mask FooBar" varnish v1 -clierr 106 "param.set vsl_mask -FooBar" diff --git a/include/tbl/cli_cmds.h b/include/tbl/cli_cmds.h index b4b9c3194..3295b5844 100644 --- a/include/tbl/cli_cmds.h +++ b/include/tbl/cli_cmds.h @@ -172,9 +172,11 @@ CLI_CMD(VCL_LABEL, CLI_CMD(PARAM_RESET, "param.reset", - "param.reset ", + "param.reset [-j] ", "Reset parameter to default value.", - "", + " The JSON output is the same as ``param.show -j `` and" + " contains the updated value as it would be represented by a" + " subsequent execution of ``param.show``.\n\n", 1,1 ) From dridi.boukelmoune at gmail.com Tue Nov 21 06:03:10 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:03:10 +0000 (UTC) Subject: [master] 23d62d28a param: Deprecate default for bits parameters Message-ID: <20231121060310.634B110F80F@lists.varnish-cache.org> commit 23d62d28a319cd01d0de35a85e19baaf9779c32e Author: Dridi Boukelmoune Date: Tue Nov 7 09:21:24 2023 +0100 param: Deprecate default for bits parameters We have had the ability to reset any parameter to its default value for a while now. diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index 6ec97c01a..07ad6ce30 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -678,6 +678,7 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, unsigned j; if (arg != NULL && !strcmp(arg, "default")) { + /* XXX: deprecated in favor of param.reset */ return (tweak_generic_bits(vsb, par, par->def, p, l, tags, desc, sign)); } diff --git a/include/tbl/params.h b/include/tbl/params.h index 36923172a..c88809da7 100644 --- a/include/tbl/params.h +++ b/include/tbl/params.h @@ -1834,7 +1834,7 @@ PARAM_BITS( "+vcl_req_reset", /* descr */ "Enable/Disable various minor features.\n" - "\tdefault\tSet default value\n" + "\tdefault\tSet default value (deprecated: use param.reset)\n" "\tnone\tDisable all features.\n\n" "Use +/- prefix to enable/disable individual feature:") #ifdef PARAM_ALL @@ -1853,7 +1853,7 @@ PARAM_BITS( "+unsafe_path", /* descr */ "Enable/Disable various VCC behaviors.\n" - "\tdefault\tSet default value\n" + "\tdefault\tSet default value (deprecated: use param.reset)\n" "\tnone\tDisable all behaviors.\n\n" "Use +/- prefix to enable/disable individual behavior:") #ifdef PARAM_ALL @@ -1885,7 +1885,7 @@ PARAM_BITS( /* descr */ "Mask individual VSL messages from being logged.\n" "\tall\tEnable all tags\n" - "\tdefault\tSet default value\n" + "\tdefault\tSet default value (deprecated: use param.reset)\n" "\nUse +/- prefix in front of VSL tag name to unmask/mask " "individual VSL messages.") PARAM_POST From dridi.boukelmoune at gmail.com Tue Nov 21 06:39:06 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:39:06 +0000 (UTC) Subject: [7.4] daedbe586 req_fsm: Ensure failed sub-requests reach transmit Message-ID: <20231121063906.6F9AE111D6B@lists.varnish-cache.org> commit daedbe5862384ce916d66305000b2cc950335e81 Author: Dridi Boukelmoune Date: Tue Nov 21 07:12:11 2023 +0100 req_fsm: Ensure failed sub-requests reach transmit A VCL failure on the client side transitions to vcl_synth, except failures from vcl_synth that lead to minimal errors. The ESI transport is not allowed to reply with minimal responses so this would lead to a panic. On top of that, the vcl_req_reset feature flag emulates `return (fail)` statements when an HTTP/2 client disconnected, resulting in the same panic scenario. For sub-requests, we masquerade the fail transition as a deliver and trade the illegal minimal response for the synthetic response. Fixes #4022 diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 789cea540..c0508cd4c 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -333,7 +333,11 @@ cnt_synth(struct worker *wrk, struct req *req) VSLb_ts_req(req, "Process", W_TIM_real(wrk)); - if (wrk->vpi->handling == VCL_RET_FAIL) { + while (wrk->vpi->handling == VCL_RET_FAIL) { + if (req->esi_level > 0) { + wrk->vpi->handling = VCL_RET_DELIVER; + break; + } VSB_destroy(&synth_body); (void)VRB_Ignore(req); (void)req->transport->minimal_response(req, 500); diff --git a/bin/varnishtest/tests/e00037.vtc b/bin/varnishtest/tests/e00037.vtc new file mode 100644 index 000000000..bfb1354e5 --- /dev/null +++ b/bin/varnishtest/tests/e00037.vtc @@ -0,0 +1,28 @@ +varnishtest "Double fail ESI sub request" + +server s1 { + rxreq + txresp -body {} +} -start + +varnish v1 -vcl+backend { + sub vcl_backend_response { + set beresp.do_esi = true; + } + + sub vcl_recv { + if (req.esi_level > 0) { + return (fail); + } + } + + sub vcl_synth { + return (fail); + } +} -start + +client c1 { + non_fatal + txreq + rxresp +} -run From dridi.boukelmoune at gmail.com Tue Nov 21 06:40:06 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:40:06 +0000 (UTC) Subject: [7.3] aeaf44a48 req_fsm: Ensure failed sub-requests reach transmit Message-ID: <20231121064006.ABD44111FE1@lists.varnish-cache.org> commit aeaf44a48eea50145b3b3c0fba93daf7fdf6bdd1 Author: Dridi Boukelmoune Date: Tue Nov 21 07:12:11 2023 +0100 req_fsm: Ensure failed sub-requests reach transmit A VCL failure on the client side transitions to vcl_synth, except failures from vcl_synth that lead to minimal errors. The ESI transport is not allowed to reply with minimal responses so this would lead to a panic. On top of that, the vcl_req_reset feature flag emulates `return (fail)` statements when an HTTP/2 client disconnected, resulting in the same panic scenario. For sub-requests, we masquerade the fail transition as a deliver and trade the illegal minimal response for the synthetic response. Fixes #4022 diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 3deb70e5a..9373ec4ef 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -333,7 +333,11 @@ cnt_synth(struct worker *wrk, struct req *req) VSLb_ts_req(req, "Process", W_TIM_real(wrk)); - if (wrk->vpi->handling == VCL_RET_FAIL) { + while (wrk->vpi->handling == VCL_RET_FAIL) { + if (req->esi_level > 0) { + wrk->vpi->handling = VCL_RET_DELIVER; + break; + } VSB_destroy(&synth_body); (void)VRB_Ignore(req); (void)req->transport->minimal_response(req, 500); diff --git a/bin/varnishtest/tests/e00037.vtc b/bin/varnishtest/tests/e00037.vtc new file mode 100644 index 000000000..bfb1354e5 --- /dev/null +++ b/bin/varnishtest/tests/e00037.vtc @@ -0,0 +1,28 @@ +varnishtest "Double fail ESI sub request" + +server s1 { + rxreq + txresp -body {} +} -start + +varnish v1 -vcl+backend { + sub vcl_backend_response { + set beresp.do_esi = true; + } + + sub vcl_recv { + if (req.esi_level > 0) { + return (fail); + } + } + + sub vcl_synth { + return (fail); + } +} -start + +client c1 { + non_fatal + txreq + rxresp +} -run From dridi.boukelmoune at gmail.com Tue Nov 21 06:57:06 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:57:06 +0000 (UTC) Subject: [6.0] d83d85a9b fix when the vcl_synth {} Process timestamp gets emitted Message-ID: <20231121065706.73C9E112DD9@lists.varnish-cache.org> commit d83d85a9be32345dac501e4cc9bc017a9cbbd1b2 Author: Nils Goroll Date: Fri Oct 11 09:38:12 2019 +0200 fix when the vcl_synth {} Process timestamp gets emitted Found by @Dridi Conflicts: doc/changes.rst diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index f24033ee1..b09484304 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -301,8 +301,6 @@ cnt_synth(struct worker *wrk, struct req *req) wrk->stats->s_synth++; - VSLb_ts_req(req, "Process", W_TIM_real(wrk)); - if (req->err_code < 100) req->err_code = 501; @@ -315,6 +313,8 @@ cnt_synth(struct worker *wrk, struct req *req) AZ(VSB_finish(synth_body)); + VSLb_ts_req(req, "Process", W_TIM_real(wrk)); + if (wrk->handling == VCL_RET_FAIL) { VSB_destroy(&synth_body); req->doclose = SC_VCL_FAILURE; diff --git a/bin/varnishtest/tests/c00018.vtc b/bin/varnishtest/tests/c00018.vtc index 10359f758..cff0bebde 100644 --- a/bin/varnishtest/tests/c00018.vtc +++ b/bin/varnishtest/tests/c00018.vtc @@ -122,7 +122,6 @@ logexpect l1 -v v1 -g raw { expect 0 1011 VCL_return {^synth$} expect 0 1011 VCL_call {^HASH$} expect 0 1011 VCL_return {^lookup$} - expect 0 1011 Timestamp {^Process:} expect 0 1011 RespProtocol {^HTTP/1.1$} expect 0 1011 RespStatus {^405$} expect 0 1011 RespReason {^Method Not Allowed$} @@ -130,6 +129,7 @@ logexpect l1 -v v1 -g raw { expect 1 1011 RespHeader {^Date:} expect 0 1011 RespHeader {^Server: Varnish$} expect 0 1011 RespHeader {^X-Varnish: 1011$} + expect * 1011 Timestamp {^Process:} } -start diff --git a/doc/changes.rst b/doc/changes.rst index f6111b256..6c747fade 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -26,6 +26,14 @@ http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. +================================= +Varnish Cache 6.0.13 (unreleased) +================================= + +* The ``Process`` timestamp for ``vcl_synth {}`` was wrongly issued + before the VCL callback, now it gets emitted after VCL returns for + consistency with ``vcl_deliver {}`` + ================================= Varnish Cache 6.0.12 (2023-11-13) ================================= From dridi.boukelmoune at gmail.com Tue Nov 21 06:57:06 2023 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 21 Nov 2023 06:57:06 +0000 (UTC) Subject: [6.0] 63db35d03 req_fsm: Ensure failed sub-requests reach transmit Message-ID: <20231121065706.8A46A112DDC@lists.varnish-cache.org> commit 63db35d03defee2f0c2e4d553ba45dd536f2d25d Author: Dridi Boukelmoune Date: Tue Nov 21 07:12:11 2023 +0100 req_fsm: Ensure failed sub-requests reach transmit A VCL failure on the client side transitions to vcl_synth, except failures from vcl_synth that lead to minimal errors. The ESI transport is not allowed to reply with minimal responses so this would lead to a panic. On top of that, the vcl_req_reset feature flag emulates `return (fail)` statements when an HTTP/2 client disconnected, resulting in the same panic scenario. For sub-requests, we masquerade the fail transition as a deliver and trade the illegal minimal response for the synthetic response. Fixes #4022 Conflicts: bin/varnishd/cache/cache_req_fsm.c diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index b09484304..43d3e6d68 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -315,7 +315,11 @@ cnt_synth(struct worker *wrk, struct req *req) VSLb_ts_req(req, "Process", W_TIM_real(wrk)); - if (wrk->handling == VCL_RET_FAIL) { + while (wrk->handling == VCL_RET_FAIL) { + if (req->esi_level > 0) { + wrk->handling = VCL_RET_DELIVER; + break; + } VSB_destroy(&synth_body); req->doclose = SC_VCL_FAILURE; VSLb_ts_req(req, "Resp", W_TIM_real(wrk)); diff --git a/bin/varnishtest/tests/e00037.vtc b/bin/varnishtest/tests/e00037.vtc new file mode 100644 index 000000000..bfb1354e5 --- /dev/null +++ b/bin/varnishtest/tests/e00037.vtc @@ -0,0 +1,28 @@ +varnishtest "Double fail ESI sub request" + +server s1 { + rxreq + txresp -body {} +} -start + +varnish v1 -vcl+backend { + sub vcl_backend_response { + set beresp.do_esi = true; + } + + sub vcl_recv { + if (req.esi_level > 0) { + return (fail); + } + } + + sub vcl_synth { + return (fail); + } +} -start + +client c1 { + non_fatal + txreq + rxresp +} -run