From dridi.boukelmoune at gmail.com Wed Jul 1 12:09:09 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 1 Jul 2020 12:09:09 +0000 (UTC) Subject: [master] 630817e46 build: Enable automake's subdir-objects option Message-ID: <20200701120909.C248EAA614@lists.varnish-cache.org> commit 630817e467bf5df8ecd457d35cfd8ad61dfd4560 Author: Dridi Boukelmoune Date: Wed Jul 1 14:00:03 2020 +0200 build: Enable automake's subdir-objects option We can get rid of autogen.des' egrep command used to silence this warning too. In fact we can even remove the subshell and simply call autoreconf directly. The problem with egrep is the loss of a meaningful exit status for the autoreconf invocation. I also enabled autoreconf's verbose output. diff --git a/autogen.des b/autogen.des index 7ef93d968..44d3f551b 100755 --- a/autogen.des +++ b/autogen.des @@ -22,8 +22,8 @@ else DST="--prefix=/opt/varnish --mandir=/opt/varnish/man" fi -rm -f configure -(cd $SRCDIR && autoreconf -i 2>&1 | egrep -v "(subdir-objects|is in a subdirectory)") +rm -f $SRCDIR/configure +autoreconf -i -v $SRCDIR # autoconf prior to 2.62 has issues with zsh 4.2 and newer CONFIG_SHELL=/bin/sh diff --git a/configure.ac b/configure.ac index cbdeb59cf..f0f5b11db 100644 --- a/configure.ac +++ b/configure.ac @@ -19,7 +19,7 @@ AC_CANONICAL_SYSTEM AC_LANG(C) AM_MAINTAINER_MODE([disable]) -AM_INIT_AUTOMAKE([1.11 foreign color-tests parallel-tests]) +AM_INIT_AUTOMAKE([1.11 foreign color-tests parallel-tests subdir-objects]) AM_SILENT_RULES([yes]) AC_DISABLE_STATIC AC_PROG_LIBTOOL From dridi.boukelmoune at gmail.com Thu Jul 2 09:58:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 09:58:07 +0000 (UTC) Subject: [master] 1d33f3ff5 build: Facilitate VPATH bootstraps Message-ID: <20200702095807.B4CDFA13E5@lists.varnish-cache.org> commit 1d33f3ff5f0b9c4e1faec4bfaa7d6f5a7ce2510b Author: Dridi Boukelmoune Date: Thu Jul 2 11:45:05 2020 +0200 build: Facilitate VPATH bootstraps In order to bootstrap a $(builddir) outside of $(srcdir) until now the first step needed was to populate the environment. If for example one wants to work from a build/ sub-directory the shortest autogen.des invocation would look like: SRCDIR=.. ../autogen.des [configure options...] Since we can infer $(srcdir) from autogen.des' relative path, it is now possible to simply do this: ../autogen.des [configure options...] It is still possible to provide a specific SRCDIR, even to a different directory but the default should now work out of the box. diff --git a/autogen.des b/autogen.des index 44d3f551b..96a488f9e 100755 --- a/autogen.des +++ b/autogen.des @@ -2,7 +2,7 @@ # # Use this when doing code development -SRCDIR=${SRCDIR:-.} +SRCDIR=${SRCDIR:-$(dirname "$0")} set -ex From dridi.boukelmoune at gmail.com Thu Jul 2 09:58:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 09:58:07 +0000 (UTC) Subject: [master] 456c12b39 build: Query clang from the $PATH Message-ID: <20200702095807.C6483A13E8@lists.varnish-cache.org> commit 456c12b39a25f294210bfc4ee94110cedcf41826 Author: Dridi Boukelmoune Date: Thu Jul 2 11:53:32 2020 +0200 build: Query clang from the $PATH It might not be installed as /usr/bin/clang, and there's a mismatch anyway between the check and the outcome. For example on my system's $PATH clang is currently driven by ccache is available at /usr/lib64/ccache/clang. diff --git a/autogen.des b/autogen.des index 96a488f9e..7fffce8c3 100755 --- a/autogen.des +++ b/autogen.des @@ -9,7 +9,7 @@ set -ex make -k distclean > /dev/null 2>&1 || true # Prefer CLANG if we have it, and have not given preferences -if [ -f /usr/bin/clang -a "x${CC}" = "x" ] ; then +if command -v clang >/dev/null && test -z "$CC" ; then CC=clang export CC fi From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:06 +0000 (UTC) Subject: [master] d6a0f6f00 Don't register VMOD symbols as wildcards Message-ID: <20200702124906.CE583A5958@lists.varnish-cache.org> commit d6a0f6f00c9a7658bac2df2466ffd2d14c046c90 Author: Dridi Boukelmoune Date: Wed Dec 11 15:52:20 2019 +0100 Don't register VMOD symbols as wildcards Instead, register all the functions and object constructors at import time. diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c index 1f28b1758..17616b861 100644 --- a/lib/libvcc/vcc_vmod.c +++ b/lib/libvcc/vcc_vmod.c @@ -129,37 +129,6 @@ vcc_json_always(struct vcc *tl, const struct vjsn *vj, const char *vmod_name) } } -static void v_matchproto_(sym_wildcard_t) -vcc_json_wildcard(struct vcc *tl, struct symbol *msym, struct symbol *tsym) -{ - const struct vjsn *vj; - const struct vjsn_val *vv, *vv1, *vv2; - - assert(msym->kind == SYM_VMOD); - CAST_OBJ_NOTNULL(vj, msym->eval_priv, VJSN_MAGIC); - VTAILQ_FOREACH(vv, &vj->value->children, list) { - assert(vv->type == VJSN_ARRAY); - vv1 = VTAILQ_FIRST(&vv->children); - assert(vv1->type == VJSN_STRING); - vv2 = VTAILQ_NEXT(vv1, list); - assert(vv2->type == VJSN_STRING); - if (!strcmp(vv1->value, "$FUNC") && - !strcmp(vv2->value, tsym->name)) { - tsym->kind = SYM_FUNC; - tsym->noref = 1; - func_sym(tsym, msym->vmod_name, VTAILQ_NEXT(vv2, list)); - return; - } else if (!strcmp(vv1->value, "$OBJ") && - !strcmp(vv2->value, tsym->name)) { - tsym->kind = SYM_OBJECT; - tsym->eval_priv = vv2; - tsym->vmod_name = msym->vmod_name; - return; - } - } - tl->err = 1; -} - static const struct vmod_data * vcc_VmodSanity(struct vcc *tl, void *hdl, struct token *mod, char *fnp) { @@ -216,6 +185,49 @@ vcc_VmodSanity(struct vcc *tl, void *hdl, struct token *mod, char *fnp) return (vmd); } +static void +vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) +{ + const struct vjsn *vj; + const struct vjsn_val *vv, *vv1, *vv2; + struct symbol *fsym; + struct vsb *buf; + + assert(msym->kind == SYM_VMOD); + CAST_OBJ_NOTNULL(vj, msym->eval_priv, VJSN_MAGIC); + + buf = VSB_new_auto(); + AN(buf); + + VTAILQ_FOREACH(vv, &vj->value->children, list) { + VSB_clear(buf); + + assert(vv->type == VJSN_ARRAY); + vv1 = VTAILQ_FIRST(&vv->children); + assert(vv1->type == VJSN_STRING); + vv2 = VTAILQ_NEXT(vv1, list); + assert(vv2->type == VJSN_STRING); + + VSB_printf(buf, "%s.%s", msym->name, vv2->value); + AZ(VSB_finish(buf)); + + if (!strcmp(vv1->value, "$FUNC")) { + fsym = VCC_MkSym(tl, VSB_data(buf), SYM_FUNC, + VCL_LOW, VCL_HIGH); + AN(fsym); + func_sym(fsym, msym->vmod_name, VTAILQ_NEXT(vv2, list)); + } else if (!strcmp(vv1->value, "$OBJ")) { + fsym = VCC_MkSym(tl, VSB_data(buf), SYM_OBJECT, + VCL_LOW, VCL_HIGH); + AN(fsym); + fsym->eval_priv = vv2; + fsym->vmod_name = msym->vmod_name; + } + } + + VSB_destroy(&buf); +} + void vcc_ParseImport(struct vcc *tl) { @@ -317,9 +329,9 @@ vcc_ParseImport(struct vcc *tl) if (!strcmp(vsym->extra, vmd->file_id)) { /* Already loaded under different name */ msym->eval_priv = vsym->eval_priv; - msym->wildcard = vsym->wildcard; msym->extra = vsym->extra; msym->vmod_name = vsym->vmod_name; + vcc_VmodSymbols(tl, msym); AZ(dlclose(vop->hdl)); free(fnpx); return; @@ -364,9 +376,9 @@ vcc_ParseImport(struct vcc *tl) XXXAZ(p); AN(vj); msym->eval_priv = vj; - msym->wildcard = vcc_json_wildcard; msym->extra = TlDup(tl, vmd->file_id); msym->vmod_name = TlDup(tl, vmd->name); + vcc_VmodSymbols(tl, msym); vcc_json_always(tl, vj, msym->vmod_name); From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:06 +0000 (UTC) Subject: [master] 1a22c9449 Kill some redundancy and regroup VSB operations Message-ID: <20200702124906.E054BA595B@lists.varnish-cache.org> commit 1a22c9449f105f1b6825d4e6df2256a79731f968 Author: Dridi Boukelmoune Date: Wed Dec 11 17:56:09 2019 +0100 Kill some redundancy and regroup VSB operations diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c index 17616b861..51ce7c5ee 100644 --- a/lib/libvcc/vcc_vmod.c +++ b/lib/libvcc/vcc_vmod.c @@ -185,12 +185,28 @@ vcc_VmodSanity(struct vcc *tl, void *hdl, struct token *mod, char *fnp) return (vmd); } +static vcc_kind_t +vcc_vmod_kind(const char *type) +{ + +#define VMOD_KIND(str, kind) \ + do { \ + if (!strcmp(str, type)) \ + return (kind); \ + } while (0) + VMOD_KIND("$OBJ", SYM_OBJECT); + VMOD_KIND("$FUNC", SYM_FUNC); +#undef VMOD_KIND + return (SYM_NONE); +} + static void vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) { const struct vjsn *vj; const struct vjsn_val *vv, *vv1, *vv2; struct symbol *fsym; + vcc_kind_t kind; struct vsb *buf; assert(msym->kind == SYM_VMOD); @@ -200,26 +216,26 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) AN(buf); VTAILQ_FOREACH(vv, &vj->value->children, list) { - VSB_clear(buf); - assert(vv->type == VJSN_ARRAY); vv1 = VTAILQ_FIRST(&vv->children); assert(vv1->type == VJSN_STRING); vv2 = VTAILQ_NEXT(vv1, list); assert(vv2->type == VJSN_STRING); + kind = vcc_vmod_kind(vv1->value); + if (kind == SYM_NONE) + continue; + + VSB_clear(buf); VSB_printf(buf, "%s.%s", msym->name, vv2->value); AZ(VSB_finish(buf)); + fsym = VCC_MkSym(tl, VSB_data(buf), kind, VCL_LOW, VCL_HIGH); + AN(fsym); - if (!strcmp(vv1->value, "$FUNC")) { - fsym = VCC_MkSym(tl, VSB_data(buf), SYM_FUNC, - VCL_LOW, VCL_HIGH); - AN(fsym); + if (kind == SYM_FUNC) { func_sym(fsym, msym->vmod_name, VTAILQ_NEXT(vv2, list)); - } else if (!strcmp(vv1->value, "$OBJ")) { - fsym = VCC_MkSym(tl, VSB_data(buf), SYM_OBJECT, - VCL_LOW, VCL_HIGH); - AN(fsym); + } else { + assert(kind == SYM_OBJECT); fsym->eval_priv = vv2; fsym->vmod_name = msym->vmod_name; } From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:07 +0000 (UTC) Subject: [master] 3d1371521 Create VMOD object instance symbols like the rest Message-ID: <20200702124907.02BD2A595F@lists.varnish-cache.org> commit 3d13715210b5136deff6c2d7dce9aefa6805663a Author: Dridi Boukelmoune Date: Wed Dec 11 18:35:07 2019 +0100 Create VMOD object instance symbols like the rest Some assertions or expectations needed to loosen to accomodate $OBJ, $FUNC and $METHOD entries under the same roof. There is a SYM_METHOD symbol kind that is currently not used, and shouldn't be until VMOD methods are per-object symbols instead of per-instance. Once we reach that point, it becomes possible to simplify further and move more things in the func_sym() function. diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c index 51ce7c5ee..fb41197a3 100644 --- a/lib/libvcc/vcc_vmod.c +++ b/lib/libvcc/vcc_vmod.c @@ -195,6 +195,7 @@ vcc_vmod_kind(const char *type) return (kind); \ } while (0) VMOD_KIND("$OBJ", SYM_OBJECT); + VMOD_KIND("$METHOD", SYM_FUNC); VMOD_KIND("$FUNC", SYM_FUNC); #undef VMOD_KIND return (SYM_NONE); @@ -209,23 +210,45 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) vcc_kind_t kind; struct vsb *buf; - assert(msym->kind == SYM_VMOD); - CAST_OBJ_NOTNULL(vj, msym->eval_priv, VJSN_MAGIC); + if (msym->kind == SYM_VMOD) { + CAST_OBJ_NOTNULL(vj, msym->eval_priv, VJSN_MAGIC); + vv = VTAILQ_FIRST(&vj->value->children); + } else if (msym->kind == SYM_INSTANCE) { + CAST_OBJ_NOTNULL(vv, msym->eval_priv, VJSN_VAL_MAGIC); + } else { + WRONG("symbol kind"); + } buf = VSB_new_auto(); AN(buf); - VTAILQ_FOREACH(vv, &vj->value->children, list) { - assert(vv->type == VJSN_ARRAY); + for (; vv != NULL; vv = VTAILQ_NEXT(vv, list)) { + if (vv->type != VJSN_ARRAY) + continue; vv1 = VTAILQ_FIRST(&vv->children); assert(vv1->type == VJSN_STRING); vv2 = VTAILQ_NEXT(vv1, list); - assert(vv2->type == VJSN_STRING); + if (vv2->type != VJSN_STRING) + continue; kind = vcc_vmod_kind(vv1->value); if (kind == SYM_NONE) continue; + /* NB: currently VMOD object methods are effectively function + * symbols (SYM_FUNC) because they are declared per instance + * instead of per object. Once they become proper methods + * symbols (SYM_METHOD) we can move the symbol creation inside + * the func_sym() function and replace the rest of the loop + * with a single statement: + * + * func_sym(kind, msym, vv2); + * + * Then based on kind, the func_sym() function would account + * for the slight differences between the 3 kinds of VMOD + * functions (function, object constructor, object method). + */ + VSB_clear(buf); VSB_printf(buf, "%s.%s", msym->name, vv2->value); AZ(VSB_finish(buf)); @@ -234,6 +257,9 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) if (kind == SYM_FUNC) { func_sym(fsym, msym->vmod_name, VTAILQ_NEXT(vv2, list)); + /* XXX: until we use SYM_METHOD, string check. */ + if (!strcmp(vv1->value, "$METHOD")) + fsym->extra = msym->rname; } else { assert(kind == SYM_OBJECT); fsym->eval_priv = vv2; @@ -409,7 +435,7 @@ vcc_ParseImport(struct vcc *tl) void v_matchproto_(sym_act_f) vcc_Act_New(struct vcc *tl, struct token *t, struct symbol *sym) { - struct symbol *isym, *osym, *msym; + struct symbol *isym, *osym; struct inifin *ifp; struct vsb *buf; const struct vjsn_val *vv, *vf; @@ -434,6 +460,10 @@ vcc_Act_New(struct vcc *tl, struct token *t, struct symbol *sym) CAST_OBJ_NOTNULL(vv, osym->eval_priv, VJSN_VAL_MAGIC); // vv = object name + isym->vmod_name = osym->vmod_name; + isym->eval_priv = vv; + vcc_VmodSymbols(tl, isym); + vv = VTAILQ_NEXT(vv, list); // vv = flags assert(vv->type == VJSN_OBJECT); @@ -467,7 +497,6 @@ vcc_Act_New(struct vcc *tl, struct token *t, struct symbol *sym) isym->def_e = tl->t; vf = VTAILQ_FIRST(&vv->children); - vv = VTAILQ_NEXT(vv, list); assert(vf->type == VJSN_STRING); assert(!strcmp(vf->value, "$FINI")); @@ -477,26 +506,4 @@ vcc_Act_New(struct vcc *tl, struct token *t, struct symbol *sym) ifp = New_IniFin(tl); VSB_printf(ifp->fin, "\t\tif (%s)\n", isym->rname); VSB_printf(ifp->fin, "\t\t\t\t%s(&%s);", vf->value, isym->rname); - - /* Instantiate symbols for the methods */ - buf = VSB_new_auto(); - AN(buf); - - while (vv != NULL) { - vf = VTAILQ_FIRST(&vv->children); - assert(vf->type == VJSN_STRING); - assert(!strcmp(vf->value, "$METHOD")); - vf = VTAILQ_NEXT(vf, list); - assert(vf->type == VJSN_STRING); - - VSB_clear(buf); - VSB_printf(buf, "%s.%s", isym->name, vf->value); - AZ(VSB_finish(buf)); - msym = VCC_MkSym(tl, VSB_data(buf), SYM_FUNC, VCL_LOW, VCL_HIGH); - AN(msym); - func_sym(msym, osym->vmod_name, VTAILQ_NEXT(vf, list)); - msym->extra = isym->rname; - vv = VTAILQ_NEXT(vv, list); - } - VSB_destroy(&buf); } From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:07 +0000 (UTC) Subject: [master] 7d27ba046 Add a list of methods to type definitions Message-ID: <20200702124907.1DA02A5965@lists.varnish-cache.org> commit 7d27ba046853884820e0630fb03848ce2cb2191b Author: Dridi Boukelmoune Date: Thu Nov 28 18:35:24 2019 +0100 Add a list of methods to type definitions For now, this is a copy of struct vcc_methods used in expression parsing and the list includes type properties as well. diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 7ba6cbb62..dfd3a45e1 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -97,13 +97,23 @@ struct token { }; /*---------------------------------------------------------------------*/ + typedef const struct type *vcc_type_t; +struct vcc_method { + vcc_type_t type; + const char *name; + const char *impl; + int func; +}; + struct type { unsigned magic; #define TYPE_MAGIC 0xfae932d9 const char *name; + const struct vcc_method *methods; + const char *tostring; vcc_type_t multype; int stringform; From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:07 +0000 (UTC) Subject: [master] a971e6c31 Inline type methods definitions Message-ID: <20200702124907.394BBA596D@lists.varnish-cache.org> commit a971e6c310b966ca8dcafad1797afdb62cd0d71d Author: Dridi Boukelmoune Date: Thu Nov 28 18:52:09 2019 +0100 Inline type methods definitions diff --git a/lib/libvcc/vcc_types.c b/lib/libvcc/vcc_types.c index 2691a0531..33efaeed4 100644 --- a/lib/libvcc/vcc_types.c +++ b/lib/libvcc/vcc_types.c @@ -42,9 +42,15 @@ const struct type ACL[1] = {{ .tostring = "((\v1)->name)", }}; +static const struct vcc_method backend_methods[] = { + { BACKEND, "resolve", "VRT_VDI_Resolve(ctx, \v1)", 1 }, + { NULL }, +}; + const struct type BACKEND[1] = {{ .magic = TYPE_MAGIC, .name = "BACKEND", + .methods = backend_methods, .tostring = "VRT_BACKEND_string(\v1)", }}; @@ -125,9 +131,17 @@ const struct type REAL[1] = {{ .multype = REAL, }}; +static const struct vcc_method stevedore_methods[] = { +#define VRTSTVVAR(nm, vtype, ctype, dval) \ + { vtype, #nm, "VRT_stevedore_" #nm "(\v1)", 0}, +#include "tbl/vrt_stv_var.h" + { NULL }, +}; + const struct type STEVEDORE[1] = {{ .magic = TYPE_MAGIC, .name = "STEVEDORE", + .methods = stevedore_methods, .tostring = "VRT_STEVEDORE_string(\v1)", }}; @@ -144,9 +158,16 @@ const struct type STRANDS[1] = {{ .tostring = "VRT_CollectStrands(ctx,\v+\n\v1\v-\n)", }}; +static const struct vcc_method strings_methods[] = { + { STRING, "upper", "VRT_UpperLowerStrands(ctx, \vT, 1)", 1 }, + { STRING, "lower", "VRT_UpperLowerStrands(ctx, \vT, 0)", 1 }, + { NULL }, +}; + const struct type STRINGS[1] = {{ .magic = TYPE_MAGIC, .name = "STRINGS", + .methods = strings_methods, .tostring = "", }}; From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:07 +0000 (UTC) Subject: [master] 96edbd290 Remove type method and property definitions from vcc_expr Message-ID: <20200702124907.55CAEA597F@lists.varnish-cache.org> commit 96edbd290e59d76e2c9ef860176a043bcefe5e09 Author: Dridi Boukelmoune Date: Thu Nov 28 19:07:50 2019 +0100 Remove type method and property definitions from vcc_expr This is a first step towards making expression parsing more agnostic and type methods more prevalent. diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index dfd3a45e1..c63335f2f 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -100,6 +100,15 @@ struct token { typedef const struct type *vcc_type_t; +/* + * A type attribute is information already existing, requiring no processing + * or resource usage. + * + * A type method is a call and may do significant processing, change things, + * eat workspace etc. + * + * XXX: type methods might move in a more comprehensive direction. + */ struct vcc_method { vcc_type_t type; const char *name; diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index a459dd757..91c9917a0 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -816,38 +816,12 @@ vcc_expr5(struct vcc *tl, struct expr **e, vcc_type_t fmt) * SYNTAX: * Expr4: * Expr5 [ '.' (type_attribute | type_method()) ]* - * - * type_attributes is information already existing, requiring no - * processing or resource usage. - * - * type_methods are calls and may do (significant processing, change things, - * eat workspace etc. */ -static const struct vcc_methods { - vcc_type_t type_from; - vcc_type_t type_to; - const char *method; - const char *impl; - int func; -} vcc_methods[] = { - //{ BACKEND, BOOL, "healthy", "VRT_Healthy(ctx, \v1, 0)" }, - -#define VRTSTVVAR(nm, vtype, ctype, dval) \ - { STEVEDORE, vtype, #nm, "VRT_stevedore_" #nm "(\v1)", 0}, -#include "tbl/vrt_stv_var.h" - - { STRINGS, STRING, "upper", "VRT_UpperLowerStrands(ctx, \vT, 1)", 1 }, - { STRINGS, STRING, "lower", "VRT_UpperLowerStrands(ctx, \vT, 0)", 1 }, - { BACKEND, BACKEND, "resolve", "VRT_DirectorResolve(ctx, \v1)", 1 }, - - { NULL, NULL, NULL, NULL}, -}; - static void vcc_expr4(struct vcc *tl, struct expr **e, vcc_type_t fmt) { - const struct vcc_methods *vm; + const struct vcc_method *vm; *e = NULL; vcc_expr5(tl, e, fmt); @@ -857,14 +831,14 @@ vcc_expr4(struct vcc *tl, struct expr **e, vcc_type_t fmt) vcc_NextToken(tl); ExpectErr(tl, ID); - for(vm = vcc_methods; vm->type_from != NULL; vm++) { - - if (vm->type_from == (*e)->fmt && - vcc_IdIs(tl->t, vm->method)) + vm = (*e)->fmt->methods; + while (vm != NULL && vm->type != NULL) { + if (vcc_IdIs(tl->t, vm->name)) break; + vm++; } - if (vm->type_from == NULL) { + if (vm == NULL || vm->type == NULL) { VSB_cat(tl->sb, "Unknown property "); vcc_ErrToken(tl, tl->t); VSB_printf(tl->sb, @@ -873,7 +847,7 @@ vcc_expr4(struct vcc *tl, struct expr **e, vcc_type_t fmt) return; } vcc_NextToken(tl); - *e = vcc_expr_edit(tl, vm->type_to, vm->impl, *e, NULL); + *e = vcc_expr_edit(tl, vm->type, vm->impl, *e, NULL); ERRCHK(tl); if ((*e)->fmt == STRING) { (*e)->fmt = STRINGS; diff --git a/lib/libvcc/vcc_types.c b/lib/libvcc/vcc_types.c index 33efaeed4..4f9db3b80 100644 --- a/lib/libvcc/vcc_types.c +++ b/lib/libvcc/vcc_types.c @@ -43,7 +43,7 @@ const struct type ACL[1] = {{ }}; static const struct vcc_method backend_methods[] = { - { BACKEND, "resolve", "VRT_VDI_Resolve(ctx, \v1)", 1 }, + { BACKEND, "resolve", "VRT_DirectorResolve(ctx, \v1)", 1 }, { NULL }, }; From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:07 +0000 (UTC) Subject: [master] 6c5d9ae0b Split the monolithic VCL symbol table Message-ID: <20200702124907.72B01A5984@lists.varnish-cache.org> commit 6c5d9ae0b4c0ebc2422286af048dd2e10aba1ab4 Author: Dridi Boukelmoune Date: Fri Jun 5 07:44:19 2020 +0200 Split the monolithic VCL symbol table This introduces two namespaces MAIN and TYPE to allow top-level symbol collisions for type methods. For example, `backend` is a reserved keyword but `backend.resolve` is a type method. It is impossible to declare such a method symbol because it would require its parent `backend` symbol to be a type symbol. There would be a conflict between two `backend` symbols of different kinds. The same would apply if the BLOB type had methods and vmod_blob was imported. At this point only the MAIN namespace is used but the split enables the symbolization of types and type methods down the line. diff --git a/lib/libvcc/Makefile.am b/lib/libvcc/Makefile.am index 628e321e6..bf352e019 100644 --- a/lib/libvcc/Makefile.am +++ b/lib/libvcc/Makefile.am @@ -13,6 +13,7 @@ libvcc_a_CFLAGS = \ libvcc_a_SOURCES = \ vcc_compile.h \ + vcc_namespace.h \ vcc_token_defs.h \ vcc_acl.c \ vcc_action.c \ diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py index 749eebd38..c0f4a4543 100755 --- a/lib/libvcc/generate.py +++ b/lib/libvcc/generate.py @@ -184,7 +184,7 @@ class vardef(object): ctyp = vcltypes[self.typ] # fo.write("\t{ \"%s\", %s,\n" % (nm, self.typ)) - fo.write("\tsym = VCC_MkSym(tl, \"%s\", " % self.nam) + fo.write("\tsym = VCC_MkSym(tl, \"%s\", SYM_MAIN," % self.nam) if self.typ == "HEADER": fo.write(" SYM_NONE, %d, %d);\n" % (self.vlo, self.vhi)) fo.write("\tAN(sym);\n") diff --git a/lib/libvcc/vcc_action.c b/lib/libvcc/vcc_action.c index 40ad52060..1ed104af5 100644 --- a/lib/libvcc/vcc_action.c +++ b/lib/libvcc/vcc_action.c @@ -49,7 +49,7 @@ vcc_act_call(struct vcc *tl, struct token *t, struct symbol *sym) (void)t; ExpectErr(tl, ID); t0 = tl->t; - sym = VCC_SymbolGet(tl, SYM_SUB, SYMTAB_CREATE, XREF_REF); + sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_SUB, SYMTAB_CREATE, XREF_REF); if (sym != NULL) { vcc_AddCall(tl, t0, sym); VCC_GlobalSymbol(sym, SUB, "VGC_function"); @@ -124,7 +124,7 @@ vcc_act_set(struct vcc *tl, struct token *t, struct symbol *sym) (void)t; ExpectErr(tl, ID); t = tl->t; - sym = VCC_SymbolGet(tl, SYM_VAR, SYMTAB_EXISTING, XREF_NONE); + sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_VAR, SYMTAB_EXISTING, XREF_NONE); ERRCHK(tl); AN(sym); vcc_AddUses(tl, t, tl->t, sym, XREF_WRITE); @@ -162,7 +162,7 @@ vcc_act_unset(struct vcc *tl, struct token *t, struct symbol *sym) /* XXX: Wrong, should use VCC_Expr(HEADER) */ ExpectErr(tl, ID); t = tl->t; - sym = VCC_SymbolGet(tl, SYM_VAR, SYMTAB_EXISTING, XREF_NONE); + sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_VAR, SYMTAB_EXISTING, XREF_NONE); ERRCHK(tl); AN(sym); if (sym->u_methods == 0) { @@ -281,7 +281,7 @@ vcc_act_return_vcl(struct vcc *tl) SkipToken(tl, '('); ExpectErr(tl, ID); - sym = VCC_SymbolGet(tl, SYM_VCL, SYMTAB_EXISTING, XREF_NONE); + sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_VCL, SYMTAB_EXISTING, XREF_NONE); ERRCHK(tl); AN(sym); if (sym->eval_priv == NULL) { @@ -402,7 +402,8 @@ vcc_act_synthetic(struct vcc *tl, struct token *t, struct symbol *sym) #define ACT(name, func, mask) \ do { \ const char pp[] = #name; \ - sym = VCC_MkSym(tl, pp, SYM_ACTION, VCL_LOW, VCL_HIGH); \ + sym = VCC_MkSym(tl, pp, SYM_MAIN, SYM_ACTION, VCL_LOW, \ + VCL_HIGH); \ AN(sym); \ sym->action = func; \ sym->action_mask = (mask); \ diff --git a/lib/libvcc/vcc_backend.c b/lib/libvcc/vcc_backend.c index 217b2366a..f6a268800 100644 --- a/lib/libvcc/vcc_backend.c +++ b/lib/libvcc/vcc_backend.c @@ -444,7 +444,7 @@ vcc_ParseHostDef(struct vcc *tl, const struct token *t_be, const char *vgcname) vcc_NextToken(tl); (void)vcc_default_probe(tl); } else { - pb = VCC_SymbolGet(tl, SYM_PROBE, + pb = VCC_SymbolGet(tl, SYM_MAIN, SYM_PROBE, SYMTAB_EXISTING, XREF_REF); ERRCHK(tl); AN(pb); diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index 78b08d04d..d726a80eb 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -839,7 +839,7 @@ VCC_New(void) for (i = 1; i < VCL_MET_MAX; i++) { sym = VCC_MkSym(tl, method_tab[i].name, - SYM_SUB, VCL_LOW, VCL_HIGH); + SYM_MAIN, SYM_SUB, VCL_LOW, VCL_HIGH); p = vcc_NewProc(tl, sym); p->method = &method_tab[i]; VSB_printf(p->cname, "VGC_function_%s", p->method->name); @@ -906,7 +906,7 @@ vcc_predef_vcl(struct vcc *vcc, const char *name) { struct symbol *sym; - sym = VCC_MkSym(vcc, name, SYM_VCL, VCL_LOW, VCL_HIGH); + sym = VCC_MkSym(vcc, name, SYM_MAIN, SYM_VCL, VCL_LOW, VCL_HIGH); AN(sym); sym->type = VCL; sym->r_methods = VCL_MET_RECV; diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index c63335f2f..d91bd00b4 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -132,6 +132,7 @@ struct type { #include "tbl/vcc_types.h" /*---------------------------------------------------------------------*/ + typedef const struct kind *vcc_kind_t; struct kind { @@ -146,6 +147,19 @@ struct kind { /*---------------------------------------------------------------------*/ +typedef const struct vcc_namespace *const vcc_ns_t; + +#define VCC_NAMESPACE(U, l) extern vcc_ns_t SYM_##U; +#include "vcc_namespace.h" + +enum vcc_namespace_e { +#define VCC_NAMESPACE(U, l) VCC_NAMESPACE_##U, +#include "vcc_namespace.h" + VCC_NAMESPACE__MAX +}; + +/*---------------------------------------------------------------------*/ + typedef void sym_expr_t(struct vcc *tl, struct expr **, struct token *, struct symbol *sym, vcc_type_t); typedef void sym_wildcard_t(struct vcc *, struct symbol *, struct symbol *); @@ -241,7 +255,7 @@ struct vcc { #define MGT_VCC(t, n, cc) t n; #include - struct symtab *syms; + struct symtab *syms[VCC_NAMESPACE__MAX]; struct inifinhead inifin; unsigned ninifin; @@ -369,7 +383,8 @@ void vcc_stevedore(struct vcc *vcc, const char *stv_name); /* vcc_symb.c */ void VCC_PrintCName(struct vsb *vsb, const char *b, const char *e); -struct symbol *VCC_MkSym(struct vcc *tl, const char *b, vcc_kind_t, int, int); +struct symbol *VCC_MkSym(struct vcc *tl, const char *b, vcc_ns_t, vcc_kind_t, + int, int); struct symxref { const char *name; }; extern const struct symxref XREF_NONE[1]; @@ -382,11 +397,11 @@ extern const struct symmode SYMTAB_CREATE[1]; extern const struct symmode SYMTAB_EXISTING[1]; extern const struct symmode SYMTAB_PARTIAL[1]; -struct symbol *VCC_SymbolGet(struct vcc *, vcc_kind_t, +struct symbol *VCC_SymbolGet(struct vcc *, vcc_ns_t, vcc_kind_t, const struct symmode *, const struct symxref *); typedef void symwalk_f(struct vcc *tl, const struct symbol *s); -void VCC_WalkSymbols(struct vcc *tl, symwalk_f *func, vcc_kind_t); +void VCC_WalkSymbols(struct vcc *tl, symwalk_f *func, vcc_ns_t, vcc_kind_t); /* vcc_token.c */ void vcc_Coord(const struct vcc *tl, struct vsb *vsb, diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index 91c9917a0..862dd6686 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -697,7 +697,8 @@ vcc_expr5(struct vcc *tl, struct expr **e, vcc_type_t fmt) switch (tl->t->tok) { case ID: t = tl->t; - sym = VCC_SymbolGet(tl, SYM_NONE, SYMTAB_PARTIAL, XREF_REF); + sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_NONE, SYMTAB_PARTIAL, + XREF_REF); ERRCHK(tl); AN(sym); if (sym->kind == SYM_FUNC && sym->type == VOID) { @@ -1070,7 +1071,7 @@ cmp_acl(struct vcc *tl, struct expr **e, const struct cmps *cp) vcc_NextToken(tl); vcc_ExpectVid(tl, "ACL"); - sym = VCC_SymbolGet(tl, SYM_ACL, SYMTAB_CREATE, XREF_REF); + sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_ACL, SYMTAB_CREATE, XREF_REF); ERRCHK(tl); AN(sym); VCC_GlobalSymbol(sym, ACL, ACL_SYMBOL_PREFIX); @@ -1483,31 +1484,31 @@ vcc_Expr_Init(struct vcc *tl) { struct symbol *sym; - sym = VCC_MkSym(tl, "regsub", SYM_FUNC, VCL_LOW, VCL_HIGH); + sym = VCC_MkSym(tl, "regsub", SYM_MAIN, SYM_FUNC, VCL_LOW, VCL_HIGH); AN(sym); sym->type = STRING; sym->eval = vcc_Eval_Regsub; sym->eval_priv = NULL; - sym = VCC_MkSym(tl, "regsuball", SYM_FUNC, VCL_LOW, VCL_HIGH); + sym = VCC_MkSym(tl, "regsuball", SYM_MAIN, SYM_FUNC, VCL_LOW, VCL_HIGH); AN(sym); sym->type = STRING; sym->eval = vcc_Eval_Regsub; sym->eval_priv = sym; - sym = VCC_MkSym(tl, "true", SYM_FUNC, VCL_LOW, VCL_HIGH); + sym = VCC_MkSym(tl, "true", SYM_MAIN, SYM_FUNC, VCL_LOW, VCL_HIGH); AN(sym); sym->type = BOOL; sym->eval = vcc_Eval_BoolConst; sym->eval_priv = sym; - sym = VCC_MkSym(tl, "false", SYM_FUNC, VCL_LOW, VCL_HIGH); + sym = VCC_MkSym(tl, "false", SYM_MAIN, SYM_FUNC, VCL_LOW, VCL_HIGH); AN(sym); sym->type = BOOL; sym->eval = vcc_Eval_BoolConst; sym->eval_priv = NULL; - sym = VCC_MkSym(tl, "default", SYM_FUNC, VCL_LOW, VCL_HIGH); + sym = VCC_MkSym(tl, "default", SYM_MAIN, SYM_FUNC, VCL_LOW, VCL_HIGH); AN(sym); sym->type = BACKEND; // ... can also (sometimes) deliver PROBE sym->eval = vcc_Eval_Default; diff --git a/lib/libvcc/vcc_namespace.h b/lib/libvcc/vcc_namespace.h new file mode 100644 index 000000000..ee9ba2333 --- /dev/null +++ b/lib/libvcc/vcc_namespace.h @@ -0,0 +1,38 @@ +/*- + * Copyright (c) 2020 Varnish Software AS + * All rights reserved. + * + * Author: Dridi Boukelmoune + * + * SPDX-License-Identifier: BSD-2-Clause + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +/*lint -save -e525 -e539 */ + +VCC_NAMESPACE(MAIN, main) +VCC_NAMESPACE(TYPE, type) +#undef VCC_NAMESPACE + +/*lint -restore */ diff --git a/lib/libvcc/vcc_parse.c b/lib/libvcc/vcc_parse.c index f5dc8c364..ec7b5e614 100644 --- a/lib/libvcc/vcc_parse.c +++ b/lib/libvcc/vcc_parse.c @@ -182,8 +182,8 @@ vcc_Compound(struct vcc *tl) tl->err = 1; return; case ID: - sym = VCC_SymbolGet(tl, SYM_NONE, SYMTAB_NOERR, - XREF_NONE); + sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_NONE, + SYMTAB_NOERR, XREF_NONE); if (sym == NULL) { VSB_printf(tl->sb, "Symbol not found.\n"); vcc_ErrWhere(tl, tl->t); @@ -228,7 +228,7 @@ vcc_ParseFunction(struct vcc *tl) ERRCHK(tl); t = tl->t; - sym = VCC_SymbolGet(tl, SYM_SUB, SYMTAB_CREATE, XREF_DEF); + sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_SUB, SYMTAB_CREATE, XREF_DEF); ERRCHK(tl); AN(sym); p = sym->proc; @@ -432,5 +432,6 @@ vcc_Parse_Init(struct vcc *tl) struct toplev *tp; for (tp = toplev; tp->name != NULL; tp++) - AN(VCC_MkSym(tl, tp->name, SYM_RESERVED, tp->vcllo, tp->vclhi)); + AN(VCC_MkSym(tl, tp->name, SYM_MAIN, SYM_RESERVED, + tp->vcllo, tp->vclhi)); } diff --git a/lib/libvcc/vcc_storage.c b/lib/libvcc/vcc_storage.c index a0b160f37..bb08bc3d3 100644 --- a/lib/libvcc/vcc_storage.c +++ b/lib/libvcc/vcc_storage.c @@ -70,7 +70,7 @@ vcc_stevedore(struct vcc *vcc, const char *stv_name) CHECK_OBJ_NOTNULL(vcc, VCC_MAGIC); bprintf(buf, "storage.%s", stv_name); - sym = VCC_MkSym(vcc, buf, SYM_VAR, VCL_LOW, VCL_41); + sym = VCC_MkSym(vcc, buf, SYM_MAIN, SYM_VAR, VCL_LOW, VCL_41); AN(sym); sym->type = STEVEDORE; sym->eval = vcc_Eval_Var; diff --git a/lib/libvcc/vcc_symb.c b/lib/libvcc/vcc_symb.c index 6784ef48b..93983c098 100644 --- a/lib/libvcc/vcc_symb.c +++ b/lib/libvcc/vcc_symb.c @@ -40,8 +40,28 @@ #include "vct.h" /*--------------------------------------------------------------------*/ + #define VCC_KIND(U,l) const struct kind SYM_##U[1] = {{ KIND_MAGIC, #l}}; #include "tbl/symbol_kind.h" + +/*--------------------------------------------------------------------*/ + +struct vcc_namespace { + unsigned magic; +#define VCC_NAMESPACE_MAGIC 0x27b842f4 + const char *name; + enum vcc_namespace_e id; +}; + +#define VCC_NAMESPACE(U, l) \ + static const struct vcc_namespace sym_##l = { \ + VCC_NAMESPACE_MAGIC, \ + #l, \ + VCC_NAMESPACE_##U \ + }; \ + vcc_ns_t SYM_##U = &sym_##l; +#include "vcc_namespace.h" + /*--------------------------------------------------------------------*/ struct symtab { @@ -235,7 +255,7 @@ const struct symmode SYMTAB_EXISTING[1] = {{"Symbol not found"}}; const struct symmode SYMTAB_PARTIAL[1] = {{"Symbol not found"}}; struct symbol * -VCC_SymbolGet(struct vcc *tl, vcc_kind_t kind, +VCC_SymbolGet(struct vcc *tl, vcc_ns_t ns, vcc_kind_t kind, const struct symmode *e, const struct symxref *x) { struct symtab *st, *st2 = NULL; @@ -243,6 +263,8 @@ VCC_SymbolGet(struct vcc *tl, vcc_kind_t kind, struct token *t0, *tn, *tn1, *tn2 = NULL; AN(tl); + CHECK_OBJ_NOTNULL(ns, VCC_NAMESPACE_MAGIC); + CHECK_OBJ_NOTNULL(kind, KIND_MAGIC); AN(e); AN(x); AN(x->name); @@ -256,7 +278,7 @@ VCC_SymbolGet(struct vcc *tl, vcc_kind_t kind, return (NULL); } - st = tl->syms; + st = tl->syms[ns->id]; t0 = tl->t; tn = tl->t; while (1) { @@ -343,19 +365,21 @@ VCC_SymbolGet(struct vcc *tl, vcc_kind_t kind, } struct symbol * -VCC_MkSym(struct vcc *tl, const char *b, vcc_kind_t kind, int vlo, int vhi) +VCC_MkSym(struct vcc *tl, const char *b, vcc_ns_t ns, vcc_kind_t kind, + int vlo, int vhi) { struct symtab *st; struct symbol *sym; AN(tl); AN(b); + CHECK_OBJ_NOTNULL(ns, VCC_NAMESPACE_MAGIC); CHECK_OBJ_NOTNULL(kind, KIND_MAGIC); assert(vlo <= vhi); - if (tl->syms == NULL) - tl->syms = vcc_symtab_new(""); - st = vcc_symtab_str(tl->syms, b, NULL); + if (tl->syms[ns->id] == NULL) + tl->syms[ns->id] = vcc_symtab_new(""); + st = vcc_symtab_str(tl->syms[ns->id], b, NULL); AN(st); sym = vcc_sym_in_tab(tl, st, kind, vlo, vhi); AZ(sym); @@ -386,10 +410,11 @@ vcc_walksymbols(struct vcc *tl, const struct symtab *root, } void -VCC_WalkSymbols(struct vcc *tl, symwalk_f *func, vcc_kind_t kind) +VCC_WalkSymbols(struct vcc *tl, symwalk_f *func, vcc_ns_t ns, vcc_kind_t kind) { - vcc_walksymbols(tl, tl->syms, func, kind); + CHECK_OBJ_NOTNULL(ns, VCC_NAMESPACE_MAGIC); + vcc_walksymbols(tl, tl->syms[ns->id], func, kind); } void @@ -434,7 +459,7 @@ VCC_HandleSymbol(struct vcc *tl, vcc_type_t fmt, const char *pfx) assert(kind != SYM_NONE); t = tl->t; - sym = VCC_SymbolGet(tl, SYM_NONE, SYMTAB_NOERR, XREF_NONE); + sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_NONE, SYMTAB_NOERR, XREF_NONE); if (sym != NULL && sym->def_b != NULL && kind == sym->kind) { p = sym->kind->name; VSB_printf(tl->sb, "%c%s '%.*s' redefined.\n", @@ -459,7 +484,8 @@ VCC_HandleSymbol(struct vcc *tl, vcc_type_t fmt, const char *pfx) return (sym); } if (sym == NULL) - sym = VCC_SymbolGet(tl, kind, SYMTAB_CREATE, XREF_NONE); + sym = VCC_SymbolGet(tl, SYM_MAIN, kind, SYMTAB_CREATE, + XREF_NONE); if (sym == NULL) return (NULL); AN(sym); diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c index fb41197a3..fbe422435 100644 --- a/lib/libvcc/vcc_vmod.c +++ b/lib/libvcc/vcc_vmod.c @@ -252,7 +252,8 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) VSB_clear(buf); VSB_printf(buf, "%s.%s", msym->name, vv2->value); AZ(VSB_finish(buf)); - fsym = VCC_MkSym(tl, VSB_data(buf), kind, VCL_LOW, VCL_HIGH); + fsym = VCC_MkSym(tl, VSB_data(buf), SYM_MAIN, kind, + VCL_LOW, VCL_HIGH); AN(fsym); if (kind == SYM_FUNC) { @@ -296,7 +297,7 @@ vcc_ParseImport(struct vcc *tl) } tmod = tl->t; - msym = VCC_SymbolGet(tl, SYM_VMOD, SYMTAB_CREATE, XREF_NONE); + msym = VCC_SymbolGet(tl, SYM_MAIN, SYM_VMOD, SYMTAB_CREATE, XREF_NONE); ERRCHK(tl); AN(msym); @@ -454,7 +455,8 @@ vcc_Act_New(struct vcc *tl, struct token *t, struct symbol *sym) SkipToken(tl, '='); ExpectErr(tl, ID); - osym = VCC_SymbolGet(tl, SYM_OBJECT, SYMTAB_EXISTING, XREF_NONE); + osym = VCC_SymbolGet(tl, SYM_MAIN, SYM_OBJECT, SYMTAB_EXISTING, + XREF_NONE); ERRCHK(tl); AN(osym); CAST_OBJ_NOTNULL(vv, osym->eval_priv, VJSN_VAL_MAGIC); diff --git a/lib/libvcc/vcc_xref.c b/lib/libvcc/vcc_xref.c index 483805f3d..b0228ec26 100644 --- a/lib/libvcc/vcc_xref.c +++ b/lib/libvcc/vcc_xref.c @@ -95,7 +95,7 @@ int vcc_CheckReferences(struct vcc *tl) { - VCC_WalkSymbols(tl, vcc_checkref, SYM_NONE); + VCC_WalkSymbols(tl, vcc_checkref, SYM_MAIN, SYM_NONE); return (tl->err); } @@ -248,7 +248,7 @@ int vcc_CheckAction(struct vcc *tl) { - VCC_WalkSymbols(tl, vcc_checkaction, SYM_SUB); + VCC_WalkSymbols(tl, vcc_checkaction, SYM_MAIN, SYM_SUB); return (tl->err); } @@ -352,7 +352,7 @@ int vcc_CheckUses(struct vcc *tl) { - VCC_WalkSymbols(tl, vcc_checkuses, SYM_SUB); + VCC_WalkSymbols(tl, vcc_checkuses, SYM_MAIN, SYM_SUB); return (tl->err); } @@ -388,7 +388,7 @@ void VCC_InstanceInfo(struct vcc *tl) { Fc(tl, 0, "\nconst struct vpi_ii VGC_instance_info[] = {\n"); - VCC_WalkSymbols(tl, vcc_instance_info, SYM_INSTANCE); + VCC_WalkSymbols(tl, vcc_instance_info, SYM_MAIN, SYM_INSTANCE); Fc(tl, 0, "\t{ .p = NULL, .name = \"\" }\n"); Fc(tl, 0, "};\n"); } @@ -414,8 +414,8 @@ VCC_XrefTable(struct vcc *tl) { Fc(tl, 0, "\n/*\n * Symbol Table\n *\n"); - VCC_WalkSymbols(tl, vcc_xreftable_len, SYM_NONE); - VCC_WalkSymbols(tl, vcc_xreftable, SYM_NONE); + VCC_WalkSymbols(tl, vcc_xreftable_len, SYM_MAIN, SYM_NONE); + VCC_WalkSymbols(tl, vcc_xreftable, SYM_MAIN, SYM_NONE); Fc(tl, 0, "*/\n\n"); } From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:07 +0000 (UTC) Subject: [master] a82916eac Register types and type methods as symbols Message-ID: <20200702124907.A4CE3A598B@lists.varnish-cache.org> commit a82916eac970cf1c1a53b4f4df57ff61120b5270 Author: Dridi Boukelmoune Date: Fri Jun 5 07:55:54 2020 +0200 Register types and type methods as symbols The SYM_METHOD kind of symbol was already present but never used so it just found its purpose. They appear in their dedicated symbol table: /* * Symbol Table MAIN * * reserved VOID 41 41 acl * reserved VOID 41 41 backend * [...] */ /* * Symbol Table TYPE * * none VOID 40 41 BACKEND * method BACKEND 40 41 BACKEND.resolve * none VOID 40 41 STEVEDORE * method BYTES 40 41 STEVEDORE.free_space * method BOOL 40 41 STEVEDORE.happy * method BYTES 40 41 STEVEDORE.used_space * none VOID 40 41 STRINGS * method STRING 40 41 STRINGS.lower * method STRING 40 41 STRINGS.upper */ Unlike VMOD functions or object methods, native type methods are never invoked as a standalone statement: strings.upper(); They are only evaluated atop an expression: (string expression).upper(); So any VMOD named after a type, like vmod_blob, should not conflict with native type methods in the symbol table. Unless a symbol already exists in the MAIN namespace, like reserved keywords acl and backend. diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index d726a80eb..caf735f53 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -647,6 +647,8 @@ vcc_CompileSource(struct vcc *tl, struct source *sp, const char *jfile) vcc_Var_Init(tl); + vcc_Type_Init(tl); + Fh(tl, 0, "\nextern const struct VCL_conf VCL_conf;\n"); /* Register and lex the main source */ diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index d91bd00b4..1a4cb44db 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -110,6 +110,8 @@ typedef const struct type *vcc_type_t; * XXX: type methods might move in a more comprehensive direction. */ struct vcc_method { + unsigned magic; +#define VCC_METHOD_MAGIC 0x594108cd vcc_type_t type; const char *name; const char *impl; @@ -423,6 +425,7 @@ void vcc_AddToken(struct vcc *tl, unsigned tok, const char *b, /* vcc_types.c */ vcc_type_t VCC_Type(const char *p); +void vcc_Type_Init(struct vcc *tl); /* vcc_var.c */ sym_wildcard_t vcc_Var_Wildcard; diff --git a/lib/libvcc/vcc_types.c b/lib/libvcc/vcc_types.c index 4f9db3b80..1292ec27a 100644 --- a/lib/libvcc/vcc_types.c +++ b/lib/libvcc/vcc_types.c @@ -43,8 +43,9 @@ const struct type ACL[1] = {{ }}; static const struct vcc_method backend_methods[] = { - { BACKEND, "resolve", "VRT_DirectorResolve(ctx, \v1)", 1 }, - { NULL }, + { VCC_METHOD_MAGIC, BACKEND, "resolve", + "VRT_DirectorResolve(ctx, \v1)", 1 }, + { VCC_METHOD_MAGIC, NULL }, }; const struct type BACKEND[1] = {{ @@ -133,9 +134,9 @@ const struct type REAL[1] = {{ static const struct vcc_method stevedore_methods[] = { #define VRTSTVVAR(nm, vtype, ctype, dval) \ - { vtype, #nm, "VRT_stevedore_" #nm "(\v1)", 0}, + { VCC_METHOD_MAGIC, vtype, #nm, "VRT_stevedore_" #nm "(\v1)", 0}, #include "tbl/vrt_stv_var.h" - { NULL }, + { VCC_METHOD_MAGIC, NULL }, }; const struct type STEVEDORE[1] = {{ @@ -159,9 +160,11 @@ const struct type STRANDS[1] = {{ }}; static const struct vcc_method strings_methods[] = { - { STRING, "upper", "VRT_UpperLowerStrands(ctx, \vT, 1)", 1 }, - { STRING, "lower", "VRT_UpperLowerStrands(ctx, \vT, 0)", 1 }, - { NULL }, + { VCC_METHOD_MAGIC, STRING, "upper", + "VRT_UpperLowerStrands(ctx, \vT, 1)", 1 }, + { VCC_METHOD_MAGIC, STRING, "lower", + "VRT_UpperLowerStrands(ctx, \vT, 0)", 1 }, + { VCC_METHOD_MAGIC, NULL }, }; const struct type STRINGS[1] = {{ @@ -207,3 +210,43 @@ VCC_Type(const char *p) return (NULL); } +static void +vcc_type_init(struct vcc *tl, vcc_type_t type) +{ + const struct vcc_method *vm; + struct symbol *sym; + struct vsb *buf; + + /* NB: Don't bother even creating a type symbol if there are no + * methods attached to it. + */ + if (type->methods == NULL) + return; + + buf = VSB_new_auto(); + AN(buf); + AN(VCC_MkSym(tl, type->name, SYM_TYPE, SYM_NONE, VCL_LOW, VCL_HIGH)); + + for (vm = type->methods; vm->type != NULL; vm++) { + VSB_printf(buf, "%s.%s", type->name, vm->name); + AZ(VSB_finish(buf)); + sym = VCC_MkSym(tl, VSB_data(buf), SYM_TYPE, SYM_METHOD, + VCL_LOW, VCL_HIGH); + VSB_clear(buf); + if (tl->err) + break; + AN(sym); + sym->type = vm->type; + sym->eval_priv = vm; + } + + VSB_destroy(&buf); +} + +void +vcc_Type_Init(struct vcc *tl) +{ + +#define VCC_TYPE(UC, lc) vcc_type_init(tl, UC); +#include "tbl/vcc_types.h" +} diff --git a/lib/libvcc/vcc_xref.c b/lib/libvcc/vcc_xref.c index b0228ec26..0bd9f2aed 100644 --- a/lib/libvcc/vcc_xref.c +++ b/lib/libvcc/vcc_xref.c @@ -358,21 +358,6 @@ vcc_CheckUses(struct vcc *tl) /*---------------------------------------------------------------------*/ -static int sym_type_len = 0; - -static void v_matchproto_(symwalk_f) -vcc_xreftable_len(struct vcc *tl, const struct symbol *sym) -{ - int len; - - (void)tl; - CHECK_OBJ_NOTNULL(sym, SYMBOL_MAGIC); - CHECK_OBJ_NOTNULL(sym->type, TYPE_MAGIC); - len = strlen(sym->type->name); - if (sym_type_len < len) - sym_type_len = len; -} - static void v_matchproto_(symwalk_f) vcc_instance_info(struct vcc *tl, const struct symbol *sym) { @@ -393,6 +378,23 @@ VCC_InstanceInfo(struct vcc *tl) Fc(tl, 0, "};\n"); } +/*---------------------------------------------------------------------*/ + +static int sym_type_len; + +static void v_matchproto_(symwalk_f) +vcc_xreftable_len(struct vcc *tl, const struct symbol *sym) +{ + int len; + + (void)tl; + CHECK_OBJ_NOTNULL(sym, SYMBOL_MAGIC); + CHECK_OBJ_NOTNULL(sym->type, TYPE_MAGIC); + len = strlen(sym->type->name); + if (sym_type_len < len) + sym_type_len = len; +} + static void v_matchproto_(symwalk_f) vcc_xreftable(struct vcc *tl, const struct symbol *sym) { @@ -413,10 +415,13 @@ void VCC_XrefTable(struct vcc *tl) { - Fc(tl, 0, "\n/*\n * Symbol Table\n *\n"); - VCC_WalkSymbols(tl, vcc_xreftable_len, SYM_MAIN, SYM_NONE); - VCC_WalkSymbols(tl, vcc_xreftable, SYM_MAIN, SYM_NONE); +#define VCC_NAMESPACE(U, l) \ + Fc(tl, 0, "\n/*\n * Symbol Table " #U "\n *\n"); \ + sym_type_len = 0; \ + VCC_WalkSymbols(tl, vcc_xreftable_len, SYM_##U, SYM_NONE); \ + VCC_WalkSymbols(tl, vcc_xreftable, SYM_##U, SYM_NONE); \ Fc(tl, 0, "*/\n\n"); +#include "vcc_namespace.h" } /*--------------------------------------------------------------------- From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:07 +0000 (UTC) Subject: [master] 2fb26ab7e Parse .method() expressions via the symbol table Message-ID: <20200702124907.C3C48A59A0@lists.varnish-cache.org> commit 2fb26ab7ee2a842ecca584975ccebdf411abca95 Author: Dridi Boukelmoune Date: Fri Nov 29 07:43:08 2019 +0100 Parse .method() expressions via the symbol table Technically the lookup is TYPE.METHOD so it's not as simple as calling VCC_SymbolGet() with the correct namespace. diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 1a4cb44db..874c63c15 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -402,6 +402,8 @@ extern const struct symmode SYMTAB_PARTIAL[1]; struct symbol *VCC_SymbolGet(struct vcc *, vcc_ns_t, vcc_kind_t, const struct symmode *, const struct symxref *); +struct symbol *VCC_TypeSymbol(struct vcc *, vcc_kind_t, vcc_type_t); + typedef void symwalk_f(struct vcc *tl, const struct symbol *s); void VCC_WalkSymbols(struct vcc *tl, symwalk_f *func, vcc_ns_t, vcc_kind_t); diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index 862dd6686..be0ee4aa3 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -823,6 +823,7 @@ static void vcc_expr4(struct vcc *tl, struct expr **e, vcc_type_t fmt) { const struct vcc_method *vm; + const struct symbol *sym; *e = NULL; vcc_expr5(tl, e, fmt); @@ -832,14 +833,8 @@ vcc_expr4(struct vcc *tl, struct expr **e, vcc_type_t fmt) vcc_NextToken(tl); ExpectErr(tl, ID); - vm = (*e)->fmt->methods; - while (vm != NULL && vm->type != NULL) { - if (vcc_IdIs(tl->t, vm->name)) - break; - vm++; - } - - if (vm == NULL || vm->type == NULL) { + sym = VCC_TypeSymbol(tl, SYM_METHOD, (*e)->fmt); + if (sym == NULL) { VSB_cat(tl->sb, "Unknown property "); vcc_ErrToken(tl, tl->t); VSB_printf(tl->sb, @@ -847,6 +842,8 @@ vcc_expr4(struct vcc *tl, struct expr **e, vcc_type_t fmt) vcc_ErrWhere(tl, tl->t); return; } + CAST_OBJ_NOTNULL(vm, sym->eval_priv, VCC_METHOD_MAGIC); + vcc_NextToken(tl); *e = vcc_expr_edit(tl, vm->type, vm->impl, *e, NULL); ERRCHK(tl); diff --git a/lib/libvcc/vcc_symb.c b/lib/libvcc/vcc_symb.c index 93983c098..7630f4e14 100644 --- a/lib/libvcc/vcc_symb.c +++ b/lib/libvcc/vcc_symb.c @@ -364,6 +364,31 @@ VCC_SymbolGet(struct vcc *tl, vcc_ns_t ns, vcc_kind_t kind, return (sym); } +struct symbol * +VCC_TypeSymbol(struct vcc *tl, vcc_kind_t kind, vcc_type_t type) +{ + struct token t[1], *t0; + struct symbol *sym; + struct vsb *buf; + + buf = VSB_new_auto(); + AN(buf); + VSB_printf(buf, "%s.%.*s", type->name, PF(tl->t)); + AZ(VSB_finish(buf)); + + /* NB: we create a fake token but errors are handled by the caller. */ + memcpy(t, tl->t, sizeof *t); + t->b = VSB_data(buf); + t->e = t->b + VSB_len(buf); + + t0 = tl->t; + tl->t = t; + sym = VCC_SymbolGet(tl, SYM_TYPE, kind, SYMTAB_NOERR, XREF_NONE); + tl->t = t0; + VSB_destroy(&buf); + return (sym); +} + struct symbol * VCC_MkSym(struct vcc *tl, const char *b, vcc_ns_t ns, vcc_kind_t kind, int vlo, int vhi) From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:07 +0000 (UTC) Subject: [master] e673718d1 Bury the knowledge of type methods in type code Message-ID: <20200702124907.E2933A59B6@lists.varnish-cache.org> commit e673718d19d2faa244858fc708fffabf594508b9 Author: Dridi Boukelmoune Date: Fri Nov 29 07:52:37 2019 +0100 Bury the knowledge of type methods in type code This enables the possibility for other kinds of SYM_METHOD symbols with different parsing rules. diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 874c63c15..14f2975a6 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -100,24 +100,6 @@ struct token { typedef const struct type *vcc_type_t; -/* - * A type attribute is information already existing, requiring no processing - * or resource usage. - * - * A type method is a call and may do significant processing, change things, - * eat workspace etc. - * - * XXX: type methods might move in a more comprehensive direction. - */ -struct vcc_method { - unsigned magic; -#define VCC_METHOD_MAGIC 0x594108cd - vcc_type_t type; - const char *name; - const char *impl; - int func; -}; - struct type { unsigned magic; #define TYPE_MAGIC 0xfae932d9 @@ -353,6 +335,7 @@ void vcc_Expr_Init(struct vcc *tl); sym_expr_t vcc_Eval_Var; sym_expr_t vcc_Eval_Handle; sym_expr_t vcc_Eval_SymFunc; +sym_expr_t vcc_Eval_TypeMethod; void vcc_Eval_Func(struct vcc *, const struct vjsn_val *, const char *, const struct symbol *); void VCC_GlobalSymbol(struct symbol *, vcc_type_t fmt, const char *pfx); @@ -427,6 +410,7 @@ void vcc_AddToken(struct vcc *tl, unsigned tok, const char *b, /* vcc_types.c */ vcc_type_t VCC_Type(const char *p); +const char * VCC_Type_EvalMethod(struct vcc *, const struct symbol *); void vcc_Type_Init(struct vcc *tl); /* vcc_var.c */ diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index be0ee4aa3..71c618a83 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -819,11 +819,23 @@ vcc_expr5(struct vcc *tl, struct expr **e, vcc_type_t fmt) * Expr5 [ '.' (type_attribute | type_method()) ]* */ +void +vcc_Eval_TypeMethod(struct vcc *tl, struct expr **e, struct token *t, + struct symbol *sym, vcc_type_t fmt) +{ + const char *impl; + + (void)t; + impl = VCC_Type_EvalMethod(tl, sym); + ERRCHK(tl); + AN(impl); + *e = vcc_expr_edit(tl, fmt, impl, *e, NULL); +} + static void vcc_expr4(struct vcc *tl, struct expr **e, vcc_type_t fmt) { - const struct vcc_method *vm; - const struct symbol *sym; + struct symbol *sym; *e = NULL; vcc_expr5(tl, e, fmt); @@ -842,21 +854,14 @@ vcc_expr4(struct vcc *tl, struct expr **e, vcc_type_t fmt) vcc_ErrWhere(tl, tl->t); return; } - CAST_OBJ_NOTNULL(vm, sym->eval_priv, VCC_METHOD_MAGIC); - vcc_NextToken(tl); - *e = vcc_expr_edit(tl, vm->type, vm->impl, *e, NULL); + AN(sym->eval); + sym->eval(tl, e, tl->t, sym, sym->type); ERRCHK(tl); if ((*e)->fmt == STRING) { (*e)->fmt = STRINGS; (*e)->nstr = 1; } - if (vm->func) { - ExpectErr(tl, '('); - vcc_NextToken(tl); - ExpectErr(tl, ')'); - vcc_NextToken(tl); - } } } diff --git a/lib/libvcc/vcc_types.c b/lib/libvcc/vcc_types.c index 1292ec27a..b49eaa9f6 100644 --- a/lib/libvcc/vcc_types.c +++ b/lib/libvcc/vcc_types.c @@ -36,6 +36,24 @@ #include "vcc_compile.h" +/* + * A type attribute is information already existing, requiring no processing + * or resource usage. + * + * A type method is a call and may do significant processing, change things, + * eat workspace etc. + * + * XXX: type methods might move in a more comprehensive direction. + */ +struct vcc_method { + unsigned magic; +#define VCC_METHOD_MAGIC 0x594108cd + vcc_type_t type; + const char *name; + const char *impl; + int func; +}; + const struct type ACL[1] = {{ .magic = TYPE_MAGIC, .name = "ACL", @@ -237,12 +255,37 @@ vcc_type_init(struct vcc *tl, vcc_type_t type) break; AN(sym); sym->type = vm->type; + sym->eval = vcc_Eval_TypeMethod; sym->eval_priv = vm; } VSB_destroy(&buf); } +const char * +VCC_Type_EvalMethod(struct vcc *tl, const struct symbol *sym) +{ + const struct vcc_method *vm; + + AN(sym); + AN(sym->kind == SYM_METHOD); + CAST_OBJ_NOTNULL(vm, sym->eval_priv, VCC_METHOD_MAGIC); + + vcc_NextToken(tl); + if (vm->func) { + Expect(tl, '('); + if (tl->err) + return (NULL); + vcc_NextToken(tl); + Expect(tl, ')'); + if (tl->err) + return (NULL); + vcc_NextToken(tl); + } + + return (vm->impl); +} + void vcc_Type_Init(struct vcc *tl) { From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:08 +0000 (UTC) Subject: [master] b437976db Create types for VMOD objects Message-ID: <20200702124908.208CFA59C8@lists.varnish-cache.org> commit b437976db3d8664ceab0ca235e32428e31f7f49d Author: Dridi Boukelmoune Date: Fri Nov 29 15:55:14 2019 +0100 Create types for VMOD objects Now they appear as ${vmod}.${constructor} instead of VOID in the symbol table. The effective type is based on the VMOD's name, regardless of whether it was imported with an alias or not. diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index caf735f53..d24ebf5e1 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -826,6 +826,7 @@ VCC_New(void) VTAILQ_INIT(&tl->procs); VTAILQ_INIT(&tl->sym_objects); VTAILQ_INIT(&tl->sym_vmods); + VTAILQ_INIT(&tl->vmod_objects); tl->nsources = 0; diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 14f2975a6..1fda4241b 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -76,6 +76,7 @@ struct expr; struct vcc; struct vjsn_val; struct symbol; +struct vmod_obj; struct source { VTAILQ_ENTRY(source) list; @@ -276,6 +277,7 @@ struct vcc { VTAILQ_HEAD(, symbol) sym_objects; VTAILQ_HEAD(, symbol) sym_vmods; + VTAILQ_HEAD(, vmod_obj) vmod_objects; unsigned unique; unsigned vmod_count; diff --git a/lib/libvcc/vcc_symb.c b/lib/libvcc/vcc_symb.c index 7630f4e14..ea2d78d4c 100644 --- a/lib/libvcc/vcc_symb.c +++ b/lib/libvcc/vcc_symb.c @@ -386,6 +386,7 @@ VCC_TypeSymbol(struct vcc *tl, vcc_kind_t kind, vcc_type_t type) sym = VCC_SymbolGet(tl, SYM_TYPE, kind, SYMTAB_NOERR, XREF_NONE); tl->t = t0; VSB_destroy(&buf); + return (sym); } diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c index fbe422435..acef1511b 100644 --- a/lib/libvcc/vcc_vmod.c +++ b/lib/libvcc/vcc_vmod.c @@ -49,6 +49,14 @@ struct vmod_open { const char *err; }; +struct vmod_obj { + unsigned magic; +#define VMOD_OBJ_MAGIC 0x349885f8 + char *name; + struct type type[1]; + VTAILQ_ENTRY(vmod_obj) list; +}; + static int vcc_path_dlopen(void *priv, const char *fn) { @@ -201,6 +209,29 @@ vcc_vmod_kind(const char *type) return (SYM_NONE); } +static void +vcc_VmodObject(struct vcc *tl, struct symbol *sym) +{ + struct vmod_obj *obj; + struct vsb *buf; + + buf = VSB_new_auto(); + AN(buf); + + VSB_printf(buf, "%s.%s", sym->vmod_name, sym->name); + AZ(VSB_finish(buf)); + + ALLOC_OBJ(obj, VMOD_OBJ_MAGIC); + AN(obj); + REPLACE(obj->name, VSB_data(buf)); + + INIT_OBJ(obj->type, TYPE_MAGIC); + obj->type->name = obj->name; + sym->type = obj->type; + VTAILQ_INSERT_TAIL(&tl->vmod_objects, obj, list); + VSB_destroy(&buf); +} + static void vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) { @@ -265,6 +296,7 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) assert(kind == SYM_OBJECT); fsym->eval_priv = vv2; fsym->vmod_name = msym->vmod_name; + vcc_VmodObject(tl, fsym); } } From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:08 +0000 (UTC) Subject: [master] 787a505a5 Add VMOD object methods to the symbol table Message-ID: <20200702124908.47370A59CE@lists.varnish-cache.org> commit 787a505a545b56538a54da4b30f74c7faaa34c40 Author: Dridi Boukelmoune Date: Fri Nov 29 16:40:25 2019 +0100 Add VMOD object methods to the symbol table Although at this point we are still using the per-instance symbols from the table. For example with the following: new fb1 = directors.fallback(); new fb2 = directors.fallback(); We will get the following entries in the symbol table: object directors.fallback 41 41 directors.fallback method VOID 40 41 directors.fallback.add_backend method BACKEND 40 41 directors.fallback.backend method VOID 40 41 directors.fallback.remove_backend instance INSTANCE 41 41 fb1 func VOID 40 41 fb1.add_backend func BACKEND 40 41 fb1.backend func VOID 40 41 fb1.remove_backend instance INSTANCE 41 41 fb2 func VOID 40 41 fb2.add_backend func BACKEND 40 41 fb2.backend func VOID 40 41 fb2.remove_backend As long as the func symbols exist, and the instance symbols have the generic type INSTANCE, the expression parser will not try to evaluate the method symbols. But the type-based symbols aren't ready to be evaluated yet so we can have peaceful cohabitation for now. This makes this part of the code even more complicated but it will eventually straighten up once the SYM_FUNC symbols are gone. diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c index acef1511b..0fc43bab2 100644 --- a/lib/libvcc/vcc_vmod.c +++ b/lib/libvcc/vcc_vmod.c @@ -244,7 +244,7 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) if (msym->kind == SYM_VMOD) { CAST_OBJ_NOTNULL(vj, msym->eval_priv, VJSN_MAGIC); vv = VTAILQ_FIRST(&vj->value->children); - } else if (msym->kind == SYM_INSTANCE) { + } else if (msym->kind == SYM_INSTANCE || msym->kind == SYM_OBJECT) { CAST_OBJ_NOTNULL(vv, msym->eval_priv, VJSN_VAL_MAGIC); } else { WRONG("symbol kind"); @@ -281,7 +281,8 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) */ VSB_clear(buf); - VSB_printf(buf, "%s.%s", msym->name, vv2->value); + VCC_SymName(buf, msym); + VSB_printf(buf, ".%s", vv2->value); AZ(VSB_finish(buf)); fsym = VCC_MkSym(tl, VSB_data(buf), SYM_MAIN, kind, VCL_LOW, VCL_HIGH); @@ -289,14 +290,19 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) if (kind == SYM_FUNC) { func_sym(fsym, msym->vmod_name, VTAILQ_NEXT(vv2, list)); - /* XXX: until we use SYM_METHOD, string check. */ - if (!strcmp(vv1->value, "$METHOD")) + /* XXX: until we use SYM_METHOD only, string check. */ + if (!strcmp(vv1->value, "$METHOD")) { fsym->extra = msym->rname; + /* XXX: cohabitation temporary hack */ + if (msym->kind == SYM_OBJECT) + fsym->kind = SYM_METHOD; + } } else { assert(kind == SYM_OBJECT); fsym->eval_priv = vv2; fsym->vmod_name = msym->vmod_name; vcc_VmodObject(tl, fsym); + vcc_VmodSymbols(tl, fsym); } } From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:08 +0000 (UTC) Subject: [master] 7cd19157b Give the dedicated type to VMOD object instances Message-ID: <20200702124908.71E66A59DE@lists.varnish-cache.org> commit 7cd19157b858d7a9b17d98b591b4e640ddaa24c1 Author: Dridi Boukelmoune Date: Fri Nov 29 17:33:03 2019 +0100 Give the dedicated type to VMOD object instances diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c index 0fc43bab2..be31db8cd 100644 --- a/lib/libvcc/vcc_vmod.c +++ b/lib/libvcc/vcc_vmod.c @@ -497,6 +497,10 @@ vcc_Act_New(struct vcc *tl, struct token *t, struct symbol *sym) XREF_NONE); ERRCHK(tl); AN(osym); + + /* Scratch the generic INSTANCE type */ + isym->type = osym->type; + CAST_OBJ_NOTNULL(vv, osym->eval_priv, VJSN_VAL_MAGIC); // vv = object name From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:08 +0000 (UTC) Subject: [master] 417bf145a Find VMOD objects methods via their typed symbols Message-ID: <20200702124908.A37EDA59E5@lists.varnish-cache.org> commit 417bf145a8682c21ee62be0ee4862cff458aa90f Author: Dridi Boukelmoune Date: Fri Dec 13 18:44:08 2019 +0100 Find VMOD objects methods via their typed symbols VMODs can now use the same facility as native types to find methods based on the type of the instance instead of the instance itself. Because SYM_METHOD symbols are tied to a type instead of an instance we lose that "extra" argument. To circumvent this, an expression will keep track of the instance before evaluating a method. With this, VCL expression evaluation is no longer aware of methods implementation details, and feeds entirely from the symbol table. diff --git a/bin/varnishtest/tests/v00018.vtc b/bin/varnishtest/tests/v00018.vtc index 572fde61e..32583760b 100644 --- a/bin/varnishtest/tests/v00018.vtc +++ b/bin/varnishtest/tests/v00018.vtc @@ -133,3 +133,17 @@ varnish v1 -syntax 4.0 -errvcl {Symbol not found:} { return (vcl(vcl_recv)); } } + +varnish v1 -errvcl {Syntax error} { + import directors; + sub vcl_recv { + set req.backend_hint = directors.round_robin.backend(); + } +} + +varnish v1 -errvcl {Syntax error} { + import directors; + sub vcl_recv { + directors.round_robin.backend(); + } +} diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 1fda4241b..47f3e51d0 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -333,6 +333,7 @@ char *TlDup(struct vcc *tl, const char *s); /* vcc_expr.c */ void vcc_Expr(struct vcc *tl, vcc_type_t typ); sym_act_f vcc_Act_Call; +sym_act_f vcc_Act_Obj; void vcc_Expr_Init(struct vcc *tl); sym_expr_t vcc_Eval_Var; sym_expr_t vcc_Eval_Handle; diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index 71c618a83..02e67d92d 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -50,6 +50,7 @@ struct expr { #define EXPR_CONST (1<<1) #define EXPR_STR_CONST (1<<2) // Last STRING_LIST elem is "..." struct token *t1, *t2; + struct symbol *instance; int nstr; }; @@ -484,6 +485,17 @@ vcc_func(struct vcc *tl, struct expr **e, const void *priv, sa = NULL; } vv = VTAILQ_NEXT(vv, list); + if (sym->kind == SYM_METHOD) { + if (*e == NULL) { + VSB_cat(tl->sb, "Syntax error."); + tl->err = 1; + return; + } + vcc_NextToken(tl); + AZ(extra); + AN((*e)->instance); + extra = (*e)->instance->rname; + } SkipToken(tl, '('); if (extra == NULL) { extra = ""; @@ -651,7 +663,7 @@ vcc_Eval_SymFunc(struct vcc *tl, struct expr **e, struct token *t, (void)t; (void)fmt; - assert(sym->kind == SYM_FUNC); + assert(sym->kind == SYM_FUNC || sym->kind == SYM_METHOD); AN(sym->eval_priv); vcc_func(tl, e, sym->eval_priv, sym->extra, sym); @@ -701,6 +713,12 @@ vcc_expr5(struct vcc *tl, struct expr **e, vcc_type_t fmt) XREF_REF); ERRCHK(tl); AN(sym); + if (sym->kind == SYM_INSTANCE) { + AZ(*e); + *e = vcc_new_expr(sym->type); + (*e)->instance = sym; + return; + } if (sym->kind == SYM_FUNC && sym->type == VOID) { VSB_cat(tl->sb, "Function returns VOID:\n"); vcc_ErrWhere(tl, tl->t); @@ -849,8 +867,7 @@ vcc_expr4(struct vcc *tl, struct expr **e, vcc_type_t fmt) if (sym == NULL) { VSB_cat(tl->sb, "Unknown property "); vcc_ErrToken(tl, tl->t); - VSB_printf(tl->sb, - " for type %s\n", (*e)->fmt->name); + VSB_printf(tl->sb, " for type %s\n", (*e)->fmt->name); vcc_ErrWhere(tl, tl->t); return; } @@ -1406,6 +1423,23 @@ vcc_Act_Call(struct vcc *tl, struct token *t, struct symbol *sym) } vcc_delete_expr(e); } + +void v_matchproto_(sym_act_f) +vcc_Act_Obj(struct vcc *tl, struct token *t, struct symbol *sym) +{ + + struct expr *e = NULL; + + assert(sym->kind == SYM_INSTANCE); + tl->t = t; + vcc_expr4(tl, &e, sym->type); + ERRCHK(tl); + vcc_expr_fmt(tl->fb, tl->indent, e); + vcc_delete_expr(e); + SkipToken(tl, ';'); + VSB_cat(tl->fb, ";\n"); +} + /*-------------------------------------------------------------------- */ diff --git a/lib/libvcc/vcc_parse.c b/lib/libvcc/vcc_parse.c index ec7b5e614..3331d961c 100644 --- a/lib/libvcc/vcc_parse.c +++ b/lib/libvcc/vcc_parse.c @@ -183,7 +183,7 @@ vcc_Compound(struct vcc *tl) return; case ID: sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_NONE, - SYMTAB_NOERR, XREF_NONE); + SYMTAB_PARTIAL, XREF_NONE); if (sym == NULL) { VSB_printf(tl->sb, "Symbol not found.\n"); vcc_ErrWhere(tl, tl->t); diff --git a/lib/libvcc/vcc_symb.c b/lib/libvcc/vcc_symb.c index ea2d78d4c..b97b20bca 100644 --- a/lib/libvcc/vcc_symb.c +++ b/lib/libvcc/vcc_symb.c @@ -364,8 +364,8 @@ VCC_SymbolGet(struct vcc *tl, vcc_ns_t ns, vcc_kind_t kind, return (sym); } -struct symbol * -VCC_TypeSymbol(struct vcc *tl, vcc_kind_t kind, vcc_type_t type) +static struct symbol * +vcc_TypeSymbol(struct vcc *tl, vcc_ns_t ns, vcc_kind_t kind, vcc_type_t type) { struct token t[1], *t0; struct symbol *sym; @@ -383,13 +383,24 @@ VCC_TypeSymbol(struct vcc *tl, vcc_kind_t kind, vcc_type_t type) t0 = tl->t; tl->t = t; - sym = VCC_SymbolGet(tl, SYM_TYPE, kind, SYMTAB_NOERR, XREF_NONE); + sym = VCC_SymbolGet(tl, ns, kind, SYMTAB_NOERR, XREF_NONE); tl->t = t0; VSB_destroy(&buf); return (sym); } +struct symbol * +VCC_TypeSymbol(struct vcc *tl, vcc_kind_t kind, vcc_type_t type) +{ + + if (strchr(type->name, '.') == NULL) + return (vcc_TypeSymbol(tl, SYM_TYPE, kind, type)); + + /* NB: type imported from a VMOD */ + return (vcc_TypeSymbol(tl, SYM_MAIN, kind, type)); +} + struct symbol * VCC_MkSym(struct vcc *tl, const char *b, vcc_ns_t ns, vcc_kind_t kind, int vlo, int vhi) diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c index be31db8cd..ecf977a16 100644 --- a/lib/libvcc/vcc_vmod.c +++ b/lib/libvcc/vcc_vmod.c @@ -203,7 +203,7 @@ vcc_vmod_kind(const char *type) return (kind); \ } while (0) VMOD_KIND("$OBJ", SYM_OBJECT); - VMOD_KIND("$METHOD", SYM_FUNC); + VMOD_KIND("$METHOD", SYM_METHOD); VMOD_KIND("$FUNC", SYM_FUNC); #undef VMOD_KIND return (SYM_NONE); @@ -244,7 +244,7 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) if (msym->kind == SYM_VMOD) { CAST_OBJ_NOTNULL(vj, msym->eval_priv, VJSN_MAGIC); vv = VTAILQ_FIRST(&vj->value->children); - } else if (msym->kind == SYM_INSTANCE || msym->kind == SYM_OBJECT) { + } else if (msym->kind == SYM_OBJECT) { CAST_OBJ_NOTNULL(vv, msym->eval_priv, VJSN_VAL_MAGIC); } else { WRONG("symbol kind"); @@ -290,13 +290,10 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) if (kind == SYM_FUNC) { func_sym(fsym, msym->vmod_name, VTAILQ_NEXT(vv2, list)); - /* XXX: until we use SYM_METHOD only, string check. */ - if (!strcmp(vv1->value, "$METHOD")) { - fsym->extra = msym->rname; - /* XXX: cohabitation temporary hack */ - if (msym->kind == SYM_OBJECT) - fsym->kind = SYM_METHOD; - } + } else if (kind == SYM_METHOD) { + func_sym(fsym, msym->vmod_name, VTAILQ_NEXT(vv2, list)); + fsym->extra = msym->rname; + fsym->kind = SYM_METHOD; } else { assert(kind == SYM_OBJECT); fsym->eval_priv = vv2; @@ -490,6 +487,7 @@ vcc_Act_New(struct vcc *tl, struct token *t, struct symbol *sym) ERRCHK(tl); AN(isym); isym->noref = 1; + isym->action = vcc_Act_Obj; SkipToken(tl, '='); ExpectErr(tl, ID); @@ -506,7 +504,6 @@ vcc_Act_New(struct vcc *tl, struct token *t, struct symbol *sym) isym->vmod_name = osym->vmod_name; isym->eval_priv = vv; - vcc_VmodSymbols(tl, isym); vv = VTAILQ_NEXT(vv, list); // vv = flags From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:08 +0000 (UTC) Subject: [master] edee8c62e Ensure VCL instance actions imply method calls Message-ID: <20200702124908.C9729A59F3@lists.varnish-cache.org> commit edee8c62e2c262e8b266ac82d94d86d9f461d314 Author: Dridi Boukelmoune Date: Sun Dec 1 10:10:43 2019 +0100 Ensure VCL instance actions imply method calls diff --git a/bin/varnishtest/tests/v00018.vtc b/bin/varnishtest/tests/v00018.vtc index 32583760b..d2728bcaf 100644 --- a/bin/varnishtest/tests/v00018.vtc +++ b/bin/varnishtest/tests/v00018.vtc @@ -147,3 +147,11 @@ varnish v1 -errvcl {Syntax error} { directors.round_robin.backend(); } } + +varnish v1 -errvcl {Expected '.' got ';'} { + import directors; + sub vcl_init { + new rr = directors.round_robin(); + rr; + } +} diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index 02e67d92d..67623e0b2 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -1431,6 +1431,7 @@ vcc_Act_Obj(struct vcc *tl, struct token *t, struct symbol *sym) struct expr *e = NULL; assert(sym->kind == SYM_INSTANCE); + ExpectErr(tl, '.'); tl->t = t; vcc_expr4(tl, &e, sym->type); ERRCHK(tl); From dridi.boukelmoune at gmail.com Thu Jul 2 12:49:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 2 Jul 2020 12:49:08 +0000 (UTC) Subject: [master] 9426f4a4c Clean up VMOD symbol registration code Message-ID: <20200702124908.E6F8EA59FC@lists.varnish-cache.org> commit 9426f4a4c0392faa2ebbe03e59163fdfeef821a6 Author: Dridi Boukelmoune Date: Fri Dec 13 18:59:21 2019 +0100 Clean up VMOD symbol registration code The temporary code that accumulated in order to implement generic type methods is now a bit more tidy. diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c index ecf977a16..a477ea335 100644 --- a/lib/libvcc/vcc_vmod.c +++ b/lib/libvcc/vcc_vmod.c @@ -73,13 +73,43 @@ vcc_path_dlopen(void *priv, const char *fn) return (0); } +static void vcc_VmodObject(struct vcc *tl, struct symbol *sym); +static void vcc_VmodSymbols(struct vcc *tl, struct symbol *sym); + static void -func_sym(struct symbol *sym, const char *vmod_name, const struct vjsn_val *v) +func_sym(struct vcc *tl, vcc_kind_t kind, struct symbol *psym, + const struct vjsn_val *v) { + struct symbol *sym; + struct vsb *buf; + + buf = VSB_new_auto(); + AN(buf); + + VSB_clear(buf); + VCC_SymName(buf, psym); + VSB_printf(buf, ".%s", v->value); + AZ(VSB_finish(buf)); + sym = VCC_MkSym(tl, VSB_data(buf), SYM_MAIN, kind, VCL_LOW, VCL_HIGH); + AN(sym); + VSB_destroy(&buf); + + if (kind == SYM_OBJECT) { + sym->eval_priv = v; + sym->vmod_name = psym->vmod_name; + vcc_VmodObject(tl, sym); + vcc_VmodSymbols(tl, sym); + return; + } + + if (kind == SYM_METHOD) + sym->extra = psym->rname; + + v = VTAILQ_NEXT(v, list); assert(v->type == VJSN_ARRAY); sym->action = vcc_Act_Call; - sym->vmod_name = vmod_name; + sym->vmod_name = psym->vmod_name; sym->eval = vcc_Eval_SymFunc; sym->eval_priv = v; v = VTAILQ_FIRST(&v->children); @@ -233,26 +263,21 @@ vcc_VmodObject(struct vcc *tl, struct symbol *sym) } static void -vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) +vcc_VmodSymbols(struct vcc *tl, struct symbol *sym) { const struct vjsn *vj; const struct vjsn_val *vv, *vv1, *vv2; - struct symbol *fsym; vcc_kind_t kind; - struct vsb *buf; - if (msym->kind == SYM_VMOD) { - CAST_OBJ_NOTNULL(vj, msym->eval_priv, VJSN_MAGIC); + if (sym->kind == SYM_VMOD) { + CAST_OBJ_NOTNULL(vj, sym->eval_priv, VJSN_MAGIC); vv = VTAILQ_FIRST(&vj->value->children); - } else if (msym->kind == SYM_OBJECT) { - CAST_OBJ_NOTNULL(vv, msym->eval_priv, VJSN_VAL_MAGIC); + } else if (sym->kind == SYM_OBJECT) { + CAST_OBJ_NOTNULL(vv, sym->eval_priv, VJSN_VAL_MAGIC); } else { WRONG("symbol kind"); } - buf = VSB_new_auto(); - AN(buf); - for (; vv != NULL; vv = VTAILQ_NEXT(vv, list)) { if (vv->type != VJSN_ARRAY) continue; @@ -266,44 +291,8 @@ vcc_VmodSymbols(struct vcc *tl, struct symbol *msym) if (kind == SYM_NONE) continue; - /* NB: currently VMOD object methods are effectively function - * symbols (SYM_FUNC) because they are declared per instance - * instead of per object. Once they become proper methods - * symbols (SYM_METHOD) we can move the symbol creation inside - * the func_sym() function and replace the rest of the loop - * with a single statement: - * - * func_sym(kind, msym, vv2); - * - * Then based on kind, the func_sym() function would account - * for the slight differences between the 3 kinds of VMOD - * functions (function, object constructor, object method). - */ - - VSB_clear(buf); - VCC_SymName(buf, msym); - VSB_printf(buf, ".%s", vv2->value); - AZ(VSB_finish(buf)); - fsym = VCC_MkSym(tl, VSB_data(buf), SYM_MAIN, kind, - VCL_LOW, VCL_HIGH); - AN(fsym); - - if (kind == SYM_FUNC) { - func_sym(fsym, msym->vmod_name, VTAILQ_NEXT(vv2, list)); - } else if (kind == SYM_METHOD) { - func_sym(fsym, msym->vmod_name, VTAILQ_NEXT(vv2, list)); - fsym->extra = msym->rname; - fsym->kind = SYM_METHOD; - } else { - assert(kind == SYM_OBJECT); - fsym->eval_priv = vv2; - fsym->vmod_name = msym->vmod_name; - vcc_VmodObject(tl, fsym); - vcc_VmodSymbols(tl, fsym); - } + func_sym(tl, kind, sym, vv2); } - - VSB_destroy(&buf); } void From nils.goroll at uplex.de Mon Jul 6 16:30:08 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 6 Jul 2020 16:30:08 +0000 (UTC) Subject: [master] 0c30fedfe shard director: explain how the ring is constructed with weights Message-ID: <20200706163008.30CA093BAA@lists.varnish-cache.org> commit 0c30fedfe418c59eb6bb900c43c075092e7daddf Author: Nils Goroll Date: Mon Jul 6 18:07:09 2020 +0200 shard director: explain how the ring is constructed with weights diff --git a/lib/libvmod_directors/vmod_directors.vcc b/lib/libvmod_directors/vmod_directors.vcc index 8ab9f12fa..940b403cd 100644 --- a/lib/libvmod_directors/vmod_directors.vcc +++ b/lib/libvmod_directors/vmod_directors.vcc @@ -319,7 +319,10 @@ hashing circular data structure gets built from the last 32 bits of SHA256 hash values of **\ ** (default *ident* being the backend name) for each backend and for a running number *n* from 1 to *replicas*. Hashing creates the seemingly random order for placement -of backends on the consistent hashing ring. +of backends on the consistent hashing ring. When +`xshard.add_backend()`_ was called with a *weight* argument, +*replicas* is scaled by that weight to add proportionally more copies +of the that backend on the ring. When `xshard.backend()`_ is called, a load balancing key gets generated unless provided. The smallest hash value in the circle From nils.goroll at uplex.de Mon Jul 6 16:30:08 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 6 Jul 2020 16:30:08 +0000 (UTC) Subject: [master] 3dc2baed6 Loosen assertion on ctx->(req|bo), fix shard and vcl_pipe Message-ID: <20200706163008.482DA93BAD@lists.varnish-cache.org> commit 3dc2baed643f7af63039b3c49d43083a5759b27f Author: Nils Goroll Date: Mon Jul 6 17:15:03 2020 +0200 Loosen assertion on ctx->(req|bo), fix shard and vcl_pipe in VRT_priv_task() we asserted that only one of ctx->req and ctx->bo is set when not in vcl_pipe {}, but we also need to extend that assertion to when ctx->method == 0 after vcl_pipe as finished because VRT_priv_task() could be called from director resolution. Being at it, I also noticed that our behavior in vcl_pipe {} is inconsistent as, from the shard director perspective, it is a backend method. So now, vcl_pipe {} is handled like vcl_backend_* {}. We still need to make up our mind about #3329 / #3330 and depending on the outcome we might need to touch some places again which were changed in this commit. Fixes #3361 diff --git a/bin/varnishd/cache/cache_vrt_priv.c b/bin/varnishd/cache/cache_vrt_priv.c index bcc7bfca5..44692af56 100644 --- a/bin/varnishd/cache/cache_vrt_priv.c +++ b/bin/varnishd/cache/cache_vrt_priv.c @@ -134,8 +134,15 @@ VRT_priv_task(VRT_CTX, const void *vmod_id) { CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + /* + * XXX when coming from VRT_DirectorResolve() in pipe mode + * (ctx->method == 0), both req and bo are set. + * see #3329 #3330: we should make up our mind where + * pipe objects live + */ + assert(ctx->req == NULL || ctx->bo == NULL || - ctx->method == VCL_MET_PIPE); + ctx->method == VCL_MET_PIPE || ctx->method == 0); if (ctx->req) { CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); diff --git a/bin/varnishtest/tests/d00029.vtc b/bin/varnishtest/tests/d00029.vtc index 698f89e21..5bacdae45 100644 --- a/bin/varnishtest/tests/d00029.vtc +++ b/bin/varnishtest/tests/d00029.vtc @@ -1,16 +1,16 @@ varnishtest "shard director LAZY - d18.vtc" -server s1 { +server s1 -repeat 3 { rxreq txresp -body "ech3Ooj" } -start -server s2 { +server s2 -repeat 3 { rxreq txresp -body "ieQu2qua" } -start -server s3 { +server s3 -repeat 3 { rxreq txresp -body "xiuFi3Pe" } -start @@ -45,10 +45,13 @@ varnish v1 -vcl+backend { } sub vcl_recv { + if (req.http.pipe) { + return (pipe); + } return(pass); } - sub vcl_backend_fetch { + sub shard_be { set bereq.backend=vd.backend(resolve=LAZY); if (bereq.url == "/1") { @@ -64,6 +67,14 @@ varnish v1 -vcl+backend { } } + sub vcl_backend_fetch { + call shard_be; + } + + sub vcl_pipe { + call shard_be; + } + sub vcl_backend_response { set beresp.http.backend = bereq.backend; } @@ -85,4 +96,20 @@ client c1 { rxresp expect resp.body == "xiuFi3Pe" expect resp.http.backend == "vd" + + txreq -url /1 -hdr "pipe: true" + rxresp + expect resp.body == "ech3Ooj" +} -run + +client c1 { + txreq -url /2 -hdr "pipe: true" + rxresp + expect resp.body == "ieQu2qua" +} -run + +client c1 { + txreq -url /3 -hdr "pipe: true" + rxresp + expect resp.body == "xiuFi3Pe" } -run diff --git a/bin/varnishtest/tests/d00030.vtc b/bin/varnishtest/tests/d00030.vtc index 3c29f33e7..912314b03 100644 --- a/bin/varnishtest/tests/d00030.vtc +++ b/bin/varnishtest/tests/d00030.vtc @@ -31,7 +31,7 @@ logexpect l2 -v v1 -g raw { expect * 1001 VCL_Error {shard .backend param invalid} } -start logexpect l3 -v v1 -g raw { - expect * 1003 VCL_Error {shard_param.set.. may only be used in vcl_init and in backend context} + expect * 1003 VCL_Error {shard_param.set.. may only be used in vcl_init and in backend/pipe context} } -start client c1 { @@ -159,7 +159,7 @@ varnish v1 -errvcl {invalid warmup argument 1.1} { } } -varnish v1 -errvcl {resolve=LAZY with other parameters can only be used in backend context} { +varnish v1 -errvcl {resolve=LAZY with other parameters can only be used in backend/pipe context} { import directors; import blob; diff --git a/doc/changes.rst b/doc/changes.rst index b9d7e44ca..cfed269df 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -38,6 +38,9 @@ NEXT (scheduled 2020-09-15) argument which allows to extend the effective privilege set of the worker process. +* The shard director and shard director parameter objects should now + work in ``vcl_pipe {}`` like in ``vcl_backend_* {}`` subs. + ================================ Varnish Cache 6.4.0 (2020-03-16) ================================ diff --git a/lib/libvmod_directors/vmod_directors.vcc b/lib/libvmod_directors/vmod_directors.vcc index 940b403cd..5238250f1 100644 --- a/lib/libvmod_directors/vmod_directors.vcc +++ b/lib/libvmod_directors/vmod_directors.vcc @@ -462,10 +462,11 @@ is _not_ the order given when backends are added. * ``HASH``: - * when called in backend context: Use the varnish hash value as - set by ``vcl_hash{}`` + * when called in backend context and in ``vcl_pipe {}``: Use the + varnish hash value as set by ``vcl_hash{}`` - * when called in client context: hash ``req.url`` + * when called in client context other than ``vcl_pipe {}``: hash + ``req.url`` * ``URL``: hash req.url / bereq.url @@ -556,9 +557,10 @@ is _not_ the order given when backends are added. In ``vcl_init{}`` and on the client side, ``LAZY`` mode can not be used with any other argument. - On the backend side, parameters from arguments or an associated - parameter set affect the shard director instance for the backend - request irrespective of where it is referenced. + On the backend side and in ``vcl_pipe {}``, parameters from + arguments or an associated parameter set affect the shard director + instance for the backend request irrespective of where it is + referenced. * *param* @@ -607,10 +609,11 @@ Parameter sets have two scopes: * per backend request scope The per-VCL scope defines defaults for the per backend scope. Any -changes to a parameter set in backend context only affect the -respective backend request. +changes to a parameter set in backend context and in ``vcl_pipe {}`` +only affect the respective backend request. -Parameter sets can not be used in client context. +Parameter sets can not be used in client context except for +``vcl_pipe {}``. The following example is a typical use case: A parameter set is associated with several directors. Director choice happens on the @@ -649,11 +652,11 @@ $Method VOID .clear() Reset the parameter set to default values as documented for `xshard.backend()`_. -* in ``vcl_init{}``, resets the parameter set default for this VCL -* in backend context, resets the parameter set for this backend - request to the VCL defaults +* in ``vcl_init{}``, resets the parameter set default for this VCL in +* backend context and in ``vcl_pipe {}``, resets the parameter set for + this backend request to the VCL defaults -This method may not be used in client context +This method may not be used in client context other than ``vcl_pipe {}``. $Method VOID .set( [ ENUM {HASH, URL, KEY, BLOB} by ], @@ -669,11 +672,11 @@ Change the given parameters of a parameter set as documented for * in ``vcl_init{}``, changes the parameter set default for this VCL -* in backend context, changes the parameter set for this backend - request, keeping the defaults set for this VCL for unspecified - arguments. +* in backend context and in ``vcl_pipe {}``, changes the parameter set + for this backend request, keeping the defaults set for this VCL for + unspecified arguments. -This method may not be used in client context +This method may not be used in client context other than ``vcl_pipe {}``. $Method STRING .get_by() @@ -709,7 +712,7 @@ shard director using this parameter object would use. See $Method BLOB .use() -This method may only be used in backend context. +This method may only be used in backend context and in ``vcl_pipe {}``. For use with the *param* argument of `xshard.backend()`_ to associate this shard parameter set with a shard director. diff --git a/lib/libvmod_directors/vmod_shard.c b/lib/libvmod_directors/vmod_shard.c index 9779ae78d..fbcd1d132 100644 --- a/lib/libvmod_directors/vmod_shard.c +++ b/lib/libvmod_directors/vmod_shard.c @@ -170,6 +170,9 @@ vmod_shard_param_read(VRT_CTX, const void *id, const struct vmod_directors_shard_param *p, struct vmod_directors_shard_param *pstk, const char *who); +// XXX #3329 #3330 revisit - for now, treat pipe like backend +#define SHARD_VCL_TASK_REQ (VCL_MET_TASK_C & ~VCL_MET_PIPE) +#define SHARD_VCL_TASK_BEREQ (VCL_MET_TASK_B | VCL_MET_PIPE) /* ------------------------------------------------------------------------- * shard vmod interface */ @@ -617,14 +620,14 @@ vmod_shard_backend(VRT_CTX, struct vmod_directors_shard *vshard, return (vshard->dir); } - if ((ctx->method & VCL_MET_TASK_B) == 0) { + if ((ctx->method & SHARD_VCL_TASK_BEREQ) == 0) { VRT_fail(ctx, "shard .backend resolve=LAZY with other " - "parameters can only be used in backend " + "parameters can only be used in backend/pipe " "context"); return (NULL); } - assert(ctx->method & VCL_MET_TASK_B); + assert(ctx->method & SHARD_VCL_TASK_BEREQ); pp = shard_param_task(ctx, vshard->shardd, vshard->shardd->param); @@ -918,11 +921,11 @@ shard_param_prep(VRT_CTX, struct vmod_directors_shard_param *p, CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); CHECK_OBJ_NOTNULL(p, VMOD_SHARD_SHARD_PARAM_MAGIC); - if (ctx->method & VCL_MET_TASK_C) { + if (ctx->method & SHARD_VCL_TASK_REQ) { VRT_fail(ctx, "%s may only be used " - "in vcl_init and in backend context", who); + "in vcl_init and in backend/pipe context", who); return (NULL); - } else if (ctx->method & VCL_MET_TASK_B) + } else if (ctx->method & SHARD_VCL_TASK_BEREQ) p = shard_param_task(ctx, p, p); else assert(ctx->method & VCL_MET_TASK_H); @@ -967,7 +970,7 @@ vmod_shard_param_read(VRT_CTX, const void *id, CHECK_OBJ_NOTNULL(p, VMOD_SHARD_SHARD_PARAM_MAGIC); (void) who; // XXX - if (ctx->method == 0 || (ctx->method & VCL_MET_TASK_B)) + if (ctx->method == 0 || (ctx->method & SHARD_VCL_TASK_BEREQ)) p = shard_param_task(ctx, id, p); if (p == NULL) From phk at FreeBSD.org Mon Jul 13 07:51:05 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 13 Jul 2020 07:51:05 +0000 (UTC) Subject: [master] b12dcc742 vcc_namespace.h is a "table" include Message-ID: <20200713075105.62919118A9C@lists.varnish-cache.org> commit b12dcc742e058da721aa05034143eac2ee272eff Author: Poul-Henning Kamp Date: Mon Jul 13 07:45:39 2020 +0000 vcc_namespace.h is a "table" include diff --git a/lib/libvcc/flint.lnt b/lib/libvcc/flint.lnt index e6c7cbd75..f060458b9 100644 --- a/lib/libvcc/flint.lnt +++ b/lib/libvcc/flint.lnt @@ -5,3 +5,5 @@ -esym(755, VCL_40) // Not used (right now) -esym(768, token) // FLINTBUG: global struct member 'token' not ref + +-efile(451, vcc_namespace.h) // No include guard From phk at FreeBSD.org Mon Jul 13 07:51:05 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 13 Jul 2020 07:51:05 +0000 (UTC) Subject: [master] d2886e6a2 Constification Message-ID: <20200713075105.76E23118A9F@lists.varnish-cache.org> commit d2886e6a260217ba61c071f9302f036497065502 Author: Poul-Henning Kamp Date: Mon Jul 13 07:50:12 2020 +0000 Constification diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c index a477ea335..039d39787 100644 --- a/lib/libvcc/vcc_vmod.c +++ b/lib/libvcc/vcc_vmod.c @@ -74,10 +74,10 @@ vcc_path_dlopen(void *priv, const char *fn) } static void vcc_VmodObject(struct vcc *tl, struct symbol *sym); -static void vcc_VmodSymbols(struct vcc *tl, struct symbol *sym); +static void vcc_VmodSymbols(struct vcc *tl, const struct symbol *sym); static void -func_sym(struct vcc *tl, vcc_kind_t kind, struct symbol *psym, +func_sym(struct vcc *tl, vcc_kind_t kind, const struct symbol *psym, const struct vjsn_val *v) { struct symbol *sym; @@ -168,7 +168,7 @@ vcc_json_always(struct vcc *tl, const struct vjsn *vj, const char *vmod_name) } static const struct vmod_data * -vcc_VmodSanity(struct vcc *tl, void *hdl, struct token *mod, char *fnp) +vcc_VmodSanity(struct vcc *tl, void *hdl, const struct token *mod, char *fnp) { char buf[256]; const struct vmod_data *vmd; @@ -263,7 +263,7 @@ vcc_VmodObject(struct vcc *tl, struct symbol *sym) } static void -vcc_VmodSymbols(struct vcc *tl, struct symbol *sym) +vcc_VmodSymbols(struct vcc *tl, const struct symbol *sym) { const struct vjsn *vj; const struct vjsn_val *vv, *vv1, *vv2; From phk at FreeBSD.org Mon Jul 13 10:09:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 13 Jul 2020 10:09:07 +0000 (UTC) Subject: [master] 7bb73c586 Slight polish while looking at #3360 Message-ID: <20200713100907.4E38C11BA61@lists.varnish-cache.org> commit 7bb73c586323fe6511ce04e6558fb2eca7073cef Author: Poul-Henning Kamp Date: Mon Jul 13 09:40:56 2020 +0000 Slight polish while looking at #3360 diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index d24ebf5e1..6bf82bfa0 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -251,7 +251,7 @@ EmitCoordinates(const struct vcc *tl, struct vsb *vsb) { struct token *t; unsigned lin, pos; - struct source *sp; + const struct source *sp; const char *p; VSB_cat(vsb, "/* ---===### Source Code ###===---*/\n"); @@ -610,7 +610,7 @@ vcc_resolve_includes(struct vcc *tl) VTAILQ_INSERT_TAIL(&tl->sources, sp, list); sp->idx = tl->nsources++; tl->t = t2; - vcc_Lexer(tl, sp); + vcc_Lexer(tl, sp, 0); VTAILQ_REMOVE(&tl->tokens, t, list); VTAILQ_REMOVE(&tl->tokens, t1, list); @@ -654,7 +654,7 @@ vcc_CompileSource(struct vcc *tl, struct source *sp, const char *jfile) /* Register and lex the main source */ VTAILQ_INSERT_TAIL(&tl->sources, sp, list); sp->idx = tl->nsources++; - vcc_Lexer(tl, sp); + vcc_Lexer(tl, sp, 0); if (tl->err) return (NULL); @@ -663,12 +663,7 @@ vcc_CompileSource(struct vcc *tl, struct source *sp, const char *jfile) assert(sp != NULL); VTAILQ_INSERT_TAIL(&tl->sources, sp, list); sp->idx = tl->nsources++; - vcc_Lexer(tl, sp); - if (tl->err) - return (NULL); - - /* Add "END OF INPUT" token */ - vcc_AddToken(tl, EOI, sp->e, sp->e); + vcc_Lexer(tl, sp, 1); if (tl->err) return (NULL); diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 47f3e51d0..1c9053b24 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -91,7 +91,7 @@ struct token { unsigned tok; const char *b; const char *e; - struct source *src; + const struct source *src; VTAILQ_ENTRY(token) list; unsigned cnt; char *dec; @@ -249,7 +249,6 @@ struct vcc { struct tokenhead tokens; VTAILQ_HEAD(, source) sources; unsigned nsources; - struct source *src; struct token *t; int indent; int hindent; @@ -404,12 +403,10 @@ void vcc_Warn(struct vcc *); void vcc__Expect(struct vcc *tl, unsigned tok, unsigned line); int vcc_IdIs(const struct token *t, const char *p); void vcc_ExpectVid(struct vcc *tl, const char *what); -void vcc_Lexer(struct vcc *tl, struct source *sp); +void vcc_Lexer(struct vcc *tl, const struct source *sp, int eoi); void vcc_NextToken(struct vcc *tl); void vcc__ErrInternal(struct vcc *tl, const char *func, unsigned line); -void vcc_AddToken(struct vcc *tl, unsigned tok, const char *b, - const char *e); /* vcc_types.c */ vcc_type_t VCC_Type(const char *p); diff --git a/lib/libvcc/vcc_token.c b/lib/libvcc/vcc_token.c index 0e3b4569d..176b74eab 100644 --- a/lib/libvcc/vcc_token.c +++ b/lib/libvcc/vcc_token.c @@ -362,8 +362,9 @@ vcc_decstr(struct vcc *tl) * Add a token to the token list. */ -void -vcc_AddToken(struct vcc *tl, unsigned tok, const char *b, const char *e) +static void +vcc_addtoken(struct vcc *tl, unsigned tok, + const struct source *sp, const char *b, const char *e) { struct token *t; @@ -372,7 +373,7 @@ vcc_AddToken(struct vcc *tl, unsigned tok, const char *b, const char *e) t->tok = tok; t->b = b; t->e = e; - t->src = tl->src; + t->src = sp; if (tl->t != NULL) VTAILQ_INSERT_AFTER(&tl->tokens, tl->t, t, list); else @@ -385,12 +386,11 @@ vcc_AddToken(struct vcc *tl, unsigned tok, const char *b, const char *e) */ void -vcc_Lexer(struct vcc *tl, struct source *sp) +vcc_Lexer(struct vcc *tl, const struct source *sp, int eoi) { const char *p, *q; unsigned u; - tl->src = sp; for (p = sp->b; p < sp->e; ) { /* Skip any whitespace */ @@ -412,9 +412,9 @@ vcc_Lexer(struct vcc *tl, struct source *sp) if (*q == '/' && q[1] == '*') { VSB_cat(tl->sb, "/* ... */ comment contains /*\n"); - vcc_AddToken(tl, EOI, p, p + 2); + vcc_addtoken(tl, EOI, sp, p, p + 2); vcc_ErrWhere(tl, tl->t); - vcc_AddToken(tl, EOI, q, q + 2); + vcc_addtoken(tl, EOI, sp, q, q + 2); vcc_ErrWhere(tl, tl->t); return; } @@ -425,7 +425,7 @@ vcc_Lexer(struct vcc *tl, struct source *sp) } if (q < sp->e) continue; - vcc_AddToken(tl, EOI, p, p + 2); + vcc_addtoken(tl, EOI, sp, p, p + 2); VSB_cat(tl->sb, "Unterminated /* ... */ comment, starting at\n"); vcc_ErrWhere(tl, tl->t); @@ -443,7 +443,7 @@ vcc_Lexer(struct vcc *tl, struct source *sp) if (*p == 'C' && p[1] == '{') { for (q = p + 2; q < sp->e; q++) { if (*q == '}' && q[1] == 'C') { - vcc_AddToken(tl, CSRC, p, q + 2); + vcc_addtoken(tl, CSRC, sp, p, q + 2); break; } } @@ -451,7 +451,7 @@ vcc_Lexer(struct vcc *tl, struct source *sp) p = q + 2; continue; } - vcc_AddToken(tl, EOI, p, p + 2); + vcc_addtoken(tl, EOI, sp, p, p + 2); VSB_cat(tl->sb, "Unterminated inline C source, starting at\n"); vcc_ErrWhere(tl, tl->t); @@ -462,7 +462,7 @@ vcc_Lexer(struct vcc *tl, struct source *sp) if (*p == '{' && p[1] == '"') { for (q = p + 2; q < sp->e; q++) { if (*q == '"' && q[1] == '}') { - vcc_AddToken(tl, CSTR, p, q + 2); + vcc_addtoken(tl, CSTR, sp, p, q + 2); break; } } @@ -476,7 +476,7 @@ vcc_Lexer(struct vcc *tl, struct source *sp) tl->t->dec[u] = '\0'; continue; } - vcc_AddToken(tl, EOI, p, p + 2); + vcc_addtoken(tl, EOI, sp, p, p + 2); VSB_cat(tl->sb, "Unterminated long-string, starting at\n"); vcc_ErrWhere(tl, tl->t); @@ -486,7 +486,7 @@ vcc_Lexer(struct vcc *tl, struct source *sp) /* Match for the fixed tokens (see generate.py) */ u = vcl_fixed_token(p, &q); if (u != 0) { - vcc_AddToken(tl, u, p, q); + vcc_addtoken(tl, u, sp, p, q); p = q; continue; } @@ -499,14 +499,14 @@ vcc_Lexer(struct vcc *tl, struct source *sp) break; } if (*q == '\r' || *q == '\n') { - vcc_AddToken(tl, EOI, p, q); + vcc_addtoken(tl, EOI, sp, p, q); VSB_cat(tl->sb, "Unterminated string at\n"); vcc_ErrWhere(tl, tl->t); return; } } - vcc_AddToken(tl, CSTR, p, q); + vcc_addtoken(tl, CSTR, sp, p, q); if (vcc_decstr(tl)) return; p = q; @@ -518,7 +518,7 @@ vcc_Lexer(struct vcc *tl, struct source *sp) for (q = p; q < sp->e; q++) if (!vct_isident(*q)) break; - vcc_AddToken(tl, ID, p, q); + vcc_addtoken(tl, ID, sp, p, q); p = q; continue; } @@ -529,20 +529,22 @@ vcc_Lexer(struct vcc *tl, struct source *sp) if (!vct_isdigit(*q)) break; if (*q != '.') { - vcc_AddToken(tl, CNUM, p, q); + vcc_addtoken(tl, CNUM, sp, p, q); p = q; continue; } for (++q; q < sp->e; q++) if (!vct_isdigit(*q)) break; - vcc_AddToken(tl, FNUM, p, q); + vcc_addtoken(tl, FNUM, sp, p, q); p = q; continue; } - vcc_AddToken(tl, EOI, p, p + 1); + vcc_addtoken(tl, EOI, sp, p, p + 1); VSB_cat(tl->sb, "Syntax error at\n"); vcc_ErrWhere(tl, tl->t); return; } + if (eoi) + vcc_addtoken(tl, EOI, sp, sp->e, sp->e); } From phk at FreeBSD.org Mon Jul 13 10:09:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 13 Jul 2020 10:09:07 +0000 (UTC) Subject: [master] 50a59f105 Fail on recursive use of vcl `include` Message-ID: <20200713100907.62FF311BA64@lists.varnish-cache.org> commit 50a59f1058017d540b0dccfb5bee18a36bf39923 Author: Poul-Henning Kamp Date: Mon Jul 13 10:07:23 2020 +0000 Fail on recursive use of vcl `include` Fixes: #3360 diff --git a/bin/varnishtest/tests/r03360.vtc b/bin/varnishtest/tests/r03360.vtc new file mode 100644 index 000000000..842988119 --- /dev/null +++ b/bin/varnishtest/tests/r03360.vtc @@ -0,0 +1,15 @@ +varnishtest "Test recusive vcl includes" + +shell {echo include '"_recurse.vcl";' > ${tmpdir}/_recurse.vcl} +shell {echo include '"_recurse2.vcl";' > ${tmpdir}/_recurse1.vcl} +shell {echo include '"_recurse1.vcl";' > ${tmpdir}/_recurse2.vcl} + +varnish v1 -arg "-p vcl_path=${tmpdir}" -errvcl "Recursive include of" { + backend b { .host = "127.0.0.1"; } + include "_recurse.vcl" ; +} + +varnish v2 -arg "-p vcl_path=${tmpdir}" -errvcl "Recursive include of" { + backend b { .host = "127.0.0.1"; } + include "_recurse1.vcl" ; +} diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index 6bf82bfa0..b181dc67d 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -553,6 +553,7 @@ vcc_resolve_includes(struct vcc *tl) { struct token *t, *t1, *t2; struct source *sp; + const struct source *sp1; struct vsb *vsb; const char *p; @@ -607,6 +608,22 @@ vcc_resolve_includes(struct vcc *tl) vcc_ErrWhere(tl, t1); return; } + + for (sp1 = t->src; sp1 != NULL; sp1 = sp1->parent) + if (!strcmp(sp1->name, sp->name)) + break; + if (sp1 != NULL) { + VSB_printf(tl->sb, + "Recursive include of \"%s\"\n\n", sp->name); + vcc_ErrWhere(tl, t1); + for (sp1 = t->src; sp1 != NULL; sp1 = sp1->parent) { + if (sp1->parent_tok) + vcc_ErrWhere(tl, sp1->parent_tok); + } + return; + } + sp->parent = t->src; + sp->parent_tok = t1; VTAILQ_INSERT_TAIL(&tl->sources, sp, list); sp->idx = tl->nsources++; tl->t = t2; diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 1c9053b24..b6db44343 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -85,6 +85,8 @@ struct source { const char *e; unsigned idx; char *freeit; + const struct source *parent; + const struct token *parent_tok; }; struct token { From phk at FreeBSD.org Mon Jul 13 11:57:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 13 Jul 2020 11:57:06 +0000 (UTC) Subject: [phk3316] d6a496d70 VMB: renovation Message-ID: <20200713115706.0C18653D3@lists.varnish-cache.org> commit d6a496d7036352ac8dc518b5c87a61db0117956f Author: Poul-Henning Kamp Date: Mon Jul 13 11:54:46 2020 +0000 VMB: renovation Use if we have it. Let Ancient FreeBSD without fall back to the __GNUCC_ inline assy. diff --git a/configure.ac b/configure.ac index f0f5b11db..b63840512 100644 --- a/configure.ac +++ b/configure.ac @@ -100,6 +100,9 @@ AC_SUBST(NET_LIBS, "${SOCKET_LIBS} ${NSL_LIBS}") # Userland slab allocator from Solaris, ported to other systems AC_CHECK_HEADERS([umem.h]) +# More portable vmb.h +AC_CHECK_HEADERS([stdatomic.h]) + # XXX: This _may_ be for OS/X AC_CHECK_LIBM AC_SUBST(LIBM) diff --git a/include/vmb.h b/include/vmb.h index 9aaa7e949..7029e29f6 100644 --- a/include/vmb.h +++ b/include/vmb.h @@ -29,60 +29,38 @@ * * Memory barriers * - * XXX: It is utterly braindamaged, that no standard facility for this - * XXX: is available. The "just use pthreads locking" excuse does not - * XXX: make sense, and does not apply to two unthreaded programs sharing - * XXX: a memory segment. */ #ifndef VMB_H_INCLUDED #define VMB_H_INCLUDED -#if defined(__FreeBSD__) -#include -#endif - -#if defined(__FreeBSD__) && __FreeBSD_version >= 800058 +#if defined(HAVE_STDATOMIC_H) -#include -#include -#define VMB() mb() -#define VWMB() wmb() -#define VRMB() rmb() +# include +# define VWMB() atomic_thread_fence(memory_order_release) +# define VRMB() atomic_thread_fence(memory_order_acquire) #elif defined(__amd64__) && defined(__GNUC__) -#define VMB() __asm __volatile("mfence;" : : : "memory") -#define VWMB() __asm __volatile("sfence;" : : : "memory") -#define VRMB() __asm __volatile("lfence;" : : : "memory") - -#elif defined(__arm__) - -#define VMB() -#define VWMB() -#define VRMB() +# define VWMB() __asm __volatile("sfence;" : : : "memory") +# define VRMB() __asm __volatile("lfence;" : : : "memory") #elif defined(__i386__) && defined(__GNUC__) -#define VMB() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") -#define VWMB() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") -#define VRMB() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") +# define VWMB() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") +# define VRMB() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") #elif defined(__sparc64__) && defined(__GNUC__) -#define VMB() __asm__ __volatile__ ("membar #MemIssue": : :"memory") -#define VWMB() VMB() -#define VRMB() VMB() +# define VWMB() __asm__ __volatile__ ("membar #MemIssue": : :"memory") +# define VRMB() __asm__ __volatile__ ("membar #MemIssue": : :"memory") #else -#define VMB_NEEDS_PTHREAD_WORKAROUND_THIS_IS_BAD_FOR_PERFORMANCE 1 - -void vmb_pthread(void); - -#define VMB() vmb_pthread() -#define VWMB() vmb_pthread() -#define VRMB() vmb_pthread() +# define VMB_NEEDS_PTHREAD_WORKAROUND_THIS_IS_BAD_FOR_PERFORMANCE 1 + void vmb_pthread(void); +# define VWMB() vmb_pthread() +# define VRMB() vmb_pthread() #endif From phk at FreeBSD.org Mon Jul 13 13:16:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 13 Jul 2020 13:16:07 +0000 (UTC) Subject: [master] 6b0c55b76 Add a BASE64 class to VCT Message-ID: <20200713131607.210C776F6@lists.varnish-cache.org> commit 6b0c55b76d3293a54758c3a339f6b9f6b4591e8f Author: Poul-Henning Kamp Date: Mon Jul 13 13:05:43 2020 +0000 Add a BASE64 class to VCT diff --git a/include/vct.h b/include/vct.h index 9bd3ce3d8..7c5825849 100644 --- a/include/vct.h +++ b/include/vct.h @@ -47,6 +47,7 @@ #define VCT_TCHAR (1<<9) #define VCT_ID (1<<10) #define VCT_IDENT (VCT_ALPHA | VCT_DIGIT | VCT_ID) +#define VCT_BASE64 (1<<11) #define VCT_VT (1<<12) #define VCT_SPACE (VCT_LWS | VCT_VT) #define VCT_UPPER (1<<13) @@ -75,6 +76,7 @@ vct_is(int x, uint16_t y) #define vct_islower(x) vct_is(x, VCT_LOWER) #define vct_isupper(x) vct_is(x, VCT_UPPER) #define vct_isalnum(x) vct_is(x, VCT_ALPHA | VCT_DIGIT) +#define vct_isbase64(x) vct_is(x, VCT_BASE64) #define vct_issep(x) vct_is(x, VCT_SEPARATOR) #define vct_issepctl(x) vct_is(x, VCT_SEPARATOR | VCT_CTL) #define vct_isident1(x) vct_isalpha(x) diff --git a/lib/libvarnish/vct.c b/lib/libvarnish/vct.c index 7b0d051ae..566f1d9ea 100644 --- a/lib/libvarnish/vct.c +++ b/lib/libvarnish/vct.c @@ -43,8 +43,8 @@ /* NB: VCT always operate in ASCII, don't replace 0x0d with \r etc. */ -#define VCT_UPALPHA (VCT_ALPHA | VCT_UPPER) -#define VCT_LOALPHA (VCT_ALPHA | VCT_LOWER) +#define VCT_UPALPHA (VCT_ALPHA | VCT_UPPER | VCT_BASE64) +#define VCT_LOALPHA (VCT_ALPHA | VCT_LOWER | VCT_BASE64) const uint16_t vct_typtab[256] = { [0x00] = VCT_CTL, @@ -90,25 +90,25 @@ const uint16_t vct_typtab[256] = { [0x28] = VCT_SEPARATOR, [0x29] = VCT_SEPARATOR, [0x2a] = VCT_TCHAR, - [0x2b] = VCT_TCHAR, + [0x2b] = VCT_TCHAR | VCT_BASE64, [0x2c] = VCT_SEPARATOR, [0x2d] = VCT_XMLNAME | VCT_TCHAR | VCT_ID, [0x2e] = VCT_XMLNAME | VCT_TCHAR, - [0x2f] = VCT_SEPARATOR, - [0x30] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, - [0x31] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, - [0x32] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, - [0x33] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, - [0x34] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, - [0x35] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, - [0x36] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, - [0x37] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, - [0x38] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, - [0x39] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, + [0x2f] = VCT_SEPARATOR | VCT_BASE64, + [0x30] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME | VCT_BASE64, + [0x31] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME | VCT_BASE64, + [0x32] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME | VCT_BASE64, + [0x33] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME | VCT_BASE64, + [0x34] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME | VCT_BASE64, + [0x35] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME | VCT_BASE64, + [0x36] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME | VCT_BASE64, + [0x37] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME | VCT_BASE64, + [0x38] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME | VCT_BASE64, + [0x39] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME | VCT_BASE64, [0x3a] = VCT_SEPARATOR | VCT_XMLNAMESTART, [0x3b] = VCT_SEPARATOR, [0x3c] = VCT_SEPARATOR, - [0x3d] = VCT_SEPARATOR, + [0x3d] = VCT_SEPARATOR | VCT_BASE64, [0x3e] = VCT_SEPARATOR, [0x3f] = VCT_SEPARATOR, [0x40] = VCT_SEPARATOR, From phk at FreeBSD.org Mon Jul 13 13:16:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 13 Jul 2020 13:16:07 +0000 (UTC) Subject: [master] 6ba917d81 Add parsing for structured-fields `sf-bytes` to VCL. Message-ID: <20200713131607.36DCB76F9@lists.varnish-cache.org> commit 6ba917d81d4df3ab9f4d4483c5a62a7851184bba Author: Poul-Henning Kamp Date: Mon Jul 13 13:10:43 2020 +0000 Add parsing for structured-fields `sf-bytes` to VCL. Currently fails with "unsupported", later will turn into BLOB. diff --git a/bin/varnishtest/tests/README b/bin/varnishtest/tests/README index 35d598dd2..1d01105de 100644 --- a/bin/varnishtest/tests/README +++ b/bin/varnishtest/tests/README @@ -22,6 +22,7 @@ Naming scheme id ~ ^f --> Security related tests id ~ ^g --> GZIP tests id ~ ^h --> HAproxy tests + id ~ ^i --> Interoperability and standards compliance id ~ ^j --> JAIL tests id ~ ^l --> VSL tests id ~ ^m --> VMOD tests excluding director diff --git a/bin/varnishtest/tests/i00000.vtc b/bin/varnishtest/tests/i00000.vtc new file mode 100644 index 000000000..14849f336 --- /dev/null +++ b/bin/varnishtest/tests/i00000.vtc @@ -0,0 +1,10 @@ +varnishtest "SF-blob parsing in VCL" + +varnish v1 -errvcl "BLOB must have n*3 base64 characters" { sub vcl_recv { :a: } } +varnish v2 -errvcl "BLOB must have n*3 base64 characters" { sub vcl_recv { :aa: } } +varnish v3 -errvcl "Illegal BLOB character:" { sub vcl_recv { :aa?: } } +varnish v4 -errvcl "BLOB must have n*3 base64 characters" { sub vcl_recv { :aaaa: } } +varnish v5 -errvcl "Wrong padding ('=') in BLOB" { sub vcl_recv { :aaa=aa: } } +varnish v6 -errvcl "Wrong padding ('=') in BLOB" { sub vcl_recv { :aaa==a: } } +varnish v7 -errvcl "Wrong padding ('=') in BLOB" { sub vcl_recv { :aaaa=a: } } +varnish v8 -errvcl "BLOB is not supported yet" { sub vcl_recv { :aaaa==: } } diff --git a/lib/libvcc/vcc_token.c b/lib/libvcc/vcc_token.c index 176b74eab..fa2d504cf 100644 --- a/lib/libvcc/vcc_token.c +++ b/lib/libvcc/vcc_token.c @@ -388,7 +388,7 @@ vcc_addtoken(struct vcc *tl, unsigned tok, void vcc_Lexer(struct vcc *tl, const struct source *sp, int eoi) { - const char *p, *q; + const char *p, *q, *r; unsigned u; for (p = sp->b; p < sp->e; ) { @@ -483,6 +483,49 @@ vcc_Lexer(struct vcc *tl, const struct source *sp, int eoi) return; } + /* Recognize BLOB (= SF-binary) */ + if (*p == ':') { + r = NULL; + for (q = p + 1; q < sp->e && vct_isbase64(*q); q++) { + if (r == NULL && *q == '=') + r = q; + } + if (q == sp->e || *q != ':') { + VSB_cat(tl->sb, + "Illegal BLOB character:\n"); + vcc_addtoken(tl, EOI, sp, q, q+1); + vcc_ErrWhere(tl, tl->t); + return; + } + if ((q - p) % 3 != 1) { + u = ((q - 1) - p) / 3; + vcc_addtoken(tl, EOI, sp, p + u * 3 + 1, q); + VSB_cat(tl->sb, + "BLOB must have n*3 base64 characters\n"); + vcc_ErrWhere(tl, tl->t); + return; + } + if (r == NULL) { + /* No padding; */ + } else if (r + 1 == q) { + /* One pad char */ + } else if (r + 2 == q && r[1] == '=') { + /* Two (valid) pad chars */ + } else { + VSB_cat(tl->sb, + "Wrong padding ('=') in BLOB:\n"); + vcc_addtoken(tl, EOI, sp, r, r+1); + vcc_ErrWhere(tl, tl->t); + return; + } + p = q + 1; + vcc_addtoken(tl, EOI, sp, p, q); + VSB_cat(tl->sb, + "BLOB is not supported yet.\n"); + vcc_ErrWhere(tl, tl->t); + return; + } + /* Match for the fixed tokens (see generate.py) */ u = vcl_fixed_token(p, &q); if (u != 0) { From nils.goroll at uplex.de Fri Jul 17 12:16:09 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 17 Jul 2020 12:16:09 +0000 (UTC) Subject: [master] e0ca9dcca gc dup docs of experimental parameter flag Message-ID: <20200717121609.6AEB91110E8@lists.varnish-cache.org> commit e0ca9dcca1ddb86f4bae57a2403b5c2b4f65ed2d Author: Nils Goroll Date: Fri Jul 17 14:14:16 2020 +0200 gc dup docs of experimental parameter flag diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst index 3c4a1d931..95c680908 100644 --- a/doc/sphinx/reference/varnishd.rst +++ b/doc/sphinx/reference/varnishd.rst @@ -501,10 +501,6 @@ flags are: The VCL programs must be reloaded for this parameter to take effect. -* `experimental` - - We're not really sure about this parameter, tell us what you find. - * `wizard` Do not touch unless you *really* know what you're doing. From dridi.boukelmoune at gmail.com Mon Jul 20 07:17:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 20 Jul 2020 07:17:07 +0000 (UTC) Subject: [master] adb48856c vcc: Plug minor leak Message-ID: <20200720071707.BFF4C5DCC@lists.varnish-cache.org> commit adb48856c138d4a4b9f127ca141d2cb3f91ef11f Author: Dridi Boukelmoune Date: Mon Jul 20 09:07:16 2020 +0200 vcc: Plug minor leak Spotted by Coverity diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index b181dc67d..f3e5b3b0b 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -523,6 +523,23 @@ vcc_new_source(const char *b, const char *e, const char *name) /*--------------------------------------------------------------------*/ +static void +vcc_destroy_source(struct source **spp) +{ + struct source *sp; + + AN(spp); + sp = *spp; + *spp = NULL; + + AN(sp); + free(sp->name); + free(sp->freeit); + free(sp); +} + +/*--------------------------------------------------------------------*/ + static struct source * vcc_file_source(const struct vcc *tl, const char *fn) { @@ -620,6 +637,7 @@ vcc_resolve_includes(struct vcc *tl) if (sp1->parent_tok) vcc_ErrWhere(tl, sp1->parent_tok); } + vcc_destroy_source(&sp); return; } sp->parent = t->src; From nils.goroll at uplex.de Mon Jul 20 07:22:30 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 20 Jul 2020 09:22:30 +0200 Subject: [master] adb48856c vcc: Plug minor leak In-Reply-To: <20200720071707.BFF4C5DCC@lists.varnish-cache.org> References: <20200720071707.BFF4C5DCC@lists.varnish-cache.org> Message-ID: <64f1d192-1811-d2f6-9fb8-202b2448c92c@uplex.de> I thought we would not care about the (many) leaks in vcc as it is running in a short lived process? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From dridi at varni.sh Mon Jul 20 07:55:54 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 20 Jul 2020 09:55:54 +0200 Subject: [master] adb48856c vcc: Plug minor leak In-Reply-To: <64f1d192-1811-d2f6-9fb8-202b2448c92c@uplex.de> References: <20200720071707.BFF4C5DCC@lists.varnish-cache.org> <64f1d192-1811-d2f6-9fb8-202b2448c92c@uplex.de> Message-ID: Agreed, and I think coverity scan doesn't consider it a leak if it's somehow tracked by the time we exit, which wasn't the case here. I only meant to silence a "false positive" (accurate but pointless since we'll exit soon after). Sent from my phone On Mon, Jul 20, 2020, 09:23 Nils Goroll wrote: > I thought we would not care about the (many) leaks in vcc as it is running > in a > short lived process? > > _______________________________________________ > varnish-commit mailing list > varnish-commit at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nils.goroll at uplex.de Fri Jul 24 10:46:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 24 Jul 2020 10:46:06 +0000 (UTC) Subject: [master] 77aba96ea solaris jail: two more privileges for unix domain sockets Message-ID: <20200724104606.EF9EC115D9E@lists.varnish-cache.org> commit 77aba96eaa9156150125ab8db21657eb46c3da88 Author: Nils Goroll Date: Fri Jul 24 12:43:16 2020 +0200 solaris jail: two more privileges for unix domain sockets When varnishd is started as root (or otherwise will a full privilege set), the user will rightly expect that whatever permissions and mode bits are set for a socket, varnishd will fix them to the specification from the -a argument. diff --git a/bin/varnishd/mgt/mgt_jail_solaris_tbl.h b/bin/varnishd/mgt/mgt_jail_solaris_tbl.h index f13d81331..062f25bb6 100644 --- a/bin/varnishd/mgt/mgt_jail_solaris_tbl.h +++ b/bin/varnishd/mgt/mgt_jail_solaris_tbl.h @@ -59,7 +59,9 @@ PRIV(MASTER_FILE, E , "file_write") PRIV(MASTER_STORAGE, E , "file_read") PRIV(MASTER_STORAGE, E , "file_write") -PRIV(MASTER_PRIVPORT, E , "file_write") // bind(AF_UNIX) +PRIV(MASTER_PRIVPORT, E , "file_write") // bind(AF_UNIX) +PRIV(MASTER_PRIVPORT, E , PRIV_FILE_CHOWN) // user= +PRIV(MASTER_PRIVPORT, E , PRIV_FILE_OWNER) // mode= PRIV(MASTER_PRIVPORT, E , "net_access") PRIV(MASTER_PRIVPORT, E , PRIV_NET_PRIVADDR) From phk at FreeBSD.org Fri Jul 24 15:32:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 24 Jul 2020 15:32:06 +0000 (UTC) Subject: [master] 24a9a5698 Make sure all headers listed in `Connection:` are hop-to-hop Message-ID: <20200724153206.D59C911BDA6@lists.varnish-cache.org> commit 24a9a5698e15444aa48e0df8b66f98c2a499339d Author: Poul-Henning Kamp Date: Fri Jul 24 12:16:40 2020 +0000 Make sure all headers listed in `Connection:` are hop-to-hop diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index cf949b765..4af545b57 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -765,9 +765,16 @@ http_DoConnection(struct http *hp) #include "tbl/http_headers.h" /*lint -restore */ - v = http_findhdr(hp, u, b); - if (v > 0) + for (v = HTTP_HDR_FIRST; v < hp->nhd; v++) { + Tcheck(hp->hd[v]); + if (hp->hd[v].e < hp->hd[v].b + u + 1) + continue; + if (hp->hd[v].b[u] != ':') + continue; + if (strncasecmp(b, hp->hd[v].b, u)) + continue; hp->hdf[v] |= HDF_FILTER; + } } return (retval); } diff --git a/bin/varnishtest/tests/c00016.vtc b/bin/varnishtest/tests/c00016.vtc index 847fb8730..8cbbfc79d 100644 --- a/bin/varnishtest/tests/c00016.vtc +++ b/bin/varnishtest/tests/c00016.vtc @@ -42,3 +42,27 @@ client c1 { rxresp expect resp.status == 400 } -run + +server s1 { + rxreq + expect req.http.baa == "" + expect req.http.baax == + txresp -hdr "Foox: 1" -hdr "foox: 2" -hdr "Connection: foox" +} -start + +varnish v1 -vcl+backend { + sub vcl_backend_fetch { + set bereq.http.baa = bereq.http.baax; + } + sub vcl_deliver { + set resp.http.foo = resp.http.foox; + } +} + +client c1 { + txreq -hdr "Baax: 1" -hdr "Baax: 2" -hdr "Connection: baax" + rxresp + expect resp.status == 200 + expect resp.http.foo == "" + expect resp.http.foox == +} -run From phk at FreeBSD.org Sat Jul 25 11:46:09 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Sat, 25 Jul 2020 11:46:09 +0000 (UTC) Subject: [master] 8bfcc907e A little light house-cleaning Message-ID: <20200725114609.F2C5B112CBA@lists.varnish-cache.org> commit 8bfcc907ef806d27219e1ddde627494f75259ad6 Author: Poul-Henning Kamp Date: Sat Jul 25 11:45:03 2020 +0000 A little light house-cleaning diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index c143565f3..c28f1ebeb 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -233,7 +233,6 @@ vbf_stp_mkbereq(struct worker *wrk, struct busyobj *bo) http_ForceHeader(bo->bereq0, H_Accept_Encoding, "gzip"); } http_ForceField(bo->bereq0, HTTP_HDR_PROTO, "HTTP/1.1"); - http_CopyHome(bo->bereq0); if (bo->stale_oc != NULL && ObjCheckFlag(bo->wrk, bo->stale_oc, OF_IMSCAND) && @@ -249,6 +248,7 @@ vbf_stp_mkbereq(struct worker *wrk, struct busyobj *bo) "If-None-Match: %s", q); } + http_CopyHome(bo->bereq0); HTTP_Setup(bo->bereq, bo->ws, bo->vsl, SLT_BereqMethod); bo->ws_bo = WS_Snapshot(bo->ws); HTTP_Clone(bo->bereq, bo->bereq0); diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index 4af545b57..9ecb5ce76 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -291,12 +291,7 @@ http_PutField(struct http *to, int field, const char *string) VSLb(to->vsl, SLT_LostHeader, "%s", string); return; } - to->hd[field].b = p; - to->hd[field].e = strchr(p, '\0'); - to->hdf[field] = 0; - http_VSLH(to, field); - if (field == HTTP_HDR_PROTO) - http_Proto(to); + http_SetH(to, field, p); } /*--------------------------------------------------------------------*/ @@ -1138,29 +1133,6 @@ HTTP_Merge(struct worker *wrk, struct objcore *oc, struct http *to) /*--------------------------------------------------------------------*/ -static void -http_filterfields(struct http *to, const struct http *fm, unsigned how) -{ - unsigned u; - - CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC); - CHECK_OBJ_NOTNULL(to, HTTP_MAGIC); - to->nhd = HTTP_HDR_FIRST; - to->status = fm->status; - for (u = HTTP_HDR_FIRST; u < fm->nhd; u++) { - Tcheck(fm->hd[u]); - if (http_isfiltered(fm, u, how)) - continue; - assert (to->nhd < to->shd); - to->hd[to->nhd] = fm->hd[u]; - to->hdf[to->nhd] = 0; - http_VSLH(to, to->nhd); - to->nhd++; - } -} - -/*--------------------------------------------------------------------*/ - static void http_linkh(const struct http *to, const struct http *fm, unsigned n) { @@ -1177,6 +1149,8 @@ http_linkh(const struct http *to, const struct http *fm, unsigned n) void http_FilterReq(struct http *to, const struct http *fm, unsigned how) { + unsigned u; + CHECK_OBJ_NOTNULL(to, HTTP_MAGIC); CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC); @@ -1184,7 +1158,19 @@ http_FilterReq(struct http *to, const struct http *fm, unsigned how) http_linkh(to, fm, HTTP_HDR_URL); http_linkh(to, fm, HTTP_HDR_PROTO); to->protover = fm->protover; - http_filterfields(to, fm, how); + to->status = fm->status; + + to->nhd = HTTP_HDR_FIRST; + for (u = HTTP_HDR_FIRST; u < fm->nhd; u++) { + Tcheck(fm->hd[u]); + if (http_isfiltered(fm, u, how)) + continue; + assert (to->nhd < to->shd); + to->hd[to->nhd] = fm->hd[u]; + to->hdf[to->nhd] = 0; + http_VSLH(to, to->nhd); + to->nhd++; + } } /*-------------------------------------------------------------------- @@ -1267,11 +1253,7 @@ http_PrintfHeader(struct http *to, const char *fmt, ...) http_fail(to); VSLbv(to->vsl, SLT_LostHeader, fmt, ap2); } else { - to->hd[to->nhd].b = p; - to->hd[to->nhd].e = p + sz; - to->hdf[to->nhd] = 0; - http_VSLH(to, to->nhd); - to->nhd++; + http_SetH(to, to->nhd++, p); } va_end(ap); va_end(ap2); @@ -1296,11 +1278,7 @@ http_TimeHeader(struct http *to, const char *fmt, vtim_real now) } strcpy(p, fmt); VTIM_format(now, strchr(p, '\0')); - to->hd[to->nhd].b = p; - to->hd[to->nhd].e = strchr(p, '\0'); - to->hdf[to->nhd] = 0; - http_VSLH(to, to->nhd); - to->nhd++; + http_SetH(to, to->nhd++, p); } /*--------------------------------------------------------------------*/ From dridi.boukelmoune at gmail.com Mon Jul 27 13:34:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 27 Jul 2020 13:34:08 +0000 (UTC) Subject: [master] 82491c2a8 vcc: PRIV_ arguments can be named too Message-ID: <20200727133408.387AE10952D@lists.varnish-cache.org> commit 82491c2a8abd65fb72b5588afbf07e8fdeeeede0 Author: Dridi Boukelmoune Date: Mon Jul 27 15:19:51 2020 +0200 vcc: PRIV_ arguments can be named too This otherwise doesn't work with [optional] arguments. diff --git a/bin/varnishtest/tests/m00052.vtc b/bin/varnishtest/tests/m00052.vtc new file mode 100644 index 000000000..580c01e1c --- /dev/null +++ b/bin/varnishtest/tests/m00052.vtc @@ -0,0 +1,21 @@ +varnishtest "priv_task with [optional] argument" + +varnish v1 -vcl { + import debug; + + backend be none; + + sub vcl_deliver { + set resp.http.none = debug.priv_task_with_option(); + set resp.http.one = debug.priv_task_with_option("one"); + set resp.http.two = debug.priv_task_with_option("two"); + } +} -start + +client c1 { + txreq + rxresp + expect resp.http.none == "" + expect resp.http.one == one + expect resp.http.two == one +} -run diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index 67623e0b2..60a29b585 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -515,6 +515,9 @@ vcc_func(struct vcc *tl, struct expr **e, const void *priv, vvp = VTAILQ_FIRST(&vv->children); if (!memcmp(vvp->value, "PRIV_", 5)) { fa->result = vcc_priv_arg(tl, vvp->value, sym); + vvp = VTAILQ_NEXT(vvp, list); + if (vvp != NULL) + fa->name = vvp->value; continue; } fa->type = VCC_Type(vvp->value); diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 4dbc5ab3b..af7598b6b 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -1162,3 +1162,14 @@ xyzzy_re_quote(VRT_CTX, VCL_STRING s) WS_MarkOverflow(ctx->ws); return (q); } + +VCL_STRING v_matchproto_(td_xyzzy_priv_task_with_option) +xyzzy_priv_task_with_option(VRT_CTX, struct VARGS(priv_task_with_option) *args) +{ + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + AN(args->priv); + if (args->priv->priv == NULL && args->valid_opt) + args->priv->priv = WS_Copy(ctx->ws, args->opt, -1); + return (args->priv->priv); +} diff --git a/lib/libvmod_debug/vmod_debug.vcc b/lib/libvmod_debug/vmod_debug.vcc index 63c326cf4..094db5311 100644 --- a/lib/libvmod_debug/vmod_debug.vcc +++ b/lib/libvmod_debug/vmod_debug.vcc @@ -304,3 +304,7 @@ Allow rollbacks. Must be called before the end of the task. $Function STRING re_quote(STRING) Quote an input string to be usable for an exact match in a regular expression. + +$Function STRING priv_task_with_option(PRIV_TASK priv, [STRING opt]) + +A function mixing a named PRIV_TASK with optional parameters. From phk at FreeBSD.org Mon Jul 27 13:43:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 27 Jul 2020 13:43:07 +0000 (UTC) Subject: [master] f3b62f8fa VMB: renovation Message-ID: <20200727134307.2A5CE109996@lists.varnish-cache.org> commit f3b62f8fa0f0aeda851a0576c0ca1fda0b955265 Author: Poul-Henning Kamp Date: Mon Jul 13 11:54:46 2020 +0000 VMB: renovation Use if we have it. Let Ancient FreeBSD without fall back to the __GNUCC_ inline assy. diff --git a/configure.ac b/configure.ac index f0f5b11db..b63840512 100644 --- a/configure.ac +++ b/configure.ac @@ -100,6 +100,9 @@ AC_SUBST(NET_LIBS, "${SOCKET_LIBS} ${NSL_LIBS}") # Userland slab allocator from Solaris, ported to other systems AC_CHECK_HEADERS([umem.h]) +# More portable vmb.h +AC_CHECK_HEADERS([stdatomic.h]) + # XXX: This _may_ be for OS/X AC_CHECK_LIBM AC_SUBST(LIBM) diff --git a/include/vmb.h b/include/vmb.h index 9aaa7e949..7029e29f6 100644 --- a/include/vmb.h +++ b/include/vmb.h @@ -29,60 +29,38 @@ * * Memory barriers * - * XXX: It is utterly braindamaged, that no standard facility for this - * XXX: is available. The "just use pthreads locking" excuse does not - * XXX: make sense, and does not apply to two unthreaded programs sharing - * XXX: a memory segment. */ #ifndef VMB_H_INCLUDED #define VMB_H_INCLUDED -#if defined(__FreeBSD__) -#include -#endif - -#if defined(__FreeBSD__) && __FreeBSD_version >= 800058 +#if defined(HAVE_STDATOMIC_H) -#include -#include -#define VMB() mb() -#define VWMB() wmb() -#define VRMB() rmb() +# include +# define VWMB() atomic_thread_fence(memory_order_release) +# define VRMB() atomic_thread_fence(memory_order_acquire) #elif defined(__amd64__) && defined(__GNUC__) -#define VMB() __asm __volatile("mfence;" : : : "memory") -#define VWMB() __asm __volatile("sfence;" : : : "memory") -#define VRMB() __asm __volatile("lfence;" : : : "memory") - -#elif defined(__arm__) - -#define VMB() -#define VWMB() -#define VRMB() +# define VWMB() __asm __volatile("sfence;" : : : "memory") +# define VRMB() __asm __volatile("lfence;" : : : "memory") #elif defined(__i386__) && defined(__GNUC__) -#define VMB() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") -#define VWMB() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") -#define VRMB() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") +# define VWMB() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") +# define VRMB() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") #elif defined(__sparc64__) && defined(__GNUC__) -#define VMB() __asm__ __volatile__ ("membar #MemIssue": : :"memory") -#define VWMB() VMB() -#define VRMB() VMB() +# define VWMB() __asm__ __volatile__ ("membar #MemIssue": : :"memory") +# define VRMB() __asm__ __volatile__ ("membar #MemIssue": : :"memory") #else -#define VMB_NEEDS_PTHREAD_WORKAROUND_THIS_IS_BAD_FOR_PERFORMANCE 1 - -void vmb_pthread(void); - -#define VMB() vmb_pthread() -#define VWMB() vmb_pthread() -#define VRMB() vmb_pthread() +# define VMB_NEEDS_PTHREAD_WORKAROUND_THIS_IS_BAD_FOR_PERFORMANCE 1 + void vmb_pthread(void); +# define VWMB() vmb_pthread() +# define VRMB() vmb_pthread() #endif From nils.goroll at uplex.de Tue Jul 28 07:44:08 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 28 Jul 2020 07:44:08 +0000 (UTC) Subject: [master] 7e144bde3 Haiku build fix Message-ID: <20200728074408.24E6F6178D@lists.varnish-cache.org> commit 7e144bde31aac46788c058351f07d4f97fe1a48f Author: David Carlier Date: Tue Jul 21 07:44:35 2020 +0000 Haiku build fix Mainly WCOREDUMP is undefine but WIFCORED has similar semantic. diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 9c3bbd02d..418851f86 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -34,7 +34,6 @@ #include "config.h" #include -#include #include #include @@ -49,6 +48,8 @@ #include "mgt.h" +#include "vapi/vsig.h" + #include "vbm.h" #include "vcli_serve.h" #include "vev.h" diff --git a/bin/varnishtest/vtc_subr.c b/bin/varnishtest/vtc_subr.c index 1d549ab60..087cb71b7 100644 --- a/bin/varnishtest/vtc_subr.c +++ b/bin/varnishtest/vtc_subr.c @@ -36,7 +36,6 @@ #include #include #include -#include #include #include "vtc.h" @@ -45,6 +44,9 @@ #include "vnum.h" #include "vre.h" +#include "vapi/voptget.h" +#include "vapi/vsig.h" + struct vsb * vtc_hex_to_bin(struct vtclog *vl, const char *arg) { @@ -145,10 +147,6 @@ vtc_expect(struct vtclog *vl, * otherwise, the process must die on exit(expect_status) */ -#ifndef WCOREDUMP -# define WCOREDUMP(s) (-1) -#endif - void vtc_wait4(struct vtclog *vl, long pid, int expect_status, int expect_signal, int allow_core) diff --git a/include/vapi/vsig.h b/include/vapi/vsig.h index 810b37dce..0c1bdcb7d 100644 --- a/include/vapi/vsig.h +++ b/include/vapi/vsig.h @@ -39,4 +39,14 @@ #include "tbl/vsig_list.h" +#include + +#ifndef WCOREDUMP +# ifdef WIFCORED +# define WCOREDUMP(s) WIFCORED(s) +# else +# define WCOREDUMP(s) (-1) +# endif +#endif + #endif /* VAPI_VSC_H_INCLUDED */ diff --git a/lib/libvarnish/vsub.c b/lib/libvarnish/vsub.c index eade83bac..56308f6b3 100644 --- a/lib/libvarnish/vsub.c +++ b/lib/libvarnish/vsub.c @@ -46,6 +46,8 @@ #include "vdef.h" +#include "vapi/vsig.h" + #include "vas.h" #include "vfil.h" #include "vlu.h" From nils.goroll at uplex.de Tue Jul 28 07:54:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 28 Jul 2020 07:54:07 +0000 (UTC) Subject: [master] 2d4da769f polish previous commit Message-ID: <20200728075407.1B59A61D91@lists.varnish-cache.org> commit 2d4da769f5f59374a6baf9f9f6efdf327d0dee8a Author: Nils Goroll Date: Tue Jul 28 09:51:51 2020 +0200 polish previous commit - remove a duplicate sys/wait.h include - remove a stray vapi/voptget.h include - move include in vsig.h to the right place diff --git a/bin/varnishtest/vtc_subr.c b/bin/varnishtest/vtc_subr.c index 087cb71b7..fe5fe2008 100644 --- a/bin/varnishtest/vtc_subr.c +++ b/bin/varnishtest/vtc_subr.c @@ -44,7 +44,6 @@ #include "vnum.h" #include "vre.h" -#include "vapi/voptget.h" #include "vapi/vsig.h" struct vsb * diff --git a/include/vapi/vsig.h b/include/vapi/vsig.h index 0c1bdcb7d..25e6647d5 100644 --- a/include/vapi/vsig.h +++ b/include/vapi/vsig.h @@ -32,6 +32,8 @@ #ifndef VAPI_VSIG_H_INCLUDED #define VAPI_VSIG_H_INCLUDED +#include + #define VSIG_SIGNAL(UPPER, lower) \ extern sig_atomic_t VSIG_##lower; \ void VSIG_Got_##lower(int sig); \ @@ -39,8 +41,6 @@ #include "tbl/vsig_list.h" -#include - #ifndef WCOREDUMP # ifdef WIFCORED # define WCOREDUMP(s) WIFCORED(s) diff --git a/lib/libvarnish/vsub.c b/lib/libvarnish/vsub.c index 56308f6b3..b528c12c4 100644 --- a/lib/libvarnish/vsub.c +++ b/lib/libvarnish/vsub.c @@ -33,8 +33,6 @@ #include "config.h" -#include - #include #include #include // Solaris closefrom(3c) From nils.goroll at uplex.de Tue Jul 28 08:01:08 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 28 Jul 2020 08:01:08 +0000 (UTC) Subject: [master] 6d4df3639 more include shuffling for vsig.h Message-ID: <20200728080108.38331621FF@lists.varnish-cache.org> commit 6d4df3639725bbec6d1657b07867ec44f4ba14f8 Author: Nils Goroll Date: Tue Jul 28 10:00:31 2020 +0200 more include shuffling for vsig.h diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c index f93ace25d..e3e76efa3 100644 --- a/bin/varnishadm/varnishadm.c +++ b/bin/varnishadm/varnishadm.c @@ -54,7 +54,6 @@ #include #include #include -#include #include #include #include diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 418851f86..6be347d92 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -36,7 +36,6 @@ #include #include -#include #include #include #include diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c index a2e94cb5d..63bf8e73e 100644 --- a/bin/varnishhist/varnishhist.c +++ b/bin/varnishhist/varnishhist.c @@ -38,7 +38,6 @@ #include #include #include -#include #include #include #include @@ -52,10 +51,10 @@ #include "vapi/vsl.h" #include "vapi/vsm.h" #include "vapi/voptget.h" +#include "vapi/vsig.h" #include "vas.h" #include "vut.h" #include "vtim.h" -#include "vapi/vsig.h" #if 1 #define AC(x) assert((x) != ERR) diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c index b8d5c7cf4..f7c486db1 100644 --- a/bin/varnishstat/varnishstat_curses.c +++ b/bin/varnishstat/varnishstat_curses.c @@ -38,7 +38,6 @@ #include #include #include -#include #include #include diff --git a/include/vapi/vsig.h b/include/vapi/vsig.h index 25e6647d5..3f896e037 100644 --- a/include/vapi/vsig.h +++ b/include/vapi/vsig.h @@ -33,6 +33,7 @@ #define VAPI_VSIG_H_INCLUDED #include +#include #define VSIG_SIGNAL(UPPER, lower) \ extern sig_atomic_t VSIG_##lower; \ diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c index 4bc283ad2..6ec9fd536 100644 --- a/lib/libvarnishapi/vsm.c +++ b/lib/libvarnishapi/vsm.c @@ -38,7 +38,6 @@ #include #include #include -#include #include #include #include diff --git a/lib/libvarnishapi/vut.c b/lib/libvarnishapi/vut.c index 165c2e34d..6e8384572 100644 --- a/lib/libvarnishapi/vut.c +++ b/lib/libvarnishapi/vut.c @@ -40,7 +40,6 @@ #include #include #include -#include #include /* for MUSL */ #include "compat/daemon.h" @@ -55,11 +54,12 @@ #include "vsb.h" #include "vfil.h" -#include "vut.h" - #include "vapi/voptget.h" #include "vapi/vsig.h" +#include "vut.h" + + static int vut_synopsis(const struct vopt_spec *); static int vut_options(const struct vopt_spec *); From dridi.boukelmoune at gmail.com Tue Jul 28 08:09:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 28 Jul 2020 08:09:07 +0000 (UTC) Subject: [master] 81a7e8513 doc: It is now possible to name PRIV_ arguments Message-ID: <20200728080907.DD4386277E@lists.varnish-cache.org> commit 81a7e851388f726ac96b7e32c387e9b7f969485c Author: Dridi Boukelmoune Date: Tue Jul 28 09:58:42 2020 +0200 doc: It is now possible to name PRIV_ arguments Contrary to what the documentation stated, PRIV_ arguments aren't unnamed by nature with the current vmodtool script. Both the generated struct and JSON descriptor would account for named PRIV_ arguments and the only limitation was in libvcc. It was really a bug in the sense that vmodtool would generate C code incompatible with the code generated by libvcc. It was simpler to fix libvcc and allow VMOD authors to give meaningful names to PRIVs if they happen to use them in conjunction with [optional arguments]. Refs 82491c2a8abd diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst index ca448bb8e..f4edad883 100644 --- a/doc/sphinx/reference/vmod.rst +++ b/doc/sphinx/reference/vmod.rst @@ -156,7 +156,7 @@ Optional arguments The vmod.vcc declaration also allows for optional arguments in square brackets like so:: - $Function VOID opt(PRIV_TASK, INT four = 4, [ STRING opt]) + $Function VOID opt(PRIV_TASK priv, INT four = 4, [STRING opt]) With any optional argument present, the C function prototype looks completely different: @@ -185,9 +185,6 @@ declarations: with `n` starting at 1 and incrementing with the argument's position. - Note that in particular also ``PRIV_*`` arguments (which are - unnamed by definition) are passed as ``arg``\ `n` - .. _ref-vmod-vcl-c-objects: Objects and methods From dridi.boukelmoune at gmail.com Tue Jul 28 09:23:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 28 Jul 2020 09:23:07 +0000 (UTC) Subject: [master] 1c3b78411 Start skeleton release notes for the next version. Message-ID: <20200728092307.4DE9F94787@lists.varnish-cache.org> commit 1c3b78411b2a16da8b3d5c2b241d49ca333464cc Author: Geoff Simmons Date: Tue Sep 25 16:31:17 2018 +0200 Start skeleton release notes for the next version. Restructured so that: * 'Upgrading' is limited to work that has to be done to upgrade from a current deployment to the new version. * 'Changes' is a comprehensive, user-level description of changes and new features. Conflicts: doc/sphinx/whats-new/index.rst diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst new file mode 100644 index 000000000..fbda9d6b9 --- /dev/null +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -0,0 +1,73 @@ +**Note: This is a working document for a future release, with running +updates for changes in the development branch. For changes in the +released versions of Varnish, see:** :ref:`whats-new-index` + +.. _whatsnew_changes_CURRENT: + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +Changes in Varnish **$NEXT_RELEASE** +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +For information about updating your current Varnish deployment to the +new version, see :ref:`whatsnew_upgrading_CURRENT`. + +A more detailed and technical account of changes in Varnish, with +links to issues that have been fixed and pull requests that have been +merged, may be found in the `change log`_. + +.. _change log: https://github.com/varnishcache/varnish-cache/blob/master/doc/changes.rst + +varnishd +======== + +Parameters +~~~~~~~~~~ + +**XXX changes in -p parameters** + +Other changes in varnishd +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Changes to VCL +============== + +VCL variables +~~~~~~~~~~~~~ + +**XXX new, deprecated or removed variables, or changed semantics** + +Other changes to VCL +~~~~~~~~~~~~~~~~~~~~ + +VMODs +===== + +**XXX changes in the bundled VMODs** + +varnishlog +========== + +**XXX changes concerning varnishlog(1) and/or vsl(7)** + +varnishadm +========== + +**XXX changes concerning varnishadm(1) and/or varnish-cli(7)** + +varnishstat +=========== + +**XXX changes concerning varnishstat(1) and/or varnish-counters(7)** + +varnishtest +=========== + +**XXX changes concerning varnishtest(1) and/or vtc(7)** + +Changes for developers and VMOD authors +======================================= + +**XXX changes concerning VRT, the public APIs, source code organization, +builds etc.** + +*eof* diff --git a/doc/sphinx/whats-new/index.rst b/doc/sphinx/whats-new/index.rst index 652248b20..29dc31d84 100644 --- a/doc/sphinx/whats-new/index.rst +++ b/doc/sphinx/whats-new/index.rst @@ -8,6 +8,19 @@ This section describes the changes and improvements between different versions of Varnish, and what upgrading between the different versions entail. +Varnish **$NEXT_RELEASE** +------------------------- + +**Note: These are working documents for a future release, with running +updates for changes in the development branch. For changes in the +released versions of Varnish, see the chapters listed below.** + +.. toctree:: + :maxdepth: 2 + + changes-trunk + upgrading-trunk + Varnish 6.4 ----------- diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst new file mode 100644 index 000000000..6143fde99 --- /dev/null +++ b/doc/sphinx/whats-new/upgrading-trunk.rst @@ -0,0 +1,33 @@ +**Note: This is a working document for a future release, with running +updates for changes in the development branch. For changes in the +released versions of Varnish, see:** :ref:`whats-new-index` + +.. _whatsnew_upgrading_CURRENT: + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +Upgrading to Varnish **$NEXT_RELEASE** +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +**XXX: how to upgrade from previous deployments to this +version. Limited to work that has to be done for an upgrade, new +features are listed in "Changes". Explicitly mention what does *not* +have to be changed, especially in VCL. May include, but is not limited +to:** + +* Elements of VCL that have been removed or are deprecated, or whose + semantics have changed. + +* -p parameters that have been removed or are deprecated, or whose + semantics have changed. + +* Changes in the CLI. + +* Changes in the output or interpretation of stats or the log, including + changes affecting varnishncsa/-hist/-top. + +* Changes that may be necessary in VTCs or in the use of varnishtest. + +* Changes in public APIs that may require changes in VMODs or VAPI/VUT + clients. + +*eof* From nils.goroll at uplex.de Tue Jul 28 10:16:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 28 Jul 2020 10:16:07 +0000 (UTC) Subject: [master] 803b9a0cb use a real target for the `$(srcdir)` to `$(builddir)` links for sphinx Message-ID: <20200728101607.DD4E795F11@lists.varnish-cache.org> commit 803b9a0cbdb22e8cda3cf9b4acb455351529df8d Author: Nils Goroll Date: Tue Jul 28 10:45:56 2020 +0200 use a real target for the `$(srcdir)` to `$(builddir)` links for sphinx The phony target was executed unconditionally and would thus race. A real target (using `index.rst`) does not race and has the additional advantage of simplifying the code (avoiding a condition in the shell code). It seems (at least GNU) `make` outsmarts the attempt to use a file from `$(srcdir)` as a marker for the link to have succeeded when `$(builddir)/index.rst` is used as the target. Using `$(abs_builddir)` seems to dtrt. Largely unrelated to this issue: The links are required for targets in `BUILT_SOURCES`, so previously I added a dependency to the `all` target. Having the link target in `BUILD_SOURCES` seems to be the better option. Fixes #3370 (I hope) diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am index 4ef49bc66..f7b67755b 100644 --- a/doc/sphinx/Makefile.am +++ b/doc/sphinx/Makefile.am @@ -18,26 +18,25 @@ help: clean: -rm -rf $(BUILDDIR)/* $(CLEANFILES) -# use index.rst as an indicator if we have copied already -.PHONY: link_srcdir -link_srcdir: - if test "x$(srcdir)" != "x$(builddir)" && test ! -f index.rst; then \ - s=`realpath $(srcdir)`; \ - for f in `cd $$s && find . -type f`; do \ - d=`dirname $$f`; \ - test -d $$d || mkdir -p $$d; \ - test -f $$f || ln -s $$s/$$f $$f; \ - done \ - fi +# sphinx does not support include paths or anything comparable: +# link source tree files for an out-of-tree build +rstlinks = $(abs_builddir)/doc/sphinx/index.rst +$(rstlinks): + s=`realpath $(srcdir)`; \ + for f in `cd $$s && find . -type f`; do \ + d=`dirname $$f`; \ + test -d $$d || mkdir -p $$d; \ + test -f $$f || ln -s $$s/$$f $$f; \ + done + +BUILT_SOURCES = $(rstlinks) # work around for make html called within doc/sphinx .PHONY: graphviz graphviz: cd ../graphviz && $(MAKE) html -sphinx_prereq: link_srcdir graphviz conf.py - -all: link_srcdir +sphinx_prereq: graphviz conf.py html: sphinx_prereq $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @@ -77,12 +76,12 @@ distclean-local: include/cli.rst: $(top_builddir)/bin/varnishd/varnishd $(top_builddir)/bin/varnishd/varnishd -x cli > ${@}_ mv ${@}_ ${@} -BUILT_SOURCES = include/cli.rst +RST_BUILT = include/cli.rst include/params.rst: $(top_builddir)/bin/varnishd/varnishd $(top_builddir)/bin/varnishd/varnishd -x parameter > ${@}_ mv ${@}_ ${@} -BUILT_SOURCES += include/params.rst +RST_BUILT += include/params.rst COUNTERS = \ $(top_srcdir)/bin/varnishd/VSC_main.vsc \ @@ -101,7 +100,7 @@ include/counters.rst: $(top_srcdir)/lib/libvcc/vsctool.py $(COUNTERS) done mv ${@}_ ${@} -BUILT_SOURCES += include/counters.rst +RST_BUILT += include/counters.rst include/varnishncsa_options.rst: $(top_builddir)/bin/varnishncsa/varnishncsa $(top_builddir)/bin/varnishncsa/varnishncsa --options > ${@}_ @@ -109,7 +108,7 @@ include/varnishncsa_options.rst: $(top_builddir)/bin/varnishncsa/varnishncsa include/varnishncsa_synopsis.rst: $(top_builddir)/bin/varnishncsa/varnishncsa $(top_builddir)/bin/varnishncsa/varnishncsa --synopsis > ${@}_ mv ${@}_ ${@} -BUILT_SOURCES += include/varnishncsa_options.rst \ +RST_BUILT += include/varnishncsa_options.rst \ include/varnishncsa_synopsis.rst include/varnishlog_options.rst: $(top_builddir)/bin/varnishlog/varnishlog @@ -118,7 +117,7 @@ include/varnishlog_options.rst: $(top_builddir)/bin/varnishlog/varnishlog include/varnishlog_synopsis.rst: $(top_builddir)/bin/varnishlog/varnishlog $(top_builddir)/bin/varnishlog/varnishlog --synopsis > ${@}_ mv ${@}_ ${@} -BUILT_SOURCES += include/varnishlog_options.rst \ +RST_BUILT += include/varnishlog_options.rst \ include/varnishlog_synopsis.rst include/varnishtop_options.rst: $(top_builddir)/bin/varnishtop/varnishtop @@ -127,7 +126,7 @@ include/varnishtop_options.rst: $(top_builddir)/bin/varnishtop/varnishtop include/varnishtop_synopsis.rst: $(top_builddir)/bin/varnishtop/varnishtop $(top_builddir)/bin/varnishtop/varnishtop --synopsis > ${@}_ mv ${@}_ ${@} -BUILT_SOURCES += include/varnishtop_options.rst \ +RST_BUILT += include/varnishtop_options.rst \ include/varnishtop_synopsis.rst include/varnishhist_options.rst: $(top_builddir)/bin/varnishhist/varnishhist @@ -136,7 +135,7 @@ include/varnishhist_options.rst: $(top_builddir)/bin/varnishhist/varnishhist include/varnishhist_synopsis.rst: $(top_builddir)/bin/varnishhist/varnishhist $(top_builddir)/bin/varnishhist/varnishhist --synopsis > ${@}_ mv ${@}_ ${@} -BUILT_SOURCES += include/varnishhist_options.rst \ +RST_BUILT += include/varnishhist_options.rst \ include/varnishhist_synopsis.rst include/varnishstat_options.rst: $(top_builddir)/bin/varnishstat/varnishstat @@ -148,14 +147,14 @@ include/varnishstat_synopsis.rst: $(top_builddir)/bin/varnishstat/varnishstat include/varnishstat_bindings.rst: $(top_builddir)/bin/varnishstat/varnishstat $(top_builddir)/bin/varnishstat/varnishstat --bindings > ${@}_ mv ${@}_ ${@} -BUILT_SOURCES += include/varnishstat_options.rst \ +RST_BUILT += include/varnishstat_options.rst \ include/varnishstat_synopsis.rst \ include/varnishstat_bindings.rst include/vsl-tags.rst: $(top_builddir)/lib/libvarnishapi/vsl2rst $(top_builddir)/lib/libvarnishapi/vsl2rst > ${@}_ mv ${@}_ ${@} -BUILT_SOURCES += include/vsl-tags.rst +RST_BUILT += include/vsl-tags.rst VTCSYN_SRC = $(top_srcdir)/bin/varnishtest/vtc.c \ $(top_srcdir)/bin/varnishtest/vtc_barrier.c \ @@ -170,43 +169,45 @@ VTCSYN_SRC = $(top_srcdir)/bin/varnishtest/vtc.c \ include/vtc-syntax.rst: vtc-syntax.py $(VTCSYN_SRC) $(AM_V_GEN) $(PYTHON) $(top_srcdir)/doc/sphinx/vtc-syntax.py $(VTCSYN_SRC) > ${@}_ @mv ${@}_ ${@} -BUILT_SOURCES += include/vtc-syntax.rst +RST_BUILT += include/vtc-syntax.rst # XXX copy/paste rules need some TLC include/vmod_std.generated.rst: $(top_builddir)/lib/libvmod_std/vmod_std.rst cp $(top_builddir)/lib/libvmod_std/vmod_std.rst $@ -BUILT_SOURCES += include/vmod_std.generated.rst +RST_BUILT += include/vmod_std.generated.rst include/vmod_directors.generated.rst: $(top_builddir)/lib/libvmod_directors/vmod_directors.rst cp $(top_builddir)/lib/libvmod_directors/vmod_directors.rst $@ -BUILT_SOURCES += include/vmod_directors.generated.rst +RST_BUILT += include/vmod_directors.generated.rst include/vmod_purge.generated.rst: $(top_builddir)/lib/libvmod_purge/vmod_purge.rst cp $(top_builddir)/lib/libvmod_purge/vmod_purge.rst $@ -BUILT_SOURCES += include/vmod_purge.generated.rst +RST_BUILT += include/vmod_purge.generated.rst include/vmod_vtc.generated.rst: $(top_builddir)/lib/libvmod_vtc/vmod_vtc.rst cp $(top_builddir)/lib/libvmod_vtc/vmod_vtc.rst $@ -BUILT_SOURCES += include/vmod_vtc.generated.rst +RST_BUILT += include/vmod_vtc.generated.rst include/vmod_blob.generated.rst: $(top_builddir)/lib/libvmod_blob/vmod_blob.rst cp $(top_builddir)/lib/libvmod_blob/vmod_blob.rst $@ -BUILT_SOURCES += include/vmod_blob.generated.rst +RST_BUILT += include/vmod_blob.generated.rst include/vmod_cookie.generated.rst: $(top_builddir)/lib/libvmod_cookie/vmod_cookie.rst cp $(top_builddir)/lib/libvmod_cookie/vmod_cookie.rst $@ -BUILT_SOURCES += include/vmod_cookie.generated.rst +RST_BUILT += include/vmod_cookie.generated.rst include/vmod_unix.generated.rst: $(top_builddir)/lib/libvmod_unix/vmod_unix.rst cp $(top_builddir)/lib/libvmod_unix/vmod_unix.rst $@ -BUILT_SOURCES += include/vmod_unix.generated.rst +RST_BUILT += include/vmod_unix.generated.rst include/vmod_proxy.generated.rst: $(top_builddir)/lib/libvmod_proxy/vmod_proxy.rst cp $(top_builddir)/lib/libvmod_proxy/vmod_proxy.rst $@ -BUILT_SOURCES += include/vmod_proxy.generated.rst +RST_BUILT += include/vmod_proxy.generated.rst + +EXTRA_DIST += $(RST_BUILT) +BUILT_SOURCES += $(RST_BUILT) +CLEANFILES = $(RST_BUILT) -EXTRA_DIST += $(BUILT_SOURCES) -CLEANFILES = $(BUILT_SOURCES) .NOPATH: $(BUILT_SOURCES) From nils.goroll at uplex.de Tue Jul 28 10:43:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 28 Jul 2020 10:43:07 +0000 (UTC) Subject: [master] d7d31e47e more sphinx build fiddling Message-ID: <20200728104307.51EE397B7F@lists.varnish-cache.org> commit d7d31e47eeafa7457bc65047367aec8ba4f062f8 Author: Nils Goroll Date: Tue Jul 28 12:37:50 2020 +0200 more sphinx build fiddling In the previous commit, I argued that adding the link job to `BUILT_SOURCES` was the better option than adding it to the `all` target, but I overlooked the fact that also the `BUILT_SOURCES` are built in parallel (only before all other targets). Thus, we still need an explicit dependency on the link job. diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am index f7b67755b..02392dc4b 100644 --- a/doc/sphinx/Makefile.am +++ b/doc/sphinx/Makefile.am @@ -208,6 +208,6 @@ RST_BUILT += include/vmod_proxy.generated.rst EXTRA_DIST += $(RST_BUILT) BUILT_SOURCES += $(RST_BUILT) CLEANFILES = $(RST_BUILT) - +$(RST_BUILT): $(rstlinks) .NOPATH: $(BUILT_SOURCES) From nils.goroll at uplex.de Tue Jul 28 11:10:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 28 Jul 2020 11:10:07 +0000 (UTC) Subject: [master] 2754e02e2 be more specific about files we need to copy from `$(srcdir)` Message-ID: <20200728111007.5B3A79B87A@lists.varnish-cache.org> commit 2754e02e245542e5d38b1419dc6d6f01a3ba0a6c Author: Nils Goroll Date: Tue Jul 28 13:07:02 2020 +0200 be more specific about files we need to copy from `$(srcdir)` in particular, do not copy files which we are about to generate (side note: If I had not before, now I really detest this... all I really want is working VTEST builds, there should be a better way) diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am index 02392dc4b..c5bcd9c22 100644 --- a/doc/sphinx/Makefile.am +++ b/doc/sphinx/Makefile.am @@ -23,7 +23,7 @@ clean: rstlinks = $(abs_builddir)/doc/sphinx/index.rst $(rstlinks): s=`realpath $(srcdir)`; \ - for f in `cd $$s && find . -type f`; do \ + for f in `cd $$s && find . -name \*.rst \! -name \*.generated.rst -type f`; do \ d=`dirname $$f`; \ test -d $$d || mkdir -p $$d; \ test -f $$f || ln -s $$s/$$f $$f; \ From nils.goroll at uplex.de Tue Jul 28 11:18:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 28 Jul 2020 11:18:07 +0000 (UTC) Subject: [master] 372155993 fix a stupid mistake which I overlooked Message-ID: <20200728111807.536DF9BE63@lists.varnish-cache.org> commit 37215599397ba5df32712b2a45c1a0e299063d5d Author: Nils Goroll Date: Tue Jul 28 13:13:11 2020 +0200 fix a stupid mistake which I overlooked ... trying to understand the reason for failing builds. Apologies! diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am index c5bcd9c22..14d0b9ad9 100644 --- a/doc/sphinx/Makefile.am +++ b/doc/sphinx/Makefile.am @@ -20,7 +20,7 @@ clean: # sphinx does not support include paths or anything comparable: # link source tree files for an out-of-tree build -rstlinks = $(abs_builddir)/doc/sphinx/index.rst +rstlinks = $(abs_builddir)/index.rst $(rstlinks): s=`realpath $(srcdir)`; \ for f in `cd $$s && find . -name \*.rst \! -name \*.generated.rst -type f`; do \ From nils.goroll at uplex.de Tue Jul 28 11:26:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 28 Jul 2020 11:26:06 +0000 (UTC) Subject: [master] 3f2be2ce8 Full abort on the sphinx links mess Message-ID: <20200728112606.D04FAA0329@lists.varnish-cache.org> commit 3f2be2ce824e7e13fcc930b6185fe414e701f40e Author: Nils Goroll Date: Tue Jul 28 13:23:41 2020 +0200 Full abort on the sphinx links mess Either I need to come up with a better idea or leave this with someone else. This reverts commit 37215599397ba5df32712b2a45c1a0e299063d5d. This reverts commit 2754e02e245542e5d38b1419dc6d6f01a3ba0a6c. This reverts commit d7d31e47eeafa7457bc65047367aec8ba4f062f8. This reverts commit 803b9a0cbdb22e8cda3cf9b4acb455351529df8d. diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am index 14d0b9ad9..4ef49bc66 100644 --- a/doc/sphinx/Makefile.am +++ b/doc/sphinx/Makefile.am @@ -18,25 +18,26 @@ help: clean: -rm -rf $(BUILDDIR)/* $(CLEANFILES) -# sphinx does not support include paths or anything comparable: -# link source tree files for an out-of-tree build -rstlinks = $(abs_builddir)/index.rst -$(rstlinks): - s=`realpath $(srcdir)`; \ - for f in `cd $$s && find . -name \*.rst \! -name \*.generated.rst -type f`; do \ - d=`dirname $$f`; \ - test -d $$d || mkdir -p $$d; \ - test -f $$f || ln -s $$s/$$f $$f; \ - done - -BUILT_SOURCES = $(rstlinks) +# use index.rst as an indicator if we have copied already +.PHONY: link_srcdir +link_srcdir: + if test "x$(srcdir)" != "x$(builddir)" && test ! -f index.rst; then \ + s=`realpath $(srcdir)`; \ + for f in `cd $$s && find . -type f`; do \ + d=`dirname $$f`; \ + test -d $$d || mkdir -p $$d; \ + test -f $$f || ln -s $$s/$$f $$f; \ + done \ + fi # work around for make html called within doc/sphinx .PHONY: graphviz graphviz: cd ../graphviz && $(MAKE) html -sphinx_prereq: graphviz conf.py +sphinx_prereq: link_srcdir graphviz conf.py + +all: link_srcdir html: sphinx_prereq $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @@ -76,12 +77,12 @@ distclean-local: include/cli.rst: $(top_builddir)/bin/varnishd/varnishd $(top_builddir)/bin/varnishd/varnishd -x cli > ${@}_ mv ${@}_ ${@} -RST_BUILT = include/cli.rst +BUILT_SOURCES = include/cli.rst include/params.rst: $(top_builddir)/bin/varnishd/varnishd $(top_builddir)/bin/varnishd/varnishd -x parameter > ${@}_ mv ${@}_ ${@} -RST_BUILT += include/params.rst +BUILT_SOURCES += include/params.rst COUNTERS = \ $(top_srcdir)/bin/varnishd/VSC_main.vsc \ @@ -100,7 +101,7 @@ include/counters.rst: $(top_srcdir)/lib/libvcc/vsctool.py $(COUNTERS) done mv ${@}_ ${@} -RST_BUILT += include/counters.rst +BUILT_SOURCES += include/counters.rst include/varnishncsa_options.rst: $(top_builddir)/bin/varnishncsa/varnishncsa $(top_builddir)/bin/varnishncsa/varnishncsa --options > ${@}_ @@ -108,7 +109,7 @@ include/varnishncsa_options.rst: $(top_builddir)/bin/varnishncsa/varnishncsa include/varnishncsa_synopsis.rst: $(top_builddir)/bin/varnishncsa/varnishncsa $(top_builddir)/bin/varnishncsa/varnishncsa --synopsis > ${@}_ mv ${@}_ ${@} -RST_BUILT += include/varnishncsa_options.rst \ +BUILT_SOURCES += include/varnishncsa_options.rst \ include/varnishncsa_synopsis.rst include/varnishlog_options.rst: $(top_builddir)/bin/varnishlog/varnishlog @@ -117,7 +118,7 @@ include/varnishlog_options.rst: $(top_builddir)/bin/varnishlog/varnishlog include/varnishlog_synopsis.rst: $(top_builddir)/bin/varnishlog/varnishlog $(top_builddir)/bin/varnishlog/varnishlog --synopsis > ${@}_ mv ${@}_ ${@} -RST_BUILT += include/varnishlog_options.rst \ +BUILT_SOURCES += include/varnishlog_options.rst \ include/varnishlog_synopsis.rst include/varnishtop_options.rst: $(top_builddir)/bin/varnishtop/varnishtop @@ -126,7 +127,7 @@ include/varnishtop_options.rst: $(top_builddir)/bin/varnishtop/varnishtop include/varnishtop_synopsis.rst: $(top_builddir)/bin/varnishtop/varnishtop $(top_builddir)/bin/varnishtop/varnishtop --synopsis > ${@}_ mv ${@}_ ${@} -RST_BUILT += include/varnishtop_options.rst \ +BUILT_SOURCES += include/varnishtop_options.rst \ include/varnishtop_synopsis.rst include/varnishhist_options.rst: $(top_builddir)/bin/varnishhist/varnishhist @@ -135,7 +136,7 @@ include/varnishhist_options.rst: $(top_builddir)/bin/varnishhist/varnishhist include/varnishhist_synopsis.rst: $(top_builddir)/bin/varnishhist/varnishhist $(top_builddir)/bin/varnishhist/varnishhist --synopsis > ${@}_ mv ${@}_ ${@} -RST_BUILT += include/varnishhist_options.rst \ +BUILT_SOURCES += include/varnishhist_options.rst \ include/varnishhist_synopsis.rst include/varnishstat_options.rst: $(top_builddir)/bin/varnishstat/varnishstat @@ -147,14 +148,14 @@ include/varnishstat_synopsis.rst: $(top_builddir)/bin/varnishstat/varnishstat include/varnishstat_bindings.rst: $(top_builddir)/bin/varnishstat/varnishstat $(top_builddir)/bin/varnishstat/varnishstat --bindings > ${@}_ mv ${@}_ ${@} -RST_BUILT += include/varnishstat_options.rst \ +BUILT_SOURCES += include/varnishstat_options.rst \ include/varnishstat_synopsis.rst \ include/varnishstat_bindings.rst include/vsl-tags.rst: $(top_builddir)/lib/libvarnishapi/vsl2rst $(top_builddir)/lib/libvarnishapi/vsl2rst > ${@}_ mv ${@}_ ${@} -RST_BUILT += include/vsl-tags.rst +BUILT_SOURCES += include/vsl-tags.rst VTCSYN_SRC = $(top_srcdir)/bin/varnishtest/vtc.c \ $(top_srcdir)/bin/varnishtest/vtc_barrier.c \ @@ -169,45 +170,43 @@ VTCSYN_SRC = $(top_srcdir)/bin/varnishtest/vtc.c \ include/vtc-syntax.rst: vtc-syntax.py $(VTCSYN_SRC) $(AM_V_GEN) $(PYTHON) $(top_srcdir)/doc/sphinx/vtc-syntax.py $(VTCSYN_SRC) > ${@}_ @mv ${@}_ ${@} -RST_BUILT += include/vtc-syntax.rst +BUILT_SOURCES += include/vtc-syntax.rst # XXX copy/paste rules need some TLC include/vmod_std.generated.rst: $(top_builddir)/lib/libvmod_std/vmod_std.rst cp $(top_builddir)/lib/libvmod_std/vmod_std.rst $@ -RST_BUILT += include/vmod_std.generated.rst +BUILT_SOURCES += include/vmod_std.generated.rst include/vmod_directors.generated.rst: $(top_builddir)/lib/libvmod_directors/vmod_directors.rst cp $(top_builddir)/lib/libvmod_directors/vmod_directors.rst $@ -RST_BUILT += include/vmod_directors.generated.rst +BUILT_SOURCES += include/vmod_directors.generated.rst include/vmod_purge.generated.rst: $(top_builddir)/lib/libvmod_purge/vmod_purge.rst cp $(top_builddir)/lib/libvmod_purge/vmod_purge.rst $@ -RST_BUILT += include/vmod_purge.generated.rst +BUILT_SOURCES += include/vmod_purge.generated.rst include/vmod_vtc.generated.rst: $(top_builddir)/lib/libvmod_vtc/vmod_vtc.rst cp $(top_builddir)/lib/libvmod_vtc/vmod_vtc.rst $@ -RST_BUILT += include/vmod_vtc.generated.rst +BUILT_SOURCES += include/vmod_vtc.generated.rst include/vmod_blob.generated.rst: $(top_builddir)/lib/libvmod_blob/vmod_blob.rst cp $(top_builddir)/lib/libvmod_blob/vmod_blob.rst $@ -RST_BUILT += include/vmod_blob.generated.rst +BUILT_SOURCES += include/vmod_blob.generated.rst include/vmod_cookie.generated.rst: $(top_builddir)/lib/libvmod_cookie/vmod_cookie.rst cp $(top_builddir)/lib/libvmod_cookie/vmod_cookie.rst $@ -RST_BUILT += include/vmod_cookie.generated.rst +BUILT_SOURCES += include/vmod_cookie.generated.rst include/vmod_unix.generated.rst: $(top_builddir)/lib/libvmod_unix/vmod_unix.rst cp $(top_builddir)/lib/libvmod_unix/vmod_unix.rst $@ -RST_BUILT += include/vmod_unix.generated.rst +BUILT_SOURCES += include/vmod_unix.generated.rst include/vmod_proxy.generated.rst: $(top_builddir)/lib/libvmod_proxy/vmod_proxy.rst cp $(top_builddir)/lib/libvmod_proxy/vmod_proxy.rst $@ -RST_BUILT += include/vmod_proxy.generated.rst +BUILT_SOURCES += include/vmod_proxy.generated.rst -EXTRA_DIST += $(RST_BUILT) -BUILT_SOURCES += $(RST_BUILT) -CLEANFILES = $(RST_BUILT) -$(RST_BUILT): $(rstlinks) +EXTRA_DIST += $(BUILT_SOURCES) +CLEANFILES = $(BUILT_SOURCES) .NOPATH: $(BUILT_SOURCES) From nils.goroll at uplex.de Tue Jul 28 12:20:08 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 28 Jul 2020 12:20:08 +0000 (UTC) Subject: [master] b41d4bea0 vtest: git-clean the tree Message-ID: <20200728122008.9AA39A2C03@lists.varnish-cache.org> commit b41d4bea0b840a9f593e0491aaa5b0d1fbfd3bea Author: Nils Goroll Date: Tue Jul 28 14:17:52 2020 +0200 vtest: git-clean the tree Some mistake in the previous commits has left vtest maschine's tree in an unclean state with bad symlinks. Try to avoid such issues in the future. diff --git a/tools/vtest.sh b/tools/vtest.sh index de1464118..88e853f80 100755 --- a/tools/vtest.sh +++ b/tools/vtest.sh @@ -217,6 +217,7 @@ do i=$((i + 1)) (cd "${SRCDIR}" && git reset --hard > /dev/null 2>&1 || true) + (cd "${SRCDIR}" && git clean -df > /dev/null 2>&1 || true) (cd "${SRCDIR}" && git pull > /dev/null 2>&1 || true) rev=`cd "${SRCDIR}" && git show -s --pretty=format:%H` if [ "${waitnext}" -gt 0 -a "x${rev}" = "x${orev}" ] ; then @@ -235,7 +236,7 @@ do rm -rf "${REPORTDIR}" mkdir "${REPORTDIR}" - echo "VTEST 1.04" > ${VTEST_REPORT} + echo "VTEST 1.05" > ${VTEST_REPORT} echo "DATE `date +%s`" >> ${VTEST_REPORT} echo "BRANCH trunk" >> ${VTEST_REPORT} echo "HOST `hostname`" >> ${VTEST_REPORT} From dridi.boukelmoune at gmail.com Wed Jul 29 10:42:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 29 Jul 2020 10:42:07 +0000 (UTC) Subject: [master] acfc992fa Complete the VSL memory fence pair Message-ID: <20200729104207.4D6D2627CB@lists.varnish-cache.org> commit acfc992fad3ada1195431f7bbb17c892050c2070 Author: Martin Blix Grydeland Date: Mon Jul 27 13:32:44 2020 +0200 Complete the VSL memory fence pair The log readers would not complete the acquire/release pair of memory fence operations when waiting for new data to appear in the shared memory log. With this, a VRMB() is issued after having waited for the VSL_ENDMARKER disappears, and before using any data after it (including the deciphered length data). diff --git a/lib/libvarnishapi/vsl_cursor.c b/lib/libvarnishapi/vsl_cursor.c index aa77fdc1c..cf5349ade 100644 --- a/lib/libvarnishapi/vsl_cursor.c +++ b/lib/libvarnishapi/vsl_cursor.c @@ -138,15 +138,6 @@ vslc_vsm_next(const struct VSL_cursor *cursor) t = *(volatile const uint32_t *)c->next.ptr; AN(t); - if (t == VSL_WRAPMARKER) { - /* Wrap around not possible at front */ - assert(c->next.ptr != c->head->log); - c->next.ptr = c->head->log; - while (c->next.priv % VSL_SEGMENTS) - c->next.priv++; - continue; - } - if (t == VSL_ENDMARKER) { if (VSM_StillValid(c->vsm, &c->vf) != VSM_valid) return (vsl_e_abandon); @@ -156,6 +147,19 @@ vslc_vsm_next(const struct VSL_cursor *cursor) return (vsl_end); } + /* New data observed. Ensure load ordering with the log + * writer. */ + VRMB(); + + if (t == VSL_WRAPMARKER) { + /* Wrap around not possible at front */ + assert(c->next.ptr != c->head->log); + c->next.ptr = c->head->log; + while (c->next.priv % VSL_SEGMENTS) + c->next.priv++; + continue; + } + c->cursor.rec = c->next; c->next.ptr = VSL_NEXT(c->next.ptr); From phk at FreeBSD.org Wed Jul 29 11:03:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 29 Jul 2020 11:03:06 +0000 (UTC) Subject: [master] 56a5b3bdf Add note about version bumps. Message-ID: <20200729110306.9010363241@lists.varnish-cache.org> commit 56a5b3bdfc226339c899e2917726dee41fb79fad Author: Poul-Henning Kamp Date: Wed Jul 29 11:02:15 2020 +0000 Add note about version bumps. diff --git a/tools/vtest.sh b/tools/vtest.sh index 88e853f80..de8cd8649 100755 --- a/tools/vtest.sh +++ b/tools/vtest.sh @@ -236,6 +236,9 @@ do rm -rf "${REPORTDIR}" mkdir "${REPORTDIR}" + # NB: Only change the report version number when the format/content + # NB: of the report changes. Corresponding changes on the backend + # NB: will be required. Coordinate with phk at . echo "VTEST 1.05" > ${VTEST_REPORT} echo "DATE `date +%s`" >> ${VTEST_REPORT} echo "BRANCH trunk" >> ${VTEST_REPORT} From martin at varnish-software.com Wed Jul 29 13:55:06 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Wed, 29 Jul 2020 13:55:06 +0000 (UTC) Subject: [master] 8d0000955 Attempt to silence coverity warning Message-ID: <20200729135506.B5155105458@lists.varnish-cache.org> commit 8d00009558559a764d79480143c27fda94f08dc6 Author: Martin Blix Grydeland Date: Wed Jul 29 15:52:19 2020 +0200 Attempt to silence coverity warning Coverity fails to see that with at least 2 OC refs on entry to EXP_Insert, and one deref if race, then at least one ref remains. Attempt to coerce acception. diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index c6bb36ef5..ac3594629 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -177,6 +177,7 @@ void EXP_Insert(struct worker *wrk, struct objcore *oc) { unsigned remove_race = 0; + struct objcore *tmpoc; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); @@ -186,6 +187,8 @@ EXP_Insert(struct worker *wrk, struct objcore *oc) if (!(oc->exp_flags & OC_EF_REFD)) return; + /* One ref held by the caller, and one that wil be owned by + * expiry. */ assert(oc->refcnt >= 2); ObjSendEvent(wrk, oc, OEV_INSERT); @@ -204,8 +207,10 @@ EXP_Insert(struct worker *wrk, struct objcore *oc) if (remove_race) { ObjSendEvent(wrk, oc, OEV_EXPIRE); - (void)HSH_DerefObjCore(wrk, &oc, 0); - AZ(oc); + tmpoc = oc; + (void)HSH_DerefObjCore(wrk, &tmpoc, 0); + AZ(tmpoc); + assert(oc->refcnt >= 1); /* Silence coverity */ } } From phk at FreeBSD.org Wed Jul 29 22:25:13 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 29 Jul 2020 22:25:13 +0000 (UTC) Subject: [master] 3fb7df711 Change the back-off algorithm to be less aggresive when builds fail Message-ID: <20200729222513.8279B111D72@lists.varnish-cache.org> commit 3fb7df7112cdb2e1f1b2a9784d1f41372e9d2663 Author: Poul-Henning Kamp Date: Wed Jul 29 22:24:26 2020 +0000 Change the back-off algorithm to be less aggresive when builds fail diff --git a/tools/vtest.sh b/tools/vtest.sh index de8cd8649..5ebb4e165 100755 --- a/tools/vtest.sh +++ b/tools/vtest.sh @@ -44,9 +44,9 @@ export TMPDIR="${TMPDIR:-`pwd`/_vtest_tmp}" MESSAGE="${MESSAGE:-}" WAITPERIOD=60 # unit: Seconds +WAITMIN=1 # unit: WAITPERIOD +WAITMAX=60 # unit: WAITPERIOD -WAITGOOD=60 # unit: WAITPERIOD -WAITBAD=1 # unit: WAITPERIOD MAXRUNS="${MAXRUNS:-0}" ####################################################################### @@ -209,7 +209,8 @@ if $enable_gcov ; then fi orev=000 -waitnext=${WAITBAD} +waitnext=0 +waitcur=${WAITMIN} i=0 while [ $MAXRUNS -eq 0 ] || [ $i -lt $MAXRUNS ] @@ -219,13 +220,23 @@ do (cd "${SRCDIR}" && git reset --hard > /dev/null 2>&1 || true) (cd "${SRCDIR}" && git clean -df > /dev/null 2>&1 || true) (cd "${SRCDIR}" && git pull > /dev/null 2>&1 || true) + (cd "${SRCDIR}" && chmod -R +w varnish-trunk && rm -rf varnish-trunk > /dev/null 2>&1 || true) rev=`cd "${SRCDIR}" && git show -s --pretty=format:%H` - if [ "${waitnext}" -gt 0 -a "x${rev}" = "x${orev}" ] ; then + if [ "x${rev}" != "x${orev}" ] ; then + waitcur=${WAITMIN} + elif [ "${waitnext}" -gt 0 ] ; then sleep ${WAITPERIOD} waitnext=`expr ${waitnext} - 1 || true` continue + else + waitcur=`expr ${waitcur} + 1` + if [ ${waitcur} -gt ${WAITMAX} ] ; then + waitcur=${WAITMAX} + fi fi - waitnext=${WAITBAD} + + waitnext=${waitcur} + orev=${rev} if ! [ -d "${SRCDIR}" ] && ! mkdir -p "${SRCDIR}" ; then @@ -265,7 +276,8 @@ do mv ${SRCDIR}/_gcov ${REPORTDIR}/ echo "MAKEGCOV GOOD" >> ${VTEST_REPORT} echo "MANIFEST _gcov" >> ${VTEST_REPORT} - waitnext=${WAITGOOD} + waitcur=${WAITMAX} + waitnext=${WAITMAX} else echo "MAKEGCOV BAD" >> ${VTEST_REPORT} echo "MANIFEST _makegcov" >> ${VTEST_REPORT} @@ -278,7 +290,8 @@ do failedtests >> ${VTEST_REPORT} else echo "MAKEDISTCHECK GOOD" >> ${VTEST_REPORT} - waitnext=${WAITGOOD} + waitnext=${WAITMAX} + waitcur=${WAITMAX} fi fi echo "VTEST END" >> ${VTEST_REPORT} From dridi.boukelmoune at gmail.com Thu Jul 30 09:41:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 30 Jul 2020 09:41:06 +0000 (UTC) Subject: [master] 72370791f build: Ignore directory stamps Message-ID: <20200730094106.F24805E29@lists.varnish-cache.org> commit 72370791ffe418e16c5836c0d8765b4ed87c332a Author: Dridi Boukelmoune Date: Thu Jul 30 11:34:18 2020 +0200 build: Ignore directory stamps I believe we get those when we have "subdir-objects", which we do now. diff --git a/.gitignore b/.gitignore index e261781ba..d98e8064e 100644 --- a/.gitignore +++ b/.gitignore @@ -43,6 +43,7 @@ _* TAGS tags cscope.*out +.dirstamp # Default vcl made from bin/varnishd/default.vcl /bin/varnishd/builtin_vcl.c