varnish core dump
Mahesh Ollalwar
mahesho at outlook.com
Wed Sep 14 07:14:53 CEST 2016
Thank you Anand.
we have set workspace_backend=262144 and the issue is resolved.
Thanks,
Mahesh.
________________________________
From: anand at rediff-inc.com <anand at rediff-inc.com> on behalf of Anand Shah <anand at rediff-inc.com>
Sent: Monday, September 12, 2016 4:09 PM
To: Mahesh Ollalwar
Cc: Dag Haavi Finstad ; varnish-misc at varnish-cache.org
Subject: Re: varnish core dump
In order to reduce contention between all the threads, Varnish uses workspaces which allow each thread to use or modify memory independently of other threads, which avoids locking. Try increasing session workspace and shm_workspace. Disable any VMOD's that you use.
Regards,
Anand Shah
From: Mahesh Ollalwar <mahesho at outlook.com>
Sent: Mon, 12 Sep 2016 15:16:52
To: Dag Haavi Finstad <daghf at varnish-software.com>, "varnish-misc at varnish-cache.org" <varnish-misc at varnish-cache.org>
Subject: Re: varnish core dump
Hi,
Thanks for the response.
We have upgraded to latest stable version 4.1.3 but still see panics. below are the logs.
Sep 12 12:45:18 Prod-SHARED-CACHE6 abrt[32688]: Saved core dump of pid 24551 (/product/varnish/sbin/varnishd) to /var/spool/abrt/ccpp-2016-09-12-12:45:15-24551 (585650176 bytes)
Sep 12 12:45:18 Prod-SHARED-CACHE6 abrtd: Directory 'ccpp-2016-09-12-12:45:15-24551' creation detected
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Executable '/product/varnish/sbin/varnishd' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551' exited with 1
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551'
Sep 12 12:45:19 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) not responding to CLI, killing it.
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) died signal=6 (core dumped)
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) Last panic at: Mon, 12 Sep 2016 07:15:20 GMT#012"Assert error in http_EstimateWS(), cache/cache_http.c line 793:#012 Condition((fm->hd[u].b) != 0) not true.#012thread = (cache-worker)#012version = varnish-4.1.3 revision 5e3b6d2#012ident = Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x432f13: pan_ic+0x163#012 0x42dec5: http_EstimateWS+0x435#012 0x421874: vbf_beresp2obj+0x84#012 0x422cb5: vbf_stp_error+0x235#012 0x42366b: vbf_fetch_thread+0x1db#012 0x449825: WRK_Thread+0x4a5#012 0x449c5b: pool_thread+0x2b#012 0x3c25e079d1: libpthread.so<http://libpthread.so>.0() [0x3c25e079d1]#012 0x3c25ae8b6d: libc.so<http://libc.so>.6(clone+0x6d) [0x3c25ae8b6d]#012busyobj = 0x7f7f505208a0 {#012 ws = 0x7f7f50520960 {#012 OVERFLOWED id = \"Bo\",#012 {s,f,r,e} = {0x7f7f50522818,+57440,(nil),+57440},#012 },#012 refcnt = 2,#012 retries = 0, failed = 1, state = 1,#012 flags = {do_gzip, do_stream, is_gunzip},#012 director_req = 0x1a16130 {#012 vcl_name = farm_anon,#012 type = round-robin {#012 },#012 },#012 director_resp = 0x1a08e98 {#012 vcl_name = PROD_LB_BE_ANON_1,#012 type = backend {#012 display_name = boot.PROD_LB_BE_ANON_1,#012 ipv4 = 10.254.4.254<http://10.254.4.254>,#012 port = 8080,#012 hosthdr = 10.254.4.254<http://10.254.4.254>,#012 health=healthy, admin_health=probe, changed=1473664247.5,#012 n_conn = 2,#012 },#012 },#012 http[bereq] = 0x7f7f50520f28 {#012 ws[Bo] = 0x7f7f50520960,#012 hdrs {#012 \"GET\",#012 \"/AVS/besc?action=GetArrayContentList&channel=ANDROID&categoryId=7022\",#012 \"HTTP/1.1\",#012 \"host: www.example.com<http://www.example.com> \",#012 \"Accept-Encoding: gzip\",#012 \"Akamai-Origin-Hop: 4\",#012 \"True-Client-IP: 183.83.249.187<http://183.83.249.187>\",#012 \"User-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; Le X526 Build/IIXOSOP5801607291S)\",#012 \"Via: 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 aka
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (386) said Child starts
Thanks in advance,
Mahesh.
________________________________
From: Dag Haavi Finstad <daghf at varnish-software.com>
Sent: Monday, June 27, 2016 2:57 PM
To: Mahesh Ollalwar
Cc: varnish-misc at varnish-cache.org
Subject: Re: varnish core dump
Hi
I believe this might be #1539 [1], which is fixed in 4.0.3 and newer.
Please upgrade to the latest release and file a bug if you still run into this.
[1]: https://www.varnish-cache.org/trac/ticket/1539
On Fri, Jun 24, 2016 at 9:06 PM, Mahesh Ollalwar <mahesho at outlook.com> wrote:
> Can anyone help me on below error. looks like core dump is happening and
> child is getting restarted. This is happening recently without any change to
> the VCL and any system configuration.
>
> During this time the requests are getting terminated abruptly.
>
> Thanks in advance.
>
>
> Varnish version - varnish-4.0.2
>
>
> Jun 24 23:49:30 Prod-CACHE2 abrt[32341]: Saved core dump of pid 26526
> (/product/varnish/sbin/varnishd) to
> /var/spool/abrt/ccpp-2016-06-24-23:48:19-26526 (9034817536 bytes)
> Jun 24 23:49:30 Prod-CACHE2 abrtd: Directory
> 'ccpp-2016-06-24-23:48:19-26526' creation detected
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Executable
> '/product/varnish/sbin/varnishd' doesn't belong to any package and
> ProcessUnpackaged is set to 'no'
> Jun 24 23:49:31 Prod-CACHE2 abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' exited with 1
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526'
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) died signal=6
> (core dumped)
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) Panic
> message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line
> 430:#012 Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread =
> (cache-worker)#012ident =
> Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012
> 0x4310e5: pan_ic+0xc5#012 0x434c30: cnt_purge+0x0#012 0x436461:
> CNT_Request+0x241#012 0x42b73b: HTTP1_Session+0x53b#012 0x439f98:
> ses_req_pool_task+0x68#012 0x43a6d4: SES_pool_accept_task+0x2a4#012
> 0x433b87: Pool_Work_Thread+0x357#012 0x4467d9: wrk_thread_real+0xb9#012
> 0x3c25e079d1: /lib64/libpthread.so<http://libpthread.so>.0() [0x3c25e079d1]#012 0x3c25ae8b6d:
> /lib64/libc.so<http://libc.so>.6(clone+0x6d) [0x3c25ae8b6d]#012req = 0x7f83baaf2cb0 {#012
> sp = 0x7f84dd71c430, vxid = 1108394165, step = R_STP_LOOKUP,#012 req_body
> = R_BODY_NONE,#012 restarts = 0, esi_level = 0#012 sp = 0x7f84dd71c430
> {#012 fd = 64, vxid = 34652201,#012 client = 10.254.9.78<http://10.254.9.78> 61217,#012
> step = S_STP_WORKING,#012 },#012 worker = 0x7f8574c0cc40 {#012 ws =
> 0x7f8574c0ce58 {#012 id = "wrk",#012 {s,f,r,e} =
> {0x7f8574c0ac30,0x7f8574c0ac30,(nil),+8192},#012 },#012 VCL::method =
> 0x0,#012 VCL::return = deliver,#012 },#012 ws = 0x7f83baaf2e48 {#012
> id = "req",#012 {s,f,r,e} = {0x7f83baaf4ca0,+1320,(nil),+57360},#012
> },#012 http[req] = {#012 ws = 0x7f83baaf2e48[req]#012 "GET",#012
> "/search-engine-web/SearchContents?action=SearchContents&moreFilters=series%3A3197&appVersion=5.2.2&query=*&startIndex=0&searchOrder=counter_week%20desc&maxResult=8&channel=ANDROID&type=CLIPS",#012
> "HTTP/1.1",#012 "host: xxx.xxx.com<http://xxx.xxx.com>",#012 "Accept-Encoding:
> gzip",#012 "Akamai-Origin-Hop: 4",#012 "True-Client-IP:
> 49.15.204.238<http://49.15.204.238>",#012 "User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1;
> 1201 Build/LMY47I)",#012 "Via: 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost)
> (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1
> v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 akamai.net<http://akamai.net>(ghost)
> (AkamaiGHost)",#012 "X-Aka
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: child (1277) Started
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: Child (1277) said Child
> starts
>
>
> Thanks,
>
> Mahesh.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
--
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!
_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc<//prolinks.rediffmailpro.com/cgi-bin/prored.cgi?red=https%3A%2F%2Fwww%2Evarnish%2Dcache%2Eorg%2Flists%2Fmailman%2Flistinfo%2Fvarnish%2Dmisc&rediffng=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160914/1babd81d/attachment.html>
More information about the varnish-misc
mailing list