Page MenuHomePhabricator
Paste P8441

cdanis@prometheus1004.eqiad.wmnet ~ % sudo journalctl --unit prometheus@ops.service --since '2019-04-25 18:41:48' --until '2019-04-25 20:42:03' | phaste
ActivePublic

Authored by ProdPasteBot on Apr 26 2019, 12:05 AM.
Tags
None
Referenced Files
F28777207: raw.txt
Apr 26 2019, 12:05 AM
Subscribers
None
-- Logs begin at Wed 2019-04-17 09:18:59 UTC, end at Fri 2019-04-26 00:05:54 UTC. --
Apr 25 19:00:29 prometheus1004 prometheus@ops[27971]: level=info ts=2019-04-25T19:00:29.394681819Z caller=compact.go:443 component=tsdb msg="write block" mint=1556208000000 maxt=1556215200000 ulid=01D9AXWSX1N1ZPFS0HQ05J1KEP
Apr 25 19:00:34 prometheus1004 prometheus@ops[27971]: level=info ts=2019-04-25T19:00:34.951390768Z caller=head.go:526 component=tsdb msg="head GC completed" duration=3.066242694s
Apr 25 19:00:54 prometheus1004 prometheus@ops[27971]: level=info ts=2019-04-25T19:00:54.845632003Z caller=head.go:573 component=tsdb msg="WAL checkpoint complete" first=12319 last=12349 duration=19.89416885s
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: fatal error: runtime: out of memory
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime stack:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.throw(0x1dd50f9, 0x16)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/panic.go:608 +0x72
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.sysMap(0xd560000000, 0x4000000, 0x31c4918)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mem_linux.go:156 +0xc7
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).sysAlloc(0x31aafe0, 0x4000000, 0x31ab068, 0x7f1de6e5a560)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/malloc.go:619 +0x1c7
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).grow(0x31aafe0, 0x3e, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mheap.go:920 +0x42
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).allocSpanLocked(0x31aafe0, 0x3e, 0x31c4928, 0x6000)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mheap.go:848 +0x337
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).alloc_m(0x31aafe0, 0x3e, 0x100, 0x7f9738fa6fff)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mheap.go:692 +0x119
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).alloc.func1()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mheap.go:759 +0x4c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).alloc(0x31aafe0, 0x3e, 0x7f9738010100, 0x7f1e6c8f0c88)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mheap.go:758 +0x8a
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.largeAlloc(0x7bee0, 0x7f1e237f0001, 0x7f1e6c8f0c88)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/malloc.go:1019 +0x97
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.mallocgc.func1()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/malloc.go:914 +0x46
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.systemstack(0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/asm_amd64.s:351 +0x66
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.mstart()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/proc.go:1229
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 79522534 [running]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.systemstack_switch()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/asm_amd64.s:311 fp=0xc0b39440d8 sp=0xc0b39440d0 pc=0x45bdd0
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.mallocgc(0x7bee0, 0x1be3920, 0x1, 0x58)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/malloc.go:913 +0x896 fp=0xc0b3944178 sp=0xc0b39440d8 pc=0x40df16
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.newarray(0x1be3920, 0x446, 0x7f9ca6b3ab58)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/malloc.go:1048 +0x6a fp=0xc0b39441a8 sp=0xc0b3944178 pc=0x40e2ea
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.makeBucketArray(0x1a70de0, 0xc0b394420a, 0x0, 0x60, 0x1d05400)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/map.go:355 +0x184 fp=0xc0b39441e0 sp=0xc0b39441a8 pc=0x40f1b4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.makemap(0x1a70de0, 0x1303, 0xc0b39445b0, 0xd55ffc0000)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/map.go:321 +0xec fp=0xc0b3944228 sp=0xc0b39441e0 pc=0x40ef1c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*evaluator).rangeEval(0xd52e1bb450, 0xd53de8bc80, 0xc0b3944da0, 0x2, 0x2, 0x0, 0xc1d85357c0, 0x2029320)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:793 +0x3e3 fp=0xc0b39448e0 sp=0xc0b3944228 pc=0x1656693
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*evaluator).eval(0xd52e1bb450, 0x201a1a0, 0xc1d4ec8380, 0xc0b3944f30, 0xf8)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:1081 +0x2128 fp=0xc0b3944ed0 sp=0xc0b39448e0 pc=0x1659f18
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*evaluator).Eval(0xd52e1bb450, 0x201a1a0, 0xc1d4ec8380, 0x0, 0x0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:699 +0x80 fp=0xc0b3944f08 sp=0xc0b3944ed0 pc=0x1655f40
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc00044b230, 0x20286a0, 0xd540b189c0, 0xd5333153e0, 0xd529e77b30, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:436 +0x441 fp=0xc0b39451e8 sp=0xc0b3944f08 pc=0x1653f51
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*Engine).exec(0xc00044b230, 0x20286a0, 0xd540b189c0, 0xd5333153e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:390 +0x59f fp=0xc0b39452a8 sp=0xc0b39451e8 pc=0x165388f
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*query).Exec(0xd5333153e0, 0x20285e0, 0xc1d4ec8300, 0xd546b50550)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:178 +0x94 fp=0xc0b3945370 sp=0xc0b39452a8 pc=0x1651cc4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1.(*API).query(0xc00018c000, 0xd53255d200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1/api.go:292 +0x33f fp=0xc0b3945580 sp=0xc0b3945370 pc=0x17e3a2f
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1.(*API).query-fm(0xd53255d200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1/api.go:224 +0x70 fp=0xc0b3945610 sp=0xc0b3945580 pc=0x17f0c80
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x201a4e0, 0xd550dc0160, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1/api.go:205 +0x8e fp=0xc0b39456a8 sp=0xc0b3945610 pc=0x17f013e
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc00016e6a0, 0x201a4e0, 0xd550dc0160, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b39456d0 sp=0xc0b39456a8 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x2009f40, 0xc00016e6a0, 0x7f9ca69ab220, 0xd529e77ae0, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7c fp=0xc0b3945708 sp=0xc0b39456d0 pc=0x17e205c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP-fm(0x7f9ca69ab220, 0xd529e77ae0, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/web.go:294 +0x57 fp=0xc0b3945740 sp=0xc0b3945708 pc=0x17f0b57
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web.(*Handler).testReady.func1(0x7f9ca69ab220, 0xd529e77ae0, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/web.go:386 +0x55 fp=0xc0b39457a0 sp=0xc0b3945740 pc=0x180bfb5
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc00016e6e0, 0x7f9ca69ab220, 0xd529e77ae0, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b39457c8 sp=0xc0b39457a0 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(0x20198e0, 0xd550dc0120, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:196 +0xe9 fp=0xc0b3945848 sp=0xc0b39457c8 pc=0x17cc839
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc0003d9080, 0x20198e0, 0xd550dc0120, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b3945870 sp=0xc0b3945848 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(0x20198e0, 0xd550dc0120, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:76 +0xb5 fp=0xc0b39458f8 sp=0xc0b3945870 pc=0x17cc3e5
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/common/route.(*Router).handle.func1(0x20198e0, 0xd550dc0120, 0xd53255d100, 0x0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/common/route/route.go:60 +0x2b6 fp=0xc0b39459d0 sp=0xc0b39458f8 pc=0x17e0806
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0003d8c90, 0x20198e0, 0xd550dc0120, 0xd53255d100)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/julienschmidt/httprouter/router.go:299 +0x6c2 fp=0xc0b3945ab0 sp=0xc0b39459d0 pc=0x17dc7f2
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/common/route.(*Router).ServeHTTP(0xc00016e520, 0x20198e0, 0xd550dc0120, 0xd53255d100)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/common/route/route.go:98 +0x4c fp=0xc0b3945ae0 sp=0xc0b3945ab0 pc=0x17e041c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.StripPrefix.func1(0x20198e0, 0xd550dc0120, 0xd53255d000)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:2003 +0x18b fp=0xc0b3945b50 sp=0xc0b3945ae0 pc=0x6d1dbb
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc0005d0240, 0x20198e0, 0xd550dc0120, 0xd53255d000)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b3945b78 sp=0xc0b3945b50 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.(*ServeMux).ServeHTTP(0xc0003d8c30, 0x20198e0, 0xd550dc0120, 0xd53255d000)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:2361 +0x127 fp=0xc0b3945bb8 sp=0xc0b3945b78 pc=0x6b3ba7
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2(0x20269a0, 0xd5325340e0, 0xd53255cf00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/opentracing-contrib/go-stdlib/nethttp/server.go:90 +0x451 fp=0xc0b3945cb0 sp=0xc0b3945bb8 pc=0x17c61e1
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc0005d02d0, 0x20269a0, 0xd5325340e0, 0xd53255cf00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b3945cd8 sp=0xc0b3945cb0 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web.withStackTracer.func1(0x20269a0, 0xd5325340e0, 0xd53255cf00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/web.go:86 +0x9d fp=0xc0b3945d20 sp=0xc0b3945cd8 pc=0x180b7dd
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc0005d0300, 0x20269a0, 0xd5325340e0, 0xd53255cf00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b3945d48 sp=0xc0b3945d20 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.serverHandler.ServeHTTP(0xc0000c9520, 0x20269a0, 0xd5325340e0, 0xd53255cf00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:2741 +0xab fp=0xc0b3945d78 sp=0xc0b3945d48 pc=0x6b4b4b
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.(*conn).serve(0xc1d92e5040, 0x20285e0, 0xc17681d580)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1847 +0x646 fp=0xc0b3945fc8 sp=0xc0b3945d78 pc=0x6b0e56
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.goexit()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0b3945fd0 sp=0xc0b3945fc8 pc=0x45deb1
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by net/http.(*Server).Serve
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:2851 +0x2f5
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 1 [chan receive, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run(0xc000617df8, 0xc0004e45e0, 0x8)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:43 +0xf0
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:690 +0x656c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 34 [syscall, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: os/signal.signal_recv(0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/sigqueue.go:139 +0x9c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: os/signal.loop()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/os/signal/signal_unix.go:23 +0x22
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by os/signal.init.0
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/os/signal/signal_unix.go:29 +0x41
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 146 [select]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery.(*Manager).sender(0xc0000e7ea0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:256 +0x110
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/discovery.(*Manager).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:170 +0x43
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 113 [select, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func5(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:465 +0xe5
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0004da100, 0xc0004e4570)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 130 [chan receive, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery.(*Manager).Run(0xc0000e7e00, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:171 +0x72
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func7(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:487 +0x40
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000e2040, 0xc0000e2080)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 131 [chan receive, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery.(*Manager).Run(0xc0000e7ea0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:171 +0x72
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func9(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:501 +0x40
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000e20c0, 0xc0000e20e0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 132 [select, 2 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/scrape.(*Manager).Run(0xc0001f33b0, 0xc0000e43c0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/scrape/manager.go:69 +0xdc
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func11(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:521 +0x7d
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc00044ad80, 0xc0000e2220)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 133 [select, 11987 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func13(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:546 +0x16a
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000a02a0, 0xc0004e45b0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 134 [chan receive, 11987 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func15(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:593 +0x351
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000a0300, 0xc0004e45c0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 135 [chan receive, 11987 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func17(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:609 +0x71
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000e22e0, 0xc0000e2300)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 136 [chan receive, 11987 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func19(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:643 +0x41b
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0004da1c0, 0xc00044b9b0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 137 [select, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web.(*Handler).Run(0xc0005b45a0, 0x20285e0, 0xc0000d9a40, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/web.go:485 +0xdcb
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func21(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:658 +0x40
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000e2320, 0xc0004e45d0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 138 [select]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/notifier.(*Manager).Run(0xc0000c4a00, 0xc0000e4480)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/notifier/notifier.go:311 +0xfd
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func23(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:681 +0x7d
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc00044b9e0, 0xc0004e45e0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 162 [select]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery.(*Manager).sender(0xc0000e7e00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:256 +0x110
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/discovery.(*Manager).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:170 +0x43
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 178 [select, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc0003d8c00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x142
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by google.golang.org/grpc.(*ccResolverWrapper).start
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x3f
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 179 [chan receive, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/prompb.RegisterAdminHandlerFromEndpoint.func1.1(0x20285e0, 0xc0000d9a40, 0xc000204000, 0x7ffc33fffe4e, 0xe)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/prompb/rpc.pb.gw.go:89 +0x4c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/prompb.RegisterAdminHandlerFromEndpoint.func1
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/prompb/rpc.pb.gw.go:88 +0x194
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 194 [select]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0000d8180)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x10f
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by google.golang.org/grpc.newCCBalancerWrapper
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x13a
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 195 [select]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc.(*addrConn).transportMonitor(0xc000105680)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/clientconn.go:1242 +0x1c1
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc.(*addrConn).connect.func1(0xc000105680)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/clientconn.go:853 +0x1b2
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by google.golang.org/grpc.(*addrConn).connect
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/clientconn.go:844 +0xe1
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 180 [chan receive]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/cockroachdb/cmux.muxListener.Accept(0x20198a0, 0xc00016e0a0, 0xc000516120, 0xc0005e5e78, 0x48fb66, 0x5cc21b6c, 0x4377ce)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/cockroachdb/cmux/cmux.go:184 +0x41
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.(*Server).Serve(0xc0000c9520, 0x2026ba0, 0xc00016e240, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:2826 +0x22f
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web.(*Handler).Run.func3(0xc0001de2a0, 0xc0000c9520, 0x2026ba0, 0xc00016e240)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/web.go:476 +0x3f
Apr 25 20:42:02 prometheus1004 systemd[1]: prometheus@ops.service: Main process exited, code=killed, status=9/KILL
Apr 25 20:42:02 prometheus1004 systemd[1]: prometheus@ops.service: Unit entered failed state.
Apr 25 20:42:02 prometheus1004 systemd[1]: prometheus@ops.service: Failed with result 'signal'.

Event Timeline

ProdPasteBot changed the title of this paste from untitled to Masterwork From Distant Lands.
CDanis changed the title of this paste from Masterwork From Distant Lands to cdanis@prometheus1004.eqiad.wmnet ~ % sudo journalctl --unit prometheus@ops.service --since '2019-04-25 18:41:48' --until '2019-04-25 20:42:03' | phaste.Apr 26 2019, 12:08 AM