Page MenuHomePhabricator
Paste P8441

cdanis@prometheus1004.eqiad.wmnet ~ % sudo journalctl --unit prometheus@ops.service --since '2019-04-25 18:41:48' --until '2019-04-25 20:42:03' | phaste
ActivePublic

Authored by ProdPasteBot on Apr 26 2019, 12:05 AM.
-- Logs begin at Wed 2019-04-17 09:18:59 UTC, end at Fri 2019-04-26 00:05:54 UTC. --
Apr 25 19:00:29 prometheus1004 prometheus@ops[27971]: level=info ts=2019-04-25T19:00:29.394681819Z caller=compact.go:443 component=tsdb msg="write block" mint=1556208000000 maxt=1556215200000 ulid=01D9AXWSX1N1ZPFS0HQ05J1KEP
Apr 25 19:00:34 prometheus1004 prometheus@ops[27971]: level=info ts=2019-04-25T19:00:34.951390768Z caller=head.go:526 component=tsdb msg="head GC completed" duration=3.066242694s
Apr 25 19:00:54 prometheus1004 prometheus@ops[27971]: level=info ts=2019-04-25T19:00:54.845632003Z caller=head.go:573 component=tsdb msg="WAL checkpoint complete" first=12319 last=12349 duration=19.89416885s
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: fatal error: runtime: out of memory
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime stack:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.throw(0x1dd50f9, 0x16)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/panic.go:608 +0x72
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.sysMap(0xd560000000, 0x4000000, 0x31c4918)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mem_linux.go:156 +0xc7
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).sysAlloc(0x31aafe0, 0x4000000, 0x31ab068, 0x7f1de6e5a560)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/malloc.go:619 +0x1c7
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).grow(0x31aafe0, 0x3e, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mheap.go:920 +0x42
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).allocSpanLocked(0x31aafe0, 0x3e, 0x31c4928, 0x6000)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mheap.go:848 +0x337
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).alloc_m(0x31aafe0, 0x3e, 0x100, 0x7f9738fa6fff)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mheap.go:692 +0x119
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).alloc.func1()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mheap.go:759 +0x4c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.(*mheap).alloc(0x31aafe0, 0x3e, 0x7f9738010100, 0x7f1e6c8f0c88)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/mheap.go:758 +0x8a
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.largeAlloc(0x7bee0, 0x7f1e237f0001, 0x7f1e6c8f0c88)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/malloc.go:1019 +0x97
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.mallocgc.func1()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/malloc.go:914 +0x46
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.systemstack(0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/asm_amd64.s:351 +0x66
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.mstart()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/proc.go:1229
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 79522534 [running]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.systemstack_switch()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/asm_amd64.s:311 fp=0xc0b39440d8 sp=0xc0b39440d0 pc=0x45bdd0
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.mallocgc(0x7bee0, 0x1be3920, 0x1, 0x58)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/malloc.go:913 +0x896 fp=0xc0b3944178 sp=0xc0b39440d8 pc=0x40df16
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.newarray(0x1be3920, 0x446, 0x7f9ca6b3ab58)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/malloc.go:1048 +0x6a fp=0xc0b39441a8 sp=0xc0b3944178 pc=0x40e2ea
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.makeBucketArray(0x1a70de0, 0xc0b394420a, 0x0, 0x60, 0x1d05400)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/map.go:355 +0x184 fp=0xc0b39441e0 sp=0xc0b39441a8 pc=0x40f1b4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.makemap(0x1a70de0, 0x1303, 0xc0b39445b0, 0xd55ffc0000)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/map.go:321 +0xec fp=0xc0b3944228 sp=0xc0b39441e0 pc=0x40ef1c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*evaluator).rangeEval(0xd52e1bb450, 0xd53de8bc80, 0xc0b3944da0, 0x2, 0x2, 0x0, 0xc1d85357c0, 0x2029320)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:793 +0x3e3 fp=0xc0b39448e0 sp=0xc0b3944228 pc=0x1656693
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*evaluator).eval(0xd52e1bb450, 0x201a1a0, 0xc1d4ec8380, 0xc0b3944f30, 0xf8)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:1081 +0x2128 fp=0xc0b3944ed0 sp=0xc0b39448e0 pc=0x1659f18
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*evaluator).Eval(0xd52e1bb450, 0x201a1a0, 0xc1d4ec8380, 0x0, 0x0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:699 +0x80 fp=0xc0b3944f08 sp=0xc0b3944ed0 pc=0x1655f40
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc00044b230, 0x20286a0, 0xd540b189c0, 0xd5333153e0, 0xd529e77b30, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:436 +0x441 fp=0xc0b39451e8 sp=0xc0b3944f08 pc=0x1653f51
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*Engine).exec(0xc00044b230, 0x20286a0, 0xd540b189c0, 0xd5333153e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:390 +0x59f fp=0xc0b39452a8 sp=0xc0b39451e8 pc=0x165388f
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql.(*query).Exec(0xd5333153e0, 0x20285e0, 0xc1d4ec8300, 0xd546b50550)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/promql/engine.go:178 +0x94 fp=0xc0b3945370 sp=0xc0b39452a8 pc=0x1651cc4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1.(*API).query(0xc00018c000, 0xd53255d200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1/api.go:292 +0x33f fp=0xc0b3945580 sp=0xc0b3945370 pc=0x17e3a2f
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1.(*API).query-fm(0xd53255d200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1/api.go:224 +0x70 fp=0xc0b3945610 sp=0xc0b3945580 pc=0x17f0c80
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x201a4e0, 0xd550dc0160, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/api/v1/api.go:205 +0x8e fp=0xc0b39456a8 sp=0xc0b3945610 pc=0x17f013e
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc00016e6a0, 0x201a4e0, 0xd550dc0160, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b39456d0 sp=0xc0b39456a8 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x2009f40, 0xc00016e6a0, 0x7f9ca69ab220, 0xd529e77ae0, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7c fp=0xc0b3945708 sp=0xc0b39456d0 pc=0x17e205c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP-fm(0x7f9ca69ab220, 0xd529e77ae0, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/web.go:294 +0x57 fp=0xc0b3945740 sp=0xc0b3945708 pc=0x17f0b57
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web.(*Handler).testReady.func1(0x7f9ca69ab220, 0xd529e77ae0, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/web.go:386 +0x55 fp=0xc0b39457a0 sp=0xc0b3945740 pc=0x180bfb5
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc00016e6e0, 0x7f9ca69ab220, 0xd529e77ae0, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b39457c8 sp=0xc0b39457a0 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(0x20198e0, 0xd550dc0120, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:196 +0xe9 fp=0xc0b3945848 sp=0xc0b39457c8 pc=0x17cc839
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc0003d9080, 0x20198e0, 0xd550dc0120, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b3945870 sp=0xc0b3945848 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(0x20198e0, 0xd550dc0120, 0xd53255d200)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:76 +0xb5 fp=0xc0b39458f8 sp=0xc0b3945870 pc=0x17cc3e5
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/common/route.(*Router).handle.func1(0x20198e0, 0xd550dc0120, 0xd53255d100, 0x0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/common/route/route.go:60 +0x2b6 fp=0xc0b39459d0 sp=0xc0b39458f8 pc=0x17e0806
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0003d8c90, 0x20198e0, 0xd550dc0120, 0xd53255d100)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/julienschmidt/httprouter/router.go:299 +0x6c2 fp=0xc0b3945ab0 sp=0xc0b39459d0 pc=0x17dc7f2
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/common/route.(*Router).ServeHTTP(0xc00016e520, 0x20198e0, 0xd550dc0120, 0xd53255d100)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/common/route/route.go:98 +0x4c fp=0xc0b3945ae0 sp=0xc0b3945ab0 pc=0x17e041c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.StripPrefix.func1(0x20198e0, 0xd550dc0120, 0xd53255d000)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:2003 +0x18b fp=0xc0b3945b50 sp=0xc0b3945ae0 pc=0x6d1dbb
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc0005d0240, 0x20198e0, 0xd550dc0120, 0xd53255d000)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b3945b78 sp=0xc0b3945b50 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.(*ServeMux).ServeHTTP(0xc0003d8c30, 0x20198e0, 0xd550dc0120, 0xd53255d000)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:2361 +0x127 fp=0xc0b3945bb8 sp=0xc0b3945b78 pc=0x6b3ba7
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2(0x20269a0, 0xd5325340e0, 0xd53255cf00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/opentracing-contrib/go-stdlib/nethttp/server.go:90 +0x451 fp=0xc0b3945cb0 sp=0xc0b3945bb8 pc=0x17c61e1
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc0005d02d0, 0x20269a0, 0xd5325340e0, 0xd53255cf00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b3945cd8 sp=0xc0b3945cb0 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web.withStackTracer.func1(0x20269a0, 0xd5325340e0, 0xd53255cf00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/web.go:86 +0x9d fp=0xc0b3945d20 sp=0xc0b3945cd8 pc=0x180b7dd
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.HandlerFunc.ServeHTTP(0xc0005d0300, 0x20269a0, 0xd5325340e0, 0xd53255cf00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1964 +0x44 fp=0xc0b3945d48 sp=0xc0b3945d20 pc=0x6b1df4
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.serverHandler.ServeHTTP(0xc0000c9520, 0x20269a0, 0xd5325340e0, 0xd53255cf00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:2741 +0xab fp=0xc0b3945d78 sp=0xc0b3945d48 pc=0x6b4b4b
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.(*conn).serve(0xc1d92e5040, 0x20285e0, 0xc17681d580)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:1847 +0x646 fp=0xc0b3945fc8 sp=0xc0b3945d78 pc=0x6b0e56
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: runtime.goexit()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0b3945fd0 sp=0xc0b3945fc8 pc=0x45deb1
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by net/http.(*Server).Serve
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:2851 +0x2f5
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 1 [chan receive, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run(0xc000617df8, 0xc0004e45e0, 0x8)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:43 +0xf0
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:690 +0x656c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 34 [syscall, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: os/signal.signal_recv(0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/runtime/sigqueue.go:139 +0x9c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: os/signal.loop()
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/os/signal/signal_unix.go:23 +0x22
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by os/signal.init.0
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/os/signal/signal_unix.go:29 +0x41
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 146 [select]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery.(*Manager).sender(0xc0000e7ea0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:256 +0x110
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/discovery.(*Manager).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:170 +0x43
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 113 [select, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func5(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:465 +0xe5
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0004da100, 0xc0004e4570)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 130 [chan receive, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery.(*Manager).Run(0xc0000e7e00, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:171 +0x72
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func7(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:487 +0x40
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000e2040, 0xc0000e2080)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 131 [chan receive, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery.(*Manager).Run(0xc0000e7ea0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:171 +0x72
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func9(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:501 +0x40
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000e20c0, 0xc0000e20e0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 132 [select, 2 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/scrape.(*Manager).Run(0xc0001f33b0, 0xc0000e43c0, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/scrape/manager.go:69 +0xdc
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func11(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:521 +0x7d
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc00044ad80, 0xc0000e2220)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 133 [select, 11987 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func13(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:546 +0x16a
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000a02a0, 0xc0004e45b0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 134 [chan receive, 11987 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func15(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:593 +0x351
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000a0300, 0xc0004e45c0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 135 [chan receive, 11987 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func17(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:609 +0x71
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000e22e0, 0xc0000e2300)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 136 [chan receive, 11987 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func19(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:643 +0x41b
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0004da1c0, 0xc00044b9b0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 137 [select, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web.(*Handler).Run(0xc0005b45a0, 0x20285e0, 0xc0000d9a40, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/web.go:485 +0xdcb
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func21(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:658 +0x40
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc0000e2320, 0xc0004e45d0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 138 [select]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/notifier.(*Manager).Run(0xc0000c4a00, 0xc0000e4480)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/notifier/notifier.go:311 +0xfd
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: main.main.func23(0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/cmd/prometheus/main.go:681 +0x7d
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0000a0360, 0xc00044b9e0, 0xc0004e45e0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 162 [select]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery.(*Manager).sender(0xc0000e7e00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:256 +0x110
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/discovery.(*Manager).Run
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/discovery/manager.go:170 +0x43
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 178 [select, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc0003d8c00)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x142
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by google.golang.org/grpc.(*ccResolverWrapper).start
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x3f
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 179 [chan receive, 11988 minutes]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/prompb.RegisterAdminHandlerFromEndpoint.func1.1(0x20285e0, 0xc0000d9a40, 0xc000204000, 0x7ffc33fffe4e, 0xe)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/prompb/rpc.pb.gw.go:89 +0x4c
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by github.com/prometheus/prometheus/prompb.RegisterAdminHandlerFromEndpoint.func1
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/prompb/rpc.pb.gw.go:88 +0x194
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 194 [select]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0000d8180)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x10f
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by google.golang.org/grpc.newCCBalancerWrapper
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x13a
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 195 [select]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc.(*addrConn).transportMonitor(0xc000105680)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/clientconn.go:1242 +0x1c1
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc.(*addrConn).connect.func1(0xc000105680)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/clientconn.go:853 +0x1b2
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: created by google.golang.org/grpc.(*addrConn).connect
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: google.golang.org/grpc/clientconn.go:844 +0xe1
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: goroutine 180 [chan receive]:
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/cockroachdb/cmux.muxListener.Accept(0x20198a0, 0xc00016e0a0, 0xc000516120, 0xc0005e5e78, 0x48fb66, 0x5cc21b6c, 0x4377ce)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/cockroachdb/cmux/cmux.go:184 +0x41
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: net/http.(*Server).Serve(0xc0000c9520, 0x2026ba0, 0xc00016e240, 0x0, 0x0)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: /usr/lib/go-1.11/src/net/http/server.go:2826 +0x22f
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web.(*Handler).Run.func3(0xc0001de2a0, 0xc0000c9520, 0x2026ba0, 0xc00016e240)
Apr 25 20:41:49 prometheus1004 prometheus@ops[27971]: github.com/prometheus/prometheus/web/web.go:476 +0x3f
Apr 25 20:42:02 prometheus1004 systemd[1]: prometheus@ops.service: Main process exited, code=killed, status=9/KILL
Apr 25 20:42:02 prometheus1004 systemd[1]: prometheus@ops.service: Unit entered failed state.
Apr 25 20:42:02 prometheus1004 systemd[1]: prometheus@ops.service: Failed with result 'signal'.

Event Timeline

ProdPasteBot edited the content of this paste. (Show Details)Apr 26 2019, 12:05 AM
ProdPasteBot changed the title of this paste from untitled to Masterwork From Distant Lands.
CDanis changed the title of this paste from Masterwork From Distant Lands to cdanis@prometheus1004.eqiad.wmnet ~ % sudo journalctl --unit prometheus@ops.service --since '2019-04-25 18:41:48' --until '2019-04-25 20:42:03' | phaste.Apr 26 2019, 12:08 AM