CodeChain crashes on SIGINT

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/codechain`
#469691130 2018-07-09 17:19:56  main INFO codechain  Starting client
#469691130 2018-07-09 17:19:57  main INFO codechain  Handshake Listening on 3485
#469691130 2018-07-09 17:19:57  main INFO discovery  Node runs with unstructured discovery
#469691130 2018-07-09 17:19:57  main INFO codechain  RPC Listening on 8080
#469691130 2018-07-09 17:19:57  IO Worker #1 INFO sync  Sync extension initialized
#469691130 2018-07-09 17:19:57  main INFO codechain  IPC Listening on /tmp/jsonrpc.ipc
#469691130 2018-07-09 17:19:57  main INFO test_script  Initialization complete
^Cpure virtual method called
terminate called without an active exception
Aborted (core dumped)

I bisected and it seems 9447945 caused this issue.

@sgkim126

@sgkim126 @majecty Any hint on this issue?

If we turn on the stratum, the crash does not occur anymore. However, if we turn off the stratum, the crash occurs again.

I've found a hint. If the RPC struct has weak pointers of dependencies, the crash disappears I will send a PR today.

☕️

Wait before shutdown to avoid the crash

I removed the Testnet milestone since there is the hotfix(#593). But I'll not close this issue until we find the reason and solve it.

FYI, the hotfix makes CodeChain wait a second before shutdown. You can use the environment variable named WAIT_BEFORE_SHUTDOWN to change the waiting time.

I found the reason. It's because the main thread exits without waiting for the child threads. The bug would be fixed if the main thread waits until the child threads join.

Closed by #1498

Bump the version to 1.3.0

Reoccurred on the master branch.

I'm suspecting that the bug is caused by rocksdb. It can be fixed if we upgrade rocksdb. Please try #1441 first.

rust-lldb gives a slightly different error, but the cause seems to be the same.
I've ran this command:

rust-lldb target/debug/codechain
(lldb) run
Process 9094 launched: '/home/builder/codechain/target/debug/codechain' (x86_64)
Process 9094 stopped and restarted: thread 1 received signal: SIGCHLD
#481054402 2019-10-07 20:14:05.524284403  main INFO client  Starting client
#481054402 2019-10-07 20:14:05.536024470  discovery.ext INFO discovery  Discovery starts with unstructured option
#481054402 2019-10-07 20:14:05.537053886  block-propagation.ext INFO sync  Sync extension initialized
#481054402 2019-10-07 20:14:05.538430996  main INFO rpc  RPC Listening on 127.0.0.1:8080
#481054402 2019-10-07 20:14:05.541738846  main INFO rpc  IPC Listening on /tmp/jsonrpc.ipc
#481054402 2019-10-07 20:14:05.545748357  main INFO ws  Listening for new connections on 127.0.0.1:8081.
#481054402 2019-10-07 20:14:05.546173063  main INFO rpc  WebSockets Listening on 127.0.0.1:8081
#481054402 2019-10-07 20:14:05.547757687  main INFO test_script  Initialization complete
Process 9094 stopped
* thread #1, name = 'codechain', stop reason = signal SIGSTOP
    frame #0: 0x00007ffff6cd69f3 libpthread.so.0`__pthread_cond_wait + 579
libpthread.so.0`__pthread_cond_wait:
->  0x7ffff6cd69f3 <+579>: cmpq   $-0x1000, %rax            ; imm = 0xF000
    0x7ffff6cd69f9 <+585>: movq   0x30(%rsp), %r8
    0x7ffff6cd69fe <+590>: ja     0x7ffff6cd6b30            ; <+896>
    0x7ffff6cd6a04 <+596>: movl   %r9d, %edi
(lldb) process handle SIGINT -p true                                                                                                                                              NAME         PASS   STOP   NOTIFY
===========  =====  =====  ======
SIGINT       true   true   true
(lldb) c
Process 9094 resuming

Then I've signalled the process with kill -INT 9094

Process 9094 stopped
* thread #1, name = 'codechain', stop reason = signal SIGINT
    frame #0: 0x00007ffff6cd69f3 libpthread.so.0`__pthread_cond_wait + 579
libpthread.so.0`__pthread_cond_wait:
->  0x7ffff6cd69f3 <+579>: cmpq   $-0x1000, %rax            ; imm = 0xF000
    0x7ffff6cd69f9 <+585>: movq   0x30(%rsp), %r8
    0x7ffff6cd69fe <+590>: ja     0x7ffff6cd6b30            ; <+896>
    0x7ffff6cd6a04 <+596>: movl   %r9d, %edi
(lldb) c
Process 9094 resuming
pure virtual method called
terminate called without an active exception

The main thread is dropped (which calls PosixEnv::~PosixEnv()) before the other threads are dropped (which calls rocksdb_close()). rocks_db is not the root cause, it was just a smoking gun indicating that we have had circular references or haven't joined the threads properly.

If you set breakpoints to b rocksdb::PosixEnv::~PosixEnv and b rocksdb_close, you can get the consistent result that the former one is called before the later one.

It is a bug by jsonrpc. jsonrpc version v9.0.0 have two methods for the server.

fn close(self);
fn wait(self);

It means either:

You can close the server's channel to prepare to terminate the server's thread, but can't wait the thread to be joined. (which is the same with Drop::drop)
or you can wait the thread to be joined, but can't close the channel.

Here's current timeline.

 * <- (1) Fire SIGINT
 * <- (2) Before exiting `run_node` scope, 
 | \      `drop::<jsonrpc::http::Server>()` is called.
 |   * <- (3) prepare to terminate `jsonrpc::http::Server`'s thread.
 * <-|--- (4) exit `run_node()`, `main()`, and `libc` cleanups static variables.
 |   |        `PosixEnv::~PosixEnv()` is called.
 X <-|--- (5) main thread is terminated.
     * <- (6) `rocksdb_close` is called in the server thread
     ! <- (7) Crash while calling `PosixEnv`'s method in rocksdb_close

What a nonsense.
close_handle() which enables close() && wait() behaviour is added after v12.0.0, but it makes cargo fail to resolve versions of packages after upgrading.

Proper exit behavior.

 * <- (1) SIGINT is fired
 * <- (2) `jsonrpc::http::Server::close()` is called
 |\
 * <\--(3) `jsonrpc::http::Server::wait()` is called to wait the termination of the server thread.
 |   * <- (4) prepare to terminate `jsonrpc::http::Server`'s thread.
 |   * <- (5) rocksdb_close is called in the thread
 |   X <- (6) the server thread is terminated
 | /
 * <- (7) exit `run_node` scope, `main()` scope, `libc` cleanups static var.
 |       `PosixEnv::~PosixEnv()` is called.
 X <- (8) main thread is terminated.

I've fixed one issue. Another issue that leads to a non-zero exit code left.
Here's the backtrace of all threads when the codechain crashes.
Thread #33 is in the middle of destructuring a thread that is created at jsonrpc_ipc_server::server::ServerBuilder.
It is clear that Parity didn't implement close & wait properly.

* thread #33, name = 'event.loop0', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00007fff5b3b5137 libsystem_pthread.dylib`pthread_mutex_lock
libsystem_pthread.dylib`pthread_mutex_lock:
->  0x7fff5b3b5137 <+0>:  cmpq   $0x4d55545a, (%rdi)       ; imm = 0x4D55545A
    0x7fff5b3b513e <+7>:  jne    0x7fff5b3b51af            ; <+120>
    0x7fff5b3b5140 <+9>:  movl   $0x1c0, %eax              ; imm = 0x1C0
    0x7fff5b3b5145 <+14>: andl   0xc(%rdi), %eax
Target 0: (codechain) stopped.

(lldb) bt all
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
Traceback (most recent call last):
  File "/Users/sclee/.rustup/toolchains/1.37.0-x86_64-apple-darwin/lib/rustlib/etc/lldb_rust_formatters.py", line 115, in print_val
    is_tuple_like = True)
  File "/Users/sclee/.rustup/toolchains/1.37.0-x86_64-apple-darwin/lib/rustlib/etc/lldb_rust_formatters.py", line 210, in print_struct_val
    body = separator.join([render_child(idx) for idx in range(field_start_index, len(fields))])
  File "/Users/sclee/.rustup/toolchains/1.37.0-x86_64-apple-darwin/lib/rustlib/etc/lldb_rust_formatters.py", line 203, in render_child
    return this + print_val(field_val.get_wrapped_value(), internal_dict)
TypeError: cannot concatenate 'str' and 'NoneType' objects
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
  thread #1, queue = 'com.apple.main-thread'
    frame #0: 0x00000001010b2dcf codechain`std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo> > >::__deallocate_node(this=&0x1026fb488, __np=0x0000000106489460) at __hash_table:1560:24
    frame #1: 0x00000001010b2d8f codechain`std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo> > >::~__hash_table(this=&0x1026fb488) at __hash_table:1516:5
    frame #2: 0x00000001010b2d55 codechain`std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo> > >::~__hash_table(this=&0x1026fb488) at __hash_table:1508:1
    frame #3: 0x00000001010b2d35 codechain`std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, rocksdb::OptionTypeInfo> > >::~unordered_map(this=&0x1026fb488) at __hash_table:1383:84
    frame #4: 0x00000001010ac9b5 codechain`std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, rocksdb::OptionTypeInfo> > >::~unordered_map(this=&0x1026fb488) at __hash_table:1383:84
    frame #5: 0x00007fff5b2693cf libsystem_c.dylib`__cxa_finalize_ranges + 319
    frame #6: 0x00007fff5b2696b3 libsystem_c.dylib`exit + 55
    frame #7: 0x00007fff5b1c33dc libdyld.dylib`start + 8
    frame #8: 0x00007fff5b1c33d5 libdyld.dylib`start + 1
  thread #5
    frame #0: 0x00007fff5b2f9bfe libsystem_kernel.dylib`__workq_kernreturn + 10
    frame #1: 0x00007fff5b3b6636 libsystem_pthread.dylib`_pthread_wqthread + 458
    frame #2: 0x00007fff5b3b63fd libsystem_pthread.dylib`start_wqthread + 13
  thread #6
    frame #0: 0x00007fff5b2f9bfe libsystem_kernel.dylib`__workq_kernreturn + 10
    frame #1: 0x00007fff5b3b6636 libsystem_pthread.dylib`_pthread_wqthread + 458
    frame #2: 0x00007fff5b3b63fd libsystem_pthread.dylib`start_wqthread + 13
  thread #32
    frame #0: 0x00007fff5b2fe78e libsystem_kernel.dylib`kevent + 10
    frame #1: 0x00000001020e2610 codechain`mio::sys::unix::kqueue::Selector::select::h7b42b85158304a2c(self=&0x106700cb0, evts=&0x70000e7de5c8, awakener=Token(18446744073709551615), timeout=<unavailable>) at kqueue.rs:84:26
    frame #2: 0x00000001020d6f69 codechain`mio::poll::Poll::poll2::hfcfbe223406c0a1b(self=&0x106700cb0, events=&0x70000e7de5c8, timeout=Option<core::time::Duration> {

}, interruptible=false) at poll.rs:1175:22
    frame #3: 0x00000001020d6cb9 codechain`mio::poll::Poll::poll1::h42f0d73ccca600ce(self=&0x106700cb0, events=&0x70000e7de5c8, timeout=Option<core::time::Duration> {

}, interruptible=false) at poll.rs:1137:18
    frame #4: 0x00000001020d6632 codechain`mio::poll::Poll::poll::h7ef088846d61d209(self=&0x106700cb0, events=&0x70000e7de5c8, timeout=<unavailable>) at poll.rs:1008:8
    frame #5: 0x0000000102026247 codechain`tokio_reactor::Reactor::poll::he988f4ee9e5da4fc(self=&0x70000e7de5c8, max_wait=Option<core::time::Duration> {

}) at lib.rs:328:14
    frame #6: 0x0000000102025f61 codechain`tokio_reactor::Reactor::turn::h0d76a7c95c7e25fc(self=&0x70000e7de5c8, max_wait=<unavailable>) at lib.rs:301:8
    frame #7: 0x0000000102032ccb codechain`tokio_reactor::background::run::hd1be9317f2c86c3b(reactor=Reactor {
events: Events {
inner: Events {
sys_events: KeventList(vec![]),
events: vec![],
event_map: HashMap<mio::token::Token, unsigned long, std::collections::hash::map::RandomState> {
base: HashMap<mio::token::Token, unsigned long, std::collections::hash::map::RandomState> {
hash_builder: RandomState {
k0: 7585609451430703067,
k1: 15808857290546738926
},
table: RawTable<(mio::token::Token, usize)> {
bucket_mask: 2047,
ctrl: NonNull<unsigned char> {
pointer: &0x1061f4000
},
data: NonNull<(mio::token::Token, usize)> {
pointer: &0x1061f4810
},
growth_left: 1792,
items: 0,
marker: PhantomData<(mio::token::Token, usize)> {

}
}
}
}
}
},
inner: Arc<tokio_reactor::Inner> {
ptr: NonNull<alloc::sync::ArcInner<tokio_reactor::Inner> > {
pointer: &0x106700ca0
},
phantom: PhantomData<tokio_reactor::Inner> {

}
},
_wakeup_registration: Registration {
inner: RegistrationInner {
node: &0x106700ac0
}
}
}, shared=Arc<tokio_reactor::background::Shared> {
ptr: NonNull<alloc::sync::ArcInner<tokio_reactor::background::Shared> > {
pointer: &0x106700ff0
},
phantom: PhantomData<tokio_reactor::background::Shared> {

}
}) at background.rs:205:8
    frame #8: 0x00000001020325a4 codechain`tokio_reactor::background::Background::new::_$u7b$$u7b$closure$u7d$$u7d$::h81f7ad685325c05a at background.rs:77:27
    frame #9: 0x000000010202f243 codechain`std::sys_common::backtrace::__rust_begin_short_backtrace::h184b46dcca4171fc(f=<unavailable>) at backtrace.rs:77:4
    frame #10: 0x0000000102033860 codechain`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h2955eb35e6c54d94 at mod.rs:470:16
    frame #11: 0x0000000102028833 codechain`_$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hfb703233ebaf7e22(self=<unavailable>, _args=<unavailable>) at panic.rs:315:8
    frame #12: 0x000000010202a1b3 codechain`std::panicking::try::do_call::hdc92d1e42481e564(data=&0x70000e7dea38) at panicking.rs:296:39
    frame #13: 0x000000010238a63f codechain`__rust_maybe_catch_panic at lib.rs:82:7 [opt]
    frame #14: 0x0000000102029ff5 codechain`std::panicking::try::h466173fe25efa93d(f=<unavailable>) at panicking.rs:275:12
    frame #15: 0x00000001020292d0 codechain`std::panic::catch_unwind::h21da07b5f43b351e(f=<unavailable>) at panic.rs:394:8
    frame #16: 0x0000000102033684 codechain`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::he73d734344c7a80d at mod.rs:469:29
    frame #17: 0x000000010202f415 codechain`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h85f04ec253993307((null)=&0x70000e7dee10, (null)=<unavailable>) at function.rs:231:4
    frame #18: 0x000000010237645e codechain`_$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::ha44ca87baccc45af at boxed.rs:746:8 [opt]
    frame #19: 0x0000000102389b6e codechain`std::sys::unix::thread::Thread::new::thread_start::hceb9fd3b79bf176c [inlined] _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hf7f4d50b605effc3 at boxed.rs:746:8 [opt]
    frame #20: 0x0000000102389b62 codechain`std::sys::unix::thread::Thread::new::thread_start::hceb9fd3b79bf176c [inlined] std::sys_common::thread::start_thread::he0046dfc66dff589 at thread.rs:13 [opt]
    frame #21: 0x0000000102389aee codechain`std::sys::unix::thread::Thread::new::thread_start::hceb9fd3b79bf176c at thread.rs:79 [opt]
    frame #22: 0x00007fff5b3b72eb libsystem_pthread.dylib`_pthread_body + 126
    frame #23: 0x00007fff5b3ba249 libsystem_pthread.dylib`_pthread_start + 66
    frame #24: 0x00007fff5b3b640d libsystem_pthread.dylib`thread_start + 13
* thread #33, name = 'event.loop0', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x00007fff5b3b5137 libsystem_pthread.dylib`pthread_mutex_lock
    frame #1: 0x00007fff583f0959 libc++.1.dylib`std::__1::mutex::lock() + 9
    frame #2: 0x00000001012d5213 codechain`std::__1::lock_guard<std::__1::mutex>::lock_guard(this=&0x70000e9ddec8, __m=0x40000000106552d2) at __mutex_base:104:27
    frame #3: 0x00000001012d188d codechain`std::__1::lock_guard<std::__1::mutex>::lock_guard(this=&0x70000e9ddec8, __m=0x40000000106552d2) at __mutex_base:104:21
    frame #4: 0x00000001012d1a72 codechain`rocksdb::ThreadPoolImpl::Impl::UnSchedule(this=&0x4000000010655282, arg=&0x10801f600) at threadpool_imp.cc:347:33
    frame #5: 0x00000001012d2c5c codechain`rocksdb::ThreadPoolImpl::UnSchedule(this=&0x106549a20, arg=&0x10801f600) at threadpool_imp.cc:436:17
    frame #6: 0x0000000100d0f4fc codechain`rocksdb::(anonymous namespace)::PosixEnv::UnSchedule(this=&0x1026f5ef8, arg=&0x10801f600, pri=BOTTOM) at env_posix.cc:891:29
    frame #7: 0x0000000100de171d codechain`rocksdb::DBImpl::~DBImpl(this=&0x10801f600) at db_impl.cc:260:13
    frame #8: 0x0000000100de3025 codechain`rocksdb::DBImpl::~DBImpl(this=&0x10801f600) at db_impl.cc:254:19
    frame #9: 0x0000000100de3049 codechain`rocksdb::DBImpl::~DBImpl(this=&0x10801f600) at db_impl.cc:254:19
    frame #10: 0x0000000100d3acae codechain`::rocksdb_close(db=&0x109816e30) at c.cc:605:3
    frame #11: 0x0000000100cef5a8 codechain`_$LT$rocksdb..rocksdb..DB$u20$as$u20$core..ops..drop..Drop$GT$::drop::h4e733176be42cc56(self=&0x1098177f8) at rocksdb.rs:837:12
    frame #12: 0x00000001000f65c5 codechain`core::ptr::real_drop_in_place::h701412558610da55((null)=&0x1098177f8) at mod.rs:197
    frame #13: 0x0000000100107f65 codechain`core::ptr::real_drop_in_place::he32ee23f5664dde7((null)=&0x1098177f8) at mod.rs:197
    frame #14: 0x00000001000eb6fa codechain`core::ptr::real_drop_in_place::h2be01d91e3961ffe((null)=&0x1098177f8) at mod.rs:197
    frame #15: 0x00000001000eb315 codechain`core::ptr::real_drop_in_place::h2aa79e8e338cd5f4((null)=&0x1098177f8) at mod.rs:197
    frame #16: 0x00000001000f7929 codechain`core::ptr::real_drop_in_place::h7788b22562a89dc6((null)=&0x1098177f0) at mod.rs:197
    frame #17: 0x00000001000fdfd3 codechain`core::ptr::real_drop_in_place::ha212f8d9b6aee641((null)=&0x1098177f0) at mod.rs:197
    frame #18: 0x00000001016513ca codechain`core::ptr::real_drop_in_place::he1d42f4056504023((null)=&mut KeyValueDB {

}) at mod.rs:197
    frame #19: 0x0000000101653f69 codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::hdf29c6a85b775623 [inlined] core::ptr::drop_in_place::h3c81f54f187a570a(to_drop=*mut KeyValueDB {

}) at mod.rs:187:4
    frame #20: 0x0000000101653f5c codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::hdf29c6a85b775623(self=&0x109817ad8) at sync.rs:549
    frame #21: 0x00000001016540e4 codechain`_$LT$alloc..sync..Arc$LT$T$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h618f96e4bf301242(self=&0x109817ad8) at sync.rs:1007:12
    frame #22: 0x00000001008e8745 codechain`core::ptr::real_drop_in_place::h4101373dfa45dc3a((null)=&0x109817ad8) at mod.rs:197
    frame #23: 0x00000001008e2c9b codechain`core::ptr::real_drop_in_place::h196af5196268db2c((null)=&0x109817918) at mod.rs:197
    frame #24: 0x00000001008fa375 codechain`core::ptr::real_drop_in_place::hbf95cea8c5642009((null)=&0x109817918) at mod.rs:197
    frame #25: 0x00000001008fffb9 codechain`core::ptr::real_drop_in_place::he98cef48b5f72a7d((null)=&0x109817910) at mod.rs:197
    frame #26: 0x00000001007643be codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::hfc7a971766697868 [inlined] core::ptr::drop_in_place::h2dedc13b0ef38df7(to_drop=&0x109817910) at mod.rs:187:4
    frame #27: 0x00000001007643b5 codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::hfc7a971766697868(self=&0x10a001e10) at sync.rs:549
    frame #28: 0x0000000100766450 codechain`_$LT$alloc..sync..Arc$LT$T$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::ha4e4e7a376fa1d28(self=&0x10a001e10) at sync.rs:1007:12
    frame #29: 0x00000001008eff65 codechain`core::ptr::real_drop_in_place::h75c402defbd24a98((null)=&0x10a001e10) at mod.rs:197
    frame #30: 0x00000001008fa0e5 codechain`core::ptr::real_drop_in_place::hbf088b9d4fd9268c((null)=&0x10a001e10) at mod.rs:197
    frame #31: 0x000000010076353e codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::h49c1a10d677d9af6 [inlined] core::ptr::drop_in_place::hd44031467594ad3e(to_drop=&0x10a001e10) at mod.rs:187:4
    frame #32: 0x0000000100763535 codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::h49c1a10d677d9af6(self=&0x10681dfc0) at sync.rs:549
    frame #33: 0x0000000100765dc0 codechain`_$LT$alloc..sync..Arc$LT$T$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h4a0503c6f8c985db(self=&0x10681dfc0) at sync.rs:1007:12
    frame #34: 0x00000001008f4c85 codechain`core::ptr::real_drop_in_place::h9d4111e17483e699((null)=&0x10681dfc0) at mod.rs:197
    frame #35: 0x00000001008f4e83 codechain`core::ptr::real_drop_in_place::h9e3216adf2c79a66((null)=&0x10681de18) at mod.rs:197
    frame #36: 0x00000001008fc850 codechain`core::ptr::real_drop_in_place::hd0583d2c7011f0b2((null)=&0x10681d610) at mod.rs:197
    frame #37: 0x000000010076307e codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::h21392c0c1faadf84 [inlined] core::ptr::drop_in_place::h84aae20f1c1afdeb(to_drop=&0x10681d610) at mod.rs:187:4
    frame #38: 0x0000000100763075 codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::h21392c0c1faadf84(self=&0x1064b2da0) at sync.rs:549
    frame #39: 0x0000000100765f80 codechain`_$LT$alloc..sync..Arc$LT$T$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h5fa93c87ba0b1e7a(self=&0x1064b2da0) at sync.rs:1007:12
    frame #40: 0x00000001000f8635 codechain`core::ptr::real_drop_in_place::h7be64656c1d1b931((null)=&0x1064b2da0) at mod.rs:197
    frame #41: 0x0000000100103de5 codechain`core::ptr::real_drop_in_place::hc7c4eaf7b873d5a0((null)=&0x1064b2da0) at mod.rs:197
    frame #42: 0x00000001001e358e codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::habffa88273aa44a0 [inlined] core::ptr::drop_in_place::h259eae16d0e0688a(to_drop=&0x1064b2da0) at mod.rs:187:4
    frame #43: 0x00000001001e3585 codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::habffa88273aa44a0(self=&0x1064b3890) at sync.rs:549
    frame #44: 0x00000001001e52e0 codechain`_$LT$alloc..sync..Arc$LT$T$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h46502c27e917cff0(self=&0x1064b3890) at sync.rs:1007:12
    frame #45: 0x00000001000f7695 codechain`core::ptr::real_drop_in_place::h76e4d7e8903bea22((null)=&0x1064b3890) at mod.rs:197
    frame #46: 0x00000001000fc5c5 codechain`core::ptr::real_drop_in_place::h9720a862956f51dd((null)=&0x1064b3890) at mod.rs:197
    frame #47: 0x00000001000f8f2a codechain`core::ptr::real_drop_in_place::h7ecfd0f113ead9eb((null)=&mut RpcMethod<()> {

}) at mod.rs:197
    frame #48: 0x00000001001e29b9 codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::h569c4032e5879bba [inlined] core::ptr::drop_in_place::hf2a8e6f3e696bf45(to_drop=*mut RpcMethod<()> {

}) at mod.rs:187:4
    frame #49: 0x00000001001e29ac codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::h569c4032e5879bba(self=&0x107875478) at sync.rs:549
    frame #50: 0x00000001001e59e4 codechain`_$LT$alloc..sync..Arc$LT$T$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::hcfb9b51b1d9d50d1(self=&0x107875478) at sync.rs:1007:12
    frame #51: 0x00000001001070c5 codechain`core::ptr::real_drop_in_place::hdd814ee9ab14c5e9((null)=&0x107875478) at mod.rs:197
    frame #52: 0x0000000100109104 codechain`core::ptr::real_drop_in_place::he9b297f212a0618b((null)=&0x107875470) at mod.rs:197
    frame #53: 0x00000001000f0dba codechain`core::ptr::real_drop_in_place::h4d6120205b438ce3((null)=&0x107875458) at mod.rs:197
    frame #54: 0x00000001002a8f7d codechain`core::ptr::_$LT$impl$u20$$BP$mut$u20$T$GT$::drop_in_place::h4944b0f848e5e1b1 [inlined] core::ptr::drop_in_place::h049b054fe7917f37(to_drop=&0x107875458) at mod.rs:187:4
    frame #55: 0x00000001002a8f74 codechain`core::ptr::_$LT$impl$u20$$BP$mut$u20$T$GT$::drop_in_place::h4944b0f848e5e1b1(self=&0x107875458) at mod.rs:2284
    frame #56: 0x00000001000026f2 codechain`hashbrown::raw::Bucket$LT$T$GT$::drop::h8bd9d94e0eff1e17(self=&0x70000e9ded78) at mod.rs:304:8
    frame #57: 0x00000001000017a8 codechain`_$LT$hashbrown..raw..RawTable$LT$T$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h76664f573ca57a82(self=&0x1064b4410) at mod.rs:1011:24
    frame #58: 0x000000010010bca5 codechain`core::ptr::real_drop_in_place::hfaea88b28e513ccc((null)=&0x1064b4410) at mod.rs:197
    frame #59: 0x0000000100109209 codechain`core::ptr::real_drop_in_place::hea3645b85c4887a2((null)=&0x1064b4400) at mod.rs:197
    frame #60: 0x00000001000eef05 codechain`core::ptr::real_drop_in_place::h414f9b16ef6e5266((null)=&0x1064b4400) at mod.rs:197
    frame #61: 0x00000001000e73d5 codechain`core::ptr::real_drop_in_place::h12657f3c40de3b45((null)=&0x1064b4400) at mod.rs:197
    frame #62: 0x00000001001e33ce codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::ha014f3a7375727c4 [inlined] core::ptr::drop_in_place::h55d4c4219a2ba498(to_drop=&0x1064b4400) at mod.rs:187:4
    frame #63: 0x00000001001e33c5 codechain`alloc::sync::Arc$LT$T$GT$::drop_slow::ha014f3a7375727c4(self=&0x70000e9df1e8) at sync.rs:549
    frame #64: 0x00000001001e5ac0 codechain`_$LT$alloc..sync..Arc$LT$T$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::hd97a4666508b1494(self=&0x70000e9df1e8) at sync.rs:1007:12
    frame #65: 0x00000001000f1ae5 codechain`core::ptr::real_drop_in_place::h52064890857a36b1((null)=&0x70000e9df1e8) at mod.rs:197
    frame #66: 0x00000001000f89d0 codechain`core::ptr::real_drop_in_place::h7d380d1abc85b167((null)=&0x70000e9df1c0) at mod.rs:197
    frame #67: 0x00000001000ec72a codechain`core::ptr::real_drop_in_place::h32b5903a25a4962b((null)=&0x70000e9df190) at mod.rs:197
    frame #68: 0x00000001001023d5 codechain`core::ptr::real_drop_in_place::hbde25a3a49848fab((null)=&0x70000e9df190) at mod.rs:197
    frame #69: 0x00000001000f6915 codechain`core::ptr::real_drop_in_place::h71ef345ae3b31b60((null)=&0x70000e9df190) at mod.rs:197
    frame #70: 0x00000001000fc645 codechain`core::ptr::real_drop_in_place::h9794e41a730c0fe9((null)=&0x70000e9df190) at mod.rs:197
    frame #71: 0x000000010010025e codechain`core::ptr::real_drop_in_place::hae21c153e5fba507((null)=&0x70000e9df188) at mod.rs:197
    frame #72: 0x00000001000e52b5 codechain`core::ptr::real_drop_in_place::h0895aeac0a59a2de((null)=&0x70000e9df188) at mod.rs:197
    frame #73: 0x00000001000eb165 codechain`core::ptr::real_drop_in_place::h2a032111b5a115fb((null)=&0x70000e9df188) at mod.rs:197
    frame #74: 0x00000001003621a7 codechain`jsonrpc_ipc_server::server::ServerBuilder$LT$M$C$S$GT$::start::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hcd9ef08364bbd272((null)=((), futures::future::select::SelectNext<futures::stream::for_each::ForEach<futures::stream::buffer_unordered::BufferUnordered<futures::stream::map::Map<parity_tokio_ipc::Incoming, closure>>, closure, core::result::Result<(), std::io::error::Error>>, futures::future::map_err::MapErr<futures::sync::oneshot::Receiver<()>, closure>>) @ 0x000070000e9df188) at server.rs:245:5
    frame #75: 0x0000000100063cd0 codechain`core::result::Result$LT$T$C$E$GT$::map::hb26852891237f7d4(self=Result<((), futures::future::select::SelectNext<futures::stream::for_each::ForEach<futures::stream::buffer_unordered::BufferUnordered<futures::stream::map::Map<parity_tokio_ipc::Incoming, closure>>, closure, core::result::Result<(), std::io::error::Error>>, futures::future::map_err::MapErr<futures::sync::oneshot::Receiver<()>, closure>>), (std::io::error::Error, futures::future::select::SelectNext<futures::stream::for_each::ForEach<futures::stream::buffer_unordered::BufferUnordered<futures::stream::map::Map<parity_tokio_ipc::Incoming, closure>>, closure, core::result::Result<(), std::io::error::Error>>, futures::future::map_err::MapErr<futures::sync::oneshot::Receiver<()>, closure>>)> {

}, op=closure(Sender<void> {
inner: Arc<futures::sync::oneshot::Inner<void> > {
ptr: NonNull<alloc::sync::ArcInner<futures::sync::oneshot::Inner<void> > > {
pointer: &0x106209f80
},
phantom: PhantomData<futures::sync::oneshot::Inner<void> > {

}
}
})) at result.rs:460:24
    frame #76: 0x00000001001f4a3a codechain`_$LT$futures..future..map..Map$LT$A$C$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h5fdcdd34d9c00011(self=&0x10620a050) at map.rs:35:8
    frame #77: 0x000000010040443a codechain`_$LT$futures..future..map_err..MapErr$LT$A$C$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h9ed5240315a37791(self=&0x10620a050) at map_err.rs:30:22
    frame #78: 0x0000000100440d74 codechain`_$LT$futures..future..either..Either$LT$A$C$B$GT$$u20$as$u20$futures..future..Future$GT$::poll::h037e309cf85d0ab1(self=&0x10620a048) at either.rs:36:36
    frame #79: 0x000000010012de52 codechain`_$LT$futures..future..lazy..Lazy$LT$F$C$R$GT$$u20$as$u20$futures..future..Future$GT$::poll::h57bbdc19c7b6cd64(self=&0x10620a040) at lazy.rs:82:8
    frame #80: 0x00000001021282a5 codechain`_$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h7b8180e676b5c17b(self=&0x1062055c0) at mod.rs:113:12
    frame #81: 0x000000010203e2d5 codechain`futures::task_impl::Spawn$LT$T$GT$::poll_future_notify::_$u7b$$u7b$closure$u7d$$u7d$::hb1a95143f2e856e5((null)=closure {

}, f=&0x1062055c0) at mod.rs:329:44
    frame #82: 0x000000010203e550 codechain`futures::task_impl::Spawn$LT$T$GT$::enter::_$u7b$$u7b$closure$u7d$$u7d$::h0a218879ac83edb7 at mod.rs:399:26
    frame #83: 0x000000010205df97 codechain`futures::task_impl::std::set::h96ec3b74657519be(task=&0x70000e9e00b8, f=closure(Sender<void> {
inner: Arc<futures::sync::oneshot::Inner<void> > {
ptr: NonNull<alloc::sync::ArcInner<futures::sync::oneshot::Inner<void> > > {
pointer: &0x70000e9e0118
},
phantom: PhantomData<futures::sync::oneshot::Inner<void> > {

}
}
})) at mod.rs:83:12
    frame #84: 0x000000010203e4f8 codechain`futures::task_impl::Spawn$LT$T$GT$::enter::hb8b1d3d5a679b60f(self=&0x106205588, unpark=<unavailable>, f=closure {

}) at mod.rs:399:8
    frame #85: 0x000000010203e168 codechain`futures::task_impl::Spawn$LT$T$GT$::poll_fn_notify::h50f36f9c60e76051(self=&0x106205588, notify=&0x70000e9e0b48, id=4397749616, f=closure {

}) at mod.rs:291:8
    frame #86: 0x000000010203e235 codechain`futures::task_impl::Spawn$LT$T$GT$::poll_future_notify::h800b301e31aac0c3(self=&0x106205588, notify=&0x70000e9e0b48, id=4397749616) at mod.rs:329:8
    frame #87: 0x000000010206727d codechain`tokio_threadpool::task::Task::run::_$u7b$$u7b$closure$u7d$$u7d$::h0ad40e15ab2acaf0 at mod.rs:122:22
    frame #88: 0x0000000102042a2d codechain`core::ops::function::FnOnce::call_once::h138bf4d3e50e7f8b((null)=<unavailable>, (null)=<unavailable>) at function.rs:231:4
    frame #89: 0x000000010203a678 codechain`_$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hcc93a23ed2142ce6(self=<unavailable>, _args=<unavailable>) at panic.rs:315:8
    frame #90: 0x0000000102049bf5 codechain`std::panicking::try::do_call::h2d1dfae5f11c2eab(data=&0x70000e9e03d8) at panicking.rs:296:39
    frame #91: 0x000000010238a63f codechain`__rust_maybe_catch_panic at lib.rs:82:7 [opt]
    frame #92: 0x00000001020498d3 codechain`std::panicking::try::h24a85fc723795907(f=<unavailable>) at panicking.rs:275:12
    frame #93: 0x000000010203a82f codechain`std::panic::catch_unwind::h0c5c13ff468a4bb2(f=<unavailable>) at panic.rs:394:8
    frame #94: 0x0000000102066d81 codechain`tokio_threadpool::task::Task::run::hae7c8f59a62e58c0(self=&0x106205570, unpark=&0x70000e9e0b48) at mod.rs:108:18
    frame #95: 0x00000001020715f1 codechain`tokio_threadpool::worker::Worker::run_task2::h75690498a8ef2f7c(self=&0x70000e9e19f0, task=&0x70000e9e0960, notify=&0x70000e9e0b48) at mod.rs:555:8
    frame #96: 0x0000000102071102 codechain`tokio_threadpool::worker::Worker::run_task::h521d2164321d0465(self=&0x70000e9e19f0, task=Arc<tokio_threadpool::task::Task> {
ptr: NonNull<alloc::sync::ArcInner<tokio_threadpool::task::Task> > {
pointer: &0x106205560
},
phantom: PhantomData<tokio_threadpool::task::Task> {

}
}, notify=&0x70000e9e0b48) at mod.rs:454:18
    frame #97: 0x00000001020709ba codechain`tokio_threadpool::worker::Worker::try_run_owned_task::h77fc67f1da4d91ec(self=&0x70000e9e19f0, notify=&0x70000e9e0b48) at mod.rs:394:16
    frame #98: 0x000000010207043d codechain`tokio_threadpool::worker::Worker::try_run_task::h8b6487c644247284(self=&0x70000e9e19f0, notify=&0x70000e9e0b48) at mod.rs:302:11
    frame #99: 0x00000001020702b9 codechain`tokio_threadpool::worker::Worker::run::h46782f8b9e285500(self=&0x70000e9e19f0) at mod.rs:246:15
    frame #100: 0x0000000101fe1978 codechain`tokio::runtime::threadpool::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h41976187efebf0c6 at builder.rs:349:32
    frame #101: 0x0000000101fe694d codechain`tokio_trace_core::dispatcher::with_default::h8cd94b017dcbf493(dispatcher=&0x1064b4ab0, f=closure(Sender<void> {
inner: Arc<futures::sync::oneshot::Inner<void> > {
ptr: NonNull<alloc::sync::ArcInner<futures::sync::oneshot::Inner<void> > > {
pointer: &0x70000e9e1400
},
phantom: PhantomData<futures::sync::oneshot::Inner<void> > {

}
}
})) at dispatcher.rs:54:4
    frame #102: 0x0000000101fe19a9 codechain`tokio::runtime::threadpool::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h822da605c28f7df3((null)=&0x70000e9e16d8) at builder.rs:348:28
    frame #103: 0x0000000101fda38c codechain`tokio_timer::timer::handle::with_default::_$u7b$$u7b$closure$u7d$$u7d$::h98fa079eefc22439(current=&0x10787ea70) at handle.rs:101:8
    frame #104: 0x0000000101fe7f97 codechain`std::thread::local::LocalKey$LT$T$GT$::try_with::h0bb59c317b509b84(self=&0x1026d50d0, f=closure(&0x70000e9e0e58, closure(&0x1064b4ab0, &0x70000e9e1400), &0x70000e9e0e60)) at local.rs:257:15
    frame #105: 0x0000000101fe7e48 codechain`std::thread::local::LocalKey$LT$T$GT$::with::hd3e28a711a87635b(self=&0x1026d50d0, f=<unavailable>) at local.rs:234:8
    frame #106: 0x0000000101fda218 codechain`tokio_timer::timer::handle::with_default::h4d7e9939c8b9f28d(handle=&0x10640e1c0, enter=&0x70000e9e16d8, f=closure(&0x1064b4ab0, &0x70000e9e1400)) at handle.rs:84:4
    frame #107: 0x0000000101fe1a02 codechain`tokio::runtime::threadpool::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h915a04e9ca791ff2(enter=&0x70000e9e16d8) at builder.rs:347:24
    frame #108: 0x0000000101fe3452 codechain`tokio_timer::clock::clock::with_default::_$u7b$$u7b$closure$u7d$$u7d$::h00368a73e97e4062(cell=&0x10787d880) at clock.rs:137:8
    frame #109: 0x0000000101fe8533 codechain`std::thread::local::LocalKey$LT$T$GT$::try_with::h5a40f68bc805c688(self=&0x1026d55c0, f=closure(&0x70000e9e10e0, closure(&0x1064b4a98, closure(&0x70000e9e1410, &0x1064b4ab0), &0x70000e9e1400), &0x70000e9e10e8)) at local.rs:257:15
    frame #110: 0x0000000101fe7d99 codechain`std::thread::local::LocalKey$LT$T$GT$::with::ha69eff4b8bf66d02(self=&0x1026d55c0, f=<unavailable>) at local.rs:234:8
    frame #111: 0x0000000101fe334f codechain`tokio_timer::clock::clock::with_default::h236e0415dd6d0082(clock=&0x1064b4a88, enter=&0x70000e9e16d8, f=<unavailable>) at clock.rs:117:4
    frame #112: 0x0000000101fe1a52 codechain`tokio::runtime::threadpool::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::he5c6732fb925b409(enter=&0x70000e9e16d8) at builder.rs:346:20
    frame #113: 0x0000000101fe6634 codechain`tokio_reactor::with_default::_$u7b$$u7b$closure$u7d$$u7d$::hf016be4623578d31(current=&0x10787ea90) at lib.rs:207:8
    frame #114: 0x0000000101fe83c1 codechain`std::thread::local::LocalKey$LT$T$GT$::try_with::h40f389bbe73e4ece(self=&0x1026d5780, f=closure(&0x70000e9e1370, closure(&0x1064b4a88, &0x1064b4a98, &0x70000e9e1410, &0x1064b4ab0, &0x70000e9e1400), &0x70000e9e1378)) at local.rs:257:15
    frame #115: 0x0000000101fe7d39 codechain`std::thread::local::LocalKey$LT$T$GT$::with::h87fff46d0bd903ad(self=&0x1026d5780, f=<unavailable>) at local.rs:234:8
    frame #116: 0x0000000101fe64b7 codechain`tokio_reactor::with_default::h7f6fe046464239ae(handle=&0x1064b3240, enter=&0x70000e9e16d8, f=<unavailable>) at lib.rs:199:4
    frame #117: 0x0000000101fe1ae7 codechain`tokio::runtime::threadpool::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h7e1eddc63d52f49d(w=&0x70000e9e19f0, enter=&0x70000e9e16d8) at builder.rs:345:16
    frame #118: 0x0000000102048976 codechain`tokio_threadpool::callback::Callback::call::h7820641a027b4d52(self=&0x1064b97a0, worker=&0x70000e9e19f0, enter=&0x70000e9e16d8) at callback.rs:21:8
    frame #119: 0x000000010206ffd6 codechain`tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::had1885700dc6ad3a(enter=&0x70000e9e16d8) at mod.rs:132:20
    frame #120: 0x00000001020497b3 codechain`tokio_executor::global::with_default::_$u7b$$u7b$closure$u7d$$u7d$::hd989dbe27e857841(cell=&0x10787e950) at global.rs:209:8
    frame #121: 0x0000000102058029 codechain`std::thread::local::LocalKey$LT$T$GT$::try_with::h7d3b37b7c4beb2d8(self=&0x1026d7938, f=closure(&0x70000e9e1670, &0x70000e9e1828, &0x70000e9e1678)) at local.rs:257:15
    frame #122: 0x0000000102057a90 codechain`std::thread::local::LocalKey$LT$T$GT$::with::hc42b2c1ed7c5c757(self=&0x1026d7938, f=<unavailable>) at local.rs:234:8
    frame #123: 0x00000001020496ac codechain`tokio_executor::global::with_default::h53e5dd673f673a49(executor=&0x70000e9e16d0, enter=&0x70000e9e16d8, f=closure(Sender<void> {
inner: Arc<futures::sync::oneshot::Inner<void> > {
ptr: NonNull<alloc::sync::ArcInner<futures::sync::oneshot::Inner<void> > > {
pointer: &0x70000e9e1828
},
phantom: PhantomData<futures::sync::oneshot::Inner<void> > {

}
}
})) at global.rs:178:4
    frame #124: 0x0000000102070067 codechain`tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::h5da40fadc9776078(c=&0x10787eab0) at mod.rs:130:12
    frame #125: 0x000000010205813c codechain`std::thread::local::LocalKey$LT$T$GT$::try_with::h89953703b4eb72bc(self=&0x1026d92e8, f=closure(Sender<void> {
inner: Arc<futures::sync::oneshot::Inner<void> > {
ptr: NonNull<alloc::sync::ArcInner<futures::sync::oneshot::Inner<void> > > {
pointer: &0x70000e9e1828
},
phantom: PhantomData<futures::sync::oneshot::Inner<void> > {

}
}
})) at local.rs:257:15
    frame #126: 0x0000000102057add codechain`std::thread::local::LocalKey$LT$T$GT$::with::hc75fdb8693e34fd7(self=&0x1026d92e8, f=closure(Sender<void> {
inner: Arc<futures::sync::oneshot::Inner<void> > {
ptr: NonNull<alloc::sync::ArcInner<futures::sync::oneshot::Inner<void> > > {
pointer: &0x70000e9e1828
},
phantom: PhantomData<futures::sync::oneshot::Inner<void> > {

}
}
})) at local.rs:234:8
    frame #127: 0x000000010206fec4 codechain`tokio_threadpool::worker::Worker::do_run::h674670ef8555a617(self=&0x70000e9e19f0) at mod.rs:121:8
    frame #128: 0x00000001020757c5 codechain`tokio_threadpool::pool::Pool::spawn_thread::_$u7b$$u7b$closure$u7d$$u7d$::h5465db0d66791494 at mod.rs:354:20
    frame #129: 0x000000010204f160 codechain`std::sys_common::backtrace::__rust_begin_short_backtrace::hac18f7d772a7346f(f=<unavailable>) at backtrace.rs:77:4
    frame #130: 0x0000000102041dd0 codechain`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h4571ef1b067aa4b1 at mod.rs:470:16
    frame #131: 0x000000010203a640 codechain`_$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hbca812925c9d3c48(self=<unavailable>, _args=<unavailable>) at panic.rs:315:8
    frame #132: 0x0000000102049c6d codechain`std::panicking::try::do_call::hc6694999c01e2440(data=&0x70000e9e1c98) at panicking.rs:296:39
    frame #133: 0x000000010238a63f codechain`__rust_maybe_catch_panic at lib.rs:82:7 [opt]
    frame #134: 0x0000000102049a75 codechain`std::panicking::try::h68180c93bf4182a3(f=<unavailable>) at panicking.rs:275:12
    frame #135: 0x000000010203a7e0 codechain`std::panic::catch_unwind::h0179e8d904d67123(f=<unavailable>) at panic.rs:394:8
    frame #136: 0x0000000102041c1e codechain`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h49a2c4453bfcd469 at mod.rs:469:29
    frame #137: 0x00000001020429b5 codechain`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h7d9b379405d69b70((null)=&0x70000e9e1e70, (null)=<unavailable>) at function.rs:231:4
    frame #138: 0x000000010237645e codechain`_$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::ha44ca87baccc45af at boxed.rs:746:8 [opt]
    frame #139: 0x0000000102389b6e codechain`std::sys::unix::thread::Thread::new::thread_start::hceb9fd3b79bf176c [inlined] _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hf7f4d50b605effc3 at boxed.rs:746:8 [opt]
    frame #140: 0x0000000102389b62 codechain`std::sys::unix::thread::Thread::new::thread_start::hceb9fd3b79bf176c [inlined] std::sys_common::thread::start_thread::he0046dfc66dff589 at thread.rs:13 [opt]
    frame #141: 0x0000000102389aee codechain`std::sys::unix::thread::Thread::new::thread_start::hceb9fd3b79bf176c at thread.rs:79 [opt]
    frame #142: 0x00007fff5b3b72eb libsystem_pthread.dylib`_pthread_body + 126
    frame #143: 0x00007fff5b3ba249 libsystem_pthread.dylib`_pthread_start + 66
    frame #144: 0x00007fff5b3b640d libsystem_pthread.dylib`thread_start + 13
  thread #37
    frame #0: 0x00007fff5b2fbf32 libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fff5b287914 libsystem_c.dylib`nanosleep + 199
    frame #2: 0x0000000102376cb0 codechain`std::thread::sleep::ha3264eefde5aa55f [inlined] std::sys::unix::thread::Thread::sleep::h805478b62361d458 at thread.rs:152:19 [opt]
    frame #3: 0x0000000102376c50 codechain`std::thread::sleep::ha3264eefde5aa55f at mod.rs:781 [opt]
    frame #4: 0x00000001002d24f8 codechain`ctrlc::CtrlC::set_handler_with_polling_rate::_$u7b$$u7b$closure$u7d$$u7d$::hc4e8940201115e91 at lib.rs:121:16
    frame #5: 0x000000010012e118 codechain`std::sys_common::backtrace::__rust_begin_short_backtrace::hbcd6ce9d1f6d80a5(f=<unavailable>) at backtrace.rs:77:4
    frame #6: 0x000000010035f978 codechain`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h42b77fb42460d3c1 at mod.rs:470:16
    frame #7: 0x00000001003c7e28 codechain`_$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hd7b9a14df127bd7e(self=<unavailable>, _args=<unavailable>) at panic.rs:315:8
    frame #8: 0x000000010042be15 codechain`std::panicking::try::do_call::h379d336dea17997a(data=&0x70000f1edcb8) at panicking.rs:296:39
    frame #9: 0x000000010238a63f codechain`__rust_maybe_catch_panic at lib.rs:82:7 [opt]
    frame #10: 0x000000010042b705 codechain`std::panicking::try::h8f2917b13f70940a(f=<unavailable>) at panicking.rs:275:12
    frame #11: 0x00000001003c8008 codechain`std::panic::catch_unwind::he5a22b6f0a5206a1(f=<unavailable>) at panic.rs:394:8
    frame #12: 0x000000010035eb9e codechain`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::hc991b4e5138a0e2d at mod.rs:469:29
    frame #13: 0x00000001000e06b5 codechain`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h7aa1bb91c143f3be((null)=&0x70000f1ede70, (null)=<unavailable>) at function.rs:231:4
    frame #14: 0x000000010237645e codechain`_$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::ha44ca87baccc45af at boxed.rs:746:8 [opt]
    frame #15: 0x0000000102389b6e codechain`std::sys::unix::thread::Thread::new::thread_start::hceb9fd3b79bf176c [inlined] _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hf7f4d50b605effc3 at boxed.rs:746:8 [opt]
    frame #16: 0x0000000102389b62 codechain`std::sys::unix::thread::Thread::new::thread_start::hceb9fd3b79bf176c [inlined] std::sys_common::thread::start_thread::he0046dfc66dff589 at thread.rs:13 [opt]
    frame #17: 0x0000000102389aee codechain`std::sys::unix::thread::Thread::new::thread_start::hceb9fd3b79bf176c at thread.rs:79 [opt]
    frame #18: 0x00007fff5b3b72eb libsystem_pthread.dylib`_pthread_body + 126
    frame #19: 0x00007fff5b3ba249 libsystem_pthread.dylib`_pthread_start + 66
    frame #20: 0x00007fff5b3b640d libsystem_pthread.dylib`thread_start + 13

Navigation Menu

Repository navigation

CodeChain crashes on SIGINT #348

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CodeChain crashes on SIGINT #348

Description

Activity

kseo commented on Aug 4, 2018

majecty commented on Aug 4, 2018

majecty commented on Aug 4, 2018

majecty commented on Aug 8, 2018

sgkim126 commented on Aug 17, 2018

sgkim126 commented on Aug 17, 2018

3 remaining items

sgkim126 commented on Apr 10, 2019

sgkim126 commented on May 13, 2019

sgkim126 commented on May 20, 2019

sgkim126 commented on Jun 14, 2019

foriequal0 commented on Oct 7, 2019

foriequal0 commented on Oct 7, 2019

foriequal0 commented on Oct 8, 2019

foriequal0 commented on Oct 11, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions