Get Transport death lock if all brokers shutdown unnormally

[ShannonDing]

Test Env:

• 1broker
• 1 push consumer

How to reproduce:

• Use push consumer to connect to broker,
• shutdown the broker
• restart the broker
• check the consumer

logs:

[2018-Dec-12 07:29:30.760170](info):namesrvIndex is:8, index:0, namesrvaddrlist size:1

[2018-Dec-12 07:29:31.240943](info):Wait till event base is looping

[2018-Dec-12 07:29:31.637820](info):eventcb: received event:11 on fd:14

[2018-Dec-12 07:29:31.637899](info):eventcb:rcv error event cb:11 on fd:14

[2018-Dec-12 07:29:31.637931](info):event_base_dispatch exit once

[2018-Dec-12 07:29:32.241539](info):Wait till event base is looping

[2018-Dec-12 07:29:33.242114](info):Wait till event base is looping

[2018-Dec-12 07:29:33.760524](error):GetTransport of:172.17.0.2:9876 get timed_mutex timeout

[2018-Dec-12 07:29:33.760650](info):updateTopicRouteInfoFromNameServer start:T_TestTopic

[2018-Dec-12 07:29:33.760759](info):namesrvIndex is:9, index:0, namesrvaddrlist size:1

[2018-Dec-12 07:29:34.242835](info):Wait till event base is looping

[2018-Dec-12 07:29:35.243643](info):Wait till event base is looping

[2018-Dec-12 07:29:36.244387](info):Wait till event base is looping

[2018-Dec-12 07:29:36.761118](error):GetTransport of:172.17.0.2:9876 get timed_mutex timeout

[2018-Dec-12 07:29:37.244875](info):Wait till event base is looping

[2018-Dec-12 07:29:38.245616](info):Wait till event base is looping

[2018-Dec-12 07:29:39.246274](info):Wait till event base is looping

[2018-Dec-12 07:29:40.246906](info):Wait till event base is looping

[2018-Dec-12 07:29:41.247591](info):Wait till event base is looping

[2018-Dec-12 07:29:42.248256](info):Wait till event base is looping

[2018-Dec-12 07:29:43.249143](info):Wait till event base is looping

[sstream536]

The same problem arises in our environment, the death lock is always happend to the consumer start function @ShannonDing Can you fix this?

[ShannonDing]

The same problem arises in our environment, the deathlock always happens to the consumer start function @ShannonDing Can you fix this?
This problem is under analysis. submitting PR is warmly welcomed if you have a good solution.

[sstream536]

@ShannonDing ok! Thank you for your attention, look at the code below

tcpConnectStatus TcpTransport::connect(const string &strServerURL,
                                       int timeOutMillisecs /* = 3000 */) {
 //....
 m_ReadDatathread = new boost::thread(boost::bind(&TcpTransport::runThread, this));
    while(!m_event_base_status) {
      LOG_INFO("Wait till event base is looping");
      boost::system_time const timeout=boost::get_system_time()+ boost::posix_time::milliseconds(1000);
      boost::unique_lock<boost::mutex> lock(m_event_base_mtx);
      m_event_base_cv.timed_wait(lock, timeout);
    }
//.....
}
void TcpTransport::runThread() {
  while (m_ReadDatathread) {
    if (m_eventBase != NULL) {
      
      if (!m_event_base_status) {
        boost::mutex::scoped_lock lock(m_event_base_mtx);
        m_event_base_status.store(true);
        m_event_base_cv.notify_all();
        LOG_INFO("Notify on event_base_dispatch");
      }
      event_base_dispatch(m_eventBase);
      // event_base_loop(m_eventBase, EVLOOP_ONCE);//EVLOOP_NONBLOCK should not
      // be used, as could not callback event immediatly
    }
    LOG_INFO("event_base_dispatch exit once");
    boost::this_thread::sleep(boost::posix_time::milliseconds(1));
    if (getTcpConnectStatus() != e_connectSuccess) return;
  }
}

I think m_ReadDatathread is nullptr when the runThread function call thread loop in the TcpTransport.cpp, so the thread loop depends on the m_ReadDatathread is unreasonable.
If the runThread function return because of m_ReadDatathread is nullptr, the m_event_base_cv condition variable will won't be able to notify and fire, so the thread called the connect function will be endless loop.

[sstream536]

I have fixed it

[ifplusor]

@KARLLEN we have changed thread model of event loop, could you review PR #60 ?

[ShannonDing]

I have fixed it
@KARLLEN Thank you for your information, Could you please review and verify this PR #60 .