今天收到报警Datanode is dead，登录上去看下发现datanode进程还“活着”，没有高负载，内存也正常，datanode日志发现只有几处block传输异常，之后就是在接收block，但是心跳超时导致被NN认为死亡：

1
2
3
4
5
6
7
8
9
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException inBlockReceiver.run():
java.net.SocketException:Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:
96
)
at java.net.SocketOutputStream.write(SocketOutputStream.java:
136
)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:
65
)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:
123
)
at java.io.DataOutputStream.flush(DataOutputStream.java:
106
)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:
1003
)
at java.lang.Thread.run(Thread.java:
662
)

查看了ThreadDump（见附1）后终于发现原因：

在pipeline失败后，需要创建一个新的DataXceiver进行pipeline recovery,1.调用recoverRbw方法会锁定FsDatasetImp；2.然后调用stopWriter:即在进行错误恢复时，如果该数据节点的写线程DataReceiver如果还没有结束，要首先中断并等待该线程退出，保证后续恢复：

1
2
3
4
5
6
7
8
9
10
public 
void 
stopWriter() 
throws 
IOException {
  
if 
(writer != 
null 
&& writer != Thread.currentThread() && writer.isAlive()) {
    
writer.interrupt();
    
try 
{
      
writer.join();
    
} 
catch 
(InterruptedException e) {
      
throw 
new 
IOException(
"Waiting for writer thread is interrupted."
);
    
}
  
}
}

3.同时老的写入线程可能正在提交block，因此需要对FsDatasetImpl进行锁定，这时死锁发生了！！！

4.而Datanode发送Heartbeat前也需要获取FsDatasetImpl的锁定获取节点容量，因此发生超时！


其实这是一个Bug，在CDH4.4.0已经修复了，引入隐藏参数dfs.datanode.xceiver.stop.timeout.millis默认1min，对stopWriter引入超时join.

附1：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
"DataXceiverfor client xxx at xxx [Receiving block xxx]" 
daemon prio=10tid=
0x00000000406bc000 
nid=
0x612d 
in Object.wait() [
0x00007ffdcb314000
]
   
java.lang.Thread.State: WAITING (on objectmonitor)
        
at java.lang.Object.wait(Native Method)
        
atjava.lang.Thread.join(Thread.java:
1186
)
        
- locked <
0x00000007ca882a30
> (aorg.apache.hadoop.util.Daemon)
        
atjava.lang.Thread.join(Thread.java:
1239
)
        
atorg.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.stopWriter(ReplicaInPipeline.java:
157
)
        
atorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverRbw(FsDatasetImpl.java:
707
)
        
- locked <
0x0000000784acf478
> (aorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
        
atorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverRbw(FsDatasetImpl.java:
91
)
        
atorg.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:
163
)
        
atorg.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:
457
)
        
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:
103
)
        
atorg.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:
67
)
        
atorg.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
221
)
        
at java.lang.Thread.run(Thread.java:
662
)

1
2
3
4
5
6
"PacketResponder:xxx, type=HAS_DOWNSTREAM_IN_PIPELINE" 
daemon prio=10tid=
0x00007ffde0248000 
nid=
0x5ed7 
waiting 
for 
monitor entry [
0x00007ffdccbf9000
]
   
java.lang.Thread.State: BLOCKED (on objectmonitor)
        
atorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeBlock(FsDatasetImpl.java:
846
)
        
- waiting to lock<
0x0000000784acf478
> (a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
        
atorg.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:
964
)
        
atjava.lang.Thread.run(Thread.java:
662
)

本文转自MIKE老毕 51CTO博客，原文链接：http://blog.51cto.com/boylook/1313655，如需转载请自行联系原作者