Due to resource constraints, the namenode fails to start with an error unable to create new native thread

Catalog

1、namenode Log error information

2、 Problem cause analysis

3、 Problem thinking ： Why is there insufficient thread resources ？

4、 Check /etc/security/limits.conf Configuration file for

5、 The solution is 90-nproc.conf Additional configuration in the configuration file

6、 Restart ,nn1 Successful startup

Problem review and reflection ：

---

1、namenode Log error information

2022-07-22 14:00:54,578 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2022-07-22 14:00:54,578 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9000: starting
2022-07-22 14:00:55,240 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at org.apache.hadoop.ipc.Server.start(Server.java:2880)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.start(NameNodeRpcServer.java:521)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:767)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:711)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:905)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:884)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1678)
2022-07-22 14:00:55,243 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1

2、 Problem cause analysis

unable to create new native thread, Unable to create new native thread .

3、 Problem thinking ： Why is there insufficient thread resources ？

Because this is a newly built hadoop colony , I went back to check the installation documents , The following configuration modifications were found on the document .65536, This thread is completely sufficient for the new cluster , Why didn't it work ？

sed -i '/soft nofile/d' /etc/security/limits.conf
echo '* soft nofile 65536' >> /etc/security/limits.conf
sed -i '/hard nofile/d' /etc/security/limits.conf
echo '* hard nofile 65536' >> /etc/security/limits.conf 

4、 Check /etc/security/limits.conf Configuration file for

It is found that , But the inspection found that it did not take effect .

Someone on the Internet said limits.conf must Restart and take effect , Our operation and maintenance personnel generally do not restart the machine easily , So we can only use other methods .

* soft nproc 65535
* hard nproc 65535


# End of file
* soft nofile 65536
* hard nofile 65536

5、 processing method
stay 90-nproc.conf Additional configuration in the configuration file

* - nproc 65536 
* - nofile 65536

6、 Restart ,nn1 Successful startup

$ jps
5379 JournalNode
19845 NameNode
24042 Jps

Problem review and reflection ：

The reason for the problem is to build hadoop The cluster before , The adjustment of resource restrictions did not take effect . So I suggest you relax the resource restrictions , Delete first /etc/security/limits.conf Related Settings , modify /etc/security/limits.d/90-nproc.conf, Relax the maximum number of processes and file handles .