[300 + selected interview questions from big companies continued to share] big data operation and maintenance sharp knife interview question column (V)

Continuous sharing is useful 、 valuable 、 Selected high-quality big data interview questions

We are committed to building the most comprehensive big data interview topic database in the whole network

41、NodeManager The node will change the current node's DataNode Report to RescourceManager, But you said before HDFS The only access to the cluster is NameNode, Whether each one NodeManager Collecting local DataNode Data must be accessed at all times NameNode Get it ？ If so, it's too wasteful HDFS Performance is up , If not, it is NodeManager What mechanism is used to realize without NameNode You can get one DataNode What about the data? ？

Refer to the answer ：

        “NodeManager The node will change the current node's DataNode Report to RescourceManager”, This is not true , Should be DataNode Report your own block to NameNode,RescourceManager and NodeManager It's resources, not data . When the job is to be read hdfs Data only comes from NameNode Where to get data on , Instead of getting the data itself , Get the location of the data and then find it DataNode Get the data itself .

42、CDH It is an open source product , however CM It's paid , We use it in a production environment CM no Directed cloudera Will the company receive a lawyer's letter when paying ？ But there's no need to CM We don't know if What management ( build , Capacity expansion , Monitoring etc. )CDH Clustered , If the company is not willing to pay, we mostly According to the operation and maintenance, how to deal with it ？
 

Refer to the answer ：

         There will be no free ones in the future , If you are not willing to pay, you can use the open source community version , If there are not many nodes Human flesh operation and maintenance , Self built monitoring platform , If there are many nodes, you can write scripts or use self-help tools Operation and maintenance , such as ansible etc. .

43、ElasticSearch and HDFS It's all distributed file systems , Can also do data storage and retrieval functions , All of them are JAVA Open source products , Why in the field of big data HDFS Than ES It's even hotter ？

Refer to the answer ：

        These two components are not comparable , The scenarios are different ,HDFS Applied to massive data storage ,ES Apply to full-text search , It is widely used in e-commerce and search engines

44、 Developers in hive How can the metadata table information created in be monitored in real time ？

Refer to the answer ：

        You can use Alibaba open source Canal To read in real time Hive Metabase of mysql Of binlog data , Achieve the change operation of real-time monitoring table .

45、HDFS The data of is deleted and cannot be monitored directly , How to prevent developers from deleting data by mistake in enterprises, so that our operation and maintenance can't be blamed ？

Refer to the answer ：

    Open the audit log , And use filebeat Collect and then write ES in , It can query all data in real time .

46、 Big data security , Rights management , Does the audit have a complete solution ？

Refer to the answer ：

      As far as the teacher knows , There is no complete plan , Possible commercial products are , Big data security can be roughly as secure as border gateways , For example, cloud desktop 、VPN etc. , The second is the security of big data components .

47、 In the production environment, we need to make use of CM Of " Chart generator " Customize icons to form a dashboard Do you ? If necessary , There are too many official measures , What kind of supervisor have you defined in the production environment Control icon ?
 

Refer to the answer ：
        We will do this in the formal class CM Explain in the monitoring chapter , Mainly the host and various components The core indicators of , When problems occur, you can check other indicators .

48、 My previous question is to understand , Hot and cold data file type and storage compression , There are two points , Teacher only Reply to compression . Compression layer , Think about it , It is only related to CPU hook , There is not much to discuss , Objective front I good p. Of yes Yes On 1000 gauge model Of Set Group writing Pieces of save Store class type Of choose Choose .ORC,TEXT,Parquert Such as these , What aspects are the design layers based on .
 

Refer to the answer ：
        The data format we collected is snappy, Think of it as ODS layer ; For precipitation data gz;parquet It's not unsuitable for counting positions , But for the specific stratification of the log warehouse, the advantage is not reflected ,parquet The advantage is obvious when you only focus on a few fields in the table , In the data warehouse ODS、DWD Layers are concerned about almost all fields of the table , This is the time parquet The advantages are not obvious , stay DM Layer and the OLAP The advantages are obvious .

49、 How to achieve automatic repair when a cluster fails ? For example, a DataNode In the morning 2 Half a sudden But I hung up 4 Nodes , At that time, our operation and maintenance personnel had entered a deep sleep state , How to make it automatically The node process is automatically pulled up ( Here we assume that the process storage is successfully started )？
 

Refer to the answer ：
        We collected CDH Serious events , Whether there are more serious events in the inventory datanode Hang up 了 , If so, research CDH API start-up datanode. Besides, we have people on duty by turns every day .

50、 Lack of a unified open source monitoring solution , All companies have to develop relevant big data monitoring by themselves Platform? ？
 

Refer to the answer ：
        Generally, it is open source plus secondary development , It's hard to have a perfect plan .

Continuous sharing is useful 、 valuable 、 Selected high-quality big data interview questions

We are committed to building the most comprehensive big data interview topic database in the whole network