当前位置:网站首页>Still have 1 requests outstanding when connection from slaveX/X.X.X.X:33202 is closed
Still have 1 requests outstanding when connection from slaveX/X.X.X.X:33202 is closed
2022-07-23 21:28:00 【格格巫 MMQ!!】
ERROR TransportResponseHandler:
Still have 1 requests outstanding when connection from is closed
我加了如下的参数
spark.driver.memory=4g
spark.hadoop.mapreduce.input.fileinputformat.split.maxsize=134217728
spark.dynamicAllocation.initialExecutors=50
spark.yarn.executor.memoryOverhead=2048
spark.dynamicAllocation.maxExecutors=1000
spark.dynamicAllocation.tasksPerExecutorSlot=1
运行的程序其实逻辑上比较简单,只是从hive表里读取的数据量很大,差不多60+G,
并且需要将某些hive表读取到dirver节点上,用来获取每个executor上某些数据的映射值,
所以driver设定的资源较大。运行时抛出的异常信息,从网上查了下原因大致是服务器的
并发连接数超过了其承载量,服务器会将其中一些连接Down掉,这也就是说在运行spark程序时,过多的申请资源并发执行。
java.util.concurrent.RejectedExecutionException: Task [email protected] rejected from java.util.concurrent.ThreadPoolExecutor
spark 日志如上所示。还有一些其他日志,
18/06/11 18:40:43 ERROR TransportResponseHandler:
边栏推荐
- TCP half connection queue and full connection queue (the most complete in History)
- Chapter1 数据清洗
- 1063 Set Similarity
- Basic principle of synchronized lock
- 集群聊天服务器:集群与分布式理论
- Leetcode hot topic hot52-100
- It's good to change jobs for a while, and it's good to change jobs all the time?
- Unity—3D数学-Vector3
- Looking for the missing class name
- 1062 Talent and Virtue
猜你喜欢

Protocol buffers 的问题和滥用

& 9 nodemon automatic restart tool

Compare kernelshap and treeshap based on speed, complexity and other factors
现在完全不知道怎么同步

Problems and abuse of protocol buffers

Typescript Basics

Network learning infrared module, 8-way emission independent control

集群聊天服务器:Model数据层的框架设计和数据库代码的封装
![[isprint function determines whether characters can be output]](/img/c8/c06dc36dbd0bf1feeba64c7753faf5.png)
[isprint function determines whether characters can be output]

221. Largest square ● &1277. Square submatrix with statistics all 1 ● ●
随机推荐
flink原理及开发总结(详细)
手机测试相关基础知识
WinDbg practice -- Introduction
Chapter 2 Regression
剑指 Offer II 115. 重建序列 : 拓扑排序构造题
Is it safe to open a mobile stock account?
& 9 nodemon automatic restart tool
SQLite database
High numbers | calculation of double integral 4 | high numbers | handwritten notes
[continuous update] collection of raspberry pie startup and failure series
Complete set of official openlayers instances
Cluster chat server: creation of project directory
Basic principle of synchronized lock
集群聊天服務器:數據庫錶的設計
Proof of green Tao theorem (1): preparation, notation and Gowers norm
Cluster chat server: cluster and distributed theory
How to introduce your project experience in the interview
05_ UE4 advanced_ Material UV scaling
Unity - 3D mathematics -vector3
Protocol buffers 的问题和滥用