当前位置:网站首页>Use reflection to convert RDD to dataframe
Use reflection to convert RDD to dataframe
2022-07-25 03:01:00 【Rookies also have dreams】
Java:
package cn.spark.sql;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SQLContext;
import java.util.List;
public class RDD2DataFrameReflection {
public static void main(String[] args){
SparkConf conf = new SparkConf().setAppName("RDD2DataFrameReflection")
.setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);
SQLContext sqlContext = new SQLContext(sc);
JavaRDD<String> lines = sc.textFile("C://Users//Desktop//students.txt");
JavaRDD<Student> students = lines.map(
new Function<String, Student>() {
private static final long serialVersionUID = 1L;
@Override
public Student call(String s) throws Exception {
String[] lineSplit = s.split(",");
Student stu = new Student();
stu.setID(Integer.valueOf(lineSplit[0].trim()));
stu.setName(lineSplit[1]);
stu.setAge(Integer.valueOf(lineSplit[2].trim()));
return stu;
}
});
/*
* Use reflection take RDD Convert to DataFrame
* take Student.class Put it in In fact, it is created by reflection DataFrame
* Because the bottom layer passes Student class On reflection To get some of the field
* requirement javabean We need to implement serializable Interface
* */
DataFrame studentDF = sqlContext.createDataFrame(
students,Student.class);
// To get a DataFrame after You can register a temporary table
studentDF.registerTempTable("students");
DataFrame teengerDF = sqlContext.sql("select * from students where age <= 18");
// Will find out DataFrame Convert again to RDD
JavaRDD<Row> teengerRDD = teengerDF.javaRDD();
// take EDD Data in mapping It maps to student
JavaRDD<Student> teengerStudentRDD = teengerRDD.map(new Function<Row, Student>() {
@Override
public Student call(Row row) throws Exception {
Student stu = new Student();
stu.setAge(row.getInt(0));
stu.setID(row.getInt(1));
stu.setName(row.getString(2));
return stu;
}
});
List<Student> studentList = teengerStudentRDD.collect();
for (Student stu : studentList){
System.out.println(stu);
}
}
}
test :

边栏推荐
- Tp5.0 background admin access
- Eslint error
- JS foundation -- regular expression
- Tp5.1 login configuration method of whether to login public functions (complete login case)
- Tp5.1 paging (with parameter transfer)
- Clothing ERP | ten advantages of clothing ERP for enterprises
- C: wechat chat software instance (wpf+websocket+webapi+entityframework)
- Daily three questions 7.19
- Use of stm32cubemonitor Part II - historical data storage and network access
- Sequence diagram of UML diagram series
猜你喜欢

Simulation Implementation of string function (Part 1)

Pagoda workman WSS reverse proxy socket legal domain name applet chat remove port

Flink's study notes

Rotating frame target detection mmrotate v0.3.1 training hrsc2016 data set (II)

Dynamic programming -- Digital DP

Riotboard development board series notes (VIII) -- building desktop system

Wechat sports field reservation of the finished works of the applet graduation project (7) mid-term inspection report

JS written test question -- realize the flat function of array

Jenkins plug-in development -- plug-in expansion

Is redis'module'not an internal or external command?
随机推荐
JS construction linked list
[Kali's sshd service is enabled]
Wechat sports field reservation of the finished works of the applet graduation project (6) opening defense ppt
Arduino + si5351 square wave generator
Domain driven model (DDD)
JS written test questions -- random numbers, array de duplication
Is redis'module'not an internal or external command?
It7259q-13, it7259ex-24 feature wearable devices
Go language path is relative, but relative Import paths are not supported in module mode
Stm32cubemx quadrature encoder
Banana pie bpi-m5 toss record (2) -- compile u-boot
JS foundation -- regular expression
Download the jar package of jsqlparser and PageHelper
JS foundation -- data
Three ways to solve your performance management problems
Resolved (the latest version of selenium reported an error) attributeerror: module 'selenium webdriver‘ has no attribute ‘PhantomJS‘
Time formatting
Keil compile download error: no algorithm found for: 08000000h - 08001233h solution
IO (1) -io layering
Vscode configuration, eslint+prettier combined with detailed configuration steps, standardized development