当前位置:网站首页>Use reflection to convert RDD to dataframe
Use reflection to convert RDD to dataframe
2022-07-25 03:01:00 【Rookies also have dreams】
Java:
package cn.spark.sql;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SQLContext;
import java.util.List;
public class RDD2DataFrameReflection {
public static void main(String[] args){
SparkConf conf = new SparkConf().setAppName("RDD2DataFrameReflection")
.setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);
SQLContext sqlContext = new SQLContext(sc);
JavaRDD<String> lines = sc.textFile("C://Users//Desktop//students.txt");
JavaRDD<Student> students = lines.map(
new Function<String, Student>() {
private static final long serialVersionUID = 1L;
@Override
public Student call(String s) throws Exception {
String[] lineSplit = s.split(",");
Student stu = new Student();
stu.setID(Integer.valueOf(lineSplit[0].trim()));
stu.setName(lineSplit[1]);
stu.setAge(Integer.valueOf(lineSplit[2].trim()));
return stu;
}
});
/*
* Use reflection take RDD Convert to DataFrame
* take Student.class Put it in In fact, it is created by reflection DataFrame
* Because the bottom layer passes Student class On reflection To get some of the field
* requirement javabean We need to implement serializable Interface
* */
DataFrame studentDF = sqlContext.createDataFrame(
students,Student.class);
// To get a DataFrame after You can register a temporary table
studentDF.registerTempTable("students");
DataFrame teengerDF = sqlContext.sql("select * from students where age <= 18");
// Will find out DataFrame Convert again to RDD
JavaRDD<Row> teengerRDD = teengerDF.javaRDD();
// take EDD Data in mapping It maps to student
JavaRDD<Student> teengerStudentRDD = teengerRDD.map(new Function<Row, Student>() {
@Override
public Student call(Row row) throws Exception {
Student stu = new Student();
stu.setAge(row.getInt(0));
stu.setID(row.getInt(1));
stu.setName(row.getString(2));
return stu;
}
});
List<Student> studentList = teengerStudentRDD.collect();
for (Student stu : studentList){
System.out.println(stu);
}
}
}
test :

边栏推荐
- Mid year summary and personal feelings
- How to take the mold for the picture of 1.54 inch TFT st7789 LCD screen
- Pypi counts the number of Downloads
- Tp5.1 login configuration method of whether to login public functions (complete login case)
- Classic network learning RESNET code implementation
- [jailhouse article] scheduling policies and system software architectures for mixed criticality
- "Introduction to interface testing" punch in to learn day09: Micro service interface: how to use mock to solve chaotic call relationships
- Uni app configuration
- Selenium framework operation steelth.min.js file hides browser fingerprint features
- Go common standard library -time
猜你喜欢

Read and upgrade st-link chip information and SWD burning media through STM32 stlink utility tool

Tp5.1 include include files (reference public files)

JS foundation -- hijacking of this keyword

JS foundation -- object static method

Learning record XIII

Ctfshow misc introduction

Request and response

Banana pie bpi-m5 toss record (2) -- compile u-boot

Tp5.1 initialize initialization method (not \u initialize)

Is redis'module'not an internal or external command?
随机推荐
Domestic edge computing organization and product research
Learning Record V
JS written test question -- deep copy of object
Routing policy interferes with routing
Flume's study notes
What should I do when the interface encounters jsonstring
Dynamic planning of force buckle punch in summary
Go multiplexing
Resolved (the latest version of selenium reported an error) attributeerror: module 'selenium webdriver‘ has no attribute ‘PhantomJS‘
PHP record
Wechat sports field reservation of the finished works of the applet graduation project (6) opening defense ppt
Define macros in makefile and pass them to source code
JS written test question -- browser kernel
Color space (1) - RGB
"Introduction to interface testing" punch in to learn day07: websocket interface: how to test a completely unfamiliar protocol interface?
Bgy development small example
Keil compile download error: no algorithm found for: 08000000h - 08001233h solution
Object.defineproperty use
Hyperchain hyperblockchain Shi Xingguo was interviewed by 36 krypton: the amount of customer cooperation consulting is doubling
Pagoda workman WSS reverse proxy socket legal domain name applet chat remove port