上一篇：spark：wordcount
通用流程
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| Unsaved
+---------+ +--------------+ 4.反向注册
| | 1.任务提交 | <-----------------------+
| client +---------------> Driver | |
| | | | |
+---------+ | 初 始 化 sc | +--+----------+
| 任 务 划 分 | | |
| 任 务 调 度 | | Executor |
+------+-------+ | 执 行 任 务|
| | textFile |
| +------------------>+ flatMap |
| | 3.启动Executor | map |
2.注册应用| | | |
| | +-------------+
| |
| |
+------v--------+
| | +-------------+
| 资 源 管 理 器 +-------------------> |
| | | Executor |
+---------------+ | |
+-------------+
|
standalone运行模式
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
| Driver
+-------------------+ +---------------------+
| | 1.register | |
| Client +-----------------------> |
+------------------> | | |
| | +-------------+ | | Master |
| | | | | | |
| | | SparkContext | 7.destroy | |
| | +-------------+ +-----------------------> |
| +-^-----------------+ +-+---------------^---+
6.report task status | | | |
until finished | | | |
| | | 2.apply | | 3.report the status
| | | for | |
| 4.register | resource | |
| | | boot | |
| | | ExecutorBackend |
| | | | |
| | 5.distribute tasks | |
| | | | |
| | | | |
| | | | |
| +--+-----------v-------+ +-------v---------------+------+
| | | | |
+-----------------+ Worker | | Worker |
| | | |
| +-----------------+ | | +------------+ +----------+ |
| | | | | | | | | |
| | executorA | | | | executorB | |executorC | |
| +-----------------+ | | +------------+ +----------+ |
+----------------------+ +------------------------------+
|
下一篇：spark standalone模式的配置与运行