基本介绍

Apache shardingsphere 子项目

  • 基于quartz 定时任务框架为基础的，因此具备quartz的大部分功能
  • 使用zookeeper做协调，调度中心，更加轻量级
  • 支持任务的分片
  • 支持弹性扩容 ， 可以水平扩展 ， 当任务再次运行时，会检查当前的服务器数量，重新分片，分片结束之后才会继续执行任务
  • 失效转移，容错处理，当一台调度服务器宕机或者跟zookeeper断开连接之后，会立即停止作业，然后再去寻找其他空闲的调度服务器，来运行剩余的任务
  • 提供运维界面，可以管理作业和注册中心

优秀的总体介绍文章：
https://www.infoq.cn/article/zcesh20kucb9qp1o1pnt
https://www.infoq.cn/article/2016/09/mesos-elastic-job-cloud

项目地址

当前版本 3.0.0 最新刚发布
https://shardingsphere.apache.org/elasticjob/index_zh.html
分为两个子项目 ElasticJob-Lite 和 ElasticJob-Cloud

项目演示

elasticjob-lite演示

1.启动zookeeper
2.开发项目 引入elastic-job-lite依赖

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<dependency>
    <groupId>org.apache.shardingsphere.elasticjob</groupId>
    <artifactId>elasticjob-lite-spring-boot-starter</artifactId>
    <version>${elasticjoblite.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.shardingsphere.elasticjob</groupId>
    <artifactId>elasticjob-error-handler-dingtalk</artifactId>
    <version>${elasticjoblite.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.shardingsphere.elasticjob</groupId>
    <artifactId>elasticjob-error-handler-wechat</artifactId>
    <version>${elasticjoblite.version}</version>
</dependency>

3.开发一些列job并配置到application.properties上

1
2
3
4
5
6
7
8
9
@Component
public class SpringSimpleJob implements SimpleJob {

    @Override
    public void execute(ShardingContext shardingContext) {
        System.out.println("run simple job");
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
@Component 
public class SpringDataflowJob implements DataflowJob {
//    @Resource 
//    private FooRepository fooRepository;

@Override
public List<Integer> fetchData(final ShardingContext shardingContext) {
    System.out.printf("Item: %s | Time: %s | Thread: %s | %s%n",
            shardingContext.getShardingItem(), new SimpleDateFormat("HH:mm:ss").format(new Date()), Thread.currentThread().getId(), "DATAFLOW FETCH");
    Integer[] myArray = { 1, 2, 3 };
    return Arrays.asList(myArray);
}

@Override
public void processData(final ShardingContext shardingContext, final List<Integer> data) {
    System.out.printf("Item: %s | Time: %s | Thread: %s | %s%n",
            shardingContext.getShardingItem(), new SimpleDateFormat("HH:mm:ss").format(new Date()), Thread.currentThread().getId(), "DATAFLOW PROCESS");
    for (Integer each : data) {
        System.out.println("get Item:" + each);
    }
}

}

3.使用打包编译的elasticjob-lite-ui.tar.gz 执行 bin/start.sh 启动管理控制台

4.执行结果如下

屏幕快照 2021-02-24 下午2.01.17.png

5.可以修改配置，修改分片 并且失效转移
屏幕快照 2021-02-24 下午2.57.42.png

分片修改了，可以看到执行也变了，分片数也对应增加或减少了

elasticjob cloud 3.0.0 RC1 cloud-example-demo

1-启动zookeeper

1
docker run --name some-zookeeper -p 2181:2181 --restart always -d zookeeper

2-启动mesos/agent
安装mesos框架 先安装homebrew brew install mesos
下面是官方二进制启动方式

The RPM installation creates the directory /var/lib/mesos that can be used as a work directory.
Start the Mesos master with the following command:

1
$ mesos-master --work_dir=/var/lib/mesos

On a different terminal, start the Mesos agent, and associate it with the Mesos master started above:

1
$ mesos-agent --work_dir=/var/lib/mesos --master=127.0.0.1:5050

我这边路径换为/usr/local/lib/mesos

1
mesos-master --work_dir=/usr/local/lib/mesos --zk=zk://127.0.0.1:2181/mesos --quorum=1 --ip=127.0.0.1
1
mesos-slave --work_dir=/usr/local/lib/mesos --master=127.0.0.1:5050 --ip=127.0.0.1

注册demo作业的快捷命令:
0-重要
先登录
curl -H "Content-Type: application/json" -X POST http://localhost:8899/api/login -d '{"username": "root", "password": "pwd"}'
{"accessToken":"eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0="}
使用返回的accessToken来做鉴权操作

  1. 注册APP:
    curl -l -H "Content-type: application/json" -H "accessToken:eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=" -X POST -d '{"appName":"exampleApp","appURL":"``[http://localhost:8001/static/](http://localhost:8001/static/elasticjob-example-cloud-2.1.4.tar.gz)``elasticjob-example-cloud-3.0.0-RC1.tar.gz","cpuCount":0.1,"memoryMB":64.0,"bootstrapScript":"bin/start.sh","appCacheEnable":true}' http://localhost:8899/api/app

返回true

  1. Java启动方式作业注册:

curl -l -H "Content-type: application/json" -H "accessToken:``eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=``" -X POST -d '{"jobName":"test_job_simple","appName":"exampleApp","jobExecutionType":"TRANSIENT","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.simple.JavaSimpleJob","cron":"0/10 * * * * ?","shardingTotalCount":1,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register

curl -l -H "Content-type: application/json" -H "accessToken:``eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=``" -X POST -d '{"jobName":"test_job_simple","appName":"exampleApp","jobExecutionType":"TRANSIENT","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.simple.JavaSimpleJob","cron":"0/10 * * * * ?","shardingTotalCount":1,"cpuCount":0.1,"memoryMB":64.0,"props":}' http://localhost:8899/api/job/register

curl -l -H "Content-type: application/json" -H "accessToken:eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=" -X POST -d '{"jobName":"test_job_dataflow","appName":"exampleApp","jobExecutionType":"DAEMON","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.dataflow.JavaDataflowJob","cron":"0/10 * * * * ?","shardingTotalCount":3,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register

curl -l -H "Content-type: application/json" -H "accessToken:eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=" -X POST -d '{"jobName":"test_job_script","appName":"exampleApp","jobExecutionType":"TRANSIENT","cron":"0/10 * * * * ?","shardingTotalCount":3,"cpuCount":0.1,"memoryMB":64.0, scriptCommandLine="script/demo.sh"}' http://localhost:8899/api/job/register

  1. Spring启动方式作业注册:

curl -l -H "Content-type: application/json" ``-H "accessToken:eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=" ``-X POST -d '{"jobName":"test_job_simple_spring","appName":"exampleApp","jobExecutionType":"TRANSIENT","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.simple.SpringSimpleJob","beanName":"springSimpleJob","applicationContext":"classpath:META-INF/applicationContext.xml","cron":"0/10 * * * * ?","shardingTotalCount":1,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register

curl -l -H "Content-type: application/json" -X POST -d '{"jobName":"test_job_dataflow_spring","appName":"exampleApp","jobExecutionType":"DAEMON","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.dataflow.SpringDataflowJob","beanName":"springDataflowJob","applicationContext":"classpath:META-INF/applicationContext.xml","cron":"0/10 * * * * ?","shardingTotalCount":3,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register

3- 执行效果
因为我的app注册 JobBootstrap.execute(script) 所以我注册方式得使用script方法

1
curl -l -H "Content-type: application/json"  -H "accessToken:eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=" -X POST -d '{"jobName":"demo","appName":"exampleApp","jobExecutionType":"TRANSIENT","jobType":"Simple","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.simple.JavaSimpleJob","cron":"0/10 * * * * ?","shardingTotalCount":1,"cpuCount":0.1,"memoryMB":64.0, "props":{"script.command.line":"script/demo.sh"}}' http://localhost:8899/api/job/register

可以看到登陆进 localhost:5050 mesos framework 沙箱容器里面看执行了到了结果

屏幕快照 2021-02-22 下午2.35.30.png

源码解析

2-x版本
http://www.iocoder.cn/Elastic-Job/cloud-job-scheduler-and-executor-first/?vip