基本介绍
Apache shardingsphere 子项目
- 基于quartz 定时任务框架为基础的,因此具备quartz的大部分功能
- 使用zookeeper做协调,调度中心,更加轻量级
- 支持任务的分片
- 支持弹性扩容 , 可以水平扩展 , 当任务再次运行时,会检查当前的服务器数量,重新分片,分片结束之后才会继续执行任务
- 失效转移,容错处理,当一台调度服务器宕机或者跟zookeeper断开连接之后,会立即停止作业,然后再去寻找其他空闲的调度服务器,来运行剩余的任务
- 提供运维界面,可以管理作业和注册中心
优秀的总体介绍文章:
https://www.infoq.cn/article/zcesh20kucb9qp1o1pnt
https://www.infoq.cn/article/2016/09/mesos-elastic-job-cloud
项目地址
当前版本 3.0.0 最新刚发布
https://shardingsphere.apache.org/elasticjob/index_zh.html
分为两个子项目 ElasticJob-Lite 和 ElasticJob-Cloud
项目演示
elasticjob-lite演示
1.启动zookeeper
2.开发项目 引入elastic-job-lite依赖
1 | <dependency> |
3.开发一些列job并配置到application.properties上
1 | @Component |
1 |
|
3.使用打包编译的elasticjob-lite-ui.tar.gz 执行 bin/start.sh 启动管理控制台
4.执行结果如下
5.可以修改配置,修改分片 并且失效转移
分片修改了,可以看到执行也变了,分片数也对应增加或减少了
elasticjob cloud 3.0.0 RC1 cloud-example-demo
1-启动zookeeper
1 | docker run --name some-zookeeper -p 2181:2181 --restart always -d zookeeper |
2-启动mesos/agent
安装mesos框架 先安装homebrew brew install mesos
下面是官方二进制启动方式
The RPM installation creates the directory /var/lib/mesos
that can be used as a work directory.
Start the Mesos master with the following command:
1 | $ mesos-master --work_dir=/var/lib/mesos |
On a different terminal, start the Mesos agent, and associate it with the Mesos master started above:
1 | $ mesos-agent --work_dir=/var/lib/mesos --master=127.0.0.1:5050 |
我这边路径换为/usr/local/lib/mesos
1 | mesos-master --work_dir=/usr/local/lib/mesos --zk=zk://127.0.0.1:2181/mesos --quorum=1 --ip=127.0.0.1 |
1 | mesos-slave --work_dir=/usr/local/lib/mesos --master=127.0.0.1:5050 --ip=127.0.0.1 |
注册demo作业的快捷命令:
0-重要
先登录curl -H "Content-Type: application/json" -X POST http://localhost:8899/api/login -d '{"username": "root", "password": "pwd"}'
{"accessToken":"eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0="}
使用返回的accessToken来做鉴权操作
- 注册APP:
curl -l -H "Content-type: application/json" -H "accessToken:eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=" -X POST -d '{"appName":"exampleApp","appURL":"``[http://localhost:8001/static/](http://localhost:8001/static/elasticjob-example-cloud-2.1.4.tar.gz)``elasticjob-example-cloud-3.0.0-RC1.tar.gz","cpuCount":0.1,"memoryMB":64.0,"bootstrapScript":"bin/start.sh","appCacheEnable":true}' http://localhost:8899/api/app
返回true
- Java启动方式作业注册:
curl -l -H "Content-type: application/json" -H "accessToken:``eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=``" -X POST -d '{"jobName":"test_job_simple","appName":"exampleApp","jobExecutionType":"TRANSIENT","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.simple.JavaSimpleJob","cron":"0/10 * * * * ?","shardingTotalCount":1,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register
curl -l -H "Content-type: application/json" -H "accessToken:``eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=``" -X POST -d '{"jobName":"test_job_simple","appName":"exampleApp","jobExecutionType":"TRANSIENT","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.simple.JavaSimpleJob","cron":"0/10 * * * * ?","shardingTotalCount":1,"cpuCount":0.1,"memoryMB":64.0,"props":}' http://localhost:8899/api/job/register
curl -l -H "Content-type: application/json" -H "accessToken:eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=" -X POST -d '{"jobName":"test_job_dataflow","appName":"exampleApp","jobExecutionType":"DAEMON","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.dataflow.JavaDataflowJob","cron":"0/10 * * * * ?","shardingTotalCount":3,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register
curl -l -H "Content-type: application/json" -H "accessToken:eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=" -X POST -d '{"jobName":"test_job_script","appName":"exampleApp","jobExecutionType":"TRANSIENT","cron":"0/10 * * * * ?","shardingTotalCount":3,"cpuCount":0.1,"memoryMB":64.0, scriptCommandLine="script/demo.sh"}' http://localhost:8899/api/job/register
- Spring启动方式作业注册:
curl -l -H "Content-type: application/json" ``-H "accessToken:eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=" ``-X POST -d '{"jobName":"test_job_simple_spring","appName":"exampleApp","jobExecutionType":"TRANSIENT","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.simple.SpringSimpleJob","beanName":"springSimpleJob","applicationContext":"classpath:META-INF/applicationContext.xml","cron":"0/10 * * * * ?","shardingTotalCount":1,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register
curl -l -H "Content-type: application/json" -X POST -d '{"jobName":"test_job_dataflow_spring","appName":"exampleApp","jobExecutionType":"DAEMON","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.dataflow.SpringDataflowJob","beanName":"springDataflowJob","applicationContext":"classpath:META-INF/applicationContext.xml","cron":"0/10 * * * * ?","shardingTotalCount":3,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register
3- 执行效果
因为我的app注册 JobBootstrap.execute(script) 所以我注册方式得使用script方法
1 | curl -l -H "Content-type: application/json" -H "accessToken:eyJhdXRoVXNlcm5hbWUiOiJyb290IiwiYXV0aFBhc3N3b3JkIjoicHdkIn0=" -X POST -d '{"jobName":"demo","appName":"exampleApp","jobExecutionType":"TRANSIENT","jobType":"Simple","jobClass":"org.apache.shardingsphere.elasticjob.lite.example.job.simple.JavaSimpleJob","cron":"0/10 * * * * ?","shardingTotalCount":1,"cpuCount":0.1,"memoryMB":64.0, "props":{"script.command.line":"script/demo.sh"}}' http://localhost:8899/api/job/register |
可以看到登陆进 localhost:5050 mesos framework 沙箱容器里面看执行了到了结果
源码解析
2-x版本
http://www.iocoder.cn/Elastic-Job/cloud-job-scheduler-and-executor-first/?vip