mirror of
https://gitee.com/jd-platform-opensource/asyncTool.git
synced 2024-12-31 15:35:34 +08:00
Merge remote-tracking branch 'origin/master'
This commit is contained in:
wuweifeng10
commit
112e88dca6
175
README.en.md
175
README.en.md
@ -1,36 +1,139 @@
|
||||
# asyncTool
|
||||
|
||||
#### Description
|
||||
解决任意的多线程并行、串行、阻塞、依赖、回调的并发框架，可以任意组合各线程的执行顺序，还带全链路回调。
|
||||
|
||||
#### Software Architecture
|
||||
Software architecture description
|
||||
|
||||
#### Installation
|
||||
|
||||
1. xxxx
|
||||
2. xxxx
|
||||
3. xxxx
|
||||
|
||||
#### Instructions
|
||||
|
||||
1. xxxx
|
||||
2. xxxx
|
||||
3. xxxx
|
||||
|
||||
#### Contribution
|
||||
|
||||
1. Fork the repository
|
||||
2. Create Feat_xxx branch
|
||||
3. Commit your code
|
||||
4. Create Pull Request
|
||||
|
||||
|
||||
#### Gitee Feature
|
||||
|
||||
1. You can use Readme\_XXX.md to support different languages, such as Readme\_en.md, Readme\_zh.md
|
||||
2. Gitee blog [blog.gitee.com](https://blog.gitee.com)
|
||||
3. Explore open source project [https://gitee.com/explore](https://gitee.com/explore)
|
||||
4. The most valuable open source project [GVP](https://gitee.com/gvp)
|
||||
5. The manual of Gitee [https://gitee.com/help](https://gitee.com/help)
|
||||
6. The most popular members [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/)
|
||||
##Concurrent framework description
|
||||
|
||||
If you have any questions, you can send an email to the author, or you can have specific scenario requirements. Thank you for your comments. wuweifeng10@jd.com
|
||||
|
||||
|
||||
|
||||
If you are interested in blockchain, please refer to another [GVP project] (https://gitee.com/tianalei/md_blockchain) of the author, Java blockchain low level introduction
|
||||
|
||||
|
||||
|
||||
If you just need to use this frame, please look down. If you need to have a deep understanding of how this framework is implemented step by step, from receiving the requirements to thinking about each step, why each class is so designed, why there are these methods, that is, how to develop this framework from 0 to 1, I opened a column in [CSDN] (https://blog.csdn.net/tianaleixiaowu/category_. HTML) to talk about how middleware is developed from 0, including and Not limited to this small frame. JD internal colleagues can search my ERP on CF and see it.
|
||||
|
||||
|
||||
|
||||
####To do
|
||||
|
||||
Recently, many students have reported that they want to support dynamic task arrangement based on file configuration. Considering this, we need to add an additional plug-in function. At present, the function does not need to be changed. However, the format of the file, how to accurately express the sequence, parallel sequence and dependency, is relatively complex. If there are related needs and interests, we can study it.
|
||||
|
||||
|
||||
|
||||
####Concurrent common scenarios
|
||||
|
||||
1. The client requests the server interface, which needs to call the interfaces of other n microservices. For example, to request my order, you need to call the user's RPC, product details RPC, inventory RPC, coupons and many other services. At the same time, these services also have interdependence, for example, you must first get a user's field, and then go to an RPC service to request data. After all the results are finally obtained or timeout, the result will be summarized and returned to the client.
|
||||
|
||||
|
||||
|
||||
2 many tasks in the form of Workflow
|
||||
|
||||
|
||||
|
||||
3. Reptiles and so on. They are dependent on each other
|
||||
|
||||
|
||||
|
||||
####Possible requirements for concurrent scenarios -- arbitrary arrangement
|
||||
|
||||
1 serial request of multiple execution units
|
||||
|
||||
|
||||
|
||||
! [enter picture description] (https://images.gitee.com/uploads/images/2019/1226/092905. PNG "screenshot. PNG")
|
||||
|
||||
|
||||
|
||||
2 parallel requests of multiple execution units
|
||||
|
||||
|
||||
|
||||
! [enter picture description] (https://images.gitee.com/uploads/images/2019/1226/092925 c01a5_. PNG "screenshot. PNG")
|
||||
|
||||
|
||||
|
||||
3 block wait, serial followed by multiple parallels
|
||||
|
||||
|
||||
|
||||
! [enter picture description] (https://images.gitee.com/uploads/images/2019/1226/092935 "[5babe488" [303698. PNG "screenshot. PNG")
|
||||
|
||||
|
||||
|
||||
4 block wait, execute a certain one after multiple parallel execution
|
||||
|
||||
|
||||
|
||||
! [enter picture description] (https://images.gitee.com/uploads/images/2019/1226/092952. PNG "screenshot. PNG")
|
||||
|
||||
|
||||
|
||||
5 series parallel interdependence
|
||||
|
||||
|
||||
|
||||
! [enter picture description] (https://images.gitee.com/uploads/images/2019/1226/093006 cd133c. PNG "screenshot. PNG")
|
||||
|
||||
|
||||
|
||||
6 complex scene
|
||||
|
||||
|
||||
|
||||
! [enter picture description] (https://images.gitee.com/uploads/images/2019/1226/093023 "screenshot. PNG")
|
||||
|
||||
|
||||
|
||||
####Possible requirements for concurrent scenarios -- callbacks for each execution result
|
||||
|
||||
Traditional future and completeablefuture can complete task choreography to some extent, and transfer the results to the next task. For example, completabilefuture has the then method, but it can't do the callback for each execution unit. For example, if the execution of a is successful, followed by B, I hope that a will have a callback result after the execution, so that I can monitor the current execution status, or log something. Failed, I can also record an exception information or something.
|
||||
|
||||
|
||||
|
||||
At this time, the traditional can do nothing.
|
||||
|
||||
|
||||
|
||||
My framework provides such a callback function. In addition, if the execution fails or times out, the default value can be set when defining the execution unit.
|
||||
|
||||
|
||||
|
||||
####The possible requirements of concurrent scenarios -- strong and weak dependence on execution order
|
||||
|
||||
As shown in Figure 3, a and B execute concurrently, and finally C.
|
||||
|
||||
|
||||
|
||||
In some scenarios, we want to execute C only after a and B have finished executing. There is an allow (futures...). Then () method in completable future.
|
||||
|
||||
|
||||
|
||||
In some scenarios, we want either a or B to finish execution, and then execute C. There is an anyof (futures...). Then () method in completable future.
|
||||
|
||||
|
||||
|
||||
My framework also provides similar functions. When setting the adddepend dependency in the wrapper, you can specify whether the dependent task must be completed. If you depend on the execution of the must, you must wait for all the must dependencies to be completed before executing yourself.
|
||||
|
||||
|
||||
|
||||
If you don't rely on must, you can execute any dependency and then you can execute yourself.
|
||||
|
||||
|
||||
|
||||
####Possible requirements for concurrent scenarios -- relying on upstream execution results as input parameters
|
||||
|
||||
For example, for the three execution units of a-b-c, the input parameter of a is string, and the output parameter is int. for B, it needs to use the result of a as its input parameter. In other words, a and B are not independent but result dependent.
|
||||
|
||||
|
||||
|
||||
Before a finishes executing, B can't get the result. It just knows the result type of A.
|
||||
|
||||
|
||||
|
||||
So, my framework also supports such scenarios. You can take the result wrapper class of a as the input parameter of B when arranging. Although it is not implemented at this time, it must be empty, but it can ensure that after the implementation of a, the participation of B will be assigned.
|
||||
|
||||
|
||||
|
||||
####Possible requirements for concurrent scenarios -- timeout of the whole group of tasks
|
||||
|
||||
For a group of tasks, although the time of each internal execution unit is not controllable, I can control that the execution time of the whole group does not exceed a certain value. Control the execution threshold of the whole group by setting timeout.
|
||||
|
||||
|
||||
|
||||
35
README.md
35
README.md
@ -1,18 +1,33 @@
|
||||
# 并发框架说明
|
||||
# 并行框架说明
|
||||
有问题可以给作者发邮件说明，或者有特定的场景需求，感谢您的意见。wuweifeng10@jd.com
|
||||
|
||||
有对区块链感兴趣的，可以参考作者另一个[GVP项目](https://gitee.com/tianyalei/md_blockchain)，java区块链底层入门
|
||||
该框架目前正在 **京东App后台** 接受苛刻、高并发、海量用户等复杂场景业务的检验测试，随时会根据实际情况发布更新和bugFix。
|
||||
|
||||
有对区块链感兴趣的，可以参考作者另一个[GVP项目](https://gitee.com/tianyalei/md_blockchain)，java区块链底层入门。
|
||||
|
||||
如果只是需要用这个框架，请往下看即可。如果需要深入了解这个框架是如何一步一步实现的，从接到需求，到每一步的思考，每个类为什么这么设计，为什么有这些方法，也就是如何从0到1开发出这个框架，我在[csdn开了专栏](https://blog.csdn.net/tianyaleixiaowu/category_9637010.html)专门讲中间件如何从0开发，包括并不限于这个小框架。京东内部同事可在cf上搜索我erp也能看到。
|
||||
|
||||
#### 并发常见的场景
|
||||
1 客户端请求服务端接口，该接口需要调用其他N个微服务的接口。譬如 请求我的订单，那么就需要去调用用户的rpc、商品详情的rpc、库存rpc、优惠券等等好多个服务。同时，这些服务还有相互依赖关系，譬如必须先拿到用户的某个字段后，再去某rpc服务请求数据。 最终全部获取完毕后，或超时了，就汇总结果，返回给客户端。
|
||||
#### 并行常见的场景
|
||||
1 客户端请求服务端接口，该接口需要调用其他N个微服务的接口
|
||||
|
||||
2 工作流式的很多个任务
|
||||
`譬如 请求我的订单，那么就需要去调用用户的rpc、商品详情的rpc、库存rpc、优惠券等等好多个服务。同时，这些服务还有相互依赖关系，譬如必须先拿到用户的某个字段后，再去某rpc服务请求数据。 最终全部获取完毕后，或超时了，就汇总结果，返回给客户端。`
|
||||
|
||||
3 爬虫之类的，有前后依赖关系
|
||||
2 并行执行N个任务，后续根据这1-N个任务的执行结果来决定是否继续执行下一个任务
|
||||
|
||||
#### 并发场景可能存在的需求之——任意编排
|
||||
`如用户可以通过邮箱、手机号、用户名登录，登录接口只有一个，那么当用户发起登录请求后，我们需要并行根据邮箱、手机号、用户名来同时查数据库，只要有一个成功了，都算成功，就可以继续执行下一步。而不是先试邮箱能否成功、再试手机号……`
|
||||
|
||||
|
||||
`再如某个接口，有5个前置任务需要处理。其中有3个是必须要执行完毕才能执行后续的，另外2个是无所谓的，只要这3个执行完就可以进行下一步，到时另外2个如果成功了就有值，如果还没执行完，就是默认值。`
|
||||
|
||||
3 需要进行线程隔离的多批次任务
|
||||
|
||||
`如多组任务， 各组任务之间彼此不相关，每组都需要一个独立的线程池，每组都是独立的一套执行单元的组合。有点类似于hystrix的线程池隔离策略。`
|
||||
|
||||
4 单机工作流任务编排
|
||||
|
||||
5 爬虫相关，有前后依赖关系
|
||||
|
||||
#### 并行场景之核心——任意编排
|
||||
1 多个执行单元的串行请求
|
||||
|
||||
|
||||
@ -37,14 +52,14 @@
|
||||
|
||||
|
||||
|
||||
#### 并发场景可能存在的需求之——每个执行结果的回调
|
||||
#### 并行场景可能存在的需求之——每个执行结果的回调
|
||||
传统的Future、CompleteableFuture一定程度上可以完成任务编排，并可以把结果传递到下一个任务。如CompletableFuture有then方法，但是却无法做到对每一个执行单元的回调。譬如A执行完毕成功了，后面是B，我希望A在执行完后就有个回调结果，方便我监控当前的执行状况，或者打个日志什么的。失败了，我也可以记录个异常信息什么的。
|
||||
|
||||
此时，传统的就无能为力了。
|
||||
|
||||
我的框架提供了这样的回调功能。并且，如果执行失败、超时，可以在定义这个执行单元时就设定默认值。
|
||||
|
||||
#### 并发场景可能存在的需求之——执行顺序的强依赖和弱依赖
|
||||
#### 并行场景可能存在的需求之——执行顺序的强依赖和弱依赖
|
||||
如上图的3，A和B并发执行，最后是C。
|
||||
|
||||
有些场景下，我们希望A和B都执行完毕后，才能执行C，CompletableFuture里有个allOf(futures...).then()方法可以做到。
|
||||
@ -70,7 +85,7 @@
|
||||
|
||||
创建线程量少。
|
||||
如这样的，A会运行在B、C执行更慢的那个单元的线程上，而不会额外创建线程。
|
||||
#### async-Tool介绍
|
||||
#### asyncTool介绍
|
||||
解决任意的多线程并行、串行、阻塞、依赖、回调的并发框架，可以任意组合各线程的执行顺序，还带全链路回调和超时控制。
|
||||
|
||||
其中的A、B、C分别是一个最小执行单元（worker），可以是一段耗时代码、一次Rpc调用等，不局限于你做什么。
|
||||
|
||||
Loading…
Reference in New Issue
Block a user