[ISSUE #135] Add gRPC connector on LiteSimpleConsumer with downstream ack and sub-topic throttling#136
Open
lizhimins wants to merge 1 commit into
Open
[ISSUE #135] Add gRPC connector on LiteSimpleConsumer with downstream ack and sub-topic throttling#136lizhimins wants to merge 1 commit into
lizhimins wants to merge 1 commit into
Conversation
…h downstream ack and sub-topic throttling Add the flink-connector-rocketmq-grpc module built on rocketmq-client-java 5.2.1+ LiteSimpleConsumer (Pop model): - Source binds one main lite topic with a wildcard subscription and never acks; each subtask shares one consumer across fetch-concurrency long-poll workers. - Emits AckableMessage<OUT> carrying a serializable, credential-free RocketMQReceiptHandle that can cross shuffle boundaries. - Downstream ack via a per-TaskManager reference-counted client pool, with two wirings: RocketMQAckProcessFunction (direct ack) and MessageThrottlePolicy + RocketMQThrottleProcessFunction (policy-based fair throttling with DLQ safety valves). - Optional invisible-duration renewal prevents premature redelivery under backpressure. - gRPC producer sink and rocketmq-grpc SQL connector; design documented in docs/grpc-connector.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #135
内容
新增独立模块
flink-connector-rocketmq-grpc(及 SQL fat-jarflink-sql-connector-rocketmq-grpc),基于rocketmq-client-java5.2.1+ 的LiteSimpleConsumer(Pop 模型):Source
bindTopic主 lite topic 泛化订阅,broker 做消息级负载均衡,不做 split 分区;fetch-concurrency个并发receive()长轮询线程,接收失败指数退避;AckableMessage<OUT>(payload + 可序列化、免凭据的RocketMQReceiptHandle),配套专用 TypeInformation/TypeSerializer(不走 Kryo)。下游 ack / 限流(DataStream-only)
RocketMQLiteAckClient:每 TM 引用计数池化,按 handle 的路由三元组 (endpoint, namespace, consumerGroup) 维护同组 consumer 池;ack RPC 有界指数退避重试,INVALID_RECEIPT_HANDLE容忍为 warn(消息重投,at-least-once);RocketMQAckProcessFunction直接ack()/changeInvisibleDuration(),或MessageThrottlePolicy<T>+RocketMQThrottleProcessFunction<T>(内置 maxDeliveryAttempt=16、maxInvisibleDuration=30min 安全阀);CredentialsResolver(TM 本地按 endpoint 解析);@Internal适配类RocketMQReceiptHandleCodec。其它
rocketmq-grpcSQL connector;docs/grpc-connector.md:动机、架构、语义、决策与被否决的备选方案。验证
mvn spotless:check clean package(JDK 11/17)全部 BUILD SUCCESS。What
Adds a separate module
flink-connector-rocketmq-grpc(plus the SQL fat-jarflink-sql-connector-rocketmq-grpc) built onrocketmq-client-java5.2.1+LiteSimpleConsumer(Pop model):Source
fetch-concurrencyconcurrentreceive()long-poll workers; receive failures back off exponentially.AckableMessage<OUT>— the payload plus a serializable, credential-freeRocketMQReceiptHandle— with dedicated TypeInformation/TypeSerializer (no Kryo fallback).Downstream ack / throttling (DataStream-only)
RocketMQLiteAckClient: per-TaskManager reference-counted pool keyed by the handle routing triple (endpoint, namespace, consumerGroup); ack RPCs retry with bounded exponential backoff, andINVALID_RECEIPT_HANDLEis tolerated as a warning (redelivery, at-least-once).RocketMQAckProcessFunctionand callack()/changeInvisibleDuration()directly, or implementMessageThrottlePolicy<T>withRocketMQThrottleProcessFunction<T>(built-in safety valves: maxDeliveryAttempt=16, maxInvisibleDuration=30min).CredentialsResolverresolved locally per TaskManager and endpoint.@Internaladapter,RocketMQReceiptHandleCodec.Other
rocketmq-grpcSQL connector.docs/grpc-connector.md: motivation, architecture, semantics, decisions and rejected alternatives.Verification
mvn spotless:check clean packagesucceeds on JDK 11 and 17.