Skip to content

Commit 6ab4638

Browse files
authored
feat: implement advanced state API for Go and Python (#214)
- Go SDK: add state package with codec, keyed state, and structures - codec: bool/bytes/float/int/string/uint/json and ordered types - keyed: Value, List, Map, Aggregating, Reducing, PriorityQueue keyed states - structures: underlying implementations for each state type - Python API: refactor store into codec, keyed, structures submodules - keyed: KeyedValueState, KeyedListState, KeyedMapState, etc. - structures: state structures aligned with Go - Runtime: update fs_context to work with new state API Made-with: Cursor <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Adds new Go and Python “advanced state” libraries and updates the Go runtime context concurrency behavior, which could affect state access and build/runtime integration. Main risk is correctness/performance of the new state encodings (ordered codecs, merge/list packing) and any assumptions about thread-safety after removing locking. > > **Overview** > Introduces a new **Go advanced SDK** (`go-sdk-advanced`) providing typed state abstractions (`ValueState`, `ListState`, `MapState`, `PriorityQueueState`, `AggregatingState`, `ReducingState`) plus keyed variants via factories, all backed by the existing low-level `Store` and a new codec layer with ordered primitive codecs and `DefaultCodecFor`. > > Adds a new **Python advanced state package** (`functionstream-api-advanced`) with parallel codec/state/keyed APIs and packaging metadata, and wires builds/docs to recognize the new advanced modules (Makefile `PYTHONPATH` update, `.gitignore` build output, README/doc links, and new advanced-state API guides in EN/ZH). > > Also tweaks the Go SDK runtime context implementation by removing internal mutex guarding `closed`/`stores`, changing thread-safety expectations for `Emit`/`GetOrCreateStore`/`Close`. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit e93c4a4. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent f58af82 commit 6ab4638

81 files changed

Lines changed: 5944 additions & 56 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ examples/python-processor/build/**
2222
examples/python-processor/dependencies/**
2323
python/functionstream-client/src/fs_client/_proto/
2424
python/functionstream-api/build
25+
python/functionstream-api-advanced/build
26+
2527

2628

2729
# python Runtime - Build artifacts and intermediate files

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ build-lite: .check-env
8282
.build-wasm:
8383
$(call log,WASM,Building Python Runtime using $(PYTHON_EXEC))
8484
@cd $(PYTHON_ROOT)/functionstream-runtime && \
85-
PYTHONPATH=../functionstream-api ../../$(PYTHON_EXEC) build.py > /dev/null
85+
PYTHONPATH=../functionstream-api:../functionstream-api-advanced ../../$(PYTHON_EXEC) build.py > /dev/null
8686
@[ -f "$(WASM_SOURCE)" ] || (printf "$(C_R)[X] WASM Build Failed$(C_0)\n" && exit 1)
8787

8888
dist: build

README-zh.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,8 @@ function-stream-<version>/
206206
| [Function 任务配置规范](docs/function-configuration-zh.md) | 任务定义规范 |
207207
| [SQL CLI 交互式管理指南](docs/sql-cli-guide-zh.md) | 交互式管理指南 |
208208
| [Function 管理与开发指南](docs/function-development-zh.md) | 管理与开发指南 |
209-
| [Python SDK 开发与交互指南](docs/python-sdk-guide-zh.md) | Python SDK 指南 |
209+
| [Go SDK 开发与交互指南](docs/Go-SDK/go-sdk-guide-zh.md) | Go SDK 指南 |
210+
| [Python SDK 开发与交互指南](docs/Python-SDK/python-sdk-guide-zh.md) | Python SDK 指南 |
210211

211212
## 配置
212213

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -205,7 +205,8 @@ We provide a robust shell script to manage the server process, capable of handli
205205
| [Function Configuration](docs/function-configuration.md) | Task Definition Specification |
206206
| [SQL CLI Guide](docs/sql-cli-guide.md) | Interactive Management Guide |
207207
| [Function Development](docs/function-development.md) | Management & Development Guide |
208-
| [Python SDK Guide](docs/python-sdk-guide.md) | Python SDK Guide |
208+
| [Go SDK Guide](docs/Go-SDK/go-sdk-guide.md) | Go SDK Guide |
209+
| [Python SDK Guide](docs/Python-SDK/python-sdk-guide.md) | Python SDK Guide |
209210

210211
## Configuration
211212

docs/Go-SDK/go-sdk-advanced-state-api-zh.md

Lines changed: 321 additions & 0 deletions
Large diffs are not rendered by default.

docs/Go-SDK/go-sdk-advanced-state-api.md

Lines changed: 322 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -263,7 +263,7 @@ create function with (
263263
);
264264
```
265265

266-
config.yaml 中需配置 `name``type: processor``input-groups``outputs`(如 Kafka)。详见 [Function 配置](function-configuration-zh.md)[examples/go-processor/README.md](../examples/go-processor/README.md)
266+
config.yaml 中需配置 `name``type: processor``input-groups``outputs`(如 Kafka)。详见 [Function 配置](../function-configuration-zh.md)[examples/go-processor/README.md](../../examples/go-processor/README.md)
267267

268268
---
269269

@@ -301,7 +301,18 @@ if err != nil {
301301

302302
---
303303

304-
## 七、目录结构参考
304+
## 七、高级状态 API(进阶文档)
305+
306+
本指南仅覆盖**低阶 go-sdk**(Driver、Context、Store、目录结构)。**高级状态 API**(Codec、ValueState、ListState、MapState、PriorityQueueState、AggregatingState、ReducingState、Keyed\* 工厂与用法)由独立库 **go-sdk-advanced** 提供,完整说明、Codec 约定、构造函数表与示例均在进阶文档中:
307+
308+
- **[Go SDK — 高级状态 API](go-sdk-advanced-state-api-zh.md)**(中文)
309+
- [Go SDK — Advanced State API](go-sdk-advanced-state-api.md)(英文)
310+
311+
---
312+
313+
## 八、目录结构参考
314+
315+
**低阶库 go-sdk**
305316

306317
```text
307318
go-sdk/
@@ -317,8 +328,20 @@ go-sdk/
317328
│ ├── runtime.go
318329
│ ├── context.go
319330
│ └── store.go
331+
├── state/
332+
│ └── common/ # 公共辅助(Store 类型别名、DupBytes)
320333
├── wit/ # processor.wit 及依赖(可由 make wit 生成)
321334
└── bindings/ # wit-bindgen-go 生成的 Go 代码(make bindings)
322335
```
323336

324-
更多示例与 SQL 操作见 [examples/go-processor/README.md](../examples/go-processor/README.md)[SQL CLI 指南](sql-cli-guide-zh.md)
337+
**高阶库 go-sdk-advanced**(依赖 go-sdk,含 Codec 与全部状态类型):
338+
339+
```text
340+
go-sdk-advanced/
341+
├── go.mod # require go-sdk
342+
├── codec/ # Codec[T]、DefaultCodecFor、内置与 JSON codec
343+
├── structures/ # ValueState、ListState、MapState、PriorityQueue、Aggregating、Reducing
344+
└── keyed/ # Keyed 状态工厂(value、list、map、PQ、aggregating、reducing)
345+
```
346+
347+
更多示例与 SQL 操作见 [examples/go-processor/README.md](../../examples/go-processor/README.md)[SQL CLI 指南](../sql-cli-guide-zh.md)
Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -263,7 +263,7 @@ create function with (
263263
);
264264
```
265265

266-
Configure `name`, `type: processor`, `input-groups`, and `outputs` (e.g. Kafka) in config.yaml. See [Function Configuration](function-configuration.md) and [examples/go-processor/README.md](../examples/go-processor/README.md).
266+
Configure `name`, `type: processor`, `input-groups`, and `outputs` (e.g. Kafka) in config.yaml. See [Function Configuration](../function-configuration.md) and [examples/go-processor/README.md](../../examples/go-processor/README.md).
267267

268268
---
269269

@@ -301,7 +301,18 @@ if err != nil {
301301

302302
---
303303

304-
## 7. Directory Layout
304+
## 7. Advanced State API (see advanced doc)
305+
306+
This guide covers only the **low-level go-sdk** (Driver, Context, Store, directory layout). The **advanced state API** (Codec, ValueState, ListState, MapState, PriorityQueueState, AggregatingState, ReducingState, Keyed\* factories and usage) is provided by a separate library **go-sdk-advanced**. Full reference, codec contract, constructor tables, and examples are in the advanced document:
307+
308+
- **[Go SDK — Advanced State API](go-sdk-advanced-state-api.md)** (English)
309+
- [Go SDK — 高级状态 API](go-sdk-advanced-state-api-zh.md) (中文)
310+
311+
---
312+
313+
## 8. Directory Layout
314+
315+
**Low-level library (go-sdk):**
305316

306317
```text
307318
go-sdk/
@@ -317,8 +328,10 @@ go-sdk/
317328
│ ├── runtime.go
318329
│ ├── context.go
319330
│ └── store.go
331+
├── state/
332+
│ └── common/ # Shared helpers (Store type alias, DupBytes)
320333
├── wit/ # processor.wit and deps (make wit)
321334
└── bindings/ # Generated by wit-bindgen-go (make bindings)
322335
```
323336

324-
For more examples and SQL operations, see [examples/go-processor/README.md](../examples/go-processor/README.md) and the [SQL CLI Guide](sql-cli-guide.md).
337+
For more examples and SQL operations, see [examples/go-processor/README.md](../../examples/go-processor/README.md) and the [SQL CLI Guide](../sql-cli-guide.md).

0 commit comments

Comments
 (0)