Conversation
- Remove protobuf-lite dependency and integrate upb runtime. - Refactor CMakeLists.txt to support SPM_USE_UPB by default and clean source file lists. - Simplify python/setup.py to compile in UPB mode and link train library. - Add index bounds checks to upb wrappers for safety. TAG=agy CONV=e46efd90-ae25-4cfc-8385-5381e11ee2ec
- Fix member variable initialization order in SentencePieceTextWrapper - Remove unused variable sub_msg in NBestSentencePieceTextWrapper::add_nbests TAG=agy CONV=e46efd90-ae25-4cfc-8385-5381e11ee2ec
taku910
force-pushed
the
upb
branch
17 times, most recently
from
July 13, 2026 16:19
f12cafb to
ce53e01
Compare
- Implement C++ wrappers in upb_wrapper.h replacing protobuf-lite with UPB. - Fix decoding space trimming by adding child-parent sync (on_change_). - Achieve thread-safe concurrent inference via eager cache loading. - Add comprehensive upb_wrapper_test.cc (OOB safety, multi-thread stress, lazy-init/UAF validation, lazy-init concurrency race). - Document API differences and anti-patterns in builtin_upb/README.txt. - Map UPB_ATOMIC to std::atomic under C++ in third_party/upb/upb.h to maintain thread-safe lock-free atomic operations. - Map upb_Atomic_* macros to std::atomic member functions under C++ to fix 32-bit Linux/GCC stdatomic compatibility. - Fix upb_StringView alignment assert in third_party/upb/upb.c to support m68k 2-byte alignment. - Install sentencepiece.pb.h, sentencepiece_model.pb.h and upb_wrapper.h as public headers in src/CMakeLists.txt to fix Python pip Wheel builds. - Suppress MSVC warnings C4334 and C5287 for Windows/ARM64 upb builds. - Resolve dynamic include paths in python/setup.py under PEP 517 build isolation. - Enable SPM_USE_UPB=ON in python/build_bundled.sh to ensure UPB headers are installed during Unix bundled builds. - Fix MSVC pointer cast and designated initializer errors in third_party/upb/upb.h. - Format modified upb.h and upb.c to fit 80 characters limit and remove trailing spaces. - Make UPB the default build mode (SPM_USE_PROTOC=OFF) and add SPM_USE_PROTOC option (default OFF) to build with legacy protoc (protobuf-lite). - Ignore python/build/ directory in .gitignore to prevent compiler cache pollution in Docker CI. - Copy C++ core files to python/sentencepiece prior to calling cibuildwheel in .github/workflows/wheel.yml. - Synchronize const cache pointers when mutable spec is obtained in ModelProtoWrapper to prevent Segmentation faults on dynamic spec overrides in Python. - Pre-initialize cache objects (normalizer_spec_cache_ and denormalizer_spec_cache_) in ModelProtoWrapper default constructor using const default specs. - Simplify normalizer_spec() and denormalizer_spec() getters to return pre-allocated cache references, avoiding dynamic lazy-initialization and eliminating data races. - Fix Use-After-Free (UAF) bug in on_change lambdas where cached wrappers were replaced with new std::unique_ptr instances, causing raw pointers stored in C++ core to become dangling. - Specify shell: bash for wheel build steps in wheel.yml to support copying core source files on Windows runners. - Link sentencepiece_train.lib conditionally on Windows in python/setup.py to resolve trainer and normalizer symbols during cibuildwheel builds. - Skip logstream redirection test (test_train_with_logstream) under Abseil Logging (Google3) in python/test/clean_sentencepiece_test_manual.py due to stderr caching. TAG=agy CONV=e46efd90-ae25-4cfc-8385-5381e11ee2ec
…tion - Fix OOB write & null dereference in ModelProto/spt/nbest wrappers. - Fix heap-buffer-overflow in builder.cc caused by untyped string_view.data() log. - DRY up wrappers using boundary-check macros and template cache initializer. - Extract proto-independent wrapper helpers and DEFINE_UPB_* macros into reusable `upb_message_wrapper.h`. - Migrate wrapper memory & arena lifecycle management from raw pointers to Composition using `UpbMessageHolder<T>`. - Maintain original reference-return spec for API compatibility. - Eliminate boilerplate setter implementations using `UPB_WRAPPER_COPY_AND_SET_SUB_MSG` macro. - DRY up boilerplate assignment operators using `DEFINE_UPB_ASSIGNMENT_OPERATOR` macro. - Refactor accessor macros to delegate actual logic to type-safe C++ helper templates. - Consolidate complex mutable spec getter cache synchronizations into `GetOrCreateMutableSpec` template method in `ModelProtoWrapper`. - DRY up string accessors in `SelfTestData_SampleWrapper` using `DEFINE_UPB_STRING_ACCESSOR` macro. TAG=agy CONV=e46efd90-ae25-4cfc-8385-5381e11ee2ec
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrate SentencePiece to UPB (protobuf-free) mode.