Skip to content

test: validate encode/decode against official spec vectors#52

Merged
seetadev merged 5 commits into
multiformats:masterfrom
sumanjeet0012:fix-issue-38-spec-vectors
Jul 13, 2026
Merged

test: validate encode/decode against official spec vectors#52
seetadev merged 5 commits into
multiformats:masterfrom
sumanjeet0012:fix-issue-38-spec-vectors

Conversation

@sumanjeet0012

Copy link
Copy Markdown
Contributor

Fixes #38

Description

This pull request adds tests/test_spec_vectors.py which executes the official multibase test vectors natively from the multibase-spec submodule (spec/tests/*.csv). This directly aligns py-multibase's regression test suite with the official specifications.

Changes Made

  • Added a dynamically parameterized pytest harness to discover all csv files in the spec repository's tests directory.
  • Mapped columns strictly adhering to whether the vector is canonical (encode + decode tested) or non-canonical (decode tested only).
  • Gracefully handled parsing escaped bytes natively in Python using ast.literal_eval.
  • Included an xfail wrapper around the core loop because these test vectors currently expose known bugs in py-multibase (e.g. BaseStringConverter losing leading zeros, base16 case-insensitivity bugs, etc.). This ensures the test suite safely introduces verification for passed encodings without instantly blocking CI builds.

Verification

  • Verified the suite accurately parses all test_values accurately into bytes.
  • The suite passes with exactly 57 passed tests and 24 xfailed tests, which are correctly attributed to existing implementation gaps.
  • make lint executed successfully.

@seetadev

Copy link
Copy Markdown
Contributor

Thanks for the great contribution, @sumanjeet0012!

This PR is indeed a meaningful improvement to our conformance and regression testing.

Aligning py-multibase with the official specification test vectors helps ensure our implementation remains consistent with the upstream multibase specification and provides much stronger confidence in interoperability across implementations.

We particularly appreciate the dynamic discovery of the official CSV vectors and the clear distinction between canonical and non-canonical test cases, which keeps the test suite maintainable as the specification evolves.

Using xfail for the currently known implementation gaps is also a practical approach, it allows us to introduce comprehensive specification coverage without disrupting CI while clearly documenting areas that still need attention.

As those implementation issues are resolved, these expected failures will naturally transition into passing tests. It is also great to see that the entire CI matrix has passed successfully across multiple Python versions and platforms, giving additional confidence in the robustness of this change.

Neat efforts for building a solid foundation for improving standards compliance and preventing future regressions. Excellent work, and thank you for putting this together.

LGTM, happy to merge.

CCing @acul71, @johannamoran and @mishmosh.

@seetadev seetadev merged commit 61ddd54 into multiformats:master Jul 13, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No spec vector tests validating encode/decode against official test vectors

2 participants