Skip to content

swift eval broken with evalscope>=1.11: EvalType.SERVICE was removed but ms-swift still uses it #10465

Open

Description

@darrentangdt

Describe the bug

swift eval --eval_backend Native crashes with AttributeError: type object 'EvalType' has no attribute 'SERVICE' when the installed evalscope is >= 1.11.0.

evalscope 1.11.0 removed EvalType.SERVICE (it was kept as a deprecated alias SERVICE = 'openai_api' up to 1.10.0), but ms-swift still references it in swift/pipelines/eval/eval.py:

eval_type=EvalType.SERVICE,

This is still the case in the latest release (4.6.0, verified from the PyPI sdist). Meanwhile requirements/eval.txt declares evalscope>=1.0.0 with no upper bound, so a fresh pip install "ms-swift[eval]" today resolves evalscope 1.11+ and swift eval is broken out of the box.

To Reproduce

pip install "ms-swift[eval]==4.6.0" "evalscope>=1.11"
swift eval --model Qwen/Qwen2.5-0.5B-Instruct \
  --eval_backend Native --infer_backend pt \
  --eval_dataset gsm8k --eval_limit 20

Traceback (trimmed):

File ".../swift/pipelines/eval/eval.py", line 96, in get_native_task_cfg
    eval_type=EvalType.SERVICE,
AttributeError: type object 'EvalType' has no attribute 'SERVICE'

Expected behavior

Either migrate to EvalType.OPENAI_API (with a fallback for older evalscope), or cap the requirement as evalscope>=1.0.0,<1.11 until the migration lands.

Workaround

Pinning evalscope==1.9.1 (or <1.11) makes swift eval work again.

Environment

  • ms-swift: 4.5.3 / 4.6.0
  • evalscope: 1.11.0+ (fails), 1.9.1/1.10.0 (works)
  • python: 3.11 / 3.12

Activity

  1. ryznren commented on Oct 29, 2026

    @ryznren

    I'm working on a PR to fix the EvalScope compatibility issue, including regression tests for the supported versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions