test(python): accept standalone live smoke acknowledgements
The installed-wheel smoke checks real model tool use and externally verifies the created file, a fresh host-only challenge, the copied receipt, and the unchanged source. DeepSeek can return the requested acknowledgement before a file-details paragraph; requiring it to be the last line makes correct SDK execution fail according to the placement of explanatory prose. Accept the exact acknowledgement on any standalone line, retaining completed turns, model-requested tool calls, exact file bytes, and session-log checks. Replay the observed response through both turns and keep negative coverage for absent or embedded acknowledgements and incorrect files.
This commit is contained in:
1 parent
a523d8ec58
commit
8c32bcf97d
5 files changed
+23
-10
No files matched your search
@@ -9,8 +9,8 @@
|
||||
en: 8e99640e4936332d
|
||||
zh: 729d5de1e0ad2869
|
||||
/python-contributor-workflows/validate-the-sdk:
|
||||
en: fbdd6bb379edd159
|
||||
zh: f3a803a5d99a7b92
|
||||
en: 1f71b1a48e153113
|
||||
zh: 61ae9412b61dfa03
|
||||
/python-contributor-workflows/run-against-node-source:
|
||||
en: 4d30c74632781443
|
||||
zh: 13191cb05f99dd31
|
||||
|
||||
@@ -38,7 +38,7 @@ SDK scenarios compare committed expected output under `scripts/snapshots/python-
|
||||
|
||||
The `advanced` and `restart` comparisons match native delivery qualifiers against each input's own Session generation, including SDK events and notifications. Captured and other delivery generations retain their numeric values. Each side must use one generation across its Session roles, and fresh logs must identify the current writer. Comparison never rewrites committed recordings.
|
||||
|
||||
Trusted pull requests and master pushes also run `--scenario sdk-live --installed-wheel` on each selected native target. That scenario performs two tool-using turns against `https://api.deepseek.com`: it checks the created file immediately, replaces its content with a host-only random challenge, and requires the second turn to copy the changed content into a fresh receipt without modifying the source. Both turns must complete with the exact sentinel answer and a model-requested tool call; external byte comparisons check the files. Missing repository secrets fail instead of self-skipping. Fork and Dependabot pull requests run the complete keyless installed-wheel path but receive no key.
|
||||
Trusted pull requests and master pushes also run `--scenario sdk-live --installed-wheel` on each selected native target. That scenario performs two tool-using turns against `https://api.deepseek.com`: it checks the created file immediately, replaces its content with a host-only random challenge, and requires the second turn to copy the changed content into a fresh receipt without modifying the source. Both turns must complete with a standalone sentinel acknowledgement and a model-requested tool call; the acknowledgement may precede or follow explanation, and external byte comparisons check the files. Missing repository secrets fail instead of self-skipping. Fork and Dependabot pull requests run the complete keyless installed-wheel path but receive no key.
|
||||
|
||||
An interactive smoke test needs `DEEPSEEK_API_KEY` in the environment or repository-root `.env`:
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ SDK 场景会比对 `scripts/snapshots/python-sdk-single-exe/` 下已提交的
|
||||
|
||||
`advanced` 与 `restart` 的比较按每份输入自身的 Session 代际匹配原生 delivery 限定值,包括 SDK 事件与通知。捕获的代际及其他 delivery 代际保留数值。每一侧的 Session 角色必须使用同一个代际,新日志必须标识当前写入器。比较绝不重写已提交的记录。
|
||||
|
||||
可信拉取请求与 master 推送还会在各自选定的原生目标上运行 `--scenario sdk-live --installed-wheel`。该场景面向 `https://api.deepseek.com` 执行两个使用工具的轮次:立即检查已创建文件,将其内容替换为仅宿主知道的随机挑战值,并要求第二轮将变更后的内容复制到全新的回执文件,且不修改源文件。两个轮次都必须完成、返回精确的哨兵答案并由模型请求调用工具;文件通过外部逐字节比较验证。仓库密钥缺失时失败,而不是自行 skip。Fork 与 Dependabot 拉取请求会运行完整的 keyless 安装后 wheel 路径,但不会获得密钥。
|
||||
可信拉取请求与 master 推送还会在各自选定的原生目标上运行 `--scenario sdk-live --installed-wheel`。该场景面向 `https://api.deepseek.com` 执行两个使用工具的轮次:立即检查已创建文件,将其内容替换为仅宿主知道的随机挑战值,并要求第二轮将变更后的内容复制到全新的回执文件,且不修改源文件。两个轮次都必须完成、返回独占一行的哨兵确认并由模型请求调用工具;确认可以位于解释之前或之后,文件通过外部逐字节比较验证。仓库密钥缺失时失败,而不是自行 skip。Fork 与 Dependabot 拉取请求会运行完整的 keyless 安装后 wheel 路径,但不会获得密钥。
|
||||
|
||||
交互式冒烟测试需要环境变量或仓库根目录 `.env` 中存在 `DEEPSEEK_API_KEY`:
|
||||
|
||||
|
||||
@@ -111,6 +111,20 @@ def test_live_smoke_requires_fresh_external_content(live_smoke: SimpleNamespace)
|
||||
assert live_smoke.closed and not live_smoke.root.exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("label", ["create", "verify"])
|
||||
def test_live_smoke_accepts_file_details_after_acknowledgement(
|
||||
live_smoke: SimpleNamespace, label: str,
|
||||
) -> None:
|
||||
setattr(live_smoke, f"{label}_result", live_result(final_response=(
|
||||
"PYTHON_SDK_LIVE_OK\n\n"
|
||||
"File created at `live-api-marker.txt` — 18 bytes, no BOM, no trailing newline."
|
||||
)))
|
||||
SMOKE["smoke_sdk_live"]()
|
||||
assert len(live_smoke.prompts) == 2
|
||||
assert len(live_smoke.checked_logs) == 1
|
||||
assert live_smoke.closed and not live_smoke.root.exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("label", ["create", "verify"])
|
||||
@pytest.mark.parametrize(("overrides", "message"), [
|
||||
({"finish_reason": "error"}, "turn ended with 'error'"),
|
||||
@@ -700,11 +714,11 @@ def test_profile_plugin_failure_reports_native_exit_status(monkeypatch: pytest.M
|
||||
|
||||
|
||||
@pytest.mark.parametrize("prefix", ["", "File created with exactly 18 bytes.\n\n"])
|
||||
def test_live_turn_accepts_explanation_before_final_sentinel(prefix: str) -> None:
|
||||
def test_live_turn_accepts_explanation_before_acknowledgement(prefix: str) -> None:
|
||||
SMOKE["assert_live_turn"]("create", live_result(final_response=prefix + SMOKE["LIVE_API_SENTINEL"]))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("answer", ["", "PYTHON_SDK_LIVE_OK but the operation failed", "PYTHON_SDK_LIVE_OK\nFailure"])
|
||||
def test_live_turn_rejects_missing_final_sentinel(answer: str) -> None:
|
||||
@pytest.mark.parametrize("answer", ["", "File created.", "PYTHON_SDK_LIVE_OK but the operation failed"])
|
||||
def test_live_turn_rejects_missing_standalone_acknowledgement(answer: str) -> None:
|
||||
with pytest.raises(AssertionError, match="turn returned"):
|
||||
SMOKE["assert_live_turn"]("create", live_result(final_response=answer))
|
||||
Reference in new issue
Block a user