test(ptc): refresh ordinary snapshots without extra retention
This commit is contained in:
1 parent
9b9496b692
commit
aeb166c42e
59 files changed
+411
-6046
No files matched your search
@@ -34,13 +34,11 @@ uv run --project python/sdk python scripts/smoke-python-runtime.py \
|
||||
--scenario sdk-minimal --exe dist-exe/deepseek-harness-sdk-runtime-macos-arm64
|
||||
```
|
||||
|
||||
SDK scenarios compare committed expected output under `scripts/snapshots/python-sdk-single-exe/`. `minimal/model-visible.json` pins the Linux/macOS `sdk-minimal` profile's assembled system prompts, advertised tool schemas, and model-visible messages; `minimal/win-x64/model-visible.json` pins its PowerShell counterpart. A plugin that contributes an unintended system section or user message therefore fails the job, and every message the profile emits is compared. `advanced-native/` pins direct tool calls, child agents, and workflows; `advanced-ptc/` pins program calls and nested denials. Both compare SDK results and session logs across every target. `restart-native/` launches two complete SDK runtime processes against one persistence root and snapshots their isolated model histories, high-level results, and separate durable logs across every target. `sdk-minimal-in-history` reuses the persistent-shell and editor scenario with a section that changes after the first successful shell call. `minimal-in-history/prompt-history.json` pins both prompt versions, the unchanged leading prompt in later requests, appended SDK system-message events, and `request/context.systemPromptUpdate`; tool schemas stay fixed and the editor file is checked independently. `sdk-dynamic-tools` pins native tool additions and removals in `dynamic-tools/tool-history.json`, including historical header references and agreement between Python SDK results, notifications, and persisted events. Rerun the owning scenario with `--update-snapshots` and review that diff before committing it.
|
||||
SDK scenarios compare committed expected output under `scripts/snapshots/python-sdk-single-exe/`. `minimal/model-visible.json` pins the Linux/macOS `sdk-minimal` profile's assembled system prompts, advertised tool schemas, and model-visible messages; `minimal/win-x64/model-visible.json` pins its PowerShell counterpart. A plugin that contributes an unintended system section or user message therefore fails the job, and every message the profile emits is compared. `advanced/` pins direct tool calls, child agents, and workflows; `advanced-ptc/` pins program calls and nested denials. Both compare SDK results and session logs across every target. `restart/` launches two complete SDK runtime processes against one persistence root and snapshots their isolated model histories, high-level results, and separate durable logs across every target. `sdk-minimal-in-history` reuses the persistent-shell and editor scenario with a section that changes after the first successful shell call. `minimal-in-history/prompt-history.json` pins both prompt versions, the unchanged leading prompt in later requests, appended SDK system-message events, and `request/context.systemPromptUpdate`; tool schemas stay fixed and the editor file is checked independently. `sdk-dynamic-tools` pins native tool additions and removals in `dynamic-tools/tool-history.json`, including historical header references and agreement between Python SDK results, notifications, and persisted events. Rerun the owning scenario with `--update-snapshots` and review that diff before committing it.
|
||||
|
||||
The retained `advanced/`, `restart/`, and `scheduler-recovery/` directories keep immutable historical output. Read-only comparison tests cover all three; executable scenarios use `advanced-native/`, `advanced-ptc/`, `restart-native/`, and `scheduler-recovery-native/`.
|
||||
`scheduler-recovery/` records a failed tool turn and a subsequent completed turn. It preserves the original `UNKNOWN` turn error, the first tool's completed result, and `TOOL_OUTCOME_UNKNOWN` / `TOOL_NOT_STARTED` results for the remaining calls. `sdk-snapshot` and `all` include this case; `--scenario sdk-recovery` runs it alone.
|
||||
|
||||
`scheduler-recovery-native/` records a failed tool turn and a subsequent completed turn. It preserves the original `UNKNOWN` turn error, the first tool's completed result, and `TOOL_OUTCOME_UNKNOWN` / `TOOL_NOT_STARTED` results for the remaining calls. `sdk-snapshot` and `all` include this case; `--scenario sdk-recovery` runs it alone.
|
||||
|
||||
The `advanced-native`, `advanced-ptc`, and `restart-native` comparisons match native delivery qualifiers against each input's own Session generation, including SDK events and notifications. Captured and other delivery generations retain their numeric values. Each side must use one generation across its Session roles, and fresh logs must identify the current writer. Comparison never rewrites committed recordings.
|
||||
The `advanced`, `advanced-ptc`, and `restart` comparisons match native delivery qualifiers against each input's own Session generation, including SDK events and notifications. Captured and other delivery generations retain their numeric values. Each side must use one generation across its Session roles, and fresh logs must identify the current writer. Without `--update-snapshots`, comparison writes no files. Refresh writes current-writer outputs and preserves earlier Session generations.
|
||||
|
||||
Trusted pull requests and master pushes also run `--scenario sdk-live --installed-wheel` on each selected native target. That scenario performs two tool-using turns against `https://api.deepseek.com`: it checks the created file immediately, replaces its content with a host-only random challenge, and requires the second turn to copy the changed content into a fresh receipt without modifying the source. Both turns must complete with a standalone sentinel acknowledgement and a model-requested tool call; the acknowledgement may precede or follow explanation, and external byte comparisons check the files. Missing repository secrets fail instead of self-skipping. Fork and Dependabot pull requests run the complete keyless installed-wheel path but receive no key.
|
||||
|
||||
|
||||
Reference in new issue
Block a user