dsh agent
3849709cc2
Sync upstream and build images / sync-and-build (push) Successful in 8m26s
The entrypoint called /usr/local/bin/dsh-workspace-doctor while the Dockerfile installs /usr/local/bin/dsh-workspace-doctor.mjs, so every start logged "...: not found" and the workspace doctor never ran: no report, no repair. The CI smoke test checked the .mjs path with a direct `docker exec`, which cannot see a mistake in the entrypoint's own invocation, so the build stayed green while the container was broken. Resolve the doctor by name (`command -v dsh-workspace-doctor.mjs`) in both the root and the non-root path, and log an explicit "not installed in this image" line when it cannot be found, instead of the shell's bare "not found". The smoke test now reads the container's boot log: it requires the doctor's own "dsh-workspace-doctor:" output to appear, and fails if the entrypoint logs its "workspace doctor:" failure line — that covers path, exec bit and invocation together, which is the class of mistake that shipped. README: manual repair through a one-off container with --entrypoint (dsh stopped, so nothing races the registry write), and the note that an existing Unraid container does not pick up the template's new Workspace Doctor variable on its own.
358 lines
18 KiB
YAML
358 lines
18 KiB
YAML
name: Sync upstream and build images
|
|
|
|
# Mirrors the official DeepSeek Harness repository and ships one container
|
|
# image per upstream release, with this repo's fixes applied:
|
|
# 1. Mirror upstream `master` and every `dsh-v*` tag — same names, same
|
|
# commits, same dates as https://github.com/deepseek-ai/deepseek-harness/tags.
|
|
# 2. For every upstream version that is also published on npm (from
|
|
# MIN_VERSION up): build the image if the registry lacks it, or rebuild it
|
|
# in place when our recipe (Dockerfile, entrypoint, doctor, wrapper, bind
|
|
# patch), pnpm, or the node base image changed since it was built. One image
|
|
# tag per version — <version>, nothing else.
|
|
# 3. Point `:latest` at the newest release-candidate version.
|
|
# 4. Publish one Gitea release per built version on the mirrored upstream
|
|
# tag, titled like upstream, carrying upstream's own notes plus a short
|
|
# image footer. Updated in place when the image is rebuilt.
|
|
# 5. A version that fails to build or push fails the run: a missed upstream
|
|
# release must be visible, not a warning in a green run.
|
|
#
|
|
# Images are pushed with regctl, in 50 MB chunks. The registry sits behind
|
|
# Cloudflare, which rejects any request body over 100 MB; `docker push` sends a
|
|
# layer as one request, and dsh's node_modules layer passed that mark with
|
|
# 0.1.6-alpha.2 (413 Payload Too Large).
|
|
|
|
on:
|
|
schedule:
|
|
- cron: '17 */6 * * *'
|
|
workflow_dispatch:
|
|
inputs:
|
|
rebuild:
|
|
description: Force-rebuild these versions (space-separated) or "all"
|
|
required: false
|
|
default: ''
|
|
push:
|
|
branches: [main]
|
|
paths:
|
|
- Dockerfile
|
|
- docker-entrypoint.sh
|
|
- dsh-workspace-doctor.mjs
|
|
- run-as-runtime-user.sh
|
|
- web.cordis.patch.yml
|
|
- .gitea/workflows/sync-upstream.yml
|
|
- .gitea/scripts/publish-releases.js
|
|
|
|
env:
|
|
UPSTREAM: https://github.com/deepseek-ai/deepseek-harness.git
|
|
REGISTRY: gitea.milesward.dev
|
|
IMAGE: mward4/deepseek-harness
|
|
BASE_IMAGE: node:22-bookworm-slim
|
|
# Oldest upstream version this project ships an image for.
|
|
MIN_VERSION: 0.1.1-rc.2
|
|
# Upstream versions that cannot be built from npm on this base image, with
|
|
# the reason, so the run neither retries them every six hours nor warns:
|
|
# 0.1.3-alpha.2 — depends on the native module fs-ext, which needs a C
|
|
# toolchain (node-gyp) the slim runtime image lacks.
|
|
SKIP_VERSIONS: 0.1.3-alpha.2
|
|
|
|
jobs:
|
|
sync-and-build:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Checkout CI branch
|
|
uses: actions/checkout@v4
|
|
with:
|
|
ref: main
|
|
|
|
# Static binary, pinned by version and checksum. Used to push images in
|
|
# chunks (see the header) and to prune untagged manifests.
|
|
- name: Install regctl
|
|
env:
|
|
REGCTL_VERSION: v0.11.6
|
|
REGCTL_SHA256: 8e0e62a497fcdb8048d18aa927a139613176ba0531f412bc541044e28f9856bd
|
|
run: |
|
|
set -eu
|
|
curl -fsSL -o /usr/local/bin/regctl "https://github.com/regclient/regclient/releases/download/${REGCTL_VERSION}/regctl-linux-amd64"
|
|
echo "${REGCTL_SHA256} /usr/local/bin/regctl" | sha256sum -c -
|
|
chmod 0755 /usr/local/bin/regctl
|
|
regctl version
|
|
|
|
- name: Mirror upstream master and dsh-v* tags
|
|
env:
|
|
TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
run: |
|
|
set -eux
|
|
git clone --bare --single-branch --branch master "$UPSTREAM" /tmp/upstream
|
|
cd /tmp/upstream
|
|
git fetch -q origin '+refs/tags/dsh-v*:refs/tags/dsh-v*'
|
|
git remote add mirror "https://oauth2:${TOKEN}@${REGISTRY}/${IMAGE}.git"
|
|
git push mirror '+refs/heads/master:refs/heads/master' '+refs/tags/dsh-v*:refs/tags/dsh-v*'
|
|
git tag -l 'dsh-v*' | sed 's/^dsh-v//' > /tmp/upstream-versions
|
|
echo "upstream versions: $(tr '\n' ' ' < /tmp/upstream-versions)"
|
|
|
|
- name: Plan builds
|
|
id: plan
|
|
env:
|
|
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
|
OWNER: ${{ github.repository_owner }}
|
|
REBUILD: ${{ inputs.rebuild }}
|
|
run: |
|
|
set -eu
|
|
PNPM_LATEST=$(npm view pnpm dist-tags.latest)
|
|
npm view @deepseek-ai/dsh versions --json | grep -o '"[0-9][^"]*"' | tr -d '"' > /tmp/npm-versions
|
|
docker pull -q "$BASE_IMAGE" > /dev/null
|
|
BASE_DIGEST=$(docker inspect --format '{{index .RepoDigests 0}}' "$BASE_IMAGE")
|
|
RECIPE_HASH=$(cat Dockerfile docker-entrypoint.sh dsh-workspace-doctor.mjs run-as-runtime-user.sh web.cordis.patch.yml | sha256sum | cut -c1-12)
|
|
WANT="${PNPM_LATEST}|${BASE_DIGEST}|${RECIPE_HASH}"
|
|
echo "pnpm=$PNPM_LATEST base=$BASE_DIGEST recipe=$RECIPE_HASH rebuild='$REBUILD'"
|
|
|
|
ACCEPT='application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json'
|
|
tag_exists() {
|
|
[ "$(curl -s -o /dev/null -w '%{http_code}' -u "${OWNER}:${TOKEN}" \
|
|
-H "Accept: $ACCEPT" "https://${REGISTRY}/v2/${IMAGE}/manifests/$1")" = 200 ]
|
|
}
|
|
fetch_manifest() { curl -s -u "${OWNER}:${TOKEN}" -H "Accept: $ACCEPT" "https://${REGISTRY}/v2/${IMAGE}/manifests/$1"; }
|
|
first_digest() { printf '%s' "$1" | grep -o '"digest": *"sha256:[a-f0-9]*"' | head -1 | grep -o 'sha256:[a-f0-9]*'; }
|
|
# Provenance labels of an image ("pnpm|base|recipe"); follows a
|
|
# multi-arch index to its first platform manifest.
|
|
tag_toolchain() {
|
|
MAN=$(fetch_manifest "$1") || return 0
|
|
if printf '%s' "$MAN" | grep -q '"manifests"'; then
|
|
CHILD=$(first_digest "$MAN"); [ -n "$CHILD" ] || return 0
|
|
MAN=$(fetch_manifest "$CHILD") || return 0
|
|
fi
|
|
CFG=$(first_digest "$MAN") || return 0
|
|
[ -n "$CFG" ] || return 0
|
|
BLOB=$(curl -s -u "${OWNER}:${TOKEN}" "https://${REGISTRY}/v2/${IMAGE}/blobs/$CFG") || return 0
|
|
P=$(printf '%s' "$BLOB" | grep -o '"dev.milesward.pnpm-version":"[^"]*"' | cut -d'"' -f4 || true)
|
|
B=$(printf '%s' "$BLOB" | grep -o '"dev.milesward.base-digest":"[^"]*"' | cut -d'"' -f4 || true)
|
|
R=$(printf '%s' "$BLOB" | grep -o '"dev.milesward.recipe-hash":"[^"]*"' | cut -d'"' -f4 || true)
|
|
echo "${P}|${B}|${R}"
|
|
}
|
|
# v1 >= v2 in version order?
|
|
at_least() { [ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | head -1)" = "$2" ]; }
|
|
forced() { case " $REBUILD " in *" all "*|*" $1 "*) return 0;; *) return 1;; esac; }
|
|
|
|
# Candidates: upstream tags that npm actually published, from MIN_VERSION up.
|
|
CANDIDATES=""
|
|
while read -r V; do
|
|
[ -n "$V" ] || continue
|
|
grep -qx "$V" /tmp/npm-versions || { echo "skip $V: not on npm"; continue; }
|
|
at_least "$V" "$MIN_VERSION" || continue
|
|
case " $SKIP_VERSIONS " in *" $V "*) echo "skip $V: listed in SKIP_VERSIONS"; continue;; esac
|
|
CANDIDATES="$CANDIDATES $V"
|
|
done < <(sort -V /tmp/upstream-versions)
|
|
echo "candidates:$CANDIDATES"
|
|
|
|
BUILDS=""
|
|
for V in $CANDIDATES; do
|
|
if ! tag_exists "$V"; then
|
|
echo "$V: missing -> build"; BUILDS="$BUILDS $V"
|
|
elif forced "$V"; then
|
|
echo "$V: rebuild requested"; BUILDS="$BUILDS $V"
|
|
else
|
|
HAVE=$(tag_toolchain "$V")
|
|
if [ "$HAVE" != "$WANT" ]; then echo "$V: stale ($HAVE) -> rebuild"; BUILDS="$BUILDS $V"; fi
|
|
fi
|
|
done
|
|
# Newest release candidate = what :latest points at.
|
|
CURRENT=$(printf '%s\n' $CANDIDATES | grep -v alpha | sort -V | tail -1)
|
|
echo "planned builds:$BUILDS"
|
|
echo "current (:latest): $CURRENT"
|
|
{
|
|
echo "builds=${BUILDS}"
|
|
echo "current=${CURRENT}"
|
|
echo "pnpm=${PNPM_LATEST}"
|
|
echo "base_digest=${BASE_DIGEST}"
|
|
echo "recipe_hash=${RECIPE_HASH}"
|
|
} >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Build, smoke-test, and push
|
|
id: build
|
|
if: steps.plan.outputs.builds != ''
|
|
env:
|
|
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
|
PNPM_VERSION: ${{ steps.plan.outputs.pnpm }}
|
|
BASE_DIGEST: ${{ steps.plan.outputs.base_digest }}
|
|
RECIPE_HASH: ${{ steps.plan.outputs.recipe_hash }}
|
|
CURRENT: ${{ steps.plan.outputs.current }}
|
|
CI_RUN: ${{ github.run_number }}
|
|
run: |
|
|
set -u
|
|
echo "$TOKEN" | regctl registry login "$REGISTRY" -u "${{ github.repository_owner }}" --pass-stdin
|
|
# Blobs above 50 MB go up in 50 MB chunks: half of Cloudflare's cap.
|
|
regctl registry set "$REGISTRY" --blob-chunk 50000000 --blob-max 50000000
|
|
GIT_SHA=$(git rev-parse HEAD)
|
|
BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
|
SMOKE="dsh-smoke-${{ github.run_id }}"
|
|
|
|
# Build one version; smoke-test on a fresh root-owned volume (what a
|
|
# new Unraid appdata dir looks like); push only if it passes.
|
|
# 200 = pre-token versions, 303/401 = 0.1.2+ launch-token gate.
|
|
build_one() (
|
|
set -eux
|
|
V="$1"; REF="${REGISTRY}/${IMAGE}:${V}"
|
|
docker build \
|
|
--build-arg "DSH_VERSION=${V}" \
|
|
--build-arg "PNPM_VERSION=${PNPM_VERSION}" \
|
|
--build-arg "BASE_DIGEST=${BASE_DIGEST}" \
|
|
--build-arg "RECIPE_HASH=${RECIPE_HASH}" \
|
|
--build-arg "GIT_SHA=${GIT_SHA}" \
|
|
--build-arg "CI_RUN=${CI_RUN}" \
|
|
--build-arg "BUILD_DATE=${BUILD_DATE}" \
|
|
-t "$REF" .
|
|
docker rm -f "$SMOKE" 2>/dev/null || true
|
|
docker volume rm -f "$SMOKE" 2>/dev/null || true
|
|
docker run -d --name "$SMOKE" --mount "type=volume,src=$SMOKE,dst=/home/node/.dsh" "$REF"
|
|
sleep 3
|
|
[ "$(docker exec "$SMOKE" dsh --version)" = "$V" ]
|
|
[ "$(docker exec "$SMOKE" pnpm --version)" = "$PNPM_VERSION" ]
|
|
[ "$(docker exec "$SMOKE" awk '/^Uid:/{print $2}' /proc/1/status)" = "1000" ]
|
|
# The workspace doctor must be installed, executable and able to read
|
|
# a state volume. --help is the strict presence check; --check may
|
|
# legitimately answer 1 (something to report) on a fresh volume, so
|
|
# only 2 — the tool itself failed — is a build failure.
|
|
docker exec "$SMOKE" dsh-workspace-doctor.mjs --help > /dev/null
|
|
DOCTOR_RC=0
|
|
docker exec "$SMOKE" dsh-workspace-doctor.mjs --check > /dev/null || DOCTOR_RC=$?
|
|
[ "$DOCTOR_RC" -le 1 ]
|
|
# ...and the entrypoint must actually invoke it. The doctor prints a
|
|
# "dsh-workspace-doctor:" line when it runs; the entrypoint logs a
|
|
# "workspace doctor:" line only when it could not. A direct
|
|
# `docker exec` above sails past a wrong path or a lost exec bit, so
|
|
# the boot log, not the binary, is what proves the wiring.
|
|
if ! docker logs "$SMOKE" 2>&1 | grep -q "^dsh-workspace-doctor:"; then
|
|
echo "the workspace doctor did not run at boot"
|
|
docker logs "$SMOKE" | tail -30
|
|
exit 1
|
|
fi
|
|
if docker logs "$SMOKE" 2>&1 | grep -q "workspace doctor:"; then
|
|
echo "the entrypoint could not run the workspace doctor"
|
|
docker logs "$SMOKE" | tail -30
|
|
exit 1
|
|
fi
|
|
OK=""
|
|
for i in $(seq 1 30); do
|
|
if docker exec "$SMOKE" node -e '
|
|
const os = require("os");
|
|
const nic = Object.values(os.networkInterfaces()).flat()
|
|
.find(i => i && i.family === "IPv4" && !i.internal);
|
|
fetch("http://" + nic.address + ":3080/").then(r => {
|
|
process.exit([200, 303, 401].includes(r.status) ? 0 : 1);
|
|
}).catch(() => process.exit(2));'
|
|
then OK=1; break; fi
|
|
sleep 2
|
|
done
|
|
if [ -z "$OK" ]; then docker logs "$SMOKE" | tail -40; exit 1; fi
|
|
docker rm -f "$SMOKE"; docker volume rm -f "$SMOKE" || true
|
|
# Export from the daemon and push with regctl (chunked). A daemon
|
|
# with the containerd store exports an index with an attestation
|
|
# manifest; copy only the image so the tag stays a plain manifest.
|
|
rm -rf /tmp/export; mkdir -p /tmp/export
|
|
docker save "$REF" -o /tmp/export/image.tar
|
|
regctl image import "ocidir:///tmp/export/oci:${V}" /tmp/export/image.tar
|
|
# A classic (non-containerd) daemon exports layers uncompressed, and
|
|
# regctl pushes what it is given: without this, dsh's 500 MB raw
|
|
# layer would reach the registry and every pull would be 4x larger.
|
|
regctl image mod "ocidir:///tmp/export/oci:${V}" --layer-compress gzip --replace
|
|
PLATFORM=""
|
|
if regctl manifest get "ocidir:///tmp/export/oci:${V}" --format raw-body | grep -q '"manifests"'; then PLATFORM="--platform linux/amd64"; fi
|
|
regctl image copy $PLATFORM "ocidir:///tmp/export/oci:${V}" "$REF"
|
|
rm -rf /tmp/export
|
|
regctl manifest head "$REF" > /dev/null
|
|
)
|
|
|
|
FAILED=""
|
|
for V in ${{ steps.plan.outputs.builds }}; do
|
|
if build_one "$V"; then
|
|
echo "::notice::built ${IMAGE}:${V}"
|
|
else
|
|
docker rm -f "$SMOKE" 2>/dev/null || true; docker volume rm -f "$SMOKE" 2>/dev/null || true
|
|
echo "::warning::${V} did not build or push; any existing image for it is left untouched"
|
|
FAILED="$FAILED $V"
|
|
fi
|
|
done
|
|
echo "failed=${FAILED}" >> "$GITHUB_OUTPUT"
|
|
if [ -n "$FAILED" ]; then
|
|
echo "failed versions:$FAILED"
|
|
case " $FAILED " in *" $CURRENT "*) echo "::error::the current version (${CURRENT}) failed to build"; exit 1;; esac
|
|
fi
|
|
|
|
# `:latest` = the newest release-candidate version. A manifest copy
|
|
# through the registry API: identical digest, nothing rebuilt.
|
|
- name: Point :latest at the current version
|
|
env:
|
|
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
|
OWNER: ${{ github.repository_owner }}
|
|
CURRENT: ${{ steps.plan.outputs.current }}
|
|
run: |
|
|
set -eu
|
|
[ -n "$CURRENT" ] || { echo "no current version"; exit 0; }
|
|
ACCEPT='application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json'
|
|
digest_of() { curl -sI -u "${OWNER}:${TOKEN}" -H "Accept: $ACCEPT" "https://${REGISTRY}/v2/${IMAGE}/manifests/$1" | grep -i '^docker-content-digest' | awk '{print $2}' | tr -d '\r'; }
|
|
WANT=$(digest_of "$CURRENT")
|
|
[ -n "$WANT" ] || { echo "::error::${CURRENT} has no image in the registry"; exit 1; }
|
|
HAVE=$(digest_of latest || true)
|
|
echo "latest -> ${CURRENT} (${WANT}); currently ${HAVE:-unset}"
|
|
[ "$WANT" != "$HAVE" ] || { echo "latest already current"; exit 0; }
|
|
curl -s -u "${OWNER}:${TOKEN}" -H "Accept: $ACCEPT" -D /tmp/hdr -o /tmp/manifest "https://${REGISTRY}/v2/${IMAGE}/manifests/${CURRENT}"
|
|
CT=$(grep -i '^content-type' /tmp/hdr | awk '{print $2}' | tr -d '\r')
|
|
CODE=$(curl -s -o /tmp/put.out -w '%{http_code}' -X PUT -u "${OWNER}:${TOKEN}" -H "Content-Type: ${CT}" \
|
|
--data-binary @/tmp/manifest "https://${REGISTRY}/v2/${IMAGE}/manifests/latest")
|
|
[ "$CODE" = 201 ] || { echo "::error::retag failed: HTTP $CODE"; cat /tmp/put.out; exit 1; }
|
|
echo "latest now -> ${CURRENT} ($(digest_of latest))"
|
|
|
|
# Every rebuild of a version tag leaves its previous manifest behind as
|
|
# an untagged package version (a `sha256:` row on the Packages page that
|
|
# keeps its layers on disk). Remove those, keeping any manifest that a
|
|
# tagged index still points at.
|
|
- name: Prune untagged manifests
|
|
env:
|
|
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
|
OWNER: ${{ github.repository_owner }}
|
|
run: |
|
|
set -eu
|
|
API="https://${REGISTRY}/api/v1"
|
|
NAME="${IMAGE#*/}"
|
|
LIST=$(curl -s -u "${OWNER}:${TOKEN}" "${API}/packages/${OWNER}?type=container&q=${NAME}&limit=100")
|
|
VERSIONS=$(printf '%s' "$LIST" | grep -o '"version":"[^"]*"' | cut -d'"' -f4 | sort -u)
|
|
KEEP=" "
|
|
for T in $VERSIONS; do
|
|
case "$T" in sha256:*) continue;; esac
|
|
MAN=$(regctl manifest get "${REGISTRY}/${IMAGE}:${T}" --format raw-body 2>/dev/null || true)
|
|
case "$MAN" in *'"manifests"'*) KEEP="$KEEP$(printf '%s' "$MAN" | grep -o '"digest":"sha256:[a-f0-9]*"' | cut -d'"' -f4 | tr '\n' ' ') ";; esac
|
|
done
|
|
N=0
|
|
for D in $VERSIONS; do
|
|
case "$D" in sha256:*) ;; *) continue;; esac
|
|
case "$KEEP" in *" $D "*) echo "keep $D (referenced by a tagged index)"; continue;; esac
|
|
CODE=$(curl -s -o /dev/null -w '%{http_code}' -X DELETE -u "${OWNER}:${TOKEN}" "${API}/packages/${OWNER}/container/${NAME}/${D/:/%3A}")
|
|
echo "pruned $D (HTTP $CODE)"; N=$((N+1))
|
|
done
|
|
echo "pruned $N untagged manifest(s)"
|
|
|
|
# One release per built version, on the mirrored upstream tag, titled
|
|
# like upstream and carrying upstream's notes; updated in place when the
|
|
# image is rebuilt. Runs on the image base so the runner needs nothing.
|
|
- name: Publish releases
|
|
env:
|
|
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
REG_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
|
OWNER: ${{ github.repository_owner }}
|
|
REPO: ${{ github.repository }}
|
|
RUN_NUMBER: ${{ github.run_number }}
|
|
run: |
|
|
set -eu
|
|
MAIN_SHA=$(git rev-parse HEAD)
|
|
docker run --rm -i \
|
|
-e "GITEA=https://${REGISTRY}" -e OWNER -e REPO -e IMAGE -e GITEA_TOKEN -e REG_TOKEN \
|
|
-e "MAIN_SHA=${MAIN_SHA}" -e RUN_NUMBER \
|
|
"$BASE_IMAGE" node - < .gitea/scripts/publish-releases.js
|
|
|
|
# Last, so :latest and the releases for everything that did ship are
|
|
# already in place when the run turns red.
|
|
- name: Fail the run if a version did not ship
|
|
if: steps.build.outputs.failed != ''
|
|
run: |
|
|
echo "::error::these versions did not build or push:${{ steps.build.outputs.failed }}"
|
|
exit 1
|