WardandClaude Fable 5.1
a10b8efa38
Sync upstream and build images / sync-and-build (push) Successful in 1m46s
0.1.6-alpha.2 built and passed its smoke test on 2026-09-17, then the push died: the registry sits behind Cloudflare, which rejects request bodies over 100 MB, and `docker push` sends a layer as one request. dsh's node_modules layer was 78 MB compressed through 0.1.6-alpha.1 and is ~137 MB from alpha.2 on, so every future version would have failed the same way. The run stayed green because a non-current version only warned. Images are now exported from the daemon and pushed with regctl (pinned v0.11.6, checksum-verified) in 50 MB chunks; the registry accepts chunked uploads and a 140 MB layer went through in 12 s. Any version that fails to build or push fails the run, as a final step so :latest and the releases for everything that did ship are in place first. Manifests left untagged by rebuilds are pruned each run, keeping anything a tagged index still points at. Co-Authored-By: Claude Fable 5.1 <[email protected]>
330 lines
17 KiB
YAML
330 lines
17 KiB
YAML
name: Sync upstream and build images
|
|
|
|
# Mirrors the official DeepSeek Harness repository and ships one container
|
|
# image per upstream release, with this repo's fixes applied:
|
|
# 1. Mirror upstream `master` and every `dsh-v*` tag — same names, same
|
|
# commits, same dates as https://github.com/deepseek-ai/deepseek-harness/tags.
|
|
# 2. For every upstream version that is also published on npm (from
|
|
# MIN_VERSION up): build the image if the registry lacks it, or rebuild it
|
|
# in place when our recipe (Dockerfile, entrypoint, wrapper, bind patch),
|
|
# pnpm, or the node base image changed since it was built. One image tag
|
|
# per version — <version>, nothing else.
|
|
# 3. Point `:latest` at the newest release-candidate version.
|
|
# 4. Publish one Gitea release per built version on the mirrored upstream
|
|
# tag, titled like upstream, carrying upstream's own notes plus a short
|
|
# image footer. Updated in place when the image is rebuilt.
|
|
# 5. A version that fails to build or push fails the run: a missed upstream
|
|
# release must be visible, not a warning in a green run.
|
|
#
|
|
# Images are pushed with regctl, in 50 MB chunks. The registry sits behind
|
|
# Cloudflare, which rejects any request body over 100 MB; `docker push` sends a
|
|
# layer as one request, and dsh's node_modules layer passed that mark with
|
|
# 0.1.6-alpha.2 (413 Payload Too Large).
|
|
|
|
on:
|
|
schedule:
|
|
- cron: '17 */6 * * *'
|
|
workflow_dispatch:
|
|
inputs:
|
|
rebuild:
|
|
description: Force-rebuild these versions (space-separated) or "all"
|
|
required: false
|
|
default: ''
|
|
push:
|
|
branches: [main]
|
|
paths:
|
|
- Dockerfile
|
|
- docker-entrypoint.sh
|
|
- run-as-runtime-user.sh
|
|
- web.cordis.patch.yml
|
|
- .gitea/workflows/sync-upstream.yml
|
|
- .gitea/scripts/publish-releases.js
|
|
|
|
env:
|
|
UPSTREAM: https://github.com/deepseek-ai/deepseek-harness.git
|
|
REGISTRY: gitea.milesward.dev
|
|
IMAGE: mward4/deepseek-harness
|
|
BASE_IMAGE: node:22-bookworm-slim
|
|
# Oldest upstream version this project ships an image for.
|
|
MIN_VERSION: 0.1.1-rc.2
|
|
# Upstream versions that cannot be built from npm on this base image, with
|
|
# the reason, so the run neither retries them every six hours nor warns:
|
|
# 0.1.3-alpha.2 — depends on the native module fs-ext, which needs a C
|
|
# toolchain (node-gyp) the slim runtime image lacks.
|
|
SKIP_VERSIONS: 0.1.3-alpha.2
|
|
|
|
jobs:
|
|
sync-and-build:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Checkout CI branch
|
|
uses: actions/checkout@v4
|
|
with:
|
|
ref: main
|
|
|
|
# Static binary, pinned by version and checksum. Used to push images in
|
|
# chunks (see the header) and to prune untagged manifests.
|
|
- name: Install regctl
|
|
env:
|
|
REGCTL_VERSION: v0.11.6
|
|
REGCTL_SHA256: 8e0e62a497fcdb8048d18aa927a139613176ba0531f412bc541044e28f9856bd
|
|
run: |
|
|
set -eu
|
|
curl -fsSL -o /usr/local/bin/regctl "https://github.com/regclient/regclient/releases/download/${REGCTL_VERSION}/regctl-linux-amd64"
|
|
echo "${REGCTL_SHA256} /usr/local/bin/regctl" | sha256sum -c -
|
|
chmod 0755 /usr/local/bin/regctl
|
|
regctl version
|
|
|
|
- name: Mirror upstream master and dsh-v* tags
|
|
env:
|
|
TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
run: |
|
|
set -eux
|
|
git clone --bare --single-branch --branch master "$UPSTREAM" /tmp/upstream
|
|
cd /tmp/upstream
|
|
git fetch -q origin '+refs/tags/dsh-v*:refs/tags/dsh-v*'
|
|
git remote add mirror "https://oauth2:${TOKEN}@${REGISTRY}/${IMAGE}.git"
|
|
git push mirror '+refs/heads/master:refs/heads/master' '+refs/tags/dsh-v*:refs/tags/dsh-v*'
|
|
git tag -l 'dsh-v*' | sed 's/^dsh-v//' > /tmp/upstream-versions
|
|
echo "upstream versions: $(tr '\n' ' ' < /tmp/upstream-versions)"
|
|
|
|
- name: Plan builds
|
|
id: plan
|
|
env:
|
|
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
|
OWNER: ${{ github.repository_owner }}
|
|
REBUILD: ${{ inputs.rebuild }}
|
|
run: |
|
|
set -eu
|
|
PNPM_LATEST=$(npm view pnpm dist-tags.latest)
|
|
npm view @deepseek-ai/dsh versions --json | grep -o '"[0-9][^"]*"' | tr -d '"' > /tmp/npm-versions
|
|
docker pull -q "$BASE_IMAGE" > /dev/null
|
|
BASE_DIGEST=$(docker inspect --format '{{index .RepoDigests 0}}' "$BASE_IMAGE")
|
|
RECIPE_HASH=$(cat Dockerfile docker-entrypoint.sh run-as-runtime-user.sh web.cordis.patch.yml | sha256sum | cut -c1-12)
|
|
WANT="${PNPM_LATEST}|${BASE_DIGEST}|${RECIPE_HASH}"
|
|
echo "pnpm=$PNPM_LATEST base=$BASE_DIGEST recipe=$RECIPE_HASH rebuild='$REBUILD'"
|
|
|
|
ACCEPT='application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json'
|
|
tag_exists() {
|
|
[ "$(curl -s -o /dev/null -w '%{http_code}' -u "${OWNER}:${TOKEN}" \
|
|
-H "Accept: $ACCEPT" "https://${REGISTRY}/v2/${IMAGE}/manifests/$1")" = 200 ]
|
|
}
|
|
fetch_manifest() { curl -s -u "${OWNER}:${TOKEN}" -H "Accept: $ACCEPT" "https://${REGISTRY}/v2/${IMAGE}/manifests/$1"; }
|
|
first_digest() { printf '%s' "$1" | grep -o '"digest": *"sha256:[a-f0-9]*"' | head -1 | grep -o 'sha256:[a-f0-9]*'; }
|
|
# Provenance labels of an image ("pnpm|base|recipe"); follows a
|
|
# multi-arch index to its first platform manifest.
|
|
tag_toolchain() {
|
|
MAN=$(fetch_manifest "$1") || return 0
|
|
if printf '%s' "$MAN" | grep -q '"manifests"'; then
|
|
CHILD=$(first_digest "$MAN"); [ -n "$CHILD" ] || return 0
|
|
MAN=$(fetch_manifest "$CHILD") || return 0
|
|
fi
|
|
CFG=$(first_digest "$MAN") || return 0
|
|
[ -n "$CFG" ] || return 0
|
|
BLOB=$(curl -s -u "${OWNER}:${TOKEN}" "https://${REGISTRY}/v2/${IMAGE}/blobs/$CFG") || return 0
|
|
P=$(printf '%s' "$BLOB" | grep -o '"dev.milesward.pnpm-version":"[^"]*"' | cut -d'"' -f4 || true)
|
|
B=$(printf '%s' "$BLOB" | grep -o '"dev.milesward.base-digest":"[^"]*"' | cut -d'"' -f4 || true)
|
|
R=$(printf '%s' "$BLOB" | grep -o '"dev.milesward.recipe-hash":"[^"]*"' | cut -d'"' -f4 || true)
|
|
echo "${P}|${B}|${R}"
|
|
}
|
|
# v1 >= v2 in version order?
|
|
at_least() { [ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | head -1)" = "$2" ]; }
|
|
forced() { case " $REBUILD " in *" all "*|*" $1 "*) return 0;; *) return 1;; esac; }
|
|
|
|
# Candidates: upstream tags that npm actually published, from MIN_VERSION up.
|
|
CANDIDATES=""
|
|
while read -r V; do
|
|
[ -n "$V" ] || continue
|
|
grep -qx "$V" /tmp/npm-versions || { echo "skip $V: not on npm"; continue; }
|
|
at_least "$V" "$MIN_VERSION" || continue
|
|
case " $SKIP_VERSIONS " in *" $V "*) echo "skip $V: listed in SKIP_VERSIONS"; continue;; esac
|
|
CANDIDATES="$CANDIDATES $V"
|
|
done < <(sort -V /tmp/upstream-versions)
|
|
echo "candidates:$CANDIDATES"
|
|
|
|
BUILDS=""
|
|
for V in $CANDIDATES; do
|
|
if ! tag_exists "$V"; then
|
|
echo "$V: missing -> build"; BUILDS="$BUILDS $V"
|
|
elif forced "$V"; then
|
|
echo "$V: rebuild requested"; BUILDS="$BUILDS $V"
|
|
else
|
|
HAVE=$(tag_toolchain "$V")
|
|
if [ "$HAVE" != "$WANT" ]; then echo "$V: stale ($HAVE) -> rebuild"; BUILDS="$BUILDS $V"; fi
|
|
fi
|
|
done
|
|
# Newest release candidate = what :latest points at.
|
|
CURRENT=$(printf '%s\n' $CANDIDATES | grep -v alpha | sort -V | tail -1)
|
|
echo "planned builds:$BUILDS"
|
|
echo "current (:latest): $CURRENT"
|
|
{
|
|
echo "builds=${BUILDS}"
|
|
echo "current=${CURRENT}"
|
|
echo "pnpm=${PNPM_LATEST}"
|
|
echo "base_digest=${BASE_DIGEST}"
|
|
echo "recipe_hash=${RECIPE_HASH}"
|
|
} >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Build, smoke-test, and push
|
|
id: build
|
|
if: steps.plan.outputs.builds != ''
|
|
env:
|
|
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
|
PNPM_VERSION: ${{ steps.plan.outputs.pnpm }}
|
|
BASE_DIGEST: ${{ steps.plan.outputs.base_digest }}
|
|
RECIPE_HASH: ${{ steps.plan.outputs.recipe_hash }}
|
|
CURRENT: ${{ steps.plan.outputs.current }}
|
|
CI_RUN: ${{ github.run_number }}
|
|
run: |
|
|
set -u
|
|
echo "$TOKEN" | regctl registry login "$REGISTRY" -u "${{ github.repository_owner }}" --pass-stdin
|
|
# Blobs above 50 MB go up in 50 MB chunks: half of Cloudflare's cap.
|
|
regctl registry set "$REGISTRY" --blob-chunk 50000000 --blob-max 50000000
|
|
GIT_SHA=$(git rev-parse HEAD)
|
|
BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
|
SMOKE="dsh-smoke-${{ github.run_id }}"
|
|
|
|
# Build one version; smoke-test on a fresh root-owned volume (what a
|
|
# new Unraid appdata dir looks like); push only if it passes.
|
|
# 200 = pre-token versions, 303/401 = 0.1.2+ launch-token gate.
|
|
build_one() (
|
|
set -eux
|
|
V="$1"; REF="${REGISTRY}/${IMAGE}:${V}"
|
|
docker build \
|
|
--build-arg "DSH_VERSION=${V}" \
|
|
--build-arg "PNPM_VERSION=${PNPM_VERSION}" \
|
|
--build-arg "BASE_DIGEST=${BASE_DIGEST}" \
|
|
--build-arg "RECIPE_HASH=${RECIPE_HASH}" \
|
|
--build-arg "GIT_SHA=${GIT_SHA}" \
|
|
--build-arg "CI_RUN=${CI_RUN}" \
|
|
--build-arg "BUILD_DATE=${BUILD_DATE}" \
|
|
-t "$REF" .
|
|
docker rm -f "$SMOKE" 2>/dev/null || true
|
|
docker volume rm -f "$SMOKE" 2>/dev/null || true
|
|
docker run -d --name "$SMOKE" --mount "type=volume,src=$SMOKE,dst=/home/node/.dsh" "$REF"
|
|
sleep 3
|
|
[ "$(docker exec "$SMOKE" dsh --version)" = "$V" ]
|
|
[ "$(docker exec "$SMOKE" pnpm --version)" = "$PNPM_VERSION" ]
|
|
[ "$(docker exec "$SMOKE" awk '/^Uid:/{print $2}' /proc/1/status)" = "1000" ]
|
|
OK=""
|
|
for i in $(seq 1 30); do
|
|
if docker exec "$SMOKE" node -e '
|
|
const os = require("os");
|
|
const nic = Object.values(os.networkInterfaces()).flat()
|
|
.find(i => i && i.family === "IPv4" && !i.internal);
|
|
fetch("http://" + nic.address + ":3080/").then(r => {
|
|
process.exit([200, 303, 401].includes(r.status) ? 0 : 1);
|
|
}).catch(() => process.exit(2));'
|
|
then OK=1; break; fi
|
|
sleep 2
|
|
done
|
|
if [ -z "$OK" ]; then docker logs "$SMOKE" | tail -40; exit 1; fi
|
|
docker rm -f "$SMOKE"; docker volume rm -f "$SMOKE" || true
|
|
# Export from the daemon and push with regctl (chunked). A daemon
|
|
# with the containerd store exports an index with an attestation
|
|
# manifest; copy only the image so the tag stays a plain manifest.
|
|
rm -rf /tmp/export; mkdir -p /tmp/export
|
|
docker save "$REF" -o /tmp/export/image.tar
|
|
regctl image import "ocidir:///tmp/export/oci:${V}" /tmp/export/image.tar
|
|
PLATFORM=""
|
|
if regctl manifest get "ocidir:///tmp/export/oci:${V}" --format raw-body | grep -q '"manifests"'; then PLATFORM="--platform linux/amd64"; fi
|
|
regctl image copy $PLATFORM "ocidir:///tmp/export/oci:${V}" "$REF"
|
|
rm -rf /tmp/export
|
|
regctl manifest head "$REF" > /dev/null
|
|
)
|
|
|
|
FAILED=""
|
|
for V in ${{ steps.plan.outputs.builds }}; do
|
|
if build_one "$V"; then
|
|
echo "::notice::built ${IMAGE}:${V}"
|
|
else
|
|
docker rm -f "$SMOKE" 2>/dev/null || true; docker volume rm -f "$SMOKE" 2>/dev/null || true
|
|
echo "::warning::${V} did not build or push; any existing image for it is left untouched"
|
|
FAILED="$FAILED $V"
|
|
fi
|
|
done
|
|
echo "failed=${FAILED}" >> "$GITHUB_OUTPUT"
|
|
if [ -n "$FAILED" ]; then
|
|
echo "failed versions:$FAILED"
|
|
case " $FAILED " in *" $CURRENT "*) echo "::error::the current version (${CURRENT}) failed to build"; exit 1;; esac
|
|
fi
|
|
|
|
# `:latest` = the newest release-candidate version. A manifest copy
|
|
# through the registry API: identical digest, nothing rebuilt.
|
|
- name: Point :latest at the current version
|
|
env:
|
|
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
|
OWNER: ${{ github.repository_owner }}
|
|
CURRENT: ${{ steps.plan.outputs.current }}
|
|
run: |
|
|
set -eu
|
|
[ -n "$CURRENT" ] || { echo "no current version"; exit 0; }
|
|
ACCEPT='application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json'
|
|
digest_of() { curl -sI -u "${OWNER}:${TOKEN}" -H "Accept: $ACCEPT" "https://${REGISTRY}/v2/${IMAGE}/manifests/$1" | grep -i '^docker-content-digest' | awk '{print $2}' | tr -d '\r'; }
|
|
WANT=$(digest_of "$CURRENT")
|
|
[ -n "$WANT" ] || { echo "::error::${CURRENT} has no image in the registry"; exit 1; }
|
|
HAVE=$(digest_of latest || true)
|
|
echo "latest -> ${CURRENT} (${WANT}); currently ${HAVE:-unset}"
|
|
[ "$WANT" != "$HAVE" ] || { echo "latest already current"; exit 0; }
|
|
curl -s -u "${OWNER}:${TOKEN}" -H "Accept: $ACCEPT" -D /tmp/hdr -o /tmp/manifest "https://${REGISTRY}/v2/${IMAGE}/manifests/${CURRENT}"
|
|
CT=$(grep -i '^content-type' /tmp/hdr | awk '{print $2}' | tr -d '\r')
|
|
CODE=$(curl -s -o /tmp/put.out -w '%{http_code}' -X PUT -u "${OWNER}:${TOKEN}" -H "Content-Type: ${CT}" \
|
|
--data-binary @/tmp/manifest "https://${REGISTRY}/v2/${IMAGE}/manifests/latest")
|
|
[ "$CODE" = 201 ] || { echo "::error::retag failed: HTTP $CODE"; cat /tmp/put.out; exit 1; }
|
|
echo "latest now -> ${CURRENT} ($(digest_of latest))"
|
|
|
|
# Every rebuild of a version tag leaves its previous manifest behind as
|
|
# an untagged package version (a `sha256:` row on the Packages page that
|
|
# keeps its layers on disk). Remove those, keeping any manifest that a
|
|
# tagged index still points at.
|
|
- name: Prune untagged manifests
|
|
env:
|
|
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
|
OWNER: ${{ github.repository_owner }}
|
|
run: |
|
|
set -eu
|
|
API="https://${REGISTRY}/api/v1"
|
|
NAME="${IMAGE#*/}"
|
|
LIST=$(curl -s -u "${OWNER}:${TOKEN}" "${API}/packages/${OWNER}?type=container&q=${NAME}&limit=100")
|
|
VERSIONS=$(printf '%s' "$LIST" | grep -o '"version":"[^"]*"' | cut -d'"' -f4 | sort -u)
|
|
KEEP=" "
|
|
for T in $VERSIONS; do
|
|
case "$T" in sha256:*) continue;; esac
|
|
MAN=$(regctl manifest get "${REGISTRY}/${IMAGE}:${T}" --format raw-body 2>/dev/null || true)
|
|
case "$MAN" in *'"manifests"'*) KEEP="$KEEP$(printf '%s' "$MAN" | grep -o '"digest":"sha256:[a-f0-9]*"' | cut -d'"' -f4 | tr '\n' ' ') ";; esac
|
|
done
|
|
N=0
|
|
for D in $VERSIONS; do
|
|
case "$D" in sha256:*) ;; *) continue;; esac
|
|
case "$KEEP" in *" $D "*) echo "keep $D (referenced by a tagged index)"; continue;; esac
|
|
CODE=$(curl -s -o /dev/null -w '%{http_code}' -X DELETE -u "${OWNER}:${TOKEN}" "${API}/packages/${OWNER}/container/${NAME}/${D/:/%3A}")
|
|
echo "pruned $D (HTTP $CODE)"; N=$((N+1))
|
|
done
|
|
echo "pruned $N untagged manifest(s)"
|
|
|
|
# One release per built version, on the mirrored upstream tag, titled
|
|
# like upstream and carrying upstream's notes; updated in place when the
|
|
# image is rebuilt. Runs on the image base so the runner needs nothing.
|
|
- name: Publish releases
|
|
env:
|
|
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
REG_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
|
OWNER: ${{ github.repository_owner }}
|
|
REPO: ${{ github.repository }}
|
|
RUN_NUMBER: ${{ github.run_number }}
|
|
run: |
|
|
set -eu
|
|
MAIN_SHA=$(git rev-parse HEAD)
|
|
docker run --rm -i \
|
|
-e "GITEA=https://${REGISTRY}" -e OWNER -e REPO -e IMAGE -e GITEA_TOKEN -e REG_TOKEN \
|
|
-e "MAIN_SHA=${MAIN_SHA}" -e RUN_NUMBER \
|
|
"$BASE_IMAGE" node - < .gitea/scripts/publish-releases.js
|
|
|
|
# Last, so :latest and the releases for everything that did ship are
|
|
# already in place when the run turns red.
|
|
- name: Fail the run if a version did not ship
|
|
if: steps.build.outputs.failed != ''
|
|
run: |
|
|
echo "::error::these versions did not build or push:${{ steps.build.outputs.failed }}"
|
|
exit 1
|