Skip to content

Restore compatible Neko release image builds - #27

Merged
sjmiller609 merged 2 commits into
masterfrom
hypeship/fix-neko-release-apt
Oct 9, 2026
Merged

sjmiller609 merged 2 commits into
masterfrom
hypeship/fix-neko-release-apt

Conversation

@sjmiller609

@sjmiller609 sjmiller609 commented Oct 9, 2026 •

Copy link
Copy Markdown

Summary

The v3.0.8-v1.6.1 release cannot build: Bullseye security package indexes advertise files that now return 404 from the normal security mirror. The existing Go builder also links the server against a newer libc than the Bullseye runtime provides.

  • Use the signed Debian archive security repository for the Bullseye runtime, Xorg builder, and server builder. No APT signature or expiry checks are disabled.
  • Build the server and plugins on Bullseye, copying the unchanged Go 1.25.0 toolchain from its official image. This preserves the runtime ABI instead of upgrading the runtime OS.
  • Set GOTOOLCHAIN=local to preserve the official Go image's pin and reject dependencies requiring a newer toolchain.
  • Keep the crypto/network dependency upgrades from PR Update Go crypto and network dependencies #25.

Verification

  • Reproduced the package-download 404s independently; the archive mirror installs the same packages successfully.
  • Reproduced the old builder binary failing to load in the published Bullseye base because GLIBC_2.32 and GLIBC_2.34 were unavailable.
  • Generated the full release Dockerfile using the release client artifact and built the complete amd64 base image successfully.
  • Ran /usr/bin/neko --version inside the new base image successfully.
  • Ran go test -race ./..., go vet ./..., and go mod verify in the new server builder; all passed.
  • Verified the packaged binary embeds x/crypto v0.54.0 and x/net v0.57.0.
  • Verified a synthetic module requiring Go 1.26.0 fails with GOTOOLCHAIN=local instead of downloading another toolchain.
  • amd64/arm64 server builds and server tests passed in CI for the toolchain-pin update.
  • Multi-architecture release builds and browser live-view e2e were not run locally.

Release

After merging, publish a new tag v3.0.8-v1.6.2; do not move the existing failed v3.0.8-v1.6.1 tag. Update the dependent browser image pins after the new base is published.

Bullseye is retained for compatibility in this bounded release fix. Migrating the runtime to a supported Debian release remains separate work.


Note

Medium Risk
Changes release build infrastructure and package mirrors for EOL Bullseye; incorrect mirror or toolchain wiring could break CI/releases, but scope is limited to Docker build paths.

Overview
Fixes broken v3.0.8 release image builds by pointing Bullseye APT security sources at archive.debian.org/debian-security in the runtime, server builder, and xorg-deps Dockerfiles so apt-get no longer hits 404s on the live security mirror.

The server image no longer uses golang:1.25.0 as the build stage base. It builds on debian:bullseye-slim and copies the Go 1.25.0 toolchain from a separate stage, with added compile packages (build-essential, git, etc.), so binaries link against Bullseye’s glibc and run in the published runtime without GLIBC_2.32/2.34 errors.

Reviewed by Cursor Bugbot for commit 1745c26. Bugbot is set up for automated code reviews on this repo. Configure here.

@sjmiller609
sjmiller609 marked this pull request as ready for review October 9, 2026 16:37
@sjmiller609

Copy link
Copy Markdown
Author

Additional runtime verification: started the complete rebuilt amd64 base image with its normal supervisord entrypoint. Neko, Xorg, and PulseAudio were all RUNNING; /health succeeded and the web client returned HTTP 200. Logs confirm video-pipeline syntax validation and startup of the filetransfer, chat, scaletozero, and telemetry plugins. This verifies full base-image startup, not browser WebRTC playback or a published multi-architecture release.

@rgarcia rgarcia left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed — lgtm, approving. a few non-blocking things:

Suggestions

  • server/Dockerfile:5-7 — the copied toolchain doesn't carry over the golang image's ENV GOTOOLCHAIN=local (verified: docker run golang:1.25.0 env shows it). without it go falls back to auto, so if a future dep bump requires a go version newer than 1.25.0, the go get in ./build will silently download a newer toolchain instead of failing. consider adding ENV GOTOOLCHAIN=local next to GOPATH/PATH so the builder actually stays pinned to 1.25.0

Questions

  • runtime/Dockerfile.intel:17 — the intel flavor writes its own sources.list with deb.debian.org/debian-security, so it would hit the same 404s if ghcr_intel.yml.bak is ever re-enabled. fine to leave since it's not released, just flagging
  • only bullseye-security moves to the archive here; main is still served from deb.debian.org and will break the same way once debian archives the rest of bullseye. is the move off bullseye tracked somewhere?
  • the tag build in ghcr.yml does amd64 + arm64 + arm/v7 under qemu, and the PR says only amd64 was built locally. worth watching the v3.0.8-v1.6.2 run before bumping the browser image pins

@sjmiller609

Copy link
Copy Markdown
Author

Added GOTOOLCHAIN=local to preserve the original official Go image behavior. Rebuilt the server builder; race-enabled tests, vet, and module verification pass. A synthetic module requiring Go 1.26.0 now fails explicitly while running Go 1.25.0 with GOTOOLCHAIN=local instead of downloading a newer toolchain. The disabled Intel flavor is unchanged. Browser-image pins will only be updated after the tagged multi-architecture base release succeeds.

@sjmiller609
sjmiller609 merged commit 1d69742 into master Oct 9, 2026
4 checks passed
@sjmiller609
sjmiller609 deleted the hypeship/fix-neko-release-apt branch October 9, 2026 17:32
sjmiller609 added a commit to kernel/kernel-images that referenced this pull request Oct 10, 2026
## Summary
- Upgrade bundled Docker tooling from 29.1.3 to 29.8.2 in headful and
headless images.
- Replace Docker's bundled containerd 2.3.6 (which still embeds gRPC
1.80.0) with containerd 2.4.1. Download only the daemon and runc shim,
verifying pinned SHA-256 hashes for amd64 and arm64.
- Pin the headful Neko base to the published 3.0.8-v1.6.2 release,
including the crypto/network updates and compatible native builder from
kernel/neko#27.
- Leave API dependencies unchanged: main already resolves x/crypto
v0.54.0, x/net v0.57.0, and gRPC v1.82.1.

## Verification
- API build and race-enabled unit suite passed (excluding browser e2e).
- Built the Docker tooling stage and verified both containerd archive
checksums.
- Inspected copied Go binaries: containerd/shim use gRPC v1.83.2 and
x/net v0.58.0; dockerd uses gRPC v1.83.2, x/crypto v0.57.0, and x/net
v0.59.0; runc uses x/net v0.55.0.
- Validated the existing daemon.json with the new dockerd.
- Started Docker 29.8.2 with containerd 2.4.1 in an isolated privileged
container and successfully ran a BusyBox container.
- Full browser image builds, browser e2e, and native arm64 execution
were not run locally.

## Published base verification
- Neko `v3.0.8-v1.6.2` was successfully built and published for amd64,
arm64, and arm/v7:
https://github.com/kernel/neko/actions/runs/37966960969.
- Pulled each published architecture by digest and inspected its Neko
binary: all embed x/crypto v0.54.0 and x/net v0.57.0.
- Booted the published amd64 base: Neko, Xorg, and PulseAudio are
RUNNING; health succeeds and the web client returns HTTP 200.
- Both headful and headless image builds, race-enabled server unit
tests, browser e2e, live-view, and launcher checks pass on the updated
pin: https://github.com/kernel/kernel-images/actions/runs/37973078200.
The first e2e attempt failed the executor-tab cleanup assertion; the
complete rerun passed without code changes. No production browser image
was deployed.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes privileged in-image Docker/containerd runtime versions used
when `WITHDOCKER` is enabled; mis-versioned or incompatible binaries
could break nested container startup despite checksum pinning.
> 
> **Overview**
> Upgrades the **Docker-in-Docker** stage in both chromium headful and
headless images from `docker:29.1.3-dind` to **`29.8.2-dind`**, and
during that stage runs a new **`shared/docker/update-containerd.sh`**
script that downloads **containerd 2.4.1** static binaries (daemon +
`containerd-shim-runc-v2`) with **pinned SHA-256 checks** for
amd64/arm64, installing them over Docker’s bundled containerd to pick up
a newer embedded gRPC stack.
> 
> The headful image also pins the **Neko base** from `3.0.8-v1.6.0` to
**`3.0.8-v1.6.1`**; the final images still copy the same Docker binaries
from the `docker` stage into `/usr/local/bin`.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
14b7267. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: sjmiller609 <7516283+sjmiller609@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants