Skip to content

Ship the benchup command with BenchUp releases #85

Open

Description

@furilo

Follow-up to #55 and #73. The live bridge and the in-app Agent connection are on main, but an ordinary user still cannot use them, because the benchup command is not shipped with BenchUp. Today the only way to get it is to build it from the source repository with cargo install --locked --path crates/benchup-cli, which assumes a Rust toolchain and a checkout. The Agent connection panel already tells people to "install the benchup command", and site/llms.txt says it is "not in the desktop releases yet", but neither can point to a download.

Current state

The release workflow (.github/workflows/build.yml) builds only benchup-app --bin benchup-app and stages that single executable into package/ before scripts/desktop_release.py package runs vpk pack. The CLI crate is compiled and tested in CI (-p benchup-tools -p benchup-cli), so the binary already builds on every platform we release for; it is just never packaged.

Proposal

I suggest we do both of the following, since they serve different users.

  1. Ship benchup inside the desktop packages. We would build -p benchup-cli --release next to the app in the same job and copy the executable into package/, so that Velopack installs and updates it together with the app. That keeps the CLI and the app on the same version, which matters because the bridge protocol and the tool layer are shared between them. The command then has to be reachable from a terminal, and each platform needs its own approach:
    • On macOS, the binary would live inside BenchUp.app/Contents/MacOS/. The app could offer an "Install command line tool" action that creates a symlink in a directory on the PATH (for example /usr/local/bin, or ~/.local/bin to avoid asking for administrator rights), similar to what code editors do. It has to be signed and notarized together with the app.
    • On Windows, Velopack installs under %LocalAppData%, so the installer or the app could add the current directory to the user's PATH.
    • On Linux, the AppImage cannot add itself to the PATH, so the same in-app action could copy or link the binary into ~/.local/bin.
  2. Publish standalone CLI archives on each GitHub Release (benchup-<version>-<platform>.tar.gz / .zip). These cover people who use the CLI with files only, without the desktop app, and agents running on machines where only the web app is used. They would be built in the same release job and attached by the same step that publishes the installers.

Whichever way it is installed, the Agent connection panel should detect whether benchup is available and, if it is not, show the right instruction for the platform instead of a generic sentence. Its copied setup commands (claude mcp add … -- benchup bridge and the MCP config) should use the absolute path of the bundled binary when the command is not on the PATH.

Things to decide

  • Should the CLI version be pinned to the app version, and should benchup bridge refuse to pair with an app whose protocol version differs, with a clear message?
  • Do we want a benchup update path for the standalone archives, or is "download the new release" enough at first?
  • Web-only users need the CLI to run the bridge. Is a standalone archive enough for them, or should we also look at Homebrew, Scoop or cargo binstall later?

Acceptance

  • A fresh install of the desktop release on macOS, Windows and Linux lets the user run benchup --version from a terminal, after at most one in-app action.
  • Each GitHub Release has CLI archives for the four desktop targets.
  • docs/cli.md, the Agent connection panel and site/llms.txt describe the new install paths, and the "built from source for now" wording is removed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions