Update docs to changelog, update setup

.github/workflows/release.yml

@@ -19,7 +19,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@v6.4.0
         with:
-          go-version: "1.26.2"
+          go-version-file: go.mod
 
       - name: Building Ghostferry
         run: .github/workflows/build-deb.sh

.github/workflows/tests.yml

@@ -32,7 +32,7 @@ jobs:
       - uses: actions/checkout@v6.0.2
       - uses: actions/setup-go@v6.4.0
         with:
-          go-version: "1.26.2"
+          go-version-file: go.mod
 
       - name: Starting up MySQL
         run: .github/workflows/start-mysql.sh
@@ -57,7 +57,7 @@ jobs:
       - uses: actions/checkout@v6.0.2
       - uses: actions/setup-go@v6.4.0
         with:
-          go-version: "1.26.2"
+          go-version-file: go.mod
 
       - name: Starting up MySQL
         run: .github/workflows/start-mysql.sh
@@ -88,7 +88,7 @@ jobs:
       - uses: actions/checkout@v6.0.2
       - uses: actions/setup-go@v6.4.0
         with:
-          go-version: "1.26.2"
+          go-version-file: go.mod
       - uses: ruby/setup-ruby@v1
         with:
           bundler-cache: true
@@ -108,7 +108,7 @@ jobs:
       - uses: actions/checkout@v6.0.2
       - uses: actions/setup-go@v6.4.0
         with:
-          go-version: "1.26.2"
+          go-version-file: go.mod
 
       - name: Building Ghostferry
         run: .github/workflows/build-deb.sh --tagged-only

.tool-versions

@@ -0,0 +1,2 @@
+golang 1.26.2
+ruby 3.4.8

README.md

@@ -15,15 +15,19 @@ database from one machine to another.
 Talk to us on IRC at [irc.freenode.net #ghostferry](https://webchat.freenode.net/?channels=#ghostferry).
 
 - **Tutorial and General Documentations**: https://shopify.github.io/ghostferry
-- Code documentations: https://godoc.org/github.com/Shopify/ghostferry
+- Code documentations: https://pkg.go.dev/github.com/Shopify/ghostferry
+  (versioned API docs; for guides tracking `main`, the source in this
+  repository is authoritative)
 
 Overview of How it Works
 ------------------------
 
-An overview of Ghostferry's high-level design is expressed in the [TLA+
-specification](https://en.wikipedia.org/wiki/TLA%2B), under the `tlaplus` directory. It may be good to consult with
-that as it has a concise definition. However, the specification might not be
-entirely correct as proofs remain elusive.
+A simplified model of Ghostferry's high-level copy algorithm is written in
+[TLA+](https://en.wikipedia.org/wiki/TLA%2B) under the `tlaplus` directory,
+together with a TLC model configuration in `tlaplus/ghostferry.toolbox`. It is
+a small finite model with explicitly stated simplifying assumptions (see the
+comment at the top of `tlaplus/ghostferry.tla`); model checking it is not a
+proof of correctness of the current Go implementation.
 
 On a high-level, Ghostferry is broken into several components, enabling it to
 copy data. This is documented at
@@ -52,34 +56,122 @@ The build writes the site to `build/docs/`; `htmlproofer` fails on broken
 internal links or anchors. The live preview is served at
 http://127.0.0.1:4000/ghostferry/main/. None of these commands deploy anything.
 
+The [Changelog](docs/changelog.md) page is populated at build time from the
+root `CHANGELOG.md`, which is the only file to edit for release notes;
+`docs/changelog.md` is just a landing page for readers browsing the source on
+GitHub. Build and serve with the commands above. The Jekyll watcher only
+watches `docs/`, so restart `dev docs` / `jekyll serve` after editing the root
+`CHANGELOG.md`.
+
 Development Setup
 -----------------
 
 ### Installation
 
+#### Prerequisites
+
+- Go 1.26.2 (the `go` directive in `go.mod` is authoritative), Git, Make and
+  a MySQL client, to build and run `ghostferry-copydb`.
+- For the tests and the documentation site additionally: Ruby 3.4.8
+  (`.ruby-version`), Bundler 4.0.10 (`Gemfile.lock`), a C compiler toolchain
+  and the MySQL client development libraries needed to compile the `mysql2`
+  gem. Run `bundle install` without excluding the test, development or docs
+  groups; `test/test_helper.rb` loads `pry-byebug` from the development group
+  unless `CI` is set.
+- Docker (or Podman with `podman-compose`) for the local MySQL servers.
+
+Go and Ruby versions are pinned in `.tool-versions`, which both
+[mise](https://mise.jdx.dev/) and [asdf](https://asdf-vm.com/) read.
+
 #### For Internal Contributors
 
 `dev up`
 
 #### For External Contributors
 
-- Have Docker installed
-- Clone the repo
-- `docker-compose up -d`
-- `nix-shell`
+Install Go and Ruby with mise or asdf from the repository root:
+
+```sh
+mise install    # or: asdf install
+```
+
+Without a version manager, any Go 1.21 or newer also works: because `go.mod`
+requires Go 1.26.2, the `go` command downloads and uses that toolchain itself.
+
+Install the MySQL client and its development libraries (needed by the `mysql2`
+gem), for example `brew install mysql-client` on macOS or
+`apt install default-mysql-client default-libmysqlclient-dev` on Debian/Ubuntu,
+then install the gems:
+
+```sh
+bundle install
+```
+
+Homebrew's `mysql-client` is keg-only; if `mysql2` cannot find it, run
+`bundle config set build.mysql2 --with-mysql-config="$(brew --prefix mysql-client)/bin/mysql_config"`
+first.
+
+Start two disposable MySQL 8.0 servers from the repository root:
+
+```sh
+docker compose -f docker-compose_8.0.yml up -d mysql-1 mysql-2
+# or: podman-compose -f docker-compose_8.0.yml up -d mysql-1 mysql-2
+```
+
+They listen on ports 29291 (source) and 29292 (target) with a passwordless
+`root` account. They are throwaway test servers, not a template for production
+credentials. Wait until both accept connections:
+
+```sh
+mysql --protocol=tcp -u root -P 29291 -e 'SELECT 1'
+mysql --protocol=tcp -u root -P 29292 -e 'SELECT 1'
+```
+
+Build `ghostferry-copydb` into the first `GOPATH` entry's `bin` directory:
+
+```sh
+export GOPATH="$(go env GOPATH)"
+export PATH="${GOPATH%%:*}/bin:$PATH"
+make copydb
+```
+
+Run the binary from the repository root: its web UI templates are loaded from
+`webui/` below `ControlServerConfig.WebBasedir`, which defaults to `.`.
+Debian packages built by `make copydb-deb` instead compile in the base
+directory `/usr/share/ghostferry` and install `webui/` beneath it; like `.`
+for source builds, the base directory is the parent of `webui/`, not the
+`webui` directory itself. Packaged builds are published on the project's
+[GitHub Releases](https://github.com/Shopify/ghostferry/releases) page; most
+of them are prereleases (see [Releasing new version](#releasing-new-version)).
 
 Testing
 ---------------
 
+Export `MYSQL_VERSION=8.0` when running tests against the MySQL 8.0 servers
+above.
+
 #### Run all tests
 
 - `make test`
 
 #### Run example copydb usage
 
-- `make copydb && ghostferry-copydb -verbose examples/copydb/conf.json`
-- For a more detailed tutorial, see the
-  [documentation](https://shopify.github.io/ghostferry).
+`examples/copydb/conf.json` copies the `abc` database created by the
+[copydb tutorial](docs/tutorialcopydb.md): seed the source with the tutorial's
+SQL first, and make sure the target has no `abc` tables for a fresh run. Then,
+from the repository root:
+
+```sh
+ghostferry-copydb -verbose examples/copydb/conf.json
+```
+
+This example uses the `Inline` verifier, binds the UI to
+`127.0.0.1:8000` and adds two Custom Script buttons. It sets
+`"SkipTargetVerification": true`, which disables target-write monitoring; the
+tutorial intentionally keeps the protected default.
+
+For a more detailed walkthrough, see the
+[documentation](https://shopify.github.io/ghostferry).
 
 ### Ruby Integration Tests
 
@@ -92,19 +184,19 @@ Examples:
 
 Run all tests
 
-`rake test`
+`bundle exec rake test`
 
 Run a single file
 
-`rake test TEST=test/integration/trivial_test.rb`
+`bundle exec rake test TEST=test/integration/trivial_test.rb`
 
 or
 
-`ruby -Itest test/integration/trivial_test.rb`
+`bundle exec ruby -Itest test/integration/trivial_test.rb`
 
 Run a specific test
 
-`DEBUG=1 ruby -Itest test/integration/trivial_test.rb -n "TrivialIntegrationTest#test_logged_query_omits_columns"`
+`DEBUG=1 bundle exec ruby -Itest test/integration/trivial_test.rb -n 'TrivialIntegrationTest#test_logged_query_omits_columns'`
 
 Releasing new version
 ---------------------
@@ -118,10 +210,12 @@ git tag --sign --message="Initial support for UUIDs as pagination keys" canary/v
 git push origin --tags
 ```
 
-This will create the release named by tag.
+This creates a GitHub prerelease named after the tag.
 
 ### Production
 
-Final releases are created automatically on merge to `main` branch, they will end up with `release-SHA` name.
+Every push to the `main` branch creates a GitHub **prerelease** named
+`release-<first seven characters of the commit SHA>`.
 
-Remember to update version prior to bigger releases in `Makefile` along with updating the `CHANGELOG.md`.
+Remember to update `VERSION` in `Makefile` along with the root `CHANGELOG.md`
+prior to releases.

benchmark/README.md

@@ -2,7 +2,73 @@ Ghostferry benchmark setup
 ==========================
 
 A benchmark "library" is provided with `./ghostferry_benchmark.rb`. An example
-benchmark case can be seen with `./studies/batch_size.rb` and the post
-processing is provided in `./studies/batch_size_vs_row_size.ipynb`. 
+benchmark case can be seen with `./studies/batch_size.rb`, which measures the
+copy speed for several row sizes and batch sizes.
 
 Some modifications are needed to make this benchmark work for other studies.
+
+Prerequisites
+-------------
+
+Follow the Development Setup in the repository's [README](../README.md):
+
+- Start the local MySQL 8.0 servers from the repository root:
+
+  ```sh
+  docker compose -f docker-compose_8.0.yml up -d mysql-1 mysql-2
+  # or: podman-compose -f docker-compose_8.0.yml up -d mysql-1 mysql-2
+  ```
+
+  The harness connects to them as passwordless `root` on ports 29291 (source)
+  and 29292 (target).
+
+  **Warning:** the harness drops and recreates the `benchmark` schema on the
+  source and drops it on the target. Only point it at disposable servers.
+
+- Build `ghostferry-copydb` and put it on your `PATH`; the harness runs the
+  `ghostferry-copydb` found there:
+
+  ```sh
+  export GOPATH="$(go env GOPATH)"
+  export PATH="${GOPATH%%:*}/bin:$PATH"
+  make copydb
+  ```
+
+- Install the gems with `bundle install`, including the test and development
+  groups of the root `Gemfile` (`mysql2`, `webrick`, `tqdm`).
+
+- Ports 8000 (the Ghostferry web UI) and 8001 (the harness's progress callback
+  server) must be free on 127.0.0.1.
+
+Running the batch size study
+----------------------------
+
+From the `benchmark/` directory:
+
+```sh
+bundle exec ruby studies/batch_size.rb
+```
+
+For each row size, the study seeds `benchmark.t` on the source, then for each
+batch size wipes the target, runs `ghostferry-copydb` for 15 seconds, stops it
+and computes the average copy speed from the progress callbacks. The generated
+configuration sets `ControlServerConfig.WebBasedir` to the repository root, so
+the web UI is found although the study runs from `benchmark/`.
+
+Outputs, relative to `benchmark/`:
+
+- `out/rs=<row size>-bs=<batch size>/conf.json`: the Ghostferry configuration;
+- `out/rs=<row size>-bs=<batch size>/ghostferry.log`: Ghostferry's output;
+- `out/rs=<row size>-bs=<batch size>/progress.json.log`: the progress callbacks;
+- `out/rs=<row size>-bs=<batch size>/rows_written.csv`: time taken, rows
+  written and state per progress callback;
+- `studies/batch_size_benchmark.csv`: `row size,batch size,rows/s` per case.
+
+Analysis notebook
+-----------------
+
+`studies/batch_size_vs_row_size.ipynb` analyses the checked-in historical
+results in `studies/benchmark.csv`; it is not fed automatically by
+`studies/batch_size.rb`. It needs Jupyter, NumPy, Matplotlib and SciPy, and must
+be run with `benchmark/studies` as its working directory. To analyse new results,
+change its CSV input, plot ranges and interpolation domain to match your study.

benchmark/ghostferry_benchmark.rb

@@ -144,7 +144,13 @@ def default_ghostferry_config
         "DumpStateOnSignal" => false,
         "VerifierType" => "Inline",
         "SkipTargetVerification" => true,
-        "DataIterationBatchSize" => 200
+        "UpdatableConfig" => {
+          "DataIterationBatchSize" => 200
+        },
+        "ControlServerConfig" => {
+          "WebBasedir" => File.expand_path("..", __dir__),
+          "ServerBindAddr" => "127.0.0.1:8000"
+        }
       }
     end
 

benchmark/studies/batch_size.rb

@@ -25,7 +25,7 @@
       FileUtils.mkdir_p(output_dir)
 
       config = GhostferryBenchmark.default_ghostferry_config
-      config["DataIterationBatchSize"] = batch_size
+      config["UpdatableConfig"]["DataIterationBatchSize"] = batch_size
 
       GhostferryBenchmark::Databases.wipe_target
       speed = GhostferryBenchmark.run_ghostferry(ghostferry_config: config, output_dir: output_dir)