Skip to content

test: describe hardware scenarios in Gherkin, play them on the adapters #105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
g-carre wants to merge 3 commits into
base: main
Choose a base branch
from
Open

test: describe hardware scenarios in Gherkin, play them on the adapters #105

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3ecd32d
test: describe hardware scenarios in Gherkin, play them on MegaRAID
g-carre Oct 12, 2026
61c993b
test(scenario): play the disk failure scenarios on storcli2 controllers
g-carre Oct 12, 2026
11b2a13
test(scenario): play the disk failure scenarios on HPE Smart Array co…
g-carre Oct 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/pre-merge.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ jobs:
go-version: ${{ env.GO_VERSION }}
- name: Set list of modules
id: set-modules
run: echo "modules=$(go list -m -json | jq -s '.' | jq -c '[.[].Dir]')" >> $GITHUB_OUTPUT
# The root module and the nested ones, such as scenario/.
run: echo "modules=$(find "$PWD" -name go.mod -not -path '*/bin/*' -printf '%h\n' | sort | jq -R . | jq -sc .)" >> $GITHUB_OUTPUT

golangci-lint:
needs: detect-modules
Expand Down Expand Up @@ -57,7 +58,8 @@ jobs:
with:
go-version: ${{ env.GO_VERSION }}
- name: Run tests
run: go test -v ${{ matrix.modules }}/...
working-directory: ${{ matrix.modules }}
run: go test -v ./...

build-e2e:
# The on-hardware integration harnesses are not run in CI (they need real
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,8 @@ jobs:
with:
go-version-file: go.mod
- name: Run tests
run: go test -v ./...
# Every module, including the nested scenario/ one.
run: make tests

release:
needs: quality-gate
Expand Down
2 changes: 2 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,15 @@ E2E_BUILD_ENV := CGO_ENABLED=0 GOOS=$(E2E_GOOS) GOARCH=$(E2E_GOARCH)
lint:
@echo "Linting..."
golangci-lint run -c .golangci.yaml ./...
cd scenario && golangci-lint run -c ../.golangci.yaml ./...
@echo "Lint done"

# TODO
.PHONY: tests
tests:
@echo "Running tests..."
go test -v ./...
cd scenario && go test -v ./...
@echo "Tests done"

# Build the on-hardware integration harnesses as static linux/amd64 binaries
Expand Down
109 changes: 109 additions & 0 deletions scenario/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Hardware scenarios

Replay what happens on a RAID controller, such as a drive failing and being
replaced, from real controller output, and check what raidmgmt reports,
without the hardware.

Scenarios are Gherkin features in [`features/`](features), one per
controller family and story, tagged with the family (`@megaraid`,
`@storcli2`, `@ssacli`). The `Background` names the capture of a real
controller; each `Scenario` plays events on its drives and gives, as a
table, the disks expected:

```gherkin
@megaraid
Feature: MegaRAID disk failure and replacement

Background:
Given the controller captured as "megaraid9560-4hdd"

Scenario: A failed drive does not take the paths of the others
When drive 251:3 fails
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 251:1 | Used | SIMD0001000000000000 | /dev/sdl | /dev/disk/by-id/wwn-0x6000… |
| 251:3 | Failed | SIMD0003 | | |
But known bug ARTESCA-17960 reports instead:
| slot | status | serial | devicePath | permanentPath |
| 251:1 | Used | SIMD0001000000000000 | | |
| 251:3 | Failed | SIMD0003 | | |
```

## Steps

| Step | Event |
|---|---|
| `Given the controller captured as "<capture>"` | Start from a capture |
| `drive <slot> fails` / `has failed` | The drive fails; its volume goes offline, no longer a block device |
| `drive <slot> is replaced` / `has been replaced by a drive with serial "<serial>" and WWN "<wwn>"` | Its volume is deleted and a new drive, in no volume, takes the slot |
| `a volume is created` / `has been created on drive <slot>, exposed as "<device>" with WWN "<wwn>"` | The volume the drive had in the capture is created again on it |
| `Then the drives are reported as:` | Check the table: status, serial, and the device and permanent paths of the volume holding each drive. An empty cell is an empty value |
| `But known bug <ticket> reports instead:` | Right after a `Then`: the table reported today because of the bug |

- Each scenario stands alone: it replays its own events on the capture, the
ones it depends on written as `Given` steps.
- Events are written the same for every controller; each backend applies
them as its controller shows them (states, serials, drive groups), as
documented in its package. A backend rejects events in an order that
cannot happen: only a drive in a volume fails, only a failed drive is
replaced, and a volume is created only once the old one is deleted.
- A known bug passes while its table is reported. It fails when the
expected table is reported (the bug is fixed: remove the step) and when
neither is (the behaviour changed).
- Steps are run by [godog](https://github.com/cucumber/godog), strictly: a
step with no definition fails. When a table differs, the error prints the
table reported, ready to paste.

## Where scenarios are played

- `megaraidsim/adapter_test.go` plays every `@megaraid` scenario (storcli64)
through the raidmgmt megaraid adapter of this repository, with
`scenario.Run`.
- `storcli2sim/adapter_test.go` plays every `@storcli2` scenario through the
raidmgmt storcli2 adapter.
- `ssaclisim/adapter_test.go` plays every `@ssacli` scenario (HPE Smart
Array) through the raidmgmt ssacli getters.
- `steps_internal_test.go` checks the steps themselves on the features of
[`testdata/`](testdata), in particular that they fail when they should.

This is a separate module (`github.com/scality/raidmgmt/scenario`), which
uses the raidmgmt code of this repository through a `replace`: godog and
the captures stay out of the dependencies and the downloads of raidmgmt.

## Adding a scenario

Add a `Scenario` to a feature, or a `.feature` file in `features/` tagged
with its controller family, on an existing capture, and run `go test ./...`
in this directory. A new kind of event needs a step in `steps.go` and its
handling in every backend.

## Adding a controller

1. Capture its CLI output on a lab machine, one file per command the adapter
runs, in `<backend>/captures/<name>/`.
2. Write a backend that answers those commands from the capture with the
events applied, as [`megaraidsim`](megaraidsim) does for storcli64, and
gives the host files the adapter checks.
3. Add a test calling `scenario.Run` with the tag of that controller and a
player reporting through its adapter; `scenario.Report` turns what an
adapter reports into the disks of a table.

## Captures

- `megaraidsim/captures/megaraid9560-4hdd`: MegaRAID 9560-8i, SEAGATE
ST18000NM000D drives in enclosure 251, one RAID0 volume each (storcli64
007.1616). Captured with `storcli64 <selector> show all J` on 12 drives,
kept to the first 4 (251:1-4, volumes 239-236) to keep scenarios short.
Serials, WWNs and controller identifiers are replaced by fake ones, also in
the hex inquiry data; keep it so when adding a capture.
- `storcli2sim/captures/storcli2-4hdd`: storcli2 controller, SEAGATE
ST10000NM018B drives in enclosure 306, one RAID0 volume each (storcli2
008.0005). Captured with `storcli2 <selector> show all J` on 24 drives,
kept to the first 4 (306:0-3, volumes 1-4), with the same fake identifiers
(drive serials, WWNs and SAS addresses, volume NAA Ids and serials).
- `ssaclisim/captures/ssacli-4hdd`: HPE Smart Array P816i-a, SAS HDDs each
in its own RAID0 array. Captured with `ssacli controller slot=0 ... show
detail` and `show config`, kept to 4 drives (1I:1:3, 1I:1:4, 2I:2:1,
2I:2:2, logical drives 2-5), with the same fake identifiers (serials,
WWIDs, unique identifiers, controller and host serials). ssacli prints
text: the backend edits the lines of the captured sections.
61 changes: 61 additions & 0 deletions scenario/features/megaraid-disk-failure.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
@megaraid
Feature: MegaRAID disk failure and replacement
A MegaRAID 9560-8i with HDDs in enclosure 251, each in its own RAID0
volume, as ARTESCA deploys them. Drive 251:3 fails, is replaced, then gets a
new volume (RD-2154, ARTESCA-17960).

The capture is real storcli output of a lab node while all its drives
were healthy, kept to its first 4 drives, with serials and WWNs replaced by
fake ones.

Each table lists, per drive, what raidmgmt must report: the drive, and
the paths of the volume holding it.

Background:
Given the controller captured as "megaraid9560-4hdd"

Scenario: All drives are healthy
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 251:1 | Used | SIMD0001000000000000 | /dev/sdl | /dev/disk/by-id/wwn-0x600062b2000000000000000000000239 |
| 251:2 | Used | SIMD0002000000000000 | /dev/sdk | /dev/disk/by-id/wwn-0x600062b2000000000000000000000238 |
| 251:3 | Used | SIMD0003000000000000 | /dev/sdj | /dev/disk/by-id/wwn-0x600062b2000000000000000000000237 |
| 251:4 | Used | SIMD0004000000000000 | /dev/sdi | /dev/disk/by-id/wwn-0x600062b2000000000000000000000236 |

Scenario: A failed drive does not take the paths of the others
When drive 251:3 fails
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 251:1 | Used | SIMD0001000000000000 | /dev/sdl | /dev/disk/by-id/wwn-0x600062b2000000000000000000000239 |
| 251:2 | Used | SIMD0002000000000000 | /dev/sdk | /dev/disk/by-id/wwn-0x600062b2000000000000000000000238 |
| 251:3 | Failed | SIMD0003 | | |
| 251:4 | Used | SIMD0004000000000000 | /dev/sdi | /dev/disk/by-id/wwn-0x600062b2000000000000000000000236 |
# The offline volume has no by-id link: its path cannot be resolved, which
# fails the listing of all the volumes of the controller.
But known bug ARTESCA-17960 reports instead:
| slot | status | serial | devicePath | permanentPath |
| 251:1 | Used | SIMD0001000000000000 | | |
| 251:2 | Used | SIMD0002000000000000 | | |
| 251:3 | Failed | SIMD0003 | | |
| 251:4 | Used | SIMD0004000000000000 | | |

Scenario: A replacement drive is reported as a new, available drive
Given drive 251:3 has failed
When drive 251:3 is replaced by a drive with serial "SIMD0013000000000000" and WWN "5000C50000000013"
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 251:1 | Used | SIMD0001000000000000 | /dev/sdl | /dev/disk/by-id/wwn-0x600062b2000000000000000000000239 |
| 251:2 | Used | SIMD0002000000000000 | /dev/sdk | /dev/disk/by-id/wwn-0x600062b2000000000000000000000238 |
| 251:3 | UnassignedGood | SIMD0013000000000000 | | |
| 251:4 | Used | SIMD0004000000000000 | /dev/sdi | /dev/disk/by-id/wwn-0x600062b2000000000000000000000236 |

Scenario: A replacement drive gets the paths of its new volume
Given drive 251:3 has failed
And drive 251:3 has been replaced by a drive with serial "SIMD0013000000000000" and WWN "5000C50000000013"
When a volume is created on drive 251:3, exposed as "/dev/sdm" with WWN "600062b2000000000000000000001237"
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 251:1 | Used | SIMD0001000000000000 | /dev/sdl | /dev/disk/by-id/wwn-0x600062b2000000000000000000000239 |
| 251:2 | Used | SIMD0002000000000000 | /dev/sdk | /dev/disk/by-id/wwn-0x600062b2000000000000000000000238 |
| 251:3 | Used | SIMD0013000000000000 | /dev/sdm | /dev/disk/by-id/wwn-0x600062b2000000000000000000001237 |
| 251:4 | Used | SIMD0004000000000000 | /dev/sdi | /dev/disk/by-id/wwn-0x600062b2000000000000000000000236 |
51 changes: 51 additions & 0 deletions scenario/features/ssacli-disk-failure.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
@ssacli
Feature: ssacli disk failure and replacement
The ssacli counterpart of megaraid-disk-failure: an HPE Smart Array
P816i-a with HDDs, each in its own RAID0 array, as ARTESCA deploys them.
Drive 1I:1:4 (array C, logical drive 3) fails, is replaced, then gets a new
array (ARTESCA-17960).

The capture is real ssacli output of a lab node while all its drives were
healthy, kept to 4 drives, with serials, WWIDs and identifiers replaced by
fake ones. ssacli getters report no permanent path.

Background:
Given the controller captured as "ssacli-4hdd"

Scenario: All drives are healthy
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 1I:1:3 | Used | SIMH0001 | /dev/sdd | |
| 1I:1:4 | Used | SIMH0002 | /dev/sde | |
| 2I:2:1 | Used | SIMH0003 | /dev/sdf | |
| 2I:2:2 | Used | SIMH0004 | /dev/sdg | |

Scenario: A failed drive does not take the paths of the others
When drive 1I:1:4 fails
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 1I:1:3 | Used | SIMH0001 | /dev/sdd | |
| 1I:1:4 | Failed | SIMH0002 | | |
| 2I:2:1 | Used | SIMH0003 | /dev/sdf | |
| 2I:2:2 | Used | SIMH0004 | /dev/sdg | |

Scenario: A replacement drive is reported as a new, available drive
Given drive 1I:1:4 has failed
When drive 1I:1:4 is replaced by a drive with serial "SIMH0013" and WWN "5000C50030000013"
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 1I:1:3 | Used | SIMH0001 | /dev/sdd | |
| 1I:1:4 | UnassignedGood | SIMH0013 | | |
| 2I:2:1 | Used | SIMH0003 | /dev/sdf | |
| 2I:2:2 | Used | SIMH0004 | /dev/sdg | |

Scenario: A replacement drive gets the paths of its new array
Given drive 1I:1:4 has failed
And drive 1I:1:4 has been replaced by a drive with serial "SIMH0013" and WWN "5000C50030000013"
When a volume is created on drive 1I:1:4, exposed as "/dev/sdh" with WWN "600508B1000000000000000000001003"
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 1I:1:3 | Used | SIMH0001 | /dev/sdd | |
| 1I:1:4 | Used | SIMH0013 | /dev/sdh | |
| 2I:2:1 | Used | SIMH0003 | /dev/sdf | |
| 2I:2:2 | Used | SIMH0004 | /dev/sdg | |
59 changes: 59 additions & 0 deletions scenario/features/storcli2-disk-failure.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
@storcli2
Feature: storcli2 disk failure and replacement
The storcli2 counterpart of megaraid-disk-failure: a controller driven by
storcli2 with HDDs in enclosure 306, each in its own RAID0 volume, as
ARTESCA deploys them. Drive 306:2 fails, is replaced, then gets a new
volume (ARTESCA-17960).

The capture is real storcli2 output of a lab node while all its drives
were healthy, kept to its first 4 drives, with serials, WWNs and volume
identifiers replaced by fake ones.

Background:
Given the controller captured as "storcli2-4hdd"

Scenario: All drives are healthy
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 306:0 | Used | SIMD2001000000000000 | /dev/sdb | /dev/disk/by-id/wwn-0x600062b2000000000000000000000001 |
| 306:1 | Used | SIMD2002000000000000 | /dev/sdc | /dev/disk/by-id/wwn-0x600062b2000000000000000000000002 |
| 306:2 | Used | SIMD2003000000000000 | /dev/sdd | /dev/disk/by-id/wwn-0x600062b2000000000000000000000003 |
| 306:3 | Used | SIMD2004000000000000 | /dev/sde | /dev/disk/by-id/wwn-0x600062b2000000000000000000000004 |

Scenario: A failed drive does not take the paths of the others
When drive 306:2 fails
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 306:0 | Used | SIMD2001000000000000 | /dev/sdb | /dev/disk/by-id/wwn-0x600062b2000000000000000000000001 |
| 306:1 | Used | SIMD2002000000000000 | /dev/sdc | /dev/disk/by-id/wwn-0x600062b2000000000000000000000002 |
| 306:2 | Failed | SIMD2003000000000000 | | |
| 306:3 | Used | SIMD2004000000000000 | /dev/sde | /dev/disk/by-id/wwn-0x600062b2000000000000000000000004 |
# The permanent path of the offline volume is still built from its NAA
# Id, although the volume has no block device and no by-id link.
But known bug ARTESCA-17960 reports instead:
| slot | status | serial | devicePath | permanentPath |
| 306:0 | Used | SIMD2001000000000000 | /dev/sdb | /dev/disk/by-id/wwn-0x600062b2000000000000000000000001 |
| 306:1 | Used | SIMD2002000000000000 | /dev/sdc | /dev/disk/by-id/wwn-0x600062b2000000000000000000000002 |
| 306:2 | Failed | SIMD2003000000000000 | | /dev/disk/by-id/wwn-0x600062b2000000000000000000000003 |
| 306:3 | Used | SIMD2004000000000000 | /dev/sde | /dev/disk/by-id/wwn-0x600062b2000000000000000000000004 |

Scenario: A replacement drive is reported as a new, available drive
Given drive 306:2 has failed
When drive 306:2 is replaced by a drive with serial "SIMD2013000000000000" and WWN "5000C50020000013"
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 306:0 | Used | SIMD2001000000000000 | /dev/sdb | /dev/disk/by-id/wwn-0x600062b2000000000000000000000001 |
| 306:1 | Used | SIMD2002000000000000 | /dev/sdc | /dev/disk/by-id/wwn-0x600062b2000000000000000000000002 |
| 306:2 | UnassignedGood | SIMD2013000000000000 | | |
| 306:3 | Used | SIMD2004000000000000 | /dev/sde | /dev/disk/by-id/wwn-0x600062b2000000000000000000000004 |

Scenario: A replacement drive gets the paths of its new volume
Given drive 306:2 has failed
And drive 306:2 has been replaced by a drive with serial "SIMD2013000000000000" and WWN "5000C50020000013"
When a volume is created on drive 306:2, exposed as "/dev/sdf" with WWN "600062b2000000000000000000001003"
Then the drives are reported as:
| slot | status | serial | devicePath | permanentPath |
| 306:0 | Used | SIMD2001000000000000 | /dev/sdb | /dev/disk/by-id/wwn-0x600062b2000000000000000000000001 |
| 306:1 | Used | SIMD2002000000000000 | /dev/sdc | /dev/disk/by-id/wwn-0x600062b2000000000000000000000002 |
| 306:2 | Used | SIMD2013000000000000 | /dev/sdf | /dev/disk/by-id/wwn-0x600062b2000000000000000000001003 |
| 306:3 | Used | SIMD2004000000000000 | /dev/sde | /dev/disk/by-id/wwn-0x600062b2000000000000000000000004 |
27 changes: 27 additions & 0 deletions scenario/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
module github.com/scality/raidmgmt/scenario

go 1.25.6

require (
github.com/cucumber/godog v0.16.0
github.com/pkg/errors v0.9.1
github.com/scality/raidmgmt v0.14.0
github.com/stretchr/testify v1.11.1
)

require (
github.com/cucumber/gherkin/go/v42 v42.0.0 // indirect
github.com/cucumber/messages/go/v34 v34.2.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-memdb v1.3.5 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/spf13/pflag v1.0.10 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

// Play the scenarios on the adapters of this repository.
replace github.com/scality/raidmgmt => ../
Loading
Loading