Skip to content

test: describe hardware scenarios in Gherkin, play them on the adapters - #105

Open
g-carre wants to merge 3 commits into
mainfrom
test/ARTESCA-17960-disk-failure-scenario-gherkin
Open

g-carre wants to merge 3 commits into
mainfrom
test/ARTESCA-17960-disk-failure-scenario-gherkin

Conversation

@g-carre

@g-carre g-carre commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Overview

This PR reproduces ARTESCA-17960 from real RAID utility output, without hardware. For each controller utility it records what raidmgmt reports today: MegaRAID and storcli2 show the bug, which is marked as known, and ssacli does not. Each commit covers one controller utility.

The fix, #107, is stacked on this PR so it can be reviewed separately. Both are meant to merge together.

Details

Adds scenario/, a test kit that replays hardware events (a drive fails, is replaced, gets a new volume) on real RAID controller output, and checks what raidmgmt reports for each drive. No hardware is needed. The scenarios are written in Gherkin and run with godog.

The first scenarios reproduce ARTESCA-17960 (RD-2154): on MegaRAID and storcli2, one failed drive blanks the device paths of every drive on the controller. This PR only adds the tests and records the bug as a known bug; #107 removes the known bug steps.

What a scenario looks like

@megaraid
Feature: MegaRAID disk failure and replacement

  Background:
    Given the controller captured as "megaraid9560-4hdd"

  Scenario: A failed drive does not take the paths of the others
    When drive 251:3 fails
    Then the drives are reported as:
      | slot  | status | serial               | devicePath | permanentPath                                          |
      | 251:1 | Used   | SIMD0001000000000000 | /dev/sdl   | /dev/disk/by-id/wwn-0x600062b2000000000000000000000239 |
      | 251:3 | Failed | SIMD0003             |            |                                                        |
      ...
    But known bug ARTESCA-17960 reports instead:
      | slot  | status | serial               | devicePath | permanentPath |
      | 251:1 | Used   | SIMD0001000000000000 |            |               |
      ...
  • The Background names a capture: real output of a lab controller with all drives healthy, cut to 4 drives, with fake serials and WWNs.
  • The steps are the same for every controller. Each simulator applies them the way its controller shows them.
  • Each Scenario is self-contained: its prerequisites are Given steps, and it runs as its own subtest.
  • A But known bug <ticket> reports instead: table after a Then gives what is reported today. The step passes while that table is reported. It fails when the expected table is reported ("the bug looks fixed, remove its known bug step") and when neither is (the behaviour changed). A failure prints the reported table, ready to paste.

Each controller family has 4 scenarios: all drives healthy, one drive failed, the failed drive replaced, and a new volume on the replacement. The failed-drive scenario carries the known bug on MegaRAID and storcli2. ssacli passes.

Commits

One commit per adapter, each passing tests and lint on its own:

  1. Framework and MegaRAID. Harness, steps, the MegaRAID simulator and its capture, the CI changes.
  2. storcli2. Simulator, capture and feature.
  3. HPE Smart Array (ssacli). Simulator, capture and feature.

How to review

Most of the diff is captures: JSON and text under */captures/. You can skim them. Their serials and WWNs are fake.

Suggested order:

  1. scenario/features/*.feature. These are the specification. Check that the expected tables are what raidmgmt should report after each event.
  2. scenario/README.md. Steps, how known bugs work, how to add a scenario or a controller.
  3. scenario/scenario.go, steps.go, report.go. The harness:
    • scenario.go: parsing and comparing tables. It rejects a wrong header, an empty table and duplicate slots.
    • steps.go: the step definitions. Undefined steps fail (strict mode). A known bug step is tied to the Then right before it.
    • report.go: turns the adapter's drives and volumes into rows.
  4. scenario/steps_internal_test.go and scenario/testdata/. They check that the harness fails when it should: a differing table, a missing drive, a fixed or changed known bug, a misplaced known bug, an undefined step, a bad table.
  5. The simulators: megaraidsim, storcli2sim, ssaclisim. Each answers the CLI commands of the raidmgmt adapter from the capture, with the events applied. The package docs say how each controller shows each event. For example, MegaRAID cuts a failed drive's serial to 8 characters, and ssacli moves a replaced drive to Unassigned. */adapter_test.go plays the features through the real raidmgmt adapter.
  6. CI and Makefile. scenario/ is a nested Go module. Module detection now finds every go.mod, tests run in each module's directory, and make tests runs both modules (used by the release workflow).

Points to look at

  • Nested module. scenario/ has its own go.mod, with a replace pointing to the repository root. This keeps godog out of raidmgmt's own dependencies. Its tests use testify, like the root module.
  • Simulator fidelity. No controller output was captured while a drive was failed. The failed and replaced states follow what storcli showed on the RD-2154 node. Each package doc says so.
  • New events. A state that the current steps cannot produce needs a new step in steps.go and its handling in every simulator.

Running locally

cd scenario && go test ./...

Refs: ARTESCA-17960

@g-carre
g-carre changed the base branch from main to test/ARTESCA-17960-megaraid-disk-failure-scenario October 12, 2026 09:50
@g-carre
g-carre changed the base branch from test/ARTESCA-17960-megaraid-disk-failure-scenario to main October 12, 2026 09:51
@g-carre
g-carre force-pushed the test/ARTESCA-17960-disk-failure-scenario-gherkin branch from 92d09e9 to 74a8271 Compare October 12, 2026 14:09
@g-carre
g-carre force-pushed the test/ARTESCA-17960-disk-failure-scenario-gherkin branch 2 times, most recently from eb4261b to ae2f0cd Compare October 15, 2026 21:21
Comment thread scenario/steps_internal_test.go Outdated
@g-carre
g-carre force-pushed the test/ARTESCA-17960-disk-failure-scenario-gherkin branch from ae2f0cd to 3791dcd Compare October 15, 2026 21:37
Comment thread scenario/megaraidsim/megaraidsim.go
@g-carre
g-carre force-pushed the test/ARTESCA-17960-disk-failure-scenario-gherkin branch from 3791dcd to a7ab071 Compare October 16, 2026 12:36
Add github.com/scality/raidmgmt/scenario, to test what raidmgmt reports
when a drive fails, is replaced and gets a new volume, without the hardware.
It is a separate module, using the raidmgmt code of this repository through
a replace, so that godog and the captures stay out of raidmgmt's
dependencies:

- features/*.feature: Gherkin scenarios tagged with their controller
  family. The Background names the capture of a real controller; each
  scenario plays events on its drives (a drive fails, is replaced, gets a
  new volume), the ones it depends on written as Given steps, and gives the
  table of disks expected. When a bug makes it differ today, a known bug
  step gives the table reported because of it.
- scenario: the step definitions, run by godog strictly (undefined steps
  fail, empty tables and duplicate slots are rejected), and the comparison
  of what is reported with a table. A known bug passes only while its table
  is reported; it fails once the expected table is (the bug is fixed) and
  when neither is (the behaviour changed). A drive reported twice is a
  difference. Run plays the features of a controller family through a
  player; Report collects the disks an adapter reports, listing the
  controllers first as disk-management-agent does.
- steps_internal_test.go: checks on the features of testdata/ that the
  steps fail when they should, for the right reason.
- megaraidsim: answers storcli64 commands from a capture with events
  applied as storcli showed them on RD-2154, and shows the
  /dev/disk/by-id links of exposed volumes. Drive groups are written as
  storcli prints them: a number, or "-" for an unconfigured drive. Events
  in an order that cannot happen (a volume created on a drive never
  replaced, a healthy drive replaced) are rejected, and a capture listing
  an unknown drive or volume is an error, not a panic.
- megaraidsim/adapter_test.go: plays every @MegaRAID scenario through the
  megaraid adapter of this repository.

The first feature, megaraid-disk-failure, replays real storcli output of a
lab node, kept to its first 4 single-drive RAID0 volumes and with fake
serials and WWNs: all drives healthy, then drive 251:3 failing, being
replaced and getting a new volume. The failure scenario documents
ARTESCA-17960: the offline volume of the failed drive makes LogicalVolumes
fail for the whole controller, so no drive gets a path.

CI and the release gate now test and lint every module.

Refs: ARTESCA-17960
Add storcli2sim, the backend of storcli2/perccli2 controllers: it answers the
commands of the raidmgmt storcli2 adapter (controller listing, drives and
volumes) from a capture with events applied. storcli2 reports a drive
status apart from its state: a failed drive keeps state Conf with status
Failed, a replacement drive is UConf with status Good. Drive groups are
written as numbers, or "-" for an unconfigured drive. The capture is the
storcli2 test data of this repository, kept to 4 RAID0 drives, with fake
serials, WWNs, SAS addresses, volume and controller identifiers.

The storcli2-disk-failure feature plays the megaraid-disk-failure scenarios
with the same steps. The healthy drives keep their paths when 306:2 fails,
but the offline volume of the failed drive is still reported with a
permanent path built from its NAA Id, although it has no block device:
documented as a known bug of ARTESCA-17960.

Refs: ARTESCA-17960
…ntrollers

Add ssaclisim, the backend of ssacli controllers: it answers the commands of
the raidmgmt ssacli getters (controller listing, physical and logical drive
details, show config) from a capture with events applied. ssacli prints
text, so the backend edits the lines of the captured sections: drive and
logical drive status, serials, disk names, and moves a replacement drive to
the Unassigned section once its array is gone. The capture is the ssacli
test data of this repository, kept to 4 single-drive RAID0 arrays, with fake
serials, WWIDs and identifiers.

The ssacli-disk-failure feature plays the megaraid-disk-failure scenarios
with the same steps. The ssacli getters already report them as expected:
the healthy drives keep their paths, the failed one is Failed, and the
replacement drive is UnassignedGood.

Refs: ARTESCA-17960
@g-carre
g-carre force-pushed the test/ARTESCA-17960-disk-failure-scenario-gherkin branch from a7ab071 to 11b2a13 Compare October 16, 2026 12:50
@g-carre
g-carre marked this pull request as ready for review October 16, 2026 12:58
@g-carre
g-carre requested a review from a team as a code owner October 16, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant