fstree: support online depth reshaping by End-rey · Pull Request #4149 · nspcc-dev/neofs-node

End-rey · 2026-08-26T14:46:45Z

Closes #3774.

codecov · 2026-08-26T14:50:25Z

Codecov Report

❌ Patch coverage is 78.21782% with 88 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.76%. Comparing base (db2dd63) to head (ed2d035).

Files with missing lines	Patch %	Lines
...kg/local_object_storage/blobstor/fstree/reshape.go	79.87%	32 Missing ⚠️
pkg/local_object_storage/blobstor/fstree/fstree.go	80.30%	26 Missing ⚠️
cmd/neofs-node/config.go	0.00%	12 Missing ⚠️
...kg/local_object_storage/blobstor/fstree/control.go	81.03%	11 Missing ⚠️
pkg/local_object_storage/blobstor/fstree/head.go	87.87%	4 Missing ⚠️
...ject_storage/blobstor/fstree/fstree_write_linux.go	71.42%	2 Missing ⚠️
cmd/neofs-node/storage.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4149      +/-   ##
==========================================
+ Coverage   30.32%   30.76%   +0.44%     
==========================================
  Files         673      674       +1     
  Lines       40673    40986     +313     
==========================================
+ Hits        12333    12611     +278     
- Misses      28340    28375      +35

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

roman-khimov · 2026-08-27T08:58:57Z

 		return fmt.Errorf("validate configuration: %w", err)
 	}
+	if blobstorLayoutChanged(oldCfg, newCfg) {
+		return errors.New("changing blobstor depth requires node restart")


Why can't it be done without it? Triggered via control service?

Runtime layout changes require more than updating configuration: the node must locate the active FSTree instance, persist the new reshape state, atomically switch primary/fallback layouts, and synchronize this transition with concurrent PUT, DELETE, GET and iteration operations. It must also define what happens if configuration reload succeeds only for some shards or the reshape worker cannot start.

A restart performs this transition during storage initialization, before the node starts serving requests. Reshaping itself is still online, and the persistent descriptor state makes it safe to resume after later restarts. Given that, adding a separate runtime control path does not seem worth the additional complexity and risk.

roman-khimov · 2026-08-27T09:33:45Z

-		return err
-	}
+	t.reshapeMtx.Lock()
+	defer t.reshapeMtx.Unlock()


Not sure what you're trying to protect from.

It prevents a delete/rename race that could resurrect a deleted object.

rename is atomic. Proper sequencing can prevent this. Delete old path first, then delete new path. If old path is deleted before reshaper rename --- reshaper gets a rename error. If reshaper is first to rename then deleting old path fails, but deleting new path does.

roman-khimov · 2026-08-27T09:45:30Z

+	if err := util.MkdirAllX(filepath.Dir(newPath), t.Permissions); err != nil {
+		return false, fmt.Errorf("create destination directory for %q: %w", newPath, err)
+	}
+	if _, err := os.Lstat(newPath); err == nil {


Why? Things can change between lstat and rename. You're trying to avoid this with reshapeMtx, but this doesn't matter much anyway, objects with the same address are the same objects and rename can be done safely. Then reshapeIterMtx will drastically affect performance, this background process will almost always hold the lock and regular operations will suffer badly.

I still think this lstat can be avoided.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

End-rey · 2026-09-01T15:57:24Z

+			info, statErr := os.Stat(entry.path)
+			if statErr != nil {
+				return fmt.Errorf("stat file %q: %w", entry.path, statErr)
+			}
+			if err = sizeHandler(entry.addr, uint64(info.Size())); err != nil {
+				return err
+			}


End-rey · 2026-09-01T15:57:38Z

+	d.Depth = d.Reshape.ToDepth
+	d.Reshape = nil
+	if err = writeDescriptor(descPath, d); err != nil {
+		return fmt.Errorf("write completed reshape descriptor: %w", err)


Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

roman-khimov · 2026-09-03T15:48:41Z

+	return nil
+}
+
+func (t *FSTree) reshapeLastProcessedPath() (string, error) {


I'd expect it to be read once on FSTree init, parsing it again just adds some potential for errors.

roman-khimov · 2026-09-03T15:49:33Z

+
+func (t *FSTree) updateReshapeProgress(lastProcessedPath string) error {
+	descPath := t.descriptorPath()
+	f, err := os.Open(descPath)


You can also keep the current version in mem, update last path there and flush it to disk, without reading the file.

roman-khimov · 2026-09-03T19:04:34Z

-		return err
-	}
+	t.reshapeMtx.Lock()
+	defer t.reshapeMtx.Unlock()


rename is atomic. Proper sequencing can prevent this. Delete old path first, then delete new path. If old path is deleted before reshaper rename --- reshaper gets a rename error. If reshaper is first to rename then deleting old path fails, but deleting new path does.

roman-khimov · 2026-09-03T19:04:59Z

 func (t *FSTree) getPath(addr oid.Address) (string, error) {
-	p := t.treePath(addr)
+	t.reshapeMtx.RLock()
+	defer t.reshapeMtx.RUnlock()


Same with all other operations, old first, new afterwards --- no locking needed.

roman-khimov · 2026-09-03T19:05:30Z

 	}

+	t.reshapeMtx.Lock()
+	defer t.reshapeMtx.Unlock()


Puts don't care at all, they always use new path.

roman-khimov · 2026-09-03T19:10:48Z

+					return err
+				}
+				*dirty = false
+				if err := t.updateReshapeProgress(relativePath); err != nil {


You can write this file and have a single syncfs call for it and other meta (avoid syncReshape).

roman-khimov · 2026-09-03T19:12:58Z

+	if err := util.MkdirAllX(filepath.Dir(newPath), t.Permissions); err != nil {
+		return false, fmt.Errorf("create destination directory for %q: %w", newPath, err)
+	}
+	if _, err := os.Lstat(newPath); err == nil {


I still think this lstat can be avoided.

roman-khimov · 2026-09-03T19:43:26Z

+			}
 		}
 	}
-


Add secondary-depth fallback and background file relocation. Require restart for FSTree layout changes. Closes #3774. Signed-off-by: Andrey Butusov <andrey@nspcc.io>

End-rey self-assigned this Aug 26, 2026

End-rey requested review from carpawell, cthulhu-rider and roman-khimov as code owners August 26, 2026 14:46

roman-khimov requested a balanced review from Copilot August 26, 2026 16:07

Copilot AI reviewed Aug 26, 2026

roman-khimov reviewed Aug 27, 2026

View reviewed changes

End-rey force-pushed the fstree-online-reshape branch 3 times, most recently from b11b791 to f7a4027 Compare August 31, 2026 12:55

End-rey requested a review from roman-khimov August 31, 2026 13:17

End-rey force-pushed the fstree-online-reshape branch from f7a4027 to 9ab6efa Compare August 31, 2026 13:48

roman-khimov requested a balanced review from Copilot August 31, 2026 16:35

Copilot AI reviewed Aug 31, 2026

End-rey requested a balanced review from Copilot August 31, 2026 17:41

Copilot AI reviewed Aug 31, 2026

roman-khimov requested a balanced review from Copilot September 1, 2026 13:01

Copilot started reviewing on behalf of roman-khimov September 1, 2026 13:02 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

End-rey force-pushed the fstree-online-reshape branch 2 times, most recently from f7e1ac1 to 314eee8 Compare September 1, 2026 15:57

End-rey requested a balanced review from Copilot September 1, 2026 16:01

Copilot AI reviewed Sep 1, 2026

roman-khimov reviewed Sep 3, 2026

View reviewed changes

End-rey force-pushed the fstree-online-reshape branch 3 times, most recently from c153a12 to ed2d035 Compare September 4, 2026 15:31

fstree: support online depth reshaping

ed2d035

Add secondary-depth fallback and background file relocation. Require restart for FSTree layout changes. Closes #3774. Signed-off-by: Andrey Butusov <andrey@nspcc.io>

Uh oh!

Conversation

End-rey commented Aug 26, 2026

Uh oh!

codecov Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Aug 26, 2026 •

edited

Loading