diff --git a/AGENTS.md b/AGENTS.md index e0adefb..19654a2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ released together and consumed separately. |---|---|---| | `git-core` | Hardened repositories, remote policy, commit and reference identity, results, transport, reading, cache repository, in-core writing, compare-and-swap, in-core merge, notes; linked working trees, commit from a working tree, working-tree status (package `worktrees`) | - | | `git-testing` | Generators and fixtures: local and in-memory repositories, a local remote | - | -| `git-store` | Repository registry, typed configuration, codec interface, outcome list and result, service interfaces with provider selection | `git-core` | +| `git-store` | Repository registry, typed configuration, codec interface, outcome list and result, service interfaces with provider selection; write path: reference layout, drafts, explicit save with lease, conflicts and change detection, draft expiry | `git-core` | Planned but not yet created: `git-forms`, `git-timestamp`, `git-directory` (see `doc/concepts/module-structure.adoc` for the order and the dependency graph). @@ -51,10 +51,15 @@ fixtures. `git-store` exports `de.cuioss.git.document.store` with the foundation closed outcome list `StoreOutcome` and the result `StoreResult` (whose outcome is a `StoreOutcome` or a non-success Git outcome passed on as `StoreResult.PassedOn`), the codec interface `Codec`, the repository registry `RepositoryRegistry` with `StoreRepository`, the typed configuration `StoreConfiguration` (built by -`StoreConfiguration.from` from an already-parsed nested map, over the package-private reader `ConfigurationTree`), the +`StoreConfiguration.from` from an already-parsed nested map, over the package-private reader `ConfigurationTree`; its +sections include `commit`, the record `Commit` with `commit.email`), the service interfaces `SigningKeySource`, `AnchorProvider` and `PersonDirectory` with `PersonEntry` and `Membership`, the -provider selection `StoreProviders` through `ServiceLoader`, and the `LogRecord` holder `GitStoreLogMessages`; its -tests run on the class path against the `git-testing` fixtures. Add an `exports` directive together with the first +provider selection `StoreProviders` through `ServiceLoader`, and the `LogRecord` holder `GitStoreLogMessages`; and the +write path: `StoreWriter` (drafts, listing, explicit save with lease, draft expiry with garbage collection) with +`StoreReferences`, `DocumentCollection`, `DocumentKey`, `Ulid`, `Draft`, `SaveResult`, `TechnicalUser` and `Author`, +over the package-private `DraftReferences`, `ExplicitSave`, `DraftExpiry`, `CommitIdentities` and `RepositoryGuard`; +the commits it writes are unsigned and carry no trailers until GDR-STORE-12/13. Its tests run on the class path against +the `git-testing` fixtures. Add an `exports` directive together with the first public class of a package: javac rejects exporting a package that contains only `package-info.java`. The build guards GDR-GEN-1 (`doc/specification/general.adoc`) with the `maven-enforcer-plugin` that `cui-parent-pom` diff --git a/doc/README.adoc b/doc/README.adoc index 6d5547a..7062f44 100644 --- a/doc/README.adoc +++ b/doc/README.adoc @@ -121,9 +121,14 @@ xref:specification/git-core.adoc#GDR-CORE-10[GDR-CORE-10], xref:specification/gi xref:specification/git-core.adoc#GDR-CORE-12[GDR-CORE-12], xref:specification/git-core.adoc#GDR-CORE-13[GDR-CORE-13], xref:specification/git-core.adoc#GDR-CORE-14[GDR-CORE-14] and xref:specification/git-core.adoc#GDR-CORE-15[GDR-CORE-15] of `git-core`, and -xref:specification/git-store.adoc#GDR-STORE-1[GDR-STORE-1] and -xref:specification/git-store.adoc#GDR-STORE-11[GDR-STORE-11] of `git-store`, together with its configuration, its -outcome list and its service interfaces. +xref:specification/git-store.adoc#GDR-STORE-1[GDR-STORE-1], +xref:specification/git-store.adoc#GDR-STORE-11[GDR-STORE-11] and the write path +xref:specification/git-store.adoc#GDR-STORE-2[GDR-STORE-2], xref:specification/git-store.adoc#GDR-STORE-3[GDR-STORE-3], +xref:specification/git-store.adoc#GDR-STORE-4[GDR-STORE-4], xref:specification/git-store.adoc#GDR-STORE-5[GDR-STORE-5], +xref:specification/git-store.adoc#GDR-STORE-7[GDR-STORE-7] and +xref:specification/git-store.adoc#GDR-STORE-8[GDR-STORE-8] of `git-store`, together with its configuration, its +outcome list and its service interfaces; xref:specification/general.adoc#GDR-NFR-2[GDR-NFR-2] and the write-path part +of xref:specification/general.adoc#GDR-NFR-3[GDR-NFR-3] are covered by tests of `git-store`. xref:specification/git-core.adoc#GDR-CORE-1[GDR-CORE-1] is met with one limit: the JVM-wide process start of JGit's `SystemReader` is closed only when the embedding application installs `RefusingFS.systemReader`, the recommended setup. For xref:specification/general.adoc#GDR-NFR-4[GDR-NFR-4], `git-core` ships its native-image metadata, covering its @@ -145,8 +150,9 @@ of a branch reservation |Repository fixtures and the credential generator |xref:specification/git-store.adoc[git-store] -|Registry, outcome list, drafts, saving, release, identity, index, listeners, codecs, signature, verification, -anchoring, own timestamp, person directory interface, configuration +|Registry, outcome list, the write path (drafts, explicit save with lease, conflicts and change detection, draft +expiry), release, identity, index, listeners, codecs, signature, verification, anchoring, own timestamp, person +directory interface, configuration |xref:specification/git-forms.adoc[git-forms] |Kinds of documents, schema validation, data layout, generated output, workflow, deadlines, step-up, links, running diff --git a/doc/concepts/module-structure.adoc b/doc/concepts/module-structure.adoc index 0cf9226..2ed1af0 100644 --- a/doc/concepts/module-structure.adoc +++ b/doc/concepts/module-structure.adoc @@ -53,7 +53,8 @@ reading, notes, working trees |Drafts, saving, release, commit signature and identity trailers, index, post-commit listeners, anchoring with the own timestamp (xref:storage-engine.adoc[Storage Engine], xref:identity-and-signing.adoc[Identity and Signing]) |implemented: repository registry, typed configuration, codec interface, outcome list and result, service interfaces -and provider selection; planned: drafts, saving, release, signature, trailers, index, listeners, anchoring +and provider selection, drafts, explicit save with lease, conflicts and change detection, draft expiry; planned: +release, signature, trailers, index, listeners, anchoring |`git-forms` |Schema-driven documents: schema, template, workflow, deadlines, step-up for releases diff --git a/doc/specification/general.adoc b/doc/specification/general.adoc index 95cd69d..0f96440 100644 --- a/doc/specification/general.adoc +++ b/doc/specification/general.adoc @@ -11,12 +11,15 @@ its requirements (xref:git-testing.adoc#GDR-TEST-1[GDR-TEST-1], xref:git-testing operations, cache repository, in-core writing, compare-and-swap, in-core merge, reading and notes; the hardening with the one limit xref:git-core.adoc#GDR-CORE-1[GDR-CORE-1] states; and the working-tree operations of its package `de.cuioss.git.document.core.worktrees`, xref:git-core.adoc#working-trees[`git-core`, Working Trees]), and -`git-store` is created: it implements -xref:git-store.adoc#GDR-STORE-1[GDR-STORE-1] and xref:git-store.adoc#GDR-STORE-11[GDR-STORE-11] and its configuration, -and defines its three service interfaces, whose providers are planned (xref:git-store.adoc[`git-store`]). Of the -requirements below, <> is implemented, and for <> `git-core` ships its native-image metadata, -which this repository does not verify yet; every other requirement of the libraries is planned, and the other modules -are not yet created. +`git-store` implements xref:git-store.adoc#GDR-STORE-1[GDR-STORE-1], xref:git-store.adoc#GDR-STORE-11[GDR-STORE-11] +and its configuration, and its write path: xref:git-store.adoc#GDR-STORE-2[GDR-STORE-2], +xref:git-store.adoc#GDR-STORE-3[GDR-STORE-3], xref:git-store.adoc#GDR-STORE-4[GDR-STORE-4], +xref:git-store.adoc#GDR-STORE-5[GDR-STORE-5], xref:git-store.adoc#GDR-STORE-7[GDR-STORE-7] and +xref:git-store.adoc#GDR-STORE-8[GDR-STORE-8]; it defines its three service interfaces, whose providers are planned +(xref:git-store.adoc[`git-store`]). Of the requirements below, <> is implemented; <> and the +write-path part of <> are covered by tests of `git-store` (`ConcurrentWritersTest`, +`WritePathSecretsTest`); and for <> `git-core` ships its native-image metadata, which this repository does +not verify yet. Every other requirement of the libraries is planned, and the other modules are not yet created. [#GDR-GEN-1] == GDR-GEN-1: Module Layering and Dependency Isolation @@ -261,7 +264,10 @@ The test is to run the writers as threads against a local remote of `git-testing * No token, password, private or secret key material, session id, `sub`, e-mail address or person name may appear in a result detail, an exception message, a log message or the metadata of a commit (author, committer, message, - trailers); the person entry of the `git` directory is the one file that holds a name. + trailers); the person entry of the `git` directory is the one file that holds a name. The one exception: the e-mail + fields of author and committer may carry an address only when the operator sets `commit.email` to `true` + (xref:git-store.adoc#store-configuration[`git-store`, Configuration]); by default they stay empty, and an address + never appears in a result detail, an exception message, a log message, a commit message or a trailer. * Public keys (commit-signing and anchor keys) are no secrets and may be stored. [cols="1,2"] @@ -272,7 +278,9 @@ The test is to run the writers as threads against a local remote of `git-testing read from the application's configuration for the call; never stored, logged, committed or put into a detail |signing and anchor keys |held by the key source or in memory; never written in clear text |`sub`, e-mail address, person name |`sub` only as the subject hash of the person entry; the name only in the person -entry; never in the metadata of a commit (author, committer, message, trailers), a document or a log +entry; never in the metadata of a commit (author, committer, message, trailers), a document or a log, except an +address in the e-mail fields of author and committer when the operator sets `commit.email` to `true` +(xref:git-store.adoc#store-configuration[`git-store`, Configuration]); by default those fields stay empty |=== Tests are to assert the rules with a JUL log capture (`cui-test-juli-logger`, declared in the test scope of `git-core` diff --git a/doc/specification/git-core.adoc b/doc/specification/git-core.adoc index 78ac20b..43f2d5a 100644 --- a/doc/specification/git-core.adoc +++ b/doc/specification/git-core.adoc @@ -172,7 +172,12 @@ The operation is `cloneIfMissing`. An existing bare repository at the git direct remote; a directory that holds anything else is `NOT_A_REPOSITORY`. Otherwise the URL is checked against the remote policy (`REFUSED_BY_POLICY` without contact), a bare repository is initialised, and its remote `origin` is configured and fetched. The fetch refspecs mirror the remote's branches, tags and notes into the clone: -`+refs/heads/*:refs/heads/*`, `+refs/tags/*:refs/tags/*` and `+refs/notes/*:refs/notes/*`. +`+refs/heads/*:refs/heads/*`, `+refs/tags/*:refs/tags/*` and `+refs/notes/*:refs/notes/*`. They are public as the +unmodifiable list `JGitRepositoryOperations.MIRROR_FETCH_REF_SPECS`, the one source of that set: a caller that prunes a +clone passes it to `fetch` explicitly, so the prune of <> deletes only references below the mirrored +namespaces, whatever fetch refspecs the clone's configuration holds (JGit 7.8.0 `Transport.fetch` falls back to the +configured fetch refspecs only for an empty list, and `FetchProcess.deleteStaleTrackingRefs` deletes only references +matching the destination of a fetched refspec). The stored origin carries no user information: the URL is configured without user name and password (`URIish.setUser(null).setPass(null)`), so no credential reaches the persistent configuration of the clone; the @@ -419,6 +424,14 @@ an update was not written `LOCK_FAILURE`, `IO_FAILURE` and every other result end it `FAILED`. The detail names the local reference and the result. +A fetch with `FetchInput.prune` set (`false` by default) also deletes every local reference that lies in the +destination of one of its reference specifications and whose source the remote no longer advertises, through +`FetchCommand.setRemoveDeletedRefs`; JGit 7.8.0 `FetchProcess.deleteStaleTrackingRefs` considers only the local +references that match a destination of the fetch's reference specifications, so a reference outside every +destination, such as a draft below `refs/drafts/` of `git-store`, is never deleted +(link:https://github.com/eclipse-jgit/jgit/blob/v7.8.0.202609011348-r/org.eclipse.jgit/src/org/eclipse/jgit/transport/FetchProcess.java[JGit +7.8.0 `FetchProcess`]). + Every transport of a clone, a fetch and a push runs with a compiled timeout of 60 seconds (`JGitRepositoryOperations.TRANSPORT_TIMEOUT_SECONDS`, set through `TransportCommand.setTimeout`; link:https://github.com/eclipse-jgit/jgit/blob/v7.8.0.202609011348-r/org.eclipse.jgit/src/org/eclipse/jgit/api/TransportCommand.java[JGit diff --git a/doc/specification/git-store.adoc b/doc/specification/git-store.adoc index 3991152..4a6f01c 100644 --- a/doc/specification/git-store.adoc +++ b/doc/specification/git-store.adoc @@ -11,8 +11,12 @@ Rationale: xref:../concepts/storage-engine.adoc[Storage Engine], xref:../concept and Signing] and xref:../concepts/evidence-and-companion-stores.adoc[Evidence and Companion Stores]. Implemented are the repository registry (<>), the codec interface (<>), the typed -configuration (<>), the outcome list with its result, and the three service interfaces below with -the selection of their providers; the providers themselves (`file`, `signed`) and every other requirement are planned. +configuration (<>), the outcome list with its result, the three service interfaces below with +the selection of their providers, and the write path of `StoreWriter`: drafts (<>), the explicit save +(<>) with its conflicts (<>) and change detection (<>), the draft expiry +(<>), and author, committer and credentials (<>). Until <> and +<> are implemented, the commits the write path writes are unsigned and carry no trailers. The providers +themselves (`file`, `signed`) and every other requirement are planned. Its third-party dependencies are `cui-java-tools` for `CuiLogger`, required transitively because the public holder `GitStoreLogMessages` exposes its `LogRecord`, Lombok (scope `provided`) and `org.jspecify:jspecify`, see xref:general.adoc#GDR-GEN-1[GDR-GEN-1]; JGit comes through `git-core`, which the module @@ -157,6 +161,8 @@ No detail carries the remote. `repositories()` returns the repositories in regis |`refs/tags/checkpoints/` |checkpoint `n` (<>), never moved |yes |`refs/notes/anchors/` |anchor proofs (<>) |yes |`refs/drafts///` |intermediate saves of one user for one document or entry |*never* +|`refs/applied///` |the draft commit a save last applied while a newer draft was kept +(<>) |*never* |=== * A collection is either per-document or numbered, never both: JGit refuses a reference nested below an existing one @@ -172,6 +178,40 @@ No detail carries the remote. `repositories()` returns the repositories in regis * The drafts live only in the store's bare clone; the remote never secures them, and they are lost with the clone unless the operator backs up its storage. +`StoreReferences` builds the layout: `MAIN`, the prefix `DRAFTS`, `target(key)` and `draft(pseudonym, key)`, and the +push reference specifications `PUSH_REF_SPECS` (`refs/heads/*`, `refs/tags/*` and `refs/notes/*`, each to the same +name on the remote), which never carry `refs/drafts/*` +(xref:../adr/0003-Drafts_are_per-user_references_that_are_never_pushed.adoc[ADR-0003]). A +`DocumentCollection` has a name of the form `[a-z0-9-]+`, neither `main` nor `checkpoints`, and a kind: +`PER_DOCUMENT` (one working branch per document), `NUMBERED` (one collection branch) or `MAIN` (content without +release, saved onto `main` itself). A `DocumentKey` is the collection with an id: the document's ULID for a +per-document collection, otherwise an id of the form `[A-Za-z0-9-]+`; `Ulid.next()` creates a ULID from a +`SecureRandom`, without the optional monotonic mode of the specification. A pseudonym is a reference segment of the +form `[A-Za-z0-9-]+`; any other is refused as `MALFORMED` with the detail `pseudonym: invalid`, which never names it. +Every refusal of a name or id is `MALFORMED`, and its detail names the field, never the value. + +`StoreWriter.open(repository, gitDir, user, configuration, operations, clock)` clones the remote into a bare clone if +none exists yet, with `PUSH_REF_SPECS` and the credentials of the technical user. An existing clone is reused only +when its `origin` has exactly one fetch URL and every push URL (the fetch URL when none is configured) equal to the +registered remote, read through JGit 7.8.0 `RemoteConfig`, which applies `insteadOf` and `pushInsteadOf`; a clone of +another repository at the path is the passed-on `NOT_A_REPOSITORY` with the detail +`: clone: remote does not match the registry`, which names no URL, credential or path. +`draft(author, key, edits, message)` +writes the next commit of the draft reference by compare-and-swap (xref:git-core.adoc#GDR-CORE-5[GDR-CORE-5]), its +edits applied onto the draft's current tree. The first commit of a draft that does not exist yet has the tip of the +local target branch as its parent, else the tip of `main`, else none, so it holds the state the user started from; +it is written in core first, and only then is the draft reference published by an update that expects it absent and +writes no reflog, so a refused first write, for example two edits of one path, leaves no draft to be listed. That +update's `LOCK_FAILURE` means that another writer published the draft first or that its lock could not be acquired, +which JGit 7.8.0 reports alike; the next attempt reads the draft again and writes on top of the published one, or +publishes again. A draft that existed when it was read but is absent when the compare-and-swap reads it, deleted +meanwhile by a save or by another process, is never replaced by a root commit holding only the new edits: no commit +is written and the next attempt writes the first commit of the draft anew. A root commit is written only when neither +branch exists. When every one of `save.max-attempts` attempts lost its draft or its publication, the write ends in the +passed-on `CONFLICT`, which may be retried. `drafts(pseudonym)` lists the drafts +of one user, sorted by reference name, each as a `Draft` (collection, id, tip commit id, and the committer time of the +tip as the time of the last intermediate save). + [#GDR-STORE-3] == GDR-STORE-3: Explicit Save @@ -204,6 +244,57 @@ then expects the branch to be absent on the remote. Content that has no release (xref:git-directory.adoc#GDR-DIR-3[GDR-DIR-3], xref:git-directory.adoc#GDR-DIR-6[GDR-DIR-6], xref:git-directory.adoc#GDR-DIR-7[GDR-DIR-7]). +`StoreWriter.save(author, key, expectedBlobIds, message)` returns a `SaveResult`, a sealed type of three variants a +caller branches on with a pattern-matching `switch`: `Saved` with the commit the target branch points at, +`Conflicting` with the conflicting paths (<>), and `NotSaved` with any other outcome and its detail; +`toResult()` gives the plain `StoreResult` shape. The save reads the draft tip once, and every attempt re-applies that +draft commit, so a draft written during the save is kept for the next save and not published with the expected blob +ids and the message of this one. An attempt fetches with the mirror reference specifications of `git-core` +(`JGitRepositoryOperations.MIRROR_FETCH_REF_SPECS`, xref:git-core.adoc#GDR-CORE-3[GDR-CORE-3]), passed explicitly, +and the credentials of the technical user, so it fetches and prunes `refs/heads/`, `refs/tags/` and `refs/notes/` +only: it prunes every reference there the remote no longer has (xref:git-core.adoc#GDR-CORE-7[GDR-CORE-7]). JGit +7.8.0 `Transport.fetch` uses the fetch reference specifications of the clone's configuration only for an empty list, +and `FetchProcess.deleteStaleTrackingRefs` deletes only the references matching the destination of a fetched +reference specification +(link:https://github.com/eclipse-jgit/jgit/blob/v7.8.0.202609011348-r/org.eclipse.jgit/src/org/eclipse/jgit/transport/FetchProcess.java[JGit +7.8.0 `FetchProcess`], checked in the sources jar of `org.eclipse.jgit` 7.8.0.202609011348-r), so a draft and a +`refs/applied/` reference are never pruned, also in a reused clone whose stored fetch reference specifications reach +further. The attempt takes the local tip of the target branch as the base, so a target branch the remote deleted starts at `main` again. +The changes of the draft are the paths changed between the merge base of draft tip and base (JGit's +`RevFilter.MERGE_BASE`) and the draft tip, read with `RepositoryReader.changes` of `git-core`; without a merge base +every file of the draft is a change. A draft kept by a save because it moved during that save is replayed only with +what it changed since: the save records the draft commit it applied in `refs/applied///`, +and the next save of a draft that descends from that commit takes the paths changed since it, so it does not +overwrite a later change of another writer to a path it saved already. The reference is deleted together with the +draft, and lives in the clone only. One commit with the base as its only parent and the changes as its edits is +written in core by `InCoreWriter` of `git-core` and pushed to the target with the lease of the base; a +target branch that does not exist yet is pushed against the zero-id lease, which expects the remote branch to be +absent. A lease refusal or a rejection starts the next attempt from the fetch; after the last of `save.max-attempts` +the save is `NotSaved` with the passed-on `REJECTED`. An unreachable remote and invalid credentials end the save at once +as the passed-on `UNREACHABLE` or `CREDENTIALS_INVALID`, with nothing changed. A save without a draft is the passed-on +`NOT_FOUND`. A draft without changes is `Saved` with the tip of its existing target branch, with no commit and no push; +when the target branch does not exist, the save is `NotSaved` with the passed-on `NOT_FOUND` and the detail +`draft: no changes and no target branch`, and the draft is kept. After the accepted +push the local target branch moves by a reference update that expects the base and writes no reflog; an I/O failure +of that local move does not undo the save, which stays `Saved` and is logged as `GIT_STORE-103` with the class of the +error only, and the next fetch mirrors the remote branch. The draft is then deleted by compare-and-delete expecting +the applied draft commit. Every Git outcome is passed on as +`StoreResult.PassedOn`. A save is logged as `GIT_STORE-2`, a save that ended without a commit for another reason than +a conflict as `GIT_STORE-101`, naming the outcome, never a credential (`git-store/doc/LogMessages.adoc`). + +Within one process, `StoreWriter` coordinates the users of one bare clone through one in-process guard per clone, +shared by every writer that opens the same git directory (keyed by its real path): draft writes +and saves take the shared side of a maintenance lock, the deletes and the garbage collection of <> its +exclusive side, one lock per draft reference serialises the saves of that draft, and one transport lock serialises the +fetch, apply and push of a save attempt. A save takes the shared side of the maintenance lock, then the lock of its +draft, then the transport lock per attempt; the expiry takes only the exclusive side, so no cycle arises. Under the +draft lock a save reads the draft and its `refs/applied/` reference only after every earlier save of that draft in the +process has settled, so it does not apply again the changes an earlier save applied. The draft lock is in-process only, +and a failed write of the `refs/applied/` reference for a kept draft is outside it: the next save then applies the +draft's changes since its merge base. The guard does not close the window +between reading a reference and moving it; the expected-id comparison under the reference's lock +(xref:git-core.adoc#GDR-CORE-5[GDR-CORE-5]) closes it, also against a writer outside the process. + [#GDR-STORE-4] == GDR-STORE-4: Conflict Handling @@ -212,6 +303,13 @@ xref:git-directory.adoc#GDR-DIR-7[GDR-DIR-7]). The user resolves a conflict in the draft and saves again. +A conflict is `SaveResult.Conflicting`, whose outcome is the passed-on `CONFLICT`: its paths are repository-relative, +sorted and distinct, and travel as data only; its detail names their number (` conflicting paths`), never a path, +and the conflict is logged as `GIT_STORE-3` with that number. Nothing is pushed, the remote branch is unchanged, and +the draft reference is not modified. The fetch that preceded the conflict check may have updated the local branches of +the clone to the remote's tips, and pruned the ones the remote deleted (<>); the conflict itself writes +no reference. + [#GDR-STORE-5] == GDR-STORE-5: Concurrent Change Detection @@ -220,6 +318,17 @@ The user resolves a conflict in the draft and saves again. overwriting it. * A save of a file without an expected blob id must overwrite a concurrent change to that file. +`expectedBlobIds` maps a repository-relative path to the full blob id the caller loaded; the zero id +(`0000000000000000000000000000000000000000`) expects the path to be absent, and an empty map checks nothing. Every +changed path of the draft that has an expected blob id is compared with its blob on the fetched base, and every path +that differs is a conflict (<>). A written path also conflicts when it overlaps the fetched base as a +file and a directory in a way the draft did not resolve: when a parent of the path is a file on the base that the +draft does not delete (the edit would turn it into a directory), or when the path is a directory on the base holding +a file the draft does not change (the edit would replace it); this holds with and without an expected blob id, and an +addition below an existing directory is no overlap. An expected blob id that is no full object id is `NotSaved` with +the passed-on `NOT_FOUND`, as for `git-core`'s own object-id inputs, and the detail +`expectedBlobIds: not a full object id`, which names no path. + [#GDR-STORE-6] == GDR-STORE-6: Release @@ -243,15 +352,56 @@ identity trailers of the user who releases (<>), so it names the a (xref:git-core.adoc#GDR-CORE-5[GDR-CORE-5]), so that a draft changed meanwhile is kept. * A garbage collection must remove objects reachable only from deleted drafts. +`StoreWriter.expireDrafts(now)` deletes every draft below `refs/drafts/` whose tip has a committer time at or before +`now` minus `drafts.expiry`, computed in UTC, by compare-and-delete expecting the tip that was read as stale; a draft +changed meanwhile ends that delete in `CONFLICT` and is kept. The result is `OK` with the full names of the deleted +draft references, sorted, possibly none, and the run is logged as `GIT_STORE-4` with their number. The deletes run +under the exclusive side of the clone's maintenance lock (<>), so no draft write or save of the process +runs meanwhile: a draft write that found its draft present cannot lose it before its compare-and-swap. Every +`refs/applied/` reference (<>) whose draft no longer exists is deleted the same way, so it keeps no +draft commit from the garbage collection. + +Then, still under the exclusive side of that lock, so that no in-process write holds objects not yet referenced, the +garbage collection runs: `Git.wrap(repository).gc()`, JGit's `GarbageCollectCommand`, with `gc.pruneExpire` and +`gc.prunePackExpire` set to `now` in the clone's configuration in memory. JGit 7.8.0 `GarbageCollectCommand.call` +creates a `GC` for a `FileRepository` and waits for `GC.gc()` to complete (`gc.gc().get()`), so the collection is +synchronous; it packs the references, repacks the reachable objects into a new pack, deletes the old packs older +than the pack expiry and prunes the unreferenced loose objects older than the expiry, both read from that configuration +(link:https://github.com/eclipse-jgit/jgit/blob/v7.8.0.202609011348-r/org.eclipse.jgit/src/org/eclipse/jgit/api/GarbageCollectCommand.java[JGit +7.8.0 `GarbageCollectCommand`], +link:https://github.com/eclipse-jgit/jgit/blob/v7.8.0.202609011348-r/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/GC.java[JGit +7.8.0 `GC`], checked in the sources jar of `org.eclipse.jgit` 7.8.0.202609011348-r). It runs no hook and starts no +process. Every reference update of the store disables the reflog, so no reflog keeps an object of a deleted draft +alive. A failed collection is the passed-on `FAILED` with the detail `gc: `, logged as +`GIT_STORE-102` and at DEBUG with the exception class only; no log record attaches the exception, and neither names +its message or its causes, which may carry a path (xref:general.adoc#GDR-NFR-3[GDR-NFR-3]). JGit 7.8.0 `GC.doGc` +returns without collecting and without an error while another collection holds the clone's `gc.pid`, which +`GC.PidLock` takes for a running collection when it was modified less than 12 hours ago, and calls the command's +progress monitor (`GarbageCollectCommand.setProgressMonitor`, passed on to `GC.setProgressMonitor`) with `start` only +after it took `gc.pid`. The expiry gives the command a progress monitor; when that monitor was never started, the +collection was skipped, however late the other collection took `gc.pid`, and the expiry logs `GIT_STORE-104` and ends +in the passed-on `CONFLICT` with the detail `gc: skipped, another collection holds gc.pid`, which may be retried, so a +skipped collection is never reported as a successful expiry; the drafts it deleted stay deleted. + [#GDR-STORE-8] == GDR-STORE-8: Author, Committer and Credentials -* Every commit of a save must carry the caller's pseudonym as author (e-mail field empty) and the technical user as - committer. +* Every commit of a save must carry the caller's pseudonym as author and the technical user as committer, both with + an empty e-mail field unless `commit.email` is `true`; then they carry the address the application supplies for the + author and for the technical user, or an empty field when none is supplied. * Every fetch and push must use the credential of the technical user configured for the repository; the store must offer no other credentials. * Commits the store writes without a user's save (checkpoints, notes) must not be combined with the commits of a save. +The caller passes the author as `Author` (pseudonym and an optional address) and the technical user as +`TechnicalUser` (committer name, an optional address and the `CredentialsProvider` of every fetch and push of the +repository); no operation of `StoreWriter` takes another credential. The author and committer of every commit the +write path writes, draft commits included, follow `commit.email` (<>), and the commit time comes +from the writer's clock. An address that is blank or contains `<`, `>`, a carriage return, a line feed or a NUL +character is refused when the record is created, and the message names the component, never its value. The +`toString()` of `Author` renders the pseudonym only, that of `TechnicalUser` the name only, never an address or the +credentials (xref:general.adoc#GDR-NFR-3[GDR-NFR-3]). The save commit is the only commit a save pushes. + [#GDR-STORE-9] == GDR-STORE-9: In-Memory Index @@ -472,8 +622,10 @@ xref:../concepts/data-protection.adoc#_the_person_directory[Data Protection, The |Key |Semantics |`repositories` |the registry (<>) -|`save.max-attempts` |required; attempts of a save or of a push of notes +|`save.max-attempts` |required; 1 to 100; attempts of a save or of a push of notes |`drafts.expiry` |required; ISO 8601 period +|`commit.email` |boolean, default `false`: whether author and committer of every commit `git-store` writes carry an +e-mail address (xref:general.adoc#GDR-NFR-3[GDR-NFR-3], <>) |`fetch.interval` |required; ISO 8601 duration of the periodic fetch |`signing.key-source`, `signing.key` |the commit-signing key; source default `file` |`signing.allowed-signers` |required for verification; public commit-signing keys with validity periods @@ -489,8 +641,8 @@ nothing is anchored |=== `StoreConfiguration.from(tree, policy)` reads the configuration at start and returns a `StoreResult` of the typed -record `StoreConfiguration`, whose nested records mirror the sections: `Save`, `Drafts`, `Fetch`, `Signing`, `Anchor` -with `Anchor.Signed`, and `Directory`, plus the `RepositoryRegistry` (<>) checked against the one +record `StoreConfiguration`, whose nested records mirror the sections: `Save`, `Drafts`, `Fetch`, `Commit`, `Signing`, +`Anchor` with `Anchor.Signed`, and `Directory`, plus the `RepositoryRegistry` (<>) checked against the one `RemotePolicy` the caller passes. The input is an already-parsed tree of nested maps as a YAML or JSON parser delivers it, keys nested by dot segment (`anchor.signed.key` is the key `key` of the map `signed` of the map `anchor`), so `git-store` needs no parser library: @@ -501,7 +653,9 @@ it, keys nested by dot segment (`anchor.signed.key` is the key `key` of the map integer. Booleans are `Boolean`, and other values non-blank strings. Durations and periods are ISO 8601 strings parsed with `java.time.Duration` and `java.time.Period` (`drafts.expiry` and `retention` are periods); every duration and period must be positive, and a text whose value overflows on parsing, such as the period `P306783379W` - whose weeks exceed `int` as days, is malformed. `save.max-attempts` must be at least 1. + whose weeks exceed `int` as days, is malformed. `save.max-attempts` must lie between 1 and 100. `commit.email` must + be a boolean; any other value is refused as `MALFORMED` with the detail `commit.email is malformed`, which names the + key, never the value. * `anchor.provider`, when present, must match `[a-z0-9-]+`, because the id names the notes reference `refs/notes/anchors/` (<>); `signing.key-source` and `directory.persons.provider` only have to be non-blank. diff --git a/git-core/src/main/java/de/cuioss/git/document/core/JGitRepositoryOperations.java b/git-core/src/main/java/de/cuioss/git/document/core/JGitRepositoryOperations.java index 406bbe6..e3bc990 100644 --- a/git-core/src/main/java/de/cuioss/git/document/core/JGitRepositoryOperations.java +++ b/git-core/src/main/java/de/cuioss/git/document/core/JGitRepositoryOperations.java @@ -115,8 +115,17 @@ public final class JGitRepositoryOperations implements RepositoryOperations { /** The remote a clone is created with. */ static final String ORIGIN = Constants.DEFAULT_REMOTE_NAME; - /** The fetch reference specifications of a clone: branches, tags and notes are mirrored. */ - static final List MIRROR_FETCH_REF_SPECS = + /** + * The fetch reference specifications of a clone, unmodifiable: branches, tags and notes are mirrored. A caller that + * prunes a clone passes them to {@link #fetch(FetchInput)} explicitly, so that the prune deletes only references + * below the mirrored namespaces whatever fetch reference specifications the clone's configuration holds: JGit + * 7.8.0 {@code Transport.fetch} falls back to the configured ones only for an empty list, and + * {@code FetchProcess.deleteStaleTrackingRefs} deletes only references matching the destination of a fetched + * reference specification. + * + * @since 0.1 + */ + public static final List MIRROR_FETCH_REF_SPECS = List.of("+refs/heads/*:refs/heads/*", "+refs/tags/*:refs/tags/*", "+refs/notes/*:refs/notes/*"); /** @@ -497,6 +506,7 @@ public GitResult> fetch(FetchInput input) { FetchResult result = git.fetch() .setRemote(input.remote()) .setRefSpecs(input.refSpecs().stream().map(RefSpec::new).toList()) + .setRemoveDeletedRefs(input.prune()) // JGit 7.8.0 FetchCommand fetches submodules of a working tree on demand, each from its own // remote that the policy check above never saw .setRecurseSubmodules(FetchRecurseSubmodulesMode.NO) diff --git a/git-core/src/main/java/de/cuioss/git/document/core/RepositoryOperations.java b/git-core/src/main/java/de/cuioss/git/document/core/RepositoryOperations.java index e99668a..5199520 100644 --- a/git-core/src/main/java/de/cuioss/git/document/core/RepositoryOperations.java +++ b/git-core/src/main/java/de/cuioss/git/document/core/RepositoryOperations.java @@ -370,10 +370,16 @@ record RefUpdate(String ref, @Nullable String oldId, @Nullable String newId, Str * @param remote the configured remote name, e.g. {@code origin} * @param refSpecs the reference specifications; empty for the remote's configured ones * @param credentials the credentials of this call; {@code null} when none + * @param prune whether the fetch also deletes every local reference that lies in the destination of one of + * its reference specifications and whose source the remote no longer advertises (JGit 7.8.0 + * {@code FetchCommand.setRemoveDeletedRefs}, carried out by + * {@code FetchProcess.deleteStaleTrackingRefs}); a reference outside every destination is never + * deleted; {@code false} by default * @since 0.1 */ @Builder - record FetchInput(Path root, String remote, List refSpecs, @Nullable CredentialsProvider credentials) { + record FetchInput(Path root, String remote, List refSpecs, @Nullable CredentialsProvider credentials, + boolean prune) { /** * Requires the repository, the remote and the reference specifications, and copies the latter. @@ -382,6 +388,7 @@ record FetchInput(Path root, String remote, List refSpecs, @Nullable Cre * @param remote the remote name * @param refSpecs the reference specifications * @param credentials the credentials + * @param prune whether stale local references of the destinations are deleted * @throws NullPointerException if {@code root}, {@code remote} or {@code refSpecs} is {@code null} */ public FetchInput { diff --git a/git-core/src/test/java/de/cuioss/git/document/core/JGitRepositoryOperationsTest.java b/git-core/src/test/java/de/cuioss/git/document/core/JGitRepositoryOperationsTest.java index e28422c..17200eb 100644 --- a/git-core/src/test/java/de/cuioss/git/document/core/JGitRepositoryOperationsTest.java +++ b/git-core/src/test/java/de/cuioss/git/document/core/JGitRepositoryOperationsTest.java @@ -687,6 +687,35 @@ void pushAndFetch(@Remote LocalRemote remote, @LocalRepository Repository pusher () -> assertNull(updates.getFirst().oldId(), "the tracking reference is new")); } + @Test + @DisplayName("GDR-CORE-7: only a pruning fetch deletes the tracking reference of a deleted remote branch") + void pruningFetch(@Remote LocalRemote remote, @LocalRepository Repository client) throws Exception { + TestCommits.addRemote(client, ORIGIN, remote.uri().toString()); + var ref = branchRef(); + TestCommits.commitOn(remote.repository(), ref); + assertTrue(operations.fetch(fetch(client)).isOk(), "the client fetches the branch"); + var tracking = Constants.R_REMOTES + ORIGIN + "/" + Repository.shortenRefName(ref); + var outside = "refs/drafts/" + TestCommits.name(); + var outsideId = TestCommits.commit(client, null); + TestCommits.setRef(client, outside, outsideId); + org.eclipse.jgit.lib.RefUpdate delete = remote.repository().updateRef(ref); + delete.setForceUpdate(true); + assertEquals(org.eclipse.jgit.lib.RefUpdate.Result.FORCED, delete.delete(), "the remote branch is deleted"); + + var plain = operations.fetch(fetch(client)); + var keptByPlain = client.exactRef(tracking); + var pruned = operations.fetch( + FetchInput.builder().root(dir(client)).remote(ORIGIN).refSpecs(List.of()).prune(true).build()); + + assertAll( + () -> assertEquals(GitOutcome.OK, plain.outcome(), "the plain fetch succeeds"), + () -> assertNotNull(keptByPlain, "a plain fetch keeps the stale tracking reference"), + () -> assertEquals(GitOutcome.OK, pruned.outcome(), "the pruning fetch succeeds"), + () -> assertNull(client.exactRef(tracking), "the pruning fetch deletes it"), + () -> assertEquals(outsideId, refId(client, outside), + "a reference outside every destination is kept")); + } + @Test @DisplayName("GDR-CORE-7: a push that is no fast-forward is rejected") void rejected(@Remote LocalRemote remote, @LocalRepository Repository client) throws Exception { diff --git a/git-core/src/test/java/de/cuioss/git/document/core/RepositoryOperationsRecordsTest.java b/git-core/src/test/java/de/cuioss/git/document/core/RepositoryOperationsRecordsTest.java index 687fb55..4728b8e 100644 --- a/git-core/src/test/java/de/cuioss/git/document/core/RepositoryOperationsRecordsTest.java +++ b/git-core/src/test/java/de/cuioss/git/document/core/RepositoryOperationsRecordsTest.java @@ -208,7 +208,7 @@ void commitInfo() { } @Test - @DisplayName("the fetch input builder builds the constructor value") + @DisplayName("the fetch input builder builds the constructor value, without pruning unless asked") void fetchInput() { var root = path(); var remote = text(); @@ -216,8 +216,15 @@ void fetchInput() { var credentials = credentials(); var built = FetchInput.builder().root(root).remote(remote).refSpecs(refSpecs).credentials(credentials).build(); + var pruning = FetchInput.builder().root(root).remote(remote).refSpecs(refSpecs).credentials(credentials) + .prune(true).build(); - assertEquals(new FetchInput(root, remote, refSpecs, credentials), built, "the builder builds the same input"); + assertAll( + () -> assertEquals(new FetchInput(root, remote, refSpecs, credentials, false), built, + "the builder builds the same input"), + () -> assertFalse(built.prune(), "no pruning by default"), + () -> assertEquals(new FetchInput(root, remote, refSpecs, credentials, true), pruning, + "the builder sets the pruning")); } @Test diff --git a/git-store/doc/LogMessages.adoc b/git-store/doc/LogMessages.adoc index d38e3c6..89d71dc 100644 --- a/git-store/doc/LogMessages.adoc +++ b/git-store/doc/LogMessages.adoc @@ -12,8 +12,10 @@ WARN 100-199, ERROR 200-299). The messages are declared in `de.cuioss.git.document.store.GitStoreLogMessages`. No message carries anything xref:../../doc/specification/general.adoc#GDR-NFR-3[GDR-NFR-3] forbids: a message names a -provider id, a configuration key, a repository alias or a path, never a configured value, a key reference, key -material, a token, a password or the user information of a remote. +provider id, a configuration key, a repository alias, a path, a reference, a commit id, an outcome or a number, never +a configured value, a key reference, key material, a token, a password, the user information of a remote, a content, +an e-mail address or a person name. The messages of a save name no path of the saved document, only the number of +conflicting paths. Debug and trace messages carry no identifier and are not listed. == INFO Level (1-99) @@ -24,6 +26,19 @@ Debug and trace messages carry no identifier and are not listed. |GIT_STORE-1 |GIT_STORE |Selected the providers: signing key source %s, person directory %s, anchor provider %s |Logged when the providers of the store were selected at start: the ids of the selected signing key source, person directory and anchor provider, the provider ids only. +|GIT_STORE-2 |GIT_STORE |Saved repository %s, reference %s at commit %s +|Logged when the remote accepted an explicit save, or a draft without changes was saved as the tip of its existing +target branch +(xref:../../doc/specification/git-store.adoc#GDR-STORE-3[GDR-STORE-3]): the repository alias, the target reference and +the full id of the commit the target branch points at. +|GIT_STORE-3 |GIT_STORE |Did not save repository %s, reference %s: %s conflicting paths +|Logged when an explicit save conflicted and pushed nothing +(xref:../../doc/specification/git-store.adoc#GDR-STORE-4[GDR-STORE-4]): the repository alias, the target reference and +the number of conflicting paths, never a path. +|GIT_STORE-4 |GIT_STORE |Expired the drafts of repository %s: %s deleted +|Logged when the draft expiry deleted the drafts unchanged for `drafts.expiry` and collected the garbage +(xref:../../doc/specification/git-store.adoc#GDR-STORE-7[GDR-STORE-7]): the repository alias and the number of deleted +drafts, never a draft reference, which names a pseudonym. |=== == WARN Level (100-199) @@ -35,6 +50,23 @@ directory and anchor provider, the provider ids only. provider selection the store does not accept: the detail of the result, which names a key, an alias or a service interface, never a value. A refused provider selection names the service interface and the configuration key (`SigningKeySource: no provider with the id of signing.key-source`), never the configured id. +|GIT_STORE-101 |GIT_STORE |Did not complete the save of repository %s, reference %s: %s |Logged when an explicit save +ended without a commit on the remote for a reason other than a conflict +(xref:../../doc/specification/git-store.adoc#GDR-STORE-3[GDR-STORE-3]), for example a lease refused or a push rejected +on every attempt, an unreachable remote or invalid credentials: the repository alias, the target reference and the +name of the outcome, never a credential. +|GIT_STORE-102 |GIT_STORE |The garbage collection of repository %s failed: %s |Logged when the garbage collection +after the draft expiry failed (xref:../../doc/specification/git-store.adoc#GDR-STORE-7[GDR-STORE-7]): the repository +alias and the simple class name of the exception, never its message, which may carry a path. +|GIT_STORE-103 |GIT_STORE |Did not move the local branch of repository %s, reference %s after the accepted save: %s +|Logged when the remote accepted an explicit save but the local target branch could not be moved +(xref:../../doc/specification/git-store.adoc#GDR-STORE-3[GDR-STORE-3]): the repository alias, the target reference and +the simple class name of the exception, never its message, which may carry a path. The save is still `Saved`; the next +fetch mirrors the remote branch into the clone. +|GIT_STORE-104 |GIT_STORE |Skipped the garbage collection of repository %s: another collection holds gc.pid +|Logged when the draft expiry did not run its garbage collection because another collection holds the clone's +`gc.pid` (xref:../../doc/specification/git-store.adoc#GDR-STORE-7[GDR-STORE-7]): the repository alias. The expiry +ends in the passed-on `CONFLICT`, which may be retried. |=== == ERROR Level (200-299) diff --git a/git-store/src/main/java/de/cuioss/git/document/store/Author.java b/git-store/src/main/java/de/cuioss/git/document/store/Author.java new file mode 100644 index 0000000..c2d95fd --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/Author.java @@ -0,0 +1,70 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.util.Objects; + + +import org.jspecify.annotations.Nullable; + +/** + * The author of a draft or a save: the user, named in the history by the pseudonym only. + *

+ * {@link #toString()} renders the pseudonym only, never the address (GDR-NFR-3). + * + * @param pseudonym the pseudonym of the user, the author name of every commit; also a segment of the draft reference, + * so it must match {@code [A-Za-z0-9-]+} for a draft (see {@link StoreReferences#draft}) + * @param email the author address; {@code null} when none is known. It is written only when {@code commit.email} + * is {@code true} (see {@link StoreConfiguration.Commit}) + * @since 0.1 + */ +public record Author(String pseudonym, @Nullable String email) { + + /** + * Checks the components. + * + * @param pseudonym the pseudonym + * @param email the address, or {@code null} + * @throws NullPointerException if {@code pseudonym} is {@code null} + * @throws IllegalArgumentException if the address is blank or contains {@code <}, {@code >}, a carriage return, a + * line feed or a NUL character; the message names the component, never its value + */ + public Author { + Objects.requireNonNull(pseudonym, "pseudonym"); + CommitIdentities.checkAddress(email); + } + + /** + * Creates an author without address. + * + * @param pseudonym the pseudonym + * @return the author + * @throws NullPointerException if {@code pseudonym} is {@code null} + */ + public static Author of(String pseudonym) { + return new Author(pseudonym, null); + } + + /** + * Renders the pseudonym only, never the address (GDR-NFR-3). + * + * @return the rendering + */ + @Override + public String toString() { + return "Author[pseudonym=" + pseudonym + "]"; + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/CommitIdentities.java b/git-store/src/main/java/de/cuioss/git/document/store/CommitIdentities.java new file mode 100644 index 0000000..ada0df3 --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/CommitIdentities.java @@ -0,0 +1,100 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.Objects; + + +import org.eclipse.jgit.lib.PersonIdent; +import org.jspecify.annotations.Nullable; + +/** + * The single place every commit identity of {@code git-store} is built: the author from an {@link Author}, the + * committer from the {@link TechnicalUser}. + *

+ * With {@code commit.email} {@code false}, the default, the e-mail field of both is empty and a supplied address is + * ignored, so no address reaches the history (GDR-NFR-3). With {@code commit.email} {@code true} the supplied address + * is written, and an absent one stays empty. Every time is written in UTC. + */ +final class CommitIdentities { + + /** The characters an address must not contain: they would break the identity line of a commit. */ + private static final String FORBIDDEN = "<>\r\n\0"; + + private final boolean email; + private final TechnicalUser user; + + /** + * Creates the identities. + * + * @param commit the section {@code commit} of the configuration + * @param user the technical user, the committer + * @throws NullPointerException if an argument is {@code null} + */ + CommitIdentities(StoreConfiguration.Commit commit, TechnicalUser user) { + this.email = Objects.requireNonNull(commit, "commit").email(); + this.user = Objects.requireNonNull(user, "user"); + } + + /** + * Builds the author identity of a commit. + * + * @param author the author + * @param when the author time + * @return the pseudonym as name, and the address or an empty one + * @throws NullPointerException if an argument is {@code null} + */ + PersonIdent author(Author author, Instant when) { + Objects.requireNonNull(author, "author"); + return identity(author.pseudonym(), author.email(), when); + } + + /** + * Builds the committer identity of a commit. + * + * @param when the commit time + * @return the name of the technical user, and its address or an empty one + * @throws NullPointerException if {@code when} is {@code null} + */ + PersonIdent committer(Instant when) { + return identity(user.name(), user.email(), when); + } + + private PersonIdent identity(String name, @Nullable String address, Instant when) { + Objects.requireNonNull(when, "when"); + String written = email && address != null ? address : ""; + return new PersonIdent(name, written, when, ZoneOffset.UTC); + } + + /** + * Checks an address of an {@link Author} or a {@link TechnicalUser}. + * + * @param address the address, or {@code null} for none + * @throws IllegalArgumentException if the address is blank or contains {@code <}, {@code >}, a carriage return, a + * line feed or a NUL character; the message names the component {@code email}, + * never the value + */ + static void checkAddress(@Nullable String address) { + if (address == null) { + return; + } + if (address.isBlank() || address.chars().anyMatch(character -> FORBIDDEN.indexOf(character) >= 0)) { + throw new IllegalArgumentException("email: invalid"); + } + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/DocumentCollection.java b/git-store/src/main/java/de/cuioss/git/document/store/DocumentCollection.java new file mode 100644 index 0000000..344bc78 --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/DocumentCollection.java @@ -0,0 +1,108 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.util.Objects; +import java.util.Optional; +import java.util.Set; + +/** + * A collection of documents of a repository and the way its documents are kept on branches (GDR-STORE-2). + *

+ * The name is a segment of every reference of the collection, so it takes the alphabet of a repository alias, + * {@code [a-z0-9-]+}, which contains no character git-check-ref-format(1) forbids, no dot and no slash. The names + * {@value #MAIN_NAME} and {@value #CHECKPOINTS_NAME} are reserved: {@code refs/heads/main} is the released state, and + * {@code refs/tags/checkpoints/} are the checkpoints. + * + * @param name the name of the collection, of the form {@code [a-z0-9-]+} and neither {@value #MAIN_NAME} nor + * {@value #CHECKPOINTS_NAME} + * @param kind the way the documents of the collection are kept + * @since 0.1 + */ +public record DocumentCollection(String name, Kind kind) { + + /** The reserved name of the branch of the released state. */ + public static final String MAIN_NAME = "main"; + + /** The reserved name of the tag namespace of the checkpoints. */ + public static final String CHECKPOINTS_NAME = "checkpoints"; + + private static final Set RESERVED = Set.of(MAIN_NAME, CHECKPOINTS_NAME); + + /** + * The way the documents of a collection are kept on branches. + * + * @since 0.1 + */ + public enum Kind { + + /** + * One working branch per document, {@code refs/heads//}, the id being the document's ULID. + */ + PER_DOCUMENT, + + /** + * One collection branch, {@code refs/heads/}, holding every entry and the counter file; the id of + * an entry is its readable number, such as {@code R-042}. + */ + NUMBERED, + + /** Content without release, saved onto {@code refs/heads/main} itself. */ + MAIN + } + + /** + * Checks the name and the kind. + * + * @param name the name of the collection + * @param kind the kind of the collection + * @throws NullPointerException if {@code name} or {@code kind} is {@code null} + * @throws IllegalArgumentException if the name lies outside {@code [a-z0-9-]+} or is reserved + */ + public DocumentCollection { + Objects.requireNonNull(name, "name"); + Objects.requireNonNull(kind, "kind"); + refusal(name).ifPresent(detail -> { + throw new IllegalArgumentException(detail); + }); + } + + /** + * Creates a collection, refusing an invalid or reserved name as a result. + * + * @param name the name of the collection + * @param kind the kind of the collection + * @return {@link StoreOutcome#OK} with the collection, or {@link StoreOutcome#MALFORMED} with the detail + * {@code collection: invalid name} for a name outside {@code [a-z0-9-]+} or + * {@code collection: reserved name} for {@value #MAIN_NAME} or {@value #CHECKPOINTS_NAME}; the detail + * never carries the name + * @throws NullPointerException if {@code name} or {@code kind} is {@code null} + */ + public static StoreResult of(String name, Kind kind) { + Objects.requireNonNull(name, "name"); + Objects.requireNonNull(kind, "kind"); + return refusal(name).>map(detail -> StoreResult.of(StoreOutcome.MALFORMED, + detail)).orElseGet(() -> StoreResult.ok(new DocumentCollection(name, kind))); + } + + /** Returns the detail refusing a name, or empty for a valid one. */ + private static Optional refusal(String name) { + if (!RepositoryRegistry.ALIAS.matcher(name).matches()) { + return Optional.of("collection: invalid name"); + } + return RESERVED.contains(name) ? Optional.of("collection: reserved name") : Optional.empty(); + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/DocumentKey.java b/git-store/src/main/java/de/cuioss/git/document/store/DocumentKey.java new file mode 100644 index 0000000..562874e --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/DocumentKey.java @@ -0,0 +1,78 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.util.Objects; +import java.util.regex.Pattern; + +/** + * The key of one document or entry of a collection: the collection and the id within it (GDR-STORE-2). + *

+ * The id of a document of a {@link DocumentCollection.Kind#PER_DOCUMENT} collection is its {@link Ulid}. The id of an + * entry of any other collection matches {@code [A-Za-z0-9-]+}: a readable number such as {@code R-042}, or a ULID for + * the draft of a new entry until its first save. Both forms contain no character git-check-ref-format(1) forbids, no + * dot and no slash, so the id is a single valid segment of a reference name. + * + * @param collection the collection + * @param id the id within the collection + * @since 0.1 + */ +public record DocumentKey(DocumentCollection collection, String id) { + + /** The form of an id of a collection that is not per-document. */ + private static final Pattern READABLE_ID = Pattern.compile("[A-Za-z0-9-]+"); + + /** The detail of a refused id; it never carries the id. */ + private static final String INVALID_ID = "id: invalid"; + + /** + * Checks the collection and the id. + * + * @param collection the collection + * @param id the id + * @throws NullPointerException if {@code collection} or {@code id} is {@code null} + * @throws IllegalArgumentException if the id does not fit the kind of the collection + */ + public DocumentKey { + Objects.requireNonNull(collection, "collection"); + Objects.requireNonNull(id, "id"); + if (!isValid(collection, id)) { + throw new IllegalArgumentException(INVALID_ID); + } + } + + /** + * Creates a key, refusing an id that does not fit the kind of the collection as a result. + * + * @param collection the collection + * @param id the id within the collection + * @return {@link StoreOutcome#OK} with the key, or {@link StoreOutcome#MALFORMED} with the detail + * {@code id: invalid} when a per-document id is no ULID or another id lies outside {@code [A-Za-z0-9-]+}; + * the detail never carries the id + * @throws NullPointerException if {@code collection} or {@code id} is {@code null} + */ + public static StoreResult of(DocumentCollection collection, String id) { + Objects.requireNonNull(collection, "collection"); + Objects.requireNonNull(id, "id"); + return isValid(collection, id) ? StoreResult.ok(new DocumentKey(collection, id)) + : StoreResult.of(StoreOutcome.MALFORMED, INVALID_ID); + } + + private static boolean isValid(DocumentCollection collection, String id) { + return collection.kind() == DocumentCollection.Kind.PER_DOCUMENT ? Ulid.isValid(id) + : READABLE_ID.matcher(id).matches(); + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/Draft.java b/git-store/src/main/java/de/cuioss/git/document/store/Draft.java new file mode 100644 index 0000000..dcd1ee5 --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/Draft.java @@ -0,0 +1,47 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.time.Instant; +import java.util.Objects; + +/** + * One draft of a user, as listed from {@code refs/drafts///} (GDR-STORE-2). + * + * @param collection the name of the collection + * @param id the id of the document or entry within the collection + * @param commitId the full id of the draft's tip commit + * @param changedAt the committer time of the draft's tip commit, the time of its last intermediate save + * @since 0.1 + */ +public record Draft(String collection, String id, String commitId, Instant changedAt) { + + /** + * Checks that every component is present. + * + * @param collection the collection name + * @param id the id + * @param commitId the tip commit id + * @param changedAt the time of the last intermediate save + * @throws NullPointerException if a component is {@code null} + */ + public Draft { + Objects.requireNonNull(collection, "collection"); + Objects.requireNonNull(id, "id"); + Objects.requireNonNull(commitId, "commitId"); + Objects.requireNonNull(changedAt, "changedAt"); + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/DraftExpiry.java b/git-store/src/main/java/de/cuioss/git/document/store/DraftExpiry.java new file mode 100644 index 0000000..7f6c60a --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/DraftExpiry.java @@ -0,0 +1,282 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.io.IOException; +import java.time.Instant; +import java.time.Period; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.Objects; + + +import de.cuioss.git.document.core.GitOutcome; +import de.cuioss.git.document.core.GitResult; +import de.cuioss.git.document.core.ReferenceUpdater; +import de.cuioss.tools.logging.CuiLogger; +import org.eclipse.jgit.api.Git; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.eclipse.jgit.api.errors.JGitInternalException; +import org.eclipse.jgit.lib.ConfigConstants; +import org.eclipse.jgit.lib.ObjectId; +import org.eclipse.jgit.lib.ProgressMonitor; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.lib.StoredConfig; +import org.eclipse.jgit.revwalk.RevWalk; + +/** + * The expiry of the drafts of one bare clone (GDR-STORE-7; checked against JGit 7.8.0). + *

+ * Every draft below {@code refs/drafts/} whose tip has a committer time at or before {@code now} minus + * {@code drafts.expiry}, computed in UTC, is deleted by {@link ReferenceUpdater#compareAndDelete} expecting the tip + * that was read as stale, so a draft changed meanwhile is kept. The deletes run under the exclusive side of the + * clone's maintenance lock, so no in-process draft write or save runs while a draft is deleted: a write that found its + * draft present cannot lose it before its compare-and-swap. Every applied-draft reference + * ({@link StoreReferences#APPLIED}) whose draft no longer exists is deleted the same way, so it keeps no draft commit + * alive. + *

+ * Then, still under the exclusive side of that lock, so that no in-process write holds objects not yet referenced, a + * garbage collection removes the objects reachable only from deleted references: {@code Git.wrap(repository).gc()}, JGit's + * {@code GarbageCollectCommand}, with {@code gc.pruneExpire} and {@code gc.prunePackExpire} set to {@code now} in the + * clone's configuration in memory. JGit 7.8.0 {@code GarbageCollectCommand.call} runs {@code GC.gc().get()}, which is + * synchronous unless {@code GC.setBackground(true)}; {@code GC.doGc} packs the references, repacks the reachable + * objects into a new pack, deletes the old packs older than the pack expiry, and prunes the unreferenced loose objects + * older than the expiry ({@code GC.prune}), both read from that configuration ({@code GC.getExpireDate}, + * {@code GC.getPackExpireDate}); with the expiry {@code now} it loosens no object of an old pack + * ({@code GC.deleteOldPacks}), so nothing unreachable survives in a fresh loose object. It runs no hook and starts no + * process: the repository's {@code FS} is asked for file times only. Every reference update of the store disables the + * reflog, so no reflog keeps an object of a deleted draft alive. JGit 7.8.0 {@code GC.doGc} returns without collecting + * and without an error while another collection holds {@code gc.pid} ({@code GC.PidLock}), and calls + * {@code ProgressMonitor.start} only after it took that file; the expiry gives the command a progress monitor + * ({@code GarbageCollectCommand.setProgressMonitor}, passed on to {@code GC.setProgressMonitor}) and, when that monitor + * was never started, ends in {@link GitOutcome#CONFLICT}, which may be retried, so a skipped collection is never + * reported as a successful expiry, however late the other collection took {@code gc.pid}. The drafts it deleted stay + * deleted. + *

+ * A failure detail and the debug log name the exception class only, never its message, which may carry a path + * (GDR-NFR-3). + */ +final class DraftExpiry { + + private static final CuiLogger LOGGER = new CuiLogger(DraftExpiry.class); + + /** The expiry of the garbage collection: every unreferenced object and old pack, git's own word. */ + private static final String NOW = "now"; + + /** The file a running JGit garbage collection holds in the git directory (JGit 7.8.0 {@code GC.PidLock}). */ + static final String GC_PID = "gc.pid"; + + /** The detail of a skipped garbage collection. */ + static final String GC_SKIPPED = "gc: skipped, another collection holds " + GC_PID; + + private final String alias; + private final Repository repository; + private final RepositoryGuard guard; + private final ReferenceUpdater updater; + private final Period expiry; + private final int attempts; + + /** + * Creates the expiry of a clone. + * + * @param alias the alias of the repository + * @param repository the bare clone + * @param guard the in-process coordination of the clone + * @param updater the compare-and-delete updater + * @param expiry the period after which an unchanged draft is deleted ({@code drafts.expiry}) + * @param attempts the attempts of a compare-and-delete, at least 1 + * @throws NullPointerException if an argument is {@code null} + */ + DraftExpiry(String alias, Repository repository, RepositoryGuard guard, ReferenceUpdater updater, Period expiry, + int attempts) { + this.alias = Objects.requireNonNull(alias, "alias"); + this.repository = Objects.requireNonNull(repository, "repository"); + this.guard = Objects.requireNonNull(guard, "guard"); + this.updater = Objects.requireNonNull(updater, "updater"); + this.expiry = Objects.requireNonNull(expiry, "expiry"); + this.attempts = attempts; + } + + /** + * Deletes every draft unchanged for {@code drafts.expiry} and collects the garbage. + * + * @param now the current time + * @return {@link StoreOutcome#OK} with the deleted draft references, sorted, possibly none; the passed-on + * {@link GitOutcome#CONFLICT} when another garbage collection holds {@value #GC_PID}; or the passed-on + * {@link GitOutcome#FAILED} of a failed read, delete or garbage collection, the detail naming the + * exception class or the reference only + * @throws NullPointerException if {@code now} is {@code null} + */ + StoreResult> expire(Instant now) { + Objects.requireNonNull(now, "now"); + Instant cutoff = now.atOffset(ZoneOffset.UTC).minus(expiry).toInstant(); + StoreResult> deleted = guard.exclusive(() -> deleteAndCollect(cutoff)); + if (deleted.isOk()) { + List names = Objects.requireNonNull(deleted.value()); + LOGGER.info(GitStoreLogMessages.INFO.DRAFTS_EXPIRED, alias, names.size()); + } + return deleted; + } + + /** Deletes the stale drafts, then collects the garbage; both under the exclusive side of the maintenance lock. */ + private StoreResult> deleteAndCollect(Instant cutoff) { + StoreResult> deleted = deleteStale(cutoff); + if (!deleted.isOk()) { + return deleted; + } + StoreResult> orphans = deleteOrphanedApplied(); + if (!orphans.isOk()) { + return orphans; + } + StoreResult> collected = collectGarbage(); + return collected.isOk() ? deleted : collected; + } + + /** + * Deletes every applied-draft reference whose draft no longer exists, by compare-and-delete expecting the tip that + * was read, so the draft commits it kept reachable are collected as well; the value of a success is unused. + */ + private StoreResult> deleteOrphanedApplied() { + try { + for (Ref ref : repository.getRefDatabase().getRefsByPrefix(StoreReferences.APPLIED)) { + ObjectId tip = ref.getObjectId(); + if (tip != null && repository.exactRef(StoreReferences.draftOf(ref.getName())) == null) { + var result = updater.compareAndDelete(repository, ref.getName(), tip.name(), attempts); + if (!result.isOk() && result.outcome() != GitOutcome.CONFLICT) { + return StoreResult.passOn(result); + } + } + } + return StoreResult.ok(List.of()); + } catch (IOException e) { + return StoreResult.passOn(DraftReferences.failure(StoreReferences.APPLIED, e)); + } + } + + /** Deletes every draft whose tip lies at or before the cutoff, expecting that tip. */ + private StoreResult> deleteStale(Instant cutoff) { + try (RevWalk walk = new RevWalk(repository)) { + List refs = new ArrayList<>(repository.getRefDatabase().getRefsByPrefix(StoreReferences.DRAFTS)); + refs.sort(Comparator.comparing(Ref::getName)); + List deleted = new ArrayList<>(); + for (Ref ref : refs) { + GitResult decided = deleteIfStale(walk, ref, cutoff); + if (!decided.isOk()) { + return StoreResult.passOn(decided); + } + if (Boolean.TRUE.equals(decided.value())) { + deleted.add(ref.getName()); + } + } + return StoreResult.ok(List.copyOf(deleted)); + } catch (IOException e) { + return StoreResult.passOn(DraftReferences.failure(StoreReferences.DRAFTS, e)); + } + } + + /** + * Deletes one draft if its tip lies at or before the cutoff, expecting that tip. + * + * @return {@code true} for a deleted draft, {@code false} for one that is kept (younger, without a tip, or changed + * meanwhile), or the outcome of a failed delete + */ + private GitResult deleteIfStale(RevWalk walk, Ref ref, Instant cutoff) throws IOException { + ObjectId tip = ref.getObjectId(); + if (tip == null || walk.parseCommit(tip).getCommitterIdent().getWhenAsInstant().isAfter(cutoff)) { + return GitResult.ok(false); + } + var result = updater.compareAndDelete(repository, ref.getName(), tip.name(), attempts); + if (result.isOk()) { + return GitResult.ok(true); + } + if (result.outcome() == GitOutcome.CONFLICT) { + LOGGER.debug("The draft %s changed meanwhile and is kept", ref.getName()); + return GitResult.ok(false); + } + return GitResult.of(result.outcome(), Objects.requireNonNull(result.detail())); + } + + /** + * Collects the garbage with both expiries {@value #NOW}; the value of a success is unused. A collection JGit skips + * because another one holds {@value #GC_PID}, which its progress monitor shows by never being started, is passed on + * as {@link GitOutcome#CONFLICT}, which may be retried. + */ + private StoreResult> collectGarbage() { + StoredConfig config = repository.getConfig(); + config.setString(ConfigConstants.CONFIG_GC_SECTION, null, ConfigConstants.CONFIG_KEY_PRUNEEXPIRE, NOW); + config.setString(ConfigConstants.CONFIG_GC_SECTION, null, ConfigConstants.CONFIG_KEY_PRUNEPACKEXPIRE, NOW); + var started = new StartRecordingMonitor(); + try (Git git = Git.wrap(repository)) { + git.gc().setProgressMonitor(started).call(); + } catch (GitAPIException | JGitInternalException e) { + String type = e.getClass().getSimpleName(); + LOGGER.debug("The garbage collection of %s failed: %s", alias, type); + LOGGER.warn(GitStoreLogMessages.WARN.GARBAGE_COLLECTION_FAILED, alias, type); + return new StoreResult<>(new StoreResult.PassedOn(GitOutcome.FAILED), null, "gc: " + type); + } + if (!started.started()) { + LOGGER.warn(GitStoreLogMessages.WARN.GARBAGE_COLLECTION_SKIPPED, alias); + return new StoreResult<>(new StoreResult.PassedOn(GitOutcome.CONFLICT), null, GC_SKIPPED); + } + return StoreResult.ok(List.of()); + } + + /** + * A progress monitor that records whether the collection started: JGit 7.8.0 {@code GC.doGc} calls + * {@link #start(int)} only after it took {@value #GC_PID}, so a monitor never started marks a skipped collection. + */ + private static final class StartRecordingMonitor implements ProgressMonitor { + + private boolean started; + + boolean started() { + return started; + } + + @Override + public void start(int totalTasks) { + started = true; + } + + @Override + public void beginTask(String title, int totalWork) { + // the progress of a running collection is not reported + } + + @Override + public void update(int completed) { + // the progress of a running collection is not reported + } + + @Override + public void endTask() { + // the progress of a running collection is not reported + } + + @Override + public boolean isCancelled() { + return false; + } + + @Override + public void showDuration(boolean enabled) { + // the progress of a running collection is not reported + } + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/DraftReferences.java b/git-store/src/main/java/de/cuioss/git/document/store/DraftReferences.java new file mode 100644 index 0000000..7372d96 --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/DraftReferences.java @@ -0,0 +1,270 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.io.IOException; +import java.time.Clock; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.Objects; +import java.util.Optional; + + +import de.cuioss.git.document.core.CommitMetadata; +import de.cuioss.git.document.core.GitOutcome; +import de.cuioss.git.document.core.GitResult; +import de.cuioss.git.document.core.InCoreWriter; +import de.cuioss.git.document.core.ReferenceUpdater; +import de.cuioss.git.document.core.TreeEdit; +import de.cuioss.git.document.core.TreeOperation; +import de.cuioss.tools.logging.CuiLogger; +import org.eclipse.jgit.lib.ObjectId; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.lib.RefUpdate; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.revwalk.RevCommit; +import org.eclipse.jgit.revwalk.RevWalk; + +/** + * The draft references of one bare clone (GDR-STORE-2): an intermediate save writes the next commit of + * {@code refs/drafts///}, and the drafts of a user are listed from + * {@code refs/drafts//}. + *

+ * The first commit of a draft that does not exist yet has the tip of the local target branch of its document + * ({@link StoreReferences#target(DocumentKey)}) as its parent, else the tip of {@code refs/heads/main}, else none, so + * it holds the state the user started from. It is written through {@link InCoreWriter} first, and only then is the + * draft reference published by a {@link RefUpdate} that expects it to be absent (the zero id) and writes no reflog, so + * a refused first write, for example two edits of one path, leaves no draft behind. That update's + * {@link RefUpdate.Result#LOCK_FAILURE} means that another writer published the draft first or that its lock could not + * be acquired (JGit 7.8.0 reports both alike): the next attempt reads the draft again and writes on top of the + * published one, or publishes again. + *

+ * Every later draft commit is written by {@link ReferenceUpdater#compareAndSwap}, unsigned and without trailers, with + * the author and committer of {@link CommitIdentities}; its edits are applied onto the draft's current tree. A draft + * that existed when it was read but is absent when the compare-and-swap reads it, deleted meanwhile by a save or + * another process, is never replaced by a root commit holding only the new edits: no commit is written and the next + * attempt writes the first commit of the draft anew on its target branch or {@code main}. After + * {@code save.max-attempts} attempts that all lost their draft or its publication, the write ends in + * {@link GitOutcome#CONFLICT}, which may be retried. + *

+ * A detail names a reference, never a content, a pseudonym beyond the reference or an address (GDR-NFR-3). + */ +final class DraftReferences { + + private static final CuiLogger LOGGER = new CuiLogger(DraftReferences.class); + + private final Repository repository; + private final ReferenceUpdater updater; + private final InCoreWriter writer = new InCoreWriter(); + private final CommitIdentities identities; + private final Clock clock; + private final int attempts; + + /** + * Creates the draft references of a clone. + * + * @param repository the bare clone + * @param updater the compare-and-swap updater + * @param identities the commit identities + * @param clock the clock of the commit times + * @param attempts the attempts of a compare-and-swap, at least 1 + * @throws NullPointerException if an argument is {@code null} + */ + DraftReferences(Repository repository, ReferenceUpdater updater, CommitIdentities identities, Clock clock, + int attempts) { + this.repository = Objects.requireNonNull(repository, "repository"); + this.updater = Objects.requireNonNull(updater, "updater"); + this.identities = Objects.requireNonNull(identities, "identities"); + this.clock = Objects.requireNonNull(clock, "clock"); + this.attempts = attempts; + } + + /** + * Writes an intermediate save: the next commit of the draft reference of the author for the document. + * + * @param author the author, whose pseudonym names the draft reference + * @param key the document or entry + * @param edits the edits, applied onto the draft's current tree + * @param message the commit message + * @return {@link StoreOutcome#OK} with the full id of the new draft commit, {@link StoreOutcome#MALFORMED} for an + * invalid pseudonym, or the passed-on Git outcome that prevented the write (for example + * {@link GitOutcome#NOT_FOUND} for an invalid path, {@link GitOutcome#CONFLICT} after every attempt lost + * its race or its draft, {@link GitOutcome#FAILED} for an I/O failure) + * @throws NullPointerException if an argument is {@code null} + */ + StoreResult write(Author author, DocumentKey key, List edits, String message) { + Objects.requireNonNull(author, "author"); + Objects.requireNonNull(key, "key"); + List copy = List.copyOf(Objects.requireNonNull(edits, "edits")); + Objects.requireNonNull(message, "message"); + var draftRef = StoreReferences.draft(author.pseudonym(), key); + if (!draftRef.isOk()) { + return draftRef; + } + String refName = Objects.requireNonNull(draftRef.value()); + for (int attempt = 1; attempt <= attempts; attempt++) { + Optional> written = attempt(author, key, refName, copy, message); + if (written.isPresent()) { + return written.get(); + } + LOGGER.debug("The draft %s vanished or its publication was lost, attempt %s of %s", refName, attempt, + attempts); + } + return StoreResult.passOn(GitResult.of(GitOutcome.CONFLICT, + refName + ": draft lost or not published on any attempt (" + attempts + ")")); + } + + /** + * One attempt of a draft write: the next commit of an existing draft by compare-and-swap, or the first commit of a + * missing one. + * + * @return the result of the write, or empty when the draft vanished before its compare-and-swap, or its creation + * met a held lock or a draft another writer published first, so the next attempt reads the draft again + */ + private Optional> attempt(Author author, DocumentKey key, String refName, List edits, + String message) { + try { + if (repository.exactRef(refName) == null) { + return create(author, key, refName, edits, message); + } + } catch (IOException e) { + return Optional.of(StoreResult.passOn(failure(refName, e))); + } + TreeOperation operation = current -> { + if (current == null) { + // deleted since it was read, e.g. by a save: a root commit would drop the draft's base + throw new DraftVanished(); + } + return edits; + }; + try { + GitResult written = updater.compareAndSwap(repository, refName, metadata(author, message), + operation, attempts); + return Optional.of(written.isOk() ? StoreResult.ok(Objects.requireNonNull(written.value())) + : StoreResult.passOn(written)); + } catch (DraftVanished vanished) { + return Optional.empty(); + } + } + + /** + * Writes the first commit of a missing draft with the tip of the local target branch as its parent, else the tip + * of {@code main}, else as a root commit, and only then publishes the draft reference, expecting it absent. A + * first write that is refused publishes nothing, so no draft is listed that holds no change of the user. + * + * @return the result of the write, or empty when the publication met a held lock or a draft another writer + * published first; JGit 7.8.0 reports both as {@link RefUpdate.Result#LOCK_FAILURE} + */ + private Optional> create(Author author, DocumentKey key, String refName, List edits, + String message) throws IOException { + Optional base = tip(StoreReferences.target(key)); + if (base.isEmpty()) { + base = tip(StoreReferences.MAIN); + } + List parents = base.map(id -> List.of(id.name())).orElse(List.of()); + GitResult written = writer.write(repository, parents, metadata(author, message), edits); + if (!written.isOk()) { + return Optional.of(StoreResult.passOn(written)); + } + String commit = Objects.requireNonNull(written.value()); + RefUpdate publish = repository.updateRef(refName); + publish.setExpectedOldObjectId(ObjectId.zeroId()); + publish.setNewObjectId(ObjectId.fromString(commit)); + publish.disableRefLog(); + RefUpdate.Result result = publish.update(); + return switch (result) { + case NEW -> Optional.of(StoreResult.ok(commit)); + case LOCK_FAILURE -> { + LOGGER.debug("The publication of the draft %s met a held lock or another writer", refName); + yield Optional.empty(); + } + case IO_FAILURE -> Optional.of(StoreResult.passOn(GitResult.of(GitOutcome.FAILED, refName + ": " + result))); + default -> Optional.of(StoreResult.passOn(GitResult.of(GitOutcome.REJECTED, refName + ": " + result))); + }; + } + + /** The metadata of a draft commit, its times read from the clock now. */ + private CommitMetadata metadata(Author author, String message) { + var now = clock.instant(); + return new CommitMetadata(identities.author(author, now), identities.committer(now), message, null); + } + + /** + * Lists the drafts of a user, sorted by reference name. + * + * @param pseudonym the pseudonym of the user + * @return {@link StoreOutcome#OK} with the drafts, possibly none; {@link StoreOutcome#MALFORMED} for an invalid + * pseudonym; the passed-on {@link GitOutcome#FAILED} for an I/O failure + * @throws NullPointerException if {@code pseudonym} is {@code null} + */ + StoreResult> list(String pseudonym) { + Objects.requireNonNull(pseudonym, "pseudonym"); + if (!StoreReferences.isPseudonym(pseudonym)) { + return StoreResult.of(StoreOutcome.MALFORMED, StoreReferences.INVALID_PSEUDONYM); + } + String prefix = StoreReferences.drafts(pseudonym); + try (RevWalk walk = new RevWalk(repository)) { + List refs = new ArrayList<>(repository.getRefDatabase().getRefsByPrefix(prefix)); + refs.sort(Comparator.comparing(Ref::getName)); + List drafts = new ArrayList<>(refs.size()); + for (Ref ref : refs) { + String[] segments = ref.getName().substring(prefix.length()).split("/"); + ObjectId id = ref.getObjectId(); + if (segments.length != 2 || id == null) { + // only this class writes below refs/drafts/, always as / + LOGGER.debug("Skipped the reference %s, which is no draft of the form /", + ref.getName()); + continue; + } + RevCommit commit = walk.parseCommit(id); + drafts.add(new Draft(segments[0], segments[1], id.name(), + commit.getCommitterIdent().getWhenAsInstant())); + } + return StoreResult.ok(List.copyOf(drafts)); + } catch (IOException e) { + return StoreResult.passOn(failure(prefix, e)); + } + } + + private Optional tip(String refName) throws IOException { + Ref ref = repository.exactRef(refName); + return ref == null ? Optional.empty() : Optional.ofNullable(ref.getObjectId()); + } + + /** + * Builds the failure of an I/O error: the reference and the class of the error, never its message, which may + * carry a path or a content; the debug log carries the same and no exception (GDR-NFR-3). + */ + static GitResult failure(String refName, IOException e) { + String type = e.getClass().getSimpleName(); + LOGGER.debug("I/O failure on %s: %s", refName, type); + return GitResult.of(GitOutcome.FAILED, refName + ": " + type); + } + + /** + * Ends a compare-and-swap whose draft vanished after it was read, before any commit is written; carries no stack + * trace, it is control flow within this class. + */ + private static final class DraftVanished extends RuntimeException { + + private static final long serialVersionUID = 1L; + + DraftVanished() { + super(null, null, false, false); + } + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/ExplicitSave.java b/git-store/src/main/java/de/cuioss/git/document/store/ExplicitSave.java new file mode 100644 index 0000000..1bd3f5b --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/ExplicitSave.java @@ -0,0 +1,605 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.io.IOException; +import java.nio.file.Path; +import java.time.Clock; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.Set; + + +import de.cuioss.git.document.core.CommitMetadata; +import de.cuioss.git.document.core.GitOutcome; +import de.cuioss.git.document.core.GitResult; +import de.cuioss.git.document.core.InCoreWriter; +import de.cuioss.git.document.core.JGitRepositoryOperations; +import de.cuioss.git.document.core.ReferenceUpdater; +import de.cuioss.git.document.core.RepositoryOperations; +import de.cuioss.git.document.core.RepositoryOperations.FetchInput; +import de.cuioss.git.document.core.RepositoryOperations.PushInput; +import de.cuioss.git.document.core.RepositoryReader; +import de.cuioss.git.document.core.RepositoryReader.ChangeType; +import de.cuioss.git.document.core.RepositoryReader.PathChange; +import de.cuioss.git.document.core.TreeEdit; +import de.cuioss.tools.logging.CuiLogger; +import org.eclipse.jgit.lib.Constants; +import org.eclipse.jgit.lib.FileMode; +import org.eclipse.jgit.lib.ObjectId; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.lib.RefUpdate; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.revwalk.RevCommit; +import org.eclipse.jgit.revwalk.RevTree; +import org.eclipse.jgit.revwalk.RevWalk; +import org.eclipse.jgit.revwalk.filter.RevFilter; +import org.eclipse.jgit.treewalk.TreeWalk; +import org.eclipse.jgit.treewalk.filter.PathFilter; +import org.jspecify.annotations.Nullable; + +/** + * The explicit save of one bare clone (GDR-STORE-3, GDR-STORE-4, GDR-STORE-5, GDR-STORE-8): the draft's changes + * applied onto the remote tip of the target branch as one commit, pushed with lease. + *

+ * A save runs under the shared side of the clone's maintenance lock and under the lock of its draft reference, and + * each of its attempts under the clone's transport lock ({@link RepositoryGuard}). The draft lock serialises the saves + * of one draft within the process: a save reads the draft tip and its applied-draft reference only after every + * earlier save of that draft has settled, so it never applies again the changes an earlier save applied; a failed + * write of the applied-draft reference for a kept draft is outside that guarantee (see {@code setApplied}). The save + * reads the draft tip once, and every attempt applies that draft commit, so a draft written during the save is not + * published with the expectations and the message of this save. An attempt fetches with the mirror reference + * specifications {@link JGitRepositoryOperations#MIRROR_FETCH_REF_SPECS}, passed explicitly, and the credentials of + * the technical user, pruning every reference below {@code refs/heads/}, {@code refs/tags/} and {@code refs/notes/} + * the remote deleted: JGit 7.8.0 prunes only the destinations of the given reference specifications, so a draft or an + * applied-draft reference is never pruned, whatever fetch reference specifications the clone's configuration holds. + * It takes the local tip of the target branch as the base; a target branch that does not exist (any more) starts at the local tip of {@code main} (else at + * nothing, so the commit is a root commit) and is pushed against the zero-id lease, which expects the remote branch to + * be absent. The changes of the draft are the paths changed since the draft commit a former save applied + * ({@link StoreReferences#APPLIED}), when the draft descends from it; otherwise between the merge base of draft tip + * and base ({@code RevFilter.MERGE_BASE}) and the draft tip ({@link RepositoryReader#changes}); without a merge base + * every file of the draft is a change. Every changed path with an expected blob id is checked against the base, the + * zero id expecting the path to be absent there, and every written path that would turn a file of the base into a + * directory, or replace a directory of the base holding a file the draft did not change, is checked for that overlap; + * every differing or overlapping path is a conflict: nothing is pushed, draft and target branch stay as they are. + * Otherwise one commit with the base as its only parent and the changes as its edits is written through + * {@link InCoreWriter}, unsigned, without trailers, with the author and committer of {@link CommitIdentities}, and + * pushed to the target with the lease of the base. + *

+ * The save is a cooperative compare-and-swap against the remote: a lease refusal or a rejection starts the next + * attempt from the fetch, up to the configured attempts, then the save is {@link GitOutcome#REJECTED}; an unreachable + * remote or invalid credentials end the save at once with nothing changed. Only after the remote accepted the commit + * the local target branch moves by a {@link RefUpdate} expecting the base, with the reflog disabled (an I/O failure + * there is logged and the save stays {@link SaveResult.Saved}), and the draft is deleted by + * {@link ReferenceUpdater#compareAndDelete} expecting the applied draft commit, so a newer draft is kept; for a kept + * draft the applied-draft reference records the applied commit, otherwise it is deleted with the draft. A draft + * without changes is saved as the tip of an existing target branch, with no commit and no push; without the target + * branch it is {@link GitOutcome#NOT_FOUND} and kept. The save commit is the only commit pushed; no other commit the + * store writes is combined with it. + *

+ * A detail names a reference or a field, never a path, a content, a pseudonym or an address (GDR-NFR-3). + */ +final class ExplicitSave { + + private static final CuiLogger LOGGER = new CuiLogger(ExplicitSave.class); + + /** The remote of every bare clone of the store. */ + private static final String ORIGIN = "origin"; + + /** The detail of a save whose draft does not exist. */ + private static final String DRAFT_NOT_FOUND = "draft: not found"; + + /** The detail of an expected blob id that is no full object id. */ + private static final String INVALID_EXPECTED_BLOB_ID = "expectedBlobIds: not a full object id"; + + /** The detail of a draft without changes in a repository without its target branch. */ + private static final String NOTHING_TO_SAVE = "draft: no changes and no target branch"; + + private final Clone clone; + private final RepositoryOperations operations; + private final TechnicalUser user; + private final CommitIdentities identities; + private final Clock clock; + private final int attempts; + private final InCoreWriter writer = new InCoreWriter(); + private final ReferenceUpdater updater = new ReferenceUpdater(writer); + private final RepositoryReader reader = new RepositoryReader(); + + /** + * The bare clone a save writes to. + * + * @param alias the alias of the repository + * @param repository the opened bare clone + * @param gitDir the git directory of the bare clone, for the transport operations + * @param guard the in-process coordination of the clone + */ + record Clone(String alias, Repository repository, Path gitDir, RepositoryGuard guard) { + + /** + * Checks the components. + * + * @param alias the alias + * @param repository the opened clone + * @param gitDir the git directory + * @param guard the guard + * @throws NullPointerException if a component is {@code null} + */ + Clone { + Objects.requireNonNull(alias, "alias"); + Objects.requireNonNull(repository, "repository"); + Objects.requireNonNull(gitDir, "gitDir"); + Objects.requireNonNull(guard, "guard"); + } + } + + /** + * Creates the explicit save of a clone. + * + * @param clone the bare clone + * @param operations the repository operations of {@code git-core}, for fetch and push + * @param user the technical user, holder of the credentials of every fetch and push + * @param identities the commit identities + * @param clock the clock of the commit times + * @param attempts the attempts of a save ({@code save.max-attempts}), at least 1 + * @throws NullPointerException if an argument is {@code null} + * @throws IllegalArgumentException if {@code attempts} is less than 1 + */ + ExplicitSave(Clone clone, RepositoryOperations operations, TechnicalUser user, CommitIdentities identities, + Clock clock, int attempts) { + this.clone = Objects.requireNonNull(clone, "clone"); + this.operations = Objects.requireNonNull(operations, "operations"); + this.user = Objects.requireNonNull(user, "user"); + this.identities = Objects.requireNonNull(identities, "identities"); + this.clock = Objects.requireNonNull(clock, "clock"); + if (attempts < 1) { + throw new IllegalArgumentException("attempts must be at least 1, was " + attempts); + } + this.attempts = attempts; + } + + /** + * Saves the draft of the author for the document onto its target branch. + * + * @param author the author, whose pseudonym names the draft reference + * @param key the document or entry + * @param expectedBlobIds the blob id the caller loaded per path; the zero id expects the path to be absent + * @param message the commit message + * @return {@link SaveResult.Saved} with the commit the target branch points at; {@link SaveResult.Conflicting} + * with every path whose blob on the fetched tip differs from its expected id; {@link SaveResult.NotSaved} + * with {@link StoreOutcome#MALFORMED} for an invalid pseudonym, {@link GitOutcome#NOT_FOUND} for a missing + * draft or an expected blob id that is no full object id, {@link GitOutcome#REJECTED} after every attempt + * lost its lease or was rejected, or the passed-on outcome that ended the save, for example + * {@link GitOutcome#UNREACHABLE} or {@link GitOutcome#CREDENTIALS_INVALID} + * @throws NullPointerException if an argument, a path or an expected blob id is {@code null} + */ + SaveResult save(Author author, DocumentKey key, Map expectedBlobIds, String message) { + Objects.requireNonNull(author, "author"); + Objects.requireNonNull(key, "key"); + Map expectations = Map.copyOf(Objects.requireNonNull(expectedBlobIds, "expectedBlobIds")); + Objects.requireNonNull(message, "message"); + String target = StoreReferences.target(key); + Map expected = new HashMap<>(); + for (Map.Entry expectation : expectations.entrySet()) { + if (!ObjectId.isId(expectation.getValue())) { + return report(target, notSaved(GitOutcome.NOT_FOUND, INVALID_EXPECTED_BLOB_ID)); + } + expected.put(expectation.getKey(), ObjectId.fromString(expectation.getValue())); + } + var draftRef = StoreReferences.draft(author.pseudonym(), key); + if (!draftRef.isOk()) { + return report(target, SaveResult.NotSaved.of(draftRef)); + } + var request = new Request(author, target, Objects.requireNonNull(draftRef.value()), expected, message); + RepositoryGuard guard = clone.guard(); + return report(target, guard.shared(() -> guard.draft(request.draft(), () -> saveDraft(request)))); + } + + /** + * Reads the draft once and runs the attempts on that draft commit, all under the shared side of the maintenance + * lock, so no expiry deletes or collects the draft between two attempts, and under the lock of the draft, so the + * draft and its applied-draft reference are read after every earlier save of the draft has settled. + */ + private SaveResult saveDraft(Request request) { + Repository repository = clone.repository(); + Ref draftTip; + Ref appliedTip; + try { + draftTip = repository.exactRef(request.draft()); + appliedTip = repository.exactRef(StoreReferences.applied(request.draft())); + } catch (IOException e) { + return failed(request, e); + } + @Nullable ObjectId draft = draftTip == null ? null : draftTip.getObjectId(); + if (draft == null) { + return notSaved(GitOutcome.NOT_FOUND, DRAFT_NOT_FOUND); + } + var drafted = new Drafted(draft, appliedTip == null ? null : appliedTip.getObjectId()); + for (int attempt = 1; attempt <= attempts; attempt++) { + Optional done = clone.guard().transport(() -> attempt(request, drafted)); + if (done.isPresent()) { + return done.get(); + } + LOGGER.debug("The save onto %s lost its lease or was rejected, attempt %s of %s", request.target(), + attempt, attempts); + } + return notSaved(GitOutcome.REJECTED, + request.target() + ": lease refused or rejected on every attempt (" + attempts + ")"); + } + + /** + * Runs one attempt on the draft commit the save started with: fetches, pruning the mirrored references the remote + * deleted, and applies the draft's changes onto the fetched base. + * + * @return the result of the save, or empty when the push lost its lease or was rejected and the next attempt + * starts from the fetch + */ + private Optional attempt(Request request, Drafted drafted) { + Repository repository = clone.repository(); + try { + var fetched = operations.fetch(FetchInput.builder().root(clone.gitDir()).remote(ORIGIN) + .refSpecs(JGitRepositoryOperations.MIRROR_FETCH_REF_SPECS).credentials(user.credentials()) + .prune(true).build()); + if (!fetched.isOk()) { + return Optional.of(passedOn(fetched)); + } + Base base = base(repository, request.target()); + try (RevWalk walk = new RevWalk(repository)) { + RevCommit draftCommit = walk.parseCommit(drafted.draft()); + @Nullable ObjectId baseId = base.commit(); + @Nullable RevCommit baseCommit = baseId == null ? null : walk.parseCommit(baseId); + var changes = changes(repository, draftCommit, baseCommit, drafted.applied()); + if (!changes.isOk()) { + return Optional.of(passedOn(changes)); + } + List changed = Objects.requireNonNull(changes.value()); + if (changed.isEmpty()) { + return Optional.of(unchanged(request, drafted.draft(), base)); + } + List conflicts = conflicts(repository, changed, request.expected(), baseCommit); + if (!conflicts.isEmpty()) { + return Optional.of(new SaveResult.Conflicting(conflicts)); + } + return apply(request, drafted.draft(), base, edits(repository, changed, draftCommit.getTree())); + } + } catch (IOException e) { + return Optional.of(failed(request, e)); + } + } + + /** The failure of an I/O error: the target and the class of the error, never its message. */ + private static SaveResult failed(Request request, IOException e) { + LOGGER.debug("I/O failure of the save onto %s: %s", request.target(), e.getClass().getSimpleName()); + return notSaved(GitOutcome.FAILED, request.target() + ": " + e.getClass().getSimpleName()); + } + + /** + * Writes the save commit, pushes it with the lease and, once accepted, moves the local branch and settles the + * draft. A local branch that cannot be moved after the remote accepted the commit does not undo the save: it is + * logged by the class of the error only, and the next fetch mirrors the remote branch into the clone. + */ + private Optional apply(Request request, ObjectId draft, Base base, List edits) + throws IOException { + var now = clock.instant(); + var metadata = new CommitMetadata(identities.author(request.author(), now), identities.committer(now), + request.message(), null); + @Nullable ObjectId parent = base.commit(); + List parents = parent == null ? List.of() : List.of(parent.name()); + GitResult written = writer.write(clone.repository(), parents, metadata, edits); + if (!written.isOk()) { + return Optional.of(passedOn(written)); + } + String commit = Objects.requireNonNull(written.value()); + var pushed = operations.push(PushInput.builder().root(clone.gitDir()).remote(ORIGIN).localRef(commit) + .remoteRef(request.target()).expectedRemoteHead(base.lease().name()).credentials(user.credentials()) + .build()); + if (pushed.outcome() == GitOutcome.LEASE_REFUSED || pushed.outcome() == GitOutcome.REJECTED) { + return Optional.empty(); + } + if (!pushed.isOk()) { + return Optional.of(passedOn(pushed)); + } + try { + moveLocal(request.target(), base.localExpected(), ObjectId.fromString(commit)); + } catch (IOException e) { + String type = e.getClass().getSimpleName(); + LOGGER.debug("The local branch %s was not moved after the accepted save: %s", request.target(), type); + LOGGER.warn(GitStoreLogMessages.WARN.LOCAL_BRANCH_NOT_MOVED, clone.alias(), request.target(), type); + } + settle(request.draft(), draft); + return Optional.of(new SaveResult.Saved(commit)); + } + + /** + * A draft without changes. On an existing target branch it is saved as the branch tip itself, with no commit and + * no push, and the draft is settled; without the target branch nothing is saved and the draft is kept, because no + * branch exists that a {@link SaveResult.Saved} could name. + */ + private SaveResult unchanged(Request request, ObjectId draft, Base base) { + @Nullable ObjectId tip = base.commit(); + if (tip == null || ObjectId.zeroId().equals(base.lease())) { + return notSaved(GitOutcome.NOT_FOUND, NOTHING_TO_SAVE); + } + settle(request.draft(), draft); + return new SaveResult.Saved(tip.name()); + } + + /** + * The base of an attempt: the local tip of the target branch, else of {@code main} with the zero-id lease, else + * nothing with the zero-id lease. The fetch before has pruned a target branch the remote deleted. + */ + private static Base base(Repository repository, String target) throws IOException { + Ref targetRef = repository.exactRef(target); + @Nullable ObjectId tip = targetRef == null ? null : targetRef.getObjectId(); + if (tip != null) { + return new Base(tip, tip, tip); + } + Ref main = repository.exactRef(StoreReferences.MAIN); + @Nullable ObjectId start = main == null ? null : main.getObjectId(); + return new Base(start, ObjectId.zeroId(), ObjectId.zeroId()); + } + + /** + * The paths the draft changed: since the draft commit a former save applied, when the draft descends from it; + * otherwise between the merge base of draft tip and base and the draft tip, or every file of the draft when there + * is no merge base. + */ + private GitResult> changes(Repository repository, RevCommit draft, @Nullable RevCommit base, + @Nullable ObjectId applied) throws IOException { + if (applied != null && descends(repository, draft, applied)) { + return reader.changes(repository, applied.name(), draft.name()); + } + @Nullable ObjectId mergeBase = base == null ? null : mergeBase(repository, draft, base); + if (mergeBase != null) { + return reader.changes(repository, mergeBase.name(), draft.name()); + } + var snapshot = reader.snapshot(repository, draft.name()); + if (!snapshot.isOk()) { + return GitResult.of(snapshot.outcome(), Objects.requireNonNull(snapshot.detail())); + } + return GitResult.ok(Objects.requireNonNull(snapshot.value()).files().stream() + .map(file -> new PathChange(file.path(), ChangeType.ADDED)).toList()); + } + + private static @Nullable ObjectId mergeBase(Repository repository, RevCommit draft, RevCommit base) + throws IOException { + try (RevWalk walk = new RevWalk(repository)) { + walk.setRevFilter(RevFilter.MERGE_BASE); + walk.markStart(walk.parseCommit(draft)); + walk.markStart(walk.parseCommit(base)); + RevCommit found = walk.next(); + return found == null ? null : found.copy(); + } + } + + /** Whether the draft commit descends from, or is, the commit a former save applied. */ + private static boolean descends(Repository repository, RevCommit draft, ObjectId applied) throws IOException { + try (RevWalk walk = new RevWalk(repository)) { + return walk.isMergedInto(walk.parseCommit(applied), walk.parseCommit(draft)); + } + } + + /** + * Every changed path with an expected blob id that differs from the path's blob on the base, and every written + * path that overlaps the base as a file and a directory in a way the draft did not resolve. + */ + private static List conflicts(Repository repository, List changed, + Map expected, @Nullable RevCommit base) throws IOException { + Set changedPaths = new HashSet<>(); + changed.forEach(change -> changedPaths.add(change.path())); + List conflicts = new ArrayList<>(); + for (PathChange change : changed) { + ObjectId expectation = expected.get(change.path()); + boolean differs = expectation != null && !expectation.equals(blob(repository, change.path(), base)); + boolean overlaps = change.type() != ChangeType.DELETED && base != null + && overlaps(repository, change.path(), changedPaths, base); + if (differs || overlaps) { + conflicts.add(change.path()); + } + } + return conflicts; + } + + /** + * Whether writing a file at the path would replace what the draft did not change on the base: a file of the base + * at one of the path's parents, which the edit turns into a directory, or a file of the base below the path, which + * the edit replaces by the file. A change of the draft at that place, such as the delete of the parent file, + * resolves the overlap. + */ + private static boolean overlaps(Repository repository, String path, Set changed, RevCommit base) + throws IOException { + for (int slash = path.indexOf('/'); slash > 0; slash = path.indexOf('/', slash + 1)) { + String parent = path.substring(0, slash); + if (!changed.contains(parent) && isFile(repository, parent, base)) { + return true; + } + } + try (TreeWalk below = new TreeWalk(repository)) { + below.addTree(base.getTree()); + below.setRecursive(true); + below.setFilter(PathFilter.create(path)); + while (below.next()) { + if (!changed.contains(below.getPathString())) { + return true; + } + } + } + return false; + } + + /** Whether the commit holds a file, anything but a directory, at the path. */ + private static boolean isFile(Repository repository, String path, RevCommit commit) throws IOException { + try (TreeWalk found = TreeWalk.forPath(repository, path, commit.getTree())) { + return found != null && !FileMode.TREE.equals(found.getRawMode(0)); + } + } + + /** The blob id of a path on a commit; the zero id when the commit or the path is absent. */ + private static ObjectId blob(Repository repository, String path, @Nullable RevCommit commit) throws IOException { + if (commit == null) { + return ObjectId.zeroId(); + } + try (TreeWalk found = TreeWalk.forPath(repository, path, commit.getTree())) { + return found == null ? ObjectId.zeroId() : found.getObjectId(0); + } + } + + /** The edits of the save: the content of every added or modified path of the draft, a delete of every other. */ + private static List edits(Repository repository, List changed, RevTree draftTree) + throws IOException { + List edits = new ArrayList<>(changed.size()); + for (PathChange change : changed) { + if (change.type() == ChangeType.DELETED) { + edits.add(new TreeEdit.Delete(change.path())); + } else { + try (TreeWalk found = TreeWalk.forPath(repository, change.path(), draftTree)) { + ObjectId blob = Objects.requireNonNull(found, "a changed path exists on the draft").getObjectId(0); + byte[] content = repository.open(blob, Constants.OBJ_BLOB).getBytes(Integer.MAX_VALUE); + edits.add(new TreeEdit.Put(change.path(), content)); + } + } + } + return edits; + } + + /** + * Moves the local target branch to the accepted commit, expecting the base. A failed move changes nothing the + * remote holds; the next fetch mirrors the remote branch into the clone. + */ + private void moveLocal(String target, ObjectId expected, ObjectId commit) throws IOException { + RefUpdate update = clone.repository().updateRef(target); + update.setExpectedOldObjectId(expected); + update.setNewObjectId(commit); + update.disableRefLog(); + RefUpdate.Result result = update.update(); + if (result != RefUpdate.Result.NEW && result != RefUpdate.Result.FAST_FORWARD) { + LOGGER.debug("The local branch %s was not moved after the accepted save: %s", target, result); + } + } + + /** + * Settles the draft after a save: deletes it by compare-and-delete, expecting the applied draft commit, together + * with its applied-draft reference. A draft that moved meanwhile is kept, and its applied-draft reference is set to + * the applied commit, so its next save applies only what it changed since. + */ + private void settle(String draftRef, ObjectId applied) { + String appliedRef = StoreReferences.applied(draftRef); + var deleted = updater.compareAndDelete(clone.repository(), draftRef, applied.name(), attempts); + if (deleted.isOk()) { + setApplied(appliedRef, null); + } else if (deleted.outcome() == GitOutcome.CONFLICT) { + LOGGER.debug("The draft moved during the save and is kept"); + setApplied(appliedRef, applied); + } else { + LOGGER.debug("The draft was kept after the save: %s", deleted.outcome()); + } + } + + /** + * Points the applied-draft reference at the commit, or deletes it for {@code null}; without a reflog. Only a save + * under the draft and transport locks and the expiry under the exclusive lock write it, so the update is forced. A + * failure is logged by the class of the error only: the next save then applies the draft's changes since its merge + * base. + */ + private void setApplied(String appliedRef, @Nullable ObjectId commit) { + Repository repository = clone.repository(); + try { + if (commit == null && repository.exactRef(appliedRef) == null) { + return; + } + RefUpdate update = repository.updateRef(appliedRef); + update.setForceUpdate(true); + update.disableRefLog(); + RefUpdate.Result result; + if (commit == null) { + result = update.delete(); + } else { + update.setNewObjectId(commit); + result = update.update(); + } + LOGGER.debug("The applied-draft reference was written: %s", result); + } catch (IOException e) { + LOGGER.debug("The applied-draft reference was not written: %s", e.getClass().getSimpleName()); + } + } + + /** Logs the result of a save and returns it. */ + private SaveResult report(String target, SaveResult result) { + switch (result) { + case SaveResult.Saved(String commit) -> LOGGER.info(GitStoreLogMessages.INFO.SAVED, clone.alias(), target, + commit); + case SaveResult.Conflicting(List paths) -> LOGGER.info(GitStoreLogMessages.INFO.SAVE_CONFLICT, + clone.alias(), target, paths.size()); + case SaveResult.NotSaved notSaved -> LOGGER.warn(GitStoreLogMessages.WARN.SAVE_NOT_COMPLETED, + clone.alias(), target, name(notSaved.outcome())); + } + return result; + } + + private static String name(StoreResult.Outcome outcome) { + return switch (outcome) { + case StoreOutcome store -> store.name(); + case StoreResult.PassedOn(GitOutcome git) -> git.name(); + }; + } + + private static SaveResult.NotSaved notSaved(GitOutcome outcome, String detail) { + return new SaveResult.NotSaved(new StoreResult.PassedOn(outcome), detail); + } + + private static SaveResult.NotSaved passedOn(GitResult result) { + GitOutcome outcome = result.outcome(); + @Nullable String detail = result.detail(); + String shown = detail == null ? outcome.name() : detail; + return new SaveResult.NotSaved(new StoreResult.PassedOn(outcome), shown); + } + + /** + * The draft a save works on, read once at its start. + * + * @param draft the draft commit every attempt applies + * @param applied the draft commit a former save applied while a newer draft was kept; {@code null} when none + */ + private record Drafted(ObjectId draft, @Nullable ObjectId applied) { + } + + /** + * The input of a save. + * + * @param author the author + * @param target the target branch + * @param draft the draft reference + * @param expected the expected blob id per path + * @param message the commit message + */ + private record Request(Author author, String target, String draft, Map expected, + String message) { + } + + /** + * The base of an attempt. + * + * @param commit the parent of the save commit; {@code null} for a root commit + * @param lease the id the remote target branch is expected at, the zero id for an absent branch + * @param localExpected the id the local target branch is expected at when it moves, the zero id for an absent one + */ + private record Base(@Nullable ObjectId commit, ObjectId lease, ObjectId localExpected) { + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/GitStoreLogMessages.java b/git-store/src/main/java/de/cuioss/git/document/store/GitStoreLogMessages.java index 0bb75a8..757204a 100644 --- a/git-store/src/main/java/de/cuioss/git/document/store/GitStoreLogMessages.java +++ b/git-store/src/main/java/de/cuioss/git/document/store/GitStoreLogMessages.java @@ -25,8 +25,9 @@ * {@code git-store/doc/LogMessages.adoc}. *

* No message carries anything GDR-NFR-3 forbids: a message names a provider id, a configuration key, a repository - * alias or a path, never a configured value, a key reference, key material, a token, a password or the user - * information of a remote. + * alias, a path, a reference, a commit id, an outcome or a number, never a configured value, a key reference, key + * material, a token, a password, the user information of a remote, a content, an e-mail address or a person name. The + * messages of a save name no path of the saved document, only the number of conflicting paths. * * @since 0.1 */ @@ -53,6 +54,30 @@ public static final class INFO { .prefix(PREFIX) .identifier(1) .build(); + + /** A save was accepted by the remote: the repository alias, the target reference and the commit id. */ + public static final LogRecord SAVED = LogRecordModel.builder() + .template("Saved repository %s, reference %s at commit %s") + .prefix(PREFIX) + .identifier(2) + .build(); + + /** + * A save conflicted and pushed nothing: the repository alias, the target reference and the number of + * conflicting paths, never a path. + */ + public static final LogRecord SAVE_CONFLICT = LogRecordModel.builder() + .template("Did not save repository %s, reference %s: %s conflicting paths") + .prefix(PREFIX) + .identifier(3) + .build(); + + /** The expired drafts were deleted and the garbage collected: the repository alias and the number deleted. */ + public static final LogRecord DRAFTS_EXPIRED = LogRecordModel.builder() + .template("Expired the drafts of repository %s: %s deleted") + .prefix(PREFIX) + .identifier(4) + .build(); } /** @@ -72,5 +97,45 @@ public static final class WARN { .prefix(PREFIX) .identifier(100) .build(); + + /** + * A save did not complete, for a reason other than a conflict: the repository alias, the target reference and + * the outcome. + */ + public static final LogRecord SAVE_NOT_COMPLETED = LogRecordModel.builder() + .template("Did not complete the save of repository %s, reference %s: %s") + .prefix(PREFIX) + .identifier(101) + .build(); + + /** + * The garbage collection after the draft expiry failed: the repository alias and the class of the exception, + * never its message. + */ + public static final LogRecord GARBAGE_COLLECTION_FAILED = LogRecordModel.builder() + .template("The garbage collection of repository %s failed: %s") + .prefix(PREFIX) + .identifier(102) + .build(); + + /** + * The remote accepted a save, but the local target branch could not be moved: the repository alias, the + * target reference and the class of the exception, never its message; the next fetch mirrors the remote branch. + */ + public static final LogRecord LOCAL_BRANCH_NOT_MOVED = LogRecordModel.builder() + .template("Did not move the local branch of repository %s, reference %s after the accepted save: %s") + .prefix(PREFIX) + .identifier(103) + .build(); + + /** + * The garbage collection after the draft expiry was not run because another collection holds the clone's + * {@code gc.pid}: the repository alias. + */ + public static final LogRecord GARBAGE_COLLECTION_SKIPPED = LogRecordModel.builder() + .template("Skipped the garbage collection of repository %s: another collection holds gc.pid") + .prefix(PREFIX) + .identifier(104) + .build(); } } diff --git a/git-store/src/main/java/de/cuioss/git/document/store/RepositoryGuard.java b/git-store/src/main/java/de/cuioss/git/document/store/RepositoryGuard.java new file mode 100644 index 0000000..4782b0d --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/RepositoryGuard.java @@ -0,0 +1,137 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.io.IOException; +import java.nio.file.Path; +import java.util.Objects; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.locks.Lock; +import java.util.concurrent.locks.ReentrantLock; +import java.util.concurrent.locks.ReentrantReadWriteLock; +import java.util.function.Supplier; + +/** + * The in-process coordination of one bare clone: one guard per clone and JVM ({@link #of(Path)}), shared by every + * {@link StoreWriter} that opens the clone. + *

+ * Draft writes and saves take the shared side of the maintenance lock, the expiry of drafts, its deletes and its garbage + * collection, the exclusive side, so no write runs while the expiry deletes references and objects. One lock per draft + * reference serialises the saves of that draft, from the read of the draft and its applied-draft reference to the + * settlement, so a save reads both only after every earlier save of the draft has settled. One transport lock + * serialises the fetch, apply and push of a save attempt, so two saves of the clone never interleave their transport. + * The order is fixed: the shared side of the maintenance lock, then the draft lock, then the transport lock per + * attempt; the expiry takes the exclusive side only, so no cycle arises. The locks are in-process only. + *

+ * The guard does not close the check-then-act window between reading a draft reference and moving it: that window is + * closed by the lock-held expected-id comparison of {@link de.cuioss.git.document.core.ReferenceUpdater}, which also + * protects against a writer outside this process. + */ +final class RepositoryGuard { + + /** The guard of every clone opened in this JVM, by the real path of its git directory; never removed. */ + private static final ConcurrentMap GUARDS = new ConcurrentHashMap<>(); + + private final ReentrantReadWriteLock maintenance = new ReentrantReadWriteLock(); + private final ReentrantLock transport = new ReentrantLock(); + + /** + * The lock of every draft reference saved through the clone, by its full name; never removed, so two saves of one + * draft always meet the same lock. + */ + private final ConcurrentMap drafts = new ConcurrentHashMap<>(); + + /** + * Returns the one guard of a clone, shared by every {@link StoreWriter} of the JVM that opens it, so the garbage + * collection of one writer never runs while another writer of the same clone holds objects not yet referenced. + * The key is the real path of the git directory ({@link Path#toRealPath}), else its normalised absolute path, so + * two spellings of one directory share the guard. + * + * @param clone the git directory of the clone + * @return the guard of the clone + * @throws NullPointerException if {@code clone} is {@code null} + */ + static RepositoryGuard of(Path clone) { + return GUARDS.computeIfAbsent(canonical(Objects.requireNonNull(clone, "clone")), path -> new RepositoryGuard()); + } + + private static Path canonical(Path clone) { + try { + return clone.toRealPath(); + } catch (IOException e) { + return clone.toAbsolutePath().normalize(); + } + } + + /** + * Runs an action under the shared side of the maintenance lock: a draft write or a save. + * + * @param action the action + * @param the result type + * @return the result of the action + */ + T shared(Supplier action) { + return locked(maintenance.readLock(), action); + } + + /** + * Runs an action under the exclusive side of the maintenance lock: the deletes and the garbage collection of the + * draft expiry. + * + * @param action the action + * @param the result type + * @return the result of the action + */ + T exclusive(Supplier action) { + return locked(maintenance.writeLock(), action); + } + + /** + * Runs an action under the lock of one draft reference: one save of that draft, from its read to its settlement. + * + * @param draftRef the full name of the draft reference + * @param action the action + * @param the result type + * @return the result of the action + * @throws NullPointerException if an argument is {@code null} + */ + T draft(String draftRef, Supplier action) { + Objects.requireNonNull(draftRef, "draftRef"); + return locked(drafts.computeIfAbsent(draftRef, name -> new ReentrantLock()), action); + } + + /** + * Runs an action under the transport lock: the fetch, apply and push of one save attempt. + * + * @param action the action + * @param the result type + * @return the result of the action + */ + T transport(Supplier action) { + return locked(transport, action); + } + + private static T locked(Lock lock, Supplier action) { + Objects.requireNonNull(action, "action"); + lock.lock(); + try { + return action.get(); + } finally { + lock.unlock(); + } + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/SaveResult.java b/git-store/src/main/java/de/cuioss/git/document/store/SaveResult.java new file mode 100644 index 0000000..de48172 --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/SaveResult.java @@ -0,0 +1,225 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.util.List; +import java.util.Objects; +import java.util.Optional; +import java.util.TreeSet; + + +import de.cuioss.git.document.core.GitIdentity; +import de.cuioss.git.document.core.GitOutcome; + +/** + * The result of an explicit save (GDR-STORE-3, GDR-STORE-4), which carries the conflicting paths as data. + *

+ * A save result has the shape GDR-GEN-3 prescribes, an {@linkplain #outcome() outcome} and a {@linkplain #detail() + * detail} for every non-success that carries nothing GDR-NFR-3 forbids, plus the data of its variant: the commit of + * {@link Saved}, the conflicting paths of {@link Conflicting}. It composes that shape instead of extending + * {@link StoreResult}, because a {@link StoreResult} that is no success carries no value, and a conflict that names its + * paths would break that invariant; it is modelled on {@code MergeResult} of {@code git-core}. A caller branches with a + * pattern-matching {@code switch} over the three variants; {@link #toResult()} gives the plain GDR-GEN-3 shape to a + * caller that needs no more. + * + * @since 0.1 + */ +public sealed interface SaveResult permits SaveResult.Saved, SaveResult.Conflicting, SaveResult.NotSaved { + + /** + * The outcome of the save. + * + * @return {@link StoreOutcome#OK} for {@link Saved}, {@link StoreResult.PassedOn} of {@link GitOutcome#CONFLICT} + * for {@link Conflicting}, the recorded outcome for {@link NotSaved}; never {@code null} + */ + StoreResult.Outcome outcome(); + + /** + * The detail of a non-success. + * + * @return the detail, present exactly when the save did not succeed + */ + Optional detail(); + + /** + * Converts the result into the plain GDR-GEN-3 shape. + * + * @return {@link StoreOutcome#OK} with the commit id for {@link Saved}; otherwise the outcome and the detail, + * without value, so the conflicting paths are not carried over + */ + default StoreResult toResult() { + return switch (this) { + case Saved(String commitId) -> StoreResult.ok(commitId); + case Conflicting conflicting -> + new StoreResult<>(conflicting.outcome(), null, conflicting.detail().orElseThrow()); + case NotSaved(StoreResult.Outcome outcome, String reason) -> new StoreResult<>(outcome, null, reason); + }; + } + + /** + * A save the remote accepted, or a draft without changes, which needs no commit. + * + * @param commitId the full id of the commit the target branch points at after the save: the new commit, or the + * unchanged base for a draft without changes + * @since 0.1 + */ + record Saved(String commitId) implements SaveResult { + + /** + * Checks the commit id. + * + * @param commitId the full id of the commit + * @throws NullPointerException if {@code commitId} is {@code null} + * @throws IllegalArgumentException if {@code commitId} is no full 40-digit object id + */ + public Saved { + Objects.requireNonNull(commitId, "commitId"); + if (!GitIdentity.isFullCommitId(commitId)) { + throw new IllegalArgumentException("commitId: not a full object id"); + } + } + + /** + * The outcome of a save. + * + * @return {@link StoreOutcome#OK} + */ + @Override + public StoreResult.Outcome outcome() { + return StoreOutcome.OK; + } + + /** + * A success carries no detail. + * + * @return an empty detail + */ + @Override + public Optional detail() { + return Optional.empty(); + } + } + + /** + * A save that conflicts on at least one path (GDR-STORE-4, GDR-STORE-5): nothing was pushed, and the draft + * reference and the target branch are unchanged. + *

+ * The paths travel as data only: the detail names their number and never a path, so a caller reads the paths from + * {@link #paths()} and never parses a detail. + * + * @param paths the conflicting paths, repository-relative, sorted and distinct; never empty + * @since 0.1 + */ + record Conflicting(List paths) implements SaveResult { + + /** + * Sorts the paths, removes duplicates and copies them, so a later change of the caller's list changes nothing. + * + * @param paths the conflicting paths, repository-relative + * @throws NullPointerException if {@code paths} or one of its elements is {@code null} + * @throws IllegalArgumentException if {@code paths} is empty or a path is empty or absolute + */ + public Conflicting { + Objects.requireNonNull(paths, "paths"); + var sorted = new TreeSet(); + for (String path : paths) { + Objects.requireNonNull(path, "paths"); + if (path.isEmpty() || path.startsWith("/")) { + throw new IllegalArgumentException("paths: not a repository-relative path"); + } + sorted.add(path); + } + if (sorted.isEmpty()) { + throw new IllegalArgumentException("paths: a conflict names at least one path"); + } + paths = List.copyOf(sorted); + } + + /** + * The outcome of a conflicting save. + * + * @return {@link StoreResult.PassedOn} of {@link GitOutcome#CONFLICT} + */ + @Override + public StoreResult.Outcome outcome() { + return new StoreResult.PassedOn(GitOutcome.CONFLICT); + } + + /** + * The detail of the conflict, naming the number of conflicting paths only. + * + * @return {@code "N conflicting paths"} + */ + @Override + public Optional detail() { + return Optional.of(paths.size() + " conflicting paths"); + } + } + + /** + * Every other save that did not succeed: a malformed input, a missing draft, a rejection after the last attempt, + * an unreachable remote, invalid credentials, a failure. + * + * @param outcome the outcome, never {@link StoreOutcome#OK} + * @param reason the detail of the non-success, not blank and free of anything GDR-NFR-3 forbids + * @since 0.1 + */ + record NotSaved(StoreResult.Outcome outcome, String reason) implements SaveResult { + + /** + * Checks the outcome and the reason. + * + * @param outcome the outcome + * @param reason the detail + * @throws NullPointerException if an argument is {@code null} + * @throws IllegalArgumentException if {@code outcome} is {@link StoreOutcome#OK} or {@code reason} is blank + */ + public NotSaved { + Objects.requireNonNull(outcome, "outcome"); + Objects.requireNonNull(reason, "reason"); + if (outcome == StoreOutcome.OK) { + throw new IllegalArgumentException("a save that did not succeed is no OK"); + } + if (reason.isBlank()) { + throw new IllegalArgumentException("a save that did not succeed needs a detail"); + } + } + + /** + * Creates the non-success of a store result that did not succeed, keeping its outcome and its detail. + * + * @param result the non-success result + * @return the non-success + * @throws IllegalArgumentException if {@code result} is a success + */ + static NotSaved of(StoreResult result) { + if (result.isOk()) { + throw new IllegalArgumentException("a success is no NotSaved"); + } + return new NotSaved(result.outcome(), Objects.requireNonNull(result.detail())); + } + + /** + * The detail of the non-success. + * + * @return the reason + */ + @Override + public Optional detail() { + return Optional.of(reason); + } + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/StoreConfiguration.java b/git-store/src/main/java/de/cuioss/git/document/store/StoreConfiguration.java index 83387a9..57553d0 100644 --- a/git-store/src/main/java/de/cuioss/git/document/store/StoreConfiguration.java +++ b/git-store/src/main/java/de/cuioss/git/document/store/StoreConfiguration.java @@ -43,15 +43,17 @@ * @param signing the section {@code signing} * @param anchor the section {@code anchor} * @param directory the section {@code directory} + * @param commit the section {@code commit} * @since 0.1 */ public record StoreConfiguration(RepositoryRegistry repositories, Save save, Drafts drafts, Fetch fetch, -Signing signing, Anchor anchor, Directory directory) { +Signing signing, Anchor anchor, Directory directory, Commit commit) { private static final CuiLogger LOGGER = new CuiLogger(StoreConfiguration.class); private static final String REPOSITORIES_KEY = "repositories"; - private static final String MAX_ATTEMPTS = "save.max-attempts"; + private static final String SAVE_MAX_ATTEMPTS = "save.max-attempts"; + private static final String COMMIT_EMAIL = "commit.email"; private static final String DRAFTS_EXPIRY = "drafts.expiry"; private static final String FETCH_INTERVAL = "fetch.interval"; static final String SIGNING_KEY_SOURCE = "signing.key-source"; @@ -77,6 +79,7 @@ public record StoreConfiguration(RepositoryRegistry repositories, Save save, Dra * @param signing the section {@code signing} * @param anchor the section {@code anchor} * @param directory the section {@code directory} + * @param commit the section {@code commit} * @throws NullPointerException if a section is {@code null} */ public StoreConfiguration { @@ -87,14 +90,17 @@ public record StoreConfiguration(RepositoryRegistry repositories, Save save, Dra Objects.requireNonNull(signing, "signing"); Objects.requireNonNull(anchor, "anchor"); Objects.requireNonNull(directory, "directory"); + Objects.requireNonNull(commit, "commit"); } /** * Reads and checks the configuration of the store at start. *

- * {@code save.max-attempts} (an integer of at least 1), {@code drafts.expiry} (a positive ISO 8601 period) and - * {@code fetch.interval} (a positive ISO 8601 duration) are required; every other key has the default its section - * documents; only an omitted key takes its default, a key supplied with a {@code null} value is malformed. A + * {@code save.max-attempts} (an integer from 1 to {@value Save#MAX_ATTEMPTS}), {@code drafts.expiry} (a positive + * ISO 8601 period) and {@code fetch.interval} (a positive ISO 8601 duration) are required; every other key has the + * default its section documents, among them {@code commit.email} (a boolean, default + * {@value Commit#DEFAULT_EMAIL}); only an omitted key takes its default, a key supplied with a {@code null} value + * is malformed. A * missing required key, a malformed value (among them an {@code anchor.provider} outside {@code [a-z0-9-]+}, the * alphabet of a repository alias), a refused repository of the registry and an anchor key * equal to the commit-signing key are refused: {@link StoreOutcome#MALFORMED}, or the passed-on @@ -146,6 +152,10 @@ private static StoreResult read(ConfigurationTree tree, Remo if (!directory.isOk()) { return retype(directory); } + var commit = readCommit(tree); + if (!commit.isOk()) { + return retype(commit); + } var registry = readRegistry(tree, policy); if (!registry.isOk()) { return retype(registry); @@ -158,7 +168,7 @@ private static StoreResult read(ConfigurationTree tree, Remo return StoreResult.ok(new StoreConfiguration(Objects.requireNonNull(registry.value()), Objects.requireNonNull(save.value()), Objects.requireNonNull(drafts.value()), Objects.requireNonNull(fetch.value()), signingSection, anchorSection, - Objects.requireNonNull(directory.value()))); + Objects.requireNonNull(directory.value()), Objects.requireNonNull(commit.value()))); } private static boolean anchorKeyIsSigningKey(Signing signing, Anchor.Signed signed) { @@ -168,15 +178,26 @@ private static boolean anchorKeyIsSigningKey(Signing signing, Anchor.Signed sign } private static StoreResult readSave(ConfigurationTree tree) { - var maxAttempts = tree.integer(MAX_ATTEMPTS); + var maxAttempts = tree.integer(SAVE_MAX_ATTEMPTS); if (!maxAttempts.isOk()) { return retype(maxAttempts); } Integer attempts = maxAttempts.value(); if (attempts == null) { - return tree.missing(MAX_ATTEMPTS); + return tree.missing(SAVE_MAX_ATTEMPTS); } - return attempts < 1 ? tree.malformed(MAX_ATTEMPTS) : StoreResult.ok(new Save(attempts)); + return attempts < 1 || attempts > Save.MAX_ATTEMPTS ? tree.malformed(SAVE_MAX_ATTEMPTS) + : StoreResult.ok(new Save(attempts)); + } + + // S4449 does not see that requireNonNullElse never returns null; the argument is never null + @SuppressWarnings("java:S4449") + private static StoreResult readCommit(ConfigurationTree tree) { + var email = tree.bool(COMMIT_EMAIL); + if (!email.isOk()) { + return retype(email); + } + return StoreResult.ok(new Commit(Objects.requireNonNullElse(email.value(), Commit.DEFAULT_EMAIL))); } private static StoreResult readDrafts(ConfigurationTree tree) { @@ -336,21 +357,32 @@ private static StoreResult required(ConfigurationTree tree, String key) /** * The section {@code save}. * - * @param maxAttempts the attempts of a save or of a push of notes ({@code save.max-attempts}), at least 1 + * @param maxAttempts the attempts of a save or of a push of notes ({@code save.max-attempts}), from 1 to + * {@value #MAX_ATTEMPTS} * @since 0.1 */ public record Save(int maxAttempts) { + /** + * The largest number of attempts, {@value}: the upper bound of {@code save.max-attempts}, well above the 10 + * attempts GDR-NFR-2 needs, so that a misconfigured value cannot keep a save retrying without end. + */ + public static final int MAX_ATTEMPTS = 100; + /** * Checks the number of attempts. * * @param maxAttempts the attempts - * @throws IllegalArgumentException if {@code maxAttempts} is less than 1 + * @throws IllegalArgumentException if {@code maxAttempts} is less than 1 or greater than + * {@value #MAX_ATTEMPTS} */ public Save { if (maxAttempts < 1) { throw new IllegalArgumentException("at least one attempt is required"); } + if (maxAttempts > MAX_ATTEMPTS) { + throw new IllegalArgumentException("at most " + MAX_ATTEMPTS + " attempts are allowed"); + } } } @@ -540,4 +572,18 @@ public record Directory(String personsProvider) { Objects.requireNonNull(personsProvider, "personsProvider"); } } + + /** + * The section {@code commit}: the identity fields of the commits {@code git-store} writes. + * + * @param email whether author and committer of every commit {@code git-store} writes carry an e-mail address + * ({@code commit.email}), default {@value #DEFAULT_EMAIL}; {@code false} keeps both e-mail fields + * empty, so that no address reaches the history (GDR-NFR-3) + * @since 0.1 + */ + public record Commit(boolean email) { + + /** The default of {@code commit.email}: no e-mail address in a commit. */ + public static final boolean DEFAULT_EMAIL = false; + } } diff --git a/git-store/src/main/java/de/cuioss/git/document/store/StoreReferences.java b/git-store/src/main/java/de/cuioss/git/document/store/StoreReferences.java new file mode 100644 index 0000000..8493e8c --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/StoreReferences.java @@ -0,0 +1,177 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.util.HashSet; +import java.util.List; +import java.util.Objects; +import java.util.Set; +import java.util.regex.Pattern; + +/** + * The reference layout of a repository of the store (GDR-STORE-2): the released state, the working and collection + * branches, the drafts, the applied drafts and the push reference specifications. + *

+ * Every reference name is built from segments that contain no character git-check-ref-format(1) forbids, no dot and no + * slash: the collection name ({@code [a-z0-9-]+}, {@link DocumentCollection}), the id ({@link DocumentKey}) and the + * pseudonym ({@code [A-Za-z0-9-]+}). A collection is either per-document or numbered, never both, because JGit refuses a + * reference nested below an existing one and one that would contain an existing one ({@code RefDatabase + * .isNameConflicting}, JGit 7.8.0), so {@code refs/heads/} and {@code refs/heads//} cannot + * exist side by side; {@link #collections(List)} refuses a name declared twice. + * + * @since 0.1 + */ +public final class StoreReferences { + + /** The branch of the released state, {@code refs/heads/main}. */ + public static final String MAIN = "refs/heads/main"; + + /** The prefix of every draft reference, {@code refs/drafts/}; never pushed (ADR-0003). */ + public static final String DRAFTS = "refs/drafts/"; + + /** + * The prefix of every applied-draft reference, {@code refs/applied/}. The reference + * {@code refs/applied///} points at the draft commit a save last applied while a newer + * draft was kept, so the next save applies only what the draft changed since then. It lives in the clone only: + * never pushed, never fetched. + */ + public static final String APPLIED = "refs/applied/"; + + /** + * The push reference specifications of the bare clone: the branches, the tags and the notes, each to the same name + * on the remote. They never carry {@code refs/drafts/*}, so no draft reaches the remote (ADR-0003). + */ + public static final List PUSH_REF_SPECS = + List.of("refs/heads/*:refs/heads/*", "refs/tags/*:refs/tags/*", "refs/notes/*:refs/notes/*"); + + /** The prefix of every branch. */ + private static final String HEADS = "refs/heads/"; + + /** The form of a pseudonym. */ + private static final Pattern PSEUDONYM = Pattern.compile("[A-Za-z0-9-]+"); + + /** The detail of a refused pseudonym; it never carries the pseudonym. */ + static final String INVALID_PSEUDONYM = "pseudonym: invalid"; + + private StoreReferences() { + } + + /** + * Returns the branch a save of the document or entry targets. + * + * @param key the key of the document or entry + * @return {@code refs/heads//} for a per-document collection, {@code refs/heads/} for a + * numbered collection, and {@value #MAIN} for content without release + * @throws NullPointerException if {@code key} is {@code null} + */ + public static String target(DocumentKey key) { + Objects.requireNonNull(key, "key"); + var collection = key.collection(); + return switch (collection.kind()) { + case PER_DOCUMENT -> HEADS + collection.name() + "/" + key.id(); + case NUMBERED -> HEADS + collection.name(); + case MAIN -> MAIN; + }; + } + + /** + * Returns the draft reference of a user for a document or entry. + * + * @param pseudonym the pseudonym of the user + * @param key the key of the document or entry + * @return {@link StoreOutcome#OK} with {@code refs/drafts///}, or + * {@link StoreOutcome#MALFORMED} with the detail {@code pseudonym: invalid} for a pseudonym outside + * {@code [A-Za-z0-9-]+}; the detail never carries the pseudonym + * @throws NullPointerException if {@code pseudonym} or {@code key} is {@code null} + */ + public static StoreResult draft(String pseudonym, DocumentKey key) { + Objects.requireNonNull(pseudonym, "pseudonym"); + Objects.requireNonNull(key, "key"); + if (!isPseudonym(pseudonym)) { + return StoreResult.of(StoreOutcome.MALFORMED, INVALID_PSEUDONYM); + } + return StoreResult.ok(drafts(pseudonym) + key.collection().name() + "/" + key.id()); + } + + /** + * Returns the prefix of every draft reference of a user, {@code refs/drafts//}. + * + * @param pseudonym the pseudonym, already checked by {@link #isPseudonym(String)} + * @return the prefix, ending in {@code /} + */ + static String drafts(String pseudonym) { + return DRAFTS + pseudonym + "/"; + } + + /** + * Returns the applied-draft reference that belongs to a draft reference. + * + * @param draftRef a draft reference, below {@value #DRAFTS} + * @return the reference of the same name below {@value #APPLIED} + * @throws IllegalArgumentException if {@code draftRef} is no draft reference + */ + static String applied(String draftRef) { + if (!draftRef.startsWith(DRAFTS)) { + throw new IllegalArgumentException("not a draft reference"); + } + return APPLIED + draftRef.substring(DRAFTS.length()); + } + + /** + * Returns the draft reference an applied-draft reference belongs to. + * + * @param appliedRef an applied-draft reference, below {@value #APPLIED} + * @return the reference of the same name below {@value #DRAFTS} + * @throws IllegalArgumentException if {@code appliedRef} is no applied-draft reference + */ + static String draftOf(String appliedRef) { + if (!appliedRef.startsWith(APPLIED)) { + throw new IllegalArgumentException("not an applied-draft reference"); + } + return DRAFTS + appliedRef.substring(APPLIED.length()); + } + + /** + * Tells whether a text is a valid pseudonym segment, {@code [A-Za-z0-9-]+}. + * + * @param pseudonym the text + * @return whether it is a valid pseudonym + */ + static boolean isPseudonym(String pseudonym) { + return PSEUDONYM.matcher(pseudonym).matches(); + } + + /** + * Checks the declared collections of a repository. + * + * @param collections the declared collections, in declaration order + * @return {@link StoreOutcome#OK} with an unmodifiable copy of the collections, or {@link StoreOutcome#MALFORMED} + * with the detail {@code collection : declared twice} for the first name declared more than once, + * whatever the kinds, since a name cannot be both per-document and numbered + * @throws NullPointerException if {@code collections} or one of its elements is {@code null} + */ + public static StoreResult> collections(List collections) { + Objects.requireNonNull(collections, "collections"); + Set names = new HashSet<>(); + for (DocumentCollection collection : collections) { + Objects.requireNonNull(collection, "collection"); + if (!names.add(collection.name())) { + return StoreResult.of(StoreOutcome.MALFORMED, "collection " + collection.name() + ": declared twice"); + } + } + return StoreResult.ok(List.copyOf(collections)); + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/StoreWriter.java b/git-store/src/main/java/de/cuioss/git/document/store/StoreWriter.java new file mode 100644 index 0000000..f71cff4 --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/StoreWriter.java @@ -0,0 +1,265 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.io.IOException; +import java.net.URISyntaxException; +import java.nio.file.Path; +import java.time.Clock; +import java.time.Instant; +import java.util.List; +import java.util.Map; +import java.util.Objects; + + +import de.cuioss.git.document.core.GitOutcome; +import de.cuioss.git.document.core.InCoreWriter; +import de.cuioss.git.document.core.ReferenceUpdater; +import de.cuioss.git.document.core.RefusingFS; +import de.cuioss.git.document.core.RepositoryOperations; +import de.cuioss.git.document.core.TreeEdit; +import org.eclipse.jgit.lib.Constants; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.storage.file.FileRepositoryBuilder; +import org.eclipse.jgit.transport.RemoteConfig; +import org.eclipse.jgit.transport.URIish; + +/** + * The write path of one repository of the store: drafts per user and document (GDR-STORE-2) and the explicit save + * with lease, conflicts and change detection (GDR-STORE-3, GDR-STORE-4, GDR-STORE-5), and the expiry of drafts + * (GDR-STORE-7). + *

+ * {@link #open} clones the remote of the repository into a bare clone if none exists yet, with the push reference + * specifications {@link StoreReferences#PUSH_REF_SPECS} (never {@code refs/drafts/*}, ADR-0003) and the credentials of + * the technical user, and opens the clone on a {@link RefusingFS}, so no operation on the clone starts a process + * (GDR-CORE-1). That covers the clone only: JGit 7.8.0 reads the user and system configuration through the process-wide + * {@code SystemReader}, whose default runs {@code git} once per JVM to locate the system configuration, so an embedding + * application that needs the guarantee for its whole JVM installs + * {@link RefusingFS#systemReader(org.eclipse.jgit.util.SystemReader)} before any JGit use. An + * existing clone is reused only when every fetch and push destination of its {@code origin} is the registered remote; + * a clone of another repository at the path is refused. Every commit identity comes from the section {@code commit} + * of the configuration and the technical user. + *

+ * Every expected non-success is a {@link StoreResult} or a {@link SaveResult}; a Git outcome is passed on. Draft writes + * and saves take the shared side of the clone's maintenance lock, every save the lock of its draft, so the saves of one + * draft run one after the other, and every save attempt the transport lock; the expiry, its deletes and its garbage + * collection, takes the exclusive side. The locks belong to the clone, not to the + * writer: every writer of the JVM that opens the same git directory (by its real path) shares them, so the garbage + * collection of one writer never runs while another writer of the clone holds objects not yet referenced. No + * operation takes a credential: every fetch and push uses those of the technical user (GDR-STORE-8). The writer holds + * the opened clone and releases it on {@link #close()}. + * + * @since 0.1 + */ +public final class StoreWriter implements AutoCloseable { + + /** The detail of a clone whose remote is not the registered one; it names no URL, credential or path. */ + static final String FOREIGN_CLONE = "clone: remote does not match the registry"; + + private final String alias; + private final Repository repository; + private final RepositoryGuard guard; + private final DraftReferences drafts; + private final ExplicitSave saves; + private final DraftExpiry expiry; + + private StoreWriter(String alias, Repository repository, RepositoryGuard guard, DraftReferences drafts, + ExplicitSave saves, DraftExpiry expiry) { + this.alias = alias; + this.repository = repository; + this.guard = guard; + this.drafts = drafts; + this.saves = saves; + this.expiry = expiry; + } + + /** + * Opens the write path of a repository, cloning its remote first if the bare clone does not exist yet. + * + * @param storeRepository the repository of the registry + * @param gitDir the git directory of the bare clone + * @param user the technical user: committer and holder of the transport credentials + * @param configuration the configuration of the store + * @param operations the repository operations of {@code git-core} + * @param clock the clock of every commit time + * @return {@link StoreOutcome#OK} with the writer, or the passed-on Git outcome of the clone or of opening it, for + * example {@link GitOutcome#UNREACHABLE}, {@link GitOutcome#CREDENTIALS_INVALID}, + * {@link GitOutcome#REFUSED_BY_POLICY}, {@link GitOutcome#FAILED}, or {@link GitOutcome#NOT_A_REPOSITORY} + * with the detail {@code : clone: remote does not match the registry} for an existing clone whose + * remote is not the registered one; no detail names a URL, a credential or a path + * @throws NullPointerException if an argument is {@code null} + */ + public static StoreResult open(StoreRepository storeRepository, Path gitDir, TechnicalUser user, + StoreConfiguration configuration, RepositoryOperations operations, Clock clock) { + Objects.requireNonNull(storeRepository, "storeRepository"); + Objects.requireNonNull(gitDir, "gitDir"); + Objects.requireNonNull(user, "user"); + Objects.requireNonNull(configuration, "configuration"); + Objects.requireNonNull(operations, "operations"); + Objects.requireNonNull(clock, "clock"); + var cloned = operations.cloneIfMissing(RepositoryOperations.CloneInput.builder() + .gitDir(gitDir) + .url(storeRepository.remote()) + .pushRefSpecs(StoreReferences.PUSH_REF_SPECS) + .credentials(user.credentials()) + .build()); + if (!cloned.isOk()) { + return StoreResult.passOn(cloned); + } + Path clone = Objects.requireNonNull(cloned.value()).gitDir(); + Repository repository; + try { + repository = new FileRepositoryBuilder().setGitDir(clone.toFile()).setFS(new RefusingFS()) + .setMustExist(true).build(); + } catch (IOException e) { + return StoreResult.passOn(DraftReferences.failure(storeRepository.alias(), e)); + } + if (!transportsTo(repository, storeRepository.remote())) { + repository.close(); + return new StoreResult<>(new StoreResult.PassedOn(GitOutcome.NOT_A_REPOSITORY), null, + storeRepository.alias() + ": " + FOREIGN_CLONE); + } + var identities = new CommitIdentities(configuration.commit(), user); + var updater = new ReferenceUpdater(new InCoreWriter()); + int attempts = configuration.save().maxAttempts(); + var drafts = new DraftReferences(repository, updater, identities, clock, attempts); + var guard = RepositoryGuard.of(clone); + var saves = new ExplicitSave(new ExplicitSave.Clone(storeRepository.alias(), repository, clone, guard), + operations, user, identities, clock, attempts); + var expiry = new DraftExpiry(storeRepository.alias(), repository, guard, updater, + configuration.drafts().expiry(), attempts); + return StoreResult.ok(new StoreWriter(storeRepository.alias(), repository, guard, drafts, saves, expiry)); + } + + /** + * Tells whether every fetch and push of the clone's {@code origin} goes to the registered remote: exactly one + * fetch URL, and every push URL (the fetch URL when none is configured), equal to it. JGit 7.8.0 + * {@code RemoteConfig} applies {@code url..insteadOf} to the fetch URLs and {@code pushInsteadOf} to the + * push URLs it derives, so the comparison sees the destinations a transport would contact. + */ + private static boolean transportsTo(Repository repository, String remote) { + try { + var expected = new URIish(remote); + var origin = new RemoteConfig(repository.getConfig(), Constants.DEFAULT_REMOTE_NAME); + List fetch = origin.getURIs(); + List push = origin.getPushURIs().isEmpty() ? fetch : origin.getPushURIs(); + return fetch.size() == 1 && expected.equals(fetch.getFirst()) && push.stream().allMatch(expected::equals); + } catch (URISyntaxException e) { + return false; + } + } + + /** + * Returns the alias of the repository this writer writes to. + * + * @return the alias + */ + public String alias() { + return alias; + } + + /** + * Writes an intermediate save: the next commit of {@code refs/drafts///}, created at + * the tip of the local target branch (else {@code main}) when the user has no draft of the document yet + * (GDR-STORE-2). The draft stays in the clone and is never pushed. + * + * @param author the author, whose pseudonym names the draft reference + * @param key the document or entry + * @param edits the edits, applied onto the draft's current tree + * @param message the commit message + * @return {@link StoreOutcome#OK} with the full id of the draft commit, {@link StoreOutcome#MALFORMED} for a + * pseudonym outside {@code [A-Za-z0-9-]+}, or the passed-on Git outcome that prevented the write + * @throws NullPointerException if an argument is {@code null} + */ + public StoreResult draft(Author author, DocumentKey key, List edits, String message) { + Objects.requireNonNull(author, "author"); + Objects.requireNonNull(key, "key"); + Objects.requireNonNull(edits, "edits"); + Objects.requireNonNull(message, "message"); + return guard.shared(() -> drafts.write(author, key, edits, message)); + } + + /** + * Lists the drafts of a user, sorted by reference name (GDR-STORE-2). + * + * @param pseudonym the pseudonym of the user + * @return {@link StoreOutcome#OK} with the drafts, possibly none; {@link StoreOutcome#MALFORMED} for a pseudonym + * outside {@code [A-Za-z0-9-]+}; or the passed-on {@link de.cuioss.git.document.core.GitOutcome#FAILED} + * @throws NullPointerException if {@code pseudonym} is {@code null} + */ + public StoreResult> drafts(String pseudonym) { + Objects.requireNonNull(pseudonym, "pseudonym"); + return guard.shared(() -> drafts.list(pseudonym)); + } + + /** + * Saves the draft of the author for the document onto its target branch (GDR-STORE-3): the draft's changes as one + * commit on the remote tip, pushed with lease by the technical user, re-applied onto a new tip up to + * {@code save.max-attempts}; then the local target branch moves and the draft is deleted by compare-and-delete. + * Every changed path with an expected blob id that differs on the fetched tip is a conflict (GDR-STORE-4, + * GDR-STORE-5); a changed path without one overwrites a concurrent change. + * + * @param author the author, whose pseudonym names the draft reference and the commit's author + * @param key the document or entry + * @param expectedBlobIds the full blob id the caller loaded per repository-relative path; the zero id expects the + * path to be absent; empty for no check + * @param message the commit message + * @return {@link SaveResult.Saved} with the commit the target branch points at, {@link SaveResult.Conflicting} + * with every conflicting path, or {@link SaveResult.NotSaved} with the outcome that prevented the save, for + * example {@link StoreOutcome#MALFORMED} for an invalid pseudonym, + * {@link de.cuioss.git.document.core.GitOutcome#NOT_FOUND} for a missing draft, + * {@link de.cuioss.git.document.core.GitOutcome#REJECTED} after the last attempt, + * {@link de.cuioss.git.document.core.GitOutcome#UNREACHABLE} or + * {@link de.cuioss.git.document.core.GitOutcome#CREDENTIALS_INVALID} + * @throws NullPointerException if an argument, a path or an expected blob id is {@code null} + */ + public SaveResult save(Author author, DocumentKey key, Map expectedBlobIds, String message) { + return saves.save(author, key, expectedBlobIds, message); + } + + /** + * Deletes every draft unchanged for {@code drafts.expiry} and removes the objects reachable only from deleted + * references (GDR-STORE-7): a draft whose tip has a committer time at or before {@code now} minus + * {@code drafts.expiry} (UTC) is deleted by compare-and-delete expecting that tip, so a draft changed meanwhile is + * kept; then a garbage collection runs. Both run under the exclusive side of the clone's maintenance lock, so no + * draft write or save of this writer runs meanwhile. + * + * @param now the current time + * @return {@link StoreOutcome#OK} with the full names of the deleted draft references, sorted, possibly none; or + * the passed-on {@link de.cuioss.git.document.core.GitOutcome#FAILED} of a failed read, delete or garbage + * collection + * @throws NullPointerException if {@code now} is {@code null} + */ + public StoreResult> expireDrafts(Instant now) { + return expiry.expire(now); + } + + /** Releases the opened clone. */ + @Override + public void close() { + repository.close(); + } + + /** + * Renders the alias only. + * + * @return the rendering + */ + @Override + public String toString() { + return "StoreWriter[alias=" + alias + "]"; + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/TechnicalUser.java b/git-store/src/main/java/de/cuioss/git/document/store/TechnicalUser.java new file mode 100644 index 0000000..d3ae38c --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/TechnicalUser.java @@ -0,0 +1,67 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.util.Objects; + + +import org.eclipse.jgit.transport.CredentialsProvider; +import org.jspecify.annotations.Nullable; + +/** + * The technical user of a repository: the committer of every commit the store writes, and the holder of the + * credentials of its transport. + *

+ * {@link #toString()} renders the name only, never the address or the credentials (GDR-NFR-3). + * + * @param name the committer name, not blank + * @param email the committer address; {@code null} when none is configured. It is written only when + * {@code commit.email} is {@code true} (see {@link StoreConfiguration.Commit}) + * @param credentials the credentials of every fetch and push of the repository + * @since 0.1 + */ +public record TechnicalUser(String name, @Nullable String email, CredentialsProvider credentials) { + + /** + * Checks the components. + * + * @param name the committer name + * @param email the committer address, or {@code null} + * @param credentials the credentials + * @throws NullPointerException if {@code name} or {@code credentials} is {@code null} + * @throws IllegalArgumentException if the name is blank, or the address is blank or contains {@code <}, {@code >}, + * a carriage return, a line feed or a NUL character; the message names the + * component, never its value + */ + public TechnicalUser { + Objects.requireNonNull(name, "name"); + Objects.requireNonNull(credentials, "credentials"); + if (name.isBlank()) { + throw new IllegalArgumentException("name: blank"); + } + CommitIdentities.checkAddress(email); + } + + /** + * Renders the name only, never the address or the credentials (GDR-NFR-3). + * + * @return the rendering + */ + @Override + public String toString() { + return "TechnicalUser[name=" + name + "]"; + } +} diff --git a/git-store/src/main/java/de/cuioss/git/document/store/Ulid.java b/git-store/src/main/java/de/cuioss/git/document/store/Ulid.java new file mode 100644 index 0000000..a4fbd62 --- /dev/null +++ b/git-store/src/main/java/de/cuioss/git/document/store/Ulid.java @@ -0,0 +1,148 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import java.security.SecureRandom; +import java.time.Instant; +import java.util.Objects; +import java.util.Random; + + +import org.jspecify.annotations.Nullable; + +/** + * The ULID, the id of a document of a per-document collection and of the draft of a new numbered entry (GDR-STORE-2). + *

+ * A ULID is 26 characters of Crockford base32 in upper case: the first 10 encode a 48-bit timestamp in milliseconds + * since the Unix epoch, the last 16 encode 80 random bits. Two ULIDs created in different milliseconds therefore sort + * lexicographically by their creation time (ULID specification, sections + * Specification, Encoding and Sorting). Ids created within the same millisecond are not + * ordered among themselves; this class does not implement the optional monotonic mode of the specification. + *

+ * The random part comes from a {@link SecureRandom}, so an id cannot be guessed from another one. The alphabet contains + * neither a slash nor any character that git-check-ref-format(1) forbids, so a ULID is a valid segment of a reference + * name by construction. + * + * @since 0.1 + */ +public final class Ulid { + + /** The length of a ULID: 26 characters. */ + public static final int LENGTH = 26; + + /** The Crockford base32 alphabet: digits and upper-case letters without I, L, O and U. */ + private static final String ALPHABET = "0123456789ABCDEFGHJKMNPQRSTVWXYZ"; + + /** The number of characters encoding the timestamp. */ + private static final int TIME_LENGTH = 10; + + /** The number of characters encoding each 40-bit half of the random part. */ + private static final int HALF_LENGTH = 8; + + /** The number of bytes of each 40-bit half of the random part. */ + private static final int HALF_BYTES = 5; + + /** The number of bits one character encodes. */ + private static final int BITS_PER_CHAR = 5; + + /** The mask of the bits one character encodes. */ + private static final int CHAR_MASK = 0x1F; + + /** The largest timestamp a ULID can encode, {@code 2^48 - 1} milliseconds. */ + private static final long MAX_TIME = (1L << 48) - 1; + + /** + * The largest first character: the 10 time characters carry 50 bits, of which only 48 are used, so the first + * character is at most {@code 7}. + */ + private static final char MAX_FIRST = '7'; + + /** The source of the random part of {@link #next()}. */ + private static final SecureRandom RANDOM = new SecureRandom(); + + private Ulid() { + } + + /** + * Creates a new ULID from the current time and 80 random bits of a {@link SecureRandom}. + * + * @return a ULID of {@value #LENGTH} characters + */ + public static String next() { + return next(Instant.now(), RANDOM); + } + + /** + * Creates a ULID from a given time and a given source of randomness, so that tests are deterministic. + * + * @param time the creation time, from the Unix epoch to {@code 2^48 - 1} milliseconds after it + * @param random the source of the 80 random bits + * @return a ULID of {@value #LENGTH} characters + * @throws NullPointerException if {@code time} or {@code random} is {@code null} + * @throws IllegalArgumentException if {@code time} lies outside the range a ULID can encode + */ + static String next(Instant time, Random random) { + Objects.requireNonNull(time, "time"); + Objects.requireNonNull(random, "random"); + if (time.isBefore(Instant.EPOCH) || time.isAfter(Instant.ofEpochMilli(MAX_TIME))) { + throw new IllegalArgumentException("the time lies outside the range of a ULID"); + } + char[] chars = new char[LENGTH]; + encode(time.toEpochMilli(), chars, 0, TIME_LENGTH); + byte[] bytes = new byte[2 * HALF_BYTES]; + random.nextBytes(bytes); + encode(toLong(bytes, 0), chars, TIME_LENGTH, HALF_LENGTH); + encode(toLong(bytes, HALF_BYTES), chars, TIME_LENGTH + HALF_LENGTH, HALF_LENGTH); + return new String(chars); + } + + /** + * Tells whether a text is a ULID in its canonical form: 26 characters of the upper-case Crockford alphabet whose + * first character is at most {@code 7}. + * + * @param value the text, possibly {@code null} + * @return whether the text is a ULID; {@code false} for {@code null} + */ + public static boolean isValid(@Nullable String value) { + if (value == null || value.length() != LENGTH || value.charAt(0) > MAX_FIRST) { + return false; + } + for (int index = 0; index < LENGTH; index++) { + if (ALPHABET.indexOf(value.charAt(index)) < 0) { + return false; + } + } + return true; + } + + /** Writes the lowest {@code count * 5} bits of a value as {@code count} characters, most significant first. */ + private static void encode(long value, char[] target, int offset, int count) { + long rest = value; + for (int index = offset + count - 1; index >= offset; index--) { + target[index] = ALPHABET.charAt((int) (rest & CHAR_MASK)); + rest >>>= BITS_PER_CHAR; + } + } + + /** Reads five bytes, big-endian, as the lowest 40 bits of a long. */ + private static long toLong(byte[] bytes, int offset) { + long value = 0; + for (int index = offset; index < offset + HALF_BYTES; index++) { + value = (value << Byte.SIZE) | (bytes[index] & 0xFFL); + } + return value; + } +} diff --git a/git-store/src/test/java/de/cuioss/git/document/store/CommitIdentitiesTest.java b/git-store/src/test/java/de/cuioss/git/document/store/CommitIdentitiesTest.java new file mode 100644 index 0000000..8c4ce78 --- /dev/null +++ b/git-store/src/test/java/de/cuioss/git/document/store/CommitIdentitiesTest.java @@ -0,0 +1,234 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import static org.junit.jupiter.api.Assertions.assertAll; +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; + +import java.time.Instant; +import java.time.ZoneOffset; + + +import de.cuioss.git.document.store.StoreConfiguration.Commit; +import de.cuioss.test.generator.Generators; +import de.cuioss.test.generator.junit.EnableGeneratorController; +import org.eclipse.jgit.transport.CredentialsProvider; +import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider; +import org.jspecify.annotations.Nullable; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +/** + * The commit identities of GDR-STORE-2 and GDR-NFR-3: both settings of {@code commit.email} with and without supplied + * addresses, the address checks of {@link TechnicalUser} and {@link Author}, whose messages never carry the value, + * and renderings of both that carry no address. + *

+ * Names, pseudonyms, addresses, credentials and times are generated; the forbidden characters and the details are the + * specified vocabulary and kept as literals. + */ +@EnableGeneratorController +@DisplayName("CommitIdentities (GDR-STORE-2, GDR-NFR-3)") +class CommitIdentitiesTest { + + private static String word() { + return Generators.letterStrings(6, 16).next(); + } + + private static String address() { + return word() + "@" + word() + ".example"; + } + + private static Instant time() { + return Instant.ofEpochSecond(Generators.integers(1_000_000_000, 2_000_000_000).next()); + } + + private static CredentialsProvider credentials(String user, String password) { + return new UsernamePasswordCredentialsProvider(user, password); + } + + private static TechnicalUser user(@Nullable String email) { + return new TechnicalUser(word(), email, credentials(word(), word())); + } + + @Nested + @DisplayName("commit.email") + class Email { + + @Test + @DisplayName("GDR-STORE-2, GDR-NFR-3: with commit.email false a supplied address is not written") + void falseIgnoresAddresses() { + var author = new Author(word(), address()); + var user = user(address()); + var when = time(); + var identities = new CommitIdentities(new Commit(false), user); + + var authorIdent = identities.author(author, when); + var committerIdent = identities.committer(when); + + assertAll( + () -> assertEquals(author.pseudonym(), authorIdent.getName(), "the author is the pseudonym"), + () -> assertEquals("", authorIdent.getEmailAddress(), "no author address"), + () -> assertEquals(user.name(), committerIdent.getName(), "the committer is the technical user"), + () -> assertEquals("", committerIdent.getEmailAddress(), "no committer address"), + () -> assertEquals(when, authorIdent.getWhenAsInstant(), "the author time"), + () -> assertEquals(when, committerIdent.getWhenAsInstant(), "the commit time"), + () -> assertEquals(ZoneOffset.UTC, authorIdent.getZoneId(), "the author time is in UTC"), + () -> assertEquals(ZoneOffset.UTC, committerIdent.getZoneId(), "the commit time is in UTC")); + } + + @Test + @DisplayName("GDR-STORE-2: with commit.email false and no addresses both e-mail fields are empty") + void falseWithoutAddresses() { + var identities = new CommitIdentities(new Commit(false), user(null)); + var when = time(); + + assertAll( + () -> assertEquals("", identities.author(Author.of(word()), when).getEmailAddress(), "author"), + () -> assertEquals("", identities.committer(when).getEmailAddress(), "committer")); + } + + @Test + @DisplayName("GDR-STORE-2: with commit.email true the supplied addresses are written") + void trueWritesAddresses() { + var author = new Author(word(), address()); + var user = user(address()); + var when = time(); + var identities = new CommitIdentities(new Commit(true), user); + + assertAll( + () -> assertEquals(author.email(), identities.author(author, when).getEmailAddress(), "author"), + () -> assertEquals(user.email(), identities.committer(when).getEmailAddress(), "committer")); + } + + @Test + @DisplayName("GDR-STORE-2: with commit.email true an absent address stays empty") + void trueWithoutAddresses() { + var identities = new CommitIdentities(new Commit(true), user(null)); + var when = time(); + + assertAll( + () -> assertEquals("", identities.author(Author.of(word()), when).getEmailAddress(), "author"), + () -> assertEquals("", identities.committer(when).getEmailAddress(), "committer")); + } + } + + @Nested + @DisplayName("Address checks") + class AddressChecks { + + @ParameterizedTest(name = "[{0}]") + @ValueSource(strings = {"<", ">", "\r", "\n", "\0"}) + @DisplayName("GDR-STORE-2, GDR-NFR-3: an address that breaks the identity line is refused without its value") + void forbiddenCharacter(String character) { + var invalid = word() + character + address(); + var pseudonym = word(); + var name = word(); + var credentials = credentials(word(), word()); + + var forAuthor = assertThrows(IllegalArgumentException.class, () -> new Author(pseudonym, invalid), + "author"); + var forUser = assertThrows(IllegalArgumentException.class, + () -> new TechnicalUser(name, invalid, credentials), "technical user"); + + assertAll( + () -> assertEquals("email: invalid", forAuthor.getMessage(), "the author message names the component"), + () -> assertEquals("email: invalid", forUser.getMessage(), "the user message names the component"), + () -> assertFalse(forAuthor.getMessage().contains(invalid), "no value in the author message"), + () -> assertFalse(forUser.getMessage().contains(invalid), "no value in the user message")); + } + + @ParameterizedTest(name = "[{0}]") + @ValueSource(strings = {"", " ", "\t"}) + @DisplayName("GDR-STORE-2, GDR-NFR-3: a blank address is refused") + void blankAddress(String blank) { + var pseudonym = word(); + var name = word(); + var credentials = credentials(word(), word()); + + var forAuthor = assertThrows(IllegalArgumentException.class, () -> new Author(pseudonym, blank)); + var forUser = assertThrows(IllegalArgumentException.class, + () -> new TechnicalUser(name, blank, credentials)); + + assertAll( + () -> assertEquals("email: invalid", forAuthor.getMessage(), "author"), + () -> assertEquals("email: invalid", forUser.getMessage(), "technical user")); + } + + @Test + @DisplayName("GDR-STORE-2: a valid address and an absent one are accepted") + void validAddresses() { + assertAll( + () -> assertDoesNotThrow(() -> new Author(word(), address()), "author with address"), + () -> assertDoesNotThrow(() -> Author.of(word()), "author without address"), + () -> assertDoesNotThrow(() -> user(address()), "user with address"), + () -> assertDoesNotThrow(() -> user(null), "user without address")); + } + + @ParameterizedTest(name = "[{0}]") + @ValueSource(strings = {"", " ", "\t"}) + @DisplayName("GDR-STORE-2: a blank name of the technical user is refused") + void blankUserName(String blank) { + var address = address(); + var credentials = credentials(word(), word()); + + var refused = assertThrows(IllegalArgumentException.class, + () -> new TechnicalUser(blank, address, credentials)); + + assertEquals("name: blank", refused.getMessage(), "the message names the component"); + } + } + + @Nested + @DisplayName("Rendering (GDR-NFR-3)") + class Rendering { + + @Test + @DisplayName("GDR-STORE-2, GDR-NFR-3: an author renders its pseudonym and no address") + void authorRendering() { + var author = new Author(word(), address()); + + var rendered = author.toString(); + + assertAll( + () -> assertEquals("Author[pseudonym=" + author.pseudonym() + "]", rendered, "the rendering"), + () -> assertFalse(rendered.contains(String.valueOf(author.email())), "no address")); + } + + @Test + @DisplayName("GDR-STORE-2, GDR-NFR-3: a technical user renders its name and neither address nor credentials") + void userRendering() { + var name = word(); + var email = address(); + var login = "~" + word() + "~"; + var password = "~" + word() + "~"; + var user = new TechnicalUser(name, email, credentials(login, password)); + + var rendered = user.toString(); + + assertAll( + () -> assertEquals("TechnicalUser[name=" + name + "]", rendered, "the rendering"), + () -> assertFalse(rendered.contains(email), "no address"), + () -> assertFalse(rendered.contains(login), "no user of the credentials"), + () -> assertFalse(rendered.contains(password), "no password of the credentials")); + } + } +} diff --git a/git-store/src/test/java/de/cuioss/git/document/store/ConcurrentWritersTest.java b/git-store/src/test/java/de/cuioss/git/document/store/ConcurrentWritersTest.java new file mode 100644 index 0000000..48e2621 --- /dev/null +++ b/git-store/src/test/java/de/cuioss/git/document/store/ConcurrentWritersTest.java @@ -0,0 +1,368 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import static org.junit.jupiter.api.Assertions.assertAll; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.Period; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.TreeMap; +import java.util.concurrent.Callable; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; + + +import de.cuioss.git.document.core.GitOutcome; +import de.cuioss.git.document.core.JGitRepositoryOperations; +import de.cuioss.git.document.core.RemotePolicy; +import de.cuioss.git.document.core.RepositoryOperations; +import de.cuioss.git.document.core.TreeEdit; +import de.cuioss.git.document.store.DocumentCollection.Kind; +import de.cuioss.git.document.testing.LocalRemote; +import de.cuioss.git.document.testing.Remote; +import de.cuioss.test.generator.Generators; +import de.cuioss.test.generator.junit.EnableGeneratorController; +import org.eclipse.jgit.lib.CommitBuilder; +import org.eclipse.jgit.lib.Constants; +import org.eclipse.jgit.lib.FileMode; +import org.eclipse.jgit.lib.ObjectId; +import org.eclipse.jgit.lib.ObjectInserter; +import org.eclipse.jgit.lib.PersonIdent; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.lib.RefUpdate; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.lib.TreeFormatter; +import org.eclipse.jgit.revwalk.RevWalk; +import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider; +import org.eclipse.jgit.treewalk.TreeWalk; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Concurrent writers of GDR-NFR-2 against one {@code git-testing} local remote with {@code save.max-attempts} 10: ten + * writers with their own bare clones save different files of one working branch at the same time and all succeed, two + * writers that change the same file from the same loaded blob end in exactly one save and one conflict naming that + * file, and ten threads sharing one writer all succeed as well. A second save of one draft, started while the first is + * about to push, waits for it and finds the draft settled instead of replaying its changes. + *

+ * Names, ids, pseudonyms, paths and contents are generated; the number of writers, the attempts and the details are + * the specified scenario and kept as literals. + */ +@EnableGeneratorController +@DisplayName("Concurrent writers (GDR-NFR-2)") +class ConcurrentWritersTest { + + /** The number of concurrent writers of the scenario. */ + private static final int WRITERS = 10; + + /** {@code save.max-attempts} of the scenario: one lost lease per other writer and one more. */ + private static final int ATTEMPTS = 10; + + /** The time a scenario may take before it counts as hanging. */ + private static final long TIMEOUT_SECONDS = 120; + + /** The time the teardown waits for the workers after the scenario ended or timed out. */ + private static final long TEARDOWN_SECONDS = 10; + + @TempDir + Path temp; + + private static String word() { + return Generators.strings("abcdefghijklmnopqrstuvwxyz", 6, 16).next(); + } + + private static String pseudonym() { + return Generators.strings("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-", 8, 24).next(); + } + + private static String text() { + return Generators.letterStrings(16, 64).next(); + } + + private static V value(StoreResult result) { + assertTrue(result.isOk(), () -> "accepted: " + result.detail()); + return Objects.requireNonNull(result.value(), "a success carries its value"); + } + + private static DocumentKey key() { + var collection = value(DocumentCollection.of("c" + word(), Kind.PER_DOCUMENT)); + return value(DocumentKey.of(collection, Ulid.next())); + } + + private static StoreConfiguration configuration() { + Map tree = new LinkedHashMap<>(); + tree.put("save", Map.of("max-attempts", ATTEMPTS)); + tree.put("drafts", Map.of("expiry", Period.ofDays(Generators.integers(1, 90).next()).toString())); + tree.put("fetch", Map.of("interval", Duration.ofMinutes(Generators.integers(1, 60).next()).toString())); + return value(StoreConfiguration.from(tree, RemotePolicy.localOnly())); + } + + private static String blob(String content) { + try (ObjectInserter.Formatter formatter = new ObjectInserter.Formatter()) { + return formatter.idFor(Constants.OBJ_BLOB, content.getBytes(StandardCharsets.UTF_8)).name(); + } + } + + private static List put(String path, String content) { + return List.of(new TreeEdit.Put(path, content.getBytes(StandardCharsets.UTF_8))); + } + + /** Writes a commit holding exactly the files onto a reference of the remote, forcing the reference. */ + private static ObjectId commit(Repository repository, String ref, Map files) throws IOException { + try (ObjectInserter inserter = repository.newObjectInserter()) { + var tree = new TreeFormatter(); + for (Map.Entry file : new TreeMap<>(files).entrySet()) { + tree.append(file.getKey(), FileMode.REGULAR_FILE, + inserter.insert(Constants.OBJ_BLOB, file.getValue().getBytes(StandardCharsets.UTF_8))); + } + var commit = new CommitBuilder(); + var ident = new PersonIdent(pseudonym(), ""); + commit.setTreeId(inserter.insert(tree)); + commit.setAuthor(ident); + commit.setCommitter(ident); + commit.setMessage(word()); + var id = inserter.insert(commit); + inserter.flush(); + RefUpdate update = repository.updateRef(ref); + update.setNewObjectId(id); + update.setForceUpdate(true); + var result = update.update(); + assertTrue(result == RefUpdate.Result.NEW || result == RefUpdate.Result.FORCED, + () -> "the fixture reference is written: " + result); + return id; + } + } + + /** The files of the tip of a reference, path to content. */ + private static Map files(Repository repository, String ref) throws IOException { + Ref tip = Objects.requireNonNull(repository.exactRef(ref), "the reference exists"); + Map files = new TreeMap<>(); + try (RevWalk walk = new RevWalk(repository); TreeWalk tree = new TreeWalk(repository)) { + tree.addTree(walk.parseCommit(tip.getObjectId()).getTree()); + tree.setRecursive(true); + while (tree.next()) { + files.put(tree.getPathString(), + new String(repository.open(tree.getObjectId(0)).getBytes(), StandardCharsets.UTF_8)); + } + } + return files; + } + + private StoreWriter open(StoreRepository storeRepository) { + return open(storeRepository, new JGitRepositoryOperations(RemotePolicy.localOnly())); + } + + private StoreWriter open(StoreRepository storeRepository, RepositoryOperations operations) { + var user = new TechnicalUser(word(), null, new UsernamePasswordCredentialsProvider(word(), word())); + return value(StoreWriter.open(storeRepository, temp.resolve(word() + ".git"), user, configuration(), + operations, Clock.fixed(Instant.now(), ZoneOffset.UTC))); + } + + /** Waits until the thread parks on a lock or ends, from a callback that may not throw a checked exception. */ + private static void awaitParked(Thread thread) { + try { + ThreadStates.awaitParkedOrEnded(thread, TIMEOUT_SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("the wait for the parked thread is interrupted", e); + } + } + + /** + * Runs the tasks on one thread each, released together, and returns their results in task order. One deadline + * bounds the whole scenario: a worker that has not ended by then fails the test with a {@code TimeoutException}, + * and the teardown cancels every outstanding task and waits for the workers at most {@value #TEARDOWN_SECONDS} + * seconds, so a blocked worker cannot hang the test. + */ + private static List concurrently(List> tasks) throws Exception { + var start = new CountDownLatch(1); + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS); + ExecutorService executor = Executors.newFixedThreadPool(tasks.size()); + List> futures = new ArrayList<>(); + List results = new ArrayList<>(); + boolean ended; + try { + for (Callable task : tasks) { + futures.add(executor.submit(() -> { + assertTrue(start.await(TIMEOUT_SECONDS, TimeUnit.SECONDS), "the threads are released"); + return task.call(); + })); + } + start.countDown(); + for (Future future : futures) { + results.add(future.get(Math.max(0, deadline - System.nanoTime()), TimeUnit.NANOSECONDS)); + } + } finally { + futures.forEach(future -> future.cancel(true)); + executor.shutdownNow(); + ended = executor.awaitTermination(TEARDOWN_SECONDS, TimeUnit.SECONDS); + } + assertTrue(ended, "the workers end"); + return results; + } + + @Test + @DisplayName("GDR-NFR-2: ten writers with their own clones save different files of one branch and all succeed") + void tenWritersWithOwnClones(@Remote LocalRemote remote) throws Exception { + var key = key(); + var target = StoreReferences.target(key); + Map expected = new TreeMap<>(Map.of(word() + "-base", text())); + commit(remote.repository(), target, expected); + var storeRepository = new StoreRepository(word(), remote.uri().toString(), word(), null); + List writers = new ArrayList<>(); + try { + List> saves = new ArrayList<>(); + for (int index = 0; index < WRITERS; index++) { + var writer = open(storeRepository); + writers.add(writer); + var author = Author.of(pseudonym()); + var path = word() + "-" + index; + var content = text(); + expected.put(path, content); + value(writer.draft(author, key, put(path, content), word())); + saves.add(() -> writer.save(author, key, Map.of(), word())); + } + + var results = concurrently(saves); + + assertAll( + () -> assertTrue(results.stream().allMatch(SaveResult.Saved.class::isInstance), + () -> "every save succeeds: " + results), + () -> assertEquals(expected, files(remote.repository(), target), "the branch holds all files")); + } finally { + writers.forEach(StoreWriter::close); + } + } + + @Test + @DisplayName("GDR-NFR-2: two writers saving the same file from the same blob end in one save and one conflict") + void twoWritersOnSameFile(@Remote LocalRemote remote) throws Exception { + var key = key(); + var target = StoreReferences.target(key); + var path = word(); + var loaded = text(); + commit(remote.repository(), target, Map.of(path, loaded)); + var storeRepository = new StoreRepository(word(), remote.uri().toString(), word(), null); + var expectedBlobIds = Map.of(path, blob(loaded)); + List contents = List.of(text() + "a", text() + "b"); + + try (var first = open(storeRepository); var second = open(storeRepository)) { + List writers = List.of(first, second); + List> saves = new ArrayList<>(); + for (int index = 0; index < writers.size(); index++) { + var writer = writers.get(index); + var author = Author.of(pseudonym()); + value(writer.draft(author, key, put(path, contents.get(index)), word())); + saves.add(() -> writer.save(author, key, expectedBlobIds, word())); + } + + var results = concurrently(saves); + + var saved = results.stream().filter(SaveResult.Saved.class::isInstance).toList(); + var conflicting = results.stream().filter(SaveResult.Conflicting.class::isInstance) + .map(SaveResult.Conflicting.class::cast).toList(); + assertAll( + () -> assertEquals(1, saved.size(), () -> "exactly one save succeeds: " + results), + () -> assertEquals(1, conflicting.size(), () -> "exactly one save conflicts: " + results), + () -> assertEquals(List.of(path), conflicting.getFirst().paths(), "the conflict names the file"), + () -> assertEquals(contents.get(results.indexOf(saved.getFirst())), + files(remote.repository(), target).get(path), "the branch holds the content of the save")); + } + } + + @Test + @DisplayName("GDR-NFR-2: a second save of one draft waits for the first and finds the draft settled, not replayed") + void secondSaveOfOneDraftWaitsForTheFirst(@Remote LocalRemote remote) throws Exception { + var key = key(); + var target = StoreReferences.target(key); + commit(remote.repository(), target, Map.of(word() + "-base", text())); + var storeRepository = new StoreRepository(word(), remote.uri().toString(), word(), null); + var operations = new ScriptedRepositoryOperations(); + var author = Author.of(pseudonym()); + AtomicReference second = new AtomicReference<>(); + + try (var writer = open(storeRepository, operations)) { + value(writer.draft(author, key, put(word(), text()), word())); + var secondSave = new Thread(() -> second.set(writer.save(author, key, Map.of(), word()))); + operations.beforeNextPush(() -> { + secondSave.start(); + awaitParked(secondSave); + }); + + var first = writer.save(author, key, Map.of(), word()); + secondSave.join(TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS)); + + var tip = Objects.requireNonNull(remote.repository().exactRef(target), "the branch exists").getObjectId(); + assertAll( + () -> assertFalse(secondSave.isAlive(), "the second save ends"), + () -> assertEquals(new SaveResult.Saved(tip.name()), first, + "the first save pushes the only commit of the branch"), + () -> assertEquals(new SaveResult.NotSaved(new StoreResult.PassedOn(GitOutcome.NOT_FOUND), + "draft: not found"), second.get(), "the second save finds the draft settled"), + () -> assertEquals(1, operations.pushes().size(), "the draft's changes are pushed once")); + } + } + + @Test + @DisplayName("GDR-NFR-2: ten threads sharing one writer save different files of one branch and all succeed") + void tenThreadsSharingOneWriter(@Remote LocalRemote remote) throws Exception { + var key = key(); + var target = StoreReferences.target(key); + Map expected = new TreeMap<>(Map.of(word() + "-base", text())); + commit(remote.repository(), target, expected); + var storeRepository = new StoreRepository(word(), remote.uri().toString(), word(), null); + + try (var writer = open(storeRepository)) { + List> saves = new ArrayList<>(); + for (int index = 0; index < WRITERS; index++) { + var author = Author.of(pseudonym()); + var path = word() + "-" + index; + var content = text(); + expected.put(path, content); + saves.add(() -> { + value(writer.draft(author, key, put(path, content), word())); + return writer.save(author, key, Map.of(), word()); + }); + } + + var results = concurrently(saves); + + assertAll( + () -> assertTrue(results.stream().allMatch(SaveResult.Saved.class::isInstance), + () -> "every save succeeds: " + results), + () -> assertEquals(expected, files(remote.repository(), target), "the branch holds all files")); + } + } +} diff --git a/git-store/src/test/java/de/cuioss/git/document/store/DraftExpiryTest.java b/git-store/src/test/java/de/cuioss/git/document/store/DraftExpiryTest.java new file mode 100644 index 0000000..e51af65 --- /dev/null +++ b/git-store/src/test/java/de/cuioss/git/document/store/DraftExpiryTest.java @@ -0,0 +1,686 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import static org.junit.jupiter.api.Assertions.assertAll; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assumptions.assumeFalse; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.Period; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; + + +import de.cuioss.git.document.core.GitOutcome; +import de.cuioss.git.document.core.InCoreWriter; +import de.cuioss.git.document.core.JGitRepositoryOperations; +import de.cuioss.git.document.core.ReferenceUpdater; +import de.cuioss.git.document.core.RemotePolicy; +import de.cuioss.git.document.core.TreeEdit; +import de.cuioss.git.document.store.DocumentCollection.Kind; +import de.cuioss.git.document.testing.LocalRemote; +import de.cuioss.git.document.testing.Remote; +import de.cuioss.test.generator.Generators; +import de.cuioss.test.generator.junit.EnableGeneratorController; +import de.cuioss.test.juli.LogAsserts; +import de.cuioss.test.juli.TestLogLevel; +import de.cuioss.test.juli.junit5.EnableTestLogger; +import org.eclipse.jgit.api.errors.JGitInternalException; +import org.eclipse.jgit.internal.storage.file.FileRepository; +import org.eclipse.jgit.lib.CommitBuilder; +import org.eclipse.jgit.lib.ConfigConstants; +import org.eclipse.jgit.lib.Constants; +import org.eclipse.jgit.lib.FileMode; +import org.eclipse.jgit.lib.ObjectId; +import org.eclipse.jgit.lib.ObjectInserter; +import org.eclipse.jgit.lib.PersonIdent; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.lib.RefUpdate; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.lib.TreeFormatter; +import org.eclipse.jgit.revwalk.ObjectWalk; +import org.eclipse.jgit.revwalk.RevCommit; +import org.eclipse.jgit.revwalk.RevObject; +import org.eclipse.jgit.revwalk.RevWalk; +import org.eclipse.jgit.storage.file.FileBasedConfig; +import org.eclipse.jgit.storage.file.FileRepositoryBuilder; +import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider; +import org.jspecify.annotations.Nullable; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * The expiry of drafts of GDR-STORE-7 with fixed clocks: a draft whose tip is at or before {@code now} minus + * {@code drafts.expiry} is deleted, a younger one kept, and a draft changed between the read and the delete kept; after + * the run the objects reachable only from deleted drafts are gone from the object database, while every object + * reachable from {@code main}, the working branches and the kept drafts is still there. A stale draft whose update is + * paused between its existence check and its compare-and-swap is not deleted beneath it. A garbage collection JGit + * skips because another process holds {@code gc.pid}, whether taken before the expiry or while it runs, is a + * {@link GitOutcome#CONFLICT}, and a failed one is passed on as {@link GitOutcome#FAILED} naming the exception class + * only. + *

+ * Names, ids, pseudonyms, paths, contents, periods and times are generated; the reference prefixes and the details are + * the specified vocabulary and kept as literals. + */ +@EnableGeneratorController +@EnableTestLogger +@DisplayName("DraftExpiry (GDR-STORE-7)") +class DraftExpiryTest { + + /** The attempts of a compare-and-delete in the tests that build the expiry directly. */ + private static final int ATTEMPTS = 3; + + /** The time a paused scenario may take before it counts as hanging. */ + private static final long TIMEOUT_SECONDS = 60; + + @TempDir + Path temp; + + private static String word() { + return Generators.strings("abcdefghijklmnopqrstuvwxyz", 6, 16).next(); + } + + private static String pseudonym() { + return Generators.strings("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-", 8, 24).next(); + } + + /** A content long enough to be unique among the generated contents of a test. */ + private static byte[] content() { + return Generators.letterStrings(32, 64).next().getBytes(StandardCharsets.UTF_8); + } + + /** A commit time in whole seconds, the resolution a commit keeps. */ + private static Instant time() { + return Instant.ofEpochSecond(Generators.integers(1_000_000_000, 2_000_000_000).next()); + } + + private static Period expiry() { + return Period.ofDays(Generators.integers(1, 90).next()); + } + + private static V value(StoreResult result) { + assertTrue(result.isOk(), () -> "accepted: " + result.detail()); + return Objects.requireNonNull(result.value(), "a success carries its value"); + } + + private static DocumentKey key(Kind kind) { + var collection = value(DocumentCollection.of("c" + word(), kind)); + return value(DocumentKey.of(collection, kind == Kind.PER_DOCUMENT ? Ulid.next() : "R-" + word())); + } + + /** A configuration tree with the three required keys, the draft expiry given. */ + private static StoreConfiguration configuration(Period expiry) { + Map tree = new LinkedHashMap<>(); + tree.put("save", Map.of("max-attempts", Generators.integers(1, 10).next())); + tree.put("drafts", Map.of("expiry", expiry.toString())); + tree.put("fetch", Map.of("interval", Duration.ofMinutes(Generators.integers(1, 60).next()).toString())); + return value(StoreConfiguration.from(tree, RemotePolicy.localOnly())); + } + + private static TechnicalUser user() { + return new TechnicalUser(word(), null, new UsernamePasswordCredentialsProvider(word(), word())); + } + + /** The id the object database gives a blob of the content. */ + private static ObjectId blobId(byte[] content) { + return new ObjectInserter.Formatter().idFor(Constants.OBJ_BLOB, content); + } + + /** Writes a commit with one file at the given committer time onto a reference, forced. */ + private static ObjectId commit(Repository repository, String ref, byte[] content, Instant when, + ObjectId... parents) throws IOException { + try (ObjectInserter inserter = repository.newObjectInserter()) { + var blob = inserter.insert(Constants.OBJ_BLOB, content); + var tree = new TreeFormatter(); + tree.append(word(), FileMode.REGULAR_FILE, blob); + var commit = new CommitBuilder(); + var ident = new PersonIdent(pseudonym(), "", when, ZoneOffset.UTC); + commit.setTreeId(inserter.insert(tree)); + commit.setParentIds(parents); + commit.setAuthor(ident); + commit.setCommitter(ident); + commit.setMessage(word()); + var id = inserter.insert(commit); + inserter.flush(); + RefUpdate update = repository.updateRef(ref); + update.setNewObjectId(id); + update.setForceUpdate(true); + update.disableRefLog(); + var result = update.update(); + assertTrue(result == RefUpdate.Result.NEW || result == RefUpdate.Result.FORCED, + () -> "the fixture reference " + ref + " is written: " + result); + return id; + } + } + + /** Every object reachable from the tips: commits, trees and blobs. */ + private static Set reachable(Repository repository, List tips) throws IOException { + Set ids = new HashSet<>(); + try (ObjectWalk walk = new ObjectWalk(repository)) { + for (ObjectId tip : tips) { + walk.markStart(walk.parseAny(tip)); + } + RevObject object; + while ((object = walk.next()) != null) { + ids.add(object.copy()); + } + while ((object = walk.nextObject()) != null) { + ids.add(object.copy()); + } + } + return ids; + } + + private static Repository openRepository(Path gitDir) throws IOException { + return new FileRepositoryBuilder().setGitDir(gitDir.toFile()).setMustExist(true).build(); + } + + /** Creates an empty bare repository and returns its git directory. */ + private Path bareRepository() throws IOException { + var gitDir = temp.resolve(word() + ".git"); + try (Repository created = new FileRepositoryBuilder().setGitDir(gitDir.toFile()).setBare().build()) { + created.create(true); + } + return gitDir; + } + + private static StoreWriter open(StoreRepository storeRepository, Path gitDir, StoreConfiguration configuration, + Instant now) { + return value(StoreWriter.open(storeRepository, gitDir, user(), configuration, + new JGitRepositoryOperations(RemotePolicy.localOnly()), Clock.fixed(now, ZoneOffset.UTC))); + } + + /** + * A repository whose first detached update of one reference is preceded by another writer moving that reference + * onto a new commit: the delete of the expiry then meets a reference changed between its read and its delete. + */ + private static final class MovingRepository extends FileRepository { + + private final String moved; + private final Instant when; + private @Nullable ObjectId movedTo; + + MovingRepository(File gitDir, String moved, Instant when) throws IOException { + super(gitDir); + this.moved = moved; + this.when = when; + } + + @Override + public RefUpdate updateRef(String ref, boolean detach) throws IOException { + if (detach && movedTo == null && moved.equals(ref)) { + Ref current = Objects.requireNonNull(exactRef(ref), "the moved reference exists"); + movedTo = commit(this, ref, content(), when, current.getObjectId()); + } + return super.updateRef(ref, detach); + } + + ObjectId movedTo() { + return Objects.requireNonNull(movedTo, "the reference was moved"); + } + } + + /** + * A repository in which another process takes {@value DraftExpiry#GC_PID} after the expiry has written its + * {@code gc.pruneExpire} into the configuration and before JGit takes that file: JGit 7.8.0 + * {@code GarbageCollectCommand} reads the configuration in its constructor, before {@code GC.doGc} takes its + * {@code GC.PidLock}. + */ + private static final class PidTakingRepository extends FileRepository { + + private boolean taken; + + PidTakingRepository(File gitDir) throws IOException { + super(gitDir); + } + + @Override + public FileBasedConfig getConfig() { + FileBasedConfig config = super.getConfig(); + if (!taken && config.getString(ConfigConstants.CONFIG_GC_SECTION, null, + ConfigConstants.CONFIG_KEY_PRUNEEXPIRE) != null) { + taken = true; + try { + Files.writeString(getDirectory().toPath().resolve(DraftExpiry.GC_PID), word() + " " + word()); + } catch (IOException e) { + throw new UncheckedIOException("the fixture gc.pid is written", e); + } + } + return config; + } + + boolean taken() { + return taken; + } + } + + @Nested + @DisplayName("Through the writer") + class ThroughWriter { + + @Test + @DisplayName("GDR-STORE-7: a stale draft is deleted, a fresh one kept, and only the stale objects collected") + void deletesStaleKeepsFreshAndCollects(@Remote LocalRemote remote) throws Exception { + var main = commit(remote.repository(), StoreReferences.MAIN, content(), time()); + var staleKey = key(Kind.PER_DOCUMENT); + var target = commit(remote.repository(), StoreReferences.target(staleKey), content(), time(), main); + var expiry = expiry(); + var configuration = configuration(expiry); + var storeRepository = new StoreRepository(word(), remote.uri().toString(), word(), null); + var gitDir = temp.resolve(word() + ".git"); + var staleAt = time(); + var now = staleAt.atOffset(ZoneOffset.UTC).plus(expiry).toInstant(); + var staleAuthor = Author.of(pseudonym()); + var staleContent = content(); + var staleRef = value(StoreReferences.draft(staleAuthor.pseudonym(), staleKey)); + var freshAuthor = Author.of(pseudonym()); + var freshKey = key(Kind.NUMBERED); + var freshRef = value(StoreReferences.draft(freshAuthor.pseudonym(), freshKey)); + + String staleCommitId; + try (var staleWriter = open(storeRepository, gitDir, configuration, staleAt)) { + staleCommitId = value(staleWriter.draft(staleAuthor, staleKey, + List.of(new TreeEdit.Put(word(), staleContent)), word())); + } + try (var writer = open(storeRepository, gitDir, configuration, now)) { + var freshCommitId = value(writer.draft(freshAuthor, freshKey, + List.of(new TreeEdit.Put(word(), content())), word())); + + ObjectId staleCommit = ObjectId.fromString(staleCommitId); + List staleOnly = new ArrayList<>(); + Set kept; + try (Repository clone = openRepository(gitDir); RevWalk walk = new RevWalk(clone)) { + RevCommit parsed = walk.parseCommit(staleCommit); + staleOnly.add(staleCommit); + staleOnly.add(parsed.getTree().copy()); + staleOnly.add(blobId(staleContent)); + List keptTips = new ArrayList<>(); + for (Ref ref : clone.getRefDatabase().getRefsByPrefix(Constants.R_REFS)) { + if (!staleRef.equals(ref.getName())) { + keptTips.add(ref.getObjectId()); + } + } + kept = reachable(clone, keptTips); + List staleBefore = List.copyOf(staleOnly); + assertAll("the fixture before the expiry", + () -> assertTrue(kept.containsAll(List.of(main, target, + ObjectId.fromString(freshCommitId))), "main, branch and fresh draft are kept tips"), + () -> assertTrue(staleBefore.stream().allMatch(id -> !kept.contains(id)), + "the stale objects are reachable from the stale draft only"), + () -> assertTrue(staleBefore.stream().allMatch(id -> has(clone, id)), + "the stale objects exist before the expiry")); + } + + var deleted = value(writer.expireDrafts(now)); + + try (Repository clone = openRepository(gitDir)) { + Ref fresh = clone.exactRef(freshRef); + assertAll( + () -> assertEquals(List.of(staleRef), deleted, "the stale draft is reported deleted"), + () -> assertNull(clone.exactRef(staleRef), "the stale draft is deleted"), + () -> assertNotNull(fresh, "the fresh draft is kept"), + () -> assertEquals(freshCommitId, Objects.requireNonNull(fresh).getObjectId().name(), + "the fresh draft is unchanged"), + () -> assertTrue(staleOnly.stream().noneMatch(id -> has(clone, id)), + "no object reachable only from the stale draft is left"), + () -> assertTrue(kept.stream().allMatch(id -> has(clone, id)), + "every object reachable from main, the branches and the kept draft is present")); + } + } + LogAsserts.assertSingleLogMessagePresentContaining(TestLogLevel.INFO, + GitStoreLogMessages.INFO.DRAFTS_EXPIRED.resolveIdentifierString()); + } + + @Test + @DisplayName("GDR-STORE-7: a draft one second younger than drafts.expiry is kept") + void keepsDraftJustYoungerThanExpiry(@Remote LocalRemote remote) throws Exception { + commit(remote.repository(), StoreReferences.MAIN, content(), time()); + var expiry = expiry(); + var storeRepository = new StoreRepository(word(), remote.uri().toString(), word(), null); + var gitDir = temp.resolve(word() + ".git"); + var draftAt = time(); + var author = Author.of(pseudonym()); + var key = key(Kind.MAIN); + var now = draftAt.atOffset(ZoneOffset.UTC).plus(expiry).toInstant().minusSeconds(1); + + try (var writer = open(storeRepository, gitDir, configuration(expiry), draftAt)) { + var commitId = value(writer.draft(author, key, List.of(new TreeEdit.Put(word(), content())), word())); + + var deleted = value(writer.expireDrafts(now)); + + try (Repository clone = openRepository(gitDir)) { + Ref draft = clone.exactRef(value(StoreReferences.draft(author.pseudonym(), key))); + assertAll( + () -> assertEquals(List.of(), deleted, "no draft is deleted"), + () -> assertNotNull(draft, "the draft is kept"), + () -> assertEquals(commitId, Objects.requireNonNull(draft).getObjectId().name(), + "the draft is unchanged"), + () -> assertTrue(has(clone, ObjectId.fromString(commitId)), "its commit is present")); + } + } + } + + @Test + @DisplayName("GDR-STORE-7: a null time is refused") + void refusesNull(@Remote LocalRemote remote) { + var storeRepository = new StoreRepository(word(), remote.uri().toString(), word(), null); + Instant nullTime = nullValue(); + + try (var writer = open(storeRepository, temp.resolve(word() + ".git"), configuration(expiry()), time())) { + assertThrows(NullPointerException.class, () -> writer.expireDrafts(nullTime)); + } + } + } + + /** + * A clock whose first reading pauses its caller until released: a draft update reads it once, between its + * existence check and its compare-and-swap. + */ + private static final class PausingClock extends Clock { + + private final Instant now; + private final AtomicBoolean armed = new AtomicBoolean(true); + private final CountDownLatch paused = new CountDownLatch(1); + private final CountDownLatch released = new CountDownLatch(1); + + PausingClock(Instant now) { + this.now = now; + } + + @Override + public ZoneId getZone() { + return ZoneOffset.UTC; + } + + @Override + public Clock withZone(ZoneId zone) { + throw new UnsupportedOperationException("the fixture clock stays in UTC"); + } + + @Override + public Instant instant() { + if (armed.compareAndSet(true, false)) { + paused.countDown(); + try { + if (!released.await(TIMEOUT_SECONDS, TimeUnit.SECONDS)) { + throw new IllegalStateException("the paused reading is never released"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("the paused reading is interrupted", e); + } + } + return now; + } + + boolean awaitPaused() throws InterruptedException { + return paused.await(TIMEOUT_SECONDS, TimeUnit.SECONDS); + } + + void release() { + released.countDown(); + } + } + + @Nested + @DisplayName("Races and failures") + class RacesAndFailures { + + @Test + @DisplayName("GDR-STORE-7: a stale draft is not deleted between an update's existence check and its swap") + void keepsDraftUpdatedDuringExpiry(@Remote LocalRemote remote) throws Exception { + commit(remote.repository(), StoreReferences.MAIN, content(), time()); + var expiry = expiry(); + var configuration = configuration(expiry); + var storeRepository = new StoreRepository(word(), remote.uri().toString(), word(), null); + var gitDir = temp.resolve(word() + ".git"); + var staleAt = time(); + var now = staleAt.atOffset(ZoneOffset.UTC).plus(expiry).toInstant(); + var author = Author.of(pseudonym()); + var key = key(Kind.PER_DOCUMENT); + var draftRef = value(StoreReferences.draft(author.pseudonym(), key)); + String staleCommitId; + try (var staleWriter = open(storeRepository, gitDir, configuration, staleAt)) { + staleCommitId = value(staleWriter.draft(author, key, List.of(new TreeEdit.Put(word(), content())), + word())); + } + var clock = new PausingClock(now); + AtomicReference>> expired = new AtomicReference<>(); + + String updated; + try (var writer = value(StoreWriter.open(storeRepository, gitDir, user(), configuration, + new JGitRepositoryOperations(RemotePolicy.localOnly()), clock)); + ExecutorService executor = Executors.newSingleThreadExecutor()) { + Future> update = executor.submit(() -> writer.draft(author, key, + List.of(new TreeEdit.Put(word(), content())), word())); + assertTrue(clock.awaitPaused(), "the update pauses before its compare-and-swap"); + var expiryRun = new Thread(() -> expired.set(writer.expireDrafts(now))); + expiryRun.start(); + ThreadStates.awaitParkedOrEnded(expiryRun, TIMEOUT_SECONDS); + clock.release(); + updated = value(update.get(TIMEOUT_SECONDS, TimeUnit.SECONDS)); + expiryRun.join(TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS)); + assertFalse(expiryRun.isAlive(), "the expiry ends"); + } + + try (Repository clone = openRepository(gitDir); RevWalk walk = new RevWalk(clone)) { + RevCommit commit = walk.parseCommit(ObjectId.fromString(updated)); + Ref draft = clone.exactRef(draftRef); + assertAll( + () -> assertEquals(List.of(ObjectId.fromString(staleCommitId)), List.of(commit.getParents()), + "the update stacks on the draft it found, not on nothing"), + () -> assertNotNull(draft, "the draft is kept"), + () -> assertEquals(updated, Objects.requireNonNull(draft).getObjectId().name(), + "the draft points at the update"), + () -> assertEquals(List.of(), value(Objects.requireNonNull(expired.get())), + "the updated draft is not deleted")); + } + } + + @Test + @DisplayName("GDR-STORE-7: a draft changed between the read and the delete is kept with its new commit") + void keepsDraftChangedMeanwhile() throws Exception { + var gitDir = bareRepository(); + var expiry = expiry(); + var draftAt = time(); + var now = draftAt.atOffset(ZoneOffset.UTC).plus(expiry).toInstant(); + var draftRef = value(StoreReferences.draft(pseudonym(), key(Kind.PER_DOCUMENT))); + ObjectId read; + try (Repository writer = openRepository(gitDir)) { + read = commit(writer, draftRef, content(), draftAt); + } + + try (var repository = new MovingRepository(gitDir.toFile(), draftRef, draftAt)) { + var expiryRun = new DraftExpiry(word(), repository, new RepositoryGuard(), + new ReferenceUpdater(new InCoreWriter()), expiry, ATTEMPTS); + + var deleted = value(expiryRun.expire(now)); + + try (Repository clone = openRepository(gitDir)) { + Ref draft = clone.exactRef(draftRef); + ObjectId movedTo = repository.movedTo(); + assertAll( + () -> assertEquals(List.of(), deleted, "no draft is deleted"), + () -> assertNotNull(draft, "the changed draft is kept"), + () -> assertEquals(movedTo, Objects.requireNonNull(draft).getObjectId(), + "the draft keeps the commit of the other writer"), + () -> assertTrue(has(clone, movedTo), "the new commit is present"), + () -> assertTrue(has(clone, read), "the commit that was read, its parent, is present")); + } + } + } + + @Test + @DisplayName("GDR-STORE-7: an applied-draft reference is deleted with its draft and kept beside a live one") + void deletesOrphanedAppliedReferences() throws Exception { + var gitDir = bareRepository(); + var expiry = expiry(); + var now = time(); + var orphanDraft = value(StoreReferences.draft(pseudonym(), key(Kind.PER_DOCUMENT))); + var liveDraft = value(StoreReferences.draft(pseudonym(), key(Kind.NUMBERED))); + var orphan = StoreReferences.applied(orphanDraft); + var live = StoreReferences.applied(liveDraft); + ObjectId orphanCommit; + ObjectId liveCommit; + try (Repository writer = openRepository(gitDir)) { + orphanCommit = commit(writer, orphan, content(), now); + liveCommit = commit(writer, live, content(), now); + commit(writer, liveDraft, content(), now, liveCommit); + } + + try (Repository repository = openRepository(gitDir)) { + var deleted = value(new DraftExpiry(word(), repository, new RepositoryGuard(), + new ReferenceUpdater(new InCoreWriter()), expiry, ATTEMPTS).expire(now)); + + assertAll( + () -> assertEquals(List.of(), deleted, "no draft is stale"), + () -> assertNull(repository.exactRef(orphan), "the reference without draft is deleted"), + () -> assertFalse(has(repository, orphanCommit), "its commit is collected"), + () -> assertEquals(liveCommit, Objects.requireNonNull(repository.exactRef(live)).getObjectId(), + "the reference of a live draft is kept")); + } + } + + @Test + @DisplayName("GDR-STORE-7: a garbage collection skipped for a held gc.pid is a retryable CONFLICT, no success") + void skippedGarbageCollectionIsNoSuccess() throws Exception { + var gitDir = bareRepository(); + var now = time(); + var draftRef = value(StoreReferences.draft(pseudonym(), key(Kind.MAIN))); + try (Repository writer = openRepository(gitDir)) { + commit(writer, draftRef, content(), now); + } + Files.writeString(gitDir.resolve(DraftExpiry.GC_PID), word() + " " + word()); + + StoreResult> result; + try (Repository repository = openRepository(gitDir)) { + result = new DraftExpiry(word(), repository, new RepositoryGuard(), + new ReferenceUpdater(new InCoreWriter()), expiry(), ATTEMPTS).expire(now); + } + + assertAll( + () -> assertEquals(new StoreResult>(new StoreResult.PassedOn(GitOutcome.CONFLICT), + null, DraftExpiry.GC_SKIPPED), result, "the skipped collection is a conflict"), + () -> LogAsserts.assertLogMessagePresentContaining(TestLogLevel.WARN, + GitStoreLogMessages.WARN.GARBAGE_COLLECTION_SKIPPED.resolveIdentifierString())); + } + + @Test + @DisplayName("GDR-STORE-7: a garbage collection skipped for a gc.pid taken after the expiry began is a CONFLICT") + void collectionSkippedForPidTakenMeanwhileIsNoSuccess() throws Exception { + var gitDir = bareRepository(); + var now = time(); + var draftRef = value(StoreReferences.draft(pseudonym(), key(Kind.MAIN))); + try (Repository writer = openRepository(gitDir)) { + commit(writer, draftRef, content(), now); + } + + StoreResult> result; + boolean taken; + try (var repository = new PidTakingRepository(gitDir.toFile())) { + result = new DraftExpiry(word(), repository, new RepositoryGuard(), + new ReferenceUpdater(new InCoreWriter()), expiry(), ATTEMPTS).expire(now); + taken = repository.taken(); + } + + assertAll( + () -> assertTrue(taken, "the other process takes gc.pid before JGit does"), + () -> assertEquals(new StoreResult>(new StoreResult.PassedOn(GitOutcome.CONFLICT), + null, DraftExpiry.GC_SKIPPED), result, "the skipped collection is a conflict"), + () -> LogAsserts.assertLogMessagePresentContaining(TestLogLevel.WARN, + GitStoreLogMessages.WARN.GARBAGE_COLLECTION_SKIPPED.resolveIdentifierString())); + } + + @Test + @DisplayName("GDR-STORE-7: a failed garbage collection is passed on as FAILED naming the exception class only") + void passesOnFailedGarbageCollection() throws Exception { + var gitDir = bareRepository(); + var now = time(); + var draftRef = value(StoreReferences.draft(pseudonym(), key(Kind.MAIN))); + try (Repository writer = openRepository(gitDir)) { + commit(writer, draftRef, content(), now); + } + File pack = Files.createDirectories(gitDir.resolve("objects").resolve("pack")).toFile(); + assertTrue(pack.setWritable(false), "the pack directory is made read-only"); + try { + assumeFalse(Files.isWritable(pack.toPath()), "the user can still write a read-only directory"); + var alias = word(); + + StoreResult> result; + try (Repository repository = openRepository(gitDir)) { + result = new DraftExpiry(alias, repository, new RepositoryGuard(), + new ReferenceUpdater(new InCoreWriter()), expiry(), ATTEMPTS).expire(now); + } + + var detail = Objects.requireNonNull(result.detail(), "a failure carries a detail"); + assertAll( + () -> assertEquals(new StoreResult.PassedOn(GitOutcome.FAILED), result.outcome(), + "the failure is passed on"), + () -> assertEquals("gc: " + JGitInternalException.class.getSimpleName(), detail, + "the detail names the exception class"), + () -> assertFalse(detail.contains(temp.toString()), "the detail names no path")); + LogAsserts.assertLogMessagePresentContaining(TestLogLevel.WARN, + GitStoreLogMessages.WARN.GARBAGE_COLLECTION_FAILED.resolveIdentifierString()); + } finally { + assertTrue(pack.setWritable(true), "the pack directory is writable again"); + } + } + } + + private static boolean has(Repository repository, ObjectId id) { + try { + return repository.getObjectDatabase().has(id); + } catch (IOException e) { + throw new IllegalStateException("the object database is readable", e); + } + } + + /** Passes {@code null} where the signature forbids it, to exercise the null check. */ + @SuppressWarnings("unchecked") + private static T nullValue() { + return (T) null; + } +} diff --git a/git-store/src/test/java/de/cuioss/git/document/store/ExplicitSaveTest.java b/git-store/src/test/java/de/cuioss/git/document/store/ExplicitSaveTest.java new file mode 100644 index 0000000..9b49f06 --- /dev/null +++ b/git-store/src/test/java/de/cuioss/git/document/store/ExplicitSaveTest.java @@ -0,0 +1,1120 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import static org.junit.jupiter.api.Assertions.assertAll; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assumptions.assumeFalse; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.Period; +import java.time.ZoneOffset; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.TreeMap; + + +import de.cuioss.git.document.core.GitOutcome; +import de.cuioss.git.document.core.JGitRepositoryOperations; +import de.cuioss.git.document.core.RemotePolicy; +import de.cuioss.git.document.core.TreeEdit; +import de.cuioss.git.document.store.DocumentCollection.Kind; +import de.cuioss.git.document.testing.LocalRemote; +import de.cuioss.git.document.testing.Remote; +import de.cuioss.test.generator.Generators; +import de.cuioss.test.generator.junit.EnableGeneratorController; +import de.cuioss.test.juli.LogAsserts; +import de.cuioss.test.juli.TestLogLevel; +import de.cuioss.test.juli.junit5.EnableTestLogger; +import org.eclipse.jgit.dircache.DirCache; +import org.eclipse.jgit.dircache.DirCacheBuilder; +import org.eclipse.jgit.dircache.DirCacheEntry; +import org.eclipse.jgit.lib.CommitBuilder; +import org.eclipse.jgit.lib.ConfigConstants; +import org.eclipse.jgit.lib.Constants; +import org.eclipse.jgit.lib.FileMode; +import org.eclipse.jgit.lib.ObjectId; +import org.eclipse.jgit.lib.ObjectInserter; +import org.eclipse.jgit.lib.PersonIdent; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.lib.RefUpdate; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.revwalk.RevCommit; +import org.eclipse.jgit.revwalk.RevWalk; +import org.eclipse.jgit.storage.file.FileRepositoryBuilder; +import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider; +import org.eclipse.jgit.treewalk.TreeWalk; +import org.jspecify.annotations.Nullable; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.EnumSource; + +/** + * The explicit save against a {@code git-testing} local remote: one squash commit on the fetched tip pushed with lease + * and re-applied after a lost lease (GDR-STORE-3), conflicts that change nothing (GDR-STORE-4), the expected blob ids + * (GDR-STORE-5), and author, committer and credentials (GDR-STORE-8). {@link ScriptedRepositoryOperations} records the + * transport inputs, scripts unreachable remotes, invalid credentials and refused leases, and moves the remote between + * fetch and push as a concurrent writer would. The pruning fetch of a save keeps the drafts of a clone whose stored + * fetch refspec reaches beyond the mirrored namespaces. + *

+ * Names, ids, pseudonyms, paths, contents, addresses and times are generated; the reference prefixes, the outcomes and + * the details are the specified vocabulary and kept as literals. + */ +@EnableGeneratorController +@EnableTestLogger +@DisplayName("ExplicitSave (GDR-STORE-3, GDR-STORE-4, GDR-STORE-5, GDR-STORE-8)") +class ExplicitSaveTest { + + /** The attempts of most tests: enough for one lost lease. */ + private static final int ATTEMPTS = 3; + + @TempDir + Path temp; + + private static String word() { + return Generators.strings("abcdefghijklmnopqrstuvwxyz", 6, 16).next(); + } + + private static String pseudonym() { + return Generators.strings("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-", 8, 24).next(); + } + + private static String address() { + return word() + "@" + word() + ".example"; + } + + private static String text() { + return Generators.letterStrings(1, 64).next(); + } + + private static Instant time() { + return Instant.ofEpochSecond(Generators.integers(1_000_000_000, 2_000_000_000).next()); + } + + private static V value(StoreResult result) { + assertTrue(result.isOk(), () -> "accepted: " + result.detail()); + return Objects.requireNonNull(result.value(), "a success carries its value"); + } + + private static DocumentKey key(Kind kind) { + var collection = value(DocumentCollection.of("c" + word(), kind)); + return value(DocumentKey.of(collection, kind == Kind.PER_DOCUMENT ? Ulid.next() : "R-" + word())); + } + + private static StoreConfiguration configuration(@Nullable Boolean email, int attempts) { + Map tree = new LinkedHashMap<>(); + tree.put("save", Map.of("max-attempts", attempts)); + tree.put("drafts", Map.of("expiry", Period.ofDays(Generators.integers(1, 90).next()).toString())); + tree.put("fetch", Map.of("interval", Duration.ofMinutes(Generators.integers(1, 60).next()).toString())); + if (email != null) { + tree.put("commit", Map.of("email", email)); + } + return value(StoreConfiguration.from(tree, RemotePolicy.localOnly())); + } + + private static TechnicalUser user(@Nullable String email) { + return new TechnicalUser(word(), email, new UsernamePasswordCredentialsProvider(word(), word())); + } + + /** + * Files with generated contents that end in the tag, by name, so files of different roles (base, draft, concurrent + * change) never share a content by chance. + */ + private static Map contents(String tag, String... names) { + Map files = new TreeMap<>(); + for (String name : names) { + files.put(name, text() + "-" + tag); + } + return files; + } + + /** Files with generated contents, by name; the names are flat, so a tree holds them without subtrees. */ + private static Map files(String... names) { + Map files = new TreeMap<>(); + for (String name : names) { + files.put(name, text()); + } + return files; + } + + /** Writes a commit holding exactly the files, nested paths included, onto a reference, forcing the reference. */ + private static ObjectId commit(Repository repository, String ref, Map files, + ObjectId... parents) { + try (ObjectInserter inserter = repository.newObjectInserter()) { + DirCache index = DirCache.newInCore(); + DirCacheBuilder builder = index.builder(); + for (Map.Entry file : files.entrySet()) { + var entry = new DirCacheEntry(file.getKey()); + entry.setFileMode(FileMode.REGULAR_FILE); + entry.setObjectId( + inserter.insert(Constants.OBJ_BLOB, file.getValue().getBytes(StandardCharsets.UTF_8))); + builder.add(entry); + } + builder.finish(); + var commit = new CommitBuilder(); + var ident = new PersonIdent(pseudonym(), ""); + commit.setTreeId(index.writeTree(inserter)); + commit.setParentIds(parents); + commit.setAuthor(ident); + commit.setCommitter(ident); + commit.setMessage(word()); + var id = inserter.insert(commit); + inserter.flush(); + RefUpdate update = repository.updateRef(ref); + update.setNewObjectId(id); + update.setForceUpdate(true); + var result = update.update(); + assertTrue(result == RefUpdate.Result.NEW || result == RefUpdate.Result.FORCED + || result == RefUpdate.Result.FAST_FORWARD, () -> "the fixture reference is written: " + result); + return id; + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + private static @Nullable ObjectId tip(Repository repository, String ref) throws IOException { + Ref found = repository.exactRef(ref); + return found == null ? null : found.getObjectId(); + } + + private static RevCommit parse(Repository repository, ObjectId id) throws IOException { + try (RevWalk walk = new RevWalk(repository)) { + RevCommit commit = walk.parseCommit(id); + walk.parseBody(commit); + return commit; + } + } + + /** The files of a commit, path to content. */ + private static Map tree(Repository repository, ObjectId id) throws IOException { + Map files = new TreeMap<>(); + try (TreeWalk walk = new TreeWalk(repository)) { + walk.addTree(parse(repository, id).getTree()); + walk.setRecursive(true); + while (walk.next()) { + files.put(walk.getPathString(), new String(repository.open(walk.getObjectId(0)).getBytes(), + StandardCharsets.UTF_8)); + } + } + return files; + } + + private static String blob(String content) { + try (ObjectInserter.Formatter formatter = new ObjectInserter.Formatter()) { + return formatter.idFor(Constants.OBJ_BLOB, content.getBytes(StandardCharsets.UTF_8)).name(); + } + } + + private static List puts(Map files) { + return files.entrySet().stream().map(file -> new TreeEdit.Put(file.getKey(), + file.getValue().getBytes(StandardCharsets.UTF_8))).toList(); + } + + private static Map merged(Map base, Map changes) { + Map merged = new TreeMap<>(base); + merged.putAll(changes); + return merged; + } + + /** The writer under test with its clone, its technical user, its clock and its transport double. */ + private record Fixture(StoreWriter writer, Path gitDir, TechnicalUser user, Instant now, + ScriptedRepositoryOperations operations) implements AutoCloseable { + + Repository openClone() throws IOException { + return new FileRepositoryBuilder().setGitDir(gitDir.toFile()).setMustExist(true).build(); + } + + @Nullable + ObjectId localTip(String ref) throws IOException { + try (Repository clone = openClone()) { + return tip(clone, ref); + } + } + + String draft(Author author, DocumentKey key, List edits) { + return value(writer.draft(author, key, edits, word())); + } + + SaveResult save(Author author, DocumentKey key, Map expected) { + return writer.save(author, key, expected, word()); + } + + @Override + public void close() { + writer.close(); + } + } + + private Fixture open(LocalRemote remote, StoreConfiguration configuration, TechnicalUser user) { + var storeRepository = new StoreRepository(word(), remote.uri().toString(), word(), null); + var gitDir = temp.resolve(word() + ".git"); + var now = time(); + var operations = new ScriptedRepositoryOperations(); + var writer = value(StoreWriter.open(storeRepository, gitDir, user, configuration, operations, + Clock.fixed(now, ZoneOffset.UTC))); + return new Fixture(writer, gitDir, user, now, operations); + } + + private Fixture open(LocalRemote remote) { + return open(remote, configuration(null, ATTEMPTS), user(null)); + } + + @Nested + @DisplayName("Explicit save (GDR-STORE-3)") + class Save { + + @Test + @DisplayName("GDR-STORE-3: a save is one commit on the remote tip, the draft deleted and never pushed") + void savesOneCommitOnRemoteTip(@Remote LocalRemote remote) { + commit(remote.repository(), StoreReferences.MAIN, files(word())); + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var base = files(word()); + var tipId = commit(remote.repository(), target, base); + var author = Author.of(pseudonym()); + var changes = files(word() + "x"); + + try (var fixture = open(remote)) { + var draft = fixture.draft(author, key, puts(changes)); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = ObjectId.fromString(saved.commitId()); + assertAll( + () -> assertEquals(commit, tip(remote.repository(), target), "the remote target moved"), + () -> assertEquals(List.of(tipId), List.of(parse(remote.repository(), commit).getParents()), + "the only parent is the remote tip"), + () -> assertEquals(merged(base, changes), tree(remote.repository(), commit), + "the draft's changes on the remote tip"), + () -> assertFalse(remote.repository().getObjectDatabase().has(ObjectId.fromString(draft)), + "no draft commit reached the remote"), + () -> assertEquals(commit, fixture.localTip(target), "the local target moved"), + () -> assertNull(fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the draft is deleted"), + () -> LogAsserts.assertLogMessagePresentContaining(TestLogLevel.INFO, + GitStoreLogMessages.INFO.SAVED.resolveIdentifierString())); + } + } + + @Test + @DisplayName("GDR-STORE-3: the save applies onto the fetched tip, not the tip the draft started from") + void appliesOntoFetchedTip(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var base = files(word()); + var start = commit(remote.repository(), target, base); + var author = Author.of(pseudonym()); + var changes = files(word() + "x"); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(changes)); + var concurrent = merged(base, files(word() + "y")); + var moved = commit(remote.repository(), target, concurrent, start); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = ObjectId.fromString(saved.commitId()); + assertAll( + () -> assertEquals(List.of(moved), List.of(parse(remote.repository(), commit).getParents()), + "the only parent is the fetched tip"), + () -> assertEquals(merged(concurrent, changes), tree(remote.repository(), commit), + "the concurrent change is kept")); + } + } + + @Test + @DisplayName("GDR-STORE-3: a path the draft deleted is deleted by the save") + void deletesPath(@Remote LocalRemote remote) throws Exception { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var kept = word(); + var deleted = kept + "d"; + var base = files(kept, deleted); + commit(remote.repository(), target, base); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, List.of(new TreeEdit.Delete(deleted))); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + assertEquals(Map.of(kept, base.get(kept)), + tree(remote.repository(), ObjectId.fromString(saved.commitId())), "only the kept file"); + } + } + + @Test + @DisplayName("GDR-STORE-3: a draft moved during the save is kept for the next save") + void keepsNewerDraft(@Remote LocalRemote remote) throws Exception { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + commit(remote.repository(), target, files(word())); + var author = Author.of(pseudonym()); + var later = word() + "l"; + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(files(word() + "x"))); + String[] newer = new String[1]; + fixture.operations().beforeNextPush(() -> newer[0] = fixture.draft(author, key, puts(files(later)))); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var remoteTree = tree(remote.repository(), ObjectId.fromString(saved.commitId())); + assertAll( + () -> assertEquals(ObjectId.fromString(newer[0]), + fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the newer draft is kept"), + () -> assertFalse(remoteTree.containsKey(later), "the newer change is not saved")); + } + } + + @Test + @DisplayName("GDR-STORE-3: the pruning fetch of a save keeps every draft of a clone with a broader fetch refspec") + void pruningFetchKeepsDraftsOfABroaderClone(@Remote LocalRemote remote) throws Exception { + var key = key(Kind.PER_DOCUMENT); + commit(remote.repository(), StoreReferences.target(key), files(word())); + var author = Author.of(pseudonym()); + var otherAuthor = Author.of(pseudonym()); + var otherKey = key(Kind.NUMBERED); + var keptRef = value(StoreReferences.draft(otherAuthor.pseudonym(), otherKey)); + var keptApplied = StoreReferences.applied(keptRef); + + try (var fixture = open(remote)) { + var kept = ObjectId.fromString(fixture.draft(otherAuthor, otherKey, puts(files(word())))); + fixture.draft(author, key, puts(files(word() + "x"))); + try (Repository clone = fixture.openClone()) { + RefUpdate applied = clone.updateRef(keptApplied); + applied.setNewObjectId(kept); + applied.disableRefLog(); + assertEquals(RefUpdate.Result.NEW, applied.update(), "the applied-draft reference is written"); + var config = clone.getConfig(); + config.setStringList(ConfigConstants.CONFIG_REMOTE_SECTION, Constants.DEFAULT_REMOTE_NAME, + ConfigConstants.CONFIG_FETCH_SECTION, List.of("+refs/*:refs/*")); + config.save(); + } + + assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + assertAll( + () -> assertEquals(kept, fixture.localTip(keptRef), "the unsaved draft is kept"), + () -> assertEquals(kept, fixture.localTip(keptApplied), + "its applied-draft reference is kept"), + () -> assertEquals(JGitRepositoryOperations.MIRROR_FETCH_REF_SPECS, + fixture.operations().fetches().getLast().refSpecs(), + "the save fetches the mirrored namespaces only")); + } + } + + @Test + @DisplayName("GDR-STORE-3: a retry after a lost lease re-applies the draft the save started with") + void retryKeepsTheDraftItStartedWith(@Remote LocalRemote remote) throws Exception { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var base = files(word()); + var start = commit(remote.repository(), target, base); + var author = Author.of(pseudonym()); + var changes = files(word() + "x"); + var later = word() + "l"; + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(changes)); + String[] newer = new String[1]; + fixture.operations().beforeNextPush(() -> { + newer[0] = fixture.draft(author, key, puts(files(later))); + commit(remote.repository(), target, merged(base, files(word() + "y")), start); + }); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var remoteTree = tree(remote.repository(), ObjectId.fromString(saved.commitId())); + assertAll( + () -> assertEquals(2, fixture.operations().pushes().size(), "a second attempt"), + () -> assertTrue(remoteTree.keySet().containsAll(changes.keySet()), "the started draft"), + () -> assertFalse(remoteTree.containsKey(later), "the draft written meanwhile is not saved"), + () -> assertEquals(ObjectId.fromString(newer[0]), + fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the newer draft is kept")); + } + } + + @Test + @DisplayName("GDR-STORE-3: a save the remote accepted is Saved although the local branch cannot be moved") + void acceptedSaveWithLocalMoveFailure(@Remote LocalRemote remote) throws Exception { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + commit(remote.repository(), target, files(word())); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(files(word() + "x"))); + File branchDirectory = Files.createDirectories(fixture.gitDir().resolve(target).getParent()).toFile(); + fixture.operations().beforeNextPush(() -> assertTrue(branchDirectory.setWritable(false), + "the directory of the local branch is made read-only")); + try { + var result = fixture.save(author, key, Map.of()); + assumeFalse(Files.isWritable(branchDirectory.toPath()), + "the user can still write a read-only directory"); + + var saved = assertInstanceOf(SaveResult.Saved.class, result, "the accepted save"); + assertAll( + () -> assertEquals(ObjectId.fromString(saved.commitId()), tip(remote.repository(), target), + "the remote holds the commit"), + () -> assertNull(fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the draft is deleted"), + () -> LogAsserts.assertLogMessagePresentContaining(TestLogLevel.WARN, + GitStoreLogMessages.WARN.LOCAL_BRANCH_NOT_MOVED.resolveIdentifierString())); + } finally { + assertTrue(branchDirectory.setWritable(true), "the directory is writable again"); + } + } + } + + @Test + @DisplayName("GDR-STORE-3: a target branch the remote deleted is created again against the zero-id lease") + void deletedRemoteBranchIsCreatedAgain(@Remote LocalRemote remote) throws Exception { + var main = commit(remote.repository(), StoreReferences.MAIN, files(word())); + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + commit(remote.repository(), target, files(word()), main); + var author = Author.of(pseudonym()); + var other = Author.of(pseudonym() + "-o"); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(files(word() + "x"))); + var otherDraft = fixture.draft(other, key, puts(files(word() + "o"))); + RefUpdate delete = remote.repository().updateRef(target); + delete.setForceUpdate(true); + assertEquals(RefUpdate.Result.FORCED, delete.delete(), "the remote branch is deleted"); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = ObjectId.fromString(saved.commitId()); + assertAll( + () -> assertEquals(commit, tip(remote.repository(), target), "the target is created again"), + () -> assertEquals(List.of(main), List.of(parse(remote.repository(), commit).getParents()), + "the parent is the tip of main"), + () -> assertEquals(ObjectId.zeroId().name(), + fixture.operations().pushes().getLast().expectedRemoteHead(), "the zero-id lease"), + () -> assertEquals(ObjectId.fromString(otherDraft), + fixture.localTip(value(StoreReferences.draft(other.pseudonym(), key))), + "the fetch leaves the drafts alone")); + } + } + + @Test + @DisplayName("GDR-STORE-3: a kept newer draft saves only what it changed since the save that kept it") + void keptDraftSavesOnlyItsLaterChanges(@Remote LocalRemote remote) throws Exception { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var path = word(); + var later = path + "l"; + commit(remote.repository(), target, Map.of(path, text() + "-base")); + var author = Author.of(pseudonym()); + var theirs = text() + "-theirs"; + var laterContent = text() + "-later"; + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(Map.of(path, text() + "-mine"))); + String[] newer = new String[1]; + fixture.operations().beforeNextPush( + () -> newer[0] = fixture.draft(author, key, puts(Map.of(later, laterContent)))); + var first = ObjectId.fromString( + assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())).commitId()); + var concurrent = commit(remote.repository(), target, + merged(tree(remote.repository(), first), Map.of(path, theirs)), first); + + var second = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = ObjectId.fromString(second.commitId()); + assertAll( + () -> assertEquals(List.of(concurrent), + List.of(parse(remote.repository(), commit).getParents()), "on the concurrent change"), + () -> assertEquals(merged(tree(remote.repository(), concurrent), Map.of(later, laterContent)), + tree(remote.repository(), commit), + "only the later change is saved, the concurrent change is kept"), + () -> assertNull(fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the draft is deleted")); + } + } + + @Test + @DisplayName("GDR-STORE-3: a missing target branch starts at main and is pushed against the zero-id lease") + void missingTargetStartsAtMain(@Remote LocalRemote remote) { + var main = commit(remote.repository(), StoreReferences.MAIN, files(word())); + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(files(word() + "x"))); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = ObjectId.fromString(saved.commitId()); + assertAll( + () -> assertEquals(commit, tip(remote.repository(), target), "the target is created"), + () -> assertEquals(List.of(main), List.of(parse(remote.repository(), commit).getParents()), + "the parent is the tip of main"), + () -> assertEquals(ObjectId.zeroId().name(), + fixture.operations().pushes().getFirst().expectedRemoteHead(), "the zero-id lease"), + () -> assertEquals(main, tip(remote.repository(), StoreReferences.MAIN), "main is unchanged")); + } + } + + @Test + @DisplayName("GDR-STORE-3: the zero-id lease is refused for a branch created meanwhile, and the save re-applies") + void zeroIdLeaseRefusesCreatedBranch(@Remote LocalRemote remote) { + var main = commit(remote.repository(), StoreReferences.MAIN, files(word())); + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(files(word() + "x"))); + ObjectId[] created = new ObjectId[1]; + fixture.operations().beforeNextPush( + () -> created[0] = commit(remote.repository(), target, files(word() + "y"), main)); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = ObjectId.fromString(saved.commitId()); + var pushes = fixture.operations().pushes(); + assertAll( + () -> assertEquals(2, pushes.size(), "a second attempt"), + () -> assertEquals(ObjectId.zeroId().name(), pushes.getFirst().expectedRemoteHead(), + "the first push expects the branch absent"), + () -> assertEquals(created[0].name(), pushes.get(1).expectedRemoteHead(), + "the second push expects the created branch"), + () -> assertEquals(List.of(created[0]), + List.of(parse(remote.repository(), commit).getParents()), + "the save is applied onto the created branch")); + } + } + + @Test + @DisplayName("GDR-STORE-3: a lease refusal re-applies the save onto the new tip") + void reappliesAfterLeaseRefusal(@Remote LocalRemote remote) { + var key = key(Kind.NUMBERED); + var target = StoreReferences.target(key); + var base = files(word()); + var start = commit(remote.repository(), target, base); + var author = Author.of(pseudonym()); + var changes = files(word() + "x"); + var concurrent = merged(base, files(word() + "y")); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(changes)); + ObjectId[] moved = new ObjectId[1]; + fixture.operations().beforeNextPush(() -> moved[0] = commit(remote.repository(), target, concurrent, + start)); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = ObjectId.fromString(saved.commitId()); + assertAll( + () -> assertEquals(2, fixture.operations().pushes().size(), "a second attempt"), + () -> assertEquals(List.of(moved[0]), List.of(parse(remote.repository(), commit).getParents()), + "the only parent is the new tip"), + () -> assertEquals(merged(concurrent, changes), tree(remote.repository(), commit), + "both changes")); + } + } + + @Test + @DisplayName("GDR-STORE-3: content without release is saved onto main") + void mainKindSavesOntoMain(@Remote LocalRemote remote) { + var main = commit(remote.repository(), StoreReferences.MAIN, files(word())); + var key = key(Kind.MAIN); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(files(word() + "x"))); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = ObjectId.fromString(saved.commitId()); + assertAll( + () -> assertEquals(commit, tip(remote.repository(), StoreReferences.MAIN), "main moved"), + () -> assertEquals(List.of(main), List.of(parse(remote.repository(), commit).getParents()), + "the parent is the former tip of main")); + } + } + + @Test + @DisplayName("GDR-STORE-3: in an empty repository the save is a root commit") + void emptyRepositoryRootCommit(@Remote LocalRemote remote) { + var key = key(Kind.MAIN); + var author = Author.of(pseudonym()); + var changes = files(word()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(changes)); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = ObjectId.fromString(saved.commitId()); + assertAll( + () -> assertEquals(0, parse(remote.repository(), commit).getParentCount(), "a root commit"), + () -> assertEquals(changes, tree(remote.repository(), commit), "the draft's files")); + } + } + + @Test + @DisplayName("GDR-STORE-3: a draft without changes is saved as the base, with no commit and no push") + void draftWithoutChanges(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var base = files(word()); + var tipId = commit(remote.repository(), target, base); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(base)); + + var result = fixture.save(author, key, Map.of()); + + assertAll( + () -> assertEquals(new SaveResult.Saved(tipId.name()), result, "saved as the base"), + () -> assertEquals(List.of(), fixture.operations().pushes(), "no push"), + () -> assertEquals(tipId, tip(remote.repository(), target), "the remote is unchanged"), + () -> assertNull(fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the draft is deleted")); + } + } + + @Test + @DisplayName("GDR-STORE-3: a draft without changes and without its target branch is not saved and kept") + void draftWithoutChangesOrTarget(@Remote LocalRemote remote) { + var base = files(word()); + var main = commit(remote.repository(), StoreReferences.MAIN, base); + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + var draft = fixture.draft(author, key, puts(base)); + + var result = fixture.save(author, key, Map.of()); + + assertAll( + () -> assertEquals(new SaveResult.NotSaved(new StoreResult.PassedOn(GitOutcome.NOT_FOUND), + "draft: no changes and no target branch"), result, "no branch to name"), + () -> assertEquals(List.of(), fixture.operations().pushes(), "no push"), + () -> assertNull(tip(remote.repository(), target), "no target branch is created"), + () -> assertEquals(main, tip(remote.repository(), StoreReferences.MAIN), "main is unchanged"), + () -> assertEquals(ObjectId.fromString(draft), + fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the draft is kept")); + } + } + + @Test + @DisplayName("GDR-STORE-3: an unreachable remote is reported and changes nothing") + void unreachableChangesNothing(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var tipId = commit(remote.repository(), target, files(word())); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + var draft = fixture.draft(author, key, puts(files(word() + "x"))); + fixture.operations().failFetch(GitOutcome.UNREACHABLE); + + var result = fixture.save(author, key, Map.of()); + + assertAll( + () -> assertEquals(new SaveResult.NotSaved(new StoreResult.PassedOn(GitOutcome.UNREACHABLE), + ScriptedRepositoryOperations.SCRIPTED), result, "unreachable"), + () -> assertEquals(List.of(), fixture.operations().pushes(), "no push"), + () -> assertEquals(tipId, tip(remote.repository(), target), "the remote is unchanged"), + () -> assertEquals(tipId, fixture.localTip(target), "the local target is unchanged"), + () -> assertEquals(ObjectId.fromString(draft), + fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the draft keeps the work"), + () -> LogAsserts.assertLogMessagePresentContaining(TestLogLevel.WARN, + GitStoreLogMessages.WARN.SAVE_NOT_COMPLETED.resolveIdentifierString())); + } + } + + @ParameterizedTest(name = "{0}") + @EnumSource(value = GitOutcome.class, names = {"UNREACHABLE", "CREDENTIALS_INVALID"}) + @DisplayName("GDR-STORE-3: a push that fails ends the save at once; the local branch and the draft stay") + void failedPushMovesNothing(GitOutcome outcome, @Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var tipId = commit(remote.repository(), target, files(word())); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + var draft = fixture.draft(author, key, puts(files(word() + "x"))); + fixture.operations().failPush(outcome); + + var result = fixture.save(author, key, Map.of()); + + assertAll( + () -> assertEquals(new SaveResult.NotSaved(new StoreResult.PassedOn(outcome), + ScriptedRepositoryOperations.SCRIPTED), result, "passed on"), + () -> assertEquals(1, fixture.operations().pushes().size(), "no second attempt"), + () -> assertEquals(tipId, fixture.localTip(target), "the local target is unchanged"), + () -> assertEquals(ObjectId.fromString(draft), + fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the draft is kept")); + } + } + + @Test + @DisplayName("GDR-STORE-3: a save whose every attempt is refused or rejected is rejected") + void rejectedAfterLastAttempt(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var tipId = commit(remote.repository(), target, files(word())); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + var draft = fixture.draft(author, key, puts(files(word() + "x"))); + fixture.operations().failPush(GitOutcome.LEASE_REFUSED, GitOutcome.REJECTED, GitOutcome.LEASE_REFUSED); + + var result = fixture.save(author, key, Map.of()); + + assertAll( + () -> assertEquals(new SaveResult.NotSaved(new StoreResult.PassedOn(GitOutcome.REJECTED), + target + ": lease refused or rejected on every attempt (" + ATTEMPTS + ")"), result, + "rejected"), + () -> assertEquals(ATTEMPTS, fixture.operations().pushes().size(), "every attempt pushed"), + () -> assertEquals(ATTEMPTS, fixture.operations().fetches().size(), + "every attempt fetched first"), + () -> assertEquals(tipId, tip(remote.repository(), target), "the remote is unchanged"), + () -> assertEquals(tipId, fixture.localTip(target), "the local target is unchanged"), + () -> assertEquals(ObjectId.fromString(draft), + fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the draft is kept")); + } + } + + @Test + @DisplayName("GDR-STORE-3: a missing draft, an invalid pseudonym and an invalid expected id are not saved") + void refusals(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + + try (var fixture = open(remote)) { + assertAll( + () -> assertEquals(new SaveResult.NotSaved(new StoreResult.PassedOn(GitOutcome.NOT_FOUND), + "draft: not found"), fixture.save(Author.of(pseudonym()), key, Map.of()), + "a missing draft"), + () -> assertEquals(new SaveResult.NotSaved(StoreOutcome.MALFORMED, "pseudonym: invalid"), + fixture.save(Author.of(word() + "/" + word()), key, Map.of()), "an invalid pseudonym"), + () -> assertEquals(new SaveResult.NotSaved(new StoreResult.PassedOn(GitOutcome.NOT_FOUND), + "expectedBlobIds: not a full object id"), + fixture.save(Author.of(pseudonym()), key, Map.of(word(), word())), + "an invalid expected id"), + () -> assertEquals(List.of(), fixture.operations().fetches(), "nothing was fetched")); + } + } + } + + @Nested + @DisplayName("Conflicts (GDR-STORE-4)") + class Conflicts { + + @Test + @DisplayName("GDR-STORE-4: a conflict names every conflicting path; nothing is pushed, draft and branch unchanged") + void namesEveryConflictingPath(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var stem = word(); + var first = stem + "a"; + var second = stem + "b"; + var added = stem + "c"; + var base = contents("base", first, second); + var start = commit(remote.repository(), target, base); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + var draft = fixture.draft(author, key, puts(contents("draft", first, second, added))); + var moved = commit(remote.repository(), target, contents("concurrent", first, second), start); + Map expected = Map.of(first, blob(base.get(first)), second, blob(base.get(second)), + added, ObjectId.zeroId().name()); + + var result = fixture.save(author, key, expected); + + assertAll( + () -> assertEquals(new SaveResult.Conflicting(List.of(first, second)), result, + "every conflicting path"), + () -> assertEquals(List.of(), fixture.operations().pushes(), "nothing is pushed"), + () -> assertEquals(moved, tip(remote.repository(), target), "the remote is unchanged"), + () -> assertEquals(moved, fixture.localTip(target), "the local target is the fetched tip"), + () -> assertEquals(ObjectId.fromString(draft), + fixture.localTip(value(StoreReferences.draft(author.pseudonym(), key))), + "the draft is unchanged"), + () -> LogAsserts.assertLogMessagePresentContaining(TestLogLevel.INFO, + GitStoreLogMessages.INFO.SAVE_CONFLICT.resolveIdentifierString())); + } + } + + @Test + @DisplayName("GDR-STORE-4: a draft adding a path below a file another writer added conflicts on that path") + void fileAddedWhereTheDraftAddsADirectory(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var base = files(word()); + var start = commit(remote.repository(), target, base); + var directory = word() + "d"; + var nested = directory + "/" + word(); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(Map.of(nested, text()))); + var moved = commit(remote.repository(), target, merged(base, Map.of(directory, text())), start); + + var result = fixture.save(author, key, Map.of(nested, ObjectId.zeroId().name())); + + assertAll( + () -> assertEquals(new SaveResult.Conflicting(List.of(nested)), result, "the nested path"), + () -> assertEquals(List.of(), fixture.operations().pushes(), "nothing is pushed"), + () -> assertEquals(moved, tip(remote.repository(), target), "the remote is unchanged")); + } + } + + @Test + @DisplayName("GDR-STORE-4: a draft adding a file where another writer added a directory conflicts on it") + void directoryAddedWhereTheDraftAddsAFile(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var base = files(word()); + var start = commit(remote.repository(), target, base); + var path = word() + "f"; + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(Map.of(path, text()))); + var moved = commit(remote.repository(), target, merged(base, Map.of(path + "/" + word(), text())), + start); + + var result = fixture.save(author, key, Map.of()); + + assertAll( + () -> assertEquals(new SaveResult.Conflicting(List.of(path)), result, "the file's path"), + () -> assertEquals(List.of(), fixture.operations().pushes(), "nothing is pushed"), + () -> assertEquals(moved, tip(remote.repository(), target), "the remote is unchanged")); + } + } + + @Test + @DisplayName("GDR-STORE-4: a draft adding a file below an existing directory is saved") + void additionBelowExistingDirectory(@Remote LocalRemote remote) throws Exception { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var directory = word() + "d"; + var base = Map.of(directory + "/" + word() + "a", text()); + commit(remote.repository(), target, base); + var changes = Map.of(directory + "/" + word() + "b", text()); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(changes)); + + var saved = assertInstanceOf(SaveResult.Saved.class, + fixture.save(author, key, Map.of(changes.keySet().iterator().next(), + ObjectId.zeroId().name()))); + + assertEquals(merged(base, changes), tree(remote.repository(), ObjectId.fromString(saved.commitId())), + "both files"); + } + } + } + + @Nested + @DisplayName("Concurrent change detection (GDR-STORE-5)") + class ChangeDetection { + + @Test + @DisplayName("GDR-STORE-5: an expected blob id that differs on the fetched tip is a conflict") + void mismatchIsConflict(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var path = word(); + commit(remote.repository(), target, contents("base", path)); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(contents("draft", path))); + + var result = fixture.save(author, key, Map.of(path, blob(text() + "-loaded"))); + + assertEquals(new SaveResult.Conflicting(List.of(path)), result, "the path conflicts"); + } + } + + @Test + @DisplayName("GDR-STORE-5: a path without an expected blob id overwrites a concurrent change") + void absentExpectationOverwrites(@Remote LocalRemote remote) throws Exception { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var path = word(); + var start = commit(remote.repository(), target, contents("base", path)); + var author = Author.of(pseudonym()); + var mine = contents("draft", path); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(mine)); + commit(remote.repository(), target, contents("concurrent", path), start); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + assertEquals(mine, tree(remote.repository(), ObjectId.fromString(saved.commitId())), + "the draft's content wins"); + } + } + + @Test + @DisplayName("GDR-STORE-5: the zero id expects the path absent: saved when absent, a conflict when added") + void zeroIdExpectsAbsence(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var stem = word(); + var absent = stem + "a"; + var addedMeanwhile = stem + "b"; + var start = commit(remote.repository(), target, files(stem)); + var author = Author.of(pseudonym()); + var other = Author.of(pseudonym() + "-o"); + var zero = ObjectId.zeroId().name(); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(files(absent))); + fixture.draft(other, key, puts(files(addedMeanwhile))); + commit(remote.repository(), target, files(stem, addedMeanwhile), start); + + var saved = fixture.save(author, key, Map.of(absent, zero)); + var conflicting = fixture.save(other, key, Map.of(addedMeanwhile, zero)); + + assertAll( + () -> assertInstanceOf(SaveResult.Saved.class, saved, "an absent path is saved"), + () -> assertEquals(new SaveResult.Conflicting(List.of(addedMeanwhile)), conflicting, + "a path added meanwhile conflicts")); + } + } + } + + @Nested + @DisplayName("Author, committer and credentials (GDR-STORE-8)") + class Identities { + + @Test + @DisplayName("GDR-STORE-8: by default the save commit carries pseudonym and technical user with empty e-mail") + void defaultIdentities(@Remote LocalRemote remote) throws Exception { + var key = key(Kind.MAIN); + var author = new Author(pseudonym(), address()); + var user = user(address()); + + try (var fixture = open(remote, configuration(null, ATTEMPTS), user)) { + fixture.draft(author, key, puts(files(word()))); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = parse(remote.repository(), ObjectId.fromString(saved.commitId())); + assertAll( + () -> assertEquals(author.pseudonym(), commit.getAuthorIdent().getName(), "author"), + () -> assertEquals("", commit.getAuthorIdent().getEmailAddress(), "no author address"), + () -> assertEquals(user.name(), commit.getCommitterIdent().getName(), "committer"), + () -> assertEquals("", commit.getCommitterIdent().getEmailAddress(), "no committer address"), + () -> assertEquals(fixture.now(), commit.getCommitterIdent().getWhenAsInstant(), + "the commit time of the clock")); + } + } + + @Test + @DisplayName("GDR-STORE-8: with commit.email true the save commit carries the supplied addresses") + void identitiesWithEmail(@Remote LocalRemote remote) throws Exception { + var key = key(Kind.MAIN); + var author = new Author(pseudonym(), address()); + var user = user(address()); + + try (var fixture = open(remote, configuration(true, ATTEMPTS), user)) { + fixture.draft(author, key, puts(files(word()))); + + var saved = assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var commit = parse(remote.repository(), ObjectId.fromString(saved.commitId())); + assertAll( + () -> assertEquals(author.email(), commit.getAuthorIdent().getEmailAddress(), "author"), + () -> assertEquals(user.email(), commit.getCommitterIdent().getEmailAddress(), "committer")); + } + } + + @Test + @DisplayName("GDR-STORE-8: every clone, fetch and push receives exactly the technical user's credentials") + void credentialsOfTechnicalUser(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + commit(remote.repository(), StoreReferences.target(key), files(word())); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(files(word() + "x"))); + assertInstanceOf(SaveResult.Saved.class, fixture.save(author, key, Map.of())); + + var credentials = fixture.operations().credentials(); + assertAll( + () -> assertEquals(1 + fixture.operations().fetches().size() + + fixture.operations().pushes().size(), credentials.size(), "every transport call"), + () -> assertTrue(fixture.operations().pushes().size() >= 1, "a push happened"), + () -> assertTrue(credentials.stream().allMatch(given -> given == fixture.user().credentials()), + "the technical user's credentials"), + () -> assertSame(fixture.user().credentials(), credentials.getFirst(), "the clone")); + } + } + + @Test + @DisplayName("GDR-STORE-8: invalid credentials on the fetch are passed on and change nothing") + void credentialsInvalidPassedOn(@Remote LocalRemote remote) { + var key = key(Kind.PER_DOCUMENT); + var target = StoreReferences.target(key); + var tipId = commit(remote.repository(), target, files(word())); + var author = Author.of(pseudonym()); + + try (var fixture = open(remote)) { + fixture.draft(author, key, puts(files(word() + "x"))); + fixture.operations().failFetch(GitOutcome.CREDENTIALS_INVALID); + + var result = fixture.save(author, key, Map.of()); + + assertAll( + () -> assertEquals(new SaveResult.NotSaved( + new StoreResult.PassedOn(GitOutcome.CREDENTIALS_INVALID), + ScriptedRepositoryOperations.SCRIPTED), result, "passed on"), + () -> assertEquals(List.of(), fixture.operations().pushes(), "no push"), + () -> assertEquals(tipId, tip(remote.repository(), target), "the remote is unchanged")); + } + } + } +} diff --git a/git-store/src/test/java/de/cuioss/git/document/store/LogMessagesCatalogueTest.java b/git-store/src/test/java/de/cuioss/git/document/store/LogMessagesCatalogueTest.java new file mode 100644 index 0000000..85d293b --- /dev/null +++ b/git-store/src/test/java/de/cuioss/git/document/store/LogMessagesCatalogueTest.java @@ -0,0 +1,93 @@ +/* + * Copyright © 2026-present CUI-OpenSource-Software (info@cuioss.de) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package de.cuioss.git.document.store; + +import static org.junit.jupiter.api.Assertions.assertAll; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.lang.reflect.Field; +import java.lang.reflect.Modifier; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; +import java.util.TreeMap; + + +import de.cuioss.tools.logging.LogRecord; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * The catalogue {@code git-store/doc/LogMessages.adoc} mirrors {@link GitStoreLogMessages} by hand (GDR-GEN-4): every + * declared {@link LogRecord} has a row with its identifier, the component {@value GitStoreLogMessages#PREFIX} and its + * template verbatim, and the catalogue lists no identifier that is not declared. The prose descriptions are not + * checked. + */ +@DisplayName("Log-message catalogue of git-store (GDR-GEN-4)") +class LogMessagesCatalogueTest { + + /** The catalogue, relative to the module directory Surefire runs the tests in. */ + private static final Path CATALOGUE = Path.of("doc", "LogMessages.adoc"); + + /** The identifier of every declared message, mapped to its template. */ + private static Map declared() throws IllegalAccessException { + Map declared = new TreeMap<>(); + for (Class level : GitStoreLogMessages.class.getDeclaredClasses()) { + for (Field field : level.getDeclaredFields()) { + if (Modifier.isStatic(field.getModifiers()) && LogRecord.class.isAssignableFrom(field.getType())) { + var logRecord = (LogRecord) field.get(null); + declared.put(logRecord.resolveIdentifierString(), logRecord.getTemplate()); + } + } + } + return declared; + } + + /** + * The identifier of every row of the catalogue, mapped to its template: a row starts with + * {@code |GIT_STORE- |GIT_STORE |