Bring PATCH in line with RFC 7644

doc/changelog.rst

@@ -4,13 +4,73 @@ Changelog
 [Unreleased]
 ------------
 
+Added
+^^^^^
+- :attr:`ScimPolicy.unmatched_path_filter <scim2_models.ScimPolicy.unmatched_path_filter>` can
+  make a PATCH ``add`` or ``replace`` create the entry its path filter describes when no entry
+  matches, as Microsoft Entra ID expects. By default, the operation still fails with ``noTarget``.
+
 Changed
 ^^^^^^^
 - Python 3.11 is now the minimum supported version.
 - The enumerations, such as :class:`~scim2_models.Mutability`, are :class:`~enum.StrEnum`:
   :class:`str` and f-strings give their value, ``readOnly`` rather than ``Mutability.read_only``.
 - In a model built from a schema, an attribute named after a member of the model, such as
   ``copy``, is held as ``copy_``. Its SCIM name is unchanged.
+- A PATCH ``add`` whose path filter matches no entry, such as ``emails[type eq "work"].value``
+  on a user without a work email, now fails with ``noTarget`` instead of silently doing nothing.
+  :meth:`Path.set <scim2_models.Path.set>` raises :class:`~scim2_models.NoTargetException` in
+  that case when strict.
+- A PATCH ``add`` or ``replace`` on a filtered path, such as ``emails[type eq "work"]``, merges
+  its value into the matching entries instead of replacing them
+  (:rfc:`RFC7644 §3.5.2.3 <7644#section-3.5.2.3>`). The entries are updated in place, so the
+  immutable sub-attributes of a group member cannot be changed this way.
+
+Fixed
+^^^^^
+- A :class:`~scim2_models.PatchOperation` with a null value keeps it when dumped. A ``replace``
+  that clears its target used to be sent without a value.
+- Setting a null value under an unset complex attribute or extension no longer creates an empty
+  one, and no longer reports the resource as modified.
+- A :class:`~scim2_models.PatchOp` with no operation, or with an operation other than ``add``,
+  ``remove`` and ``replace``, fails with ``invalidValue`` instead of a validation error without
+  ``scimType``.
+- A PATCH ``add`` or ``replace`` on a complex attribute keeps the sub-attributes its value leaves
+  out, instead of replacing the whole attribute (:rfc:`RFC7644 §3.5.2.3 <7644#section-3.5.2.3>`).
+- A PATCH ``add`` without a path adds to the multi-valued attributes in its value, like an
+  ``add`` with a path, instead of replacing their values.
+- A key of a PATCH value can be an attribute path, such as ``name.givenName`` or
+  ``urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:employeeNumber``, as Microsoft
+  Entra ID and its SCIM Validator send. It used to be rejected as an undeclared attribute.
+- Undeclared attributes and sub-attributes in a PATCH value follow
+  :attr:`ScimPolicy.unknown <scim2_models.ScimPolicy.unknown>`.
+- PATCH checks immutable attributes at every level and in every operation, including operations
+  with no path, an empty path or a schema URN, and the removal of an extension. For example, the
+  ``value`` of a group member can be added and removed, but not changed
+  (:rfc:`RFC7643 §4.2 <7643#section-4.2>`). Setting a first value with ``replace``, or writing
+  back the current value, is accepted.
+- PATCH checks read-only attributes at every level, such as the ``displayName`` of the
+  enterprise ``manager``. A path to one is rejected. A value that contains one is
+  rejected only if it changes it, so an attribute sent back as it was read is accepted. Okta, for
+  example, sends back the ``id`` of a group it renames.
+- PATCH rejects a change to a required attribute only when it leaves the attribute unset
+  (:rfc:`RFC7644 §3.5.2.2 <7644#section-3.5.2.2>`). Removing some values of a required
+  multi-valued attribute, removing a sub-attribute of a required complex attribute, or adding an
+  empty list is now accepted. Required sub-attributes and extensions are checked too.
+- A PATCH ``replace`` without a value fails with ``invalidValue`` instead of clearing its
+  target. So does an operation whose path targets the resource or an extension with a value that
+  is not an object, which used to be ignored.
+- A PATCH ``replace`` that sets several ``primary`` entries fails with ``invalidValue``, like
+  ``add`` already did, instead of keeping one of them at random.
+- :meth:`PatchOp.patch <scim2_models.PatchOp.patch>` raises
+  :class:`~scim2_models.InvalidValueException` when the attribute rejects a value, instead of a
+  pydantic :class:`~pydantic.ValidationError` without ``scimType``.
+- :meth:`PatchOp.patch <scim2_models.PatchOp.patch>` no longer reports a resource as modified
+  when an operation writes a complex or multi-valued value it already has.
+- A PATCH path to an undeclared attribute follows
+  :attr:`ScimPolicy.unknown <scim2_models.ScimPolicy.unknown>`, like a value does. With ``ignore``
+  or ``keep``, the operation changes nothing instead of failing the whole patch with
+  ``invalidPath``. An undeclared sub-attribute in a filter still fails with ``invalidFilter``.
 
 Security
 ^^^^^^^^
@@ -21,7 +81,6 @@ Security
 - :func:`~scim2_models.get_model_by_payload` matches no model when ``schemas`` is not a list
   of strings.
 
-
 [0.8.2] - 2026-09-25
 --------------------
 

doc/explanation/patch.rst

@@ -13,19 +13,82 @@ Validation and application
 
 Validating a PATCH message in :attr:`~scim2_models.Context.RESOURCE_PATCH_REQUEST` rejects what
 the message alone settles: a missing operation value, a ``remove`` without a path or carrying a
-value, a read-only target, and an operation that would unassign a required attribute.
+value, a path to a read-only attribute, and an operation that removes a required
+attribute or sets it to an empty value.
 
 :meth:`~scim2_models.PatchOp.patch` then applies the message to the stored resource. Operations
 run in their listed order. This is where an immutable value can be compared with the value it
-replaces, and where the method reports whether any operation changed the resource. Splitting the
-two keeps a parsed :class:`~scim2_models.PatchOp` useful before the resource is loaded.
+replaces, and where the method reports whether any operation changed the resource. It also
+rejects an operation that unassigns a required attribute in another way, such as removing its
+last entry through a filter. Read-only attributes in the value are checked there too. A client
+that sends back the ``id`` or ``meta`` it read is accepted, since
+:rfc:`RFC7643 §3.1 <7643#section-3.1>` says to ignore them. A client that changes them gets a
+``mutability`` error. Splitting the two keeps a parsed :class:`~scim2_models.PatchOp` useful
+before the resource is loaded.
 
 Operation outcomes
 ------------------
 
 ``add`` appends a value to a multi-valued attribute rather than replacing its list. ``replace``
-replaces its target, creating an unassigned single-valued complex parent when necessary.
-``remove`` removes a selected list entry or unassigns the targeted attribute.
+replaces the value of a simple or multi-valued target. ``remove`` removes a selected list entry
+or unassigns the targeted attribute.
+
+A path to a sub-attribute of an absent complex attribute, such as ``name.givenName`` on a user
+without a name, creates that attribute.
+
+A complex attribute is merged rather than replaced. ``add`` and ``replace`` set the
+sub-attributes in their value and keep the others, as
+:rfc:`RFC7644 §3.5.2.3 <7644#section-3.5.2.3>` requires:
+
+.. doctest::
+
+    >>> from scim2_models import PatchOp, PatchOperation, User
+
+    >>> user = User(user_name="bjensen", name={"family_name": "Jensen", "given_name": "Barbara"})
+    >>> patch = PatchOp[User](
+    ...     operations=[
+    ...         PatchOperation(
+    ...             op=PatchOperation.Op.replace_, path="name", value={"givenName": "Babs"}
+    ...         )
+    ...     ]
+    ... )
+    >>> patch.patch(user)
+    True
+    >>> user.name.given_name, user.name.family_name
+    ('Babs', 'Jensen')
+
+Operations without a path
+-------------------------
+
+The value of an ``add`` or ``replace`` without a path holds the attributes to write. Each
+attribute is handled like an operation with that attribute as its path. A key can also be an
+attribute path, such as ``name.givenName`` or the full URN of an extension attribute. Microsoft
+Entra ID and its SCIM Validator send such keys. :rfc:`RFC7643 §2.1 <7643#section-2.1>` forbids
+dots and colons in attribute names, so such a key cannot be confused with a name:
+
+.. doctest::
+
+    >>> from scim2_models import EnterpriseUser, PatchOp, PatchOperation, User
+
+    >>> user = User[EnterpriseUser](user_name="bjensen", name={"family_name": "Jensen"})
+    >>> patch = PatchOp[User[EnterpriseUser]](
+    ...     operations=[
+    ...         PatchOperation(
+    ...             op=PatchOperation.Op.replace_,
+    ...             value={
+    ...                 "name.givenName": "Barbara",
+    ...                 "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:employeeNumber": "42",
+    ...             },
+    ...         )
+    ...     ]
+    ... )
+    >>> patch.patch(user)
+    True
+    >>> user.name.given_name, user.name.family_name, user[EnterpriseUser].employee_number
+    ('Barbara', 'Jensen', '42')
+
+A key with a filter is not read as a path, and neither is a key that matches no declared
+attribute. Both follow :attr:`ScimPolicy.unknown <scim2_models.ScimPolicy.unknown>`.
 
 What a path selects
 -------------------
@@ -60,6 +123,32 @@ of a multi-valued attribute an operation applies to:
     >>> [email.value for email in user.emails]
     ['new@example.com', 'home@example.com']
 
+When a filter selects whole entries, ``add`` and ``replace`` merge their value into each selected
+entry, as they do for a complex attribute. :rfc:`RFC7644 §3.5.2.3 <7644#section-3.5.2.3>` says
+"all matching record values" are replaced, and keeps the sub-attributes the value does not
+specify:
+
+.. doctest::
+
+    >>> patch = PatchOp[User](
+    ...     operations=[
+    ...         PatchOperation(
+    ...             op=PatchOperation.Op.replace_,
+    ...             path='emails[type eq "home"]',
+    ...             value={"display": "Home"},
+    ...         )
+    ...     ]
+    ... )
+    >>> patch.patch(user)
+    True
+    >>> user.emails[1].type.value, user.emails[1].value, user.emails[1].display
+    ('home', 'home@example.com', 'Home')
+
+Each selected entry is updated in place, not replaced by a new one. So a client can change the
+``display`` of a group member without repeating its immutable ``value``. A different ``value`` is
+rejected with ``mutability``. To unassign a sub-attribute, give it a null value, or remove it with
+a path such as ``emails[type eq "home"].display``.
+
 A selection matching nothing
 ----------------------------
 
@@ -100,16 +189,27 @@ nothing changed:
     >>> patch.patch(user)
     False
 
-``add`` behaves the same way. :rfc:`RFC7644 §3.5.2.1 <7644#section-3.5.2.1>` does not say what a
-selection matching nothing means for it, so the operation is a no-op and not a failure.
-`Errata 8097 <https://errata.rfc-editor.org/eid8097/>`_ asks the RFC to say whether ``add``
-accepts a value selection at all, since implementations differ on it.
-
-.. note::
-
-   Microsoft Entra ID sends ``add`` operations whose selection matches nothing, and expects the
-   selected entry to be created. scim2-models does not create it, so an integration serving that
-   client handles the case before applying the operation.
+``add`` fails like ``replace``. :rfc:`RFC7644 §3.5.2.1 <7644#section-3.5.2.1>` does not say what
+a selection matching nothing means for it, and Table 9 of :rfc:`RFC7644 §3.12 <7644#section-3.12>`
+defines ``noTarget`` for a filter that "yields no match". The filter selects the entries to
+write. It does not describe an entry to create. So the operation fails instead of silently doing
+nothing.
+
+Implementations differ here. UnboundID's SCIM 2 SDK, Apache SCIMple and scim-patch create the
+entry on ``add``. SCIM-SDK returns ``noTarget``, and so does WSO2 Charon unless the attribute is
+unassigned. `Errata 8097 <https://errata.rfc-editor.org/eid8097/>`_ describes the creation
+Microsoft Entra ID expects. It is held, with no corrected text. The editor of :rfc:`7644`
+`replied <https://mailarchive.ietf.org/arch/msg/scim/kQ8B5YiYjDdGj8rX441FcsbOH4Q>`_ that
+such an implementation is "going outside the spec at the cost of their interoperability".
+
+Entra sends such operations to fill an attribute it has not set yet. It sends ``add`` by
+default, and ``replace`` when its ``aadOptscim062020`` flag is set. For example, it targets
+``emails[type eq "work"].value`` on a user without a work email, and expects a new entry
+``{"type": "work", "value": ...}``.
+:attr:`ScimPolicy.UnmatchedPathFilter.create <scim2_models.ScimPolicy.UnmatchedPathFilter.create>`
+creates that entry for both operations. It only works with ``eq`` comparisons joined by ``and``.
+The new entry must still match the filter after the value is written, so that later operations
+with the same filter find it. See :doc:`../how-to/tolerate-a-nonconformant-peer`.
 
 What a remove selects
 ---------------------
@@ -164,9 +264,12 @@ Rejected paths
 Which ``scimType`` a rejected path answers follows Table 9 of
 :rfc:`RFC7644 §3.12 <7644#section-3.12>`. ``invalidPath`` covers "the ``path`` attribute was
 invalid or malformed (see Figure 7)". It answers a path the grammar refuses, and an attribute the
-model does not declare. Table 9 lists ``invalidFilter`` as applying to a "PATCH (Path Filter)",
+model does not declare, unless :attr:`ScimPolicy.unknown <scim2_models.ScimPolicy.unknown>` drops
+it. With ``ignore`` or ``keep``, an operation on an undeclared attribute changes
+nothing. Table 9 lists ``invalidFilter`` as applying to a "PATCH (Path Filter)",
 so it answers what goes wrong between the brackets. That covers an unknown sub-attribute, a
-comparison the attribute cannot take, and a selection over an attribute holding a single value.
+comparison the attribute cannot take, and a selection over an attribute holding a single value,
+under any policy.
 
 Building a patch rather than applying one
 -----------------------------------------

doc/explanation/policies.rst

@@ -43,16 +43,30 @@ Telling the two apart would take a notion of role, and a model has none — a re
 by the client that wrote it as readily as by the server that received it, so the context cannot
 stand in for one.
 
+Tolerances that need no setting
+-------------------------------
+
+A setting is worth its cost when a tolerance could confuse one payload with another. Some cases
+the specification does not cover carry no such risk, and scim2-models accepts them without a
+setting.
+
+A key of a PATCH value that is an attribute path, such as ``name.givenName``, is read as that
+path. Microsoft Entra ID and its SCIM Validator send such keys. A strict reading would reject them
+as unknown attributes. But :rfc:`RFC7643 §2.1 <7643#section-2.1>` forbids dots and colons in
+attribute names, so the key cannot mean anything else. A setting that rejects it by default would
+only break that client. :doc:`patch` describes how such a key is read.
+
 What a policy leaves alone
 --------------------------
 
-**PATCH paths stay strict.** An operation whose ``path`` names an attribute no model declares is
-refused with ``invalidPath``, whatever the policy says. Path resolution and unknown attributes are
-two separate mechanisms, and making them uniform would take a third. The default that would come
-out of it is the wrong one: a server would answer 200 to a modification it never applied, where
-:rfc:`RFC7644 §3.5.2 <7644#section-3.5.2>` asks for an error. Inside the body of a resource the
-trade is different, since dropping one unknown attribute still lands everything the peer and the
-model both knew.
+**PATCH filters stay strict.** Under :attr:`~scim2_models.ScimPolicy.Unknown.ignore` or
+:attr:`~scim2_models.ScimPolicy.Unknown.keep`, a PATCH operation on an attribute no model
+declares changes nothing, like an undeclared attribute in its value. A filter that
+compares such an attribute is still rejected with ``invalidFilter``, and a malformed path with
+``invalidPath``. In both cases, there is no attribute the policy could drop. Dropping an
+operation has a cost: the server returns 200 for a change it never applied, where
+:rfc:`RFC7644 §3.5.2 <7644#section-3.5.2>` asks for an error. A tolerant policy already accepts
+that cost for the body of a resource, and the strict default keeps the error.
 
 **Building a model in Python stays strict.** ``User(bogus=1)`` and ``user.bogus = 1`` raise under
 every policy. Pydantic only offers a hook for extra keys during validation, so a keyword argument

doc/explanation/scim-contexts.rst

@@ -17,12 +17,13 @@ SCIM declares rules on attributes rather than on whole Python models:
 
 ``required``
    Requires a value in creation and replacement requests. A partial PATCH request does not have
-   to repeat every required attribute.
+   to repeat every required attribute, but it cannot leave one unassigned.
 
 ``mutability``
    Controls whether a client may write an attribute. ``readOnly`` is removed from creation and
-   replacement payloads, while a PATCH targeting it is rejected. ``immutable`` needs the current
-   resource value, so replacement and PATCH enforce it as the change is applied.
+   replacement payloads. A PATCH path to it is rejected. A PATCH value that contains it
+   is rejected only if it changes the current value. ``immutable`` needs the current resource
+   value, so replacement and PATCH enforce it as the change is applied.
 
 ``returned``
    Controls response projection. ``always`` cannot be excluded, ``never`` cannot be returned,
@@ -55,8 +56,8 @@ Where rules are enforced
 Creation and replacement validation check required values. Response serialization applies
 ``returned`` and attribute projection. The replacement workflow calls
 :meth:`~scim2_models.Resource.replace` after parsing so it can compare immutable values with the
-stored resource. PATCH first validates the operation message, then checks immutable values while
-applying it.
+stored resource. PATCH first validates the operation message, then checks immutable, read-only and
+required values on the result of each operation.
 
 :doc:`../how-to/validate-and-serialize` presents the validation and serialization procedure for
 each resource operation.

doc/how-to/build-a-patch.rst

@@ -20,8 +20,9 @@ Pass the state the peer holds first, then the state it should hold:
    [{'op': 'replace', 'path': 'name.givenName', 'value': 'Babs'}]
 
 A complex attribute is compared sub-attribute by sub-attribute, and each one gets its own path.
-Targeting ``name`` as a whole would replace it entirely and drop what the operation does not
-carry.
+Per :rfc:`RFC7644 §3.5.2.3 <7644#section-3.5.2.3>`, a ``replace`` on ``name`` keeps the
+sub-attributes it does not carry. But some peers replace the whole attribute and drop them. One
+path per sub-attribute gives the same result on every peer.
 
 Leave alone what the wanted state does not name
 -----------------------------------------------