Skip to content
Open
10 changes: 10 additions & 0 deletions Documentation/RCU/whatisRCU.rst
Original file line number Diff line number Diff line change
Expand Up @@ -952,6 +952,16 @@ unfortunately any spinlock in a ``SLAB_TYPESAFE_BY_RCU`` object must be
initialized after each and every call to kmem_cache_alloc(), which renders
reference-free spinlock acquisition completely unsafe. Therefore, when
using ``SLAB_TYPESAFE_BY_RCU``, make proper use of a reference counter.
If using refcount_t, the specialized refcount_{add|inc}_not_zero_acquire()
and refcount_set_release() APIs should be used to ensure correct operation
ordering when verifying object identity and when initializing newly
allocated objects. Acquire fence in refcount_{add|inc}_not_zero_acquire()
ensures that identity checks happen *after* reference count is taken.
refcount_set_release() should be called after a newly allocated object is
fully initialized and release fence ensures that new values are visible
*before* refcount can be successfully taken by other users. Once
refcount_set_release() is called, the object should be considered visible
by other tasks.
(Those willing to use a kmem_cache constructor may also use locking,
including cache-friendly sequence locking.)

Expand Down
37 changes: 31 additions & 6 deletions Documentation/core-api/refcount-vs-atomic.rst
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,19 @@ Memory ordering guarantee changes:
* none (both fully unordered)


case 2) - increment-based ops that return no value
case 2) - non-"Read/Modify/Write" (RMW) ops with release ordering
-----------------------------------------------------------------

Function changes:

* atomic_set_release() --> refcount_set_release()

Memory ordering guarantee changes:

* none (both provide RELEASE ordering)


case 3) - increment-based ops that return no value
--------------------------------------------------

Function changes:
Expand All @@ -98,7 +110,7 @@ Memory ordering guarantee changes:

* none (both fully unordered)

case 3) - decrement-based RMW ops that return no value
case 4) - decrement-based RMW ops that return no value
------------------------------------------------------

Function changes:
Expand All @@ -110,7 +122,7 @@ Memory ordering guarantee changes:
* fully unordered --> RELEASE ordering


case 4) - increment-based RMW ops that return a value
case 5) - increment-based RMW ops that return a value
-----------------------------------------------------

Function changes:
Expand All @@ -126,7 +138,20 @@ Memory ordering guarantees changes:
result of obtaining pointer to the object!


case 5) - generic dec/sub decrement-based RMW ops that return a value
case 6) - increment-based RMW ops with acquire ordering that return a value
---------------------------------------------------------------------------

Function changes:

* atomic_inc_not_zero() --> refcount_inc_not_zero_acquire()
* no atomic counterpart --> refcount_add_not_zero_acquire()

Memory ordering guarantees changes:

* fully ordered --> ACQUIRE ordering on success


case 7) - generic dec/sub decrement-based RMW ops that return a value
---------------------------------------------------------------------

Function changes:
Expand All @@ -139,7 +164,7 @@ Memory ordering guarantees changes:
* fully ordered --> RELEASE ordering + ACQUIRE ordering on success


case 6) other decrement-based RMW ops that return a value
case 8) other decrement-based RMW ops that return a value
---------------------------------------------------------

Function changes:
Expand All @@ -154,7 +179,7 @@ Memory ordering guarantees changes:
.. note:: atomic_add_unless() only provides full order on success.


case 7) - lock-based RMW
case 9) - lock-based RMW
------------------------

Function changes:
Expand Down
106 changes: 106 additions & 0 deletions include/linux/refcount.h
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,15 @@
* The decrements dec_and_test() and sub_and_test() also provide acquire
* ordering on success.
*
* refcount_{add|inc}_not_zero_acquire() and refcount_set_release() provide
* acquire and release ordering for cases when the memory occupied by the
* object might be reused to store another object. This is important for the
* cases where secondary validation is required to detect such reuse, e.g.
* SLAB_TYPESAFE_BY_RCU. The secondary validation checks have to happen after
* the refcount is taken, hence acquire order is necessary. Similarly, when the
* object is initialized, all stores to its attributes should be visible before
* the refcount is set, otherwise a stale attribute value might be used by
* another task which succeeds in taking a refcount to the new object.
*/

#ifndef _LINUX_REFCOUNT_H
Expand Down Expand Up @@ -125,6 +134,31 @@ static inline void refcount_set(refcount_t *r, int n)
atomic_set(&r->refs, n);
}

/**
* refcount_set_release - set a refcount's value with release ordering
* @r: the refcount
* @n: value to which the refcount will be set
*
* This function should be used when memory occupied by the object might be
* reused to store another object -- consider SLAB_TYPESAFE_BY_RCU.
*
* Provides release memory ordering which will order previous memory operations
* against this store. This ensures all updates to this object are visible
* once the refcount is set and stale values from the object previously
* occupying this memory are overwritten with new ones.
*
* This function should be called only after new object is fully initialized.
* After this call the object should be considered visible to other tasks even
* if it was not yet added into an object collection normally used to discover
* it. This is because other tasks might have discovered the object previously
* occupying the same memory and after memory reuse they can succeed in taking
* refcount to the new object and start using it.
*/
static inline void refcount_set_release(refcount_t *r, int n)
{
atomic_set_release(&r->refs, n);
}

/**
* refcount_read - get a refcount's value
* @r: the refcount
Expand Down Expand Up @@ -177,6 +211,52 @@ static inline __must_check bool refcount_add_not_zero(int i, refcount_t *r)
return __refcount_add_not_zero(i, r, NULL);
}

static inline __must_check bool
__refcount_add_not_zero_acquire(int i, refcount_t *r, int *oldp)
{
int old = refcount_read(r);

do {
if (!old)
break;
} while (!atomic_try_cmpxchg_acquire(&r->refs, &old, old + i));

if (oldp)
*oldp = old;

if (unlikely(old < 0 || old + i < 0))
refcount_warn_saturate(r, REFCOUNT_ADD_NOT_ZERO_OVF);

return old;
}

/**
* refcount_add_not_zero_acquire - add a value to a refcount with acquire ordering unless it is 0
*
* @i: the value to add to the refcount
* @r: the refcount
*
* Will saturate at REFCOUNT_SATURATED and WARN.
*
* This function should be used when memory occupied by the object might be
* reused to store another object -- consider SLAB_TYPESAFE_BY_RCU.
*
* Provides acquire memory ordering on success, it is assumed the caller has
* guaranteed the object memory to be stable (RCU, etc.). It does provide a
* control dependency and thereby orders future stores. See the comment on top.
*
* Use of this function is not recommended for the normal reference counting
* use case in which references are taken and released one at a time. In these
* cases, refcount_inc_not_zero_acquire() should instead be used to increment a
* reference count.
*
* Return: false if the passed refcount is 0, true otherwise
*/
static inline __must_check bool refcount_add_not_zero_acquire(int i, refcount_t *r)
{
return __refcount_add_not_zero_acquire(i, r, NULL);
}

static inline void __refcount_add(int i, refcount_t *r, int *oldp)
{
int old = atomic_fetch_add_relaxed(i, &r->refs);
Expand Down Expand Up @@ -234,6 +314,32 @@ static inline __must_check bool refcount_inc_not_zero(refcount_t *r)
return __refcount_inc_not_zero(r, NULL);
}

static inline __must_check bool __refcount_inc_not_zero_acquire(refcount_t *r, int *oldp)
{
return __refcount_add_not_zero_acquire(1, r, oldp);
}

/**
* refcount_inc_not_zero_acquire - increment a refcount with acquire ordering unless it is 0
* @r: the refcount to increment
*
* Similar to refcount_inc_not_zero(), but provides acquire memory ordering on
* success.
*
* This function should be used when memory occupied by the object might be
* reused to store another object -- consider SLAB_TYPESAFE_BY_RCU.
*
* Provides acquire memory ordering on success, it is assumed the caller has
* guaranteed the object memory to be stable (RCU, etc.). It does provide a
* control dependency and thereby orders future stores. See the comment on top.
*
* Return: true if the increment was successful, false otherwise
*/
static inline __must_check bool refcount_inc_not_zero_acquire(refcount_t *r)
{
return __refcount_inc_not_zero_acquire(r, NULL);
}

static inline void __refcount_inc(refcount_t *r, int *oldp)
{
__refcount_add(1, r, oldp);
Expand Down
9 changes: 9 additions & 0 deletions include/linux/slab.h
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,15 @@
* rcu_read_lock before reading the address, then rcu_read_unlock after
* taking the spinlock within the structure expected at that address.
*
* Note that object identity check has to be done *after* acquiring a
* reference, therefore user has to ensure proper ordering for loads.
* Similarly, when initializing objects allocated with SLAB_TYPESAFE_BY_RCU,
* the newly allocated object has to be fully initialized *before* its
* refcount gets initialized and proper ordering for stores is required.
* refcount_{add|inc}_not_zero_acquire() and refcount_set_release() are
* designed with the proper fences required for reference counting objects
* allocated with SLAB_TYPESAFE_BY_RCU.
*
* Note that it is not possible to acquire a lock within a structure
* allocated with SLAB_TYPESAFE_BY_RCU without first acquiring a reference
* as described above. The reason is that SLAB_TYPESAFE_BY_RCU pages
Expand Down
6 changes: 4 additions & 2 deletions include/net/inet_timewait_sock.h
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,10 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk,
struct inet_timewait_death_row *dr,
const int state);

void inet_twsk_hashdance(struct inet_timewait_sock *tw, struct sock *sk,
struct inet_hashinfo *hashinfo);
void inet_twsk_hashdance_schedule(struct inet_timewait_sock *tw,
struct sock *sk,
struct inet_hashinfo *hashinfo,
int timeo);

void __inet_twsk_schedule(struct inet_timewait_sock *tw, int timeo,
bool rearm);
Expand Down
2 changes: 1 addition & 1 deletion net/core/filter.c
Original file line number Diff line number Diff line change
Expand Up @@ -7555,7 +7555,7 @@ BPF_CALL_3(bpf_sk_assign, struct sk_buff *, skb, struct sock *, sk, u64, flags)
if (sk_unhashed(sk))
return -EOPNOTSUPP;
if (sk_is_refcounted(sk) &&
unlikely(!refcount_inc_not_zero(&sk->sk_refcnt)))
unlikely(!refcount_inc_not_zero_acquire(&sk->sk_refcnt)))
return -ENOENT;

skb_orphan(skb);
Expand Down
3 changes: 1 addition & 2 deletions net/dccp/minisocks.c
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,10 @@ void dccp_time_wait(struct sock *sk, int state, int timeo)
* we complete the initialization.
*/
local_bh_disable();
inet_twsk_schedule(tw, timeo);
/* Linkage updates.
* Note that access to tw after this point is illegal.
*/
inet_twsk_hashdance(tw, sk, &dccp_hashinfo);
inet_twsk_hashdance_schedule(tw, sk, &dccp_hashinfo, timeo);
local_bh_enable();
} else {
/* Sorry, if we're out of memory, just CLOSE this
Expand Down
20 changes: 12 additions & 8 deletions net/ipv4/inet_connection_sock.c
Original file line number Diff line number Diff line change
Expand Up @@ -1038,8 +1038,8 @@ static void reqsk_timer_handler(struct timer_list *t)
* hold another count to prevent use-after-free and
* call reqsk_put() just before return.
*/
refcount_set(&nreq->rsk_refcnt, 2 + 1);
timer_setup(&nreq->rsk_timer, reqsk_timer_handler, TIMER_PINNED);
refcount_set_release(&nreq->rsk_refcnt, 2 + 1);
reqsk_queue_migrated(&inet_csk(nsk)->icsk_accept_queue, req);

req = nreq;
Expand Down Expand Up @@ -1094,8 +1094,8 @@ static void reqsk_timer_handler(struct timer_list *t)
return;

if (!inet_ehash_insert(req_to_sk(nreq), req_to_sk(oreq), NULL)) {
/* delete timer */
__inet_csk_reqsk_queue_drop(sk_listener, nreq, true);
if (timer_delete(&nreq->rsk_timer))
reqsk_put(nreq);
goto no_ownership;
}

Expand All @@ -1114,10 +1114,14 @@ static void reqsk_timer_handler(struct timer_list *t)
*/
if (nreq) {
__NET_INC_STATS(net, LINUX_MIB_TCPMIGRATEREQFAILURE);
/* The timer was not armed, drop its reference here. */
reqsk_put(nreq);
Comment thread
sourcery-ai[bot] marked this conversation as resolved.
Resolved
no_ownership:
reqsk_migrate_reset(nreq);
reqsk_queue_removed(queue, nreq);
__reqsk_free(nreq);
/* Drop the temporary reference and the one reserved for ehash. */
reqsk_put(nreq);
reqsk_put(nreq);
Comment thread
Copilot marked this conversation as resolved.
Resolved
}

drop:
Expand Down Expand Up @@ -1388,7 +1392,7 @@ struct sock *inet_csk_complete_hashdance(struct sock *sk, struct sock *child,
goto child_put;
}

refcount_set(&nreq->rsk_refcnt, 1);
refcount_set_release(&nreq->rsk_refcnt, 1);
if (inet_csk_reqsk_queue_add(sk, nreq, child)) {
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMIGRATEREQSUCCESS);
reqsk_migrate_reset(req);
Expand All @@ -1398,7 +1402,7 @@ struct sock *inet_csk_complete_hashdance(struct sock *sk, struct sock *child,

__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMIGRATEREQFAILURE);
reqsk_migrate_reset(nreq);
__reqsk_free(nreq);
reqsk_put(nreq);
} else if (inet_csk_reqsk_queue_add(sk, req, child)) {
return child;
}
Expand Down Expand Up @@ -1442,7 +1446,7 @@ void inet_csk_listen_stop(struct sock *sk)
if (nsk) {
nreq = inet_reqsk_clone(req, nsk);
if (nreq) {
refcount_set(&nreq->rsk_refcnt, 1);
refcount_set_release(&nreq->rsk_refcnt, 1);

rcu_read_lock();
if (inet_csk_reqsk_queue_add(nsk, nreq, child)) {
Expand All @@ -1454,7 +1458,7 @@ void inet_csk_listen_stop(struct sock *sk)
__NET_INC_STATS(sock_net(nsk),
LINUX_MIB_TCPMIGRATEREQFAILURE);
reqsk_migrate_reset(nreq);
__reqsk_free(nreq);
reqsk_put(nreq);
}
rcu_read_unlock();

Expand Down
7 changes: 4 additions & 3 deletions net/ipv4/inet_hashtables.c
Original file line number Diff line number Diff line change
Expand Up @@ -511,10 +511,11 @@ struct sock *__inet_lookup_established(struct net *net,
if (sk->sk_hash != hash)
continue;
if (likely(inet_match(net, sk, acookie, ports, dif, sdif))) {
if (unlikely(!refcount_inc_not_zero(&sk->sk_refcnt)))
if (unlikely(!refcount_inc_not_zero_acquire(&sk->sk_refcnt)))
goto out;
if (unlikely(!inet_match(net, sk, acookie,
ports, dif, sdif))) {
if (unlikely(hlist_nulls_unhashed_lockless(&sk->sk_nulls_node) ||
!inet_match(net, sk, acookie, ports, dif,
sdif))) {
sock_gen_put(sk);
goto begin;
}
Expand Down
Loading