Introduction
In 2022, we are likely to see experimental Rust programming language support in the Linux kernel become mainstream. On the morning of December 6, 2021, an updated patch was issued to introduce the initial support and infrastructure for handling Rust in the kernel.
This update includes:
- Upgraded to the latest Stable compiler and Rust 2021 edition. So you can get rid of const_fn_transmuteļ¼const_panic,const_unreachable_unchecked,core_panic and try_reserve these previously unstable features. Unstable characteristic wish list [1].
- Customize core and alloc. Added a more modular option to alloc to disable some features they don't need: no_rc and no_sync is mainly added for upstream Rust projects.
- Stricter code, documentation, and new lint.
- Abstraction and driver updates. Add the abstraction of sequence lock and power management callback, io memory (readX/writeX), irq chip and advanced stream handler, gpio chip (including irq chip), device, amba device and driver, and certificate. In addition, the Ref (refcount_t support) object is improved and simplified, and all instances of Rust's arc are replaced with it. Arc and Rc are completely removed from alloc crit.
From now on, the Rust for linux team will begin to submit patches regularly, about every two weeks.
In addition to the support from Arm, Google and Microsoft, the team received another letter from red hat this time: red hat is also very interested in using Rust for kernel work that red hat is considering.
- v2 patch: https://lore.kernel.org/lkml/20211206140313.5653-1-ojeda@kernel.org/[2]
- https://www.phoronix.com/scan.php?page=news_item&px=Rust-For-Linux-v2[3]
- Kernel crash document [4]
Why should Ref be introduced instead of Arc
In the kernel crate of Rust for Linux, Arc was used before, but now Ref is used. Check the relevant PR Trust: update Ref to use the kernel's refcount_ T [5], we can understand that there are two main reasons:
- Maximize the use of existing C code and eliminate Panic (Panic). Refcount is already implemented in the kernel_ t. Moreover, when it exceeds the threshold of reference count, it returns the maximum value (saturation addition) instead of Panic (abort). For this reason, 'RBTree' (mangrove) [6] is also used instead of 'btreemap' [7].
- Weak references are not required.
"Arc has a MAX_REFCOUNT limit, which is the size of isize:: Max as use. If the reference count addition exceeds this size, it will overflow, and then Panic(abort) will occur.
Therefore, the difference between the final Ref and Arc is:
- Ref is a kernel based refcount_t to support
- It does not support weak references, so the size is reduced by half
- When it exceeds the threshold, it saturates rather than aborts the reference count
- It does not provide get_mut method, so the reference count object is Pin.
Ref source code analysis
Next, analyze the implementation of 'ref' [8].
Ref structure
Ref structure is defined as follows:
/// A reference-counted pointer to an instance of `T`.
///
/// The reference count is incremented when new instances of [`Ref`] are created, and decremented
/// when they are dropped. When the count reaches zero, the underlying `T` is also dropped.
///
/// # Invariants
///
/// The reference count on an instance of [`Ref`] is always non-zero.
/// The object pointed to by [`Ref`] is always pinned.
pub struct Ref<T: ?Sized> {
ptr: NonNull<RefInner<T>>,
_p: PhantomData<RefInner<T>>,
}
It maintains an Invariants: the reference count Ref is always a non-zero instance, and the objects referenced by Ref are always Pin (immovable).
Nonnull < T > is used in this structure instead of * mut T. covariant rather than invariant is required here. Refer to the following example:
use std::ptr::NonNull;
struct Ref<T: ?Sized> {
x: NonNull<T>,
// x: *mut T, / / if you change to * mut T, the compilation will not pass
}
fn take<'a>(r: Ref<&'a u32>, y: &'a u32) {}
fn give() -> Ref<&'static u32> { todo!() }
fn test() {
let y = 1;
// Covariant, which can pass in the take function of ref < & 'a U32 >, and can also receive the parameters of ref < &' static U32 >, because 'static:' a can accept both subtypes and parent types
take(give(), &y);
}
NonNull is a covariant version of * mut T, and also represents a non null pointer, which means that the reference count object is always non null, because it will be released when the count is zero.
PhatomData is used here for Drop check. It means that Ref type has refiner < T > and refiner < T > can be dropped when Ref is dropped.
RefInner structure
Let's look at the RefInner structure:
#[repr(C)]
struct RefInner<T: ?Sized> {
refcount: Opaque<bindings::refcount_t>,
data: T,
}
RefInner contains the reference count structure refcount implemented by C language in the kernel_ t. This is to reuse C code.
Among them, 'Opaque' type [9] is a package type built in kernel crite and provided specifically for dealing with C, which is defined as follows:
pub struct Opaque<T>(MaybeUninit<UnsafeCell<T>>);
impl<T> Opaque<T> {
/// Creates a new opaque value.
pub fn new(value: T) -> Self {
Self(MaybeUninit::new(UnsafeCell::new(value)))
}
/// Creates an uninitialised value.
pub fn uninit() -> Self {
Self(MaybeUninit::uninit())
}
/// Returns a raw pointer to the opaque data.
pub fn get(&self) -> *mut T {
UnsafeCell::raw_get(self.0.as_ptr())
}
}
The Opaque type means that you never need Rust code to interpret an FFi object. Therefore, in order to use the reference count structure that already exists in the kernel, here we use Opaque < bindings:: refcount_ t> Type.
About refcount_t
Refcount defined in the Linux kernel_ T structure is defined as follows:
// from: https://github.com/torvalds/linux/blob/master/tools/include/linux/refcount.h
typedef struct refcount_struct {
atomic_t refs;
} refcount_t;
refcount_ The goal of the T API is to provide a minimum API for the reference counter of the implementation object. Although atomic operations are used internally, some refcounts_* () and atomic_* The () function has many differences in memory order guarantee.
refcount_t in 2018, there was a security vulnerability of reference count overflow, that is, when the reference count reaches the maximum value, if you add another one, the reference count will return to zero. Therefore, the referenced object will be released incorrectly. In this way, it becomes a UAF (use after free) vulnerability, which is easy to be exploited.
So now refcount_t reference count detection added:
// from: https://github.com/torvalds/linux/blob/master/tools/include/linux/refcount.h#L69
static inline __refcount_check
bool refcount_inc_not_zero(refcount_t *r)
{
unsigned int old, new, val = atomic_read(&r->refs);
for (;;) {
new = val + 1;
if (!val)
return false;
if (unlikely(!new))
return true;
old = atomic_cmpxchg_relaxed(&r->refs, val, new);
if (old == val)
break;
val = old;
}
REFCOUNT_WARN(new == UINT_MAX, "refcount_t: saturated; leaking memory.\n");
return true;
}
saturated addition is used when the maximum reference count is reached, that is, the maximum value is returned instead of zero. Note that the compare and swap atomic operation is used here and the memory order is not provided (using relaxed).
Some trait s implemented for Ref
In order for ref < T > to have some behavior similar to arc < T >, some built-in trait s are implemented for it.
// This is to allow [`Ref`] (and variants) to be used as the type of `self`.
impl<T: ?Sized> core::ops::Receiver for Ref<T> {}
// This is to allow [`RefBorrow`] (and variants) to be used as the type of `self`.
impl<T: ?Sized> core::ops::Receiver for RefBorrow<'_, T> {}
// This is to allow coercion from `Ref<T>` to `Ref<U>` if `T` can be converted to the
// dynamically-sized type (DST) `U`.
impl<T: ?Sized + Unsize<U>, U: ?Sized> core::ops::CoerceUnsized<Ref<U>> for Ref<T> {}
// This is to allow `Ref<U>` to be dispatched on when `Ref<T>` can be coerced into `Ref<U>`.
impl<T: ?Sized + Unsize<U>, U: ?Sized> core::ops::DispatchFromDyn<Ref<U>> for Ref<T> {}
// SAFETY: It is safe to send `Ref<T>` to another thread when the underlying `T` is `Sync` because
// it effectively means sharing `&T` (which is safe because `T` is `Sync`); additionally, it needs
// `T` to be `Send` because any thread that has a `Ref<T>` may ultimately access `T` directly, for
// example, when the reference count reaches zero and `T` is dropped.
unsafe impl<T: ?Sized + Sync + Send> Send for Ref<T> {}
// SAFETY: It is safe to send `&Ref<T>` to another thread when the underlying `T` is `Sync` for
// the same reason as above. `T` needs to be `Send` as well because a thread can clone a `&Ref<T>`
// into a `Ref<T>`, which may lead to `T` being accessed by the same reasoning as above.
unsafe impl<T: ?Sized + Sync + Send> Sync for Ref<T> {}
It can be seen from the above code that the trait s used are:
- `core::ops::Receiver`[10]: it is a receiver_trait feature, which indicates that a structure can be used as a method receiver and does not need an array_ self_ Types property. Some smart pointers in the standard library implement the trait, such as box < T > / arc < T > / RC < T > / & T / pin < p >.
- `core::ops::CoerceUnsized`[11]: it is also a coerce_unsized feature, which means converting Size type to DST type.
- `core::ops::DispatchFromDyn`[12]: it is also an unstable feature (dispatch_from_dyn features), which is used to check object security (dynamic security dyn safe). A type that implements DispatchFromDyn can safely be used as a self type in an object safe method.
- Send/Sync, a stable feature in Rust, is used to mark types that can be safely passed and shared between threads.
Now that these trait s are implemented for ref < T >, ref < T > has the corresponding behavior. Basically, the behavior of ref < T > is similar to that of arc < T > except for the differences mentioned above.
Reference count management
Because ref < T > is a reusable kernel C code, you only need to implement the corresponding trait for reference count management.
For example, Clone should increase the reference count automatically, while Drop should decrease the reference count automatically. So let's look at the two implementations separately.
// Implement Clone trait
impl<T: ?Sized> Clone for Ref<T> {
fn clone(&self) -> Self {
// INVARIANT: C `refcount_inc` saturates the refcount, so it cannot overflow to zero.
// SAFETY: By the type invariant, there is necessarily a reference to the object, so it is
// safe to increment the refcount.
unsafe { bindings::refcount_inc(self.ptr.as_ref().refcount.get()) };
// SAFETY: We just incremented the refcount. This increment is now owned by the new `Ref`.
unsafe { Self::from_inner(self.ptr) }
}
}
The implementation of Clone trait is very simple, directly through bindings::refcount_inc to call refcount in the kernel_ Auto increment method refcount of T_ Inc.
Because refcount_inc already has reference count overflow detection and uses saturation addition, so there is no need to worry about zeroing.
// Implement Drop trait
impl<T: ?Sized> Drop for Ref<T> {
fn drop(&mut self) {
// SAFETY: By the type invariant, there is necessarily a reference to the object. We cannot
// touch `refcount` after it's decremented to a non-zero value because another thread/CPU
// may concurrently decrement it to zero and free it. It is ok to have a raw pointer to
// freed/invalid memory as long as it is never dereferenced.
let refcount = unsafe { self.ptr.as_ref() }.refcount.get();
// INVARIANT: If the refcount reaches zero, there are no other instances of `Ref`, and
// this instance is being dropped, so the broken invariant is not observable.
// SAFETY: Also by the type invariant, we are allowed to decrement the refcount.
let is_zero = unsafe { bindings::refcount_dec_and_test(refcount) };
if is_zero {
// The count reached zero, we must free the memory.
// SAFETY: This thread holds the only remaining reference to `self`, so it is safe to
// get a mutable reference to it.
let inner = unsafe { self.ptr.as_mut() };
let layout = Layout::for_value(inner);
// SAFETY: The value stored in inner is valid.
unsafe { core::ptr::drop_in_place(inner) };
// SAFETY: The pointer was initialised from the result of a call to `alloc`.
unsafe { dealloc(self.ptr.cast().as_ptr(), layout) };
}
}
}
The drop trail is also implemented directly through bindings::refcount_dec_and_test calls the kernel refcount_dec_and_test function, which also contains reference count overflow check. However, memory needs to be freed when the reference count is reset to zero.
Note that the implementation of the above two trait s, Clone and Drop, is a classic example of Unsafe Rust abstracted as Safe Rust, mainly the Safety annotation, considering the security boundary and explaining it.
Create a new reference count object
Next, you need to focus on how ref < T > creates a new reference count object.
impl<T> Ref<T> {
/// Constructs a new reference counted instance of `T`.
pub fn try_new(contents: T) -> Result<Self> {
let layout = Layout::new::<RefInner<T>>();
// SAFETY: The layout size is guaranteed to be non-zero because `RefInner` contains the
// reference count.
let inner = NonNull::new(unsafe { alloc(layout) })
.ok_or(Error::ENOMEM)?
.cast::<RefInner<T>>();
// INVARIANT: The refcount is initialised to a non-zero value.
let value = RefInner {
// SAFETY: Just an FFI call that returns a `refcount_t` initialised to 1.
refcount: Opaque::new(unsafe { bindings::REFCOUNT_INIT(1) }),
data: contents,
};
// SAFETY: `inner` is writable and properly aligned.
unsafe { inner.as_ptr().write(value) };
// SAFETY: We just created `inner` with a reference count of 1, which is owned by the new
// `Ref` object.
Ok(unsafe { Self::from_inner(inner) })
}
}
This try_ The new method uses the core::alloc::Layout structure to define the memory layout.
Allocate new memory through NonNull::new and customized core::alloc::alloc functions, convert it to refiner < T > > type, and use bindings::REFCOUNT_INIT calls the kernel C function to initialize it to 1. The customized core::alloc module will be synchronized to the t rust core in the future.
Where Error::ENOMEM represents OOM error. In ` kernel / error RS ` [13] defines many errors corresponding to kernel error codes.
Many error codes are defined using integers in the Linux kernel. In the kernel rate, NewType mode is used to encapsulate them instead of using integer error codes directly:
macro_rules! declare_err {
($err:tt) => {
pub const $err: Self = Error(-(bindings::$err as i32));
};
($err:tt, $($doc:expr),+) => {
$(
#[doc = $doc]
)*
pub const $err: Self = Error(-(bindings::$err as i32));
};
}
#[derive(Clone, Copy, PartialEq, Eq)]
pub struct Error(c_types::c_int);
impl Error {
declare_err!(EPERM, "Operation not permitted.");
declare_err!(ENOENT, "No such file or directory.");
declare_err!(ESRCH, "No such process.");
declare_err!(ENOMEM, "Out of memory.");
// ...
}
Construct ref < T > from existing refinner < T >
Try above_ As you can see in the new method, the last step is to use from_ The inner method constructs a bare pointer as the final ref < T >. And it is an internal method, not an open API.
Note that it is an unsafe method, because the caller is required to ensure that the inner pointer is valid and non empty. The document comments on this point are also clear.
impl<T: ?Sized> Ref<T> {
/// Constructs a new [`Ref`] from an existing [`RefInner`].
///
/// # Safety
///
/// The caller must ensure that `inner` points to a valid location and has a non-zero reference
/// count, one of which will be owned by the new [`Ref`] instance.
unsafe fn from_inner(inner: NonNull<RefInner<T>>) -> Self {
// INVARIANT: By the safety requirements, the invariants hold.
Ref {
ptr: inner,
_p: PhantomData,
}
}
}
RefBorrow<T>
There is no variable borrowing from the underlying reference count structure, but there is an immutable borrowing and the lifecycle needs to be maintained manually.
/// A borrowed [`Ref`] with manually-managed lifetime.
///
/// # Invariants
///
/// There are no mutable references to the underlying [`Ref`], and it remains valid for the lifetime
/// of the [`RefBorrow`] instance.
pub struct RefBorrow<'a, T: ?Sized + 'a> {
inner: NonNull<RefInner<T>>,
_p: PhantomData<&'a ()>,
}
impl<T: ?Sized> Clone for RefBorrow<'_, T> {
fn clone(&self) -> Self {
*self
}
}
impl<T: ?Sized> Copy for RefBorrow<'_, T> {}
RefBorrow structure uses phantomdata < &'a() > to hold life cycle parameters and implement Copy trait for them. Its behavior is similar to that of ordinary immutable references.
Then implement an as for ref < T >_ ref_ Using the borrow method, you can get refborrow < T > from ref < T >.
impl<T> Ref<T> {
/// Returns a [`RefBorrow`] from the given [`Ref`].
///
/// This is useful when the argument of a function call is a [`RefBorrow`] (e.g., in a method
/// receiver), but we have a [`Ref`] instead. Getting a [`RefBorrow`] is free when optimised.
#[inline]
pub fn as_ref_borrow(&self) -> RefBorrow<'_, T> {
// SAFETY: The constraint that lifetime of the shared reference must outlive that of
// the returned `RefBorrow` ensures that the object remains alive.
unsafe { RefBorrow::new(self.ptr) }
}
}
In fact, according to the Rust naming convention, here as_ref_ Change border to as_ref is better. But here it is_ Ref is also useful:
impl<T: ?Sized> AsRef<T> for Ref<T> {
fn as_ref(&self) -> &T {
// SAFETY: By the type invariant, there is necessarily a reference to the object, so it is
// safe to dereference it.
unsafe { &self.ptr.as_ref().data }
}
}
To pass as_ The ref method gets & T from ref < T >.
Then implement Deref trait for refborrow < T > or get & T from refborrow < T >.
impl<T: ?Sized> Deref for RefBorrow<'_, T> {
type Target = T;
fn deref(&self) -> &Self::Target {
// SAFETY: By the type invariant, the underlying object is still alive with no mutable
// references to it, so it is safe to create a shared reference.
unsafe { &self.inner.as_ref().data }
}
}
Unique reference type uniqueref < T >
In addition to ref < T >, a uniqueref < T > type is implemented. As the name suggests, this type represents the case where there is only one reference count.
pub struct UniqueRef<T: ?Sized> {
inner: Ref<T>,
}
impl<T> UniqueRef<T> {
/// Tries to allocate a new [`UniqueRef`] instance.
pub fn try_new(value: T) -> Result<Self> {
Ok(Self {
// INVARIANT: The newly-created object has a ref-count of 1.
inner: Ref::try_new(value)?,
})
}
/// Tries to allocate a new [`UniqueRef`] instance whose contents are not initialised yet.
pub fn try_new_uninit() -> Result<UniqueRef<MaybeUninit<T>>> {
Ok(UniqueRef::<MaybeUninit<T>> {
// INVARIANT: The newly-created object has a ref-count of 1.
inner: Ref::try_new(MaybeUninit::uninit())?,
})
}
}
The Clone and Drop trait s are not implemented for it, so it can only hold a unique reference. The introduction of this type may provide more convenience for kernel development.
other
Ref < T > also implements other trait s, such as From/TryFrom, which can be converted from bare pointer to ref < T >.
One thing worth noting is:
impl<T> Ref<T> {
/// Deconstructs a [`Ref`] object into a raw pointer.
///
/// It can be reconstructed once via [`Ref::from_raw`].
pub fn into_raw(obj: Self) -> *const T {
let ret = &*obj as *const T;
core::mem::forget(obj);
ret
}
}
When converting ref < T > into a bare pointer, pay attention to using core::mem::forget(obj) to avoid calling the Drop of obj, otherwise the reference count will be reduced and problems will be caused.
Summary
You can learn a lot of Unsafe Rust related skills from the Rust for Linux source code, especially some good practices dealing with C language. If you are interested, you can also learn some contents related to the Linux kernel and make some preparations for writing Linux kernel drivers for Rust in the future.
reference material
[1] Unstable feature wish list: https://github.com/Rust-for-Linux/linux/issues/2
[2]v2 patch: https://lore.kernel.org/lkml/20211206140313.5653-1-ojeda@kernel.org/: https://lore.kernel.org/lkml/20211206140313.5653-1-ojeda@kernel.org/
[3]https://www.phoronix.com/scan.php?page=news_item&px=Rust-For-Linux-v2: https://www.phoronix.com/scan.php?page=news_item&px=Rust-For-Linux-v2
[4] Kernel crash documentation: https://rust-for-linux.github.io/docs/kernel/
[5]rust: update Ref to use the kernel's refcount_t: https://github.com/Rust-for-Linux/linux/pull/377
[6]RBTree: https://github.com/Rust-for-Linux/linux/blob/rust/rust/kernel/rbtree.rs
[7] Replaced BTreeMap: https://github.com/Rust-for-Linux/linux/pull/403
[8]Ref: https://github.com/Rust-for-Linux/linux/blob/rust/rust/kernel/sync/arc.rs#L42
[9] Opera type: https://github.com/Rust-for-Linux/linux/blob/rust/rust/kernel/types.rs#L277
[10]core::ops::Receiver: https://github.com/rust-lang/rust/blob/master/library/core/src/ops/deref.rs#L191
[11]core::ops::CoerceUnsized: https://github.com/rust-lang/rust/blob/master/library/core/src/ops/unsize.rs#L36
[12]core::ops::DispatchFromDyn: https://github.com/rust-lang/rust/blob/master/library/core/src/ops/unsize.rs#L117
[13]kernel/error.rs: https://github.com/Rust-for-Linux/linux/blob/rust/rust/kernel/error.rs#L64