numapp Namespace Reference

Namespaces
namespace	ll
	Defines lowlevel functions that should not be used directly.

Classes
class	Bitmask
	Generic bitmask. More...
class	CpuAffinity
	Create CPU affinity and apply to current thread. More...
class	Cpumask
	Type-safe CPU mask. More...
class	DynamicScheduler
	Normal non-realtime scheduler that use dynamic priority (nice value). More...
class	HugePageResource
	Polymorphic memory resource allocating huge pages with specified NUMA policy. More...
class	HugePageSize
	Describes a huge page size and maintains the invariant that page size is an integral power of 2, compatible with mmap() expectations. More...
class	IdleScheduler
	Represents SCHED_IDLE scheduler policy. More...
struct	IsFlagEnum
	Trait type that should be specialized for an enumeration to enable use of bitwise operators. More...
class	LockResource
	Lock memory allocated from upstream memory resource using specified LockFlag. More...
class	MemPolicy
	Class representing a memory policy that can be modified and used to apply to the current thread or a memory range. More...
struct	Nodemask
	Type-safe NUMA node mask. More...
class	NumaPolicies
	Combines the the available NUMA policy types in one object. More...
class	PageResource
	Polymorphic memory resource allocating full system pages with specified NUMA policy. More...
class	Scheduler
	A sum-type of all supported schedulers. More...
class	ScopedMemPolicy
	Applies memory policy to this thread at construction and reverts to previous at destruction. More...
class	StaticScheduler
	Static priority scheduler (real-time). More...

Typedefs
using	NumaBitmaskPtr = std::unique_ptr<struct bitmask, void (*)(struct bitmask*)>
	Lowlevel bitmask type.
using	SchedulerVariant = std::variant<DynamicScheduler, StaticScheduler, IdleScheduler>
	Variant of possible schedulers.

Enumerations
enum class	HugePagePreset : std::size_t {   HugePagePreset::Huge8k = 8 * 1024 , HugePagePreset::Huge16k = 16 * 1024 , HugePagePreset::Huge64k = 64 * 1024 , HugePagePreset::Huge256k = 256 * 1024 ,   HugePagePreset::Huge1M = 1 * 1024 * 1024 , HugePagePreset::Huge2M = 2 * 1024 * 1024 , HugePagePreset::Huge4M = 4 * 1024 * 1024 , HugePagePreset::Huge16M = 16 * 1024 * 1024 ,   HugePagePreset::Huge256M = 256 * 1024 * 1024 , HugePagePreset::Huge1G = 1ul * 1024 * 1024 * 1024 }
	Preset huge page sizes. More...
enum class	MemPolicyFlag : unsigned
	Flag that modify the behaviour of applying a memory policy to a range of memory.

Functions
std::ostream &	FormatBitmask (std::ostream &os, bitmask const *mask, int min_bits)
	Formats mask and inserts it to os.
template<class Class, class F>
void	ForEach (Bitmask< Class > const &bitmask, F &&f, bool value=true)
	Invoke function for each bit in mask that matches value.
template<class Class>
std::ostream &	FormatBitmask (std::ostream &os, Bitmask< Class > const &mask, int min_bits)
	Formats mask and inserts it to os.
std::ostream &	operator<< (std::ostream &os, Cpumask const &mask)
	Formats mask and inserts it to os.
auto	EncodeMmapFlags (HugePageSize page_size) noexcept -> int
	Encodes page size into bit representation expected by mmap().
std::ostream &	operator<< (std::ostream &os, Nodemask const &mask)
	Formats mask and inserts it to os.
bool	NumaAvailable () noexcept
	Query whether system has NUMA support.
std::ostream &	operator<< (std::ostream &os, NumaPolicies const &policies)
	Formats policies and inserts it to os.
std::ostream &	operator<< (std::ostream &os, StaticScheduler::Policy const &policy)
	Formats policy and inserts it to os.
std::ostream &	operator<< (std::ostream &os, SchedulerVariant const &scheduler)
	Formats scheduler and inserts it to os.
template<class Class, class F>
void	ForEach (Bitmask< Class > const &bitmask, F &&f, bool value=true)
	Invoke function for each bit in mask that matches value.
template<class Class>
std::ostream &	FormatBitmask (std::ostream &os, Bitmask< Class > const &mask, int min_bits)
	Formats mask and inserts it to os.
std::ostream &	FormatBitmask (std::ostream &os, bitmask const *mask, int min_bits)
	Formats mask and inserts it to os.
std::ostream &	operator<< (std::ostream &os, CpuAffinity const &affinity)
	Formats affinity and inserts it to os.
std::ostream &	operator<< (std::ostream &os, MemPolicy::Mode const &mode)
	Formats mode and inserts it to os.
std::ostream &	operator<< (std::ostream &os, MemPolicy const &mempolicy)
	Formats mempolicy and inserts it to os.
std::ostream &	operator<< (std::ostream &os, IdleScheduler const &scheduler)
	Formats scheduler and inserts it to os.
std::ostream &	operator<< (std::ostream &os, StaticScheduler const &scheduler)
	Formats scheduler and inserts it to os.
std::ostream &	operator<< (std::ostream &os, DynamicScheduler const &scheduler)
	Formats scheduler and inserts it to os.
std::ostream &	operator<< (std::ostream &os, DynamicScheduler const &policy)
	Formats policy and inserts it to os.
std::ostream &	operator<< (std::ostream &os, Scheduler const &scheduler)
	Formats scheduler and inserts it to os.
Bitwise operators for enabled enums
Convenience function to compose bit flags using bitwise OR and AND operators. Example: using numapp::operator|; // Bring in numapp::operator| in this scope MyEnum f = MyEnum::A | MyEnum::B; f &= MyEnum::A; MyFunc(f); MyFunc(MyEnum::A | MyEnum::B);
template<class Enum, typename = typename std::enable_if<IsFlagEnum<Enum>::value>::type>
constexpr Enum	operator| (Enum lhs, Enum rhs) noexcept
template<class Enum, typename = typename std::enable_if<IsFlagEnum<Enum>::value>::type>
constexpr Enum &	operator|= (Enum &lhs, Enum rhs) noexcept
	Sets lhs with logical OR between lhs and rhs.
template<class Enum, typename = typename std::enable_if<IsFlagEnum<Enum>::value>::type>
constexpr Enum	operator& (Enum lhs, Enum rhs) noexcept
template<class Enum, typename = typename std::enable_if<IsFlagEnum<Enum>::value>::type>
constexpr Enum &	operator&= (Enum &lhs, Enum rhs) noexcept
	Sets lhs with logical AND between lhs and rhs.
Hardware Queries
Query host system/hardware.
std::size_t	GetPageSize () noexcept
	Fast query of system page size.
int	GetNumNodes () noexcept
	Query number of configured NUMA nodes.
std::optional< int >	GetNodeDistance (int node1, int node2) noexcept
	Get NUMA distance between two nodes.
std::optional< int >	GetNodeOfCpu (int cpu) noexcept
	Get NUMA node of the given CPU.
Makes a std::thread or std::jthread with provided NUMA policies
Create a named thread with optional CPU affinity, scheduler and memory policies. Function has the following effects in this thread (P)arent and new thread (C)hild: (P)Create std::thread with an unspecified thread-setup function as target. (P)Wait for child to be created and complete setup. (C)Function will set thread name and apply policies. (C)Function will apply memory policy to thread stack memory which will move physical pages if necessary. This is done with strict MemPolicyFlag such that if moving pages fails the thread creation will fail. (C)Function signals parent thread with success/failure. (C)If setup was successful invoke func with args, and in case of std::jthread the associated std::stop_token if func is invocable with it, otherwise return. (P)Wake on signal from child: If setup of new thread failed it will join with thread and throw std:system_error containing error. On success it returns thread. Parameters thread_name Name of thread, Must be maximum 16 characters (15 + '\0'). policies The NUMA policies to apply. func Callable invoked in new thread. args Arguments for func which will be decay-copied. Use std::reference_wrapper to pass references (remember: caller must ensure the life-time of references objects). Exceptions std::system_error if new thread failed to apply policies. Template Parameters Func MoveConstructible function to invoke in new thread. Args MoveConstructible arguments to decay copy and call Func with in new thread. Minimum application example with a main() function: #include <chrono> #include <iostream> #include <numapp/numa.hpp> #include <numapp/thread.hpp> void ThreadFunc(std::chrono::milliseconds duration) { std::cout << "Thread affinity: " << numapp::CpuAffinity::MakeFromActive() << std::endl; std::this_thread::sleep_for(duration); } int main() { using namespace numapp; using namespace std::chrono_literals; if (!NumaAvailable()) { std::cout << "NUMA not available"; return -1; } NumaPolicies policies; policies.SetCpuAffinity(CpuAffinity::MakeFromCpuStringAll("0-3")); auto thread = MakeThread("myThread", policies, &ThreadFunc, 500ms); thread.join(); } Example function that create pinned thread with local NUMA node memory policy: #include <numapp/memory.hpp> #include <numapp/thread.hpp> template <class F, class... Args> auto MakePinnedThread(int cpu, std::string_view name, F&& f, Args... args) -> std::thread { using namespace numapp; NumaPolicies policies; policies.SetCpuAffinity(CpuAffinity::MakeBindCpu(cpu)); auto node = GetNodeOfCpu(cpu); if (!node.has_value()) { throw std::invalid_argument("cpu invalid"); } policies.SetMemPolicy(MemPolicy::MakeBindNode(*node)); return MakeThread(name, policies, std::forward<F>(f), std::forward<Args>(args)...); } Similar to first example, but uses member function instead: #include <chrono> #include <iostream> #include <numapp/numa.hpp> #include <numapp/thread.hpp> struct Example { Example(std::chrono::milliseconds duration) : m_duration(duration) { } Example(Example&&) = default; void ThreadFunc() { std::cout << "Thread affinity: " << numapp::CpuAffinity::MakeFromActive() << std::endl; std::this_thread::sleep_for(m_duration); } std::chrono::milliseconds m_duration; }; int main() { using namespace numapp; using namespace std::chrono_literals; if (!NumaAvailable()) { std::cout << "NUMA not available"; return -1; } NumaPolicies policies; policies.SetCpuAffinity(CpuAffinity::MakeFromCpuStringAll("0-3")); auto thread = MakeJthread("myThread", policies, &Example::ThreadFunc, Example{500ms}); }
template<class Func, class... Args>
std::thread	MakeThread (std::string_view thread_name, NumaPolicies const &policies, Func &&func, Args &&... args)
	Primary overload accepting string-view for thread_name.
template<class Func, class... Args>
std::thread	MakeThread (char const *thread_name, NumaPolicies const &policies, Func &&func, Args &&... args)
	Compatibility overload accepting null terminated C string for thread_name.

Memory Locking
These functions provide the means to prevent memory from being paged to the swap area. Note Care should be taken when using these as resident memory consumption will increase as the risk of out-of-memory scenarios. In particular it should be noted that applications may allocate memory that is never resident in normal use. Forcing this memory to be faulted-in may lead to significant memory pressure. One way to avoid that is to use LockFlag.OnFault and LockAllFlag.OnFault which will lock resident memory only. For real-time applications this should not be used however as page faults should be avoided in those scenarios. Permissions Locking may require additional permissions if it exceeds resource limits (- getrlimit(2)). Permissions can be provided by CAP_IPC_LOCK (- capabilities(7)). See also page section Memory Locking.
enum class	LockFlag : unsigned int { LockFlag::PreFault = 0u , LockFlag::OnFault = MLOCK_ONFAULT }
	Mutually exclusive flags that modifies behaviour of MemLock(). More...
enum class	LockAllFlag : int { LockAllFlag::Current = MCL_CURRENT , LockAllFlag::Future = MCL_FUTURE , LockAllFlag::OnFault = MCL_ONFAULT }
	Flags that are combined to modify behaviour of MemLockAll(). More...
std::error_code	MemLock (void const *addr, std::size_t len, LockFlag flag) noexcept
	Lock memory pages in the specified address range.
std::error_code	MemUnlock (void const *addr, std::size_t len) noexcept
	Unlock memory pages in the specified address range.
std::error_code	MemLockAll (LockAllFlag flags) noexcept
	Lock all memory pages as specified by provided flags.
std::error_code	MemUnlockAll () noexcept
	Unlock all locked memory in this process.

Apply CPU Affinity to Specific Thread
Applies specified policy to calling thread.
std::error_code	Apply (pid_t thread, CpuAffinity const &affinity) noexcept
	Apply policy to specified thread.
std::error_code	Apply (pid_t thread, CpuAffinity const &affinity) noexcept
	Apply policy to specified thread.

Allocate and free memory with specified memory policy
Allocate numapp.Allocate() function is equivalent to performing the following: Create new memory mapping with mmap (anonymous read/write) memory Applying memory policy to allocated memory using provided policy and flag (1) using mbind. (1) Overloads without flag use no modifiers (i.e. won't move pages). Important Memory is not pre-faulted by default. If memory is used by real-time threads consider using numapp.MemLock() after allocation to guarantee that pages as faulted in. However if process locks memory when it is mapped (as controlled by numapp.MemLockAll() with numapp.LockAllFlag.Future) memory may have to be moved to apply the requested policy. It is in this case the flags argument has an important role to control if pages should be moved. Free numapp.Free() frees memory previously allocated with numapp.Allocate(). Parameters [in] size Number of bytes to allocate/free, will be rounded up to multiples of system page size. [in] policy Memory policy to apply. [in] flags Controls behaviour if memory have to be moved because it was already paged-in due to a numapp.MemLockAll policy. [in] ptr Page-aligned pointer to memory previously allocated with numapp.Allocate(). [out] ec Error code for non-throwing overloads. If there is an out of memory situation the error code std::errc::not_enough_memory is used. Exceptions std::system_error on failure (only applicable to throwing-overloads without std::error_code return value). Returns numapp.Allocate() returns pointer to memory block that is at least size big or nullptr on failure. Example allocating with bind policy and then lock and pre-fault the newly allocated memory: #include <numapp/memory.hpp> #include <numapp/mempolicy.hpp> void* BindAlloc(std::size_t size, int node, std::error_code& ec) { using namespace numapp; MemPolicy policy = MemPolicy::MakeBindNode(node); void* ptr = Allocate(size, policy, MemPolicyFlag::Strict, ec); if (ec) { return nullptr; } ec = MemLock(ptr, size, LockFlag::PreFault); if (ec) { return nullptr; } return ptr; } Related man-page(s): mmap(2) mbind(2)
void *	Allocate (std::size_t size, MemPolicy const &policy, std::error_code &ec) noexcept
	See group for details.
void *	Allocate (std::size_t size, MemPolicy const &policy, MemPolicyFlag flags, std::error_code &ec) noexcept
	See group for details.
void *	Allocate (std::size_t size, MemPolicy const &policy)
	See group for details.
void *	Allocate (std::size_t size, MemPolicy const &policy, MemPolicyFlag flags)
	See group for details.
void	Free (void *ptr, std::size_t size, std::error_code &ec) noexcept
	See group for details.
void	Free (void *ptr, std::size_t size)
	See group for details.
void *	Allocate (std::size_t size, MemPolicy const &policy, std::error_code &ec) noexcept
	See group for details.
void *	Allocate (std::size_t size, MemPolicy const &policy, MemPolicyFlag flags, std::error_code &ec) noexcept
	See group for details.
void *	Allocate (std::size_t size, MemPolicy const &policy)
	See group for details.
void *	Allocate (std::size_t size, MemPolicy const &policy, MemPolicyFlag flags)
	See group for details.
void	Free (void *ptr, std::size_t size, std::error_code &ec) noexcept
	See group for details.
void	Free (void *ptr, std::size_t size)
	See group for details.

Allocate and free huge pages
Allocation is similar to Allocate and free but with the additional argument specifying the (huge) page size. Like non-huge allocations the pages are not pre-faulted by the functions so to avoid page fault failurs it is recommended to do that before use. #include <numapp/memory.hpp> void Example() { // Allocate 16MiB from NUMA node 1 in 2MiB pages. auto const size = 16 * 1024 * 1024; auto const page_size = numapp::HugePagePreset::Huge2M; // 1) AllocateHuge will allocate size, rounding up to nearest page_size // multiple void* ptr = numapp::AllocateHuge(size, page_size, numapp::MemPolicy::MakeBindNode(1), numapp::MemPolicyFlag::Strict); // 2) It is particularly important to pre-fault huge-pages as the likelihood // for failure is higher. If page-fault would fail when memory is accessed // the result is a segmentation fault. if (auto ec = numapp::MemLock(ptr, size, numapp::LockFlag::PreFault); ec) { // Page-fault failed; this would have caused a segmentation fault if not // for `numapp::MemLock()`. throw std::system_error(ec, "Page fault failed"); } // 3) FreeHuge will free memory, rounding up to nearest page_size multiple // as required by mmap. numapp::FreeHuge(ptr, size, page_size); } Parameters Parameters [in] size Number of bytes to allocate/free, will be rounded up to multiples of page_size. [in] page_size Page size. [in] policy Memory policy to apply. [in] flags Controls behaviour if memory have to be moved because it was already paged-in due to a numapp.MemLockAll policy. [in] ptr Page-aligned pointer to memory previously allocated with numapp.AllocateHuge(). [out] ec Error code for non-throwing overloads. If there is an out of memory situation the error code std::errc::not_enough_memory is used. Exceptions std::system_error on failure (only applicable to throwing-overloads without std::error_code return value). Returns numapp.AllocateHuge() returns pointer to memory block that is at least size big or nullptr on failure. Related man-page(s): mmap(2) mbind(2)
void *	AllocateHuge (std::size_t size, HugePageSize page_size, MemPolicy const &policy, MemPolicyFlag flags, std::error_code &ec) noexcept
	Non-throwing version of AllocateHuge()
void *	AllocateHuge (std::size_t size, HugePageSize page_size, MemPolicy const &policy, MemPolicyFlag flags)
	Throwing version of AllocateHuge()
void	FreeHuge (void *ptr, std::size_t size, HugePageSize page_size, std::error_code &ec) noexcept
	Free huge pages previously allocated with AllocateHuge.
void	FreeHuge (void *ptr, std::size_t size, HugePageSize page_size)
	Throwing version of FreeHuge().
void *	AllocateHuge (std::size_t size, HugePageSize page_size, MemPolicy const &policy, MemPolicyFlag flags, std::error_code &ec) noexcept
	Non-throwing version of AllocateHuge()
void *	AllocateHuge (std::size_t size, HugePageSize page_size, MemPolicy const &policy, MemPolicyFlag flags)
	Throwing version of AllocateHuge()

Apply Memory Policy to Memory Range
std::error_code	Apply (void *address, std::size_t length, MemPolicy const &policy, MemPolicyFlag flags) noexcept
	Applies memory policy to the memory pages that spans the range [address, adress + length].
std::error_code	Apply (void *address, std::size_t length, MemPolicy const &policy, MemPolicyFlag flags) noexcept
	Applies memory policy to the memory pages that spans the range [address, adress + length].

Apply Scheduler to Specified Thread
Applies scheduler to specified thread.
std::error_code	Apply (pid_t thread, IdleScheduler const &scheduler) noexcept
	Apply idle scheduler to specified thread.
std::error_code	Apply (pid_t thread, DynamicScheduler const &scheduler) noexcept
	Apply dynamic scheduler to specified thread.
std::error_code	Apply (pid_t thread, StaticScheduler const &scheduler) noexcept
	Apply static scheduler to specified thread.
std::error_code	Apply (pid_t thread, Scheduler const &scheduler) noexcept
	Apply variadic scheduler to specified thread.
std::error_code	Apply (pid_t thread, IdleScheduler const &scheduler) noexcept
	Apply idle scheduler to specified thread.
std::error_code	Apply (pid_t thread, DynamicScheduler const &scheduler) noexcept
	Apply dynamic scheduler to specified thread.
std::error_code	Apply (pid_t thread, StaticScheduler const &scheduler) noexcept
	Apply static scheduler to specified thread.
std::error_code	Apply (pid_t thread, Scheduler const &scheduler) noexcept
	Apply variadic scheduler to specified thread.

Detailed Description

Copyright

(c) Copyright ESO 2024 All Rights Reserved ESO (eso.org) is an Intergovernmental Organisation, and therefore special legal conditions apply.

Function Documentation

Apply() [1/3]

std::error_code Apply ( void * address, std::size_t length, MemPolicy const & policy, MemPolicyFlag flags )

nodiscardnoexcept

Applies memory policy to the memory pages that spans the range [address, adress + length].

Note

The memory policy is by default only applied to new physical page allocations. This can be modified using MemPolicyFlag::Move and MemPolicyFlag::MoveAll.

Read related man-page of mbind() to understand the various behaviours if memory is mapped or shared.

Parameters

address	start of address range.
length	length of range.
policy	Policy to apply.
flags	combination of MemPolicyFlag values that modify the behaviour.

Returns

0 if successful.

non-zero if it failed.

Related man-page(s):

• mbind(2)

operator&()

template<class Enum, typename = typename std::enable_if<IsFlagEnum<Enum>::value>::type>

Enum numapp::operator& ( Enum lhs, Enum rhs )

nodiscardconstexprnoexcept

Returns

logical AND between lhs and rhs

Definition at line 91 of file flags.hpp.

operator<<() [1/10]

std::ostream & numapp::operator<<	(	std::ostream &	os,
		NumaPolicies const &	policies )

Formats policies and inserts it to os.

Parameters

os	output stream to insert into.
policies	policy to format.

Returns

os

Definition at line 36 of file numapolicies.cpp.

operator<<() [2/10]

std::ostream & numapp::operator<<	(	std::ostream &	os,
		StaticScheduler::Policy const &	policy )

Formats policy and inserts it to os.

Parameters

os	output stream to insert into.
policy	Policy to format.

Returns

os

operator|()

template<class Enum, typename = typename std::enable_if<IsFlagEnum<Enum>::value>::type>

Enum numapp::operator| ( Enum lhs, Enum rhs )

nodiscardconstexprnoexcept

Returns

logical OR between lhs and rhs

Definition at line 74 of file flags.hpp.