Initialization

int rumpuser_init(int version, struct rump_hyperup *hyp)

Initialize the hypervisor.

version

hypercall interface version number that the kernel expects to be used. In case the hypervisor cannot provide an exact match, this routine must return a non-zero value.

hyp

pointer to a set of upcalls the hypervisor can make into the rump kernel

Memory allocation

int rumpuser_malloc(size_t len, int alignment, void **memp)

len

amount of memory to allocate

alignment

size the returned memory must be aligned to. For example, if the value passed is 4096, the returned memory must be aligned to a 4k boundary.

memp

return pointer for allocated memory

void rumpuser_free(void *mem, size_t len)

mem

memory to free

len

length of allocation. This is always equal to the amount the caller requested from the rumpuser_malloc() which returned mem.

Files and I/O

int rumpuser_open(const char *name, int mode, int *fdp)

Open name for I/O and associate a file descriptor with it. Notably, there needs to be no mapping between name and the host's file system namespace. For example, it is possible to associate the file descriptor with device I/O registers for special values of name.

name

the identifier of the file to open for I/O

mode

combination of the following:

RUMPUSER_OPEN_RDONLY

open only for reading

RUMPUSER_OPEN_WRONLY

open only for writing

RUMPUSER_OPEN_RDWR

open for reading and writing

RUMPUSER_OPEN_CREATE

do not treat missing name as an error

RUMPUSER_OPEN_EXCL

combined with RUMPUSER_OPEN_CREATE, flag an error if name already exists

RUMPUSER_OPEN_BIO

the caller will use this file for block I/O, usually used in conjunction with accessing file system media. The hypervisor should treat this flag as advisory and possibly enable some optimizations for *fdp based on it.

Notably, the permissions of the created file are left up to the hypervisor implementation.

fdp

An integer value denoting the open file is returned here.

int rumpuser_close(int fd)

Close a previously opened file descriptor.

int rumpuser_getfileinfo(const char *name, uint64_t *size, int *type)

name

file for which information is returned. The namespace is equal to that of rumpuser_open().

size

If non-NULL, size of the file is returned here.

type

If non-NULL, type of the file is returned here. The options are RUMPUSER_FT_DIR, RUMPUSER_FT_REG, RUMPUSER_FT_BLK, RUMPUSER_FT_CHR, or RUMPUSER_FT_OTHER for directory, regular file, block device, character device or unknown, respectively.

void rumpuser_bio(int fd, int op, void *data, size_t dlen, int64_t off, rump_biodone_fn biodone, void *donearg);

Initiate block I/O and return immediately.

fd

perform I/O on this file descriptor. The file descriptor must have been opened with RUMPUSER_OPEN_BIO.

op

Transfer data from the file descriptor with RUMPUSER_BIO_READ and transfer data to the file descriptor with RUMPUSER_BIO_WRITE. Unless RUMPUSER_BIO_SYNC is specified, the hypervisor may cache a write instead of committing it to permanent storage.

data

memory address to transfer data to/from

dlen

length of I/O. The length is guaranteed to be a multiple of 512.

off

offset into fd where I/O is performed

biodone

To be called when the I/O is complete. Accessing data is not legal after the call is made.

donearg

opaque arg that must be passed to biodone.

int rumpuser_iovread(int fd, struct rumpuser_iovec *ruiov, size_t iovlen, int64_t off, size_t *retv);

int rumpuser_iovwrite(int fd, struct rumpuser_iovec *ruiov, size_t iovlen, int64_t off, size_t *retv);

These routines perform scatter-gather I/O which is not block I/O by nature and therefore cannot be handled by rumpuser_bio().

fd

file descriptor to perform I/O on

ruiov

an array of I/O descriptors. It is defined as follows:

struct rumpuser_iovec {
	void *iov_base;
	size_t iov_len;
};

iovlen

number of elements in ruiov

off

offset of fd to perform I/O on. This can either be a non-negative value or RUMPUSER_IOV_NOSEEK. The latter denotes that no attempt to change the underlying objects offset should be made. Using both types of offsets on a single instance of fd results in undefined behavior.

retv

number of bytes successfully transferred is returned here

int rumpuser_syncfd(int fd, int flags, uint64_t start, uint64_t len);

Synchronizes fd with respect to backing storage. The other arguments are:

flags

controls how synchronization happens. It must contain one of the following:

RUMPUSER_SYNCFD_READ

Make sure that the next read sees writes from all other parties. This is useful for example in the case that fd represents memory to write a DMA read is being performed.

RUMPUSER_SYNCFD_WRITE

Flush cached writes.

The following additional parameters may be passed in flags:

RUMPUSER_SYNCFD_BARRIER

Issue a barrier. Outstanding I/O operations which were started before the barrier complete before any operations after the barrier are performed.

RUMPUSER_SYNCFD_SYNC

Wait for the synchronization operation to fully complete before returning. For example, this could mean that the data to be written to a disk has hit either the disk or non-volatile memory.

start

offset into the object.

len

the number of bytes to synchronize. The value 0 denotes until the end of the object.

Clocks

The hypervisor should support two clocks, one for wall time and one for monotonically increasing time, the latter of which may be based on some arbitrary time (e.g. system boot time). If this is not possible, the hypervisor must make a reasonable effort to retain semantics.

int rumpuser_clock_gettime(int enum_rumpclock, int64_t *sec, long *nsec)

enum_rumpclock

specifies the clock type. In case of RUMPUSER_CLOCK_RELWALL the wall time should be returned. In case of RUMPUSER_CLOCK_ABSMONO the time of a monotonic clock should be returned.

sec

return value for seconds

nsec

return value for nanoseconds

int rumpuser_clock_sleep(int enum_rumpclock, int64_t sec, long nsec)

enum_rumpclock

In case of RUMPUSER_CLOCK_RELWALL, the sleep should last at least as long as specified. In case of RUMPUSER_CLOCK_ABSMONO, the sleep should last until the hypervisor monotonic clock hits the specified absolute time.

sec

sleep duration, seconds. exact semantics depend on clk.

nsec

sleep duration, nanoseconds. exact semantics depend on clk.

Parameter retrieval

int rumpuser_getparam(const char *name, void *buf, size_t buflen)

Retrieve a configuration parameter from the hypervisor. It is up to the hypervisor to decide how the parameters can be set.

name

name of the parameter. If the name starts with an underscore, it means a mandatory parameter. The mandatory parameters are RUMPUSER_PARAM_NCPU which specifies the amount of virtual CPUs bootstrapped by the rump kernel and RUMPUSER_PARAM_HOSTNAME which returns a preferably unique instance name for the rump kernel.

buf

buffer to return the data in as a string

buflen

length of buffer

Termination

void rumpuser_exit(int value)

Terminate the rump kernel with exit value value. If value is RUMPUSER_PANIC the hypervisor should attempt to provide something akin to a core dump.

Console output

Console output is divided into two routines: a per-character one and printf-like one. The former is used e.g. by the rump kernel's internal printf routine. The latter can be used for direct debug prints e.g. very early on in the rump kernel's bootstrap or when using the in-kernel routine causes too much skew in the debug print results (the hypercall runs outside of the rump kernel and therefore does not cause any locking or scheduling events inside the rump kernel).

void rumpuser_putchar(int ch)

Output ch on the console.

void rumpuser_dprintf(const char *fmt, ...)

Do output based on printf-like parameters.

Signals

A rump kernel should be able to send signals to client programs due to some standard interfaces including signal delivery in their specifications. Examples of these interfaces include setitimer(2) and write(2). The rumpuser_kill() function advises the hypercall implementation to raise a signal for the process containing the rump kernel.

int rumpuser_kill(int64_t pid, int sig)

pid

The pid of the rump kernel process that the signal is directed to. This value may be used as the hypervisor as a hint on how to deliver the signal. The value RUMPUSER_PID_SELF may also be specified to indicate no hint. This value will be removed in a future version of the hypercall interface.

sig

Number of signal to raise. The value is in NetBSD signal number namespace. In case the host has a native representation for signals, the value should be translated before the signal is raised. In case there is no mapping between sig and native signals (if any), the behavior is implementation-defined.

A rump kernel will ignore the return value of this hypercall. The only implication of not implementing rumpuser_kill() is that some application programs may not experience expected behavior for standard interfaces.

As an aside,the rump_sp(7) protocol provides equivalent functionality for remote clients.

Random pool

int rumpuser_getrandom(void *buf, size_t buflen, int flags, size_t *retp)

buf

buffer that the randomness is written to

buflen

number of bytes of randomness requested

flags

The value 0 or a combination of RUMPUSER_RANDOM_HARD (return true randomness instead of something from a PRNG) and RUMPUSER_RANDOM_NOWAIT (do not block in case the requested amount of bytes is not available).

retp

The number of random bytes written into buf.

Threads

int rumpuser_thread_create(void *(*fun)(void *), void *arg, const char *thrname, int mustjoin, int priority, int cpuidx, void **cookie);

Create a schedulable host thread context. The rump kernel will call this interface when it creates a kernel thread. The scheduling policy for the new thread is defined by the hypervisor. In case the hypervisor wants to optimize the scheduling of the threads, it can perform heuristics on the thrname, priority and cpuidx parameters.

fun

function that the new thread must call. This call will never return.

arg

argument to be passed to fun

thrname

Name of the new thread.

mustjoin

If 1, the thread will be waited for by rumpuser_thread_join() when the thread exits.

priority

The priority that the kernel requested the thread to be created at. Higher values mean higher priority. The exact kernel semantics for each value are not available through this interface.

cpuidx

The index of the virtual CPU that the thread is bound to, or -1 if the thread is not bound. The mapping between the virtual CPUs and physical CPUs, if any, is hypervisor implementation specific.

cookie

In case mustjoin is set, the value returned in cookie will be passed to rumpuser_thread_join().

void rumpuser_thread_exit(void)

Called when a thread created with rumpuser_thread_create() exits.

int rumpuser_thread_join(void *cookie)

Wait for a joinable thread to exit. The cookie matches the value from rumpuser_thread_create().

void rumpuser_curlwpop(int enum_rumplwpop, struct lwp *l)

Manipulate the hypervisor's thread context database. The possible operations are create, destroy, and set as specified by enum_rumplwpop:

RUMPUSER_LWP_CREATE

Inform the hypervisor that l is now a valid thread context which may be set. A currently valid value of l may not be specified. This operation is informational and does not mandate any action from the hypervisor.

RUMPUSER_LWP_DESTROY

Inform the hypervisor that l is no longer a valid thread context. This means that it may no longer be set as the current context. A currently set context or an invalid one may not be destroyed. This operation is informational and does not mandate any action from the hypervisor.

RUMPUSER_LWP_SET

Set l as the current host thread's rump kernel context. A previous context must not exist.

RUMPUSER_LWP_CLEAR

Clear the context previous set by RUMPUSER_LWP_SET. The value passed in l is the current thread and is never NULL.

struct lwp * rumpuser_curlwp(void)

Retrieve the rump kernel thread context associated with the current host thread, as set by rumpuser_curlwpop(). This routine may be called when a context is not set and the routine must return NULL in that case. This interface is expected to be called very often. Any optimizations pertaining to the execution speed of this routine should be done in rumpuser_curlwpop().

void rumpuser_seterrno(int errno)

Set an errno value in the calling thread's TLS. Note: this is used only if rump kernel clients make rump system calls.

Mutexes, rwlocks and condition variables

The locking interfaces have standard semantics, so we will not discuss each one in detail. The data types struct rumpuser_mtx, struct rumpuser_rw and struct rumpuser_cv used by these interfaces are opaque to the rump kernel, i.e. the hypervisor has complete freedom over them.

Most of these interfaces will (and must) relinquish the rump kernel CPU context in case they block (or intend to block). The exceptions are the "nowrap" variants of the interfaces which may not relinquish rump kernel context.

void rumpuser_mutex_init(struct rumpuser_mtx **mtxp, int flags)

void rumpuser_mutex_enter(struct rumpuser_mtx *mtx)

void rumpuser_mutex_enter_nowrap(struct rumpuser_mtx *mtx)

int rumpuser_mutex_tryenter(struct rumpuser_mtx *mtx)

void rumpuser_mutex_exit(struct rumpuser_mtx *mtx)

void rumpuser_mutex_destroy(struct rumpuser_mtx *mtx)

void rumpuser_mutex_owner(struct rumpuser_mtx *mtx, struct lwp **lp)

Mutexes provide mutually exclusive locking. The flags, of which at least one must be given, are as follows:

RUMPUSER_MTX_SPIN

Create a spin mutex. Locking this type of mutex must not relinquish rump kernel context even when rumpuser_mutex_enter() is used.

RUMPUSER_MTX_KMUTEX

The mutex must track and be able to return the rump kernel thread that owns the mutex (if any). If this flag is not specified, rumpuser_mutex_owner() will never be called for that particular mutex.

void rumpuser_rw_init(struct rumpuser_rw **rwp)

void rumpuser_rw_enter(int enum_rumprwlock, struct rumpuser_rw *rw)

int rumpuser_rw_tryenter(int enum_rumprwlock, struct rumpuser_rw *rw)

int rumpuser_rw_tryupgrade(struct rumpuser_rw *rw)

void rumpuser_rw_downgrade(struct rumpuser_rw *rw)

void rumpuser_rw_exit(struct rumpuser_rw *rw)

void rumpuser_rw_destroy(struct rumpuser_rw *rw)

void rumpuser_rw_held(int enum_rumprwlock, struct rumpuser_rw *rw, int *heldp);

Read/write locks provide either shared or exclusive locking. The possible values for lk are RUMPUSER_RW_READER and RUMPUSER_RW_WRITER. Upgrading means trying to migrate from an already owned shared lock to an exclusive lock and downgrading means migrating from an already owned exclusive lock to a shared lock.

void rumpuser_cv_init(struct rumpuser_cv **cvp)

void rumpuser_cv_destroy(struct rumpuser_cv *cv)

void rumpuser_cv_wait(struct rumpuser_cv *cv, struct rumpuser_mtx *mtx)

void rumpuser_cv_wait_nowrap(struct rumpuser_cv *cv, struct rumpuser_mtx *mtx)

int rumpuser_cv_timedwait(struct rumpuser_cv *cv, struct rumpuser_mtx *mtx, int64_t sec, int64_t nsec);

void rumpuser_cv_signal(struct rumpuser_cv *cv)

void rumpuser_cv_broadcast(struct rumpuser_cv *cv)

void rumpuser_cv_has_waiters(struct rumpuser_cv *cv, int *waitersp)

Condition variables wait for an event. The mtx interlock eliminates a race between checking the predicate and sleeping on the condition variable; the mutex should be released for the duration of the sleep in the normal atomic manner. The timedwait variant takes a specifier indicating a relative sleep duration after which the routine will return with ETIMEDOUT. If a timedwait is signaled before the timeout expires, the routine will return 0.

The order in which the hypervisor reacquires the rump kernel context and interlock mutex before returning into the rump kernel is as follows. In case the interlock mutex was initialized with both RUMPUSER_MTX_SPIN and RUMPUSER_MTX_KMUTEX, the rump kernel context is scheduled before the mutex is reacquired. In case of a purely RUMPUSER_MTX_SPIN mutex, the mutex is acquired first. In the final case the order is implementation-defined.