TAIL OSv0.9.0

Aperiodic Framework — Rust API

Declare an interface once; the framework generates the typed client every caller uses and the dispatch every server runs. A server is an impl block with an attribute — there is no main, no receive loop and no reply bookkeeping to write.

An aperiodic service is a server whose work arrives when something asks for it, not on a clock. Each request goes through the same three steps: the framework receives it, dispatches it to your handler, and sends exactly one reply. A driver that answers "read this pin", a log server that hands out rings, a planner that answers "is this path clear?" are all aperiodic services.

Its sibling, the periodic node, runs a body once per declared period and exchanges data over topics. Use a periodic node for a control loop that runs at a fixed rate. Use an aperiodic service for a question that one caller asks and one server answers, at any time.

The path of one requestA caller thread's request is copied by the kernel to the server's receiving thread, decoded with its operation and length checked, handed to your handler, and answered with exactly one reply back to the caller.caller threadlimits.get(..)kernelcopies the requestreceiving threadof the serverdecodeoperation and length checkedyour handlerfn getexactly one reply

A caller's request is copied by the kernel to the server's receiving thread, checked and decoded, handed to your handler, and answered with exactly one reply.

Quick start

Three files in the BSP's app/ directory: the interface, defined once, and two programs that each hold one end of it.

File Is
app/speed_limit_interface.rs the interface: service name, operations, payloads
app/speed_limit_server/src/main.rs the server that holds a speed limit per wheel
app/drive_setup/src/main.rs a client that sets one limit and reads it back

The interface, defined once

// app/speed_limit_interface.rs

// Each program uses part of this file: the server the trait, the client the client.
#![allow(dead_code)]

use aperiodic::{interface, ErrorKind};
use wire_plain::WirePlain;

/// The name the service registers under and clients resolve.
pub const SPEED_LIMIT_SERVICE: &str = "speed_limit";

pub const OP_SET: u64 = 1;
pub const OP_GET: u64 = 2;

#[derive(Clone, Copy, Debug, WirePlain)]
#[repr(C)]
pub struct SetLimit {
    /// Wheel index, 0..=3.
    pub wheel: u32,
    /// Largest wheel speed allowed [mrad/s].
    pub max_mrad_s: u32,
}

#[derive(Clone, Copy, Debug, WirePlain)]
#[repr(C)]
pub struct GetLimit {
    /// Wheel index, 0..=3.
    pub wheel: u32,
}

#[derive(Clone, Copy, Debug, WirePlain)]
#[repr(C)]
pub struct Limit {
    /// Largest wheel speed allowed [mrad/s].
    pub max_mrad_s: u32,
}

/// Per-wheel speed limits.
#[interface(name = SPEED_LIMIT_SERVICE)]
pub trait SpeedLimit {
    /// Set one wheel's limit. Answers with nothing.
    #[op(OP_SET, reply = ())]
    fn set(&self, args: SetLimit) -> Result<(), ErrorKind>;

    /// Read one wheel's limit.
    #[op(OP_GET, reply = Limit)]
    fn get(&self, args: GetLimit) -> Result<Limit, ErrorKind>;
}

From this one declaration #[interface] generates:

Generated For
the trait SpeedLimit, as you wrote it the server, which implements it
SpeedLimitClient, with connect(), set(SetLimit) and get(GetLimit) every caller
the module speed_limit_ops, with a marker type GetOp the server, when it answers get later (see Deferred replies)

Both programs compile the same file, so they cannot disagree on an operation number or a payload type: change one and both sides change together.

The server

// app/speed_limit_server/src/main.rs
#[path = "../../speed_limit_interface.rs"]
mod speed_limit_interface;

use std::sync::atomic::{AtomicU32, Ordering};

use aperiodic::{aperiodic, ErrorKind, Lifecycle, LogTag};
use speed_limit_interface::{GetLimit, Limit, SetLimit, SpeedLimit};

struct SpeedLimitService {
    max_mrad_s: [AtomicU32; 4],
}

impl Lifecycle for SpeedLimitService {
    fn start(_log: LogTag) -> Result<Self, ErrorKind> {
        Ok(Self { max_mrad_s: Default::default() })
    }
}

#[aperiodic(tag = "speed_limit")]
impl SpeedLimit for SpeedLimitService {
    fn set(&self, args: SetLimit) -> Result<(), ErrorKind> {
        let wheel = self.max_mrad_s.get(args.wheel as usize).ok_or(ErrorKind::InvalidInput)?;
        wheel.store(args.max_mrad_s, Ordering::Relaxed);
        Ok(())
    }

    fn get(&self, args: GetLimit) -> Result<Limit, ErrorKind> {
        let wheel = self.max_mrad_s.get(args.wheel as usize).ok_or(ErrorKind::InvalidInput)?;
        Ok(Limit { max_mrad_s: wheel.load(Ordering::Relaxed) })
    }
}

#[aperiodic] writes main. It builds the handler with Lifecycle::start, registers the name speed_limit, records service ready in the system log under the tag speed_limit, and then serves requests until the process ends. Each handler returns a Result: Ok becomes the reply, Err(kind) becomes the caller's Err(kind).

The client

// app/drive_setup/src/main.rs
#[path = "../../speed_limit_interface.rs"]
mod speed_limit_interface;

use speed_limit_interface::{GetLimit, SetLimit, SpeedLimitClient};

fn main() -> Result<(), aperiodic::ErrorKind> {
    let limits = SpeedLimitClient::connect()?;
    limits.set(SetLimit { wheel: 0, max_mrad_s: 12_000 })?;
    let limit = limits.get(GetLimit { wheel: 0 })?;
    println!("wheel 0 limited to {} mrad/s", limit.max_mrad_s);
    Ok(())
}

connect() resolves the name, waiting for the server to register if it has not yet. Each method sends one request and blocks the calling thread until the reply arrives. The caller writes no operation number, no byte packing and no reply decoding.

Build and run

Add /usr/bin/speed_limit_server=speed_limit_server and /usr/bin/drive_setup=drive_setup to the [disk] section of the BSP's tail.build, then run ./make-disk.sh. See Installing TAIL OS on Raspberry Pi 3. aperiodic, wire_plain and log are libraries of the SDK, so a program names them with a use and nothing more.

On the booted image, start them in either order: speed_limit_server &, then drive_setup. If the client starts first, connect() waits for the name.

In a cargo workspace the interface is a library crate instead, depending on aperiodic and wire_plain; the server and every client list it in their Cargo.toml. The server also lists log, because the generated main creates its log tag. The interface crate's public signatures name ErrorKind and CallerIdentity, which reach it through std; put #![allow(exported_private_dependencies)] on the module that declares the trait to silence the resulting lint.

Declaring an interface

#[interface(name = <&str constant>)] goes on a trait. Every method of the trait is one operation, and carries #[op(..)]:

#[op(OP_GET, reply = Limit)]
fn get(&self, args: GetLimit) -> Result<Limit, ErrorKind>;
Part Rule
OP_GET the operation number, a u64 constant. Two operations with the same number fail to build
reply = Limit the reply type; reply = () for an operation that answers with nothing
&self always &self. &mut self is refused with a message that says where mutable state goes
args: GetLimit the request payload, always present. An operation with nothing to send takes a payload with one reserved field
a third parameter caller: CallerIdentity optional: the kernel-stamped identity of the calling process (see Who is calling)
return type Result<Reply, ErrorKind>, matching reply = ..

Further keys on #[op]:

Key Effect
defer the handler answers later, through a reply token; see Deferred replies
trailing the request is the payload followed by a run of bytes; see Bulk data

Payloads

A request payload is a #[repr(C)] struct that derives WirePlain, the same derive the periodic framework uses for topic payloads. The derive proves at the definition that any bytes another process sends form a valid value: fields are fixed-width integers, floats, arrays of them or other such structs, and the struct has no padding. Declare a reply type the same way: the client builds it from bytes the server sent.

The service checks each request's length against the declared payload, and its operation number against the interface. It does not compare layouts beyond that, so build both programs from the same interface definition.

What the build refuses

You write The build says
a request payload over 32 bytes request payload `X` exceeds the register fastpath bound (<= 32 bytes)
a reply type over 32 bytes reply type `X` exceeds the register fastpath bound (<= 32 bytes)
two operations with one number `b` declares the same discriminant value as `a`; the second is unreachable
a method without #[op] every method of an `#[interface]` trait declares an operation: ..
a payload that does not derive WirePlain an unsatisfied WirePlain bound on the payload
#[aperiodic] without tag `#[aperiodic]` needs the subsystem its records are filed under, ..

Writing a server

A server is a type that implements Lifecycle and the interface trait, with #[aperiodic(tag = "..")] on the trait's impl. The tag names the subsystem the server's log records are filed under.

Start-up

The order a server starts inLifecycle start builds the handler, before_register maps hardware and spawns threads, the name is registered, after_register publishes other signals, the service ready record is written, and the server then serves requests.Lifecycle::startbuild the handlerbefore_registermap hardware, spawn threadsthe name is registeredafter_registerpublish other signalsrecord: service readyserve requests

The handler is built, prepared before its name exists, registered, given a step after registration, announced as ready, and then serves.

Hook Runs Use it for
fn start(log: LogTag) -> Result<Self, ErrorKind> first, before anything is logged or registered building the handler. Required
fn before_register(&mut self, log: LogTag) -> Result<(), Unavailable> before the name exists, so no client can reach the service mapping hardware, attaching interrupts, spawning helper threads. Default: nothing
fn after_register(&mut self, log: LogTag) -> Result<(), Unavailable> after the name exists; clients that arrive now wait in the queue publishing a second readiness signal that must not appear before the name. Default: nothing

A hook stops start-up by returning an error:

Returned Recorded The process
Err(ErrorKind) from start startup failed: <kind>, at error exits 1; the name is never registered
Err(Unavailable::Failed(kind)) startup failed: <kind>, at error exits 1
Err(Unavailable::NotPresent) not present on this board, at info exits 0: a board without the device is not a fault

? works inside a hook on any Result<_, ErrorKind>: an ErrorKind converts into Unavailable::Failed.

The hooks take &mut self. The handler methods take &self, and the handler type must be Sync, because it is shared with the thread that receives requests. Keep state you set up once in plain fields; put state that requests change behind a Mutex or atomics, as the quick start's AtomicU32 limits are.

One request at a time

Each service is served by one receiving thread, so its handler runs one request at a time. While a handler works, every other caller of that service waits. A handler that must wait for something slow — a device, an event, another program's action — defers its reply instead of blocking.

Logging

LogTag is Copy. Keep the one start receives in a field and log through it:

self.log.warning(format_args!("wheel {} limit cut to {}", args.wheel, args.max_mrad_s));

Its levels are start, success, info, warning, error and trace. Records go to the process's log ring under the server's tag, alongside the framework's own; read them with logview.

Who is calling

An operation that declares a third parameter caller: CallerIdentity receives the calling process's identity. The kernel stamps it, so a client cannot claim to be another process.

Method Returns
caller.pid() the process id, for looking a process up or logging
caller.generation() which occupant of that process slot this is

Key per-client state on the whole CallerIdentity, not on pid(). A pid is a reused slot: state keyed on it alone passes to whatever process takes the slot next, while an identity does not match a later process. CallerIdentity is Copy, Eq and Hash, and prints as pid 7 gen 1.

Deferred replies

An operation declared defer receives the request's ClientId and returns an Outcome. It either answers now or takes a reply token and answers later, from any thread.

// app/start_gate_interface.rs
#![allow(dead_code)]

use aperiodic::{interface, ClientId, ErrorKind, Outcome};
use wire_plain::WirePlain;

pub const OP_WAIT: u64 = 1;
pub const OP_OPEN: u64 = 2;

#[derive(Clone, Copy, Debug, WirePlain)]
#[repr(C)]
pub struct Wait { pub reserved: u32 }

#[derive(Clone, Copy, Debug, WirePlain)]
#[repr(C)]
pub struct Open { pub mission: u32 }

#[derive(Clone, Copy, Debug, WirePlain)]
#[repr(C)]
pub struct Opened { pub mission: u32 }

#[interface(name = "start_gate")]
pub trait StartGate {
    /// Answers once the operator opens the gate.
    #[op(OP_WAIT, reply = Opened, defer)]
    fn wait(&self, args: Wait, client: ClientId) -> Outcome;

    /// Open the gate, answering every waiting caller.
    #[op(OP_OPEN, reply = ())]
    fn open(&self, args: Open) -> Result<(), ErrorKind>;
}
// app/start_gate_server/src/main.rs
#[path = "../../start_gate_interface.rs"]
mod start_gate_interface;

use std::sync::Mutex;

use aperiodic::{aperiodic, ClientId, ErrorKind, Lifecycle, LogTag, Outcome, Reply, ReplyToken};
use start_gate_interface::{start_gate_ops::WaitOp, Open, Opened, StartGate, Wait};

const MAX_WAITERS: usize = 8;

struct StartGateService {
    waiters: Mutex<Vec<ReplyToken<WaitOp>>>,
}

impl Lifecycle for StartGateService {
    fn start(_log: LogTag) -> Result<Self, ErrorKind> {
        Ok(Self { waiters: Mutex::new(Vec::with_capacity(MAX_WAITERS)) })
    }
}

#[aperiodic(tag = "start_gate")]
impl StartGate for StartGateService {
    fn wait(&self, _args: Wait, client: ClientId) -> Outcome {
        let mut waiters = self.waiters.lock().unwrap();
        if waiters.len() == MAX_WAITERS {
            return client.reply(Reply::err(ErrorKind::ResourceBusy));
        }
        let (outcome, token) = client.defer::<WaitOp>();
        waiters.push(token);
        outcome
    }

    fn open(&self, args: Open) -> Result<(), ErrorKind> {
        for token in self.waiters.lock().unwrap().drain(..) {
            let _ = token.reply(Opened { mission: args.mission });
        }
        Ok(())
    }
}

The caller sees no difference: StartGateClient::wait(Wait { .. }) blocks and returns Ok(Opened { .. }) once the gate opens. The server keeps serving other requests while callers wait.

ClientId is the caller of one request:

Method Does
client.reply(reply) answers now; reply is Reply::ok(), Reply::err(kind) or Reply::of::<Op>(value)
client.defer::<Op>() returns (Outcome, ReplyToken<Op>); return the Outcome, keep the token. Op is the operation's marker in <interface>_ops
client.defer_untyped() the same with an untyped token, for an operation declared reply = () (which has no marker)
client.identity(), client.pid() the caller, as in Who is calling

Each of reply and defer consumes the ClientId, so a request cannot be answered twice: the second attempt does not compile.

A ReplyToken is Send, so it can move to another thread — one that waits on an interrupt, for example. Every way of ending it sends exactly one reply:

Ending The caller gets
token.reply(value) Ok(value); value must be the operation's declared reply type
token.fail(kind) Err(kind)
token.abandon(kind) Err(kind); the way to cancel on purpose, for example when the waiter's owner is gone
the token is dropped Err(ConnectionAborted), and the server logs an error naming the caller: a dropped token is a bug in the server

reply and fail return the transport's result. Err(NotFound) means the caller is no longer waiting — it exited or was killed — and there is nothing more to do. A failure is already logged under the service's tag, so let _ = loses no evidence.

Calling a service

Constructor Waits for the name
SpeedLimitClient::connect() up to 100 attempts, 20 ms apart: at most about 2 s
SpeedLimitClient::connect_bounded(attempts, backoff) attempts times, backoff apart; connect_bounded(1, Duration::ZERO) checks once and never sleeps

Both return Err(NotFound) when the name is still not registered. The client is Copy: connect once and hand a copy to each thread that calls.

A call has no timeout. It returns when the server answers or when the server's process ends. Design a service so that every request is answered: answer at once, or defer and answer when the awaited event happens or is abandoned.

Priority

A request is served at the caller's priority. When the kernel hands a request to the receiving thread it raises that thread to at least the caller's priority, and it ends the raise at the reply. An idle receiving thread runs at the default thread priority, 31. A high-priority caller is therefore not served at a lower server's priority, and a server need not be started at the priority of its most urgent client.

Errors a caller sees

Every error is an ErrorKind, carried unchanged from where it arose:

Err(..) Means
the kind the handler returned the server refused the request: InvalidInput, ResourceBusy, PermissionDenied, or any other kind it chose
NotFound from connect: the name was not registered in time. From a call: the service ended before answering — its process exited or crashed
Unsupported the server has no such operation: the two programs were built from different interface versions
InvalidInput the request's length is not the declared payload's: likewise a version mismatch, unless the handler returned it
ConnectionAborted the server dropped the token for this deferred request
WouldBlock the service already has 16 requests waiting, or a message over 256 bytes found no free buffer in the kernel's message pool. Nothing was done; the call may be repeated
ArgumentListTooLong the request is longer than the server accepts (see Message sizes)

A panic in a handler ends the server process. Every caller waiting on it receives NotFound.

Message sizes

Message Limit Checked
a request payload 32 bytes at build time
a reply value 32 bytes at build time
a trailing request: payload plus bytes 256 KiB at the call: ArgumentListTooLong

The kernel carries a message of up to 32 bytes in registers. Up to 256 bytes it copies the message through the two threads' own buffers, which no other traffic can exhaust. A longer message travels through a shared message pool of 4 KiB, 64 KiB and 256 KiB buffers, so only messages over 256 bytes can meet the pool's WouldBlock.

Bulk data: trailing

An operation declared trailing carries a fixed payload followed by any number of bytes. The handler takes them as its next parameter:

#[op(OP_WRITE, reply = Written, trailing)]
fn write(&self, args: WriteHeader, data: &[u8]) -> Result<Written, ErrorKind>;

The generated client method takes the bytes beside the payload and returns the reply as bytes, which aperiodic::decode turns back into the reply type:

let bytes = storage.write(WriteHeader { handle, data_len: block.len() as u64 }, &block)?;
let written: Written = aperiodic::decode(&bytes)?;

A service with a trailing operation gives its receiving thread a 256 KiB request buffer, because such a request may be that long. A service without one sizes the buffer to its largest declared payload.

Several services in one process

#[aperiodic(tag = "..", no_main)] generates everything but main. Your own main then does its own preparation and ends in aperiodic::run::<StatusService>(log::log_tag!("status")), which never returns.

To serve a second service in the same process, give it a thread of its own. aperiodic::bind::<H>(log) builds and registers one service and returns a Bound<H>; aperiodic::serve(bound) serves it for good. A Bound cannot move between threads, so bind and serve on the same thread. A process registers at most 16 services; a seventeenth bind fails with OutOfMemory, and registering a name that is already taken fails with AlreadyExists.

The enum form

An interface may instead be declared as an enum, one variant per operation. Device drivers that match on the request directly use this form:

#[interface(name = BEEPER_SERVICE)]
pub enum BeeperRequest {
    #[op(OP_BEEP, reply = ())]
    Beep(BeepArgs),
}

It generates BeeperClient with connect() and beep(BeepArgs). The server implements aperiodic::Handler, whose dispatch(&self, request: BeeperRequest, client: ClientId) -> Outcome matches on the decoded enum and answers through the ClientId exactly as a deferring operation does. Its main names the service at run time:

fn main() -> ! {
    aperiodic::run_instance(BEEPER_SERVICE, log::log_tag!("beeper"), Dispatcher::new(Beeper::new()))
}

The enum form also accepts #[op(.., pool = MAX)] for an operation whose reply is longer than 32 bytes: the payload type implements aperiodic::FramedRequest, the handler answers with Reply::framed::<Op>(header, tail), and the client method takes and returns bytes, at most MAX and never more than 256 KiB.

What the framework guarantees

  • Exactly one reply per request while the server lives. A handler's Ok and Err, a decode failure, an unknown operation and every ending of a reply token each send one reply. Answering twice does not compile.
  • The name appears only when the server is ready. No client reaches the service before before_register has returned. A client that arrives during after_register waits in the queue.
  • A request is checked before your code sees it. An unknown operation is answered Unsupported and a payload of the wrong length InvalidInput, without calling the handler.
  • A dead server releases its callers. When the server process ends, for any reason, every waiting caller returns NotFound.
  • The caller's identity is the kernel's, never a value the client sent.

What you must not do

  • Do not block in a handler. The service's other callers wait behind it. Defer the reply, and answer when the event happens.
  • Do not call your own service from its handler, directly or through another service that calls back: the only receiving thread is busy with you, and the call never returns.
  • Do not drop a reply token to cancel. Call abandon(kind). A dropped token answers ConnectionAborted and logs a server bug.
  • Do not hold more tokens than you have bounded. Refuse past your limit with an error reply, as the start gate does with ResourceBusy.
  • Do not key ownership on pid(). Use the whole CallerIdentity.
  • Do not mix interface versions. Rebuild every program that uses an interface when the interface changes.
Generated from framework/aperiodic/doc/aperiodic_framework_public.md in the TAIL OS repository.