TAIL OSv0.9.0

Shell Tools for Robot Applications

Four programs on the target shell (tsh) show what a running robot application is doing, without changing a line of it:

Tool Answers
topic Which topics exist, how fast are they published, what do they carry? Record them, print a recording, play it back.
node Which periodic nodes are running, and how is each one declared?
param What are a node's run-time parameters, and set them while it runs.
logview What did each program write to the system log, including programs that have exited?

They work the same for nodes written in Rust and in Python: each node announces itself, journals its releases and serves its parameters through the framework, so the tools see every node alike.

What each shell tool reads from a nodeA periodic node publishes its topics, read by topic; its description, read by node; and its parameters, read by param, which also writes requests back to the node. The node's log ring is read by logview.its topicsnode/__nodenode/__paramsnode/__param_setlog ringperiodic nodetopicnodeparamlogview

A node publishes its topics, its description and its parameters; topic, node and param read them, param also writes a request back, and logview reads the node's log ring.

Every tool prints usage: ... and exits 2 on a malformed command line, and exits 1 with a one-line reason on standard error when it cannot do what was asked. Output examples on this page are illustrative: process ids, counts and layout identities differ on your system.

A first session

tsh> lidar & brake & drive &
tsh> node list
brake  pid 9   brake
drive  pid 10  drive
lidar  pid 8   lidar
tsh> topic list
brake/__node        kernel   1 writer    0/8 readers       96 B   layout 0x...
brake/__releases    kernel   1 writer    0/8 readers      120 B   layout 0x...
cmd/velocity        kernel   1 writer    1/8 readers       48 B   Twist
sensor/range        safe     1 writer    1/8 readers       32 B   Range
...
tsh> topic hz -n 3 sensor/range
# times are when topic saw each publication (1 ms poll); they may not match the publisher's timing
average rate: 50.00 Hz   min: 19.0 ms   max: 21.0 ms   window: 50
average rate: 50.00 Hz   min: 19.0 ms   max: 21.0 ms   window: 50
average rate: 50.00 Hz   min: 19.0 ms   max: 21.0 ms   window: 50
tsh> topic echo -n 1 cmd/velocity
# cmd/velocity: kernel lane, Twist, 48 B, layout 0x...
linear.x=0.5 linear.y=0 linear.z=0 angular.x=0 angular.y=0 angular.z=-0.25

The topics ending in /__node, /__releases, /__params and /__param_set belong to the framework: a node's description, its release journal and its parameter channel. They are listed like any other topic.

topic

usage: topic list
       topic hz     [-n REPORTS] [--lane kernel|safe|fast] NAME
       topic echo   [-n COUNT]   [--lane kernel|safe|fast] [--as "DECL"] NAME
       topic record [-o PATH] [-d SECONDS] [NAME...]
       topic dump   FILE [--topic NAME] [-n COUNT]
       topic play   FILE [-r RATE] [-s SECONDS] NAME...

topic COMMAND --help prints the full description of one command.

topic list

One line per topic on every lane, sorted by name: the name, the lane (kernel, safe or fast), its writers, its readers out of its reader capacity, its payload size, and its type. The type is a standard payload's name (Twist, Range, Odometry, ...) when the topic's layout identity and size match one, and layout 0x... otherwise.

Listing opens nothing and changes no count. The counts can over-report:

  • a running topic hz or topic echo is an ordinary reader, and is counted as one;
  • a fast-lane reader, or a safe- or fast-lane writer, that dropped its end stays counted until its process exits.

topic hz

Measures the rate a topic is published at and prints one report a second:

average rate: 50.00 Hz   min: 19.0 ms   max: 21.0 ms   window: 50
Field Meaning
window publications in the last second, counted from the publisher's own sequence numbers, so exact even for publications the poll did not see
average rate window divided by the time between the two observations that bound it
min, max shortest and longest gap between two consecutive publications the poll saw; n/a when every change it saw spanned several publications
not seen publications counted in the window that the poll did not see one by one

hz reads as one ordinary reader, polling every 1 ms at the lowest priority. Its times are when it saw each publication, not when the publisher published it, which is what its first line says: under load max grows and a rate can read low, but window stays exact. no new messages is printed for a second with nothing new.

Option
-n REPORTS exit 0 after REPORTS reports; without it, hz runs until stopped
--lane LANE read NAME on this lane, when a topic of that name exists on more than one

hz exits 1 when the topic has no writer, or once its writer is gone.

topic echo

Prints a topic's payloads as field=value pairs. The first line names the lane, the type, the size and the layout identity. echo polls every 10 ms and prints the newest publication it has not printed yet; publications it skipped are counted on a line of their own, # missed N.

A payload is decoded by the first of these that applies:

  1. --as "DECL": a declaration in the Python Payload grammar, used only when its layout identity and size equal the topic's. Otherwise echo says why and prints hex.
    tsh> topic echo --as "stamp:u64 range_mm:u32 quality:u32" lidar/raw
    tsh> topic echo --as "linear:Vector3 angular:Vector3" cmd/velocity
    
  2. A standard payload of robot_messages, found by layout identity and size.
  3. Hex: the first 256 bytes and the total length.

-n COUNT exits 0 after COUNT printed publications; --lane is as for hz. echo exits 1 when the topic has no writer, or once its writer is gone.

topic record

Records every publication of every topic on every lane, or of the topics named, into an MCAP file, each with its lane sequence number and the time it was read. Standard MCAP tools on a development host can open the file.

tsh> topic record -o /recordings/run7.mcap &        start it BEFORE the nodes
recording to /recordings/run7.mcap
tsh> lidar & brake & drive &
Option
-o PATH write PATH, which must not exist. Without it: the next /recordings/run<N>.mcap, the directory created if absent
-d SECONDS stop after SECONDS, write each topic's counts and the file's footer, and exit 0. Without it, record runs until stopped, and the file is readable up to its last whole record
  • Start record before the nodes. A kernel-lane topic hands a late reader only the newest 8 publications it still holds, and replaying a node needs its journal from its first release.
  • Nothing is lost silently. A topic record cannot attach to (every reader seat taken, or a payload over 65489 bytes) is in the file with the reason and named on standard error. A publication overwritten before it was read is a gap in its topic's sequence, which topic dump reports.
  • record reads at priority 30, below every node, polls every 1 ms and looks for new topics every 100 ms. It exits 1 if PATH exists, or as soon as a write fails or the disk is full, saying how many bytes the file holds.

topic dump

Prints a recording: its time base, each topic where it first appears, and each publication as

RECEIPT_NS seq SEQUENCE NAME: field=value ...

Standard payloads are decoded; anything else is printed as hex, with the reason on its topic's line. Every sequence number from 1 to the last one read is either a record or accounted for on a line of its own:

# sensor/range: 12 before the recording (seq 1..12)
# sensor/range: missed 3 (seq 140..142)
# sensor/range: missed 1 after the last record (seq 2104..2104)

--topic NAME prints one topic only; -n COUNT stops after COUNT records.

topic play

Publishes again what a recording holds for the topics named: each payload byte for byte, on its recorded topic name and lane, at its recorded distance from the first one played.

tsh> topic play /recordings/run7.mcap sensor/range
playing sensor/range  safe  32 B  Range
sensor/range  safe  2104 published  0 skipped  worst lateness 0.412 ms
Option
-r RATE publish RATE times as fast, e.g. 0.5 or 2; default 1. Only the data's pace changes, never a node's period
-s SECONDS start SECONDS into the recording; earlier records are counted as skipped
  • At least one topic must be named; topic dump FILE lists what a recording holds.
  • play refuses a topic that already has a live writer: stop the node that publishes it first. This is how a node is fed its recorded inputs live, with the real sensors off.
  • A record is never published early and never skipped: one found late is published at once, and the summary reports each topic's worst lateness.
  • At -r 0.5, an input whose stale_after is under twice the topic's period reads stale on some releases; at -r 2, a node misses publications. That is the data's pace changing, not a fault.

Replaying one node

Replaying a node is not a topic command: the node itself does it, from a recording, when --replay FILE ends its command line. It runs the node's body once per recorded release against exactly what that release was handed, and compares the outputs:

tsh> brake --replay /recordings/run7.mcap
replay: brake life 1: 2104 releases, 2103 compared, identical
tsh> python steer.py --replay /recordings/run7.mcap

The periodic framework guides explain it in full: Rust, Python.

node

usage: node list | node info NODE

Every periodic node, Rust or Python, announces itself when it starts. node reads those announcements; the node does nothing extra for it.

tsh> node list
balance  pid 7  balance
tsh> node info balance
balance
  Process: pid 7 (balance), thread 0
  Period: 5ms, on_overrun Skip
  Priority: 31
  Subscribers:
    imu/data: safe, stale_after 12ms
  Publishers:
    motor/command: kernel
  Action Servers:
  Parameters: 2, settable

node list prints one line per running node: its name, its process id and its program. node info prints, for each running node of that name, its process and thread, its period and overrun policy, its priority when it started, every input with its lane and stale_after, every output with its lane, the actions it serves, and how many run-time parameters it has.

  • A node that has stopped is not listed.
  • A node replaying a recording announces nothing, so it is not listed.
  • Two running nodes with one name are both listed, with a warning: node names should be unique within an image.
  • If the image's topic storage is full, a node runs unlisted and says so once in its log: brake: no node description (topic storage full); node list does not show this node.

param

usage: param list [NODE] | param set NODE NAME VALUE [NAME VALUE...]

Lists and sets the run-time parameters a node declares, such as a controller's gains, while it runs.

tsh> param list
balance  2 parameters
tsh> param list balance
kp = 42.0  (0.0 .. 200.0)
kd = 1.5   (0.0 .. 20.0)
tsh> param set balance kp 45.5
balance: kp = 45.5 from release 18342
tsh> param list balance
kp = 45.5  (0.0 .. 200.0, default 42.0)
kd = 1.5   (0.0 .. 20.0)
tsh> param set balance kp 250
param: balance kp = 250.0 is outside 0.0 .. 200.0; nothing was set
  • A set takes effect at the start of a release. The release param set names is the first whose body sees the new value; a body never sees a value change while it runs.
  • Set coupled values in one command. param set balance kp 50 kd 2.5 is one request, applied whole at one release. Two commands are two requests, and at least one release runs between them with the new kp and the old kd.
  • The node checks every request against its declared ranges, both ends included. A value outside its range, a NaN, or an unknown name is refused, and nothing changes.
  • param set succeeds only on the node's own answer. It exits 1 with what was refused, or when the node does not answer within eight of its periods plus 250 ms.
  • Values last for the node's life. A restart starts at the defaults again; param list shows each changed value beside its default, so you can copy it into the source.
  • Each applied change is logged by param in its own log ring: balance: kp 42.0 -> 45.5 from release 18342. Read it with logview -s param.
  • param list names only nodes that declare parameters; node list names every node. A node declared safety has no run-time parameters.

How a node declares parameters: Rust, Python.

logview

usage: logview [-s <ring>] [-l <level>] [-f]
       logview console <all|info|status|warning|error|none>

Each process that logs has a log ring of its own, in memory shared with the log server. Writing a record costs no system call; logview reads the rings directly and prints their records merged by time stamp. A ring stays readable after its process exits, so the last words of a crashed program are still there.

tsh> logview
logview: 5 ring(s)
  brake            pid=9      1 page(s) live
  param            pid=14     1 page(s) dead
  ...
tsh> logview -s brake -l warning
tsh> logview -f
Option
-s RING only rings whose name starts with RING: -s wifi matches wifi_rpi3
-l LEVEL only records at LEVEL or worse: trace, info, status (also start, success), warning, error
-f follow: keep printing new records until stopped
console LEVEL set which records the log server also prints to the console: all, info, status (the default: the boot narrative and anything worse), warning, error or none

If records were overwritten in a ring before logview read them, it says how many.

What lands where:

  • A Rust program writes records with log!(INFO, "..."). The periodic framework writes a Rust node's own reports (a refused publish, a missing journal) to the node's ring, so logview -s <node> shows them.
  • A Python node reports a run of refused publishes on the console, not in a log ring, so logview does not show it.
  • param logs every applied change in its ring, param.
Generated from utility/doc/shell_tools_public.md in the TAIL OS repository.