Shell Tools for Robot Applications
Four programs on the target shell (tsh) show what a running robot application is doing,
without changing a line of it:
| Tool | Answers |
|---|---|
topic |
Which topics exist, how fast are they published, what do they carry? Record them, print a recording, play it back. |
node |
Which periodic nodes are running, and how is each one declared? |
param |
What are a node's run-time parameters, and set them while it runs. |
logview |
What did each program write to the system log, including programs that have exited? |
They work the same for nodes written in Rust and in Python: each node announces itself, journals its releases and serves its parameters through the framework, so the tools see every node alike.
A node publishes its topics, its description and its parameters; topic, node and
param read them, param also writes a request back, and logview reads the node's log
ring.
Every tool prints usage: ... and exits 2 on a malformed command line, and exits 1 with a
one-line reason on standard error when it cannot do what was asked. Output examples on this
page are illustrative: process ids, counts and layout identities differ on your system.
A first session
tsh> lidar & brake & drive &
tsh> node list
brake pid 9 brake
drive pid 10 drive
lidar pid 8 lidar
tsh> topic list
brake/__node kernel 1 writer 0/8 readers 96 B layout 0x...
brake/__releases kernel 1 writer 0/8 readers 120 B layout 0x...
cmd/velocity kernel 1 writer 1/8 readers 48 B Twist
sensor/range safe 1 writer 1/8 readers 32 B Range
...
tsh> topic hz -n 3 sensor/range
# times are when topic saw each publication (1 ms poll); they may not match the publisher's timing
average rate: 50.00 Hz min: 19.0 ms max: 21.0 ms window: 50
average rate: 50.00 Hz min: 19.0 ms max: 21.0 ms window: 50
average rate: 50.00 Hz min: 19.0 ms max: 21.0 ms window: 50
tsh> topic echo -n 1 cmd/velocity
# cmd/velocity: kernel lane, Twist, 48 B, layout 0x...
linear.x=0.5 linear.y=0 linear.z=0 angular.x=0 angular.y=0 angular.z=-0.25
The topics ending in /__node, /__releases, /__params and /__param_set belong to
the framework: a node's description, its release journal and its parameter channel. They
are listed like any other topic.
topic
usage: topic list
topic hz [-n REPORTS] [--lane kernel|safe|fast] NAME
topic echo [-n COUNT] [--lane kernel|safe|fast] [--as "DECL"] NAME
topic record [-o PATH] [-d SECONDS] [NAME...]
topic dump FILE [--topic NAME] [-n COUNT]
topic play FILE [-r RATE] [-s SECONDS] NAME...
topic COMMAND --help prints the full description of one command.
topic list
One line per topic on every lane, sorted by name: the name, the lane (kernel, safe or
fast), its writers, its readers out of its reader capacity, its payload size, and its
type. The type is a standard payload's name (Twist, Range, Odometry, ...) when the
topic's layout identity and size match one, and layout 0x... otherwise.
Listing opens nothing and changes no count. The counts can over-report:
- a running
topic hzortopic echois an ordinary reader, and is counted as one; - a fast-lane reader, or a safe- or fast-lane writer, that dropped its end stays counted until its process exits.
topic hz
Measures the rate a topic is published at and prints one report a second:
average rate: 50.00 Hz min: 19.0 ms max: 21.0 ms window: 50
| Field | Meaning |
|---|---|
window |
publications in the last second, counted from the publisher's own sequence numbers, so exact even for publications the poll did not see |
average rate |
window divided by the time between the two observations that bound it |
min, max |
shortest and longest gap between two consecutive publications the poll saw; n/a when every change it saw spanned several publications |
not seen |
publications counted in the window that the poll did not see one by one |
hz reads as one ordinary reader, polling every 1 ms at the lowest priority. Its times are
when it saw each publication, not when the publisher published it, which is what its first
line says: under load max grows and a rate can read low, but window stays exact.
no new messages is printed for a second with nothing new.
| Option | |
|---|---|
-n REPORTS |
exit 0 after REPORTS reports; without it, hz runs until stopped |
--lane LANE |
read NAME on this lane, when a topic of that name exists on more than one |
hz exits 1 when the topic has no writer, or once its writer is gone.
topic echo
Prints a topic's payloads as field=value pairs. The first line names the lane, the type,
the size and the layout identity. echo polls every 10 ms and prints the newest
publication it has not printed yet; publications it skipped are counted on a line of their
own, # missed N.
A payload is decoded by the first of these that applies:
--as "DECL": a declaration in the PythonPayloadgrammar, used only when its layout identity and size equal the topic's. Otherwiseechosays why and prints hex.tsh> topic echo --as "stamp:u64 range_mm:u32 quality:u32" lidar/raw tsh> topic echo --as "linear:Vector3 angular:Vector3" cmd/velocity- A standard payload of
robot_messages, found by layout identity and size. - Hex: the first 256 bytes and the total length.
-n COUNT exits 0 after COUNT printed publications; --lane is as for hz. echo exits
1 when the topic has no writer, or once its writer is gone.
topic record
Records every publication of every topic on every lane, or of the topics named, into an MCAP file, each with its lane sequence number and the time it was read. Standard MCAP tools on a development host can open the file.
tsh> topic record -o /recordings/run7.mcap & start it BEFORE the nodes
recording to /recordings/run7.mcap
tsh> lidar & brake & drive &
| Option | |
|---|---|
-o PATH |
write PATH, which must not exist. Without it: the next /recordings/run<N>.mcap, the directory created if absent |
-d SECONDS |
stop after SECONDS, write each topic's counts and the file's footer, and exit 0. Without it, record runs until stopped, and the file is readable up to its last whole record |
- Start
recordbefore the nodes. A kernel-lane topic hands a late reader only the newest 8 publications it still holds, and replaying a node needs its journal from its first release. - Nothing is lost silently. A topic
recordcannot attach to (every reader seat taken, or a payload over 65489 bytes) is in the file with the reason and named on standard error. A publication overwritten before it was read is a gap in its topic's sequence, whichtopic dumpreports. recordreads at priority 30, below every node, polls every 1 ms and looks for new topics every 100 ms. It exits 1 if PATH exists, or as soon as a write fails or the disk is full, saying how many bytes the file holds.
topic dump
Prints a recording: its time base, each topic where it first appears, and each publication as
RECEIPT_NS seq SEQUENCE NAME: field=value ...
Standard payloads are decoded; anything else is printed as hex, with the reason on its topic's line. Every sequence number from 1 to the last one read is either a record or accounted for on a line of its own:
# sensor/range: 12 before the recording (seq 1..12)
# sensor/range: missed 3 (seq 140..142)
# sensor/range: missed 1 after the last record (seq 2104..2104)
--topic NAME prints one topic only; -n COUNT stops after COUNT records.
topic play
Publishes again what a recording holds for the topics named: each payload byte for byte, on its recorded topic name and lane, at its recorded distance from the first one played.
tsh> topic play /recordings/run7.mcap sensor/range
playing sensor/range safe 32 B Range
sensor/range safe 2104 published 0 skipped worst lateness 0.412 ms
| Option | |
|---|---|
-r RATE |
publish RATE times as fast, e.g. 0.5 or 2; default 1. Only the data's pace changes, never a node's period |
-s SECONDS |
start SECONDS into the recording; earlier records are counted as skipped |
- At least one topic must be named;
topic dump FILElists what a recording holds. playrefuses a topic that already has a live writer: stop the node that publishes it first. This is how a node is fed its recorded inputs live, with the real sensors off.- A record is never published early and never skipped: one found late is published at once, and the summary reports each topic's worst lateness.
- At
-r 0.5, an input whosestale_afteris under twice the topic's period reads stale on some releases; at-r 2, a node misses publications. That is the data's pace changing, not a fault.
Replaying one node
Replaying a node is not a topic command: the node itself does it, from a recording, when
--replay FILE ends its command line. It runs the node's body once per recorded release
against exactly what that release was handed, and compares the outputs:
tsh> brake --replay /recordings/run7.mcap
replay: brake life 1: 2104 releases, 2103 compared, identical
tsh> python steer.py --replay /recordings/run7.mcap
The periodic framework guides explain it in full: Rust, Python.
node
usage: node list | node info NODE
Every periodic node, Rust or Python, announces itself when it starts. node reads those
announcements; the node does nothing extra for it.
tsh> node list
balance pid 7 balance
tsh> node info balance
balance
Process: pid 7 (balance), thread 0
Period: 5ms, on_overrun Skip
Priority: 31
Subscribers:
imu/data: safe, stale_after 12ms
Publishers:
motor/command: kernel
Action Servers:
Parameters: 2, settable
node list prints one line per running node: its name, its process id and its program.
node info prints, for each running node of that name, its process and thread, its period
and overrun policy, its priority when it started, every input with its lane and
stale_after, every output with its lane, the actions it serves, and how many run-time
parameters it has.
- A node that has stopped is not listed.
- A node replaying a recording announces nothing, so it is not listed.
- Two running nodes with one name are both listed, with a warning: node names should be unique within an image.
- If the image's topic storage is full, a node runs unlisted and says so once in its log:
brake: no node description (topic storage full); node list does not show this node.
param
usage: param list [NODE] | param set NODE NAME VALUE [NAME VALUE...]
Lists and sets the run-time parameters a node declares, such as a controller's gains, while it runs.
tsh> param list
balance 2 parameters
tsh> param list balance
kp = 42.0 (0.0 .. 200.0)
kd = 1.5 (0.0 .. 20.0)
tsh> param set balance kp 45.5
balance: kp = 45.5 from release 18342
tsh> param list balance
kp = 45.5 (0.0 .. 200.0, default 42.0)
kd = 1.5 (0.0 .. 20.0)
tsh> param set balance kp 250
param: balance kp = 250.0 is outside 0.0 .. 200.0; nothing was set
- A set takes effect at the start of a release. The release
param setnames is the first whose body sees the new value; a body never sees a value change while it runs. - Set coupled values in one command.
param set balance kp 50 kd 2.5is one request, applied whole at one release. Two commands are two requests, and at least one release runs between them with the newkpand the oldkd. - The node checks every request against its declared ranges, both ends included. A value outside its range, a NaN, or an unknown name is refused, and nothing changes.
param setsucceeds only on the node's own answer. It exits 1 with what was refused, or when the node does not answer within eight of its periods plus 250 ms.- Values last for the node's life. A restart starts at the defaults again;
param listshows each changed value beside its default, so you can copy it into the source. - Each applied change is logged by
paramin its own log ring:balance: kp 42.0 -> 45.5 from release 18342. Read it withlogview -s param. param listnames only nodes that declare parameters;node listnames every node. A node declaredsafetyhas no run-time parameters.
How a node declares parameters: Rust, Python.
logview
usage: logview [-s <ring>] [-l <level>] [-f]
logview console <all|info|status|warning|error|none>
Each process that logs has a log ring of its own, in memory shared with the log server.
Writing a record costs no system call; logview reads the rings directly and prints
their records merged by time stamp. A ring stays readable after its process exits, so the
last words of a crashed program are still there.
tsh> logview
logview: 5 ring(s)
brake pid=9 1 page(s) live
param pid=14 1 page(s) dead
...
tsh> logview -s brake -l warning
tsh> logview -f
| Option | |
|---|---|
-s RING |
only rings whose name starts with RING: -s wifi matches wifi_rpi3 |
-l LEVEL |
only records at LEVEL or worse: trace, info, status (also start, success), warning, error |
-f |
follow: keep printing new records until stopped |
console LEVEL |
set which records the log server also prints to the console: all, info, status (the default: the boot narrative and anything worse), warning, error or none |
If records were overwritten in a ring before logview read them, it says how many.
What lands where:
- A Rust program writes records with
log!(INFO, "..."). The periodic framework writes a Rust node's own reports (a refused publish, a missing journal) to the node's ring, sologview -s <node>shows them. - A Python node reports a run of refused publishes on the console, not in a log ring, so
logviewdoes not show it. paramlogs every applied change in its ring,param.