matatu blog

Brian writes about computing

Writing tmeta in Raku

Aug 20, 2020

A Better Terminal Automator

A little while ago, I had an idea to improve my terminal experience -- described here -- the idea was this:

Split the terminal in half -- the bottom will just have what is typed. The top will have the 'interactive' session.

The point is to declutter the input, and to keep a persistent local history across a variety of environments.

I've since expanded that idea into a bigger project -- tmeta.

The github project page describes most of the functionality, this article is about the implementation.

$ echo hello                 
hello                        
$ echo world                 
world                        
$                            
────────────────────────────────────────────────────
> echo hello                 
> echo world                 
> █                          
                             
                             
                                 

I'm writing this partially because of the language in which it's written -- Raku. Raku isn't widely known. So I want to write about why this language is a great fit for an application like this.

But also, this is a guide for potential contributors or users. Let me know what you think.

As a companion to this article, I've added a link from the documentation of every command to its corresponding source code. Check there for the complete versions of the snippets below.

Anyway, here are some tmeta commands and how they are implemented in Raku:

\delay, \send

Let's start with a simple one -- send -- which just sends a file, one line at a time, to the other pane. It's basically a copy-and-paste operation, but you can control how fast the lines are pasted in. This can be handy for REPLs or prompts that don't respond right away. Typing \delay will adjust the delay between lines.

The implementations of delay and send look like this

#= delay [num] -- set the delay
$*delay = val($meta.words[1])    

The $* indicates that $*delay is a dynamic variable -- more on this below.

#= send <file> -- send a file
confirm-send( $file.IO.slurp , :big);    

and confirm-send prompts (as you can see on the right) then sleeps for $*delay between calls to run --

$ date
Sun Aug 16 22:15:25 EDT 2020
$ date
Sun Aug 16 22:15:26 EDT 2020
$



────────────────────────────────────────────────────
$ cat sendme
date
date
$ tmeta
Welcome to tmeta v0.0.2
> \delay 1
> \send sendme
~> date
date

 [q to abort, e to edit (from history)]>
>    
  run <<tmux send-keys -t "$window.$pane" -l "$send">>;  

So, there's nothing deep here -- I will just mention

  • words breaks up the input using whitespace
  • val turns a Str into an IntStr
  • An IntStr is an allomorphic type, which behaves as an Int or a Str depending on the context.
  • The call to run works with quotes and interpolation and whitespace separating arguments.

\capture

Now suppose we want to capture the output to a file. We use \capture. The implementation is also simple

my $file = $meta.words[1];
tmux-start-pipe(:$*window, :$*pane, :$file);    
$ date
...
────────────────────────────────────────────────────
> \capture /tmp/dates
> \send dates
...    

and tmux-start-pipe is

 sub tmux-start-pipe(:$window,:$pane,:$file) is export {
   shell "tmux pipe-pane -t $window.$pane 'cat >> $file'";
}  

and we have more examples of dynamic variables -- $*window and $*pane -- let's talk about them --

Dynamic variables

These are a way to have a variable that has a scope that is neither global nor lexical. Think of it like a hidden parameter -- declare my $*delay lexically, and then $*delay is available any place lower in the call stack.

sub hello {
 say $*who;
}
my $*who = 'world';
hello; # prints "world";  

You can also pass it explicitly if you want. Named parameters are sent as foo => "bar", but :$foo sends the variable named foo as the value foo, and similarly :$*foo sends the dynamic variable $*foo as the foo named argument.

Why would we do that? Because in the multi-threaded world variables may be changed by other threads -- within tmux-start-pipe we want to ensure that the variable we are using is not changed before we run the shell command. This is sort of an easy intuitive way to avoid a potential race condition.

\repeat, \await

So speaking of multi threaded -- let's send a date command repeatedly until we see the string :46. This is

> date
> \repeat
> \await :46    

and is implemented like this

# repeat
%repeating{"$window.$pane"} = Supply.interval($interval).tap: {
  for @repeat {
    sendit($_,...);
  }
}    

and the await part is basically

react whenever output-stream(:$*window,:$*pane) -> $l {
    done if $l ~~ $regex;
}    

When did threaded programming become so much fun? And what are these constructs?

$ date
Sun Aug 16 22:58:31 EDT 2020
$ date
Sun Aug 16 22:58:36 EDT 2020
$ date
Sun Aug 16 22:58:41 EDT 2020
$ date
Sun Aug 16 22:58:46 EDT 2020
$
────────────────────────────────────────────────────
$ tmeta
Welcome to tmeta v0.0.2
> date
> \repeat
repeating (in @11.0) every 5 seconds: date
> \await :46
Waiting for ":46"
Done: saw ":46"
stopping @11.0
nothing queued
>    

Supplies, Taps, and Event Loops

So Supply.interval emits increasing values every $interval seconds. A tap on a supply runs whenever a value is emitted.

The output-streams use the same mechanism as tmux-start-pipe, they run in their own threads (and we keep track of the taps in a %repeating hash so that we can close the taps)

Writing react whenever $supply {...} is basically equivalent to writing $supply.tap { ... } -- code runs whenever the supply produces a value.

\find

The \find command is for searching the history. It uses fzf. The implementation looks like this:

#= find <phrase> -- Find commands in the history.
my $what = arg($meta);

my $proc = run <<fzf -e --no-sort --layout=reverse -q "$what">>, :in, :out;

$proc.in.put($_)
  for $*log-file.IO.lines.reverse.unique;

my $send = $proc.out.get or return;

confirm-send($send, :add-to-history);    
───────────────────────────────────────────
~ $ tmeta
Welcome to tmeta v0.0.2
> echo "this is the first command"
> echo "this is the second command"
> \find this is the    
───────────────────────────────────────────
> this is the █                          
\ 2/8084
  echo "this is the second command"
> echo "this is the first command"    

Here run starts fzf with a few options, and :in and :out indicate that we are going to connect to stdin and stout. Note that the <<...>> construct for quoting allow us to put quotation marks in a word and interpolate $what without any escape characters.

We just send all of the lines in our log file to fzf in reverse order, and wait for the output. This is a nice example of how easy it is to interact with external programs. The implementations of do and dosh are similar -- we spawn our own programs, and send the output to the other pane.

The shell command used to start tmux is an alternative to run that uses a shell.

Running programs

There are a few way to run programs, including run, shell, Proc::Async.new, depending on whether you want to be synchronous, asynchronous, use a shell or just spawn a program. They all turn stdin and stdout into Supplies (see above).

Finally a quick look at how we generate the documentation.

\help

Generating the docs and the inline help use the same code. Here's how it works:

There are two ways to introspect the code and get the documentation -- one is using WHY

  #| Run a command in a shell
  method shell {
     ...
  }

  for self.^methods -> $m {
     my $desc = $m.WHY.Str.trim;
  ...
}    
$
────────────────────────────────────────────────────
> \help clear
     \clear            clear this pane
> \help shell
     \shell            Run a command in a shell
> \help help
     \help             this help
>    

Calling WHY returns documentation associated with something. This is called declarator pod, since it's associated with the declaration in the code. #| means it is associated with the next thing that is declared. ( #= means the previous entity)

Declarator pod

You can go the other way too -- i.e. given the documentation, find the associated entity. The inverse of WHY is WHEREFORE.

   for @$=pod -> $p {
     my $method = $p.WHEREFORE;
     my $line = $method.line;
    ...
   }  

When the implementation is only a line or two, a method is overkill, so it's nice to look through the docs instead of looking through the methods. If there's no declaration nearby we can fall back to grepping:

without $file {
  state @lines = $?FILE.words[0].IO.lines;
  $line = @lines.first(:k, {.contains("$pod")})
  $file = $?FILE if $line;
}  

The without statement is short for if not defined, and a state variable is only initialized once. Calling the first method with :k -- a named argument, sometimes called an adverb -- means instead of returning the line, returning the key of the key-value pair -- and for arrays, the key-value pair is the index + line pair -- i.e. this returns the line number.

Conclusions

Thanks for reading this far -- or if you didn't, here's the tl;dr --

  • Raku supports a variety of scoping rules including lexical, static, and dynamic.
  • A lot of Raku methods are available to make literate programming easier and fun.
  • There are convenient idioms to make thread-safe programming more intuitive.
  • Meta programming makes writing documentation easier.
  • There are many threading primitives which are first class citizens of the language.
  • Mechanisms for spawning processes are also built in. Data streams from these programs use built in asynchronous constructs.
  • Writing something that does automation in the spirit of the expect command can make use these features.