matatu blog

Brian writes about computing

September 4, 2018

tailgrep

Following a file, grepping for lines, watching a spinner.

I sometimes use tail -f to watch a file, and also grep the output for a particular phrase.

For instance, I might look for a URL in an nginx log file. I might start a screen session, send the output to a log, and filter for phrases in a long running command. Or use script, which I recently learned about, to generate a real time log of a shell session.

In a perfect world, that's all I need. But sometimes things go wrong. The web server stops. The long running command gets killed. And tail -f | grep don't help me realize that the output has stopped.

$ tail -f /var/log/nginx/access.log | grep special
1.2.3.4 - - [29/Aug/2018 -0400] "GET /special"…
█
$ screen -L
$ script -F /tmp/out
$ echo "Read me back the last line"
$ tail -f screenlog.0 | grep Read
Read me back the last line
█
$ tail -f /tmp/out | grep Read
Read me back the last line
█

What would be nice would be a spinner to indicate that output is still being generated.

It would also be nice if the spinner stops spinning if there is no output.


/


So, here's a little program to accomplish that. I called it tailgrep.

I don't reimplement tail -- I'm too lazy to think about opening and seeking to the end of a file -- I just use make a Proc::Async object and then watch stdout.

There are some subtleties -- whenever looks like a loop but isn't. It sets up callbacks. And react declares an event loop. start puts it in a separate thread.

Also you have to get the order right: Make the Proc::Async object, then call .stdout to create a Supply. Set up your the callbacks on your event loop with whenevers. Then start the process.

It returns a Promise which you can await.

What else.

shell "tput 'civis'"; (and cnorm) make the cursor invisible (or visible). signal(SIGINT) is another Supply that is called whenever ^C is pressed (i.e. a sigint signal is received).

< > splits up a string into an array. <<...>> interpolates too. $++ maintains a stateful variable and (post-)increments it.

#!/usr/bin/env perl6

sub spinner() {
  <\ - | - / ->[$++ % 6]
}

sub MAIN($expr, $filename) {
  shell "tput 'civis'";
  my $proc = Proc::Async.new:
    <<tail -f $filename>>;
  my $out = $proc.stdout;
  start react {
    whenever $out.lines.grep( / "$expr" / ) {
      .say
    }
    whenever $out.lines {
      print spinner() ~ "\r";
    }
    whenever signal(SIGINT) {
      shell 'tput cnorm';
      exit;
    }
  }
  await $proc.start;
}

That's better:
$ tailgrep special /var/log/nginx/access.log
1.2.3.4 - - [29/Aug/2018 -0400] "GET /special"…


Oh, yeah, MAIN($expr,$filename) declares both the parameters and the command line arguments, and also generates a usage message which is displayed when you start the program with -h.

$ ./tailgrep -h
Usage:
  ./tailgrep <expr> <filename>

Not bad for 25 lines of code, but let's do a bit more.


First I want to ensure that the file exists. To do this, I add a constraint on the $filename parameter.

sub MAIN($expr, $filename where *.IO.e) {

 

Also let's write a message if there hasn't been output for a few seconds.

We make another Supply that calls a little routine every second to check for that.


my $last-seen = DateTime.now;
...
  whenever $out.lines {
    print spinner() ~ "\r";
    $last-seen = DateTime.now;
  }
  whenever Supply.interval(1) {
    if DateTime.now - $last-seen > $wait {
      say "--no lines for $wait seconds--";
    }
  }


 

And by "a few seconds" I mean whatever numeric value is given on the command line, but let's default to 2.

sub MAIN(
  $expr,                   #= what to search for
  $filename where *.IO.e,  #= a filename to grep
  Numeric :$wait = 2,      #= when to notify
) {

Did I mention that inline comments attached to the parameters become messages in the help output?

Or that named parameters (which start with a :) become named command line arguments?

./tailgrep -h
Usage:
  ./tailgrep [--wait=<Numeric>] <expr> <filename>

    <expr>              what to search for
    <filename>          a filename to grep
    --wait=<Numeric>    when to notify

That about wraps it up, but just for fun, I also decided to look for some nicer unicode spinners and add some color. For these features I needed to use a few modules that may need to be installed separately.
(zef install JSON::Fast Terminal::ANSIColor).

So, I found some spinners here, with a nice JSON file, so we just download that and store it locally.

#!/usr/bin/env perl6
use JSON::Fast;
use Terminal::ANSIColor;

my @frames = < / - | - \ - >;
sub spinner {
  @frames[$++ % +@frames];
}

sub download-spinners {
    my $store = "{ %*ENV<HOME> }/.spinners.json";
    unless $store.IO.e {
        say "downloading spinners";
        my $url='https://raw.githubusercontent.com'
          ~ '/sindresorhus/cli-spinners'
          ~ '/HEAD/spinners.json';
        shell "curl -s $url > $store";
    }
    from-json( $store.IO.slurp );
}

Also multi-dispatch is handy -- a named boolean argument for another dispatch candidate can show a list of all the spinners.

multi MAIN(
    Bool :$list-spinners!, #= list spinners
) {
    say download-spinners.keys.sort.join("\t");
}
...

Here's the final program with the fancy unicode spinners. And here's the less fancy one (no dependencies).

If you want to try them out, here are some handy urls for curling.
curl -L https://git.io/fA8MV > ~/bin/tg-fancy
curl -L https://git.io/fA8MM > ~/bin/tailgrep
Just chmod +x and there you are.
$ tg-fancy -h
Usage:
  tg-fancy --list-spinners
  tg-fancy [--wait=<Numeric>] [--spinner=<Any>] <expr> <filename>

    --list-spinners     list spinners
    <expr>              what to search for
    <filename>          a filename to grep
    --wait=<Numeric>    when to notify
    --spinner=<Any>     which spinner
$ tg-fancy --spinner=pong tailgrep access.log

$ tg-fancy --spinner=weather tailgrep access.log


Conclusions

  • Mix and match your CLI utils for fun to improve your quality of life.
  • Threads, Promises, and Supplies are handy tools for asynchronous programming.
  • Spinners are not just for fidgeting. Sometimes they are useful.

edit: 2018-09-07: updated animations