Today I Learned

A Hashrocket project

227 posts about #command-line

asdf Global Versions 🌏

Spend a bit of time with asdf, and you might see an error like this:

$ npm install
asdf: No version set for command npm
you might want to add one of the following in your .tool-versions file:

nodejs 10.15.3
nodejs 8.9.1

The project I’m trying to set up via npm install doesn’t specify a Node version in .tool-versions, and since I have multiple Nodes on my machine, asdf isn’t sure which to use.

I don’t want to edit .tool-versions in this project; I’d rather asdf had a global default.

Here’s how I made that happen:

$ asdf global node 10.15.3   

How to clear a Mac terminal and its scroll-back?

Just type this: clear && printf '\e[3J'

Or even better create an alias for that, here’s mine:

alias clear='clear && printf "\e[3J"';

Here’s what I’ve learned today:

On Mac a regular clear is pretty much the same as typing Control + L on iTerm2. This clears the screen what’s good but sometimes I want to clear all the scroll-back to clean the noise and find things faster.

In order to clean the scroll-back I was performing a Command + K. This cleans the screen and the scroll-back. That’s great, except that it messes up with tmux rendering and tmux holds all the scroll-back per pane, so it won’t work at all.

So my new alias solves that as after clear the screen it also sends a terminal command to reset the scroll back through printf '\e[3J' and this keeps tmux working just fine!

Print Calendar With Week Number 📅

I use the week number (1-52) in my notetaking. How do I know what week number it is? ncal.

$ ncal -w
    May 2019
Mo     6 13 20 27
Tu     7 14 21 28
We  1  8 15 22 29
Th  2  9 16 23 30
Fr  3 10 17 24 31
Sa  4 11 18 25
Su  5 12 19 26
   18 19 20 21 22

Though not shown in the output above, today’s date (the 10th) is highlighted in my terminal output. At the bottom of the column containing today’s date is the week number (19).

Don't truncate when redirecting

A common problem in shell languages is truncating a file that you’re trying to transform by redirecting into it.

Let’s say I have a file full of “a”‘s

PROMPT> cat test.txt
aaaaaa

And I want to switch them all to “b”‘s

PROMPT> cat test.txt | tr 'a' 'b'
bbbbbb

But when I try to redirect into the same file I truncate file:

PROMPT> cat test.txt | tr 'a' 'b' > test.txt
PROMPT> cat test.txt
# nothing

This is because the shell truncates “test.txt” for redirection before cat reads the file.

An alternate approach is to use tee.

PROMPT> cat test.txt | tr 'a' 'b' | tee test.txt
bbbbbb
PROMPT> cat test.txt
bbbbbb

tee will write the output to both the file and to standard out, but will do so at the end of the pipe, after the cat command reads from the file.

Delete lines from file with sed

Imagine the following file:

sed.test

hised
hellosed
goodbyesed

If you want to delete a line matching a regular expression (e.g. hellosed), you can use d at the end of your regular expression.

sed '/hellosed/d' sed.test

Output:

hised
goodbyesed

However the file did not change:

cat sed.test

hised
hellosed
goodbyesed

To write the file in place use the -i [suffix] option. This argument allows you to specify the suffix of the backup file to be saved before committing your changes. For example:

sed -i '.bak' '/hellosed/d' sed.test

Now the file will be modified with our changes but we will also get a backup of the original file in sed.test.bak.

If you like living on the edge 🛩, and don’t want those pesky backup files littering your system, you can supply -i with an empty suffix, causing no backup file to be saved.

sed -i '' '/hellosed/d' sed.test

killall 💀

Today while pairing I learned about the killall command.

This BSD program kills processes by name, unlike kill, which uses PID.

Here’s me killing my Mac calculator:

$ ps aux | grep Calc
grep Calc
/Applications/Calculator.app/Contents/MacOS/Calculator
$ killall Calculator
$ ps aux | grep Calc
grep Calc

Thanks for the tip, Mark!

Two arguments in a command with xargs and bash -c

You can use substitution -I{} to put the argument into the middle of the command.

> echo "a\nb\nc\nd" | xargs -I{} echo {}!
a!
b!
c!
d!

I can use -L2 to provide exactly 2 arguments to the command:

> echo "a\nb\nc\nd" | xargs -I{} -L2 echo {}!
a b!
c d!

But I want to use two arguments, the first in one place, the next in another place:

> echo "a\nb\nc\nd" | xargs -I{} -L2 echo {}x{}!
a bxa b!
c dxa b!

I wanted axb! but got a bxa b!. In order to achieve this you have to pass arguments to a bash command.

> echo "a\nb\nc\nd" | xargs -L2 bash -c 'echo $0x$1!'
axb!
cxd!

Just like calling

bash -c 'echo $0x$1!' a b

Where $0 represents the first argument and $1 represents the second argument.

Parsing CSV at the command line with `csvcut`

Parsing csv at the command line is easy with the csvcut tool from csvkit.

csvkit is installable with pip

> pip install csvcut 

You can print only the columns you are interested in.

> echo "a,b,c,d" | csvcut -c 1,4
a,d
> echo "a,b,c,d" | csvcut -c 1,5
Column 5 is invalid. The last column is 'd' at index 4.

It also handles quoted csv columns:

echo 'a,"1,2,3,4",c,d' | csvcut -c 2,3

It handles new lines:

> echo 'a,b\nc,d' | csvcut -c 2
b
d

There are a virtual plethora of options check out csvcut --help.

csvkit has a number of csv processing tools, check them out here.

Get pip installed executables into the asdf path

Given I am using asdf when I install a python executable via pip, then I expect to be able to run that executable at the command line.

> asdf global python 3.7.2
> pip install some-executable-package
> some-executable
zsh: command not found: some-executable

This is because a shim has not yet been created for this executable. If you look into your shims dir for the executable:

> ls $(dirname $(which pip)) | grep some-executable

It doesn’t exist.

To place a shim into the shims dir of asdf you must reshim.

> asdf reshim python

And now you have a shim that points to the executable:

> ls $(dirname $(which pip)) | grep some-executable
some-executable
> some-executable
Hello World!

Confirming operations with `xargs -p`

xargs is a great tool to take a lot of input and execute a lot of different commands based on that input. Sometimes though, if you are performing destructive or mutative actions with xargs you want to proceed more cautiosly.

> echo "banana apple orange" | tr ' ' '\n' | xargs -n1 echo "I like"

This outputs:

I like banana
I like apple
I like orange

But maybe I don’t like some of those things, please ask! Including the p flag with xargs forces a prompt.

> echo "banana apple orange" | tr ' ' '\n' | xargs -p -n1 echo "I like"
echo I like banana ?...n
echo I like apple ?...y
I like apple
echo I like orange ?...n

Yep, I only like apples.

Get first image of animated gif

Image Magick’s convert tool has a no-option, very simple way to access the first frame of an animated gif.

convert 'animated.gif[0]' animated.first.gif

The square brackets after the file name above can contain any index for any frame of the image. 0 is the index of the first image.

To discover how many frames an animated gif has you can use:

identify animated.gif

Which will return a line for every frame in the animated gif. Those lines will look like this:

animated.gif[32] GIF 736x521 756x594+4+70 8-bit sRGB 256c 421707B 0.000u 0:00.000

What is my Tmux Previous Window?

Today I noticed a symbol in the Tmux bar that I had not noticed before, the dash (-) next to frontend:

0:code* 2:api  3:frontend-

In a Tmux session, asterisk (*) is the window in focus. The dash is the previous window, which you can easily return to. From window 0, code, I can return window 1, frontend, using CTRL + bb.

Send Tmux Pane to Window

A scenario I find myself in frequently: I’ve started a server in a Tmux pane, and realize I don’t need to see the server logging in my ‘home’ Tmux pane (pane 0 for me).

To send a Tmux pane to its own window, use :break-pane.

A nice addition is the -n flag, which lets you set the new window name while breaking the pane.

:break-pane -n frontend-elm

Resize App Windows With AppleScript

I showed in a previous TIL how we can run AppleScript commands inline from the terminal. Here is an inline command for positioning and resizing your iTerm2 window.

osascript -e 'tell application "iTerm2"
  set the bounds of the first window to {50, 50, 1280, 720}
end tell'

The first two values tell the command the x and y coordinates of where to position the upper left corner of the window relative to the upper left corner of your screen. The next two values are the width and height that the window should be resized to.

source

Get Matching Filenames As Output From Grep

Standard use of the grep command outputs the lines that match the specified pattern. You can instead output just the names of the files where those matches occur. To do this, include the -l flag.

$ grep -Rl hashrocket .
./elixir/run-exunit-tests-in-a-deterministic-order.md
./git/show-file-diffs-when-viewing-git-log.md
./git/single-key-presses-in-interactive-mode.md
./internet/enable-keyboard-shortcuts-in-gmail.md
...

This recursive grep finds all the files where hashrocket appears. It only looks for the first match in a file, so each file will only be listed once even if there may have been multiple matches.

See man grep for more details.

Configure cd To Behave Like pushd In Zsh

The Zsh environment has a setting that allows you to make the cd command behave like the pushd command. Normally when you use cd the remembered directory stack is not effected. However, if you add the following setting to your ~/.zshrc file:

setopt auto_pushd

then using cd to navigate directories will cause those directories to be added to the dirs stack.

This is the default in the oh-my-zsh configuration of zsh.

List The Stack Of Remembered Directories

When you open a new Unix shell, you start in some directory, probably your home (~/) directory. If you use pushd to navigate to different directories, there is a paper trail of your movements, a listing of where you’ve been. You can view this listing of directories with the dirs command.

$ dirs
~/
$ pushd code
$ dirs
~/code ~/
$ pushd /usr/bin
$ dirs
/usr/bin ~/code ~/

Each time you pushd, the directory you have moved to is pushed onto the stack of visited directories. Alternatively, you can use the popd command to return to the previous directory, removing the current directory from the stack.

source

Parallel shell processing with xargs

Today I learned how to parallel run a slow command on my shell. We can use xargs combined with the flags -n and -P flags. Let’s see how this works:

find . -type f | xargs -n1 -P8 slow_command
  • slow_command your slow command that receives a file as the first arg
  • -n to specify how many arguments are passed to the slow_command
  • -P how many parallel workers xargs will spawn to run the slow_command

Check this out watch -d -n 0.1 "seq 10 | xargs -n2 -P8 echo":

watch-xargs

On this example xargs are spawning up to 8 workers to run the echo command and for each echo execution xargs will pass 2 arguments. The arguments are produced by a seq 10 and as multiple executions of echo runs in parallel we can highlight the output changes with watch.

Remotely control your desktop over ssh on macOS

Using ssh port forwarding and vnc you can connect to your remote desktop using the Screen Sharing application.

First connect to your machine over ssh and port forward 5900.

ssh user@machine.somehwere -L 5900:localhost:5900

Then in another terminal, on your local machine, open Screen Sharing by passing a open a vnc url.

open 'vnc://localhost'

Screen Sharing should open and ask you for credentials for the remote machine. And then you do cool things on your remote desktop!

Read more about VNC here in this wikipedia article.

Hat Tip to Dorian Karter!

Capture and View screenshot on macOS remotely

First, wake up the desktop with caffeinate.

caffeinate -u -t 2 # assert that the user is active

Then switch to the application you want to have focus on the desktop with open

open -a Google\ Chrome

Then call MacOS’s screencapture command.

sudo screencapture /Users/dev/Desktop/FullScreen.png

Finally, if you are daring, using iTerm2, have downloaded imgcat from the iTerm imgcat site, have chmod +x that file, and have copied that file to /usr/local/bin, then view the captured image in your terminal with imgcat.

imgcat /Users/dev/Desktop/FullScreen.png

SSH config Include

Today I learned that’s ssh config has an Include directive to load other ssh configs.

Now I can split my ssh configs between ~/.ssh/config and ~/.ssh/config.local.

That’s my current ~/.ssh/config:

Host github.com
  HostName github.com
  IdentityFile ~/.ssh/github_id_rsa

Include ~/.ssh/config.local

Delimiters for sed shell command

Today I learned that the sed shell command accepts other delimiters than /. I was trying to run the following:

folder="screens/components"
echo "~/FOLDER/index.js" | sed -e "s/FOLDER/${folder}/g"
# sed: 1: "s/{{folder}}/screens/co ...": bad flag in substitute command: 'c'

But I got a bad flag error. So I changed my delimiter to | and all works fine:

folder="screens/components"
echo "~/FOLDER/index.js" | sed -e "s|FOLDER|${folder}|g"

Quiet noisy ssh port forwarding errors

When you are connecting via ssh to another machine and portfowarding like:

ssh person@machine.name -L 8000:localhost:8000

And there is no server running on port 8000, then you might be getting errors like:

channel 2: open failed: connect failed: Connection refused

If this is the case, you can add the -q flag to your ssh command. The ssh man page documents -q as:

-q      Quiet mode.  Causes most warning and diagnostic messages to be

So the whole ssh command would look like:

ssh person@machine.name -L 8000:localhost:8000 -q

Hopefully this solves your problem!

H/T Brian Dunn

Easily delete a long word in terminal

When you type that long feature branch name or URL but then want to delete it you have a few options, but hopefully you don’t hold your backspace/delete key until every character has been deleted. That would take too long.

I recently found a new shortcut other than the well-known ctrl-w. If you press the ESC key followed by backspace the last word will be deleted!

demo

Pretty Print JSON responses from `curl` - Part 3

If you thought that the output was pretty enough from last TIL, you were wrong.

Dennis Carlsson tweeted me about a tool called bat that has automatically syntax highlighting for a bunch of different languages and can also display line numbers.

Just pipe bat after jq and you are good to go:

> curl 'https://til.hashrocket.com/api/developer_posts.json?username=gabrielreis' | jq  | bat

       │ STDIN
───────┼──────────────────────────────────────────────────────────────────────────────
   1   │ {
   2   │   "data": {
   3   │     "posts": [
   4   │       {
   5   │         "title": "Pretty Print JSON responses from `curl` - Part 2",
   6   │         "slug": "utpch45mba"
   7   │       },
   8   │       {
   9   │         "title": "Pretty Print JSON responses from `curl`",
  10   │         "slug": "pgyjvtuwba"
  11   │       },
  12   │       {
  13   │         "title": "Display line break content in React with just CSS",
  14   │         "slug": "mmzlajavna"
  15   │       }
  16   │     ]
  17   │   }
  18   │ }

If you know any other tricks on making stdout prettier I would love to learn them.

Pretty Print JSON responses from `curl` - Part 2

After posting my last TIL , Vinicius showed me another tool that goes beyond just pretty printing: jq

If you don’t pass any args to jq it will just pretty print same as json_pp:

> curl 'https://til.hashrocket.com/api/developer_posts.json?username=gabrielreis' | jq

{
  "data": {
    "posts": [
      {
        "title": "Pretty Print JSON responses from `curl`",
        "slug": "pgyjvtuwba"
      },
      {
        "title": "Display line break content in React with just CSS",
        "slug": "mmzlajavna"
      },
      {
        "title": "Mutations with the graphql-client Ruby gem",
        "slug": "xej7xtsnit"
      }
    ]
  }
}

What if you only want to display the first post on the response? Just pass an argument to filter the keys you want. It’s like Xpath for JSON: jq '.data.posts[0]'

> curl 'https://til.hashrocket.com/api/developer_posts.json?username=gabrielreis' | jq '.data.posts[0]'

{
  "title": "Pretty Print JSON responses from `curl`",
  "slug": "pgyjvtuwba"
}

See Part 3

Pretty Print JSON responses from `curl`

When you use curl to manually make API calls, sometimes the response is not formatted:

> curl 'https://til.hashrocket.com/api/developer_posts.json?username=gabrielreis'`

{"data":{"posts":[{"title":"Display line break content in React with just CSS","slug":"mmzlajavna"},{"title":"Mutations with the graphql-client Ruby gem","slug":"xej7xtsnit"},{"title":"The rest of keyword arguments 🍕","slug":"o2wiclcyjf"}]}}%

You can pipe json_pp at the end so you have a prettier json response:

> curl 'https://til.hashrocket.com/api/developer_posts.json?username=gabrielreis' | json_pp

{
   "data" : {
      "posts" : [
         {
            "slug" : "mmzlajavna",
            "title" : "Display line break content in React with just CSS"
         },
         {
            "title" : "Mutations with the graphql-client Ruby gem",
            "slug" : "xej7xtsnit"
         },
         {
            "title" : "The rest of keyword arguments 🍕",
            "slug" : "o2wiclcyjf"
         }
      ]
   }
}

See Part 2

Resize Tmux Pane 🖥

Sometimes, after a long day of coding, I resize a Tmux pane using my mouse instead of my keyboard. It’s a habit from my GUI-informed past.

Here’s how to accomplish this without a mouse.

To resize the focused pane left one cell (from the Tmux prompt):

:resize-pane -L

Resize pane number 3 right 10 cells:

:resize-pane -t 3 -R 10

Etc.

Change Prompt in Z Shell

When I live code, or share terminal commands in a demonstration, I don’t want my customized terminal prompt included in that information. It’s noisy.

Right now I’m changing this in Z Shell via that PROMPT variable.

# Complex prompt
jake@computer-name: echo $PROMPT
%{%}%n@%m%{%}:

# Simple prompt
jake@computer-name: PROMPT="$ "
$ echo 'ready to live code'
ready to live code
$

Pipe all output from a command (stderr & stdout)

When you write a bash/zsh script relying on pipes normally you will not be able to pipe through text from the stderr output with a normal pipe.

For example, curl -v prints some information about the request, including it’s headers and status into stderr.

If we simply try to pipe the output of curl -v into less we will not see the verbose header and request info:

curl -v https://hashrocket.com | less

Output:

<html lang='en-US'>
<meta charset='UTF-8'>
<title>Ruby on Rails, Elixir, React, mobile design and development | Hashrocket</title>
...

But if we want the stderr output as well we can use the |& syntax:

curl -v https://hashrocket.com |& less

Output:

* Rebuilt URL to: https://hashrocket.com/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
...
* Connected to hashrocket.com (52.55.191.55) port 443 (#0)
...
<html lang='en-US'>
<meta charset='UTF-8'>
...

🍒 Bonus:

We can also pipe through ONLY the stderr:

curl -v https://hashrocket.com |1>& less

Output (will not contain the html response):

* Rebuilt URL to: https://hashrocket.com/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
...

h/t Thomas Allen

Get ONLY PIDs for processes listening on a port

The lsof utility on Linux is useful among other things for checking which process is listening on a specific port.

If you need to kill all processes listening on a particular port, normally you would reach for something like awk '{ print $2 }', but that would fail to remove the PID column header, so you would also need to pipe through tail -1. It get pretty verbose for something that should be pretty simple.

Fortunatly, lsof provides a way to list all the pids without the PID header specifically so you can pipe the output to the kill command.

The -t flag removes everything from the output except the pids of the resulting processes from your query.

In this example I used a query to return all processes listening on port 3000 and return their PID:

lsof -ti tcp:3000

The output of which will look something like:

6540
6543
21715

This is perfect for piping into kill using xargs:

lsof -ti tcp:3000 | xargs kill

No awks or tails necessary! 🐕

Find The Process Using A Specific Port On Mac

The netstat utility is often recommended for finding the PID (process ID) bound to a specific port. Unfortunately, Mac’s version of netstat does not support the -p (process) flag. Instead, you’ll want to use the lsof utility.

$ sudo lsof -i tcp:4567

Running this will produce a nicely formatted response that tells you several pieces of information about the process bound to :4567 including the PID.

source

Grep For Files With Multiple Matches

The grep utility is a great way to find files that contain a certain pattern:

$ grep -r ".class-name" src/css/

This will recursively look through all the files in your css directory to find matches of .class-name.

Often times these kinds of searches can turn up too many results and you’ll want to pare it back by providing some additional context.

For instance, we may only want results where @media only screen also appears, but on a different line. To do this, we need to chain a series of greps together.

$ grep -rl "@media only screen" src/css |
    xargs grep -l ".class-name"

This will produce a list of filenames (hence the -l flag) that contain both a line with @media only screen and a line with .class-name.

If you need to, chain more grep commands on to narrow things down even farther.

See man grep for more details.

Zsh file name without the extension

Zsh provides a weird way to get the different parts a of a file name.

If you want the full path without the extension:

> myfile=/path/to/story.txt
> echo ${myfile:r}
/path/to/story
> myfile=story.txt
> echo ${myfile:r}
story

If you want just the file name minus the path:

> myfile=/path/to/story.txt
> echo ${myfile:t}
story.txt

Check this out you can combine those two symbols!

> myfile=/path/to/story.txt
> echo ${myfile:t:r}
story

Copy files with progress in terminal w/rsync

When you need to transfer a lot of files from one location to another it’s sometimes useful to have some progress indication and maybe even a speed measure, or time remaining.

I recently had to transfer a few gigabytes of data from one computer to another. For this task I chose to use Rsync, since it is a command line utility that can preserve file metadata (permissions) and easily resume in case of an error.

Rsync ships with macOS by default, but if you want to get a more recent version, you can install it from homebrew.

There are two options for showing progress:

If you are transferring a few really big files you can use the --progress flag.

rsync -ah --progress source destination

This will list each file as it being transferred and show you the progress and speed in which the file is being transferred.

In my case I had a lot of small files so I chose to use --info=progress2.

rsync -ah --info=progress2 source destination

This will output something like this

2.26G  16%    6.13MB/s    0:05:51 (xfr#375313, to-chk=0/1165396)

Which represents the progress, speed and estimated time remaining for the entire transfer.

List Stats For A File

The ls command is good for listing files. Tacking on the -la flags gives you a bunch of info about each of the listed files. To get even more info, we can use the stat command.

$ stat README.md
16777220 143994676 -rw-r--r-- 1 jbranchaud staff 0 53557 "Jul 14 14:53:44 2018" "Jul 10 14:54:39 2018" "Jul 10 14:54:39 2018" "Jul 10 14:54:39 2018" 4096 112 0 README.md

That’s definitely more info, but it is unlabeled and a lot to parse. We can improve the output with the -x flag.

$ stat -x README.md
  File: "README.md"
  Size: 53557        FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/jbranchaud)  Gid: (   20/   staff)
Device: 1,4   Inode: 143994676    Links: 1
Access: Sat Jul 14 14:53:44 2018
Modify: Tue Jul 10 14:54:39 2018
Change: Tue Jul 10 14:54:39 2018

See man stat for more details.

source

Show escaped bash color codes in less #linux

My ls command colors directories and files according to their type and permissions:

ls with color

But when the window is too small to fit the content I pipe the result into less:

less broken

Which cannot correctly parse the escape code from ls and turn them into color. To fix that add -r to the less command:

solution

Notes:

My l alias is gls -F -G --color --group-directories-first -lah (gls is GNU ls)

You can alias less=less -r if you want this to be the default behavior for less.

Delete all node_modules dirs recursively with find

If you have hundreds of past JavaScript projects sitting in your workspace folder, you probably also have hundreds of node_modules folders nested inside of them, and hundreds of thousands actual npm modules resting peacefully in those.

Often enough all you care about is the code that uses the modules and not the modules themselves, so to save yourself some precious laptop diskspace you can just delete all those folders! When you need them again cd into the project directory and run yarn install or npm install.

First let’s do a dry run:

find . -name "node_modules" -type d -prune

and now that you checked the output of the above command you can delete all the nested node_module folders.

If you are still feeling paranoid (and you’re on macOS) you can simply move those to the Trash:

find . -name "node_modules" -type d -prune -exec trash '{}' +

If you feel a little braver just unlatch the airlock and toss them into a black hole 🕳 using rm -rf

find . -name "node_modules" -type d -prune -exec rm -rf '{}' +

I saved a whopping 80GB with this technique 🤑. Hope you find it helpful.

Generate Zeropadded Ranges

Need to generate 100 directories, named 01/ to 99/? Today I learned that command line brace expansion supports zeropadded (starting with one or more zeroes) ranges. The following command will create 100 zeropadded, numbered directories:

$ mkdir {01..99}

Hit tab to see the expanded command.

The second zeropad, if there is one, can be omitted— the following creates a range of 01-05, even thought there’s no zero in front of the 5:

$ mkdir {01..5}


Which expands to:

$ mkdir 01 02 03 04 05