Today I Learned

A Hashrocket project

138 posts about #command-line

Override ssh command for git

Git supports a number of environment variables one of them being GIT_SSH. You can override the default ssh command (ssh of course) with that environment variable.

GIT_SSH=./my_ssh git pull origin master

In this case my_ssh is a bash executable file that looks like this:

#!/bin/bash

ssh -i ~/.ssh/id_rsa $1 $2

An idea I got from this stack overflow answer.

Clearly the private key file I used is the default private key file so a bit redundant but its a way to add any options that might be necessary for the ssh connection, although specifying a host configuration in the ssh_config file might be simpler.

Specifying the private key to use in ssh

A private key is necessary in ssh to authenticate the connection securely. Generally ssh will look for a file in the ~/.ssh/ dir with a name of id_rsa (or if not using rsa a file beginning with id_ and the encryption algo name). That file should contain the private key.

You can, however, specify a different primary key at the command line with the -i flag like so:

ssh -i ~/.ssh/my_other_key.pem

Additionally, you may use the IdentityFile option the the ssh_config file to permanently configure the private key for a specific connection.

Select to clipboard at the ubuntu command line

Copying and pasting without a mouse - or programmatically - can be incredibly challenging. Ubuntu provides an apt-get installable program xclip which can provide X11 clipboard access at the command line.

> echo PASTEME | xclip -sel clip

The value PASTEME is now in the clipboard buffer.

The -sel or -selection indicates the buffer that will be used, primary, secondary or clipboard. Generally, clipboard is the buffer that we want.

To output the value of the buffer use the -o or -output flag:

> xclip -sel clip -o
PASTEME

Keeping track of your CPU Heat

If you’ve ever built your own computer, you’ve had anxiety about not having put everything together right and maybe even something critical, like the mechanisms that keep the computer and the CPU cool.

On Ubuntu you can intall the lm-sensors software which provides you with the facilities to get information the various heat levels inside the computer.

> sudo apt-get install lm-sensors
> sensors
asus-isa-0000
Adapter: ISA adapter
cpu_fan:        0 RPM

acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +119.0°C)
temp2:        +29.8°C  (crit = +119.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +26.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:        +23.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:        +24.0°C  (high = +80.0°C, crit = +100.0°C)
Core 2:        +24.0°C  (high = +80.0°C, crit = +100.0°C)
Core 3:        +24.0°C  (high = +80.0°C, crit = +100.0°C)

Restart or shutdown ubuntu

Its easy to shutdown down Ubuntu from the commandline (given the correct permissions). Shutdown with shutdown which gives you 60 seconds to reverse that decision to shutdown by cancelling with shutdown -c. You can also reboot which takes effect right away.

shutdown # shutdown
shutdown -c # cancel that shutdown.
reboot # reboot the computer now

Ubuntu default desktop manager

What I like about linux is that there are configurations. You should be able to manipulate the whole system without a gui. The configuration for your default display manager, for instance, is in the default-display-manager file:

> cat /etc/X11/default-display-manager
/usr/sbin/lightdm

There’s a startup script /etc/init.d/lightdm that checks that the value stored in this file points to a lightdm program and then executes that program.

Change To That New Directory

The $_ variable provided by bash is always set to the last argument of the previous command. One handy use of this is for changing directories into a newly created directory.

$ mkdir new_dir && cd $_

This command will leave you in your newly created directory, new_dir.

We can imagine using this bash variable in a number of similar scenarios as well. What if we are using some language specific command that creates a directory? Will it work when creating a new Phoenix or Rails project?

It sure will.

Give it a try with Phoenix:

mix phx.new my_app && cd $_

or with Rails:

rails new app && cd $_

source

A Mnemonic for Changing Modes

Making a script file executable involves an incantation that has something to do with chmod but what exactly? chmod has two modes, absolute and symbolic. In absolute mode, you set the entire set of permissions with a numeric value like 755.

In symbolic mode, you can add or subtract permissions to either the user (u), group (g) or owner (o). To give read permissions to the group type:

> chmod g+r myfile.txt

And to give execution priviledges to the owner of a file:

> chmod o+x myscript.rb

Customizing the output of `ps`

There’s a lot to know about processes, although typically we’re looking for a mapping of pids to commands.

You can customize the output of ps with -o. The -o flag expects a comma separated list of keywords which indicate the columns that you’d like to display for each process.

> ps -o pid,command

The above command shows only the pid and the command w/args for each process. You can see all the possible keywords with ps -L.

Am I connected to a terminal via ssh?

When connecting to another computer via ssh the computer will have some environment variables that are not set otherwise. You can examine the environment variables with SSH in the name with:

> printenv | grep SSH

And in both linux and macos you’ll see an environment variable:

SSH_CONNECTION=192.168.50.3 65102 192.168.50.12 22

The first ip address and corresponding port represents the ip address if one exists and the second is the machine you are currently ssh’d into.

Case-Insensitive Search With Ack

Use the -i flag to perform a case-insensitive search with ack.

$ ack -i easter

ack/ack-bar.md
3:The [`ack`](https://beyondgrep.com/) utility has a fun Easter egg that dumps

postgres/configure-the-timezone.md
18:Eastern time.

If you are a Vim user, you may be familiar with smart-case. The --smart-case option is a related Ack feature worth checking out.

See man ack for more details.

cat from stdin and historical notes

Generally, cat takes a file as an argument and prints the file to stdout. cat can also print to stdout from stdin like this:

> echo abc | cat -
abc

Which is the equivalent to:

> echo abc | cat
abc

But when you need to concatenate a line to the beginning of a file and then process that result it comes in handy:

> echo abc | cat - 123.txt
abc
123
> echo abc | cat - 123.txt | grep abc
abc

The macosx cat man page (here) has a history section and mentions that:

Dennis Ritchie designed and wrote the first man page. It appears to have been cat(1).

So cat was the first man page, interesting! (The ubuntu man page for cat does not mention this)

Redirect outputs with subshell

Today I learned how to use a Subshell to redirect multiple outputs as a single one. This way I can run a bunch of commands and unify all the results as it was a single command.

(git diff --name-only; git diff --staged --name-only) | sort -u

The above example will show me a sorted list of changed files (including the staged ones) in a git project without duplicates.

Here’s another example:

$ echo "world"; echo "hello" | sort -u
#=> world
#=> hello

$ (echo "world"; echo "hello") | sort -u
#=> hello
#=> world

Shhh.... Curl silently.

The curl program generally displays a progress bar as its making a request and receiving a response. This can be awkward when you are trying to wrap curl programmatically.

To shut off the progress bar use:

curl -sS http://www.google.com

The first option (-s) hides the progress bar and the errors. The second option (-S) will expose an error on fail if used in conjunction with -s.

List Available File Types For Ack

The ack utility allows you to filter the searched files based on file type. If you’d like to know all of the file types available, you can use the --help=types flag. This will include file types you’ve specified in your .ackrc file.

Here is a sample of some of the output.

$ ack --help=types
    ...
    --[no]css          .css .less .scss
    --[no]dart         .dart
    --[no]delphi       .pas .int .dfm .nfm .dof .dpk .dproj .groupproj .bdsgroup .bdsproj
    --[no]elisp        .el
    --[no]elixir       .ex .exs
    --[no]erlang       .erl .hrl
    --[no]fortran      .f .f77 .f90 .f95 .f03 .for .ftn .fpp
    --[no]go           .go
    --[no]groovy       .groovy .gtmpl .gpp .grunit .gradle
    --[no]haskell      .hs .lhs
    --[no]hh           .h
    --[no]html         .html .mustache .handlebars .tmpl
    --[no]jade         .jade
    --[no]java         .java .properties
    --[no]js           .js
    ...

See man ack for more details.

Extended file attributes on macOS

You can store metadata on any file in the mac filesystem(HFS+). If you want to ensure the file has a specific encoding, or if you want to place a checksum with the file you can use file attributes.

Setting and reading are easy on macOS with the xattr utility:

> touch something.txt
> xattr -w xyz 123 something.txt
> xattr -p xyz something.txt
123

In this example we wrote(xattr -w) an attribute onto the file and printed it(xattr -p) to see that it was written.

You can list the file attributes with a cryptic option for ls, ls -l@. Try doing this on your ~/Downloads dir to learn something cool about how macOS treats files coming from the internet.

Bits of Characters

If you are curious what the binary of an ascii/utf-8 char is you can use this string of commands at the command line:

echo 'A' | xxd -b

The A character is 65 in ascii which is 64 + 1. 01000000 is 64. 00000001 is of course 1, 01000001 is 65.

echo 'a' | xxd -b

The a character is 97 in ascii which is 64 + 32 + 1. 00100000 is 32 in binary, given this and the above, 01100001 is 97.

echo '🤓' | xxd -b

This ridiculous emoji is a utf-8 char. When you look at the binary for it:

11110000 10011111 10100100 10010011

You can see that every byte begins with a 1 which means that it will combine with any subsequent byte beginning with 1 to form a unique character.

Fuzzy awesome copy to sys clipboard w/yank & fzf

Say you want to copy a pid of a process to system clipboard, you could run ps ax, maybe grep on the result, connect your trusty mouse and try to select the value and hit ⌘ + c.

Or you can use the amazing fuzzy finder FZF (brew install fzf) in combination with Yank (brew install yank).

ps ax | fzf | yank

Now simply start typing the name of the process. When you press return you will get the columns broken down into a selectable prompt - choose one and press return. It is now in your system clipboard.

Here’s a demo:

demo

This will work with any column spaced or even multiline response. Try running ps ax | yank.

First and last arguments to a command line command

If I start with the command:

> echo a b c
a b c

Sometimes I want the last argument to a command, which in this case is c but many times can be a long file path or a hard to spell word. In that case I would use !$:

> echo !$
c

This substitutes !$ with the last argument of the previous command.

To substitute the first argument instead, use !^ like so:

> echo !^
a  

The ^ character and the $ character are ofter used as first and last. In vim those commands move the cursor to the beginning and the end of the line respectively. In regex those characters match the beginning or end of the line.

Search Files Specific To A Language

The ack command makes it easy to narrow the set of searched files to those of a specific programming language. For instance, if you have a rails project and only want to search the ruby files, use the --ruby flag with your ack command.

$ ack --ruby Active

With the --ruby flag, I get a manageable number of results. Without it, not so much.

$ ack --ruby Active | wc -l
      26
$ ack Active | wc -l
    5253

See man ack for more details.

A Better File Wiper 📂

A tried-and-true debugging technique is to tail the test log:

$ tail -f log/test.log

When the output gets noisy, we might ‘wipe’ the log by removing it, so that it only contains the output from a single action:

$ rm log/test.log && rspec spec/features/user_logs_in_spec.rb

A better way to do this is to overwrite the file, like so:

$ echo '' > log/test.log

This wipes the file without replacing it, which preserves its serial number. It’s less destructive, and allows other processes interacting with the file to finish their work.

Here’s an example of this principle in action, using ls -i. Notice how the file’s serial number changes when we replace the file, but not when we ovewrite it.

$ ls -i *.txt
27232432 foobar.txt
$ rm foobar.txt
$ touch foobar.txt
$ ls -i *.txt
27232453 foobar.txt
$ echo '' > foobar.txt
$ ls -i *.txt
27232453 foobar.txt

h/t Brian Dunn

Capitalized Letter is the Default

Today I learned something fun about command line interfaces— when presented with a list of choices such as yes, no, or quit, the capitalized choice is the default in many Unix-based CLIs.

Here’s some aptitude:

% sudo apt-get install whatever
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  python-cssselect python-glade2 python-pyquery
0 upgraded, 3 newly installed, 0 to remove and 43 not upgraded.
Need to get 497 kB of archives.
After this operation, 2,207 kB of additional disk space will be used.
Do you want to continue? [Y/n]

The interface waits for your Y (‘yes’) or n (‘no’).

Because ‘yes’ is capitalized, it’s the default, simply pressing ENTER is the same as pressing Y, then ENTER. Enjoy your extra keystrokes!

Pipe | after | grep

The grep command may not work properly if you want to pipe another command after it, specially when you are tailing a file for example. Just call grep with --line-buffered

heroku logs -t | grep --line-buffered "heroku\[router\]" | awk '{print $11" "$5}'

This command outputs an easy-to-read performance log for each HTTP request:

service=266ms path="/api/posts"
service=142ms path="/api/users"

Shell ! command

!! is well known to repeat the last shell command in a *nix terminal, it turns out there are other useful variants:

Given a shell history:

$ history
10034  echo $PATH
10035  history
10036  type ruby
10037  which ruby
10038  history
10039  ls

!n - execute a specific item:

$ !10039
ls
Applications
Desktop
...

!?<match>?: matches a pattern and runs the first command that matches:

$ !?uby?
type ruby
ruby is /Users/jason/.rvm/rubies/ruby-2.2.2/bin/ruby
````

Verify downloaded files from the web #security

If you download a file from the web on a public WiFi and want to run on your machine you might want to check if the file has not been tampered with by a man-in-the-middle-attack or if the file host has been breached.

The easiest way to do this is to check the publised md5 or sha-1 hash for that file (you can do that via your phone if you want to be extra secure). Not every package publishes that but if they do it will be on their website usually next to the download link.

To verify the file you will need to hash the file you downloaded using openssl. For example:

 $ openssl sha1 Kali-Linux-2016.1-vm-amd64.7z
 SHA1(Kali-Linux-2016.1-vm-amd64.7z)= 2b49bf1e77c11ecb5618249ca69a46f23a6f5d2d

Which matches the site’s published sha-1 hash:

kalisha

If you want to check md5, simply replace sha1 in the command with md5.

non recursive find

The find command is recursive by default. Let’s say you want to find all the screen shots on your ~/Desktop, but ignore any screens in the directories in Desktop. Setting the maxdepth flag will tell how far to recurse, using the value 1 only searches the directory you specified, using 2 would go into each directory, but only not search any sub directories, and so on.

#this will recurse by default
$ find ~/Desktop -name "*Screen*"

#this will only search the Desktop and it's immediate subdirectories
$ find ~/Desktop -maxdepth 2 -name "*Screen*"

Jump To The Ends Of Your Shell History

There are all sorts of ways to do things in your shell environment without reaching for the arrow keys. For instance, if you want to move up to the previous command, you can hit Ctrl-p. To move down to the next command in your shell history, you can hit Ctrl-n.

But what if you want to move to the beginning and end of your entire shell history?

Find your meta key (probably the one labeled alt) and hit META-< and META-> to move to the end and beginning of your shell history, respectively.

source

zsh variable editing

Lets say I made a typo in an important environment variable and I wanted a quick fix, just to see if yes, that was indeed why my application wasn’t working.

$ echo $RAILS_ENV
> porduction
# ^^^ YIKES ^^^

The Z shell has a builtin program called vared that helps you edit an existing environment/shell variable.

$ vared RAILS_ENV
porduction

Now you are in an editing context, you can edit the value with all your configured command line editing tools. Fix the error, hit RETURN, and you’re good!

List Names Of Files With Matches

I often use grep and ag to search for patterns in a group or directory of files. Generally I am interested in looking at the matching lines themselves. However, sometimes I just want to know the set of files that have matches. Both grep and ag can be told to output nothing more than the names of the files with matches when given the -l flag.

This can come in particularly handy if you just want a list of files that can be piped (or copied) for use with another command. This eliminates all the extra noise.

h/t Dillon Hafer

whois that url?

Let’s say you just got a spammy text message that contains a suspicious url and you want to learn more but clicking the link just seems like a bad idea.

You can learn more from the whois program that comes installed on macosx. It sends queries to databases that store domain registration information.

When you use the command:

whois spam.com

You can see the important result

Registrant Organisation: Hormel Foods Corporation

The wikipedia article for whois contains some deep internet history.

Grep through compressed (gzipped) log files

The logrotate linux utility automatically compresses your ever-growing production log files.

If you encountered an error and wanted to search the history including all compressed logs, you may have considered unzipping all of them into a directory and run a grep in that directory.

Fortunately linux offers a more idiomatic way for grepping gzipped files: zgrep.

From the manual:

zgrep, zegrep, and zfgrep act like grep, egrep, and fgrep, respectively, but accept input files compressed with the compress(1) or gzip(1) compression utilities.

h/t Jack Christensen

Encrypting with gpg

Encrypting and decrypting is easy with gpg.

To encrypt the text “ABC” and write it to a file called “abc.enc”:

$ echo "ABC" | gpg --symmetric --force-mdc > abc.enc

To decrypt the “abc.enc” file:

$ gpg -d < abc.enc
gpg: CAST5 encrypted data
gpg: gpg-agent is not available in this session
gpg: encrypted with 1 passphrase
ABC

Each step asks for a passphrase. Symmetric encryption is encryption that can be encrypted and decrypted with the same information, in this case that information is the passphrase.

--force-mdc enables the modification detection code, so that gpg will be able to know if the encrypted text has been modified or not.

Bracket Ranges

Today while making a bunch of throwaway directories, I tried this command in ZSH:

$ mkdir test_app/{1..5}

This tab-completed to, you guessed it:

$ mkdir test_app/1 test_app/2 test_app/3 test_app/4 test_app/5

Three cheers for laziness! It’s useful to know you can use ranges this way (did not work in Bash).

Is that process running? pgrep!

If I want to ask the operating system if a process is running I typically use something like:

$ ps -ax | grep rails

But the output reports two processes, the rails process that is running and the actual grep. Once trick to get past this is:

$ ps -ax | grep [r]ails 

So that the grep looks for a process that starts with r followed by ails but can’t find it because the r is followed by ].

But this is an old problem with a fairly old solution. Both mac and linux have a pgrep program.

$ pgrep ruby
44761

Which returns the pid of the process you’re looking for. To get more information about the program and its arguments use the -lf flags.

$ pgrep -lf ruby
44761 /Users/devuser/.rvm/rubies/ruby-2.3.1/bin/ruby bin/rails s

See how long a process has been running on #linux

If you started a long-running process and want to know how long it has been “on the run” so to speak, you can use the -eo switch on ps to specify you want the elapsed time like so:

ps -eo pid,cmd,etime

This will yield something like:

112 [aws/0]                  2-10:27:00
114 [aws/1]                  2-10:27:00
115 [aws/2]                  2-10:27:00
123 [aws/3]                  2-10:27:00
  ...

Which means that process aws has been running for 2 days, 10 hours and 27 minutes.

You can of course pipe the result to grep:

ps -eo pid,cmd,etime | grep aws