Today I Learned

A Hashrocket project

cat from stdin and historical notes

Generally, cat takes a file as an argument and prints the file to stdout. cat can also print to stdout from stdin like this:

> echo abc | cat -
abc

Which is the equivalent to:

> echo abc | cat
abc

But when you need to concatenate a line to the beginning of a file and then process that result it comes in handy:

> echo abc | cat - 123.txt
abc
123
> echo abc | cat - 123.txt | grep abc
abc

The macosx cat man page (here) has a history section and mentions that:

Dennis Ritchie designed and wrote the first man page. It appears to have been cat(1).

So cat was the first man page, interesting! (The ubuntu man page for cat does not mention this)

Size of RSA encrypted content

RSA encryption (publickey/privatekey) is powerful encryption but its really only meant to encrypt small things like other encryption keys. The maximum size of the amount of data an rsa public key can encrypt is:

(key_size/8).floor - 11

So in the case of a key thats 1024 bits it can only encrypt data up to 117 bytes long. Not long enough for documents of any size or variable length.

Add email to github account

You can have multiple emails associated with your github account so that if your commits have user_x@example.com or on another machine your commits have user_y@example.com they'll both show up as you on the github system.

In my case, all commits on hashrocket workstations are dev+somebody@hashrocket.com and we change who that somebody is with the pear application. So when I'm working on a hashrocket workstation I commit as dev+chriserin@hashrocket.com.

When editing a profile on github there is a left hand menu labeled 'personal settings' and in that menu there is an emails link. When you click emails you have the opportunity to 'Add email address'. There ya go! Happy committing!

H/T Matt Polito

Redirect outputs with subshell

Today I learned how to use a Subshell to redirect multiple outputs as a single one. This way I can run a bunch of commands and unify all the results as it was a single command.

(git diff --name-only; git diff --staged --name-only) | sort -u

The above example will show me a sorted list of changed files (including the staged ones) in a git project without duplicates.

Here's another example:

$ echo "world"; echo "hello" | sort -u
#=> world
#=> hello

$ (echo "world"; echo "hello") | sort -u
#=> hello
#=> world

The Registry is back

The Registry is a bad Windows dream, something left behind when I switched off Windows for good circa 2011.

It's back. It's a new Elixir 1.4 module where you can store values that you'd like to access globally. Potentially, you want to store a pid of a worker you started. You can store that in the registry like so.

{:ok, pid} = MyWorker.start_link()
supervisor(Registry, [:unique, Registry.WorkerPids])
Registry.register(Registry.WorkerPids, :my_worker, worker_pid)
[{_, found_worker_pid}] = Registry.lookup(Registry.WorkerPids, :my_worker)

A lot is going on here.

  • We start a worker
  • We start the registry process for register Registry.WorkerPids
  • We register the key/value in a specific register (Registry.WorkerPids)
  • We lookup the value in that register with that key

Its important to note that the Registry.WorkerPids symbol is just an atom, not a module. There is no WorkerPids module implementing any special functionality anywhere.

Definitely a lot of ceremony for a key/value store. I expect to find practical uses for it as I experience more complex Elixir systems.

Shhh.... Curl silently.

The curl program generally displays a progress bar as its making a request and receiving a response. This can be awkward when you are trying to wrap curl programmatically.

To shut off the progress bar use:

curl -sS http://www.google.com

The first option (-s) hides the progress bar and the errors. The second option (-S) will expose an error on fail if used in conjunction with -s.

Re-mark a Word as Bad

Vim's spell checking feature is pretty great. I use it as a default for .md and .rdoc files on my machine.

Sometimes it incorrectly marks technical words like 'ExUnit' as bad (misspelled). To fix this, I type zg over the word in normal mode, which adds it to a list of white-listed words that are no longer considered bad.

The opposite of that command is zw, which comments out the line in the dictionary file. Run :runtime spell/cleanadd.vim to clean up those comments.

Revealing Rails Scopes

I've been working on some Rails code that brings in ActiveRecord models from multiple gems. Often these models have default scopes, that bane of a legacy Rails codebase, and figuring that out requires source diving one or more gems. Today I hacked my way to a faster solution: just read the SQL Rails generates.

Here's a post without a default scope, and then one with a default scope:

pry(main)> Post.all.to_sql
=> "SELECT \"posts\".* FROM \"posts\""
pry(main)> Developer.all.to_sql
=> "SELECT \"developers\".* FROM \"developers\" ORDER BY \"developers\".\"username\" ASC"

I see you, ORDER BY.

Age in Postgres

On this day every year I get curious about how old I actually am. Postgres has some utilities to help me out with that, like subtraction of dates.

> select now() - '1978-3-24';
14245 days 17:39:55.623866

Ah yes, 14245 days on this earth, it really makes you think. What got returned anyways, that looks weird:

> select pg_typeof(now() - '1978-3-24');
interval

Got it, that's an interval. It doesn't really tell me much, is this interval infinite?

> select isfinite(now() - '1978-3-24');
t

I can squint and read isfinite as infinite and suddenly my outlook changes, but only momentarily.

Postgres, just tell me how many years I've been alive ok? None of this days stuff.

> select age('1978-3-24'::date);
39 years

Chaining Multiple RSpec Change Matchers

It can be handy to use RSpec's change matchers to determine if some method or process creates a new record.

expect{ Registration.create(attrs) }.to change{ User.count }.by(1)

But what if we are testing a method that creates a couple different records in the system?

RSpec allows us to chain together change matchers with and. Consider this additional contrived example.

expect {
  Project.generate(attrs)
}.to change{ Project.count }.by(1).and \
     change{ User.count }.by(1)

In addition to keeping our tests tight and concise, this approach gives some pretty nice output on failure.

If we were just beginning our implementation with a failing test, we'd see a multi-part failure like the following.

Failure/Error:
  expect {
    Project.generate(attrs)
  }.to change{ Project.count }.by(1).and \
       change{ User.count }.by(1)

     expected result to have changed by 1, but was changed by 0

  ...and:

     expected result to have changed by 1, but was changed by 0

Disk usage for just the top level dirs

Every so often your hard drive runs out of space and you need to do a survey of what could be possibly be taking up so much space. The du disk-usage command is great for that.

It can be noisy though as it traverses through all the directories and gives you a report on the size of each file. What we really want is just the size of each directory under the root dir.

du has a depth flag (-d) to help control the depth of directories that the command reports on.

du -h -d1 /

The above command gives you a report on all the directories and files at the top level.

Invisible components in React Native

Sometimes you want a component to be rendered but not visible to the user and in React Native 0.42 and below, you could accomplish that by setting the component's height and width to 0.

The good news is that React Native 0.43 will make it easier and introduce the CSS property you are already familiar with: display: 'none'.

You can already try it installing the pre-release version 0.43.0-rc.4.

Run Prettier on all #JavaScript files in a dir

If you are like me you must like formatters such as Prettier which probably prompted you to set your editor to auto format the file on save.

That's great for new projects but when working on an existing project, every file you touch will have a huge diff in git that can obscure the real changes made to the file.

To solve that you must run prettier on all your javascript files as an independent commit. You can do it with the following command:

find ./src/**/*.js | xargs prettier --write --print-width 80 --single-quote --trailing-comma es5

The flags after the prettier are all my personal preferences except for --write which tells prettier to write the file in place.

Note 1: Make sure you have all the files you are about to change committed to source control so that you can check them out if this did not go well.

Note 2: When committing this change it would be a good idea to use git add -p and go through the changes one by one (which is always a good idea...)

Note 3: To dry run and see which files will be changed run the find ./src/**/*.js by itself.

Erlang documentation with `erl -man`

Erlang can be tricky, so documentation is a must, but when you're 30,000 feet high in an airplane with no internet you might regret not having downloaded DashApp and with it all the Erlang documentation.

But it was on your machine the whole time! You have access to all the Erlang documentation with erl -man used like so:

erl -man calendar

Which pops open a man page for the calendar module.

When you are learning Erlang the docs are at your fingertips.

List Available File Types For Ack

The ack utility allows you to filter the searched files based on file type. If you'd like to know all of the file types available, you can use the --help=types flag. This will include file types you've specified in your .ackrc file.

Here is a sample of some of the output.

$ ack --help=types
    ...
    --[no]css          .css .less .scss
    --[no]dart         .dart
    --[no]delphi       .pas .int .dfm .nfm .dof .dpk .dproj .groupproj .bdsgroup .bdsproj
    --[no]elisp        .el
    --[no]elixir       .ex .exs
    --[no]erlang       .erl .hrl
    --[no]fortran      .f .f77 .f90 .f95 .f03 .for .ftn .fpp
    --[no]go           .go
    --[no]groovy       .groovy .gtmpl .gpp .grunit .gradle
    --[no]haskell      .hs .lhs
    --[no]hh           .h
    --[no]html         .html .mustache .handlebars .tmpl
    --[no]jade         .jade
    --[no]java         .java .properties
    --[no]js           .js
    ...

See man ack for more details.

Extended file attributes on macOS

You can store metadata on any file in the mac filesystem(HFS+). If you want to ensure the file has a specific encoding, or if you want to place a checksum with the file you can use file attributes.

Setting and reading are easy on macOS with the xattr utility:

> touch something.txt
> xattr -w xyz 123 something.txt
> xattr -p xyz something.txt
123

In this example we wrote(xattr -w) an attribute onto the file and printed it(xattr -p) to see that it was written.

You can list the file attributes with a cryptic option for ls, ls -l@. Try doing this on your ~/Downloads dir to learn something cool about how macOS treats files coming from the internet.

Remove One List From Another

The --/2 operator allows you to subtract two lists, that is, remove all elements in the right list from the left list. Each occurrence of an element is removed if there is a corresponding element. If there is no corresponding element, it is ignored.

Here are some examples.

> [1, 2, 3] -- [2, 4]
[1, 3]
> [:a, :b, :c, :a, :d, :a] -- [:a, :a]
[:b, :c, :d, :a]

This kind of list operation is not particularly efficient, so for large lists it can be quite slow. The following example took several minutes to run.

> Enum.into(1..1000000, []) -- Enum.into(2..1000000, [])
[1]

To achieve a true set difference, you'll note that the docs for this operator recommend checking out MapSet.difference/2.

See h Kernel.-- for more details.

RSpec Covers a Range

Today I used a feature of RSpec I've never used before, cover.

cover returns true if it's argument is covered by the given range:

expect(0..1).to cover(order.risk_score)

In my use case, I had an attribute risk_score that is calculated by a third-party API. It should always be between zero and one, but it changes on every test run because the API considers the test user more and more risky each time. cover allowed me to test this attribute, and the underlying logic, in a flexible way.

Inspect with Label

IO.inspect returns the value you pass in as the first argument. This is great when you want to debug the middle state of a series of piped calls.

"ABCDEFG"
|> String.graphemes
|> Enum.each_with_index
|> IO.inspect
|> Enum.map(fn({letter, i}) -> "#{letter}#{i}" end)
|> Enum.join

Here, inspect will write to stdout the values [{"A", 1}, {"B", 2"}...].

IO.inspect also takes a label option, which decorates the values written to stdout but does not disrupt the piped calls.

"ABCDEFG"
|> String.graphemes
|> Enum.each_with_index
|> IO.inspect(label: "Letters with index:")
|> Enum.map(fn({letter, i}) -> "#{letter}#{i}" end)
|> Enum.join

Which outputs Letters with index: [{"A", 1}, {"B", 2"}...] to stdout.

Bits of Characters

If you are curious what the binary of an ascii/utf-8 char is you can use this string of commands at the command line:

echo 'A' | xxd -b

The A character is 65 in ascii which is 64 + 1. 01000000 is 64. 00000001 is of course 1, 01000001 is 65.

echo 'a' | xxd -b

The a character is 97 in ascii which is 64 + 32 + 1. 00100000 is 32 in binary, given this and the above, 01100001 is 97.

echo '🤓' | xxd -b

This ridiculous emoji is a utf-8 char. When you look at the binary for it:

11110000 10011111 10100100 10010011

You can see that every byte begins with a 1 which means that it will combine with any subsequent byte beginning with 1 to form a unique character.

Jaro Distance determines "closeness" of 2 strings

In Elixir, the String module includes a function named jaro_distance that determines the similarity between two strings:

iex> String.jaro_distance("Chris", "Crhis")
0.9333333333333332
iex> String.jaro_distance("Chris", "Bob")
0.0
iex> String.jaro_distance("Chris", "Dennis")
0.5777777777777778

The docs say this works best with short strings. The Jaro distance article on wikipedia has lots of math if you're into that.

Referencing Values In IEx's History

Each time we execute a statement in an iex session, the counter is incremented. These numbers are references to the history of the session. We can use these references to refer to previously executed values using v/1. This is particularly handy for multi-line statements or when we forget to bind to the result of some function.

Consider the following iex session:

iex(1)> :one
:one
iex(2)> 1 + 1
2
iex(3)> "three" |> String.to_atom()
:three

If we execute v() on its own, it is the same as v(-1) in that it will give us the latest value in the history.

iex(4)> v()
:three

Providing any positive number will refer to the references we see next to each statement.

iex(5)> v(1)
:one

Negative numbers, as we saw with v(-1), will count backwards in the history from where we are.

iex(6)> v(-4)
2

See h v for more details.

Creating A PID

Often times, when invoking a function that spawns a process, the PID of the spawned process is returned and we bind to it. That PID is a reference to some BEAM process in our system.

We can create our own references using the pid/3 function.

Let's assume we have the following processes, among others, in our system at the moment.

> Process.list |> Enum.reverse |> Enum.take(3)
[#PID<0.284.0>, #PID<0.283.0>, #PID<0.282.0>]

We can create a reference to any of them using the three number parts that they are made up of.

> pid(0, 284, 0)
#PID<0.284.0>

See, it's alive.

> pid(0, 284, 0) |> Process.alive?
true

What if we make up a PID that doesn't actually reference any process?

> pid(0, 333, 0) |> Process.alive?
false

Note: there is also a pid/1 version of the function. See h pid for more details.

CSS Variables

CSS3 supports native variables that are assigned with the following syntax:

html {
  --foo-color: #FF0000;
  --bar-color: #00FF00;
}

Note the -- that precedes the variable's name. Now, the <html> element and all its descendants will be able to access the variables with the following syntax.

a {
  color: var(--foo-color);
}

button {
  background: var(--bar-color);
}

CSS uses the var function to look up variables.

You don't have to assign the variable to the <html> tag. It can be assigned to any element or selector's style block. Just remember, only that element and its children will have access to the variable ;-)

Futhermore, you can access the variables in Chrome's DevTool/Element Inspector!

NOTE: This feature is not supported at all in Internet Explorer and only has partial support in Microsoft Edge 15, the most recent Edge at this time of writing. Opera Mini users are out of luck as well :-(

Browser Support Chart

Accept Your Own Changes During Git Rebase

During a git rebase you may encounter conflicts in files between your current, HEAD, branch and the branch you're rebasing. Ordinarily, you'll want to go through each file and individually resolve each conflict to ensure that the proper changes are preserved.

Sometimes, however, you already know that you want to accept ALL the changes in a file on your local branch and discard the other branch's edits. Instead of opening the file, finding the conflict regions, then making the appropriate changes, you can more succinctly prefer your changes with the following command:

git checkout --ours /path/to/file.js

Conversely, if you want to keep the other branch's changes, run:

git checkout --theirs /path/to/file.ex

You can also do this for an entire directory:

git checkout --ours /path/to/dir/
git checkout --theirs . # Current working directory

When you [re]open the conflict files, you'll see that your preferred branch's changes have been written and the other branch's have been discarded.

After you've finished, stage the the conflict files, and continue your rebase:

git add /path/to/conflict_file.rb
git rebase --continue # or --skip

Fuzzy awesome copy to sys clipboard w/yank & fzf

Say you want to copy a pid of a process to system clipboard, you could run ps ax, maybe grep on the result, connect your trusty mouse and try to select the value and hit ⌘ + c.

Or you can use the amazing fuzzy finder FZF (brew install fzf) in combination with Yank (brew install yank).

ps ax | fzf | yank

Now simply start typing the name of the process. When you press return you will get the columns broken down into a selectable prompt - choose one and press return. It is now in your system clipboard.

Here's a demo:

demo

This will work with any column spaced or even multiline response. Try running ps ax | yank.

Determine the type of buffer

The :set <option> command will show the value of an option. In the case of :set buftype this is generally set to nothing. It's empty. This is because we're usually working with a normal buffer. The exception is the quick window. Generally, the buftype of the quick window is quickwindow. The help window also has its own buftype, help.

These values are used programmatically to help create features with vimscript.

See more with: :help buftype

Create jsonb data structure from columns

You can create a jsonb object by casting a string version of the object to jsonb:

> select '{"a": 1, "b": 2}'::jsonb;
      jsonb
------------------
 {"a": 1, "b": 2}

When using values from columns to create a jsonb object its not practical to construct a string. In this case you want to use jsonb_build_object:

> select jsonb_build_object('a', a.x) from (select x from generate_series(1, 10) as x) as a;
 jsonb_build_object
--------------------
 {"a": 1}
 {"a": 2}
 {"a": 3}
 {"a": 4}
 {"a": 5}
 {"a": 6}
 {"a": 7}
 {"a": 8}
 {"a": 9}
 {"a": 10}

jsonb_build_object is a variatic function and takes a variable number of arguments. So you can add additional values like this:

jsonb_build_object('a', 1, 'b', 2)

Making your JS files prettier on Vim

We all use VIM at Hashrocket and we can run prettier everytime we save a JS file:

yarn global add prettier
autocmd FileType javascript set formatprg=prettier\ --single-quote\ --trailing-comma\ es5\ --stdin
autocmd BufWritePre *.js :normal gggqG
autocmd BufWritePre *.js exe "normal! gggqG\<C-o>\<C-o>"

And yes. I like trailing comma,

**** Update: There is an open issue on Github when prettier fails to parse the JS file but there is a workaround for it.

Show commits in the order they were committed

When reviewing a series of commits its sometimes helpful to review them in the order they were committed.

By default, if you pass a range to git show, like git show HEAD..master (assuming you are on a different branch and master is ahead by a number of commits) the commits will be shown in the order of most recent to least recent.

To view the commits in least recent to most recent order use the --reverse flag:

git show HEAD..master --reverse

Debug the `--exclude-pattern` option in rspec.

You can exclude certain files from being run by rspec with the --exclude-pattern option like so:

rspec --exclude-pattern run_me_not_spec.rb

You can place this option into your .rspec file.

When doing this and then committing the .rspec file its helpful to make sure the exclude pattern is correct. Try this command and pipe it into grep.

rspec --dry-run -fdoc | grep 'excluded test name'

If no results are returned, then you are successfully excluding the test! The --dry-run option is important because actually running the entire test suite would be too time consuming.

Specifying The Phoenix Server Port

Running mix phx.server for a Phoenix project with the default settings will attach the server to port 4000.

If you'd like to use a different port in development, you can change it in config/dev.exs.

config :my_app, MyApp.Web.Endpoint,
  http: [port: 4444],
  ...

Alternatively, you can allow it to be configurable from the command line with an environment variable and a fallback port.

config :my_app, MyApp.Web.Endpoint,
  http: [port: System.get_env("PORT") || 4000],
  ...

Running

$ PORT=4444 mix phx.server

will launch the server on port 4444.

Logrotation for a Rails App

This week I did a some Linux server logrotation for a Ruby on Rails application. log/your_log.log can get large on a server with traffic, and sometimes we must control how long to retain data.

Here's the basic configuration file I wrote, with comments:

# Logrotater:
# - Daily
# - Timestamped
# - Doesn't error on missing log files
# - Keeps seven copies
# - Truncates log files (allows Rails to start writing 
#  to the new log file without a restart)

/var/app/current/log/your_log.log {
    daily
    dateext
    missingok
    rotate 7
    copytruncate
}

There are many other options. Check out man logrotate on a Linux machine for more info.

Prepare, Execute, and Deallocate Statements

In PostgreSQL, you can prepare a named statement to be executed later using prepare.

> prepare column_names (text) as
    select column_name from information_schema.columns where table_name = $1;
PREPARE

These statements are kept around for the duration of the session. To see the available statements, check out the pg_prepared_statements view.

> select * from pg_prepared_statements;
     name     |                                  statement                                  |         prepare_time          | parameter_types | from_sql
--------------+-----------------------------------------------------------------------------+-------------------------------+-----------------+----------
 column_names | prepare column_names (text) as                                             +| 2017-03-10 15:01:09.154528-06 | {text}          | t
              |   select column_name from information_schema.columns where table_name = $1; |                               |                 |

To run a prepared statement, use execute with the name of the statement and any arguments.

> execute column_names('users');
   column_name
-----------------
 id
 email
 password_digest
 created_at
 updated_at
 first_name
 last_name

You can also delete a statement with deallocate if you'd like.

> deallocate column_names;
DEALLOCATE

yank full line without new line character

I am often in a scenario where I want to take all the contents of the current line and paste them into the middle of another line, perhaps into some parentheses or quotation marks.

If I'm in the middle of a line I can type ^y$ which will goto the first character of the line and then yank to the end without the newline.

This is the rare palindramic vim command. It works the same backwards as it does forwards ($y^).

There is also a plugin that treats the line as a vim object called vim-textobj-line. Check it out!

Matching on directories for vim's autocmd

Generally, vim users use the autocmd command to set specific options for filetypes. Like, if you want all Ruby files to have the shiftwidth option set to 2 then you would include an autocmd line like this in your ~/.vimrc file:

autocmd BufRead *.rb set shiftwidth=2

I wanted to do something a bit different. I wanted to set the color scheme for each file in a directory rather than for a file type, and I tried this:

autocmd BufRead *writing* color shine

Nothing happened.

Via :help autocmd-patterns I was able to learn that if there isn't a slash char (/) anywhere in the pattern then it just tries to match on the filename. If a slash is included however, the whole path for that file is matched. All I needed was to add a slash:

autocmd BufRead */writing* color shine

Shine On.

Submit a pull request from the command line

Github offers a git extension for augmenting the git command line tool with github specific functionality. This extension is called hub.

hub adds the git alias pull-request to git. This alias/command creates a pull request from the current branch to master by default:

git pull-request

Additionally, you can specify which branch would be the base with the first argument and which branch would be the head with the second argument, like so:

git pull-request devolop feature/add_colors

jQuery.map returns wrapped elements

There are thousands of ways to write JavaScript and many ways to map over a collection in JavaScript. The jQuery variant of the function has a couple of peculiarities that might trip you up however.

First, the arguments to the map function are not (item, index) like they are for the Array.prototype.map function. Instead, they are (index, item).

Second, the jQuery map function does not return an array consisting of the values returned from the map function. It instead returns an array of those values wrapped by jQuery.

This can be unexpected when iterating over a set of elements to create an array of derived values.

Debug :hover selectors in CSS

When styling HTML you may need to define a hover style the css of which might look like this:

div.something:hover {
  background-color: red;
}

The div with class something will now have a red background when your mouse hovers over it.

This can be hard to debug however, if this were a regular style we could inspect it and change the style of the element in dev tools. With hovers, the style is only available in dev tools when the mouse is physically over the element.

Chrome has a facility to declare that a hover state should be on for an element. Simply open dev tools and in the Elements tab right click an element to get a context menu. The menu will have options :active, :hover, :focus, and :visited. Choose :hover and the hover state will be permenantly affixed to that element. You can now play with the hover style until it looks right.