Today I Learned

A Hashrocket project

220 posts by chriserin @mcnormalmode

Add files with matches to the args list

If I want to change all instances of the word Apple to Orange across my entire project I can use the vim command:

:argdo %s/Apple/Orange/gc

But only if the files that have the word Apple are in the args list. To populate the args list you can use the args command in combination with shelling out to git grep with the -l flag.

:args `git grep -l Apple`

Then, when you see the list of files with :args, you’ll see all the files with the word Apple.

A Mnemonic for Changing Modes

Making a script file executable involves an incantation that has something to do with chmod but what exactly? chmod has two modes, absolute and symbolic. In absolute mode, you set the entire set of permissions with a numeric value like 755.

In symbolic mode, you can add or subtract permissions to either the user (u), group (g) or owner (o). To give read permissions to the group type:

> chmod g+r myfile.txt

And to give execution priviledges to the owner of a file:

> chmod o+x myscript.rb

Chaining expectations in Rspec

Generally, we think about expectations in RSpec one at a time. If the first expectation fails, then don’t go any further. Expectations in RSpec however are chainable, meaning, I can attach one expectation to another for the same subject and then know about the failures or successes for both expections, that looks like this.

expect(1).to eq(2).and eq(3)

Which produces output like this:

 Failure/Error: expect(1).to eq(2).and eq(3)

          expected: 2
               got: 1

          (compared using ==)

       ...and:

          expected: 3
               got: 1

          (compared using ==)

The same result can be got from the below code which may appeal to you a bit more:

def chain_exp(*expects)
  expects.inject {|exps, exp| exps.and(exp)}
end

expect(1).to chain_exp(eq(2), eq(3))

Get the association with Ecto

In the schema of my post model I have this line:

belongs_to :developer

So a post is associated with a developer. When I have a post struct and try to access the developer I might see this:

> Tilex.Repo.get(Tilex.Post, 42).developer
#Ecto.Association.NotLoaded<association :developer is not loaded>

Getting the developer is now a two step process:

  1. Construct a query based on the association
  2. Use that query to acquire the struct

To construct a query based on the association we can use the Ecto.assoc/2 function:

query = Ecto.assoc(post, :developer)

Then this query is executed with the Repo.one function:

developer = Tilex.Repo.one(query)

There ya go!

Login for feature test with warden test helpers

Warden provides a way to login as a user without having to go through the web interface that a user generally sees for sign in.

user = FactoryGirl.create(:user)
login_as user, scope: :user

In the config block for Rspec you would include this statement:

config.include Warden::Test::Helpers, type: :feature

If you have different models for different types of users in your system you can sign in with different scopes. Lets say you have a student user concept, you can sign in with:

student = FactoryGirl.create(:student)
login_as student, scope: :student

Customizing the output of `ps`

There’s a lot to know about processes, although typically we’re looking for a mapping of pids to commands.

You can customize the output of ps with -o. The -o flag expects a comma separated list of keywords which indicate the columns that you’d like to display for each process.

> ps -o pid,command

The above command shows only the pid and the command w/args for each process. You can see all the possible keywords with ps -L.

erlang memory usage

You might be be curious how much memory your erlang vm is taking. You can get that and more from :erlang.memory. Below is what I get from running this after starting iex normally.

> :erlang.memory
[total: 18133840, processes: 4975616, processes_used: 4974616, system: 13158224,
 atom: 264529, atom_used: 256719, binary: 228728, code: 6640396, ets: 414904]

This list is confusing. All numbers are bytes, so in total 18133840 bytes are allocated to the erlang process. There are two subcategories processes and system and the values of those categories sum to equal the value of total.

The remaining categories are either a component of system or a component of processes.

Use Elixir plug for only some controller actions

Phoenix allows for a flexible request processing pipeline with plugs. Your controllers are basically plugs in the plug pipeline. You can specify plugs that will execute before your controller actions with the plug macro.

defmodule MyApp.ThingController do
  use MyApp.Web, :controller

  plug MyApp.InterestingPlug
end

You can also specify a guard for this macro that will only enable this plug to be run before certain actions:

plug MyApp.InterestingPlug when action in [:new, :create]

Am I connected to a terminal via ssh?

When connecting to another computer via ssh the computer will have some environment variables that are not set otherwise. You can examine the environment variables with SSH in the name with:

> printenv | grep SSH

And in both linux and macos you’ll see an environment variable:

SSH_CONNECTION=192.168.50.3 65102 192.168.50.12 22

The first ip address and corresponding port represents the ip address if one exists and the second is the machine you are currently ssh’d into.

cat from stdin and historical notes

Generally, cat takes a file as an argument and prints the file to stdout. cat can also print to stdout from stdin like this:

> echo abc | cat -
abc

Which is the equivalent to:

> echo abc | cat
abc

But when you need to concatenate a line to the beginning of a file and then process that result it comes in handy:

> echo abc | cat - 123.txt
abc
123
> echo abc | cat - 123.txt | grep abc
abc

The macosx cat man page (here) has a history section and mentions that:

Dennis Ritchie designed and wrote the first man page. It appears to have been cat(1).

So cat was the first man page, interesting! (The ubuntu man page for cat does not mention this)

Size of RSA encrypted content

RSA encryption (publickey/privatekey) is powerful encryption but its really only meant to encrypt small things like other encryption keys. The maximum size of the amount of data an rsa public key can encrypt is:

(key_size/8).floor - 11

So in the case of a key thats 1024 bits it can only encrypt data up to 117 bytes long. Not long enough for documents of any size or variable length.

Add email to github account

You can have multiple emails associated with your github account so that if your commits have user_x@example.com or on another machine your commits have user_y@example.com they’ll both show up as you on the github system.

In my case, all commits on hashrocket workstations are dev+somebody@hashrocket.com and we change who that somebody is with the pear application. So when I’m working on a hashrocket workstation I commit as dev+chriserin@hashrocket.com.

When editing a profile on github there is a left hand menu labeled ‘personal settings’ and in that menu there is an emails link. When you click emails you have the opportunity to ‘Add email address’. There ya go! Happy committing!

H/T Matt Polito

The Registry is back

“The Registry” is a bad Windows dream, something left behind when I switched off Windows for good circa 2011.

It’s back. It’s a new Elixir 1.4 module where you can store values that you’d like to access globally. Potentially, you want to store a pid of a worker you started. You can store that in the registry like so.

{:ok, pid} = MyWorker.start_link()
supervisor(Registry, [:unique, Registry.WorkerPids])
Registry.register(Registry.WorkerPids, :my_worker, worker_pid)
[{_, found_worker_pid}] = Registry.lookup(Registry.WorkerPids, :my_worker)

A lot is going on here.

  • We start a worker
  • We start the registry process for register Registry.WorkerPids
  • We register the key/value in a specific register (Registry.WorkerPids)
  • We lookup the value in that register with that key

Its important to note that the Registry.WorkerPids symbol is just an atom, not a module. There is no WorkerPids module implementing any special functionality anywhere.

Definitely a lot of ceremony for a key/value store. I expect to find practical uses for it as I experience more complex Elixir systems.

Shhh.... Curl silently.

The curl program generally displays a progress bar as its making a request and receiving a response. This can be awkward when you are trying to wrap curl programmatically.

To shut off the progress bar use:

curl -sS http://www.google.com

The first option (-s) hides the progress bar and the errors. The second option (-S) will expose an error on fail if used in conjunction with -s.

Age in Postgres

On this day every year I get curious about how old I actually am. Postgres has some utilities to help me out with that, like subtraction of dates.

> select now() - '1978-3-24';
14245 days 17:39:55.623866

Ah yes, 14245 days on this earth, it really makes you think. What got returned anyways, that looks weird:

> select pg_typeof(now() - '1978-3-24');
interval

Got it, that’s an interval. It doesn’t really tell me much, is this interval infinite?

> select isfinite(now() - '1978-3-24');
t

I can squint and read isfinite as infinite and suddenly my outlook changes, but only momentarily.

Postgres, just tell me how many years I’ve been alive ok? None of this days stuff.

> select age('1978-3-24'::date);
39 years

Disk usage for just the top level dirs

Every so often your hard drive runs out of space and you need to do a survey of what could be possibly be taking up so much space. The du disk-usage command is great for that.

It can be noisy though as it traverses through all the directories and gives you a report on the size of each file. What we really want is just the size of each directory under the root dir.

du has a depth flag (-d) to help control the depth of directories that the command reports on.

du -h -d1 /

The above command gives you a report on all the directories and files at the top level.

Erlang documentation with `erl -man`

Erlang can be tricky, so documentation is a must, but when you’re 30,000 feet high in an airplane with no internet you might regret not having downloaded DashApp and with it all the Erlang documentation.

But it was on your machine the whole time! You have access to all the Erlang documentation with erl -man used like so:

erl -man calendar

Which pops open a man page for the calendar module.

When you are learning Erlang the docs are at your fingertips.

Extended file attributes on macOS

You can store metadata on any file in the mac filesystem(HFS+). If you want to ensure the file has a specific encoding, or if you want to place a checksum with the file you can use file attributes.

Setting and reading are easy on macOS with the xattr utility:

> touch something.txt
> xattr -w xyz 123 something.txt
> xattr -p xyz something.txt
123

In this example we wrote(xattr -w) an attribute onto the file and printed it(xattr -p) to see that it was written.

You can list the file attributes with a cryptic option for ls, ls -l@. Try doing this on your ~/Downloads dir to learn something cool about how macOS treats files coming from the internet.

Inspect with Label

IO.inspect returns the value you pass in as the first argument. This is great when you want to debug the middle state of a series of piped calls.

"ABCDEFG"
|> String.graphemes
|> Enum.each_with_index
|> IO.inspect
|> Enum.map(fn({letter, i}) -> "#{letter}#{i}" end)
|> Enum.join

Here, inspect will write to stdout the values [{"A", 1}, {"B", 2"}...].

IO.inspect also takes a label option, which decorates the values written to stdout but does not disrupt the piped calls.

"ABCDEFG"
|> String.graphemes
|> Enum.each_with_index
|> IO.inspect(label: "Letters with index:")
|> Enum.map(fn({letter, i}) -> "#{letter}#{i}" end)
|> Enum.join

Which outputs Letters with index: [{"A", 1}, {"B", 2"}...] to stdout.

Bits of Characters

If you are curious what the binary of an ascii/utf-8 char is you can use this string of commands at the command line:

echo 'A' | xxd -b

The A character is 65 in ascii which is 64 + 1. 01000000 is 64. 00000001 is of course 1, 01000001 is 65.

echo 'a' | xxd -b

The a character is 97 in ascii which is 64 + 32 + 1. 00100000 is 32 in binary, given this and the above, 01100001 is 97.

echo '🤓' | xxd -b

This ridiculous emoji is a utf-8 char. When you look at the binary for it:

11110000 10011111 10100100 10010011

You can see that every byte begins with a 1 which means that it will combine with any subsequent byte beginning with 1 to form a unique character.

Jaro Distance determines "closeness" of 2 strings

In Elixir, the String module includes a function named jaro_distance that determines the similarity between two strings:

iex> String.jaro_distance("Chris", "Crhis")
0.9333333333333332
iex> String.jaro_distance("Chris", "Bob")
0.0
iex> String.jaro_distance("Chris", "Dennis")
0.5777777777777778

The docs say this works best with short strings. The Jaro distance article on wikipedia has lots of math if you’re into that.

Determine the type of buffer

The :set <option> command will show the value of an option. In the case of :set buftype this is generally set to nothing. It’s empty. This is because we’re usually working with a normal buffer. The exception is the quick window. Generally, the buftype of the quick window is quickwindow. The help window also has its own buftype, help.

These values are used programmatically to help create features with vimscript.

See more with: :help buftype

Create jsonb data structure from columns

You can create a jsonb object by casting a string version of the object to jsonb:

> select '{"a": 1, "b": 2}'::jsonb;
      jsonb
------------------
 {"a": 1, "b": 2}

When using values from columns to create a jsonb object its not practical to construct a string. In this case you want to use jsonb_build_object:

> select jsonb_build_object('a', a.x) from (select x from generate_series(1, 10) as x) as a;
 jsonb_build_object
--------------------
 {"a": 1}
 {"a": 2}
 {"a": 3}
 {"a": 4}
 {"a": 5}
 {"a": 6}
 {"a": 7}
 {"a": 8}
 {"a": 9}
 {"a": 10}

jsonb_build_object is a variatic function and takes a variable number of arguments. So you can add additional values like this:

jsonb_build_object('a', 1, 'b', 2)

Show commits in the order they were committed

When reviewing a series of commits its sometimes helpful to review them in the order they were committed.

By default, if you pass a range to git show, like git show HEAD..master (assuming you are on a different branch and master is ahead by a number of commits) the commits will be shown in the order of most recent to least recent.

To view the commits in least recent to most recent order use the --reverse flag:

git show HEAD..master --reverse

Debug the `--exclude-pattern` option in rspec.

You can exclude certain files from being run by rspec with the —exclude-pattern option like so:

rspec --exclude-pattern run_me_not_spec.rb

You can place this option into your .rspec file.

When doing this and then committing the .rspec file its helpful to make sure the exclude pattern is correct. Try this command and pipe it into grep.

rspec --dry-run -fdoc | grep 'excluded test name'

If no results are returned, then you are successfully excluding the test! The --dry-run option is important because actually running the entire test suite would be too time consuming.

yank full line without new line character

I am often in a scenario where I want to take all the contents of the current line and paste them into the middle of another line, perhaps into some parentheses or quotation marks.

If I’m in the middle of a line I can type ^y$ which will goto the first character of the line and then yank to the end without the newline.

This is the rare palindramic vim command. It works the same backwards as it does forwards ($y^).

There is also a plugin that treats the line as a vim object called vim-textobj-line. Check it out!

Matching on directories for vim's autocmd

Generally, vim users use the autocmd command to set specific options for filetypes. Like, if you want all Ruby files to have the shiftwidth option set to 2 then you would include an autocmd line like this in your ~/.vimrc file:

autocmd BufRead *.rb set shiftwidth=2

I wanted to do something a bit different. I wanted to set the color scheme for each file in a directory rather than for a file type, and I tried this:

autocmd BufRead *writing* color shine

Nothing happened.

Via :help autocmd-patterns I was able to learn that if there isn’t a slash char (/) anywhere in the pattern then it just tries to match on the filename. If a slash is included however, the whole path for that file is matched. All I needed was to add a slash:

autocmd BufRead */writing* color shine

Shine On.

Submit a pull request from the command line

Github offers a git extension for augmenting the git command line tool with github specific functionality. This extension is called hub.

hub adds the git alias pull-request to git. This alias/command creates a pull request from the current branch to master by default:

git pull-request

Additionally, you can specify which branch would be the base with the first argument and which branch would be the head with the second argument, like so:

git pull-request devolop feature/add_colors

jQuery.map returns wrapped elements

There are thousands of ways to write JavaScript and many ways to map over a collection in JavaScript. The jQuery variant of the function has a couple of peculiarities that might trip you up however.

First, the arguments to the map function are not (item, index) like they are for the Array.prototype.map function. Instead, they are (index, item).

Second, the jQuery map function does not return an array consisting of the values returned from the map function. It instead returns an array of those values wrapped by jQuery.

This can be unexpected when iterating over a set of elements to create an array of derived values.

Debug :hover selectors in CSS

When styling HTML you may need to define a hover style the css of which might look like this:

div.something:hover {
  background-color: red;
}

The div with class something will now have a red background when your mouse hovers over it.

This can be hard to debug however, if this were a regular style we could inspect it and change the style of the element in dev tools. With hovers, the style is only available in dev tools when the mouse is physically over the element.

Chrome has a facility to declare that a hover state should be on for an element. Simply open dev tools and in the “Elements” tab right click an element to get a context menu. The menu will have options :active, :hover, :focus, and :visited. Choose :hover and the hover state will be permenantly affixed to that element. You can now play with the hover style until it looks right.

Selected Text Character Count

When highlighting a word in vim (Visual Mode), type:

g<CTRL-g>

In the command section of vim you’ll see something like this:

Selected 1 of 90 Lines; 2 of 715 Words; 15 of 4171 Chars; 15 of 4173 Bytes

In this case, 15 of 4171 Chars is the number of total characters and the number of characters selected.

Its not as straight forward or as easy to parse as you might expect, but when you know what to look for it becomes easy to read.

Full page screenshots in Firefox

Firefox has an interesting feature called the Developer Toolbar. This toolbar is entirely separate from the JS console and provides a cli for custom commands for specific functionality. It can opened with shift-F2 or Tools > Developers > Developer Toolbar.

In that toolbar, type:

screenshot --fullpage my-page.png

A png file with the given name will be saved to the Downloads folder. The file format will always be png.

Source

More Developer Toolbar information can be found here.

Type help in the toolbar to see all the commands.

Images in markdown

Images in markdown are the same as links except for the bang (!) character preceding the syntax.

The syntax for links is squares-parens. I have a hard time remembering this, and I try parens-squares as often as not, but its squares-parens, like this:

[My text](https://mylink.com)

The syntax for images is also squares-parens but preceded by a bang (!), like this:

![My alt text](https://mylink.com/oh_my.png)

Installing the Golang tools with Vim

Go has a set of tools to help aid in go development such as a test coverage tool for go or go guru (a tool for answering questions about the go source code.

While in vim if you would like to install these tools just use the command:

:GoInstallBinaries

And likewise if you would like to update these tools use the command

:GoUpdateBinaries

These commands are provided by the vim-go vim plugin. The binaries are installed into your $GOPATH directory or if you’d like to override that dir set the g:go_bin_path vim variable.

Start rails with a different web server

The webserver for my current project is puma, which is a multi-threaded ruby server. This multithreaded nature makes it hard to but a pry statement in and break in specific places. There are multiple threads that will listen to the input the user provides at the REPL.

Using webrick would allow us to debug and step through our code but changing the applications configuration in order to enable that seems unreasonable.

Fortunately, rails provides an easy way to change servers at the command line.

> rails server webrick

Just pass the name of the server you would rather use as a command line argument.

When typing rails server --help you’ll see this option available on the first line:

Usage: rails server [mongrel, thin etc] [options]

Raid 0 offers performance with worse reliability

RAID is defined:

Redundant Array of Independent Drives

Whenever I think of RAID I think of data drives that each contain the same information, for fault tolerance. Different RAID configurations are given different numeric identifiers. Recently, when shopping for motherboards I saw RAID 0 listed as a selling point for many of the boards and I assumed this was a fault tolerance feature on these motherboards meant for gamers.

RAID 0 has nothing to do with fault tolerance. Nor redundance. In a Raid 0 configuration data is spread across multiple drives with the intention of increasing the bandwidth of data from the motherboard/CPU to the drives. It is a technique for increasing performance that actually reduces fault tolerance.

Read more about RAID here -> https://en.wikipedia.org/wiki/Standard_RAID_levels

First and last arguments to a command line command

If I start with the command:

> echo a b c
a b c

Sometimes I want the last argument to a command, which in this case is c but many times can be a long file path or a hard to spell word. In that case I would use !$:

> echo !$
c

This substitutes !$ with the last argument of the previous command.

To substitute the first argument instead, use !^ like so:

> echo !^
a  

The ^ character and the $ character are ofter used as first and last. In vim those commands move the cursor to the beginning and the end of the line respectively. In regex those characters match the beginning or end of the line.

Use Mkdir! to create dir of present file in vim

In vim its easy to :edit, :write, or :saveas a file to a path that doesn’t yet exist. Such as:

:edit non/existant/path/file.txt

Vim will let you have a buffer with that path, but when you try to write the file vim will throw an error:

E212: Can't open file for writing

Enter the Mkdir! command. The Mkdir command has been added by the eunuch.vim plugin to make creating directories easier from vim. The bang (!) variant takes the path of the current buffer and creates the directory from that path.

When we get the E212 error we can type:

:Mkdir!

And walla! The path will be created. No more errors when writing the file.

Alias Loaders in webpack

A loader might not be named conveniently for your project. Maybe the path to it is too long and obscures the actual configuration of the loader, in that case you can create an alias with the resolveLoader.alias webpack configuration.

module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        loader: 'smrt',
      }
    ]
  },
  resolveLoader: {
    alias: {
      'smrt': require.resolve(__dirname, 'really', 'long', 'path', 'smart-loader.js')
    }
  }
}

In addition, you may only want to transform one file. Its possible to do that in a require statement and use the bang character (!) prefix syntax to declare the loader like this.

require 'smrt!dumbfile';

Clearly in this case a long loader name would obscure intent.

Export from old school libraries in Webpack

As you are refactoring your legacy Rails project towards webpack and away from the asset pipeline you discovered an old version of the facebook_sdk that is absolutely critical to the ongoing success of the legacy application. This file doesn’t play nicely with CommonJS though and exports its constant with global declarations like:

var FB = {};

This isn’t very global in CommonJS and your app doesn’t have access to that constant anymore.

The problem can be solved with the exports-loader used like so:

module.exports = {
  module: {
    rules: [
      {
        test: /facebook_sdk/,
        loader: 'exports-loader?FB',
      }
    ]
  }
}

This is just a cute way of tacking an export line to the bottom of the file like this:

module.exports = FB;

Custom loaders for webpack

Perhaps there is a transformation you’d like to perform on your javascript files that is unique to your project. Perphaps replacing the word dumb with smart is a requirement but you don’t have control over the actual files and can’t make that change to the source files themseles.

In this situation, you can use a custom loader.

module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        loader: './smart-loader',
      }
    ]
  },
}

The loader itself (smart-loader.js) would look something like this.

module.exports = function(source) {
  return source.replace('dumb', 'smart');
}

Exclude files from webpack transformation

Sometimes you want every file that has the .dumb extension to be transformed by the dumb-loader except for one particularly dumb file that should remain as is, as a lesson for to all the other dumb files. Just use the handy exclude configuration on modules.rules.loader to make sure that file doesn’t get transformed.

module.exports = {
  module: {
    rules: [
      {
        test: /\.dumb$/,
        loader: 'dumb-loader',
        exclude: /really\.dumb/
      }
    ]
  }
}

Now that really dumb file won’t get transformed by the dumb loader.

Responding with :stop allows call of `terminate`

A GenServer in Elixir has two lifecycle methods, init and terminate. init is called when the GenServer is started with GenServer.start_link.

What I learned today is that there is a terminate method in case there’s any resources that need to be cleaned up when the GenServer is shut down. The terminate method is called when the :stop message is returned from either handle_cast or handle_call:

defmodule Cache do
  use GenServer
  
  #... a lot of other code

  def handle_cast({:something, 1}, state) do
    IO.puts "This executes first"
    {:stop, "This is my reason for stopping", state}
  end

  def terminate(reason, state)
    IO.puts "Then this executes"
  end
end

Pivoting a 2-dimensional list

In one of the exercism.io exercises, a grid is represented by a list of lists where each inner list represents a row.

The exercise then asks you to pivot the grid, that is, transform the list of lists so that each list represents a column.

From someone else’s solution I learned that an easy way to conduct this transform is to use List.zip and Tuple.to_list.

grid = [[1,2,3, 4], [1,2,3, 4], [1, 2, 3, 4]]
List.zip(grid) 
|> Enum.map(&Tuple.to_list/1)
# [[1, 1, 1], [2, 2, 2], [3, 3, 3], [4, 4, 4]]

Tuple’s to_list method is needed because List.zip returns an array of tuples.