Today I Learned

A Hashrocket project

235 posts by chriserin @mcnormalmode

`with` statement has an `else` clause

with statements are used to ensure a specific result from a function or series of functions, using the results of those functions to take actions within the with block. If a function does not return a specific result (think :error instead of :ok) then you can either define specific clauses for the things you expected to go wrong, or you can just return the result that did not conform to the with clause expectations.

This is the general form:

with {:ok, a} <- {:ok, 123} do
  IO.puts "Everythings OK"
end

A with with an else block:

with {:ok, a} <- {:error, 123} do
  IO.puts "Everythings OK"
else 
  result -> IO.puts("Not OK")
end

A with else clause with pattern matching:

with {:ok, a} <- {:error, "something went wrong"} do
  IO.puts "Everythings OK"
else 
  {:error, message} -> IO.puts(message)
  error -> IO.puts("I'm not sure what went wrong
end

A with without an else clause where the error is returned from the with block:

result = with {:ok, a} <- {:error, "something went wrong"} do
IO.puts "Everythings OK"
end

{:error, message} = results
IO.puts "This went wrong #{message}"

Implied applications and `extra_applications`

In your mix file (mix.exs) the application function returns a keyword list. Two options in that list determine what applications are started at runtime.

The applications is by default implied based on your app’s dependencies. The default list is thrown away through if this option is set in your mix.exs file.

If you want to add an extra application without disrupting the default, implied list then you can add the optionextra_applications. This leaves the default, implied list of applications untouched.

H/T Jose Valim PR

def application do
[
  mod: {Tilex, []},
  extra_applications: [:logger]
]
end

`cd` in subshell

With many of our projects sequestering the front end javascript code into an assets directory I find myself moving between the root project directory and the assets directory to perform all the npm or yarn related tasks in that assets dir. Inevitably I’ll start doing something like this:

cd assets; npm install; cd ..

or this

pushd assets; npm install; popd

In both cases using ; instead of && puts me back in the original directory regardless of the result of the npm command.

I just learned that using cd in a subshell does not change the directory of the current shell, so I can also do this:

(cd assets; npm install)

Ruby srand returns the previous seed

srand is a method on Kernel that seeds the pseudo random number generator. It takes a new seed as an argument or calls Random.new_seed if you don’t pass an argument. What’s interesting about it is that it returns the old seed. This has the effect of return a new large random number every time you call srand.

2.4.1 :007 > srand
 => 94673047677259675797540834050294260538
2.4.1 :008 > srand
 => 314698890309676898144014783014808654061
2.4.1 :009 > srand
 => 102609070680693453063563677087702518073
2.4.1 :010 > srand
 => 81598494819438432908893265364593292061

Which can come in handy if you’re playing some Ruby golf and need to generate a huge random number in as few characters as possible.

H/T Dillon Hafer

Converting strings to atoms safely

If your elixir system accepts any outside inputs and takes any part of those outside inputs and calls String.to_atom with the input as an argument then your elixir system is subject to a denial of service attack.

Malicious actors can submit input designed to dynamically create a large number of atoms until the atom limit is reached, knocking out your elixir applications.

Consider using String.to_existing_atom instead. If the argument to this function cannot be converted to an existing atom then an exception will be thrown.

> String.to_existing_atom("I don't exist")
** (ArgumentError) argument error
    :erlang.binary_to_existing_atom("nothere", :utf8)
> String.to_atom("I don't exist")
:"I don't exist"
> String.to_existing_atom("I don't exist")
:"I don't exist"

Current number of atoms in the atoms table

In Elixir and Erlang there is a hard limit on the number of atoms you can create. Atoms are not garbage collected so its important to ensure you don’t exceed the limit. You can check what the limit is with:

> :erlang.system_info(:atom_limit)
1048576

Likewise, you can check the current number of atoms in the atoms table with:

> :erlang.system_info(:atom_count)
9654

On my system using Elixir 1.5.1 I use 9654 atoms just to start iex.

Xargs from a file

I’ve struggled with xargs conceptually for long time, but actually its pretty easy conceptually. For commands that don’t read from stdin but do take arguments, like echo or kill, you can turn newline separated values from stdin in into arguments.

Piping to echo does not work.

> echo 123 | echo
# nothing

Using xargs it does.

> echo 123 | xargs echo
123

xargs can also read a file with the -a flag, turning each line of the file into an argument.

> echo "123\nabc" > test.txt
> cat test.txt
123
abc
> xargs -a test.txt echo
123 abc

H/T Brian Dunn

Kill rogue shell processes

There is a particular type of attack where an inserted usb stick can act like a keyboard, open a terminal, and start something like this:

while (true); do something_malicious; sleep 3600; done & disown

This process endlessly loops and wakes every hour to do something malicious. The & puts it in the background and the disown will end its attachment to the current terminal. When the terminal is closed the process will get a parent of 1.

This process is still detectable and killable at the command line by finding all shell programs with a parent pid of 1 and killing them with -9.

ps ax -o pid,command,ppid | grep '.*zsh.*\s1$' | awk '{print $1}' | xargs kill -9

This will kill all running rogue zsh processes. There may be reasons why you’d want a process to be detached from its parent terminal, but you could easily decide that this isn’t something you want ever and place the above command into a cron job that runs every 2 seconds.

Run vim command from .... the command line!

Generally vim is started at the command line with the vim command and it comes equipped with the -c flag for running commands.

-c will open vim, run a command, and stay open, so if you wanted to open vim with a smile you could run:

vim -c ':smile'

If you wanted to run a subsitution on a file with vim’s substitute command, then save, then quit, that would look like:

vim -c '%s/frown/upside-down/g | write | quit' frowns.txt

Capturing stderr when shelling out in ruby

All of the usual methods of shelling out in ruby (backticks, system, etc.) don’t capture stderr, but there’s an oddly named library open3 that can help with that.

require 'open3'

result_or_err, process_status = Open3.capture2e('cat abc')
puts result_or_err
# 'cat: abc: No such file or directory'
puts process_status
# <Process::Status: pid 6729 exit 1>]

In this case cat abc errors out because there is no abc file, so the command writes to stderr. If there was an abc file, result_or_err would contain the contents of the abc file.

source

Examine or Diff a file on another #git branch

At times, I’m curious about the changes that have been made for a particular file on a branch that is not the branch I’m on.

The vim-fugitive plugin provides an easy way to open a buffer for a file on another branch with the :Gedit command.

For instance:

:Gedit readme_updates:README.md

Will open a buffer with the contents of the README.md file from the readme_updates branch.

You may want to view that file in comparison to the current file. The :Gdiff command provides us with that functionality:

:Gdiff readme_updates:README.md

Non-ActiveRecord objects in FactoryGirl

Creating non-ActiveRecord objects with FactoryGirl is possible. Classically, a constructor is used to set all the data attributes of an object.

class ParsedString
  attr_reader :abc, :def
  def initialize(string)
    @abc, @def = string.split(?|)
  end
end

In the above class the attributes are set in the constructor by passing in a string that gets split into two parts on \|. Lets use instantiate_with to determine how the object gets instantiated in FactoryGirl.

factory :parsed_string do
  initial_value "123|456"

  initialize_with { new(initial_value) }
end

And then in the test we can FactoryGirl.build, like:

ps = FactoryGirl.build(:parsed_string)
expect(ps.abc).to eq 123
expect(ps.def).to eq 456

Find an npm library from the command line

When looking for a new package to use with npm you can use npm search <string>. This will query npm and return 20 - 25 results that match your search.

> npm search react-router
NAME                      | DESCRIPTION          | AUTHOR          | DATE       | VERSION  | KEYWORDS
react-router              | Declarative routing… | =ryanflorence   | 2017-08-24 |          | react router 
react-router-dom          | DOM bindings for…    | =mjackson…      | 2017-08-24 |          | react router 
react-router-redux        | Ruthlessly simple…   | =jlongster…     | 2017-02-10 |          | react redux r
react-router-native       | React Native…        | =jmurzy…        | 2017-08-24 |          |
react-router-config       | Static route config… | =mjackson…      | 2017-08-24 |          | react router 
react-router-bootstrap    | Integration between… | =monastic.panic… | 2017-04-19 |          | react react-
react-router-scroll       | React Router scroll… | =taion          | 2017-04-10 |          | react react r
react-native-router-flux  | React Native Router… | =aksonov        | 2017-08-23 |          |
...

There doesn’t seem to be a way to search by date, version or popularity, but if you pass the --long flag you’ll be able to see the entire description. Without the --long flag the results will be truncated to fit onto one line.

Check that an executable exists on the $PATH

When writing a vim plugin interacting with different programs on the server might be necessary. For instance you might want to use ruby to make a network request or perform some file management. But first, the vim plugin should check to see if the program ruby exists which can be accomplished with the executable() vimscript function.

:echo executable('ruby')
1
:echo executable('doesnotexist')
0

This can be used in a condition before running the ruby command.

if executable('ruby')
  system('ruby -e "puts YAML"')
endif

CamelCase to underscore and back again w/Elixir

If you have a mix project name with multiple words, then those multiple words are generally separated with underscores in the project directory name, like honey_bears. The module name for the project however is HoneyBears.

Converting a string from underscore to CamelCase is built into Elixir. Its in the Macro module:

> Macro.camelize("honey_bear")
"HoneyBear"

The reverse case also can be solved with a Macro function.

> Macro.underscore("HoneyBear")
"honey_bear"

The Macro module has a number of convience functions for working with macros.

Specifying the location of your vim plugin

If you are working on a vim plugin and you want to keep this plugin amongst your projects rather than with your other vim plugins you can add a new entry to the runtimepath or rtp.

Place the following in your vimrc:

set runtimepath+=~/projects/vim-my-plugin

Or

set rtp+=~/projects/vim-my-plugin

You can confirm that the path was added correctly by checking that setting with:

set rtp

Resize vim window to the size of its content

If you’re writing a vim plugin and in that plugin you are opening a new window to display some content, you may want to resize that window so that it is only the size of the content within. For instance, if you have 10 lines of content but the window is half the screen, then you want to resize the window to 10 lines.

Resizing a window is accomplished with the resize command like:

:resize 10

Getting the number of content lines is accomplished by getting the line number of the last last:

:echo line('$')

Putting the two together is tricky because the argument to the resize command is interpreted literally. :resize line('$') does not work. You have to combine the two into a string and then pass it to the execute command.

:execute('resize ' . line('$'))

Configure max http header size in Elixir Phoenix

If you store lots of data in cookies, you may eventually run up against the maximum header value length in Cowboy which by default is 4096.

While storing that much data in cookies is maybe not the best idea, there is a configuration setting that can help you work around this issue:

config :myapp, MyAppWeb.Endpoint,                                                                                                                 
  http: [protocol_options: [max_request_line_length: 8192, max_header_value_length: 8192]]

There ya go! Now you can accept http header values up to 8192 bytes in length!

Check to see if a command exists at #shell

If you want to check to see if a command exists on a user’s machine you can use command -v. command without flags will run the command passed as an argument.

> command echo "A"
"A"

command -v with the -v flag will return the path of the command, and most importantly, a non-zero status code if that command does not exist.

> command -v brew
/usr/local/bin/brew

Or on Linux

> command -v brew
# returns nothing and also returns a status code of 1

Importantly, you can check for a command before using it.

command -v brew && brew install something

Phoenix will watch your JS for you, just watch!

With es6 javascript ecosystems most compilation/transpilation is done with watchers, programs that watch for changes in your code and then trigger compilation based on the changes you’ve made.

Phoenix has a method of integrating those watchers into the server itself.

config :myapp, MyApp.Endpoint,
  watchers: [yarn: ["run", "watch", cd: Path.expand("../assets", __DIR__)]]

The above configuration goes in config/dev.exs and runs the watch command whenever you start the server/endpoint. The keyword option cd is as of yet undocumented, but does what you think it does, changes the directory to the path you configure, which by convention in Phoenix 1.3 would be the assets directory. cd is the only keyword option.

The watchers start when you run your server. Just make a change to your javascript and refresh the page!

String concatentation in the Bourne Again Shell

String concatenation must be different in every lanaguage, or so it seems, and Bash is no different! You can string concat just by putting too strings next to each other.

> echo "a""b"
ab 

Its ok if one of these strings is a variable.

> a = "x" 
> echo $a"b"
xb

Its ok if you put a double quoted string next to a single quoted string.

> echo 'a'"b"
ab

Its ok if you put spaces in between the strings, but that space now will be part of the new string.

> echo "a" "b"
a b

But multiple spaces in between strings will be squashed to one string.

> echo "a"      "b"
a b

This works when setting a variable too, but only surrounded by parens.

> c=('a'      'b')
> echo $c
a b

Don’t worry about the parens though when they are right next to each other.

> c = "a""b"
> echo $c
ab

Edit bash command at certain point in history

Jack introduced our team to the fc command a couple weeks ago. We had a command in our history that we wanted to edit, but it was not the last command. In fact, it was several 100 commands up in our history.

It turns out that the fc command takes an id argument, and each item in history has an id. To find the id of the particular command we wanted we ran:

> history | grep curl
10256 curl www.google.com
10500 curl www.amazon.com

To edit the first command listed in curl we just use:

fc 10256

H/T Ifu Aniemeka

Execute raw parameterized sql with Ecto in Elixir

Using sql directly is a good way to use some of the more interesting parts of sqlthat Ecto does not provide a ready abstraction for in its dsl. And parameterization is necessary to avoid sql injection and just to be able to deal with single quotes correctly. Its fairly straight forward once you find the right module (Ecto.Adapters.SQL) and function (query or query!). Parameters are indicated by $1.

sql = """
  select * from users where name = $1;
"""

results = Ecto.Adapters.SQL.query!(MyApp.Repo, sql, ["chris"])

There is also a stream function on the Ecto.Adapters.SQL module with the same signature as query but the documentation doesn’t necessarily state the advantages or situations where it may be useful. More to learn.f8

Load all records for an association in Ecto

Lets say you get a collection of Posts with:

posts = Repo.all(Post)

If you try to access the associated developer for one of these posts, you get an error:

hd(posts).developer
#Ecto.Association.NotLoaded<association :developer is not loaded>

The same is true for all posts in the collection, none of them have a loaded developer. You can load them one by one with:

developer = Tilex.Repo.one(Ecto.assoc(post, :developer))

But if you need to do this for each developer you’ll get into an N+1 situation, making an additional trip to the database for each post.

Instead you can load all the developers at one time with:

posts = Repo.preload(posts, :developer)

Clearing Phoenix npm cache on Heroku

I recently encountered an issue where the solution, as found on the internet, was to remove all the npm_modules, re-install them and build again. My issue however was on Heroku rather than local.

If you’re using Elixier and Phoenix on Heroku then you’re likely using the heroku-buildpack-phoenix-static. This buildpack compiles the static assets for Phoenix.

This buildpack caches npm_modules by default, to clear the cache you must provide a configuration file at the root level of your project named phoenix_static_buildpack.config that looks like:

clean_cache=true

The buildpack will now download the npm_modules on each deployment.

Suppress errors when substituting in vim

The substitute (or s) command in vim throws an error the search pattern is not found.

:substitute/banana/apple
E486: Pattern not found: banana

This error can be suppressed with the e flag at the end of the statement.

:substitute/banana/apple/e

When operating in just one file or on just one line the error is a minor annoyance but when combining the substitute command with :argdo you receive an error for every file in the args list that does not contain the pattern in question.

A typical argdo command would look like this.

:argdo s/banana/apple/gec

The flags are:

g - global

e - suppress errors

c - confirm

tmux copy-mode -u, useless or useful?

Tmux comes with a key binding for both copy-mode and copy-mode -u. The first is straight forward, you can navigate through the window history, highlight something and place it into your tmux buffer. I was curious about what the -u flag was for, and the man page says:

The -u option scrolls one page up.

So the -u option just places you into copy-mode one page up. Useless! Or wait, maybe that’s something I want…

Difference between `union` and `union all` in psql

union is an operator that operates on two sets of results, combining them:

chris=# select 1 union select 2;
 ?column?
----------
        1
        2
(2 rows)

It filters duplicate results though:

chris=# select 1 union select 1;
 ?column?
----------
        1
(1 row)

If you want to not worry about duplicate results and just stack the result sets on top of each other use union all:

chris=# select 1 union all select 1;
 ?column?
----------
        1
        1
(2 rows)

Postgres Natural joins squash unneeded columns

Dealing mostly with Rails style databases we generally write inner joins on ids and foreign keys but there are other ways to create joins, the most fun being the natural join.

The below query works like a natural join but uses inner join syntax. When two tables having columns with the same name, join on that column. You can see that both columns have been returned even though they have the same value.

chris=# select * from (select 1 x) foo inner join (select 1 x) bar on bar.x = foo.x ;
 x | x
---+---
 1 | 1
(1 row)

When using the natural join syntax the query is less verbose and only returns one column. Postgres knows that they have the same value, thats the whole point! So only present one column.

chris=# select * from (select 1 x) foo natural join (select 1 x) bar;
 x
---
 1
(1 row)

Skip all changes in this file

git add --patch gives you the opportunity to make a decision on every code change in all files individually, providing you a menu that looks like this:

Stage this hunk [y,n,q,a,d,/,j,J,g,e,?]?

Option d is

d - do not stage this hunk or any of the later hunks in the file

This comes in handy when there is a host of whitespace changes in a file that you don’t want to commit. Just hit d to skip all those whitespace changes and go to the next file.

Postgres `coalesce` errors with multiple types

The Postgres function coalesce takes a variable number of arguments and returns the first non-null argument.

psql> select coalesce(null, null, null, null, 'hi!');
hi!

But if you include a number and a string, then Postgres throws an error.

psql> select coalesce(1, 'hi!');
ERROR:  invalid input syntax for integer: "hi!"

Coalesce will try to coerce value to the type of the first argument if possible.

psql> select coalesce(1, '2');
1

Mysql behaves a bit differently, it allows values of different types.

mysql> select coalesce(1, 'hi!');
1

The postgres way makes a bit more sense to me. What is the type of a column if there are multiple possible types? Is it always a string? But I wanted a number? I feel like the additional type safety will help catch errors a bit earlier.

Has ActiveRecord::Relation already been grouped?

I’ve recently run into a situation where I needed to apply a group statement to an ActiveRecord scope if a group statement had already been applied. In this case we need to examine the current scope to see if in fact a group statement has already been applied. ActiveRecord::Relation fortunately has a group_values method that returns an array of all the columns that the query has been grouped by as symbols.

if current_scope.group_values.length > 1
  current_scope.group(:another_column)
else
  current_scope
end

Operators to get a string or more json in Postgres

There are two operators in Postgres that I have trouble telling apart. The -> operator and the ->> operator. They both take a json object on the left and a key on the right, but result in different types.

Given json in postgres that looks like this:

> select '{"a": {"b": 2}}'::jsonb;
{"a": {"b": 2 }}

I can traverse the data structure with the -> operator because it returns a data structure.

> select '{"a": {"b": 2}}'::jsonb  -> 'a' -> 'b';
2

I can get a value as a string from the json with the ->> operator.

> select '{"a": {"b": 2}}'::jsonb ->> 'a';
{"b": 2 } -- ITS A STRING

But if you use the ->> operator to get a string, then you can’t traverse the data structure with the -> operator anymore.

> select '{"a": {"b": 2}}'::jsonb ->> 'a' -> 'b';
ERROR:  operator does not exist: text -> unkown

Expect page to have a link... an exact link

Sometimes, while writing integration tests, we expect there to not be a link on a page, for instance if our page looked like this:

<a href="#create_jetpack">Create Jetpack</a>

And we want to make sure there is no general create link so we expect this:

expect(page).to_not have_link('Create')

But the expectation fails! By default capybara will match if the substring is anywhere in the content.

To avoid this problem use the exact flag like:

expect(page).to_not have_link('Create', exact: true)

Weekly jobs in crontab

The syntax for crontab entries is confusing at best. Fortunately, cron provides easy to read special strings for common cases.

If you want a job to run once a week your crontab would look like this:

@weekly /usr/bin/my_job

And thats it! @weekly in this case means 0 0 * * 0. Every Sunday at midnight.

You can check other crontab special strings with man 5 crontab.

H/T Dorian Karter

Vim move cursor within a visual selection

When expanding a visual selection you might want to increase both sides. For instance, if you have made a selection from line 20 to line 40 you may want to expand the selection to line 18 to line 42. The visual selection can only be expanded with the cursor and the cursor is only on one side, but the cursor can be moved to the other side of the selection with o.

Read more about it with: :help visual-change.

Typing card suits in vim ♢♡♣♠

Digraphs are two letter combinations that provide a easy way to type and remember non-ascii characters in vim. In insert mode you can type a digraph with Ctrl-k and then two letters that map to the unicode character.

For card suits the pneumonic is lower case c for card and upper case C for clubs, D for diamonds, H for hearts, and S for spades.

cD ♢
cC ♣
cH ♡
cS ♠

You can see all the digraphs with :digraphs.

Now you have all the necessary unicode characters to easily create card games in the terminal!

Resetting the bundler --path

When running bundler install you can pass an option --path <dir>:

bundle install --path ./gems

Which will specify where to install the gems. This is a one time option. Everytime you invoke bundle from here on in you’ll see that its using the directory you specificed with the —path variable as the gem directory.

Bundler keeps track of this option in the .bundler/config file in the folder from where bundler was initially run. That file looks like this:

---
BUNDLE_PATH: "./gems"
BUNDLE_DISABLE_SHARED_GEMS: "true"

To reset the gems path back to the default, remove the BUNDLE_PATH line from the bundle/config file.f

Override ssh command for git

Git supports a number of environment variables one of them being GIT_SSH. You can override the default ssh command (ssh of course) with that environment variable.

GIT_SSH=./my_ssh git pull origin master

In this case my_ssh is a bash executable file that looks like this:

#!/bin/bash

ssh -i ~/.ssh/id_rsa $1 $2

An idea I got from this stack overflow answer.

Clearly the private key file I used is the default private key file so a bit redundant but its a way to add any options that might be necessary for the ssh connection, although specifying a host configuration in the ssh_config file might be simpler.

Reinstall all rubies with rvm

When compiling ruby the executable is attached to various C system libs like libreadline. If libreadline is updated then each ruby on the system attached to the old libreadline must be re-compiled. If you are using rvm then this can be done with one command:

rvm reinstall all --force

The --force flag will skip any question the process asks.

Be careful, if you are working on a machine that has installed many rubies and has had many projects for those rubies this could take an extraordinarily long time.

Specifying the private key to use in ssh

A private key is necessary in ssh to authenticate the connection securely. Generally ssh will look for a file in the ~/.ssh/ dir with a name of id_rsa (or if not using rsa a file beginning with id_ and the encryption algo name). That file should contain the private key.

You can, however, specify a different primary key at the command line with the -i flag like so:

ssh -i ~/.ssh/my_other_key.pem

Additionally, you may use the IdentityFile option the the ssh_config file to permanently configure the private key for a specific connection.

Hack to size the container of a background image

An element that contains a background image cannot have its height and width derived from that background image. An element can have a height and width that is determined by the size of a child element however. Combine the two to ensure that the container of the background image has the height and width of the image itself.

<div style="background-image: url('http://example.com/image.png')">
    <img src="http://example.com/image.png">
</div>

`border-collapse` to collapse borders

There are two settings for the css property border-collapse, separate and collapse. Table cells and the table itself can both have borders. If the value separate is set, then each cell has its own border, with the space in between the borders governed by the border-spacing property.

If collapse is set, then the table cells and the surrounding table will share the borders surrounding them.

`any` and `all` special Postgres constructs

To determine if a value exists in a subquery or scalar value list you can use the in operator like:

> select 1 in (1, 2)
true

any and all go a step further by allowing for an operator in the special construct which is used against each value in the subquery or array (no scalar value list for these constructs).

any examples:

> select 1 = any(Array[1, 2, 3])
true
> select 1 > any(Array[1, 2, 3])
false

all examples

> select 17 <> any(Array[2, 4, 8])
true
> select 17 < any(Array[2, 4, 8])
false

some is a synonym for any.

Remove, Get, And Compile a dependency

To get a better feel for how a dependency works I’ll do some IO.puts style debugging, but when I’m done I need to clean up all those IO.puts statements.

mix deps.clean ecto

The above command will remove two directories, the _build/dev/lib/ecto dir and the deps/ecto dir.

mix deps.get ecto

This command will resolve the dependency tree and get the appropriate version of ecto based on the information in your mix.exs file. It will not reach out to the internet if the package is already cached as a tar file in the .hex/packages/hexpm dir. It will unpack the tar file into the local deps/ecto dir but it will not compile it.

mix deps.compile ecto

Will compile ecto to the _build/dev/lib/ecto directory.

What file is this?

If you are working in an environment where the status line is not displayed and you don’t know what file you are currently editing, you can use the :file command to give you that information.

:file
"app/models/post.rb" line 43 of 139 --30%-- col 4

:f is the abbreviated version