Today I Learned

A Hashrocket project

218 posts by chriserin @mcnormalmode

Specifying the location of your vim plugin

If you are working on a vim plugin and you want to keep this plugin amongst your projects rather than with your other vim plugins you can add a new entry to the runtimepath or rtp.

Place the following in your vimrc:

set runtimepath+=~/projects/vim-my-plugin

Or

set rtp+=~/projects/vim-my-plugin

You can confirm that the path was added correctly by checking that setting with:

set rtp

Resize vim window to the size of its content

If you’re writing a vim plugin and in that plugin you are opening a new window to display some content, you may want to resize that window so that it is only the size of the content within. For instance, if you have 10 lines of content but the window is half the screen, then you want to resize the window to 10 lines.

Resizing a window is accomplished with the resize command like:

:resize 10

Getting the number of content lines is accomplished by getting the line number of the last last:

:echo line('$')

Putting the two together is tricky because the argument to the resize command is interpreted literally. :resize line('$') does not work. You have to combine the two into a string and then pass it to the execute command.

:execute('resize ' . line('$'))

Configure max http header size in Elixir Phoenix

If you store lots of data in cookies, you may eventually run up against the maximum header value length in Cowboy which by default is 4096.

While storing that much data in cookies is maybe not the best idea, there is a configuration setting that can help you work around this issue:

config :myapp, MyAppWeb.Endpoint,                                                                                                                 
  http: [protocol_options: [max_request_line_length: 8192, max_header_value_length: 8192]]

There ya go! Now you can accept http header values up to 8192 bytes in length!

Check to see if a command exists at #shell

If you want to check to see if a command exists on a user’s machine you can use command -v. command without flags will run the command passed as an argument.

> command echo "A"
"A"

command -v with the -v flag will return the path of the command, and most importantly, a non-zero status code if that command does not exist.

> command -v brew
/usr/local/bin/brew

Or on Linux

> command -v brew
# returns nothing and also returns a status code of 1

Importantly, you can check for a command before using it.

command -v brew && brew install something

Phoenix will watch your JS for you, just watch!

With es6 javascript ecosystems most compilation/transpilation is done with watchers, programs that watch for changes in your code and then trigger compilation based on the changes you’ve made.

Phoenix has a method of integrating those watchers into the server itself.

config :myapp, MyApp.Endpoint,
  watchers: [yarn: ["run", "watch", cd: Path.expand("../assets", __DIR__)]]

The above configuration goes in config/dev.exs and runs the watch command whenever you start the server/endpoint. The keyword option cd is as of yet undocumented, but does what you think it does, changes the directory to the path you configure, which by convention in Phoenix 1.3 would be the assets directory. cd is the only keyword option.

The watchers start when you run your server. Just make a change to your javascript and refresh the page!

String concatentation in the Bourne Again Shell

String concatenation must be different in every lanaguage, or so it seems, and Bash is no different! You can string concat just by putting too strings next to each other.

> echo "a""b"
ab 

Its ok if one of these strings is a variable.

> a = "x" 
> echo $a"b"
xb

Its ok if you put a double quoted string next to a single quoted string.

> echo 'a'"b"
ab

Its ok if you put spaces in between the strings, but that space now will be part of the new string.

> echo "a" "b"
a b

But multiple spaces in between strings will be squashed to one string.

> echo "a"      "b"
a b

This works when setting a variable too, but only surrounded by parens.

> c=('a'      'b')
> echo $c
a b

Don’t worry about the parens though when they are right next to each other.

> c = "a""b"
> echo $c
ab

Edit bash command at certain point in history

Jack introduced our team to the fc command a couple weeks ago. We had a command in our history that we wanted to edit, but it was not the last command. In fact, it was several 100 commands up in our history.

It turns out that the fc command takes an id argument, and each item in history has an id. To find the id of the particular command we wanted we ran:

> history | grep curl
10256 curl www.google.com
10500 curl www.amazon.com

To edit the first command listed in curl we just use:

fc 10256

H/T Ifu Aniemeka

Execute raw parameterized sql with Ecto in Elixir

Using sql directly is a good way to use some of the more interesting parts of sqlthat Ecto does not provide a ready abstraction for in its dsl. And parameterization is necessary to avoid sql injection and just to be able to deal with single quotes correctly. Its fairly straight forward once you find the right module (Ecto.Adapters.SQL) and function (query or query!). Parameters are indicated by $1.

sql = """
  select * from users where name = $1;
"""

results = Ecto.Adapters.SQL.query!(MyApp.Repo, sql, ["chris"])

There is also a stream function on the Ecto.Adapters.SQL module with the same signature as query but the documentation doesn’t necessarily state the advantages or situations where it may be useful. More to learn.f8

Load all records for an association in Ecto

Lets say you get a collection of Posts with:

posts = Repo.all(Post)

If you try to access the associated developer for one of these posts, you get an error:

hd(posts).developer
#Ecto.Association.NotLoaded<association :developer is not loaded>

The same is true for all posts in the collection, none of them have a loaded developer. You can load them one by one with:

developer = Tilex.Repo.one(Ecto.assoc(post, :developer))

But if you need to do this for each developer you’ll get into an N+1 situation, making an additional trip to the database for each post.

Instead you can load all the developers at one time with:

posts = Repo.preload(posts, :developer)

Clearing Phoenix npm cache on Heroku

I recently encountered an issue where the solution, as found on the internet, was to remove all the npm_modules, re-install them and build again. My issue however was on Heroku rather than local.

If you’re using Elixier and Phoenix on Heroku then you’re likely using the heroku-buildpack-phoenix-static. This buildpack compiles the static assets for Phoenix.

This buildpack caches npm_modules by default, to clear the cache you must provide a configuration file at the root level of your project named phoenix_static_buildpack.config that looks like:

clean_cache=true

The buildpack will now download the npm_modules on each deployment.

Suppress errors when substituting in vim

The substitute (or s) command in vim throws an error the search pattern is not found.

:substitute/banana/apple
E486: Pattern not found: banana

This error can be suppressed with the e flag at the end of the statement.

:substitute/banana/apple/e

When operating in just one file or on just one line the error is a minor annoyance but when combining the substitute command with :argdo you receive an error for every file in the args list that does not contain the pattern in question.

A typical argdo command would look like this.

:argdo s/banana/apple/gec

The flags are:

g - global

e - suppress errors

c - confirm

tmux copy-mode -u, useless or useful?

Tmux comes with a key binding for both copy-mode and copy-mode -u. The first is straight forward, you can navigate through the window history, highlight something and place it into your tmux buffer. I was curious about what the -u flag was for, and the man page says:

The -u option scrolls one page up.

So the -u option just places you into copy-mode one page up. Useless! Or wait, maybe that’s something I want…

Difference between `union` and `union all` in psql

union is an operator that operates on two sets of results, combining them:

chris=# select 1 union select 2;
 ?column?
----------
        1
        2
(2 rows)

It filters duplicate results though:

chris=# select 1 union select 1;
 ?column?
----------
        1
(1 row)

If you want to not worry about duplicate results and just stack the result sets on top of each other use union all:

chris=# select 1 union all select 1;
 ?column?
----------
        1
        1
(2 rows)

Postgres Natural joins squash unneeded columns

Dealing mostly with Rails style databases we generally write inner joins on ids and foreign keys but there are other ways to create joins, the most fun being the natural join.

The below query works like a natural join but uses inner join syntax. When two tables having columns with the same name, join on that column. You can see that both columns have been returned even though they have the same value.

chris=# select * from (select 1 x) foo inner join (select 1 x) bar on bar.x = foo.x ;
 x | x
---+---
 1 | 1
(1 row)

When using the natural join syntax the query is less verbose and only returns one column. Postgres knows that they have the same value, thats the whole point! So only present one column.

chris=# select * from (select 1 x) foo natural join (select 1 x) bar;
 x
---
 1
(1 row)

Skip all changes in this file

git add --patch gives you the opportunity to make a decision on every code change in all files individually, providing you a menu that looks like this:

Stage this hunk [y,n,q,a,d,/,j,J,g,e,?]?

Option d is

d - do not stage this hunk or any of the later hunks in the file

This comes in handy when there is a host of whitespace changes in a file that you don’t want to commit. Just hit d to skip all those whitespace changes and go to the next file.

Postgres `coalesce` errors with multiple types

The Postgres function coalesce takes a variable number of arguments and returns the first non-null argument.

psql> select coalesce(null, null, null, null, 'hi!');
hi!

But if you include a number and a string, then Postgres throws an error.

psql> select coalesce(1, 'hi!');
ERROR:  invalid input syntax for integer: "hi!"

Coalesce will try to coerce value to the type of the first argument if possible.

psql> select coalesce(1, '2');
1

Mysql behaves a bit differently, it allows values of different types.

mysql> select coalesce(1, 'hi!');
1

The postgres way makes a bit more sense to me. What is the type of a column if there are multiple possible types? Is it always a string? But I wanted a number? I feel like the additional type safety will help catch errors a bit earlier.

Has ActiveRecord::Relation already been grouped?

I’ve recently run into a situation where I needed to apply a group statement to an ActiveRecord scope if a group statement had already been applied. In this case we need to examine the current scope to see if in fact a group statement has already been applied. ActiveRecord::Relation fortunately has a group_values method that returns an array of all the columns that the query has been grouped by as symbols.

if current_scope.group_values.length > 1
  current_scope.group(:another_column)
else
  current_scope
end

Operators to get a string or more json in Postgres

There are two operators in Postgres that I have trouble telling apart. The -> operator and the ->> operator. They both take a json object on the left and a key on the right, but result in different types.

Given json in postgres that looks like this:

> select '{"a": {"b": 2}}'::jsonb;
{"a": {"b": 2 }}

I can traverse the data structure with the -> operator because it returns a data structure.

> select '{"a": {"b": 2}}'::jsonb  -> 'a' -> 'b';
2

I can get a value as a string from the json with the ->> operator.

> select '{"a": {"b": 2}}'::jsonb ->> 'a';
{"b": 2 } -- ITS A STRING

But if you use the ->> operator to get a string, then you can’t traverse the data structure with the -> operator anymore.

> select '{"a": {"b": 2}}'::jsonb ->> 'a' -> 'b';
ERROR:  operator does not exist: text -> unkown

Expect page to have a link... an exact link

Sometimes, while writing integration tests, we expect there to not be a link on a page, for instance if our page looked like this:

<a href="#create_jetpack">Create Jetpack</a>

And we want to make sure there is no general create link so we expect this:

expect(page).to_not have_link('Create')

But the expectation fails! By default capybara will match if the substring is anywhere in the content.

To avoid this problem use the exact flag like:

expect(page).to_not have_link('Create', exact: true)

Vim move cursor within a visual selection

When expanding a visual selection you might want to increase both sides. For instance, if you have made a selection from line 20 to line 40 you may want to expand the selection to line 18 to line 42. The visual selection can only be expanded with the cursor and the cursor is only on one side, but the cursor can be moved to the other side of the selection with o.

Read more about it with: :help visual-change.

Typing card suits in vim ♢♡♣♠

Digraphs are two letter combinations that provide a easy way to type and remember non-ascii characters in vim. In insert mode you can type a digraph with Ctrl-k and then two letters that map to the unicode character.

For card suits the pneumonic is lower case c for card and upper case C for clubs, D for diamonds, H for hearts, and S for spades.

cD ♢
cC ♣
cH ♡
cS ♠

You can see all the digraphs with :digraphs.

Now you have all the necessary unicode characters to easily create card games in the terminal!

Resetting the bundler --path

When running bundler install you can pass an option --path <dir>:

bundle install --path ./gems

Which will specify where to install the gems. This is a one time option. Everytime you invoke bundle from here on in you’ll see that its using the directory you specificed with the —path variable as the gem directory.

Bundler keeps track of this option in the .bundler/config file in the folder from where bundler was initially run. That file looks like this:

---
BUNDLE_PATH: "./gems"
BUNDLE_DISABLE_SHARED_GEMS: "true"

To reset the gems path back to the default, remove the BUNDLE_PATH line from the bundle/config file.f

Override ssh command for git

Git supports a number of environment variables one of them being GIT_SSH. You can override the default ssh command (ssh of course) with that environment variable.

GIT_SSH=./my_ssh git pull origin master

In this case my_ssh is a bash executable file that looks like this:

#!/bin/bash

ssh -i ~/.ssh/id_rsa $1 $2

An idea I got from this stack overflow answer.

Clearly the private key file I used is the default private key file so a bit redundant but its a way to add any options that might be necessary for the ssh connection, although specifying a host configuration in the ssh_config file might be simpler.

Reinstall all rubies with rvm

When compiling ruby the executable is attached to various C system libs like libreadline. If libreadline is updated then each ruby on the system attached to the old libreadline must be re-compiled. If you are using rvm then this can be done with one command:

rvm reinstall all --force

The --force flag will skip any question the process asks.

Be careful, if you are working on a machine that has installed many rubies and has had many projects for those rubies this could take an extraordinarily long time.

Specifying the private key to use in ssh

A private key is necessary in ssh to authenticate the connection securely. Generally ssh will look for a file in the ~/.ssh/ dir with a name of id_rsa (or if not using rsa a file beginning with id_ and the encryption algo name). That file should contain the private key.

You can, however, specify a different primary key at the command line with the -i flag like so:

ssh -i ~/.ssh/my_other_key.pem

Additionally, you may use the IdentityFile option the the ssh_config file to permanently configure the private key for a specific connection.

Hack to size the container of a background image

An element that contains a background image cannot have its height and width derived from that background image. An element can have a height and width that is determined by the size of a child element however. Combine the two to ensure that the container of the background image has the height and width of the image itself.

<div style="background-image: url('http://example.com/image.png')">
    <img src="http://example.com/image.png">
</div>

`border-collapse` to collapse borders

There are two settings for the css property border-collapse, separate and collapse. Table cells and the table itself can both have borders. If the value separate is set, then each cell has its own border, with the space in between the borders governed by the border-spacing property.

If collapse is set, then the table cells and the surrounding table will share the borders surrounding them.

`any` and `all` special Postgres constructs

To determine if a value exists in a subquery or scalar value list you can use the in operator like:

> select 1 in (1, 2)
true

any and all go a step further by allowing for an operator in the special construct which is used against each value in the subquery or array (no scalar value list for these constructs).

any examples:

> select 1 = any(Array[1, 2, 3])
true
> select 1 > any(Array[1, 2, 3])
false

all examples

> select 17 <> any(Array[2, 4, 8])
true
> select 17 < any(Array[2, 4, 8])
false

some is a synonym for any.

Remove, Get, And Compile a dependency

To get a better feel for how a dependency works I’ll do some IO.puts style debugging, but when I’m done I need to clean up all those IO.puts statements.

mix deps.clean ecto

The above command will remove two directories, the _build/dev/lib/ecto dir and the deps/ecto dir.

mix deps.get ecto

This command will resolve the dependency tree and get the appropriate version of ecto based on the information in your mix.exs file. It will not reach out to the internet if the package is already cached as a tar file in the .hex/packages/hexpm dir. It will unpack the tar file into the local deps/ecto dir but it will not compile it.

mix deps.compile ecto

Will compile ecto to the _build/dev/lib/ecto directory.

What file is this?

If you are working in an environment where the status line is not displayed and you don’t know what file you are currently editing, you can use the :file command to give you that information.

:file
"app/models/post.rb" line 43 of 139 --30%-- col 4

:f is the abbreviated version

Fingerpring of an ssh key in multiple formats

You might need to confirm which keys you have attached to your github account. Github provides a list of your keys with the associated fingerprints. If you have a public key on your machine you can see what fingerprint it has with:

> ssh-keygen -lf ~/.ssh/id_rsa.pub
4096 SHA256:/6Rat4zVZ0auHEwWtF4QG8NA0j4NAKWisFuXV0ZP5zk chris.erin@gmail.com (RSA)

Github displays an md5 fingerprint rather than a sha256 fingerprint. You can use the -E flag to choose a specific hash type.

> ssh-keygen -E md5 -lf ~/.ssh/id_rsa.pub
4096 MD5:e3:19:85:1c:8a:31:0f:4b:de:cb:6d:a7:64:b4:4e:e7 chris.erin@gmail.com (RSA)

Logging to a file in Elixir test environment

The default configuration for a logger in the test env is:

config :logger, level: :warn

That is, of the four logging levels (defug, info, warn and error) only log warnings and errors.

By default this logger logs to the console making the below redundant:

config :logger, level: :warn, backends: [:console]

:console is the only option available by default, but there are other backends available via hex package.

With hex package logger_file_backend you might have a different config:

config :logger, backends: [{LoggerFileBackend, :error_log}]

config :logger, :error_log, path: 'log/here.log'

Compile an Elixir dependency as you change it

In an Elixir project, the dependencies are one-time compiles. That makes sense when you know the dependency is not going to change. It stops making sense, however, when you need to explore how a dependency works by adding debug statements like IO.puts. Or when you find something in a dependency you want to change and need to test that change in an application.

In those cases Elixir provides the facility to compile a dependency each time it has a change in one of the component files just like how compilation in your mix app works. You can enable this by providing a path to the location of the dependency in the mix.exs file:

defp dep do
  [
  ...
  {:wallaby, '~> 0.16.1', path: 'deps/wallaby'}
  ...
  ]
end

In this case the path is set to the local deps dir, where the dependency already exists, but you can also set it to a working directory for that library.

Select to clipboard at the ubuntu command line

Copying and pasting without a mouse - or programmatically - can be incredibly challenging. Ubuntu provides an apt-get installable program xclip which can provide X11 clipboard access at the command line.

> echo PASTEME | xclip -sel clip

The value PASTEME is now in the clipboard buffer.

The -sel or -selection indicates the buffer that will be used, primary, secondary or clipboard. Generally, clipboard is the buffer that we want.

To output the value of the buffer use the -o or -output flag:

> xclip -sel clip -o
PASTEME

Keeping track of your CPU Heat

If you’ve ever built your own computer, you’ve had anxiety about not having put everything together right and maybe even something critical, like the mechanisms that keep the computer and the CPU cool.

On Ubuntu you can intall the lm-sensors software which provides you with the facilities to get information the various heat levels inside the computer.

> sudo apt-get install lm-sensors
> sensors
asus-isa-0000
Adapter: ISA adapter
cpu_fan:        0 RPM

acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +119.0°C)
temp2:        +29.8°C  (crit = +119.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +26.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:        +23.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:        +24.0°C  (high = +80.0°C, crit = +100.0°C)
Core 2:        +24.0°C  (high = +80.0°C, crit = +100.0°C)
Core 3:        +24.0°C  (high = +80.0°C, crit = +100.0°C)

Restart or shutdown ubuntu

Its easy to shutdown down Ubuntu from the commandline (given the correct permissions). Shutdown with shutdown which gives you 60 seconds to reverse that decision to shutdown by cancelling with shutdown -c. You can also reboot which takes effect right away.

shutdown # shutdown
shutdown -c # cancel that shutdown.
reboot # reboot the computer now

Ubuntu default desktop manager

What I like about linux is that there are configurations. You should be able to manipulate the whole system without a gui. The configuration for your default display manager, for instance, is in the default-display-manager file:

> cat /etc/X11/default-display-manager
/usr/sbin/lightdm

There’s a startup script /etc/init.d/lightdm that checks that the value stored in this file points to a lightdm program and then executes that program.

Opening Gnome terminal emulator

In an attempt to get better with keyboard shortcuts on Ubuntu I’ve - for the time being - ditched the mouse. I generally only use two applications, Firefox and Terminal.

Opening a new terminal is as easy as:

Ctrl + Shift + T

It starts rather small but I resize it by using left or right with the Super (window/apple/command) key.

Super + Left
Super + Right

Then its resized to either the left half of the screen or the right half of the screen.

Create a github repository from the cmd line

In the effort to never leave the command line anything to do with github is always a frustration. The hub command line tool - a tool that augments git with github specific commands - helps out tremendously.

In this case I want to create a github repository to push my code to without opening a browser, going to github, creating a repo, copying the remote address for that repo and then setting up the remote locally.

It can all be done with one step.

hub create "exciting_open_source_tech_repo"

git init must be run first, so that there is a local repo to link to the github repo, but then your good. Start modifying, adding and commiting as you normally would.

Webmock assertion that an http request was made

Webmock is a library that helps you manager the http requests that are made during a test. Its not advisable to hit production systems with test-generated requests and with webmock you can stub those requests. Sometimes to ensure functionality of your system you might want to assert that the request was made.

stub_request(:get, 'http://www.google.com')
assert_requested(:get, 'http://www.google.com')

The stub_request method also returns a variable that can be passed to the assert function.

get_google = stub_request(:get, 'http://www.google.com')
assert_requested get_google

Time travelling in rspec/rails

When basing logic on the current time its helpful for testing to have a stable time. A time that does not change. Rails has a module ActiveSupport::Testing::TimeHelpers that was added in Rails 4.2 to provide methods that manipulate the time during testing.

travel_to(Time.parse("2017-01-19")) do
  puts Time.now.strftime(:date)
end

puts Time.now.strftime(:date)

The above code outputs 2017-01-19 and 2017-05-02 (the current date). A fun way to time travel in modern ruby.

Private vs Public Struct members

In Golang, identifiers can be either Exported or Unexported. In other languages this would be known as public or private, but in golang the question is whether the identifiers should be able to be seen outside of the module. If not? Don’t export them.

The rule is simple, if an identifier is capitalized it will be exported. This manifests itself in golang when converting a struct to JSON.

type Apple struct {
    color string
    weight int
}

a := json.Marshal(Apple{"green", 10})
fmt.Println(a)

In this case this code will print {}. To print what you would expect the struct should be defined with capitalized identifiers:

type Apple struct {
    Color string
    Weight int
}

Now the marshalled struct should be {"Color": "green", "Weight": 10}

Add files with matches to the args list

If I want to change all instances of the word Apple to Orange across my entire project I can use the vim command:

:argdo %s/Apple/Orange/gc

But only if the files that have the word Apple are in the args list. To populate the args list you can use the args command in combination with shelling out to git grep with the -l flag.

:args `git grep -l Apple`

Then, when you see the list of files with :args, you’ll see all the files with the word Apple.

A Mnemonic for Changing Modes

Making a script file executable involves an incantation that has something to do with chmod but what exactly? chmod has two modes, absolute and symbolic. In absolute mode, you set the entire set of permissions with a numeric value like 755.

In symbolic mode, you can add or subtract permissions to either the user (u), group (g) or owner (o). To give read permissions to the group type:

> chmod g+r myfile.txt

And to give execution priviledges to the owner of a file:

> chmod o+x myscript.rb