Today I Learned

A Hashrocket project

Logging to a file in Elixir test environment

The default configuration for a logger in the test env is:

config :logger, level: :warn

That is, of the four logging levels (defug, info, warn and error) only log warnings and errors.

By default this logger logs to the console making the below redundant:

config :logger, level: :warn, backends: [:console]

:console is the only option available by default, but there are other backends available via hex package.

With hex package logger_file_backend you might have a different config:

config :logger, backends: [{LoggerFileBackend, :error_log}]

config :logger, :error_log, path: 'log/here.log'

Compile an Elixir dependency as you change it

In an Elixir project, the dependencies are one-time compiles. That makes sense when you know the dependency is not going to change. It stops making sense, however, when you need to explore how a dependency works by adding debug statements like IO.puts. Or when you find something in a dependency you want to change and need to test that change in an application.

In those cases Elixir provides the facility to compile a dependency each time it has a change in one of the component files just like how compilation in your mix app works. You can enable this by providing a path to the location of the dependency in the mix.exs file:

defp dep do
  [
  ...
  {:wallaby, '~> 0.16.1', path: 'deps/wallaby'}
  ...
  ]
end

In this case the path is set to the local deps dir, where the dependency already exists, but you can also set it to a working directory for that library.

Set foreign key to null on delete in #Postgres

Say you have a table with a foreign key:

posts
------
id serial primary key
...
primary_image_id references images (id)

images
------
id serial primary key
post_id references posts (id)

If you attempted to delete an image from the images table while that image's ID is referenced in one of the posts you will receive an error preventing you from deleting the image. Postgres is trying to protect your data integrity by preventing foreign keys that point to records which don't exist.

To prevent errors and allow deleting records from the images table freely you must define the on delete strategy on the posts table.

One of the options available to you is set null so if you are creating a new table it will look like this:

create table posts (
  id int serial primary key,
  -- ...
  primary_image_id int references images (id) on delete set null
);

Now if the primary image is deleted it will set the primary_image_id to null.

This an alternative to on delete cascade which in this case will delete the post from the posts table and is not what we want.

Read the full documentation under '5.3.5. Foreign Keys'

Create A List Of Atoms

The ~w sigil makes it easy to create a word list -- a list of strings -- where each word is separated by a space.

> ~w(bulbasaur charmander squirtle)
["bulbasaur", "charmander", "squirtle"]

By appending an a onto that sigil construct, you are instructing Elixir that you would instead like a list of atoms.

> ~w(bulbasaur charmander squirtle)a
[:bulbasaur, :charmander, :squirtle]

source

Compute group proportions of a whole in SQL

It's easy with SQL to use aggregates to count members of a group. But what if you also want the proportion that group is of the whole. The answer is to use a window function combined with the aggregate. The window function is evaluated after the normal aggregation. i.e. it can reference the values of other post-aggregation rows.

select type, count(*), count(*)::numeric / sum(count(*)) over ()
from pets
group by type;
  type  | count |        ?column?        
--------+-------+------------------------
 dog    |    13 | 0.52000000000000000000
 rabbit |     3 | 0.12000000000000000000
 cat    |     9 | 0.36000000000000000000

Select to clipboard at the ubuntu command line

Copying and pasting without a mouse - or programmatically - can be incredibly challenging. Ubuntu provides an apt-get installable program xclip which can provide X11 clipboard access at the command line.

> echo PASTEME | xclip -sel clip

The value PASTEME is now in the clipboard buffer.

The -sel or -selection indicates the buffer that will be used, primary, secondary or clipboard. Generally, clipboard is the buffer that we want.

To output the value of the buffer use the -o or -output flag:

> xclip -sel clip -o
PASTEME

Keeping track of your CPU Heat

If you've ever built your own computer, you've had anxiety about not having put everything together right and maybe even something critical, like the mechanisms that keep the computer and the CPU cool.

On Ubuntu you can intall the lm-sensors software which provides you with the facilities to get information the various heat levels inside the computer.

> sudo apt-get install lm-sensors
> sensors
asus-isa-0000
Adapter: ISA adapter
cpu_fan:        0 RPM

acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +119.0°C)
temp2:        +29.8°C  (crit = +119.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +26.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:        +23.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:        +24.0°C  (high = +80.0°C, crit = +100.0°C)
Core 2:        +24.0°C  (high = +80.0°C, crit = +100.0°C)
Core 3:        +24.0°C  (high = +80.0°C, crit = +100.0°C)

Pretty Printing JSONB Rows in PostgreSQL

Who needs a document store when you can just use PostgreSQL's JSONB data type? Viewing rows of JSONB output can be challenging though because it defaults to printing them as a single line of text.

> select '{"what": "is this", "nested": {"items 1": "are the best", "items 2": [1, 2, 3]}}'::jsonb;
                                      jsonb
----------------------------------------------------------------------------------
 {"what": "is this", "nested": {"items 1": "are the best", "items 2": [1, 2, 3]}}
(1 row)

Fortunately, Postgres comes with a function for prettying up the format of the output of these rows -- jsonb_pretty

> select jsonb_pretty('{"what": "is this", "nested": {"items 1": "are the best", "items 2": [1, 2, 3]}}'::jsonb);
            jsonb_pretty
------------------------------------
 {                                 +
     "what": "is this",            +
     "nested": {                   +
         "items 1": "are the best",+
         "items 2": [              +
             1,                    +
             2,                    +
             3                     +
         ]                         +
     }                             +
 }
(1 row)

h/t Jack Christensen

Restart or shutdown ubuntu

Its easy to shutdown down Ubuntu from the commandline (given the correct permissions). Shutdown with shutdown which gives you 60 seconds to reverse that decision to shutdown by cancelling with shutdown -c. You can also reboot which takes effect right away.

shutdown # shutdown
shutdown -c # cancel that shutdown.
reboot # reboot the computer now

Ubuntu default desktop manager

What I like about linux is that there are configurations. You should be able to manipulate the whole system without a gui. The configuration for your default display manager, for instance, is in the default-display-manager file:

> cat /etc/X11/default-display-manager
/usr/sbin/lightdm

There's a startup script /etc/init.d/lightdm that checks that the value stored in this file points to a lightdm program and then executes that program.

Opening Gnome terminal emulator

In an attempt to get better with keyboard shortcuts on Ubuntu I've - for the time being - ditched the mouse. I generally only use two applications, Firefox and Terminal.

Opening a new terminal is as easy as:

Ctrl + Shift + T

It starts rather small but I resize it by using left or right with the Super (window/apple/command) key.

Super + Left
Super + Right

Then its resized to either the left half of the screen or the right half of the screen.

Generating And Executing SQL

Rails' ActiveRecord can easily support 90% of the querying we do against the tables in our database. However, there is the occasional exceptional query that is more easily written in SQL -- perhaps that query cannot even be written with the ActiveRecord DSL. For these instances, we need a way to generate and execute SQL safely. The sanitize_sql_array method is invaluable for this.

First, let's get a connection and some variables that we can use downstream in our query.

> conn = ActiveRecord::Base.connection
=> #<ActiveRecord::ConnectionAdapters::PostgreSQLAdapter ...>
> one, ten = 1, 10
=> [1, 10]

Now, we are ready to safely generate our SQL query as a string. We have to use send because it is not publicly available. Generally, this is frowned upon, but in my opinion it is worth breaking the private interface to ensure our SQL is sanitized.

> sql = ActiveRecord::Base.send(:sanitize_sql_array, ["select generate_series(?, ?);", one, ten])
=> "select generate_series(1, 10);"

Lastly, we can execute the query with our connection and inspect the results.

> result = conn.execute(sql)
   (0.4ms)  select generate_series(1, 10);
=> #<PG::Result:0x007facd93128a0 status=PGRES_TUPLES_OK ntuples=10 nfields=1 cmd_tuples=10>
> result.to_a
=> [{"generate_series"=>1},
 {"generate_series"=>2},
 {"generate_series"=>3},
 {"generate_series"=>4},
 {"generate_series"=>5},
 {"generate_series"=>6},
 {"generate_series"=>7},
 {"generate_series"=>8},
 {"generate_series"=>9},
 {"generate_series"=>10}]

Create a github repository from the cmd line

In the effort to never leave the command line anything to do with github is always a frustration. The hub command line tool - a tool that augments git with github specific commands - helps out tremendously.

In this case I want to create a github repository to push my code to without opening a browser, going to github, creating a repo, copying the remote address for that repo and then setting up the remote locally.

It can all be done with one step.

hub create "exciting_open_source_tech_repo"

git init must be run first, so that there is a local repo to link to the github repo, but then your good. Start modifying, adding and commiting as you normally would.

Serve static files/directories in Phoenix

Phoenix will by default server some files and directories from priv/static. More specifically css/, fonts/, images/, js/, favicon.ico and robots.txt. If you need to add a new directory however simply creating it in priv/static will not make Phoenix serve it. You need to explicitly add it.

Say your project is called ChicagoElixir you will need to go to your endpoint configuration, typically in: lib/chicago_elixir/web/endpoint.ex. There you will find the following configuration:

  plug Plug.Static,
    at: "/", from: :chicago_elixir, gzip: false,
    only: ~w(css fonts images js favicon.ico robots.txt)

Simply add the new folder or file name to the list in only and restart your Phoenix server.

Webmock assertion that an http request was made

Webmock is a library that helps you manager the http requests that are made during a test. Its not advisable to hit production systems with test-generated requests and with webmock you can stub those requests. Sometimes to ensure functionality of your system you might want to assert that the request was made.

stub_request(:get, 'http://www.google.com')
assert_requested(:get, 'http://www.google.com')

The stub_request method also returns a variable that can be passed to the assert function.

get_google = stub_request(:get, 'http://www.google.com')
assert_requested get_google

Time travelling in rspec/rails

When basing logic on the current time its helpful for testing to have a stable time. A time that does not change. Rails has a module ActiveSupport::Testing::TimeHelpers that was added in Rails 4.2 to provide methods that manipulate the time during testing.

travel_to(Time.parse("2017-01-19")) do
  puts Time.now.strftime(:date)
end

puts Time.now.strftime(:date)

The above code outputs 2017-01-19 and 2017-05-02 (the current date). A fun way to time travel in modern ruby.

Change To That New Directory

The $_ variable provided by bash is always set to the last argument of the previous command. One handy use of this is for changing directories into a newly created directory.

$ mkdir new_dir && cd $_

This command will leave you in your newly created directory, new_dir.

We can imagine using this bash variable in a number of similar scenarios as well. What if we are using some language specific command that creates a directory? Will it work when creating a new Phoenix or Rails project?

It sure will.

Give it a try with Phoenix:

mix phx.new my_app && cd $_

or with Rails:

rails new app && cd $_

source

Mass-Delete Git Tags

Building off this post:

I'm an advocate of Semantic Version tagging. It communicates to a team about every deploy and makes that rare rollback easier. So when does it not make sense to use a tag?

When you're the only developer (nobody to communicate with except yourself), and also using a platform like Heroku that tags every release (your tags are redundant). This the case with my blog, so today I set out to delete all my Git tags.

First, delete them remotely (assuming a remote named origin):

$ git tag | xargs git push --delete origin

We also have to delete our local tags, or a tag push with create them again on the remote:

$ git tag | xargs git tag -d

$ git tag now returns nothing, and there are no remote tags.

Ins And Outs Of Pry

When executing commands during a Pry session, you'll see an incrementing number for each prompt as you enter each statement. These numbers can be used to look up the inputs and outputs of each statement executed during the session. The statements and their results are made available in the array-like _in_ and _out_ objects.

[1] pry(main)> :one
=> :one
[2] pry(main)> 1 + 1
=> 2
[3] pry(main)> ["t", "h", "r", "e", "e"].join
=> "three"
[4] pry(main)> _in_.to_a
=> [nil, ":one\n", "1 + 1\n", "[\"t\", \"h\", \"r\", \"e\", \"e\"].join\n"]
[5] pry(main)> _out_.to_a
=> [nil, :one, 2, "three", [nil, ":one\n", "1 + 1\n", "[\"t\", \"h\", \"r\", \"e\", \"e\"].join\n"]]
[6] pry(main)> _out_[2]
=> 2
[7] pry(main)> _in_[2]
=> "1 + 1\n"

source

Write A Query Result To File With Postgres

Generally when writing a query in psql a statement will be terminated with a semicolon. An alternative approach is to end it with a \g instead. This will also send the query to the Postgres server for execution.

select 1 \g

If a filename is included after the \g, then the result of the query will be written to that file instead of output to the psql session.

> select 1, 2, 3 \g query_result.txt

If we cat that file, we can see the query result.

Time: 4.293 ms
> \! cat query_result.txt
 ?column? | ?column? | ?column?
----------+----------+----------
        1 |        2 |        3
(1 row)

See man psql for more details.

What Is On The Runtime Path?

All of the plugins, syntax highlighting, language-specific indentation that extend the default behavior of Vim are on the runtime path. If something isn't on Vim's runtime path, then Vim won't know about and as a result will not load it at runtime.

How do we see what is on the runtime path?

The rtp option is the shorthand for runtimepath. Calling set on either of these will show us the list of runtime paths, or at least some of them.

:set rtp

This will generally be a truncated list if you have a lot of plugins. To be sure you are seeing all of them, use echo instead.

:echo &rtp

See :h rtp for more details.

h/t Chris Erin

Private vs Public Struct members

In Golang, identifiers can be either Exported or Unexported. In other languages this would be known as public or private, but in golang the question is whether the identifiers should be able to be seen outside of the module. If not? Don't export them.

The rule is simple, if an identifier is capitalized it will be exported. This manifests itself in golang when converting a struct to JSON.

type Apple struct {
    color string
    weight int
}

a := json.Marshal(Apple{"green", 10})
fmt.Println(a)

In this case this code will print {}. To print what you would expect the struct should be defined with capitalized identifiers:

type Apple struct {
    Color string
    Weight int
}

Now the marshalled struct should be {"Color": "green", "Weight": 10}

Add files with matches to the args list

If I want to change all instances of the word Apple to Orange across my entire project I can use the vim command:

:argdo %s/Apple/Orange/gc

But only if the files that have the word Apple are in the args list. To populate the args list you can use the args command in combination with shelling out to git grep with the -l flag.

:args `git grep -l Apple`

Then, when you see the list of files with :args, you'll see all the files with the word Apple.

A Mnemonic for Changing Modes

Making a script file executable involves an incantation that has something to do with chmod but what exactly? chmod has two modes, absolute and symbolic. In absolute mode, you set the entire set of permissions with a numeric value like 755.

In symbolic mode, you can add or subtract permissions to either the user (u), group (g) or owner (o). To give read permissions to the group type:

> chmod g+r myfile.txt

And to give execution priviledges to the owner of a file:

> chmod o+x myscript.rb

Netrw Special Command

Building off this previous post, today I learned a new command in Netrw: the 'special command'.

When navigating, enter x over any file, and your operating system will open it with its tool of choice (Preview for images and Finder for directories on a Mac, etc.). This is great for exploring a directory.

From the Netrw help:

Certain files, such as html, gif, jpeg, (word/office) doc, etc, files, are
best seen with a special handler (ie. a tool provided with your computer's
operating system).  Netrw allows one to invoke such special handlers by:

        * when Exploring, hit the "x" key
        * when editing, hit gx with the cursor atop the special filename
          (latter not available if the g:netrw_nogx variable exists)

Send SQL to a Tmux Window

Hashrocket.vim provides a mapping that is awesome:

autocmd FileType sql nmap <buffer> <leader>t :<C-U>w \| call Send_to_Tmux("\\i ".expand("%")."\n")<CR>

With this command, if you're editing an SQL file in a Tmux session, run <leader>t from normal mode, and your file will be read into the session and window of your choice. If there's a psql session listening there, you've got a fast way to iterate on an SQL statement from Vim.

The command even writes the file for you before sending the SQL.

Chaining expectations in Rspec

Generally, we think about expectations in RSpec one at a time. If the first expectation fails, then don't go any further. Expectations in RSpec however are chainable, meaning, I can attach one expectation to another for the same subject and then know about the failures or successes for both expections, that looks like this.

expect(1).to eq(2).and eq(3)

Which produces output like this:

 Failure/Error: expect(1).to eq(2).and eq(3)

          expected: 2
               got: 1

          (compared using ==)

       ...and:

          expected: 3
               got: 1

          (compared using ==)

The same result can be got from the below code which may appeal to you a bit more:

def chain_exp(*expects)
  expects.inject {|exps, exp| exps.and(exp)}
end

expect(1).to chain_exp(eq(2), eq(3))

Inspecting The Process Message Queue

A core tenant of Elixir is message passing between processes. So, if a process is sent a message, where does that message go? What happens if it gets sent many messages? The Process.info/2 function allows us to inspect the message queue.

First, let's send some messages (to ourself) and then keep an eye on the length of the message queue as we go.

> send self(), {:error, "this is bad"}
{:error, "this is bad"}
> Process.info(self(), :message_queue_len)
{:message_queue_len, 1}
> send self(), {:hello, "world"}
{:hello, "world"}
> Process.info(self(), :message_queue_len)
{:message_queue_len, 2}

Now, I am curious what those specific messages are. Let's ask Process.info/2 for the messages that are in the message queue.

> Process.info(self(), :messages)
{:messages, [error: "this is bad", hello: "world"]}

There are a lot of other things that Process.info/2 can tell us about a process. See the Erlang docs for process_info for more details.

Get the association with Ecto

In the schema of my post model I have this line:

belongs_to :developer

So a post is associated with a developer. When I have a post struct and try to access the developer I might see this:

> Tilex.Repo.get(Tilex.Post, 42).developer
#Ecto.Association.NotLoaded<association :developer is not loaded>

Getting the developer is now a two step process:

  1. Construct a query based on the association
  2. Use that query to acquire the struct

To construct a query based on the association we can use the Ecto.assoc/2 function:

query = Ecto.assoc(post, :developer)

Then this query is executed with the Repo.one function:

developer = Tilex.Repo.one(query)

There ya go!

Cherry Pick A Range Of Commits

Git's cherry-pick command allows you to specify a range of commits to be cherry picked onto the current branch. This can be done with the A..B style syntax -- where A is the older end of the range.

Consider a scenario with the following chain of commits: A - B - C - D.

$ git cherry-pick B..D

This will cherry pick commits C and D onto HEAD. This is because the lower-bound is exclusive. If you'd like to include B as well. Try the following:

$ git cherry-pick B^..D

See man git-cherry-pick for more details.

Rails/PG Statement Timeout 🐘⏰

By default, Rails does not set a timeout on database statements. For example, this will run for a full day, even if your ruby process goes away.

ActiveRecord::Base.connection.execute(<<~SQL)
  select pg_sleep(86400);
SQL

The good news is that Rails provides a way to prevent this from happening from the database.yml with the statement_timeout variable.

default: &default
  adapter: postgresql
  ...
  variables:
    statement_timeout: 5000

I'm starting to think that this should always be set low. And then explicitly set higher on a per-query basis, when one is expecting something to take a long time.

Login for feature test with warden test helpers

Warden provides a way to login as a user without having to go through the web interface that a user generally sees for sign in.

user = FactoryGirl.create(:user)
login_as user, scope: :user

In the config block for Rspec you would include this statement:

config.include Warden::Test::Helpers, type: :feature

If you have different models for different types of users in your system you can sign in with different scopes. Lets say you have a student user concept, you can sign in with:

student = FactoryGirl.create(:student)
login_as student, scope: :student

Customizing the output of `ps`

There's a lot to know about processes, although typically we're looking for a mapping of pids to commands.

You can customize the output of ps with -o. The -o flag expects a comma separated list of keywords which indicate the columns that you'd like to display for each process.

> ps -o pid,command

The above command shows only the pid and the command w/args for each process. You can see all the possible keywords with ps -L.

erlang memory usage

You might be be curious how much memory your erlang vm is taking. You can get that and more from :erlang.memory. Below is what I get from running this after starting iex normally.

> :erlang.memory
[total: 18133840, processes: 4975616, processes_used: 4974616, system: 13158224,
 atom: 264529, atom_used: 256719, binary: 228728, code: 6640396, ets: 414904]

This list is confusing. All numbers are bytes, so in total 18133840 bytes are allocated to the erlang process. There are two subcategories processes and system and the values of those categories sum to equal the value of total.

The remaining categories are either a component of system or a component of processes.

Listing Files In IEx

When you start an IEx session, you do so in the context of some directory -- the current working directory. This context can be important if you need to do something like import a file. In fact, you may want to know what files are available in the current working directory.

You can list them all out within IEx using ls/0.

iex(1)> ls()
           .git     .gitignore      README.md         _build         assets         config
           deps            lib        mix.exs       mix.lock           priv           test
            tmp

You can also list the contents of some other specific directory by naming it when invoking ls/1.

See h() within IEx for more details.

When Was This File Added to Version Control?

I learned a helpful Git command today. To determine when a file was added to version control, run the following:

$ git log --follow --diff-filter=A <filename>

This returns the Git history, following files through renames, filtering on when a file was added to version control. Here's the output for the README of 'Today I Learned':

$ git log --follow --diff-filter=A README.md
commit 9a7984a48db19489bb4113378298ddaa97121c7a
Author: Jake Worth <dev+jwworth@hashrocket.com>
Date:   Sat Mar 28 12:52:19 2015 -0400

    Add a basic README

Useful for explaining mystery files in the repository 🔎.

Related File in Rails.vim

Rails.vim is a staple of our Hashrocket workstations. It continues to surprise me.

Today I discovered the :R (related) command, which is a counterpart to the :A (alternate) command. :R looks for the file related to your current buffer.

This command is sophisticated. Assuming you're following Rails conventions for file organization, :R over most templates will open the associated controller, with your cursor on the line where the action that corresponds with the view name (show for a template called show.html.erb) is defined. The same thing works in reverse; :R over the show method leads you back to show.html.erb.

Use Elixir plug for only some controller actions

Phoenix allows for a flexible request processing pipeline with plugs. Your controllers are basically plugs in the plug pipeline. You can specify plugs that will execute before your controller actions with the plug macro.

defmodule MyApp.ThingController do
  use MyApp.Web, :controller

  plug MyApp.InterestingPlug
end

You can also specify a guard for this macro that will only enable this plug to be run before certain actions:

plug MyApp.InterestingPlug when action in [:new, :create]

Am I connected to a terminal via ssh?

When connecting to another computer via ssh the computer will have some environment variables that are not set otherwise. You can examine the environment variables with SSH in the name with:

> printenv | grep SSH

And in both linux and macos you'll see an environment variable:

SSH_CONNECTION=192.168.50.3 65102 192.168.50.12 22

The first ip address and corresponding port represents the ip address if one exists and the second is the machine you are currently ssh'd into.

Update The URL Of A Remote

I just changed the name of a Github repository. One of the implications of this is that the remote URL that my local git repository has on record is now out of date. I need to update it.

If I use git-remote with the -v flag. I can see what remotes I currently have.

$ git remote -v
origin  git@github.com:jbranchaud/pokemon.git (fetch)
origin  git@github.com:jbranchaud/pokemon.git (push)

Now, to update the URL for that remote, I can use git remote set-url specifying the name of the remote and the updated URL.

$ git remote set-url origin git@github.com:jbranchaud/pokemon_deluxe.git

If I check again, I can see it has been updated accordingly.

$ git remote -v
origin  git@github.com:jbranchaud/pokemon_deluxe.git (fetch)
origin  git@github.com:jbranchaud/pokemon_deluxe.git (push)

Defining Multiple Clauses In An Anonymous Function

Anonymous functions often take the approach of doing a single thing with the inputs, regardless of their shape or values. There is no need to limit ourselves though. The same pattern matching that we use all over our Elixir programs can be utilized to define multiple clauses in an anonymous function as well.

Consider the following example:

iex> my_function = fn
  {:ok, x} -> "Everything is ok: #{x}"
  {:error, x} -> "There was an error: #{x}"
end
#Function<6.52032458/1 in :erl_eval.expr/5>

We can then invoke our anonymous function using the bound variable to see what results we get with different kinds of inputs.

iex> my_function.({:ok, 123})
"Everything is ok: 123"
iex> my_function.({:error, "be warned"})
"There was an error: be warned"

source