Today I Learned

A Hashrocket project

438 posts by chriserin @mcnormalmode

`sleep` is just `receive after`

While using sleep in your production application might not be for the best, it’s useful in situations where you’re simulating process behaviour in a test or when you’re trying to diagnose race conditions.

The Elixir docs say this:

Use this function with extreme care. For almost all situations where you would use sleep/1 in Elixir, there is likely a more correct, faster and precise way of achieving the same with message passing.

There’s nothing special about sleep though. The implementation for both :timer.sleep and Process.sleep is equivalent. In Elixir syntax, it’s:

receive after: (timeout -> :ok)

So, it’s waiting for a message, but doesn’t provide any patterns that would successfully be matched. It relies on the after of receive to initiate the timeout and then it returns :ok.

If you’re building out some production code that requires waiting for a certain amount of time, it might be useful to just use the receive after code so that if later you decide that the waiting can be interrupted you can quickly provide a pattern that would match the interrupt like:

timeout = 1000

receive do
  :interrupt -> 
    :ok_move_on_now
  
  after
    timeout -> :ok_we_waited
end

Assert one process gets message from another

Erlang has a very useful-for-testing function :erlang.trace/3, that can serve as a window into all sorts of behaviour.

In this case I want to test that one process sent a message to another. While it’s always best to test outputs rather than implementation, when everything is asynchronous and paralleized you might need some extra techniques to verify your code works right.

pid_a =
  spawn(fn ->
    receive do
      :red -> IO.puts("got red")
      :blue -> IO.puts("got blue")
    end
  end)

:erlang.trace(pid_a, true, [:receive])

spawn(fn ->
  send(pid_a, :blue)
end)

assert_receive({:trace, captured_pid, :receive, captured_message})

assert captured_pid == pid_a
assert :blue == captured_message

In the above example we setup a trace on receive for the first process:

:erlang.trace(pid_a, true, [:receive])

Now, the process that called trace will receive a message whenever traced process receives a message. That message will look like this:

{
  :trace,
  pid_that_received_the_message,
  :receive, # the action being traced
  message_that_was_received
}

This in combination with assert_receive allows you to test that the test process receives the trace message.

Assert Linked Process Raised Error

Linked processes bubble up errors, but not in a way that you can catch with rescue:

test "catch child process error?" do
  spawn_link(fn -> 
    raise "3RA1N1AC"
  end)
rescue 
  e in RuntimeError ->
    IO.puts e
end

This test fails because the error wasn’t caught. The error bubbles up outside of normal execution so you can’t rely on procedural methods of catching the error.

But, because the error causes on exit on the parent process (the test process) you can trap the exit with Process.flag(:trap_exit, true). This flag changes exit behavior. Instead of exiting, the parent process will now receive an :EXIT message.

test "catch child process error?" do
  Process.flag(:trap_exit, true)

  child_pid = spawn_link(fn -> 
    raise "3RA1N1AC"
  end)

  assert_receive {
    :EXIT,
    ^child_pid,
    {%RuntimeError{message: "3RA1N1AC"}, _stack}
  }
end

The error struct is returned in the message tuple so you can pattern match on it and assert about.

This method is still subject to race conditions. The child process must throw the error before the assert_receive times out.

There is a different example in the Elixir docs for catch_exit.

Assert Test Process Did or Will Receive A Message

The ExUnit.Assertions module contains a function assert_receive which the docs state:

Asserts that a message matching pattern was or is going to be received within the timeout period, specified in milliseconds.

It should possibly in addition say “received by the test process”. Let’s see if we can send a message from a different process and assert that the test process receives it:

test_process = self()

spawn(fn ->
  :timer.sleep(99)
  send(test_process, :the_message)
end)

assert_receive(:the_message, 100)
# It Passes!!!

In the above code, :the_message is sent 1 millisecond before the timeout, and the assertion passes.

Now let’s reverse the assertion to refute_receive, and change the sleep to the same time as the timeout.

test_process = self()

spawn(fn ->
  :timer.sleep(100)
  send(test_process, :the_message)
end)

refute_receive(:the_message, 100)
# It Passes!!!

Yep, it passes.

Are All Values True in Postgres

If you have values like this:

chriserin=# select * from (values (true), (false), (true)) x(x);
 x
---
 t
 f
 t

You might want to see if all of them are true. You can do that with bool_and:

chriserin=# select bool_and(x.x) from (values (true), (false), (true)) x(x);
bool_and 
---
 f

And when they are all true:

chriserin=# select bool_and(x.x) from (values (true), (true), (true)) x(x);
bool_and 
---
 t

Hiding and Revealing Struct Info with `inspect`

There are two ways to hide information when printing structs in Elixir.

Hiding by implementing the inspect protol.

defmodule Thing do
    defstruct color: "blue", tentacles: 7
end

defimpl Inspect, for: Thing do
  def inspect(thing, opts) do
    "A #{thing.color} thing!"
  end
end

So now in iex I can’t tell how many tentacles the thing has:

> monster = %Thing{color: "green", tentacles: 17}
> IO.inspect(monster, label: "MONSTER")
MONSTER: A green thing!
A green thing!

Note that iex uses inspect to output data which can get confusing.

NEW IN ELIXIR 1.8: You can also hide data with @derive

defmodule Thing do
  @derive {Inspect, only: [:color]}
  defstruct color: "blue", tentacles: 7
end

And now you won’t see tentacles on inspection

> monster = %Thing{color: "green", tentacles: 17}
> IO.inspect(monster)
#Thing<color: "green", ...>

In both cases, you can reveal the hidden information with the structs: false option:

> monster = %Thing{color: "green", tentacles: 17}
> IO.inspect(monster)
#Thing<color: "green", ...>
> IO.inspect(monster, structs: false)
%{__struct__: Thing, color: "green", tentacles: 17}

Custom Validation in Ecto

Sometimes the standard validation options aren’t enough.

You can create a custom validator in Ecto using the validate_change function.

In this particular case, I want to validate that thing has an odd number of limbs. I can pass the changeset to validate_odd_number.

def changeset(thing, attrs) do
  thing
  |> cast(attrs, [
    :number_of_limbs
  ])
  |> validate_odd_number(
    :number_of_limbs
  )
end

And then define validate_odd_number like this:

def validate_odd_number(changeset, field) when is_atom(field) do
  validate_change(changeset, field, fn (current_field, value) ->
    if rem(value, 2) == 0 do
      [{f, "This field must be an odd number"}]
    else 
      []
    end
  end)
end

We pass a function to validate_change that takes the field atom and the value for that field. In that function we test for oddness and return a keyword list that contains the field name and an error message.

Remember that when f is :number_of_limbs:

    [{f, "hi"}] == [number_of_limbs: "hi"]
    # true, these data structures are equal

Ecto's `distinct` adds an order by clause

When using distinct on you may encounter this error:

select distinct on (color) id, color from fruits order by id;
-- ERROR:  SELECT DISTINCT ON expressions must match initial ORDER BY expressions

This is because distinct on needs to enounter the rows in a specific order so that it can make the determination about which row to take. The expressions must match the initial ORDER BY expressions!

So the above query works when it looks like this:

select distinct on (color) id, color from fruits order by color, id;

OK, now it works.

Ecto’s distinct function helps us avoid this common error by prepending an order by clause ahead of the order by clause you add explicitly.

This elixir statement:

Fruit |> distinct([f], f.color) |> order_by([f], f.id) |> Repo.all

Produces this sql (cleaned up for legibility):

select distinct on (color) id, color from fruits order by color, id;

Ecto added color to the order by!

Without any order by at all, distinct does not prepend the order by.

Read the docs!

Transactions can timeout in Elixir

In Ecto, transactions can timeout. So this type of code:

  Repo.transaction fn -> 
    # Many thousands of expensive queries
    # And inserts
  end

This type of code might fail after 15 seconds, which is the default timeout.

In Ecto you can specify what the timeout should be for each operation. All functions that make a request to the database have the same same shared options of which :timeout is one.

Repo.all(massive_query, timeout: 20_000)

The above query now times out after 20 seconds.

These shared options apply to a transaction as well. If you don’t care that a transaction is taking a long time you can set the timeout to :infinity.

  Repo.transaction(fn -> 
    # Many thousands of expensive queries
    # And inserts
  end, timeout: :infinity)

Now this operation will be allowed to finish, despite the time it takes.

Timing A Function In Elixir

Erlang provides the :timer module for all things timing. The oddly named function tc will let you know how long a function takes:

{uSecs, :ok} = :timer.tc(IO, :puts, ["Hello World"])

Note that uSecs is microseconds not milliseconds so divide by 1_000_000 to get seconds.

microseconds are helpful though because sometimes functions are just that quick.

:timer.tc(IO, :puts, ["Hello World"])
# {22, :ok}

You can also call :timer.tc with a function and args:

adding = fn (x, y) ->  x + y end

:timer.tc(adding, [1,3])
# {5, 4}

Or just a function:

:timer.tc(fn -> 
    # something really expensive
    :ok
    end)
# {1_302_342, :ok}

Ack ignores node_modules by default

When searching through your JavaScript project it doesn’t make sense to search through your node_modules. But if your are on a spelunking journey into the depths of your dependencies, you may want to search through all your node_modules!

ack ignores node_modules by default, and ack being ack you can ack through ack to check it out:

> cat `which ack` | ack node_modules
--ignore-directory=is:node_modules

This is different behaviour from ag and rg which also ignore node_modules but not explicitly. They both ignore node_modules by ignoring all entries in the .gitignore file.

rg claims to implement full support for the .gitignore file while also claiming other search tools do not. The open issues list for ag bears that out.

With each of these tools, explicitly stating the directory to search through overrides the ignore.

> ack autoprefix node_modules
> rg autoprefix node_modules
> ag autoprefix node_modules

`user-select:none` needs prefixes for each browser

The user-select css property governs if text is selectable for a given element. user-select: none means that it is not selectable.

What’s interesting about this property is that while each browser supports it, they each require their own prefix, except Chrome, which does not need a prefix.

In create react app, what starts out as:

user-select: none;

Gets expanded to:

-webkit-user-select: none;
-moz-user-select: none;
-ms-user-select: none;
user-select: none;

But the default browserList configuration for development in your package.json file is:

"development": [
    "last 1 chrome version",
    "last 1 firefox version",
    "last 1 safari version"
]

And so in development, it gets expanded to:

-webkit-user-select: none;
-moz-user-select: none;
user-select: none;

Sorry Micrsoft.

How does create react app know which prefixes to render? The caniuse-lite npm package has up-to-date support data for each property and user-select:none is defined here.

That data is compressed. Here’s how to uncompress it using the node cli to see what it represents:

const compressedData = require('caniuse-lite/data/features/user-select-none');
const caniuse = require('caniuse-lite');
const unpackFunction = caniuse.feature;
unpackFunction(compressedData);

This is accomplished in create react app by the npm package autoprefixer;

Custom React Hook Must Use `use`

You can build your own hooks by composing existing hooks.

Here, I create a custom hook useBoolean by wrapping useState:

const useBoolean = () => useState(true);

Which I can then use in my component:

function Value() {
  const [value, setValue] = useBoolean();

  return <div onClick={() => setValue(!value)}>Click me {String(value)}</div>;
}

The react documentation very politely asks that you start the name of your hook with use. This is isn’t strictly necessary, and it will still work if you call it:

const doBoolean = () => useState(true);

But that violates the Rules of Hooks.

You can include an eslint plugin that will prevent you from breaking the rules. This plugin is installed by default in create-react-app version 3.

Aggregate Arrays In Postgres

array_agg is a great aggregate function in Postgres but it gets weird when aggregating other arrays.

First let’s look at what array_agg does on rows with integer columns:

select array_agg(x) from (values (1), (2), (3), (4)) x (x);
-- {1,2,3,4}

It puts each value into an array. What if are values are arrays?

select array_agg(x)
from (values (Array[1, 2]), (Array[3, 4])) x (x);
-- {{1,2},{3,4}}

Put this doesn’t work when the arrays have different numbers of elements:

select array_agg(x)
from (values (Array[1, 2]), (Array[3, 4, 5])) x (x);
-- ERROR:  cannot accumulate arrays of different dimensionality

If you are trying to accumulate elements to process in your code, you can use jsonb_agg.

select jsonb_agg(x.x)
from (values (Array[1, 2]), (Array[3, 4, 5])) x (x);
-- [[1, 2], [3, 4, 5]]

The advantage of using Postgres arrays however is being able to unnest those arrays downstream:

select unnest(array_agg(x))
from (values (Array[1, 2]), (Array[3, 4])) x (x);
--      1
--      2
--      3
--      4

Count true values with postgres

I can try to count everything that is not null in postgres:

select count(x.x)
from (
  values (null), ('hi'), (null), ('there')
) x (x);
# 2

But if I try to count everything that is true in postgres, I don’t get what I want:

select count(x.x)
from (
  values (false), (true), (false), (true)
) x (x);
# 4

I can, however, take advantage of the postgres ability to cast boolean to int:

select true::int;
# 1
select false::int;
# 0

Using ::int I get:

select count(x.x::int)
from (
  values (false), (true), (false), (true)
) x (x);
# 4

Postgres is still counting everything that is not null, but what if use sum instead?

select sum(x.x::int)
from (
  values (false), (true), (false), (true)
) x (x);
# 2

Because everything is either a 0 or a 1, sum behaves like count.

Now you can do something like this:

select
  sum((status = 'Awesome')::int) as awesomes,  
  sum((status = 'Terrible')::int) as terribles
from statuses;

Print the current stacktrace in Elixir

Stacktrace, backtrace, callstack, in Elixir its stacktrace and it’s available via Process.info/2 using the :current_stacktrace item:

Process.info(self(), :current_stacktrace)

And to print it:

IO.inspect(Process.info(self(), :current_stacktrace), label: "STACKTRACE")

I’m also learning that Process.info/2 takes a pid and an item as arguments. When you call Process.info/1 with just the pid you only get a subset of the info available, not everything.

The items available via Process.info/1 are listed in the erlang documentation here.

The additional items available via Process.info/2 are listed in the erlang documentation here.

You may note that backtrace is also an item that is available via Process.info but it contains more information than you are might need to figure out where you are in the code.

Give a commit a new parent

Given I have these logical commits on two branches:

normalbranch: A - B - C - D

funkybranch: Z - Y - X
git co normalbranch
git rebase --onto X C

Then the logical commits of normalbranch will now be:

normalbranch: Z - Y - X - C - D

We’ve given commit C the new parent of X. C’s hash will change and D’s hash will change.

I use this when I build on top of a branch that I’ve already submitted for a PR. That branch will get merged into the new root branch, and then I’ll need the new parent on the root branch for the commits that I’m currently working on.

`mix phx.digest` creates the manifest

mix phx.digest creates production ready assets: hashed, zipped and compressed.

In production, when you call:

Endpoint.static_path('asset_name.css')

This will look for a file in the priv/static directory and return the path for that file. But what you want is the hashed version of that file, because modern browsers are greedy cachers and will you use the use to bust that cache.

The static_path function can return a path that represents the cached version, but it needs to read the manifest, which maps the file to it’s hashed, zipped and compressed versions.

mix phx.digest creates the manifest along with the hashed, zipped and compressed versions of all the files in the priv/static directory.

Update Map Syntax

There is a special syntax for updating a map in elixir.

thing = %{a: 1, b: 2, c: 3}
updated_thing = %{thing | b: 4}
# %{a: 1, b: 4, c: 3}

But be careful, it throws an error if you try to update a key that doesn’t exist!!

thing = %{a: 1, b: 2, c: 3}
# ** (KeyError) key :x not found in: %{a: 1, b: 2, c: 3}

Skip Pending Tests in ExUnit

In ExUnit you have the ability to tag a test with any atom:

@tag :awesome
test "my awesome test" do
end

@tag :terrible
test "my terrible test" do
end

And then you can exclude those tags at the command line so that your terrible test does not run:

mix test --exclude terrible

So you can make your own tag :pending and exclude those. If you don’t want to have to set the flag at the command line everytime, you can call ExUnit.configure. This is typically placed in the test/test_helper.exs file before ExUnit.start().

ExUnit.configure(exclude: :pending)
ExUnit.start()

Now your pending tests will not run.

That is how you can use custom tags to skip tests, but you can also just use the built in skip tag to skip tests.

@tag :skip
test "my terrible test" do
end

Timex `between?` is exclusive but can be inclusive

Timex.between? is a great function for determining if a time is in a specific time period, but it’s exclusive, so if you are testing for a time at the boundary the result will be negative.

iex> end_time = Timex.now()
iex> start_time = Timex.shift(end_time, days: -7)
iex> time = start_time
iex> Timex.between?(time, start_time, end_time)
false

Never fear, you can pass an inclusive option:

iex> Timex.between?(time, start_time, end_time, inclusive: true)
true

But you don’t have to be inclusive on both sides! Here, I pass :end so that I am only inclusive at the end of my time period.

iex> Timex.between?(time, start_time, end_time, inclusive: :end)
false

And of course I can be only inclusive at the beginning of the period if I prefer:

iex> Timex.between?(time, start_time, end_time, inclusive: :start)
true

Match across lines with Elixir Regex `dotall` (s)

Elixir Regex is PCRE compliant and Ruby Regex isn’t in at least one specific way. The m (multiline) flag behaves differently in Elixir and Ruby.

Without any modifiers, Regex does not match across new lines:

iex> Regex.scan( ~r/.*/, "a\nb")
[["a"], [""], ["b"], [""]]
iex> Regex.scan( ~r/.*/, "ab")
[["ab"], [""]]

With the m (multiline) modifier, Regex will match the beginning of each line when using the ^ char.

iex> Regex.scan( ~r/.*/m, "a\nb")
[["a"], [""], ["b"], [""]]
iex> Regex.scan( ~r/^.*/, "a\nb")
[["a"]]
iex> Regex.scan( ~r/^.*/m, "a\nb")
[["a"], ["b"]]

The s (dotall) modifier is the right way in Elixir to match across newlines.

iex> Regex.scan( ~r/.*/s, "a\nb")
[["a\nb"], [""]]
iex> Regex.scan( ~r/^.*/s, "a\nb")
[["a\nb"]]

In ruby, you can match across lines with the m (modifier) and the s is ignored.

irb> "a\nb".scan(/.*/)
=> ["a", "", "b", ""]
irb> "a\nb".scan(/.*/m)
=> ["a\nb", ""]
irb> "a\nb".scan(/.*/s)
=> ["a", "", "b", ""]

Read about the history of dotallhere

Get _just one_ value from Ecto query

With the data structure you use in select’s expression argument you can specify what type of data structure the query will return for a row. I’ve used [] and %{} and %Something{} but you can also specify that each row is just one value without using a data structure at all.

Combine that with Repo.one to just return one row and you can just get the specific value you are looking for without any destructuring.

age = 
  User
  |> where(id: 42)
  |> select([u], u.age)
  |> Repo.one()

Don't truncate when redirecting

A common problem in shell languages is truncating a file that you’re trying to transform by redirecting into it.

Let’s say I have a file full of “a”‘s

PROMPT> cat test.txt
aaaaaa

And I want to switch them all to “b”‘s

PROMPT> cat test.txt | tr 'a' 'b'
bbbbbb

But when I try to redirect into the same file I truncate file:

PROMPT> cat test.txt | tr 'a' 'b' > test.txt
PROMPT> cat test.txt
# nothing

This is because the shell truncates “test.txt” for redirection before cat reads the file.

An alternate approach is to use tee.

PROMPT> cat test.txt | tr 'a' 'b' | tee test.txt
bbbbbb
PROMPT> cat test.txt
bbbbbb

tee will write the output to both the file and to standard out, but will do so at the end of the pipe, after the cat command reads from the file.

Get pids for each beam application in Elixir

In Elixir, it’s hard to determine what pids belong to what applications just by using Process.info

iex> pid(0,45,0) |> Process.info
[
  current_function: {:application_master, :main_loop, 2},
  initial_call: {:proc_lib, :init_p, 5},
  status: :waiting,
  message_queue_len: 0,
  links: [#PID<0.46.0>, #PID<0.43.0>],
  dictionary: [
    "$ancestors": [#PID<0.44.0>],
    "$initial_call": {:application_master, :init, 4}
  ],
  trap_exit: true,
  error_handler: :error_handler,
  priority: :normal,
  group_leader: #PID<0.45.0>,
  total_heap_size: 376,
  heap_size: 376,
  stack_size: 7,
  reductions: 49,
  garbage_collection: [
    max_heap_size: %{error_logger: true, kill: true, size: 0},
    min_bin_vheap_size: 46422,
    min_heap_size: 233,
    fullsweep_after: 65535,
    minor_gcs: 0
  ],
  suspending: []
]

With the above output, I can’t tell what application this is!

I can use the :running key of Erlang’s :application.info to get a map of applications to pids.

iex> :application.info[:running]
[
  logger: #PID<0.93.0>,
  iex: #PID<0.85.0>,
  elixir: #PID<0.79.0>,
  compiler: :undefined,
  stdlib: :undefined,
  kernel: #PID<0.45.0>
]

Oh ok, pid(0,45,0) is the Kernel application.

Test that an email was sent with correct params

assert_email_delivered_with is a function that checks if an email was delivered using Bamboo.

Sending an email is a side-effect in functional programming parlance. When testing a function where the email is sent several calls deep, you generally don’t get the email as a return value of that function.

When using Bamboo’s TestAdaptor in test mode, Bamboo captures the delivered email so that you can assert about it.

Lottery.notify_winner()
assert_email_delivered_with(subject: "You Won!!")

You can also assert on: to, cc, text_body, html_body, from, and bcc.

Happy Testing!

Formatting the email address fields with Bamboo

When you create an email with Bamboo and deliver it:

 email = new_email(
      to: "chris@example.com",
      from: "vitamin.bot@example.com",
      subject: "Reminder",
      text_body: "Take your vitamins"
    )
 |> Mailer.deliver_now()

The to field will not be what you expect it to be:

email.to
# {nil, "chris@example.com"}

This is the normalized format for a to field. The first element in the tuple is the name of the recipient.

Bamboo allows you to format this field yourself with protocols.

defimpl Bamboo.Formatter, for: Person do
  def format_email_address(person, _opts) do
    {person.nickname, person.email}
  end
end

And now if I send an email with Person in the to field:

person = %Person{
    nickname: "shorty", 
  email: "chris@example.com"
}

email = new_email(
      to: person,
      from: "vitamin.bot@example.com",
      subject: "Reminder",
      text_body: "Take your vitamins"
    )
|> Mailer.deliver_now()

Then the to field gets formatted with our protocol.

email.to
# {"shorty", "chris@example.com"}

Read more about the Bamboo Formatter here.

Fuzzy translation key merging with Gettext

Gettext is a great tool for i18n in elixir. It provides a mix task for extracting translation keys from your code. The translation keys (or message ids) are natural language and look like this:

gettext("Hi, Welcome to Tilex!")

After running mix gettext.extract && mix gettext.merge, an already translated Italian locale file would look like:

msgid "Hi, Welcome to Tilex!"
msgstr "Italian version of Welcome!"

There’s a chance that the natural language key (which also serves as the default string) will change.

If it changes just a little bit then the Italian locale file will look like:

#, fuzzy
msgid "Hi, Welcome to Tilex!!"
msgstr "Italian version of Welcome!"

It gets marked as #, fuzzy, and the new msgid replaced the old msgid.

Gettext determines how big of a change will constitute a fuzzy match with String.jaro_distance.

iex> String.jaro_distance("something", "nothing")
0.8412698412698413
iex> String.jaro_distance("peanuts", "bandersnatch")
0.576984126984127

The higher the number the closer the match. fuzzy_threshold is the configuration that determines whether a msgid is fuzzy or not and the default for fuzzy_threshold is 0.8, set here.

Two arguments in a command with xargs and bash -c

You can use substitution -I{} to put the argument into the middle of the command.

> echo "a\nb\nc\nd" | xargs -I{} echo {}!
a!
b!
c!
d!

I can use -L2 to provide exactly 2 arguments to the command:

> echo "a\nb\nc\nd" | xargs -I{} -L2 echo {}!
a b!
c d!

But I want to use two arguments, the first in one place, the next in another place:

> echo "a\nb\nc\nd" | xargs -I{} -L2 echo {}x{}!
a bxa b!
c dxa b!

I wanted axb! but got a bxa b!. In order to achieve this you have to pass arguments to a bash command.

> echo "a\nb\nc\nd" | xargs -L2 bash -c 'echo $0x$1!'
axb!
cxd!

Just like calling

bash -c 'echo $0x$1!' a b

Where $0 represents the first argument and $1 represents the second argument.

Weighted Shuffle in Elixir

Shuffling a list is easy, but what if you want to perform a weighted shuffle? That is, what if some elements in a list had a greater chance of being selected, or higher in the list, than others?

Given these colors and these weights:

weighted_colors = [{1, "orange"}, {3, "green"}, {5, "red"}]

I can create a list that is mostly represents each item the number of times of it’s weight by utilizing List.duplicate:

weighted_colors
|> Enum.map(fn ({weight, color}) -> 
  List.duplicate(color, weight)
end)
|> Enum.flatten
# ["orange", "green", "green", "green", "red", "red", "red", "red", "red"]

Then shuffle it:

|> Enum.shuffle
# ["red", "red", "orange", "red", "green", "green", "red", "green", "red"]

And then use Enum.uniq attribute of preserving the order in which it encounters each unique element from left to right.

|> Enum.uniq
# ["red", "orange", "green"]

Run this 100 times over the same weighted list and the most outcome that you would experience most often is:

# ["red", "green", "orange"]

Create a temp table from values

In postgres, if you are looking for a way to create a quick data set to experiment with you can create a temporary table using the values expression.

create temp table test as values ('a', 1), ('b', 2), ('c', 3);

In postgres, a temp table is a table that will go away at the end of the session in which it was created.

The types of the columns are inferred as you can see when examining the table in psql:

> \d test
               Table "pg_temp_3.test"
 Column  |  Type   | Collation | Nullable | Default
---------+---------+-----------+----------+---------
 column1 | text    |           |          |
 column2 | integer |           |          |

What happens if we try to mix types?

> create temp table test as values ('a', 1), (1, 'c');
ERROR:  invalid input syntax for integer: "a"

You can also be specific about types by casting the values of the first row.

> create temp table test as values ('a'::varchar(1), 1::decimal), ('b', 1);
> \d test
                    Table "pg_temp_3.test"
 Column  |       Type        | Collation | Nullable | Default
---------+-------------------+-----------+----------+---------
 column1 | character varying |           |          |
 column2 | numeric           |           |          |

Choose only one row for a given value w Distinct

If you have a table with many rows but you only want one row for a given value, you can use distinct on (<column>)

select distinct on (letter) * 
from (
values 
('y', 512), 
('y',128), 
('z', 512), 
('x',128), 
('z', 256)) 
as x (letter, number);

Which produces

letter | number
--------+--------
 x      |    128
 y      |    512
 z      |    512
(3 rows)

Distinct implicitly orders by the specified column ascending. Putting order by letter at the end of the select statement produces the exact same output (and execution plan).

Distinct chooses the first row for the given column after sorting, so changing the sort order for the second column will change the results.

Here are the results after adding a order by letter, number asc to the above select statement.

 letter | number
--------+--------
 x      |    128
 y      |    128
 z      |    256
(3 rows)

Values clause in a select statement

You’ve all seen the VALUES clause before. It is typically used to insert data.

insert into colors (name, brightness) 
values ('red', 10), ('black', 0), ('blue', 5);

You can also use the VALUES clause in a select statement:

select * from (values ('red', 10), ('black', 0), ('blue', 5)
);

This is great when experimenting with different parts of sql at the command line.

Additionally, values is a first class expression on its own:

> values (1,2), (2,3);

 column1 | column2
---------+---------
       1 |       2
       2 |       3
(2 rows)

Generating a postgres query with no rows

For testing purposes, it is nice to be able to simulate a query that does not result in any rows.

select * from generate_series(0, -1) x;
 x
---
(0 rows)

This is because (from the docs):

When step is positive, zero rows are returned if start is greater than stop.

Step (the third argument to generate series) defaults to 1, so anything where the from has to go backwards to get to the to will result in 0 rows.

select * from generate_series(4, 3) x;
 x
---
(0 rows)

And now you can see how sum() behaves without any rows:

select sum(x) from generate_series(1, 0) x;
sum
-----
   ø
(1 row)

Ohhh, it returns null. I hope you’re planning for that.

Parsing CSV at the command line with `csvcut`

Parsing csv at the command line is easy with the csvcut tool from csvkit.

csvkit is installable with pip

> pip install csvcut 

You can print only the columns you are interested in.

> echo "a,b,c,d" | csvcut -c 1,4
a,d
> echo "a,b,c,d" | csvcut -c 1,5
Column 5 is invalid. The last column is 'd' at index 4.

It also handles quoted csv columns:

echo 'a,"1,2,3,4",c,d' | csvcut -c 2,3

It handles new lines:

> echo 'a,b\nc,d' | csvcut -c 2
b
d

There are a virtual plethora of options check out csvcut --help.

csvkit has a number of csv processing tools, check them out here.

Get pip installed executables into the asdf path

Given I am using asdf when I install a python executable via pip, then I expect to be able to run that executable at the command line.

> asdf global python 3.7.2
> pip install some-executable-package
> some-executable
zsh: command not found: some-executable

This is because a shim has not yet been created for this executable. If you look into your shims dir for the executable:

> ls $(dirname $(which pip)) | grep some-executable

It doesn’t exist.

To place a shim into the shims dir of asdf you must reshim.

> asdf reshim python

And now you have a shim that points to the executable:

> ls $(dirname $(which pip)) | grep some-executable
some-executable
> some-executable
Hello World!

Find duplicate routes in Elixir Phoenix

If you have duplicated routes in your route file like this:

scope "/api", MyAppWeb.Api, as: :api do
  pipe_through [:this]
  
  resources "/users", UserController, except: [:new, :edit]
  scope 
end

scope "/api", MyAppWeb.Api, as: :api do
  pipe_through [:that]

  resources "/users", UserController, except: [:new, :edit]
end

Then you’ll get a warning like this:

warning: this clause cannot match because a previous clause at line 2 always matches
  lib/idea_web/router.ex:2

The warning doesn’t really let you know which routes are duplicated, but it’s really ease to find the duplicated routes by utilizing the uniq command.

mix phx.routes | sort | uniq -d

The -d flag for uniq is duplicates only. -d only considers similar lines in consecutive order as duplicates, so you need to run it through sort first.

The output looks like this:

api_user_path  DELETE  /api/users/:id                   MyAppWeb.Api.UserController :delete
api_user_path  GET     /api/users                       MyAppWeb.Api.UserController :index
api_user_path  GET     /api/users/:id                   MyAppWeb.Api.UserController :show
api_user_path  PATCH   /api/users/:id                   MyAppWeb.Api.UserController :update
api_user_path  POST    /api/users                       MyAppWeb.Api.UserController :create

And those are your duplicate routes!

Named bindings in Ecto (vs positional bindings)

Positional bindings in Ecto are meant to confuse when buildings large queries across several different functions.

query = Thing
|> join(:inner, [t], x in X, on: t.x_id = x.id)
|> join(:inner, [_, x], y in Y, on: x.y_id = y.id)

Using that query in another function might look like this:

query
|> where([_, x, _], x.type == "Articulated")

But what if the positions change? How am I supposed to know which positions are which when I’m adding to this query out of context?

Named bindings help tremendously here (note the :as option):

query = Thing
|> join(:inner, [t], x in X, as: :x, on: t.x_id = x.id)
|> join(:inner, [_, x], y in Y, as: :y, on: x.y_id = y.id)

Now I can refer to these things without knowing the position. If the position changes it’s all good the additive where statement does not have to change:

query
|> where([y: y, x: x], x.type == "Articulated", y.feel == "Good")

How to actually _load_ the resource with Guardian

Guardian, like all auth libraries in all languages, is tough to wrap my head around.

I know there is a plug in the pipeline called plug Guardian.Plug.LoadResource. I know there is a function called Guardian.Plug.current_resource(conn) that takes the conn and returns that returns the resource placed in the conn by the Guardian.Plug.LoadResource plug.

What I don’t know is how the LoadResource plug knows what resource to get.

In Guardian, you configure the pipeline with:

use Guardian.Plug.Pipeline, otp_app: :my_app,
                              module: GuardianImpl,
                              error_handler: ErrorHandler

The GuardianImpl is a module that uses the Guardian behaviour.

The Guardian behaviour has a callback resource_from_claims that might be implemented like this:

def resource_from_claims(claims) do
  {:ok, Repo.get(User, claims["sub"])}
end

So when you need to modify how you load the resource, you should look to see how the resource_from_claims callback is implemented.

Read more here.

Confirming operations with `xargs -p`

xargs is a great tool to take a lot of input and execute a lot of different commands based on that input. Sometimes though, if you are performing destructive or mutative actions with xargs you want to proceed more cautiosly.

> echo "banana apple orange" | tr ' ' '\n' | xargs -n1 echo "I like"

This outputs:

I like banana
I like apple
I like orange

But maybe I don’t like some of those things, please ask! Including the p flag with xargs forces a prompt.

> echo "banana apple orange" | tr ' ' '\n' | xargs -p -n1 echo "I like"
echo I like banana ?...n
echo I like apple ?...y
I like apple
echo I like orange ?...n

Yep, I only like apples.

Updating the ExUnit test context with setup

When using ExUnit the second argument to the test macro is context.

test "1 + 1 = value", context do
  assert 1 + 1 == 2
end

This context can provide setup values so that you can share setup across tests.

Use the setup macro to update the context. The keyword list you return from the setup function will be merged into the map of the context.

setup do
  [result: 2]  # this gets merged into the context
end

setup do
  # create some database records
  :ok # this does not get merged into the context
end

Also in the setup macro you can have access to the context, allowing you to potentially change the context based on the needs of the test.

setup context do
  %{result: r} = context
    [result: r + 1]
end

test "1 + 1 = value", %{result: value} do
  assert 1 + 1 == value
end

Read more about ExUnit setup here.

Get first image of animated gif

Image Magick’s convert tool has a no-option, very simple way to access the first frame of an animated gif.

convert 'animated.gif[0]' animated.first.gif

The square brackets after the file name above can contain any index for any frame of the image. 0 is the index of the first image.

To discover how many frames an animated gif has you can use:

identify animated.gif

Which will return a line for every frame in the animated gif. Those lines will look like this:

animated.gif[32] GIF 736x521 756x594+4+70 8-bit sRGB 256c 421707B 0.000u 0:00.000

Go to next ALE error

Has ALE overtaken your vim setup like it has mine? It constantly runs linters, compilers and formatters, just waiting for you to slip up so that it can put an X in the gutter.

Those X’s are really quite handy. They generally point me to the next place in the code that I need to make a change.

To get there quickly you can goto the next ALE error with:

:ALENext

This will stop at the last error in the file though. To have it wrap around use:

:ALENextWrap

I really enjoy vim-unimpaired’s handy bracket mappings, but I don’t use ]a that move between args (because I don’t use args very often).

To setup my own handy bracket mappings for ALE:

:nmap ]a :ALENextWrap<CR>
:nmap [a :ALEPreviousWrap<CR>
:nmap ]A :ALELast
:nmap [A :ALEFirst

Prevent npm high level security errors in CI

In npm 6.6, a feature was added to provide security audit information for the packages that are used in your application.

This is run with:

node audit

This exits with a non-zero exit code if any ‘low’, ‘medium’, ‘high’, or ‘critical’ errors were detected.

You can use that non-zero return code in your CI to fail a check, which should notify you of the security vulnerability which you can then resolve.

If you care about ‘high’ or ‘critical’ errors but don’t care about ‘low’ or ‘medium’ you can set the audit-level npm config value to ‘high’ in you npm configuration for your CI server.

Set the relative path of assets in a CRA app

When I build my CRA app I get a path for my assets (css, images) that begins with /static. If I deploy my app to https://something.com/myapp, then the app will try to access those asset paths at https://something.com/static/asset.css. That’s not where the asset lives. The asset lives at https://something.com/myapp/static/asset.css.

Create React App allows you to change the prefix for a the built assets with the homepage attribute in your package.json file.

You could set it to myapp:

"homepage": "/myapp"

And then the asset will have the path of /myapp/static/asset.css, but what if you want to change paths?

"homepage": "."

Setting homepage to . will make the asset always relative to index.html, allowing you to not be concerned with the path the application is deployed to.

This actually repurposes a property of the package json file that npm uses to set the homepage of an npm package, so you may find this property used in a different way in other package.json files.

See the npm docs here.

See the Create React App docs here

Override Create React App conf w/react-app-rewired

A common problem when using Create React App is changing the configured behaviour of webpack. Generally, if you want to change the webpack configuration provided by Create React App you need to eject, but eject at the very minimum adds a lot of files to your project that you may not want.

An alternative is to use react-app-rewired in combination with customize-cra.

react-app-rewired provides a file, config-overrides.js placed in your project root directory where you can override existing behaviour.

customize-cra provides a set of handy utility functions to help you override specific configurations.

For example, when using the ant design library, you can import both the component and the css for that component with one import line if you use babel-loader.

Here is an example of a config-overrides.js file that would provide that behaviour.

const { override, fixBabelImports } = require('customize-cra');

module.exports = override(
  fixBabelImports('import', {
    libraryName: 'antd',
    libraryDirectory: 'es',
    style: 'css',
  })
);

Run side effect when a prop changes w/Hooks

There is a React Hook for side effects useEffect. You can pass useEffect a function and that function will run after each render.

useEffect(() => console.log('rendered!'));

In many cases it’s inefficient and unnecessary to call the effect function after every render. useEffect has a second argument of an array of values. If passing in this second argument, the effect function will only run when the values change.

useEffect(() => console.log('value changed!'), [props.isOpen]);

Now, you will see “value changed!” both on the first render and everytime isOpen changes.

Reminder: React Hooks are for functional components not class components. Check out the hooks api here

Get a ref to a dom element with react hooks

React Hooks are now available in React 16.8. There are 10 different hooks, you can read about them here. When I needed a ref to a dom element yesterday I reached for useRef.

const containerRef = useRef(null);

This isn’t a ref to anything unless you pass the ref to a tag.

return (<div ref={containerRef}></div>);

Now the ref will be assigned a dom element that you can use. In this example I’m using the useEffect hook to execute a side effect after the render takes place. Use the current attribute to access the current dom node.

useEffect(() => {
    containerRef.current.style = 'background-color: green;'
})

Destructuring Record in Fn Argument

Elm has destructuring/pattern matching that feels typical for an ML. One pattern matching feature I like is record destructuring in a function argument.

myRecord = {a = 1}

myFunc {a} = a

myFunc myRecord
# 1

Here, we destructure the key a out of the record. What’s cool about this is the record does not need to match exactly, but can be any record with the property of a.

myRecord = {a = 1, b = 2}

myFunc {a} = a

myFunc myRecord
# 1

This enables us to be able to grow the record over time without changing the signature of the function. In Elm, not having to accomodate that change across the entire program is great.

Formatting Elm Code

Elm comes with it’s own formatter.

elm-format src/

It’s got options like --upgrade which will help you get from 0.18 code to 0.19 code, and its got --validate which you can use in continuous integration to ensure all PRs are properly formatted.

In vim, formatting is enabled by default when you use elm-vim.