Today I Learned

A Hashrocket project

210 posts about #elixir

Get a writable temporary directory in Elixir

When you need to write a temporary file to disk it is common to assume a particular path exists based on the operating system the app is running on:

tmp_file = Path.join("/tmp", filename)
File.write!(tmp_file)

However depending on the configuration of the app as it is deployed in production, it may not have access to the /tmp directory. Fortunately Elixir provides a function that returns a writable path for temporary files:

dir = System.tmp_dir!()
tmp_file = Path.join(dir, filename)
File.write!(tmp_file)

From the docs (h System.tmp_dir):

Writable temporary directory.

Returns a writable temporary directory. Searches for directories in the following order:

    1. the directory named by the TMPDIR environment variable
    2. the directory named by the TEMP environment variable
    3. the directory named by the TMP environment variable
    4. C:\TMP on Windows or /tmp on Unix
    5. as a last resort, the current working directory

Returns nil if none of the above are writable.

I prefer to use the “bang” version System.tmp_dir! described as:

Same as tmp_dir/0 but raises RuntimeError instead of returning nil if no temp dir is set.

On my machine it returns something like this:

"/var/folders/66/dj7rwns53vn4db4_1npvqtrh0000gn/T/"

🔒 Serve Phoenix App Locally with HTTPS 🔒

The Phoenix Framework provides an easy mix task to automatically generate a self-signed SSL cert. This is useful if you want to test the app locally with HTTPS.

mix phx.gen.cert

The self-sigend certs will be stored in priv/cert so make sure you add that path to your .gitignore.

When finished the command will prompt you to update your endpoint configuration and a few imporant warnings:

If you have not already done so, please update your HTTPS Endpoint
configuration in config/dev.exs:

  config :tilex, TilexWeb.Endpoint,
    http: [port: 4000],
    https: [
      port: 4001,
      cipher_suite: :strong,
      certfile: "priv/cert/selfsigned.pem",
      keyfile: "priv/cert/selfsigned_key.pem"
    ],
    ...

WARNING: only use the generated certificate for testing in a closed network
environment, such as running a development server on `localhost`.
For production, staging, or testing servers on the public internet, obtain a
proper certificate, for example from [Let's Encrypt](https://letsencrypt.org).

NOTE: when using Google Chrome, open chrome://flags/#allow-insecure-localhost
to enable the use of self-signed certificates on `localhost`.

Precise timings with `monotonic_time`

Monotonic time is time from a clock that only moves forward. The system clock on your CPU can be set and reset. Even when tied to the LAN ntp protocol the system clock can be out-of-sync by a couple of milliseconds. When measuring in microseconds, that’s a lot of time, and time drift can occur at the microsecond level even when attached to NTP, requiring system clock resets.

To get monotonic time in Elixir use, System.monotonic_time:

iex> System.monotonic_time
-576460718338896000
iex> System.monotonic_time
-576460324867892860

It’s ok that this number is negative, it’s always moving positive.

The number has a time unit of :native. To get a duration in millseconds you could convert from :native to millisecond.

iex> event_time = System.monotonic_time
-576459417748861340
iex> System.convert_time_unit(System.monotonic_time - event_time, :native, :millisecond)
38803

Or you could get a millisecond duration by using the one argument of monotonic_time to specify the time unit you want.

iex> event_time = System.monotonic_time(:millisecond)
-576459079381
iex> System.monotonic_time(:millisecond) - event_time
14519

Check out the elixir docs on time for more info.

IO.inspect Label

When doing puts-driven-development in Elixir, IO.inspect/1 and IO.inspect/2 are very useful. These functions return their items unchanged, allowing us to spy on a value in our Elixir code.

I’m a fan of the :label option, which decorates our output with a label, making it easier to understand.

def channel_name_split(channel) do
  channel.name
  |> IO.inspect(label: "channel name before")
  |> String.split("")
  |> IO.inspect(label: "channel name after")
end

And here’s the output in our server log:

channel name before: "workflow"
channel name after: ["", "w", "o", "r", "k", "f", "l", "o", "w", ""]

Check out h IO.inspect in IEX for more info.

telemetry handler detaches automatically on error

Telemetry handlers should be designed to run efficiently over and over again without making a noticeable performance impact on your production system. But what happens if an occurs? Your monitoring should have no bearing on the success of your business logic.

So what happens if your monitoring has an error? With telemetry, it’s ok, the offending handler is just detached, never to run again. Business logic unaffected.

Here’s a test that demonstrates the handler being removed.

handlerFn = fn _, _, _, _ ->
  raise "something"
end

:telemetry.attach(
  :bad_handler,
  [:something_happened],
  handlerFn,
  :no_config
)

# the handler is in ets
assert [[[:something_happened]]] =
         :ets.match(:telemetry_handler_table, {:handler, :bad_handler, :"$1", :_, :_})

:telemetry.execute([:something_happened], %{}, %{})

# the handler is gone!
assert [] = :ets.match(:telemetry_handler_table, {:handler, :bad_handler, :"$1", :_, :_})

You can see the try/catch block in the telemetry source code here.

List all telemetry event handlers

Telemetry is a new library for application metrics and logging in beam applications. It was added to Phoenix in version 1.4.7 released in June 2019.

Telemetry consists of registered handler functions that are executed when specific events occur.

To see a list of what handlers are registered for which events you can call:

:telemetry.list_handlers([])

When returns a list of maps.

To see a list of handlers that have a specific event prefix, you can pass in a prefix as the only argument.

:telemetry.list_handlers([:phoenix, :endpoint])

Which returns:

[
  %{
    config: :ok,
    event_name: [:phoenix, :endpoint, :start],
    function: #Function<2.82557494/4 in Phoenix.Logger.install/0>,
    id: {Phoenix.Logger, [:phoenix, :endpoint, :start]}
  },
  %{
    config: :ok,
    event_name: [:phoenix, :endpoint, :stop],
    function: #Function<3.82557494/4 in Phoenix.Logger.install/0>,
    id: {Phoenix.Logger, [:phoenix, :endpoint, :stop]}
  }
]

`telemetry_event` Overrides Repo Query Event

Ecto gives you a single telemetry event out of the box, [:my_app, :repo, :query], where the [:my_app, :repo] is the telemetry prefix option for ecto.

This event is called whenever any request to the database is made:

  handler = fn _, measurements, _, _ ->
    send(self(), :test_message)
  end

  :telemetry.attach(
    "query",
    [:test_telemetry, :repo, :query],
    handler,
    %{}
  )

  Repo.all(TestTelemetry.Colour)

  assert_receive(:test_message)

This event is overriden when using the the :telemetry_event option, a shared option for all Repo query functions.

  handler = fn _, measurements, _, _ ->
    send(self(), :test_message)
  end

  custom_handler = fn _, measurements, _, _ ->
    send(self(), :custom_message)
  end

  :telemetry.attach(
    "query",
    [:test_telemetry, :repo, :query],
    handler,
    %{}
  )

  :telemetry.attach(
    "custom",
    [:custom],
    custom_handler,
    %{}
  )

  Repo.all(TestTelemetry.Colour, telemetry_event: [:custom])

  assert_receive(:custom_message)
  refute_receive(:test_message)

Which means for any given query you can only broadcast one event. If you have a system that keeps track of expensive queries but you also need to debug a particular query in production, you will take that query out of the system to track expensive queries.

Telemetry Attach and Execute

The telemetry api is simpler than the name tends to imply. There are only two primary functions, attach/4 and execute/4. Check out the telemetry docs to see the full api.

Attach is simple. Essentially, you want a function to be called when a certain event occurs:

handler = fn [:a_certain_event], _measuremnts, _metadata, _config ->
  IO.puts("An event occurred!!")
end

:telemetry.attach(
  :unique_handler_id,
  [:a_certain_event],
  handler,
  :no_config
)

The first argument is handler_id and can be anything be must be unique. The last argument is config and can also be anything. It will be passed along, untouched, to the handler function as the last argument.

Execute is simple. When the program calls execute, the handler that matches the event is called and the measurements and metadata are passed to the handler.

measurements = %{}
metadata = %{}
:telemetry.execute([:a_certain_event], measurements, metadata)

It’s important to note that the event name must be a list of atoms.

A simple test for attach and execute would look like this:

test "attach and execute" do
  handler = fn _, _, _, _ ->
    send(self(), :test_message)
  end

  :telemetry.attach(
    :handler_id,
    [:something_happened],
    handler,
    :no_config
  )

  :telemetry.execute([:something_happened], %{}, %{})

  assert_receive :test_message
end

Accumulating Attributes In Elixir

Typically, if you declare an attribute twice like this:

@unit_of_measure :fathom
@unit_of_measure :stone

The second declaration will override the first:

IO.inspect(@unit_of_measure)
# :stone

But by registering the attribute by calling register_attribute you get the opportunity to set the attribute to accumulate. When accumulating, each declaration will push the declared value onto the head of a list.

defmodule TriColarian do
  @moduledoc false

  Module.register_attribute(__MODULE__, :colors, accumulate: true)

  @colors :green
  @colors :red
  @colors :yellow

  def colors do
    @colors
  end
end

TriColarian.colors()
# [:yellow, :red, :green]

At compile time, perhaps when executing a macro, you have the opportunity to dynamically build a list.

I learned this when Andrew Summers gave a talk on DSLs at Chicago Elixir this past Wednesday. You can see his slides here

`sleep` is just `receive after`

While using sleep in your production application might not be for the best, it’s useful in situations where you’re simulating process behaviour in a test or when you’re trying to diagnose race conditions.

The Elixir docs say this:

Use this function with extreme care. For almost all situations where you would use sleep/1 in Elixir, there is likely a more correct, faster and precise way of achieving the same with message passing.

There’s nothing special about sleep though. The implementation for both :timer.sleep and Process.sleep is equivalent. In Elixir syntax, it’s:

receive after: (timeout -> :ok)

So, it’s waiting for a message, but doesn’t provide any patterns that would successfully be matched. It relies on the after of receive to initiate the timeout and then it returns :ok.

If you’re building out some production code that requires waiting for a certain amount of time, it might be useful to just use the receive after code so that if later you decide that the waiting can be interrupted you can quickly provide a pattern that would match the interrupt like:

timeout = 1000

receive do
  :interrupt -> 
    :ok_move_on_now
  
  after
    timeout -> :ok_we_waited
end

Assert one process gets message from another

Erlang has a very useful-for-testing function :erlang.trace/3, that can serve as a window into all sorts of behaviour.

In this case I want to test that one process sent a message to another. While it’s always best to test outputs rather than implementation, when everything is asynchronous and paralleized you might need some extra techniques to verify your code works right.

pid_a =
  spawn(fn ->
    receive do
      :red -> IO.puts("got red")
      :blue -> IO.puts("got blue")
    end
  end)

:erlang.trace(pid_a, true, [:receive])

spawn(fn ->
  send(pid_a, :blue)
end)

assert_receive({:trace, captured_pid, :receive, captured_message})

assert captured_pid == pid_a
assert :blue == captured_message

In the above example we setup a trace on receive for the first process:

:erlang.trace(pid_a, true, [:receive])

Now, the process that called trace will receive a message whenever traced process receives a message. That message will look like this:

{
  :trace,
  pid_that_received_the_message,
  :receive, # the action being traced
  message_that_was_received
}

This in combination with assert_receive allows you to test that the test process receives the trace message.

Assert Linked Process Raised Error

Linked processes bubble up errors, but not in a way that you can catch with rescue:

test "catch child process error?" do
  spawn_link(fn -> 
    raise "3RA1N1AC"
  end)
rescue 
  e in RuntimeError ->
    IO.puts e
end

This test fails because the error wasn’t caught. The error bubbles up outside of normal execution so you can’t rely on procedural methods of catching the error.

But, because the error causes on exit on the parent process (the test process) you can trap the exit with Process.flag(:trap_exit, true). This flag changes exit behavior. Instead of exiting, the parent process will now receive an :EXIT message.

test "catch child process error?" do
  Process.flag(:trap_exit, true)

  child_pid = spawn_link(fn -> 
    raise "3RA1N1AC"
  end)

  assert_receive {
    :EXIT,
    ^child_pid,
    {%RuntimeError{message: "3RA1N1AC"}, _stack}
  }
end

The error struct is returned in the message tuple so you can pattern match on it and assert about.

This method is still subject to race conditions. The child process must throw the error before the assert_receive times out.

There is a different example in the Elixir docs for catch_exit.

Assert Test Process Did or Will Receive A Message

The ExUnit.Assertions module contains a function assert_receive which the docs state:

Asserts that a message matching pattern was or is going to be received within the timeout period, specified in milliseconds.

It should possibly in addition say “received by the test process”. Let’s see if we can send a message from a different process and assert that the test process receives it:

test_process = self()

spawn(fn ->
  :timer.sleep(99)
  send(test_process, :the_message)
end)

assert_receive(:the_message, 100)
# It Passes!!!

In the above code, :the_message is sent 1 millisecond before the timeout, and the assertion passes.

Now let’s reverse the assertion to refute_receive, and change the sleep to the same time as the timeout.

test_process = self()

spawn(fn ->
  :timer.sleep(100)
  send(test_process, :the_message)
end)

refute_receive(:the_message, 100)
# It Passes!!!

Yep, it passes.

Hiding and Revealing Struct Info with `inspect`

There are two ways to hide information when printing structs in Elixir.

Hiding by implementing the inspect protol.

defmodule Thing do
  defstruct color: "blue", tentacles: 7
end

defimpl Inspect, for: Thing do
  def inspect(thing, opts) do
    "A #{thing.color} thing!"
  end
end

So now in iex I can’t tell how many tentacles the thing has:

> monster = %Thing{color: "green", tentacles: 17}
> IO.inspect(monster, label: "MONSTER")
MONSTER: A green thing!
A green thing!

Note that iex uses inspect to output data which can get confusing.

NEW IN ELIXIR 1.8: You can also hide data with @derive

defmodule Thing do
  @derive {Inspect, only: [:color]}
  defstruct color: "blue", tentacles: 7
end

And now you won’t see tentacles on inspection

> monster = %Thing{color: "green", tentacles: 17}
> IO.inspect(monster)
#Thing<color: "green", ...>

In both cases, you can reveal the hidden information with the structs: false option:

> monster = %Thing{color: "green", tentacles: 17}
> IO.inspect(monster)
#Thing<color: "green", ...>
> IO.inspect(monster, structs: false)
%{__struct__: Thing, color: "green", tentacles: 17}

Custom Validation in Ecto

Sometimes the standard validation options aren’t enough.

You can create a custom validator in Ecto using the validate_change function.

In this particular case, I want to validate that thing has an odd number of limbs. I can pass the changeset to validate_odd_number.

def changeset(thing, attrs) do
  thing
  |> cast(attrs, [
    :number_of_limbs
  ])
  |> validate_odd_number(
    :number_of_limbs
  )
end

And then define validate_odd_number like this:

def validate_odd_number(changeset, field) when is_atom(field) do
  validate_change(changeset, field, fn (current_field, value) ->
    if rem(value, 2) == 0 do
      [{f, "This field must be an odd number"}]
    else 
      []
    end
  end)
end

We pass a function to validate_change that takes the field atom and the value for that field. In that function we test for oddness and return a keyword list that contains the field name and an error message.

Remember that when f is :number_of_limbs:

    [{f, "hi"}] == [number_of_limbs: "hi"]
    # true, these data structures are equal

Ecto's `distinct` adds an order by clause

When using distinct on you may encounter this error:

select distinct on (color) id, color from fruits order by id;
-- ERROR:  SELECT DISTINCT ON expressions must match initial ORDER BY expressions

This is because distinct on needs to enounter the rows in a specific order so that it can make the determination about which row to take. The expressions must match the initial ORDER BY expressions!

So the above query works when it looks like this:

select distinct on (color) id, color from fruits order by color, id;

OK, now it works.

Ecto’s distinct function helps us avoid this common error by prepending an order by clause ahead of the order by clause you add explicitly.

This elixir statement:

Fruit |> distinct([f], f.color) |> order_by([f], f.id) |> Repo.all

Produces this sql (cleaned up for legibility):

select distinct on (color) id, color from fruits order by color, id;

Ecto added color to the order by!

Without any order by at all, distinct does not prepend the order by.

Read the docs!

Transactions can timeout in Elixir

In Ecto, transactions can timeout. So this type of code:

  Repo.transaction fn -> 
    # Many thousands of expensive queries
    # And inserts
  end

This type of code might fail after 15 seconds, which is the default timeout.

In Ecto you can specify what the timeout should be for each operation. All functions that make a request to the database have the same same shared options of which :timeout is one.

Repo.all(massive_query, timeout: 20_000)

The above query now times out after 20 seconds.

These shared options apply to a transaction as well. If you don’t care that a transaction is taking a long time you can set the timeout to :infinity.

  Repo.transaction(fn -> 
    # Many thousands of expensive queries
    # And inserts
  end, timeout: :infinity)

Now this operation will be allowed to finish, despite the time it takes.

Timing A Function In Elixir

Erlang provides the :timer module for all things timing. The oddly named function tc will let you know how long a function takes:

{uSecs, :ok} = :timer.tc(IO, :puts, ["Hello World"])

Note that uSecs is microseconds not milliseconds so divide by 1_000_000 to get seconds.

microseconds are helpful though because sometimes functions are just that quick.

:timer.tc(IO, :puts, ["Hello World"])
# {22, :ok}

You can also call :timer.tc with a function and args:

adding = fn (x, y) ->  x + y end

:timer.tc(adding, [1,3])
# {5, 4}

Or just a function:

:timer.tc(fn -> 
    # something really expensive
    :ok
    end)
# {1_302_342, :ok}

Print the current stacktrace in Elixir

Stacktrace, backtrace, callstack, in Elixir its stacktrace and it’s available via Process.info/2 using the :current_stacktrace item:

Process.info(self(), :current_stacktrace)

And to print it:

IO.inspect(Process.info(self(), :current_stacktrace), label: "STACKTRACE")

I’m also learning that Process.info/2 takes a pid and an item as arguments. When you call Process.info/1 with just the pid you only get a subset of the info available, not everything.

The items available via Process.info/1 are listed in the erlang documentation here.

The additional items available via Process.info/2 are listed in the erlang documentation here.

You may note that backtrace is also an item that is available via Process.info but it contains more information than you are might need to figure out where you are in the code.

Flunk Your ExUnit Tests ❌

Today I got to use the flunk/1 function in an ExUnit test. I like everything about this function, from its name to its signature.

Here’s how we used it:

case result do
  "bad" ->
    flunk("Your result is bad, and you should feel bad.")

  "good" ->
    assert result_passes_other_tests(result)
end

It fit well into a helper function that should fail if my result ever comes back a certain way.

flunk docs

`mix phx.digest` creates the manifest

mix phx.digest creates production ready assets: hashed, zipped and compressed.

In production, when you call:

Endpoint.static_path('asset_name.css')

This will look for a file in the priv/static directory and return the path for that file. But what you want is the hashed version of that file, because modern browsers are greedy cachers and will you use the use to bust that cache.

The static_path function can return a path that represents the cached version, but it needs to read the manifest, which maps the file to it’s hashed, zipped and compressed versions.

mix phx.digest creates the manifest along with the hashed, zipped and compressed versions of all the files in the priv/static directory.

Update Map Syntax

There is a special syntax for updating a map in elixir.

thing = %{a: 1, b: 2, c: 3}
updated_thing = %{thing | b: 4}
# %{a: 1, b: 4, c: 3}

But be careful, it throws an error if you try to update a key that doesn’t exist!!

thing = %{a: 1, b: 2, c: 3}
# ** (KeyError) key :x not found in: %{a: 1, b: 2, c: 3}

Skip Pending Tests in ExUnit

In ExUnit you have the ability to tag a test with any atom:

@tag :awesome
test "my awesome test" do
end

@tag :terrible
test "my terrible test" do
end

And then you can exclude those tags at the command line so that your terrible test does not run:

mix test --exclude terrible

So you can make your own tag :pending and exclude those. If you don’t want to have to set the flag at the command line everytime, you can call ExUnit.configure. This is typically placed in the test/test_helper.exs file before ExUnit.start().

ExUnit.configure(exclude: :pending)
ExUnit.start()

Now your pending tests will not run.

That is how you can use custom tags to skip tests, but you can also just use the built in skip tag to skip tests.

@tag :skip
test "my terrible test" do
end

Timex `between?` is exclusive but can be inclusive

Timex.between? is a great function for determining if a time is in a specific time period, but it’s exclusive, so if you are testing for a time at the boundary the result will be negative.

iex> end_time = Timex.now()
iex> start_time = Timex.shift(end_time, days: -7)
iex> time = start_time
iex> Timex.between?(time, start_time, end_time)
false

Never fear, you can pass an inclusive option:

iex> Timex.between?(time, start_time, end_time, inclusive: true)
true

But you don’t have to be inclusive on both sides! Here, I pass :end so that I am only inclusive at the end of my time period.

iex> Timex.between?(time, start_time, end_time, inclusive: :end)
false

And of course I can be only inclusive at the beginning of the period if I prefer:

iex> Timex.between?(time, start_time, end_time, inclusive: :start)
true

Match across lines with Elixir Regex `dotall` (s)

Elixir Regex is PCRE compliant and Ruby Regex isn’t in at least one specific way. The m (multiline) flag behaves differently in Elixir and Ruby.

Without any modifiers, Regex does not match across new lines:

iex> Regex.scan( ~r/.*/, "a\nb")
[["a"], [""], ["b"], [""]]
iex> Regex.scan( ~r/.*/, "ab")
[["ab"], [""]]

With the m (multiline) modifier, Regex will match the beginning of each line when using the ^ char.

iex> Regex.scan( ~r/.*/m, "a\nb")
[["a"], [""], ["b"], [""]]
iex> Regex.scan( ~r/^.*/, "a\nb")
[["a"]]
iex> Regex.scan( ~r/^.*/m, "a\nb")
[["a"], ["b"]]

The s (dotall) modifier is the right way in Elixir to match across newlines.

iex> Regex.scan( ~r/.*/s, "a\nb")
[["a\nb"], [""]]
iex> Regex.scan( ~r/^.*/s, "a\nb")
[["a\nb"]]

In ruby, you can match across lines with the m (modifier) and the s is ignored.

irb> "a\nb".scan(/.*/)
=> ["a", "", "b", ""]
irb> "a\nb".scan(/.*/m)
=> ["a\nb", ""]
irb> "a\nb".scan(/.*/s)
=> ["a", "", "b", ""]

Read about the history of dotallhere

Get _just one_ value from Ecto query

With the data structure you use in select’s expression argument you can specify what type of data structure the query will return for a row. I’ve used [] and %{} and %Something{} but you can also specify that each row is just one value without using a data structure at all.

Combine that with Repo.one to just return one row and you can just get the specific value you are looking for without any destructuring.

age = 
  User
  |> where(id: 42)
  |> select([u], u.age)
  |> Repo.one()

Elixir sigil for DateTime with timezone

Elixir have a new sigil_U to be relased in the upcomming 1.9.0 version.

~U[2019-05-18 21:25:06.098765Z]

This new sigil creates a UTC DateTime.

Now Elixir will have all these sigils for dates & times:

Date.new(2019, 5, 18)
=> {:ok, ~D[2019-05-18]}

Time.new(23, 55, 6, 98_765)
=> {:ok, ~T[23:55:06.098765]}

NaiveDateTime.new(2019, 5, 18, 23, 55, 6, 98_765)
=> {:ok, ~N[2019-05-18 23:55:06.098765]}

DateTime.from_iso8601("2019-05-18T23:55:06.098765+02:30")
=> {:ok, ~U[2019-05-18 21:25:06.098765Z], 9000}

Install the last Elixir version with asdf

If you are using asdf for managing Elixir version through asdf-elixir then you can install the master version. This way we can use new features yet to be release as 1.9. Let’s see how:

asdf install elixir master
asdf global elixir master

Then:

elixir --version
Erlang/OTP 21 [erts-10.3.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Elixir 1.9.0-dev (6ac1b99) (compiled with Erlang/OTP 20)

Get pids for each beam application in Elixir

In Elixir, it’s hard to determine what pids belong to what applications just by using Process.info

iex> pid(0,45,0) |> Process.info
[
  current_function: {:application_master, :main_loop, 2},
  initial_call: {:proc_lib, :init_p, 5},
  status: :waiting,
  message_queue_len: 0,
  links: [#PID<0.46.0>, #PID<0.43.0>],
  dictionary: [
    "$ancestors": [#PID<0.44.0>],
    "$initial_call": {:application_master, :init, 4}
  ],
  trap_exit: true,
  error_handler: :error_handler,
  priority: :normal,
  group_leader: #PID<0.45.0>,
  total_heap_size: 376,
  heap_size: 376,
  stack_size: 7,
  reductions: 49,
  garbage_collection: [
    max_heap_size: %{error_logger: true, kill: true, size: 0},
    min_bin_vheap_size: 46422,
    min_heap_size: 233,
    fullsweep_after: 65535,
    minor_gcs: 0
  ],
  suspending: []
]

With the above output, I can’t tell what application this is!

I can use the :running key of Erlang’s :application.info to get a map of applications to pids.

iex> :application.info[:running]
[
  logger: #PID<0.93.0>,
  iex: #PID<0.85.0>,
  elixir: #PID<0.79.0>,
  compiler: :undefined,
  stdlib: :undefined,
  kernel: #PID<0.45.0>
]

Oh ok, pid(0,45,0) is the Kernel application.

Test that an email was sent with correct params

assert_email_delivered_with is a function that checks if an email was delivered using Bamboo.

Sending an email is a side-effect in functional programming parlance. When testing a function where the email is sent several calls deep, you generally don’t get the email as a return value of that function.

When using Bamboo’s TestAdaptor in test mode, Bamboo captures the delivered email so that you can assert about it.

Lottery.notify_winner()
assert_email_delivered_with(subject: "You Won!!")

You can also assert on: to, cc, text_body, html_body, from, and bcc.

Happy Testing!

Hide an Elixir Module from Documentation

Recently I’ve noticed @moduledoc false sprinkled through some Elixir projects. What’s going on? To quote the docs:

Conveniently, Elixir allows developers to hide modules and functions from the documentation, by setting @doc false to hide a particular function, or @moduledoc false to hide the whole module. If a module is hidden, you may even document the functions in the module, but the module itself won’t be listed in the documentation…

Why would you see this in a project that doesn’t autogenerate docs? Possibly to satisfy a linter like Credo. In a codebase with public APIs, I think this kind of explicit statement is a good practice.

Writing Elixir Docs

Formatting the email address fields with Bamboo

When you create an email with Bamboo and deliver it:

 email = new_email(
      to: "chris@example.com",
      from: "vitamin.bot@example.com",
      subject: "Reminder",
      text_body: "Take your vitamins"
    )
 |> Mailer.deliver_now()

The to field will not be what you expect it to be:

email.to
# {nil, "chris@example.com"}

This is the normalized format for a to field. The first element in the tuple is the name of the recipient.

Bamboo allows you to format this field yourself with protocols.

defimpl Bamboo.Formatter, for: Person do
  def format_email_address(person, _opts) do
    {person.nickname, person.email}
  end
end

And now if I send an email with Person in the to field:

person = %Person{
    nickname: "shorty", 
  email: "chris@example.com"
}

email = new_email(
      to: person,
      from: "vitamin.bot@example.com",
      subject: "Reminder",
      text_body: "Take your vitamins"
    )
|> Mailer.deliver_now()

Then the to field gets formatted with our protocol.

email.to
# {"shorty", "chris@example.com"}

Read more about the Bamboo Formatter here.

Fuzzy translation key merging with Gettext

Gettext is a great tool for i18n in elixir. It provides a mix task for extracting translation keys from your code. The translation keys (or message ids) are natural language and look like this:

gettext("Hi, Welcome to Tilex!")

After running mix gettext.extract && mix gettext.merge, an already translated Italian locale file would look like:

msgid "Hi, Welcome to Tilex!"
msgstr "Italian version of Welcome!"

There’s a chance that the natural language key (which also serves as the default string) will change.

If it changes just a little bit then the Italian locale file will look like:

#, fuzzy
msgid "Hi, Welcome to Tilex!!"
msgstr "Italian version of Welcome!"

It gets marked as #, fuzzy, and the new msgid replaced the old msgid.

Gettext determines how big of a change will constitute a fuzzy match with String.jaro_distance.

iex> String.jaro_distance("something", "nothing")
0.8412698412698413
iex> String.jaro_distance("peanuts", "bandersnatch")
0.576984126984127

The higher the number the closer the match. fuzzy_threshold is the configuration that determines whether a msgid is fuzzy or not and the default for fuzzy_threshold is 0.8, set here.

Weighted Shuffle in Elixir

Shuffling a list is easy, but what if you want to perform a weighted shuffle? That is, what if some elements in a list had a greater chance of being selected, or higher in the list, than others?

Given these colors and these weights:

weighted_colors = [{1, "orange"}, {3, "green"}, {5, "red"}]

I can create a list that is mostly represents each item the number of times of it’s weight by utilizing List.duplicate:

weighted_colors
|> Enum.map(fn ({weight, color}) -> 
  List.duplicate(color, weight)
end)
|> Enum.flatten
# ["orange", "green", "green", "green", "red", "red", "red", "red", "red"]

Then shuffle it:

|> Enum.shuffle
# ["red", "red", "orange", "red", "green", "green", "red", "green", "red"]

And then use Enum.uniq attribute of preserving the order in which it encounters each unique element from left to right.

|> Enum.uniq
# ["red", "orange", "green"]

Run this 100 times over the same weighted list and the most outcome that you would experience most often is:

# ["red", "green", "orange"]

Legible milliseconds in Elixir using Erlang timer

In Elixir, we often need to set milliseconds. This is especially common when working with GenServer, Task.Supervisor, Agent and others.

While we could write our milliseconds directly:

10800000

or perhaps

10_800_000

Did you know that was 3 hours just by glancing at it? Sure the math is fairly easy, but why make us think? Erlang’s timer to the rescue!

iex()> :timer.hours(3)
10800000

Well that makes our 3 hours pretty legible if you ask me.

We also have minutes and seconds:

iex()> :timer.minutes(11)
660000
iex()> :timer.seconds(455)
455000

Thanks Erlang!

Find duplicate routes in Elixir Phoenix

If you have duplicated routes in your route file like this:

scope "/api", MyAppWeb.Api, as: :api do
  pipe_through [:this]
  
  resources "/users", UserController, except: [:new, :edit]
  scope 
end

scope "/api", MyAppWeb.Api, as: :api do
  pipe_through [:that]

  resources "/users", UserController, except: [:new, :edit]
end

Then you’ll get a warning like this:

warning: this clause cannot match because a previous clause at line 2 always matches
  lib/idea_web/router.ex:2

The warning doesn’t really let you know which routes are duplicated, but it’s really ease to find the duplicated routes by utilizing the uniq command.

mix phx.routes | sort | uniq -d

The -d flag for uniq is duplicates only. -d only considers similar lines in consecutive order as duplicates, so you need to run it through sort first.

The output looks like this:

api_user_path  DELETE  /api/users/:id                   MyAppWeb.Api.UserController :delete
api_user_path  GET     /api/users                       MyAppWeb.Api.UserController :index
api_user_path  GET     /api/users/:id                   MyAppWeb.Api.UserController :show
api_user_path  PATCH   /api/users/:id                   MyAppWeb.Api.UserController :update
api_user_path  POST    /api/users                       MyAppWeb.Api.UserController :create

And those are your duplicate routes!

Named bindings in Ecto (vs positional bindings)

Positional bindings in Ecto are meant to confuse when buildings large queries across several different functions.

query = Thing
|> join(:inner, [t], x in X, on: t.x_id = x.id)
|> join(:inner, [_, x], y in Y, on: x.y_id = y.id)

Using that query in another function might look like this:

query
|> where([_, x, _], x.type == "Articulated")

But what if the positions change? How am I supposed to know which positions are which when I’m adding to this query out of context?

Named bindings help tremendously here (note the :as option):

query = Thing
|> join(:inner, [t], x in X, as: :x, on: t.x_id = x.id)
|> join(:inner, [_, x], y in Y, as: :y, on: x.y_id = y.id)

Now I can refer to these things without knowing the position. If the position changes it’s all good the additive where statement does not have to change:

query
|> where([y: y, x: x], x.type == "Articulated", y.feel == "Good")

How to actually _load_ the resource with Guardian

Guardian, like all auth libraries in all languages, is tough to wrap my head around.

I know there is a plug in the pipeline called plug Guardian.Plug.LoadResource. I know there is a function called Guardian.Plug.current_resource(conn) that takes the conn and returns that returns the resource placed in the conn by the Guardian.Plug.LoadResource plug.

What I don’t know is how the LoadResource plug knows what resource to get.

In Guardian, you configure the pipeline with:

use Guardian.Plug.Pipeline, otp_app: :my_app,
                              module: GuardianImpl,
                              error_handler: ErrorHandler

The GuardianImpl is a module that uses the Guardian behaviour.

The Guardian behaviour has a callback resource_from_claims that might be implemented like this:

def resource_from_claims(claims) do
  {:ok, Repo.get(User, claims["sub"])}
end

So when you need to modify how you load the resource, you should look to see how the resource_from_claims callback is implemented.

Read more here.

Updating the ExUnit test context with setup

When using ExUnit the second argument to the test macro is context.

test "1 + 1 = value", context do
  assert 1 + 1 == 2
end

This context can provide setup values so that you can share setup across tests.

Use the setup macro to update the context. The keyword list you return from the setup function will be merged into the map of the context.

setup do
  [result: 2]  # this gets merged into the context
end

setup do
  # create some database records
  :ok # this does not get merged into the context
end

Also in the setup macro you can have access to the context, allowing you to potentially change the context based on the needs of the test.

setup context do
  %{result: r} = context
    [result: r + 1]
end

test "1 + 1 = value", %{result: value} do
  assert 1 + 1 == value
end

Read more about ExUnit setup here.

Elixir ExDoc has version dropdown

ExDoc released a new version that allow developers to show a version dropdown on their documentation.

Here’s how I added to my library:

Open the mix.exs file and add javascript_config_path to the docs option on your project function.

def project do
  [
    ...
    docs: [
      main: "readme",
      extras: ~w(README.md),
      javascript_config_path: "../.doc-versions.js"
    ],
    ...
end

And on my Makefile I have this:

docs: ## Generate documentation.
docs: setup
    echo "var versionNodes = [" > .doc-versions.js
    app=`mix run -e 'IO.puts(Mix.Project.config()[:app])'`; \
    for v in $$(git tag | tail -r); do echo "{version: \"$$v\", url: \"https://hexdocs.pm/$$app/$$v/\"}," >> .doc-versions.js; done
    echo "]" >> .doc-versions.js
    mix docs

So if I run make docs this will generate or update a file .doc-versions.js from what I have on my git tag

And here is how it looks like:

image

Here’s the ExDoc changelog.

Elixir Pattern Matching with Variables

Let’s say you have a variable that you want to pattern match.

By default Elixir won’t use the variable’s value to do the pattern matching and it will do a regular assignment, overriding the original variable’s value:

iex(1)> year = 2020
2020

iex(2)> car = %{year: 2019}
%{year: 2019}

iex(3)> %{year: year} = car
%{year: 2019}

iex(4)> year
2019

Elixir has the pin opertator ^ that does exactly what we need. So in our example we can use the pin operator and if it doesn’t match you get an error:

iex(1)> year = 2020
2020

iex(2)> car = %{year: 2019}
%{year: 2019}

iex(3)> %{year: ^year} = car
** (MatchError) no match of right hand side value: %{year: 2019}

Pattern matching structs (why order is important)

When pattern matching structs with maps, many people don’t consider the consequences of order.

Say we have a user:

defmodule User do; defstruct [:name, :age] end

This will match just fine:

iex()> %{name: nil, age: nil} = %User{}
%User{age: nil, name: nil}

While the reverse will not:

iex()> %User{} = %{name: nil, age: nil}
Bug Bug ..!!** (MatchError) no match of right hand side value: %{age: nil, name: nil}

If you care to know why:

When pattern matching maps, the right side value must contain all the keys. In our example above, it appears they do:

iex()> inspect(%User{})
"%User{age: nil, name: nil}"

But looks can be deceiving:

iex()> inspect(%User{}, structs: false)
"%{__struct__: User, age: nil, name: nil}"

With our newfound knowledge, lets try again:

iex()> %User{} = %{name: nil, age: nil, __struct__: User}
%User{age: nil, name: nil}

🎉

How to assert Elixir doctest raises an error

Today I learned how to assert an Elixir doctest raises an error. Check this out:

defmodule MyModule do
  @doc """
  This function raises ArgumentError.

  ## Examples

      iex> MyModule.my_func()
      ** (ArgumentError) something is wrong
  """
  def my_func() do
    raise(ArgumentError, "something is really wrong")
  end
end

The previous doctest will fail with this message:

  1) doctest MyModule.my_func/0 (1) (MyModuleTest)
     test/my_module_test.exs:3
     Doctest failed: wrong message for ArgumentError
     expected:
       "something is wrong"
     actual:
       "something is really wrong"
     code: MyModule.my_func()
     stacktrace:
       lib/my_module.ex:10: MyModule (module)

Compute Intermediate Values In A With Construct

The expressions you use in a with construct do not have to contain the <- syntax. You can pattern match and bind values along the way as well.

with %{id: id} <- get_user(),
     url = "/api/#{id}/blogs",
     %{status_code: 200, body: body} <- HTTPoison.get(url),
     {:ok, decoded_body} <- Poison.decode(body) do
  {:ok, decoded_body}
end

In the above (sorta contrived) example we were able to construct a URL in the middle of the series of expressions.

The values we compute inline will be closed into the with construct, so they won’t leak.

See the with docs for more details.