Today I Learned

hashrocket A Hashrocket project

285 posts about #elixir surprise

Phoenix chunk huge files

So Phoenix, actually Plug, has a way to chunkify a response back to the client. This allowed us to create a stream out of an ecto query, then pipe that stream into a csv encoder and finally to Plug.Conn.chunk/2, which end up solving the huge memory spikes that we had when downloading a CSV. Here's a pseudo-code:

my_ecto_query
|> Repo.stream()
|> CSV.encode() # external lib that encodes from a stream into CSV
|> Enum.reduce_while(conn, fn chunk, conn ->
  case Conn.chunk(conn, chunk) do
    {:ok, conn} -> {:cont, conn}
    _ -> {:halt, conn}
  end
end)

Add an error to an Ecto Changeset

If you want to create custom validations with ecto changeset, chances are you are going to need to add errors to the changeset to be shown to the user.

Using add_error/4 allows you to add an error to a changeset.

changeset = change(%BlogPost{}, %{content: "Hello World ...cont"})
add_error(changeset, :content, "Your blog content is too long!")

changeset.errors
[content: {"Your blog content is too long!", []}]

Compare NaiveDateTimes by a specific unit

In Elixir, you can get the difference between two NaiveDateTimes using the NaiveDateTime.diff/3 function.

The third arg allows you to pass a unit such as :microsecond, :millisecond, or any unit returned from System.time_unit. By default, the unit is set to :second.

NaiveDateTime.diff(~N[2014-10-02 00:29:18], ~N[2014-10-02 00:29:10])
#=> 8
NaiveDateTime.diff(~N[2014-10-04 00:29:18], ~N[2014-10-02 00:29:10], :day)
#=> 2

Interestingly enough :day works as a unit, but not :month, or :year. 🤔

String Concat Pattern Matching In Elixir

This is a neat pattern matching trick in elixir, its best explained with a simple example:

invoice_message = "You owe $34"
"You owe " <> dollar_amount = invoice

IO.inspect(dollar_amount)
# => "$34"

With a slightly different situation, It may seem like you could do this:

invoice_message = "You owe 34 dollars"
"You owe " <> dollar_amount  <> " dollars"= invoice

IO.inspect(dollar_amount)
# => ** (ArgumentError) the left argument of <> operator inside
# a match should always be a literal binary because its size can't be verified. Got: dollar_amount

But sadly you'll need to use regex to do that because elixir will throw an error.

Custom Sigils In Elixir

You can create custom sigils by following the sigil_{character} definition pattern. Let's make an addition sigil sigil_a that sums up space separated numbers.

defmodule CustomSigil do
  def sigil_a(string, []) do
    string
    |> String.split(" ")
    |> Enum.map(& String.to_integer(&1))
    |> Enum.sum()
  end
end

# ~a(2 4 6)
#=> 12
# ~a(12 12)
#=> 24

Elixir's Info function

The elixir Kernel module has an interesting function i/1 that returns information about whatever you pass to it.

It will provide the argument's data type, byte size, raw representation, a description, and reference modules

i("hello world")

#=>
# Term
#  "hello world"
# Data type
#  BitString
# Byte size
#  11
# Description
#  This is a string: a UTF-8 encoded binary. It's printed surrounded by
#  "double quotes" because all UTF-8 encoded code points in it are printable.
# Raw representation
#  <<104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100>>
# Reference modules
#  String, :binary
# Implemented protocols
#  Collectable, IEx.Info, Inspect, List.Chars, String.Chars

How to debug in Elixir

If you want to debug an elixir code execution you can use the dbg/2 macro. I was already using that nice macro to show the results on each step of a pipe execution on the logs, but I learned that if we use iex to start phoenix server then the dbg/2 calls will act like adding a breakpoints so we can pry on it. We need to start the server with:

iex -S mix phx.server

As a side note this debugging worked on the elixir 1.14.3 version but I saw that on the latest versions there's a new option to be passed to iex --dbg pry in order to swap the dbg implementation from the fancy IO output to the IEx.pry one.

Elixir Data Type Comparison

Elixir has an interesting defined sorting order for its data types. The order is as follows:

number < atom < reference < function < port < pid < tuple < map < list < bitstring

For example:

random_data = [["Hello World"], {:hello, :world}, 16, %{key: "value"}, :atom]
Enum.sort(random_data)
#=> [16, :atom, {:hello, :world}, %{key: "value"}, ["Hello World"]]

Or even

4 < :atom
#=> true

Unix Timestamps in Elixir

Unix timestamps are a very simple way to compare times in an integer format. They are the total seconds elapsed since January 1st, 1970 UTC (The Unix Epoch).

In elixir, if you want to do some operations on a DateTime struct, and want to keep things simple, you can convert a DateTime struct to an integer with to_unix/2. By default, it will use seconds as the unit.

How to get javascript params in LiveView?

I found out that Phoenix LiveView has a function get_connect_params to be used when you want to get some value sent at the connection time from the javascript.

This way I can send params like:

// app.js
const params = {
  _csrf_token: csrfToken,
  timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
};

const liveSocket = new LiveSocket('/live', Socket, { params });

And get those like on my LiveView:

# my_live_view.ex
@impl true
def mount(_params, _session, socket) do
  tz = get_connect_params(socket)["timezone"] || "America/New_York"
  {:ok, assign(socket, :timezone, tz)}
end

Skip hidden fields on Phoenix inputs_for

I just found out that Phoenix Component inputs_for accepts a skip_hidden attr which was very useful today. We have a has_many relationship that we are using <.inputs_for> to decompose the nested fields and as it was generating some hidden fields the style was breaking.

We are actually using a Tailwind divide on the parent tag and those hidden fields were adding extra separation borders. A way to solve that was calling <.inputs_for twice as this:

<div>

  <.inputs_for field={@form[:posts]} />

  <div class="divide-y">
    <.inputs_for :let={form} field={@form[:posts]} skip_hidden>
      <.input type="text" field={form[:title]} label="Title" />
    </.inputs_for>
  </div>
</div>

Macro Guard Clauses in Elixir

Have you ever wanted to pass your own custom function into a guard clause?

Lets start by looking at a super basic guard clause here:

def greater_than_10(num) when num > 10, do: true
def greater_than_10(_), do: false

Let's say we want to get more specific with the operation inside of the when guard. This is overkill for our situation, but lets say we want the when to use a custom function of ours instead of a basic operation, but when guards don't allow you to directly pass functions to them.

Elixir has a fancy macro called defguard that allows you to define custom guards like functions. Let's change our function up and define a guard that checks if our argument is both a integer and even.

defmodule Guards do
  defguard is_even_num(num) when is_integer(value) and rem(value, 2) == 0
end

Let's use that guard in our function

import Guards

def even_num_greater_than_10(num) when is_even_num(num) do
  case num > 10 do
    true -> true
    _ -> false
  end
end

Even though this example is a bit overkill, the option to use custom guards is a cool elixir feature.

Casting Associations in Phoenix

If you want to cast changes on a struct's associations, you can use the changeset function cast_assoc/3. This allows you to make changes to a parent struct and its associations at the same time.

Example

Lets say you have a struct called Artist and an associated schema called Album. The schema for the Album looks something like this:

def module MyApp.Music.Album do
 use Ecto.Schema
 alias MyApp.Music.Artist

  schema "albums" do
    field :title, :string
    
    belongs_to :artist, Artist
  end

  def changeset(artist, params \\ %{}) do
    artist
    |> cast(params, [:title, :artist_id])
    |> validate_required([:title])
  end
end

and the schema for artist looks like this:

def module MyApp.Music.Artist do
 use Ecto.Schema
 alias MyApp.Music.Album

  schema "albums" do
    field :name, :string
    
    has_many :albums, Album
  end
end

Now, if you wanted to build a changeset for the Artist where you can change the artist's name and the title field on an associated album:

def changeset(artist, params \\ %{}) do
    artist
    |> cast(params, [:name])
    |> cast_assoc(:albums, with: &MyApp.Album.changeset/2)
    |> validate_required([:name])
end

Convert Changeset Into Schema With `apply_changes`

Often times, it's necessary to perform some intermediate operations on a Ecto changeset before saving it to the database. It can be easier to deal with the underlying schema. And you might need the entire schema rather than only the changes in the current changeset.

You can use the apply_changes/1 function to apply that changeset and receive schema with which you can perform operations. It's worth noting that the data is not validated when changes are applied, so care needs to be taken to ensure validity before an attempt to save that record.

iex> lead_schema = Lead.changeset(%{name: "Andrew", email: "Andrew.Vogel@hashrocket.com"})
                   |> apply_changes() 
%Lead{name: "Andrew", email: "Andrew.Vogel@hashrocket.com"}

iex> lead_schema |> downcase_email() |> do_some_other_processing()

https://hexdocs.pm/ecto/Ecto.Changeset.html#apply_changes/1

Phoenix LiveView slot attributes

Today I learned how to define an attribute on a Phoenix LiveView slot using a do block:

slot :column
  attr :field, :atom
  attr :sortable, :boolean, default: false
end

def table(assigns) do
  ~H"""
  <table>
    <tr>
      <th :for={col <- @column}>
        <%= col.label %>
        <.sort_by :if={col.sortable} field={col.field}/>
      </th>
    </tr>
  </table>
  """
end

This way we can use our slots with attributes:

def render(assigns) do
  ~H"""
    <.table>
      <:colum field={:name} sortable>Name</:colum>
      <:colum field={:price}>Price</:colum>
      <:colum>Actions</:colum>
      ...
    </.table>
  """
end

Check this out

Pattern Matching Args with Exact Values

In elixir there is a handy trick for pattern matching for exact values. Let's say we have a function head_match that takes two arguments, a string and a list, that checks if the string argument is the same as the head of the list. We could use pattern matching like this:

def head_match(head, [head, _tail]) do
  IO.puts("Thats the head of the List!")
end
#this will match if we called it like head_match("first_word", ["first_word", "second_word"])

See how we named the first argument head and used that same name as the list head? This means that the function will only match if those two values are the exact same rather than just matching the structure of the arguments.

Shorthand Elixir Anonymous Functions

Elixir has a really cool syntax for writing anonymous functions (unnamed functions). It goes like this:

# The '&' operator is used to define the function and its arguments
putter = &(IO.puts &1)
putter.("Today I learned")
#...> Today I learned
#...> :ok

&1 refers to the functions first argument; in this example, we used "Today I learned". You can use more arguments with &2, &3, etc... For example:

combiner = &(&1 <> &2)
combiner.("Foo", "Bar")
#...> "FooBar"

Also notice that we are not calling either of these functions like function(arg1, arg2), instead we are calling them like function.(arg1, arg2). We have to use . because we are not actually naming these functions, we are only assigning them a reference, hence "anonymous" functions.

Pattern Match Keyword List in Function Def

TIL that you can pattern match a keyword list in a function definition.

Sometimes you'll receive the last argument as an empty keyword list; in this case, we're calling it opts. You can pattern match key-values by defining the shape in a function.

  defmodule Example do
    def hello(opts \\ [])
 
   # will match when message is the only keyword included in opts
    def hello([message: message]) do
      IO.puts("Hello #{message}")
    end

    # will match when there are multiple keywords but message is the first
    def hello([{:message, message} | _rest] = opts) do
      IO.puts("Hello #{message}")
    end
  end

> Example.hello(message: "World")
Hello World
:ok

# could also call in the more verbose way
> Example.hello([message: "World"])
Hello World
:ok

# :message but with other args after
> Example.hello(message: "World", something_else: "hi")
Hello World
:ok

Note: As a TIL reader pointed out, pattern matching Keywords will make your function args order dependent. The following would not work:

> Example.hello(foo: "bar", message: "World")
** (FunctionClauseError) no function clause matching in Example.hello/1 

If you need them to be order independent, use a map or just match on single argument, then check for each option appropriately with the Keyword module.

https://elixir-lang.org/getting-started/keywords-and-maps.html

Composite Primary Keys using Elixir Ecto

Ecto allows us to map a table with composite primary keys into our schemas. The way to do that is by using the primary_key: true option for the Ecto.Schema.field/3 instead of dealing with the special module attr @primary_key. Check this out:

defmodule MyApp.OrderItem do
  use Ecto.Schema

  @primary_key false
  schema "order_items" do
    field :product_id, :integer, primary_key: true
    field :order_id, :integer, primary_key: true
    field :quantity, :integer
  end
end

Identity Primary Keys on Ecto

Ecto, allow the type :identity to be used since 3.5 which is cool:

create table(:user, primary_key: false) do
  add :id, :identity, primary_key: true
  add :name, :string, null: false

  timestamps()
end

That generates this SQL:

CREATE TABLE public.users (
    id bigint NOT NULL,
    name character varying(255) NOT NULL,
    inserted_at timestamp(0) without time zone NOT NULL,
    updated_at timestamp(0) without time zone NOT NULL
);

ALTER TABLE public.users ALTER COLUMN id ADD GENERATED BY DEFAULT AS IDENTITY (
    SEQUENCE NAME public.users_id_seq
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1
);

The only issue is that there's no option to change from BY DEFAULT to ALWAYS 😕

Phoenix component attr definition

The new Phoenix.Component attr/3 function is awesome. It does compile time validations to all the places that are calling the component and also it comes with nice and useful options. For instance the default option:

attr :name, :string, default: "there"

def greet(assigns) do
  ~H"""
  <h1>Hey <%= @name %>!</h1>
  """
end

That's very useful. That would totally replace most usages of assign_new like I used to do:

def greet(assigns) do
  assigns = assign_new(assigns, :name, fn -> "there" end)

  ~H"""
  <h1>Hey <%= @name %>!</h1>
  """
end

This way we can call:

<.greet />

And have this html generated:

<h1>Hey there!</h1>

Migrating Data in Ecto

I usually have created a mix task for data migrations to avoid putting the app down, but today I learned that ecto migrations accept an arg --migrations-path to their commands which allow us to have 2 separate folders for migrations.

With that we can easily use the default priv/repo/migrations folder for automatic migrations (for running on deployments) and a separate folder, let's say priv/repo/data_migrations that we can run when it's more convenient.

So in prod we run migrations on deploy and data_migrations on a quite time for the app to avoid downtime. And in dev and test env we just run them all as we usually have smaller dataset, so not a big deal.

Here's a good post about this strategy.

Use created_at in Ecto

You can use created_at in Ecto/phoenix app with timestamps/1. When migrating data from a rails application to a phoenix application you will have many tables with a created_at column.

defmodule Phoenix.Accounts.User do
  use Ecto.Schema
  import Ecto.Changeset

  schema "users" do
    field :email, :string
    field :password, :string, virtual: true, redact: true
    field :hashed_password, :string, redact: true
    field :confirmed_at, :utc_datetime

    timestamps(inserted_at: :created_at, type: :utc_datetime)
  end

end

Elixir Compilation Cycles with `--fail-above`

Elixir 1.13.0 introduced a --fail-above flag for the mix xref task which will fail that task execution under a certain criteria.

With that we can verify, for example, that our code is ok to have 1 cycle that goes over length of 4, but not 2 cycles. Let's see how it works:

$ mix xref graph --format cycles --min-cycle-size 4 --fail-above 1
2 cycles found. Showing them in decreasing size:

Cycle of length 6:

    lib/my_app_web/endpoint.ex
    lib/my_app_web/router.ex
    lib/my_app_web/views/layout_view.ex
    lib/my_app_web/live/components/sorting_link.ex
    lib/my_app_web/live/components/icon.ex
    lib/my_app_web/endpoint.ex

Cycle of length 5:

    lib/my_app_web/live/components/sorting_link.ex
    lib/my_app_web/live/components/icon.ex
    lib/my_app_web/router.ex
    lib/my_app_web/views/layout_view.ex
    lib/my_app_web/live/components/sorting_link.ex

** (Mix) Too many cycles (found: 2, permitted: 1)

In this case xref found 2 cycles with a length greater than 4, and as I allowed only 1 then we can see the error.

Elixir __struct__/0

When we define a struct via defstruct macro we end up getting a __struct__/0 function on both struct module definition and on each struct map. The intriguing part is that the implementation of the module and the map are different, check this out:

iex(1)> defmodule Book do
...(1)>   defstruct title: nil, pages_count: 0
...(1)> end

iex(2)> Book.__struct__()
%Book{pages_count: 0, title: nil}

iex(3)> Book.__struct__().__struct__()
Book

As we can see Book.__struct__() returns a new %Book{} struct with its defaults, meanwhile %Book{}.__struct() returns the Book module.

Elixir IEX multi-line command

A change on Elixir 1.12.0 made possible to pipe |> multi-line commands in iex where the |> operator is in the beginning of new lines.

That means that we can:

iex(1)> :foo
:foo
iex(2)>       |> to_string()
"foo"
iex(3)>       |> String.upcase()
"FOO"

The docs also mention that all other binary operators works the same way, except +/2 and -/2, so that's also valid:

iex(1)> [:foo]
[:foo]
iex(2)> ++ [:bar]
[:foo, :bar]
iex(3)> |> Enum.join(" ")
"foo bar"
iex(4)> |> String.upcase()
"FOO BAR"

Each line will run at a new command, so if you assign a variable in the first line you may not have what you expect, so watch out.

Phoenix Live View enable Profiling

Phoenix LiveView has a way to enable Profiling in the client side by just adding this into the app.js file:

// app.js
liveSocket.enableProfiling();

That will enable a log into your browser console such as:

toString diff (update): 1.224853515625 ms
premorph container prep: 0.006103515625 ms
morphdom: 397.676025390625 ms
full patch complete: 411.117919921875 ms

In this case we can see that the morphdom library is taking a considerable time to apply my DOM patches as my page has a huge html table full of data.

BTW, this function adds to 2 other very useful ones for debugging the client:

  • enableDebug ()
  • enableLatencySim(ms)

Get the Values for a Ecto Schema Enum Column

I recently started learning Elixir and had a model with an enum column with the following attributes:

 schema "keyboards" do
   field :nickname, :string
   field :form_factor, Ecto.Enum, values: [:macro, :num, :custom, :split, :forty, :sixty, :sixty_five, :seventy_five, :tkl, :full]

   timestamps()
 end

In my view, I had a form object where I wanted to have a select input with the values from my enum column, form_factor. Luckily, the Ecto.Enum module has a few functions that can help with this - mappings/2, values/2, and dump_values/2.

I ended up using the following in my form:

  <%= label f, :form_factor %>
  <%= select f, :form_factor, Ecto.Enum.mappings(Keyboard, :form_factor)  %>
  <%= error_tag f, :form_factor %>

https://hexdocs.pm/ecto/Ecto.Enum.html

Get Elixir GenServer current state

Today I learned that we can use :sys.get_state/1 to get the current state of a GenServer.

Check this out:

iex(1)> pid = Process.whereis(MyApp.Repo)
iex(2)> :sys.get_state(pid)

{:state, {:local, MyApp.Repo}, :one_for_one,
 {[DBConnection.ConnectionPool],
  %{
    DBConnection.ConnectionPool => {:child, #PID<0.421.0>,
     DBConnection.ConnectionPool,
     {Ecto.Repo.Supervisor, :start_child,
      [
        {DBConnection.ConnectionPool, :start_link,
         [
           {Postgrex.Protocol,
            [
              types: Postgrex.DefaultTypes,
              repo: MyApp.Repo,
              telemetry_prefix: [:my_app, :repo],
              otp_app: :my_app,
              timeout: 15000,
              database: "my_app_dev",
              hostname: "localhost",
              port: 5432,
              show_sensitive_data_on_connection_error: true,
              pool_size: 10,
              pool: DBConnection.ConnectionPool
            ]}
         ]},
        MyApp.Repo,
        Ecto.Adapters.Postgres,
        %{
          cache: #Reference<0.1645792067.2416050178.91334>,
          opts: [
            timeout: 15000,
            pool_size: 10,
            pool: DBConnection.ConnectionPool
          ],
          repo: MyApp.Repo,
          sql: Ecto.Adapters.Postgres.Connection,
          telemetry: {MyApp.Repo, :debug, [:my_app, :repo, :query]}
        }
      ]}, :permanent, false, 5000, :worker, [Ecto.Repo.Supervisor]}
  }}, :undefined, 0, 5, [], 0, :never, Ecto.Repo.Supervisor,
 {MyApp.Repo, MyApp.Repo, :my_app, Ecto.Adapters.Postgres, []}}

Split a string in elixir

You can split a string in elixir with String.split

defmodule Phoenix.Repo.Migrations.CreateUsers do
  use Ecto.Migration

  def up do
    execute_sql("""
      create extension citext;

      create table users (
        id bigint generated by default as identity primary key,
        email citext unique not null,
        inserted_at timestamptz not null default now(),
        updated_at timestamptz not null default now()
      );
    """)
  end

  def down do
    execute_sql("""
      drop table users;
      drop extension citext;
    """)
  end
   

  def execute_sql(sql_statements) do
     sql_statements
     |> String.split(";")
     |> Enum.filter(fn s -> String.trim(s) != "" end)
     |> Enum.each(&execute/1)
  end
end

Elixir: Sandboxing an `iex` session

We can change the default .iex.exs file for another to be pre-loaded by iex using --dot-iex:

MIX_ENV=test iex --dot-iex .iex-sandbox.exs -S mix

This way we could create a rudimentary way to "protect" our existing database of changes on the iex session. Check this out:

# .iex-sandbox.exs
:ok = Ecto.Adapters.SQL.Sandbox.checkout(MyApp.Repo, ownership_timeout: 300_000)

This approach works for existing databases, so if the app we are working has a very difficult data setup then maybe this is the way to go. Although, making the db to hold data changes for as long as iex sessions are running it can be a bit too much in terms of db memory. Row & Table locks can definately be a problem as well.

Another important note is that we may have to call Sandbox.allow/3 if we need to call a GenServer or another process that touches the db.

The way to go

If we have a simpler data, with a nice seeds file, we'd be much better this way:

(export DB_NAME=my_app_sandbox && mix ecto.reset && iex -S mix)

Don't forget to change our ecto config to use that ENV var.