Today I Learned

A Hashrocket project

39 posts about #devops

Killing heroku dynos

Yesterday we encountered an odd situation. A rake task running with heroku that did not finish. Through no fault of our own.

We killed the local processes kicked off by the heroku command line tool and restarted the rake task but got an error message about only allowing one free dyno. Apparently, the dyno supporting our rake task was still in use.

First we had to examine whether that was true with:

heroku ps -a myapp

Next step was to kill the dyno with the identifier provided by the above command.

heroku ps:kill dyno.1 -a myapp

We ran the rake task again, everything was fine, it worked great.

sub_filter + proxy_pass requires no gzip encoding

The nginx sub module is useful for injecting html into web requests at the nginx level. It looks like this:

location / {
  sub_filter '</head>' '<script>doSomething();</script></head>';
}

Replace the end closing tag with a script tag and the head closing tag.

This does not work however with the proxy_pass module response coming from the proxy_pass url is gzip encoded.

In this case, you want to specify to the destination server that you will accept no encoding whatoever with the proxy_set_header directive, like so:

  proxy_set_header Accept-Encoding "";

Altogether it looks like this:

location / {
  proxy_set_header Accept-Encoding "";
  proxy_pass http://destination.example.com
  sub_filter '</head>' '<script>doSomething();</script></head>';
}

Examine owner and permissions of paths

The ownership of an entire path hierarchy is important when nginx needs to read static content. For instance, the path /var/www/site.com/one/two/three/numbers.html may be problematic when directory three is owned by root rather than www-data. Nginx will respond with status code 403 (forbidden) when http://site.com/one/two/three/numbers.html is accessed.

To debug this scenario we need to examine the owners and pemissions of each of the directories that lead to numbers.html. This can be accomplished in linux with the handy namei command in combination with realpath.

Real path will return the fullpath for the file.

> realpath numbers.html
/var/www/site.com/one/two/three/numbers.html

And namei -l <full_path> will examine each step in the file’s directory structure.

> namei -l $(realpath numbers.html)
drwxr-xr-x root  root  /
drwxr-xr-x root  root  var
drwxr-xr-x www-data www-data www
drwxr-xr-x www-data www-data site.com
drwxr-xr-x www-data www-data one
drwxr-xr-x root root two
drwxr-xr-x www-data www-data three
-rw-r--r-- www-data www-data numbers.html

Ah there it is. The directory two is owned by root rather that www-data. chown will help you from here.

Disk usage for just the top level dirs

Every so often your hard drive runs out of space and you need to do a survey of what could be possibly be taking up so much space. The du disk-usage command is great for that.

It can be noisy though as it traverses through all the directories and gives you a report on the size of each file. What we really want is just the size of each directory under the root dir.

du has a depth flag (-d) to help control the depth of directories that the command reports on.

du -h -d1 /

The above command gives you a report on all the directories and files at the top level.

Logrotation for a Rails App

This week I did a some Linux server logrotation for a Ruby on Rails application. log/your_log.log can get large on a server with traffic, and sometimes we must control how long to retain data.

Here’s the basic configuration file I wrote, with comments:

# Logrotater:
# - Daily
# - Timestamped
# - Doesn't error on missing log files
# - Keeps seven copies
# - Truncates log files (allows Rails to start writing 
#  to the new log file without a restart)

/var/app/current/log/your_log.log {
    daily
    dateext
    missingok
    rotate 7
    copytruncate
}

There are many other options. Check out man logrotate on a Linux machine for more info.

Raid 0 offers performance with worse reliability

RAID is defined:

Redundant Array of Independent Drives

Whenever I think of RAID I think of data drives that each contain the same information, for fault tolerance. Different RAID configurations are given different numeric identifiers. Recently, when shopping for motherboards I saw RAID 0 listed as a selling point for many of the boards and I assumed this was a fault tolerance feature on these motherboards meant for gamers.

RAID 0 has nothing to do with fault tolerance. Nor redundance. In a Raid 0 configuration data is spread across multiple drives with the intention of increasing the bandwidth of data from the motherboard/CPU to the drives. It is a technique for increasing performance that actually reduces fault tolerance.

Read more about RAID here -> https://en.wikipedia.org/wiki/Standard_RAID_levels

Push Variables to Heroku

If you use Heroku to deploy your apps, I have a great command to try.

Included in the Heroku toolbelt command-line interface is heroku config:push.

This command pushes your local environmental variables, defaulting to a file called .env in your root directory, to any Heroku remote. It favors existing remote configs in the event of a collision.

No more copying and pasting to the command line, or pointless typos while setting remote configs.

Bonus: also check out heroku config:pull— equally useful.

Copy that ssh pub key to the remote server

When setting up a server its customary to copy the ssh key to the authorized keys file so that you don’t rely on password auth to sign in to the remote server.

Its generally something like a 4 or 5 step process but linux has a utility to get it down to one step, ssh-copy-id.

$ ssh-copy-id dev@myserver.com

After entering your password you’re all set! Not turn off that password authentication in your ssh config and start deploying your app!

Restore a Heroku PG Backup From Another App

If your staging and production instances are both hosted on Heroku, you can restore your production Postgres database to your staging instance from the Heroku CLI. No posting the dump on AWS, no backing up to a local dump, etc.

$ heroku pg:backups capture -rproduction

This will print the name of your new backup; let’s call it b234, and the production app name my-blog.

$ heroku pg:backups restore my-blog::b234 DATABASE_URL -rstaging

The databases are now in sync.

.cert vs .pem

Fellow Rocketeer Dillon Hafer had the explanation for my SSL issue mentioned at https://til.hashrocket.com/posts/c38666d448-react-native-heroku-and-ssl-heroku- reactnative -

“DNSimple gives you a cert and a pem file. The cert file is just the certificate, while the pem file is actually 4 certificates mushed into one file. The 3 extra certificates are the intermediate certificates required by some operating systems that only include Root Certificates. Hope that helps explain why the pem from DNSimple works while the cert doesn’t.” - Dillon Hafer

Chaining TLS Certificates 🐱

Apache allows you to declare an intermediate TLS certificate along with your regular certificate in your configuration, but many web servers only allow you to provide one certificate option. Like Nginx, go, or heroku.

In those cases, you will need to concatenate the entire certificate chain into one certificate file. This may sound daunting, but the process is very simple. Let me introduce cat -- concatenate and print files.

cat is normally used for printing files, but in this case we actually want to concatenate files. Below is a simple example on how we can do this:

cp example_com.crt example_com.chained.crt
cat AddTrustExternalCARoot.crt >> example_com.chained.crt
cat COMODORSAAddTrustCA.crt >> example_com.chained.crt
cat COMODORSADomainValidationSecureServerCA.crt >> example_com.chained.crt

To shorten it, we only need to use cat once:

cp example_com{,.chained}.crt &&
cat AddTrustExternalCARoot.crt COMODORSAAddTrustCA.crt COMODORSADomainValidationSecureServerCA.crt >> example_com.chained.crt

Verify TLS cert with private key

Hopefully you’re never in a situation where you don’t know what private key you used to generate your TLS certificate, but if you do… here’s how you can check.

Note: this is better than uploading the certs to production to check on them 😉

Assuming we have generated a private key named example.com.key and a certificate named example.com.crt we can use openssl to check that the MD5 hashes are the same:

openssl x509 -noout -modulus -in example.com.crt | openssl md5
openssl rsa -noout -modulus -in example.com.key | openssl md5

To make things better, you can write a script:

#!/bin/bash
CERT_MD5=$(openssl x509 -noout -modulus -in example.com.crt | openssl md5)
 KEY_MD5=$(openssl rsa  -noout -modulus -in example.com.key | openssl md5)
 
if [ "$CERT_MD5" == "$KEY_MD5" ]; then
  echo "Private key matches certificate"
else
  echo "Private key does not match certificate"
fi

Intel Speedstep and Ubuntu 14.04 Performance

Intel Speedstep works with the OS to adjust the clock speed of the CPU in real-time to save power. Older Linux kernels had a poor interaction with Speedstep that could cause the CPU to be downclocked even when running something demanding like a test suite. This can be fixed by disabling Speedstep in BIOS or upgrading the kernel. I was on kernel 3.13 and upgraded to 4.2. I saw 15-40% speed increase when running these tests.

Ubuntu 14.04.4 ships with a new kernel, but older installs of 14.04 will not be automatically upgraded. Run uname -a to see what kernel you are running. If it is not at least 4.2, then you may want to upgrade your kernel 1. Using aptitude this is simple as:

sudo aptitude install linux-generic-lts-wily

1 Upgrading the kernel in this way should be safe, but changing the kernel always has the potential to break something, so proceed at your own risk.

Test Your Nginx Configuration

Nginx misconfiguration can produce vague messages like these:

jake@myserver-2:~# sudo service nginx reload
 * Reloading nginx configuration nginx
   [fail]

Turn up the verbosity with this command:

jake@myserver-2:~# nginx -c /etc/nginx/nginx.conf -t

The -c flag indicates a certain configuration file will follow; the -t flag tells Nginx to test our configuration. This produces much more useful errors like this:

nginx: [emerg] unknown directive "erver_name" in /etc/nginx/sites-enabled/default:3
nginx: configuration file /etc/nginx/nginx.conf test failed

erver_name should be server_name— we have an actionable error message.

List The Statuses Of All Upstart Jobs

To see a list of all known upstart jobs and their statuses, use the following command:

$ initctl list
...
console stop/waiting
mounted-run stop/waiting
acpid start/running, process 2927
checkfs.sh start/running
checkroot-bootclean.sh start/running
kmod stop/waiting
mountnfs.sh start/running
nginx stop/waiting
plymouth-stop stop/waiting
rcS stop/waiting
ufw start/running
...

It will tell you for each job if it is stopped or started.

See man initctl for more details.

h/t Josh Davey

Reuse a Mac OS Installer

Once you’ve downloaded a Mac OS Installer, you know it takes a long time. Multiply that time and the half-hour installation by many workstations, and you have potentially days of work to upgrade them all.

We can slash that time by reusing the installer. ssh into a machine with the package, and ls /Applications/. We should see our directory, named Install\ OS\ X\ El\ Capitan.app/ (insert your OS version name). scp -r that directory to your local /Applications/, which should take a few minutes.

Back on your machine, find ‘Install OS El Capitan’ with the Finder, and follow the installation steps.

h/t Dillon Hafer

-y apt-get?

When installing packages on Ubuntu, you may find it really tiring to constantly confirm ‘yes’ all the time. I know I did. And when it comes to scripting your installs… that really becomes a nuisance. Today I learned that apt-get has a -y flag:

Automatic yes to prompts;
assume "yes" as answer to all prompts and run non-interactively.

Aliasing An Ansible Host

When specifying the hosts that Ansible can interact with in the /etc/ansible/hosts file, you can put just the IP address of the host server, like so:

192.168.1.50

IP addresses are not particularly meaningful for a person to look at though. Giving it a name serves as better documentation and makes it easier to see what host servers are in play during a task.

Ansible makes it easy to alias host servers. For example, we can name our host staging like so:

staging ansible_host=192.168.1.50

source

Keep mina fast

One thing I’ve loved about mina is the speed of deployments. One way mina achieves fast deploys is by avoiding unnecessary tasks.

I recently started using webpack with rails and soon found I needed that same performance boost. Because I was using mina/rails I had a nice little macro already available for me 😁

Example of #check_for_changes_script

desc "Install npm dependencies"
task :install do
  queue check_for_changes_script \
    check: 'package.json',
    at: ['package.json'],
    skip: %[echo "-----> Skipping npm installation"],
    changed: %[
      echo "-----> #{message}"
      #{echo_cmd %[NODE_ENV=#{ENV['to']} npm install]}
    ],
    default: %[
      echo "-----> Installing npm modules"
      #{echo_cmd %[NODE_ENV=#{ENV['to']} npm install]}
    ]
end

Encrypt a zip archive

When you’re using the zip CLI on your machine or a remote server and you need some extra security you can use the -e flag to encrypt a zip archive. Be sure to use a super long random password for this.

zip -e secure-files.zip ~/Documents/*.pdf

-e       
--encrypt
       
  Encrypt the contents of the zip archive using a password which is entered on
  the  terminal  in response to a prompt (this will not be echoed; if standard
  error is not a tty, zip will exit with an error).  The  password  prompt  is
  repeated to save the user from typing errors.

Debian alternatives

Debian/Ubuntu has a number of chores for which the application to use for that chore can be configured by you. Editor and pager are 2 of the most common and generally I’ve set the environment variables for those in the past. Setting EDITOR and PAGER determines what happens when an application puts you into a page mode or which an application requires that you edit something.

Debian/Ubuntu though has a system for determining what application should be used in that instance, the alternatives system.

Running ls -l /usr/bin | grep 'alternative' will show you all the programs that are linked to the /etc/alternatives directory. The /etc/alternatives dir in turn has symlinks that point to the application choices a user configures.

update-alternatives is the program which manipulates the symlinks in the /etc/alternatives directory. Running sudo update-alternatives --config editor will give you a menu from which you can choose your favorite editor.

Running Out Of inode Space

Unix systems have two types of storage limitations. The first, and more common, is a limitation on physical storage used for storing the contents of files. The second is a limitation on inode space which represents file location and other data.

Though it is uncommon, it is possible to run out of inode space before running out of disk space (run df and df -i to see the levels of each). When this happens, the system will complain that there is No space left on device. Both inode space and disk space are needed to create a new file.

How can this happen? If lots of directories with lots of empty, small, or duplicate files are being created, then the inode space can be used up disproportionately to the amount of respective disk space. You’ll need to clean up some of those files before you can continue.

Sources: this and this

Uncomplicated Fire Wall

Lets say you have a malicious program on your server submitting http posts over and over over. What’s important is shutting down those requests before the server provider shuts them down for you. An uncomplicated way to do that is with ucf uncomplicated fire wall.

First enable ssh so that you don’t lock yourself out.

ufw allow ssh

Then deny all other outgoing traffic

ufw default deny outgoing

Check out the rules with status

ufw status

And then enable the fire wall.

ufw enable 

If you’ve ever dealt with iptables, this might seem a little less…. complicated.

Check The Syntax Of nginx Files

The syntax of nginx configuration files can be a bit finicky. On top of that, some nginx server commands can fail silently. Get more confidence by using the following command to check for syntax errors in those files:

$ [sudo] nginx -t

If there is an error, you might see something like this:

$ sudo nginx -t
nginx: [emerg] unexpected ";" in /etc/nginx/nginx.conf:16
nginx: configuration file /etc/nginx/nginx.conf test failed

If all looks good, then you’ll see this:

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Use rsync with a custom port

You can rsync over a custom port for cases when you have custom ssh tunnels setup using rsync’s -e flag. The -e flag tells rsync to use a specific shell, in this case it’s still ssh, but with the addition of the -p flag.

Setup a custom tunnel

ssh -N -L 2222:10.0.0.2:22 user@remote-machine

Use rsync over the custom tunnel

rsync -avz -e 'ssh -p 2222' localhost:/files /myfiles

Be careful when editing the sudoers file

This website says: Because improper syntax in the sudoers file can leave you with a system where it is impossible to obtain elevated privileges, it is important to use the visudo command to edit the file.

RIGHT. If you screw up the syntax you may remove access to the system. visudo checks the syntax for you and won’t let you save a file with incorrect syntax. It responds to the EDITOR env variable so you can edit it in an editor of your choice.

Another command that helps you edit a resource with safety restrictions is vipw.

Limit SSH by IP address

SSH has some nice security features like password-less authentication. For the sysadmin who wants even more fine-grained control, the ssh configuration has a setting for what users are allowed to authenticate based on their IP address.

# /etc/ssh/sshd_config
AllowUsers admin@127.0.0.1 deployer@127.0.0.1 git@*

This configuration says that admin & deployer are only allowed to login from 127.0.0.1, but git can login from any IP address. You can also use wildcards the other way:

AllowUsers *@127.0.0.1

The most useful thing here is the ability to use * to match users and hosts.

h/t Chris Erin

Reload The nginx Configuration

If you’ve modified or replaced the configuration file, nginx will not immediately start using the updated nginx configuration. Once a restart or reload signal is received by nginx, it will apply the changes. The reload signal

$ service nginx reload

tells nginx to reload the configuration. It will check the validity of the configuration file and then spawn new worker processes for the latest configuration. It then sends requests to shut down the old worker processes. This means that during a reload nginx is always up and processing requests.

source

Check The Status Of All Services

In a Linux environment, you can quickly check the status of a number of different services. By running [sudo] service --status-all, the status command will be invoked for all services under the /etc/init.d/ directory.

So, if you want to check the status of something like nginx or apache, just run service --status-all and find it in the list. The - symbol means it isn’t running, the + symbol means it is up, and the ? symbol means that it cannot determine the status.

How busy are your cores?

Linux talks about core utilization in terms of “Load Average”. This is, the average load over 1 minute or over 5 minutes or over 15 minutes.

For a 1 core machine a Load Average of 1.00 is 100%. For an 8 core machine 8.00 is 100%. On my 4 core mac I can check my Load Average with uptime

chriserin@:~% uptime
20:48  up 17 days,  1:42, 7 users, load averages: 1.23 1.53 1.59

Over the last 1 minute, 1.23. Over the last 5 minutes, 1.53. Over the last 15 minutes, 1.59.

My CPU is doing great! You can also see the same numbers in htop. Source

Wipe A Heroku Postgres Database

If you have some sort of non-production version of an application running on Heroku, you may encounter a time when you need to wipe your database and start fresh. For a rails project, it may be tempting to do heroku run rake db:drop db:setup. Heroku doesn’t want you to accidentally do anything stupid, so it prevents you from running rake db:drop. Instead, you must send a more explicit command

$ heroku pg:reset DATABASE_URL

Heroku will ask you to confirm that you indeed want to wipe out the database and will then proceed.

For the curious reader, running heroku config will list the values of a number of variables including DATABASE_URL.

Push Non-master Branch To Heroku

When using git to deploy your app to Heroku, it is expected that you push to the master branch. When you run the following command

$ git push heroku master

Heroku will attempt to build and run your app. However, if you have a staging branch for your application that you want to push to your staging environment on Heroku, you cannot simply run

$ git push heroku staging

Heroku will only perform a build on pushes to the remote master branch. You can get around this, though, by specifying that your staging branch should be pushed to the remote master branch, like so

$ git push heroku staging:master

source