My First Rails Deployment Journey with Kamal

Deploying a Rails application for the first time can feel like navigating uncharted waters. After running my blog successfully with Docker Compose, I thought deploying a Rails app would follow the same pattern. Spoiler alert: it didn't, and I learned some valuable lessons along the way.

The Initial Plan (That Didn't Work)

My original idea was simple: deploy everything to one server. After all, my blog was already running smoothly there with Docker Compose and Caddy handling the routing. How hard could it be to add a Rails app to the mix?

Turns out, pretty hard (for me).

The problem? Kamal-proxy and Caddy don't play nice together. Both want to be the traffic manager, and having two reverse proxies fighting for control is a recipe for confusion. I researched ways to make them coexist—either migrate everything to kamal-proxy or keep Caddy in charge of all services.

But here's the thing: I'm not a DevOps expert, and honestly, I didn't want to become one just to deploy an app. Plus, keeping everything on one server meant upgrading the specs, which would double my hosting costs.

So I took the pragmatic route: spin up a new server with the same specs and deploy the Rails app there. Sometimes the easiest path is the best path.

Setting Up Kamal Deploy Configuration

Here's my working config/deploy.yml file. I've annotated the key parts that tripped me up:

service: <your-application-name>
image: <your-name>/<your-application-name>

ssh:
  user: <your-ssh-user-name>
  
servers:
  web:
    - <server ip address>
  job:
    hosts:
      - <server ip address>
    cmd: bin/jobs

# This is crucial if you're using Cloudflare
proxy:
  ssl: true
  host: <your-host-name>
  forward_headers: true  # Don't forget this!

registry:
  username: <docker-hub-username>
  password:
    - KAMAL_REGISTRY_PASSWORD

env:
  secret:
    - RAILS_MASTER_KEY
  clear:
    HOST: <your-host-name>
    RAILS_SERVE_STATIC_FILES: true
    RAILS_LOG_TO_STDOUT: true
    DB_HOST: <same-as-server-name>-db

aliases:
  console: app exec --interactive --reuse "bin/rails console"
  shell: app exec --interactive --reuse "bash"
  logs: app logs -f
  dbc: app exec --interactive --reuse "bin/rails dbconsole"

asset_path: /rails/public/assets

volumes:
 - "pulse_storage:/rails/storage"

builder:
  arch: amd64

accessories:
  db:
    image: postgres:18
    host: <your-server-ip>
    env:
      clear:
        POSTGRES_USER: <application-name>
        POSTGRES_DB: <application-name>_production
      secret:
        - POSTGRES_PASSWORD
    files:
      - config/init.sql:/docker-entrypoint-initdb.d/setup.sql
    volumes:  # Note: volumes, not directories!
      - <application>_db_data:/var/lib/postgresql/data

The Cloudflare Detail That Matters

Since I use Cloudflare with the orange cloud enabled (Proxied), I needed to add forward_headers: true to my proxy configuration. This ensures that the original client IP and other important headers are passed through Cloudflare's proxy to your app.

proxy:
  ssl: true
  host: example.com
  forward_headers: true

Without this, your Rails app might see Cloudflare's IP instead of your actual users' IPs.

When Things Went Wrong: The Directory vs. Volumes Mystery

After running kamal deploy, I hit a wall. The deployment failed with this cryptic error:

docker: Error response from daemon: failed to create task for container: 
failed to create shim task: OCI runtime create failed: runc create failed: 
unable to start container process: error during container init: 
error mounting "/home/<username>/<application>-db/data" to rootfs at 
"/var/lib/postgresql/data": change mount propagation through procfd: 
open o_path procfd: open /var/lib/docker/overlay2/[...]/merged/var/lib/postgresql/data: 
no such file or directory: unknown

Not exactly beginner-friendly, right?

After some digging through Kamal's documentation, I discovered the distinction between directories and volumes:

Directories

These are created on the host before being mounted into the container:

directories:
  - mysql-logs:/var/log/mysql

Volumes

These are Docker volumes that aren't automatically created or managed by Kamal:

volumes:
  - /path/to/mysql-logs:/var/log/mysql

Most tutorials use directories, but in my case, switching to volumes solved the problem. I'm still not entirely sure why, but it works, and sometimes that's good enough when you're learning.

Database Initialization Made Easy

One neat trick I learned: the init.sql file executes automatically when the PostgreSQL container is created. This is perfect for setting up multiple databases:

CREATE DATABASE <application>_production;
CREATE DATABASE <application>_production_cable;
CREATE DATABASE <application>_production_cache;
CREATE DATABASE <application>_production_queue;

Drop this file in your config folder, reference it in your deploy.yml, and you're set. No manual database creation needed!

Lessons Learned

Looking back, here's what I wish I knew before starting:

1. Don't overcomplicate things. If your existing setup works, sometimes it's better to separate concerns rather than cramming everything onto one server.
2. Read the error messages carefully. That "no such file or directory" error was actually telling me exactly what was wrong—I just needed to understand Kamal's volume system.
3. Documentation is your friend, but so is experimentation. Sometimes the "wrong" approach (using volumes instead of directories) turns out to be right for your specific situation.
4. Cloudflare users: remember forward_headers: true. Save yourself the debugging headache.

What's Next?

Now that my Rails app is deployed and running smoothly, I'm planning to explore:

• Better monitoring and logging strategies
• Understanding why volumes worked better than directories (still curious!)

If you're deploying your first Rails app with Kamal, I hope this helps you avoid some of the pitfalls I encountered. Feel free to reach out if you run into similar issues—sometimes just knowing someone else faced the same problem makes it less daunting.

Happy deploying! 🚀