Clojure Talks To Watch

I often come across videos on software engineering and Clojure that are recommended to me by colleagues.

Here are some:

Rich Hickey – Simple Made Easy

For those of you who don’t know, Rich Hickey is the creator of Clojure. This is probably his most famous talk. Definitely a must watch.

Stuart Sierra – Components Just Enough Structure

If you’re building lots of services or a web application, this is the talk for you. Stuart Sierra is the creator of the component library that allows you to keep the state of your application in components. 

David Nolen – Om Next

David Nolen, the creator of Om, a wrapper library for Facebook’s React.js, talks about what’s coming next. If you’re a user of Om or React.js, or interested at all in the topic, watch this.

A Tutorial Of Prismatic’s Schema Library For Clojure

I haven’t had many good ideas for blog posts, but I’ve gotten this bright idea to do tutorials/overviews of the libraries I use in Clojure.

The first one is Prismatic’s Schema.

Schema is an awesome library once you understand it.

Let’s go over some of the more important concepts via the schema readme.

The first thing that jumps out when you first see Schema is that it allows you to validate functions and data. You can think of it as optional type checking in a way.

(def Data
  "A schema for a nested data type"
  {:a {:b s/Str
       :c s/Int}
   :d [{:e s/Keyword
        :f [s/Num]}]})

The above Data schema validates a map. It says the key :a should be a map with keys :b as a string and :c as an integer. Additionally, key :d should have a seq of maps with :e as a keyword and :f as a seq of numbers.

(s/validate
  Data
  {:a {:b "abc"
       :c 123}
   :d [{:e :bc
        :f [12.2 13 100]}
       {:e :bc
        :f [-1]}]})
;; Success!

(s/validate
  Data
  {:a {:b 123
       :c "ABC"}})
;; Exception -- Value does not match schema:
;;  {:a {:b (not (instance? java.lang.String 123)),
;;       :c (not (integer? "ABC"))},
;;   :d missing-required-key}

Once you’ve defined a schema, you can validate it, by calling the validate function in the schema.core namespace.

If it’s a success, you’ll get your data structure back. If not, you get an exception.

The idea is that you can generate schemas to make sure the ‘shape’ of your data is the way you wanted. For instance, if you wanted to make sure your web server’s requests parameters had only these keys of these types, you can do that.

And there are lots of ways to create schemas. You can create schemas that validate on a predicate, meaning you can just pass in a function that returns a boolean and have it validate:

(require '[schema.core :as s])

(defn divisible-by-two? 
  [x]
  (= (mod x 2) 0))

(def DivisibleByTwo (s/pred divisible-by-two? 'divisible-by-two?))

(s/validate DivisibleByTwo 2)

;; => 2

The second most important concept to nail down is coercers.

Coercion took me a long time to grasp, but once you have it, you’ll be amazed at how modular and reusable the code becomes.

Prismatic talks a lot of custom walkers for coercions, but I’ve found in practice that’s not really necessary most of the time. It’s there for more advanced users. For almost every use case, the default walker is fine. I would ignore that part for now, otherwise it will confused you like me.

Essentially what coercion means is you’re able to take a type and transform it into the type you want.

Let’s be more concrete. I won’t use the sample from the readme because I feel like doesn’t explain it well enough to beginners.

Say we had these schemas:

(require '[schema.core :as s]
         '[schema.coerce :as coerce])

(def RequestCommentId s/Str)

(s/defschema RequestParams
  {;; Path param /comments/:comment-id -> /comments/4
   :comment-id RequestCommentId
   ;; Form params
   :comment-body s/Str
  })

(s/defschema DatabaseParams
  {:id s/Int
   :body s/Str})

The RequestCommentId is our representation of a string. You’ll see why we need this wrapper in a moment, but it’s also important that we have some semantics to our representation, other than strings, integers, etc.

RequestParams are the parameters that come in from a web request. For example, in this case, let’s say we have a simple blog API. Somebody sends a PUT request to update the blog post with the id they specify to “/comments/:comment-id”. If you’re using compojure, that :comment-id is going to be a string and you would need to coerce to an integer.

Then we have our DatabaseParams. In our little scenario, we’re getting an updated blog post from our API and coercing it to something we can insert into our database. In this case, that means renaming the keys from :comment-id/:comment-body to just :id and :body. Additionally, it means we want an integer for our :id.

Here’s the coercion part:

(defn RequestParams->DatabaseParams
  [{:keys [comment-id comment-body] :as request-params}]
  {:id comment-id
   :body comment-body})

(def RequestParams->DatabaseParams-coercer
  (coerce/coercer DatabaseParams {DatabaseParams RequestParams->DatabaseParams
                                  s/Int #(Integer/parseInt %)}))

(RequestParams->DatabaseParams-coercer {:comment-id "123"
                                        :comment-body "Hello World!"})

;; => {:body "Hello World!", :id 123}

I start by defining a function that renames the keys. This function affects the entire request-params map. In my experience, it’s much better to put separate out the functions that go into the coercion map, as they’re easily re-useable elsewhere.

Then I define the coercer.

The first parameter to coerce/coercer is the schema I’m coercing into. The second is a map that tells the coercer how to do its job.

The map’s keys are the types the schema wants, and its values are the functions that do the coercion.

Recall we made a schema for s/Str called RequestCommentId. We did that for two reasons:

1. We want our own type for it instead of just using s/Str.

2. The body is also an s/Str. There’s no way for schema to know how to differentiate between an s/Str that’s a comment body and an s/Str that’s a comment id. Since we called it RequestCommentId instead, we can have two different schema types that at the lower level mean String.

Back to the coercion map.

DatabaseParams is what we want and a key in the map. This means that when we want DatabaseParams, we get the entire parameter passed to the coercion function, in this case, RequestParams. Think about it as a way to grab the entire thing you’re coercion. It’s useful so you can massage the keys of the map into the shape you want before the types get coerced.

Finally, the other key, s/Int says, whenever an s/Int is specified in our schema map, apply this function to the value in that map. Recall the DatabaseParams map has an :id key with the value s/Int. Since we’ve renamed the key from :comment-id to :id, the coercion walker sees that :id should be an s/Int, looks into the coercion map for the function to do it, and returns the result under the key :id.

That’s it! An ultra quick overview of Schema, one of the libraries that we use a lot at RentPath. We’re hiring senior clojure developers by the way (100% remote), send me an email if you’re interested and are a U.S. citizen.

It might’ve been a little too quick so if there are any questions/confusions, please leave a comment.

 

A Neat Pattern With Juxt

I recently added the function juxt to my toolbox in Clojure.

A good pattern I’ve discovered is using juxt to perform operations on a data structure and destructuring its return value.

(defn read-files
  [^String path]
  (let [data-files (find-data-files path)
        [prices receipts] ((juxt find-prices-file find-receipt-files) data-files)]
    {:prices (read-prices prices)
     :receipts (apply merge (read-receipts receipts))}))

Setting Up A Node.cljs Project With Leiningen and NPM

Before we begin, make sure you can start a node.cljs repl first

One important thing I want to highlight is that you should have a stable version of node 0.12.X as the linked wiki page says. Additionally, you want to make sure you have npm installed so you can install node packages later on. Finally, I’m also assuming that you already have Clojure and Clojurescript setup with Leiningen.

Let’s begin.

Start by making a new leiningen project. 

lein new node_project

And then move into that directory.

Start by editing your project.clj file to look something like this:

(defproject node_test "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.7.0-RC1"]
                 [org.clojure/clojurescript "0.0-3269"]
                 [org.clojure/core.async "0.1.346.0-17112a-alpha"]]
  :profiles {:dev
             {:dependencies [[com.cemerick/piggieback "0.2.1"]
                             [org.clojure/tools.nrepl "0.2.10"]]
              :plugins [[lein-cljsbuild "1.0.5"]]
              :source-paths ["src" "dev"]
              :repl-options {:nrepl-middleware [cemerick.piggieback/wrap-cljs-repl]}}}
  :clean-targets
  [[:cljsbuild :builds 0 :compiler :output-to]
   :target-path
   :compile-path]
  :cljsbuild {:builds
              [{:id "dev"
                :source-paths ["src"]
                :compiler {:output-dir "out"
                           :output-to "index.js"
                           :optimizations :none
                           :source-map "src-maps"
                           :target :nodejs}}]})

It’s very important that you use versions of Clojure and Clojurescript greater than or equal to the ones I’ve listed. I’ve ran into a lot of issues trying a setup like this before finding one that worked. It was mostly due to old versions of piggieback and Clojurescript.

I added the piggieback related dependencies in dev for the node.js REPL we’re going to be using later. The clean-targets are for the command 

lein clean

Because cljsbuild deprecated its clean function.

A few more things to note:

We’re using no optimizations and targeting node.js. Having optimizations set to :simple, I ran into issues with core.async macros. 

Next, we need to setup the node side of the project. Inside the same project folder, run:

npm init

You can pretty much press enter for every single prompt, except the entry point one. There you want it to be named app.js, not index.js as we specified in the project.clj. We can’t run index.js directly due to optimizations being set to none, so cljsbuild will output multiple files.

Next, you can install a web framework using npm like express:

npm install --save express

Now that we have Express.js installed, we can just make a simple hello world web application in clojurescript.

Rename src/node_test/core.clj to src/node_test/core.cljs.

Then in the file put this:

(ns node-test.core
  (:require [cljs.nodejs :as node]))

(def express (node/require "express"))

(def app (new express))

(defn -main
  []
  (.listen app
           3000
           (fn []
             (js/console.log "App Started at http://localhost:3000"))))

(set! *main-cli-fn* -main)

Then you want to compile your clojurescript to javascript with:

lein cljsbuild auto

Now that we have our index.js file, we need to make the file for node to run the application.

Make a file called app.js in the root of your project folder with this as its contents:

// Bootstrap node.js
require('./out/goog/bootstrap/nodejs');

// Our app compiled by cljsbuild
require('./index.js');

// The core of our code
require('./out/node_test/core');

// The core of cljs
require('./out/cljs/core');

// Run main
// NOTE: Dashes in namespaces are replaced by underscores.
node_test.core._main();

If you’ve changed anything, replace the paths with the correct one.

Now, you can start the app with node.js:

$ node app.js
App Started at http://localhost:3000

And if you visit that url, you’ll receive an error message about not being able to GET /. That’s fine, we didn’t specify any routes.

If you liked this post and want to learn more about Node.cljs, sign up for my mailing list to get my latest posts in your inbox. Next post will be on how to develop this project shell further by adding in a cljs repl.

I’m also available for hire as a Clojure freelancer.

Becoming A Finisher

I’ve always known that I’m a starter and not much of a finisher. I’ll start a project and quickly get excited about a new idea and switch to the next shiny thing.

But I didn’t realize the extent of my problem until I had a conversation with some of my friends while playing Dota. My friends were talking about video games that they’ve completed, and that made me realize. I can’t even finish video games.

The games I’ve played are all competitive, like Dota or Hearthstone. No story to them. But even the competitive games with stories, like Starcraft 2, I can’t finish. Even single player games I can’t get halfway through. 

I started doing research on the subject by searching Hacker News, as it seems like a common problem and found a thread where Paul Graham gives some advice:

Taking on projects that are too big creates a situation like an airplane stalling because the angle of attack is too steep. Better only to take on projects so small that you know you can finish them, however humiliatingly small that may be initially. Then, with always finishing as a given, start to gradually crank up the ambition. But never so sharply that you stall.

Paul Graham

It may seem silly, but it looks like finishing is something you have to learn and practice. It’s a skill.

I’m going to take Paul Graham’s advice. 

Two things I’m going to start doing:

  • Using my Apple Watch as a pomodoro timer. Essentially the idea is you have periods of working and periods of relaxing. Usually it’s 25 minutes of work and 5 minutes of taking a break.
  • Breaking tasks up by single pomodoros to begin with. That’s humiliatingly small. 

Goal for tomorrow: Write the first post on Node.cljs in one pomodoro.

Stay tuned.

How To Refer To A Clojure Record In Another Namespace

I ran into an issue like usual. This time it seemed simple. I wanted to refer to a record that I def’d in another namespace.

The problem is, you can’t just do a :require and :refer it in regular Clojure. However, in Clojurescript this is how you would do it.

What you have to do is :import it. And the key trick to keep in mind when using :import on records and Clojure namespaces is that dashes are converted to underscores like packages in Java.

If you had a namespace like (ns foo-bar) in which there was a record called MyRecord, you would do something like this:

(:import [foo_bar MyRecord])

And you’ll be off to the races.

Node.cljs – The Secret Weapon Nobody Knows About

I came across a discussion on the Clojure mailing list.

Summary:

The original poster wants a web framework in Clojure. An actual framework, not a wimpy routing library like Compojure. Something closer to Rails or Django, where you don’t have to make your own framework every single time you start a project.

Of course, all the Clojure purists, hereby known as weenies, responded like so:

No Rage Face

Clojure weenies hate frameworks. They want composable libraries to make their own frameworks. Which sounds like a great idea! How could you argue against composable libraries? Composable! Modular! Sounds like a programmer’s wet dream. Libraries with single responsibilities that we can use anywhere!

The other people, hereby known as people that want to get shit done the other people, love Clojure. They want to use Clojure for everything. They bought into the claims that Clojure gives you a huge productivity boost. That functional programming is the future. But, the issue is they don’t see these benefits when doing web development. They’re used to being able to use Rails or Django to make CRUD applications in no time.


Two sides with reasonable arguments.

For the weenies, you can keep using pure Clojure.

For the other people, I have a solution that not many know about.

Node.cljs

As you might’ve guessed from the title, I think the middle ground and best of both worlds is Node.cljs. That’s node.js, but written with Clojurescript.

Why?

Clojurescript gives you Clojure that compiles to javascript. Which means you can get the entire Clojure language on the node platform. That includes macros, REPL based development, and all the libraries that support Clojurescript like Prismatic’s Schema or Core Async.

On the other end, node’s platform is amazing for web development. You can pick your favorite web framework, that’s right, framework. Like Express.js or Hapi.js. In case you cared about speed, node benchmarks a bit slower than Clojure.

Pros:

  • No JVM
  • Node platform -> web frameworks and libraries
  • Clojurescript libraries
  • Faster startup speed
  • No callback hell -> Use core.async for async operations.
  • Clojure features like integrated REPL, macros, etc.
  • Faster application development speed

Cons:

  • No JVM
  • Cljs tooling is still a little rough. You might run into some issues.
  • You have to know javascript.
  • A lot of code would be javascript interop.
  • A teeny bit complicated to setup a project.
  • Less functional code -> slightly more object oriented
  • Have to deal with clj->js and js->clj data structure conversions
  • Slower (according to Techempower’s benchmarks)

Interested in seeing how to get something like this setup? Stay tuned. Join my mailing list to get my posts automatically in your inbox.

Third Party Libraries In Clojurescript REPL

I ran into an issue at work earlier this week. I needed to include a phone number validation library in the Clojurescript side. It was libphonenumber by Google, which meant it was compiled with the Google closure library.

Great! That gets us halfway there.

That classifies it as a third party library, meaning in lein-cljsbuild you would specify it under the :libs key instead of the :preamble, :externs, or :foreign-libs.

I just want to make it clear that the :foreign-libs option is used for non Google closure code like jquery.

So if you’re using lein-cljsbuild, it’s just as easy as putting the directory path into :libs in a vector. But my problem was that I needed to access the third party library in my Clojurescript REPL while developing.

Under this documentation, it says:

Note: conveying the :libs option to the REPL so that it can find third-party JavaScript libraries has not yet been implemented.

I’ve found that to be untrue for the Rhino and Nashorn case, and I’m not completely sure if it’ll work in every single scenario yet so I’m hesitent to just change the wiki. I plan on doing some more investigation and then updating the wiki so other folks that come across this entry later won’t waste their time looking into the source code like I had to.

An example of how to use this with piggieback as the nrepl middleware:

(require 'cemerick.piggieback 'cljs.repl.rhino)
  (cemerick.piggieback/cljs-repl (cljs.repl.rhino/repl-env)
                                 :libs ["resources/js"])

Leiningen Profile Overrides

When developing applications in Clojure, one thing you might need is to be able to somehow keep different settings for each environment. For instance, you might want to be able to have different database settings for each environment.

This can be achieved with leiningen profiles:

:profiles {:dev-overrides {}
             :dev
             [:dev-overrides
              {:dependencies [[com.cemerick/austin "0.1.5"]
                              [com.cemerick/piggieback "0.1.4-SNAPSHOT"]
                              [ring-mock "0.1.5"]]
               :plugins [[com.keminglabs/cljx "0.5.0"]
                         [com.cemerick/austin "0.1.5"]
                         [lein-cljsbuild "1.0.3"]]}]
             :test-overrides {}
             :test
             [:dev
              :test-overrides]}

Assuming you have that :profiles sub-map somewhere in your project.clj map, it will override what’s in the root of the project map.

For instance:

(defproject ring_to_curl "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [cheshire "5.4.0"]]
  :profiles {:dev
             {:dependencies [[com.cemerick/austin "0.1.5"]
                             [com.cemerick/piggieback "0.1.4-SNAPSHOT"]
                             [ring-mock "0.1.5"]]
              :plugins [[com.keminglabs/cljx "0.5.0"]
                        [com.cemerick/austin "0.1.5"]
                        [lein-cljsbuild "1.0.3"]]
              :test {}}})

Notice how there’s a :dependencies key in the root of the project map and there’s one in the :dev profile.

What this means is whatever is in your currently active profile in your [:profiles :<PROFILE>] will be merged into the root project map.

Therefore, you can merge in dependencies like for say, browser REPLs in your dev profile. Merge in testing related dependencies in your test profile and so forth.


That’s the simple case of leiningen profiles. There’s a lot more you can do in the project.clj file.

Say I was using some library for databases and it was reading the database information from the project.clj map. I could then specify different databases for different environments.

I could have one for development, one for testing that gets rebuilt every time I run tests and so forth.

And this is fine for a single developer team.

What about for teams with multiple people?

This scenario comes up a lot more often then you would think. You could have a coworker that has a mysql database installed locally and another one that has their database installed in a docker container. They would have different connection information, usernames, and passwords.

In this scenario, one solution is to use what I call leiningen overrides.


There exists a file called profiles.clj in your ~/.lein/ directory. You might’ve played with it before if you’re an emacs user to add the cider nrepl middleware to it.

This profile gets loaded for every single leiningen project you work with.

You can override what’s in this folder by going to your project folder, the one that includes project.clj, and create another file named profiles.clj.

That profiles.clj in your project will take precedence over the one in your ~/.lein/ directory.

Also, notice that the map specified in the ~/.lein/profiles.clj file has a :user key (usually). That corresponds to the profile leiningen is using, which defaults to :user.

My ~/.lein/profiles.clj looks like this:

{:user {:plugins [[lein-ancient "0.6.2"]
                  [cider/cider-nrepl "0.9.0-SNAPSHOT"]
                  [lein-create-template "0.1.1"]]}}

Again, just like before, this map gets merged into the project.clj one. The :plugins key isn’t some random key that is specific to the ~/.lein/profiles.clj file. It appears in your project.clj map.

You can add more profiles for each of your environments in your project.clj file or your project’s profiles.clj file.

I make sure every environment my application will run in is defined in the project.clj file and then add overrides in the project’s profiles.clj file.

This is how it’s done:

project.clj

:profiles {:dev-overrides {}
             :dev
             [:dev-overrides
              {:dependencies [[com.cemerick/austin "0.1.5"]
                              [com.cemerick/piggieback "0.1.4-SNAPSHOT"]
                              [ring-mock "0.1.5"]]
               :plugins [[com.keminglabs/cljx "0.5.0"]
                         [com.cemerick/austin "0.1.5"]
                         [lein-cljsbuild "1.0.3"]]}]
             :test-overrides {}
             :test
             [:dev
              :test-overrides]}

There’s a few concepts that need to be explained in this map.

Firstly, by specifying the submap :dev-overrides, which isn’t a special name at all but one I defined myself, I can merge it into other maps in the profiles map.

If you look at the :dev, it’s a vector. The vector has a list of maps that will be merged together. So, by first listing the keyword :dev-overrides, I’m saying, get the value of the keyword :dev-overrides, which in this case is an empty map, and put it here. The next element in the vector is a map, so it merges that with the empty map I had in there originally from :dev-overrides.

With the knowledge that you can merge in maps in the project map, and the knowledge of profiles.clj, you might start to see where I’m going.

Say your project.clj file contains that :profiles map I specified above with the :dev-overrides key. You can now create a profiles.clj file in your project folder like this:

profiles.clj

{:dev-overrides
 {:dependencies [[reagent "0.5.0"]]}}

This is a simple, arbitrary case I came up with where I wanted the dependency reagent to be merged in to my :dev profile locally. Normally you might see things like AWS credentials and so forth, if for instance you’re using weavejester’s env library.

If you’re working with other people, what I would do is make a file called profiles.clj.sample or something and commit that to git. Then when a new developer comes to your project, they can just check it out, rename it to profiles.clj and stick in their own settings to be merged in.

Cider Refresh And Undefining Vars

A neat trick I learned a few months back is that you can reset all of your Clojure namespaces in the repl you’re working on if you’re using Emacs + Cider. Just use M-x cider-refresh.

That command ‘refreshes’ your repl by undefining the vars and reloading them.

This way, if you have run-away code in your repl state, you can refresh it without doing a cider-restart.

This comes in handy especially when you’re writing tests with clojure.test. Let’s say you have a few tests written and delete one. If you evaluate the namespace again, the test you deleted is still defined and will run every time you run the tests in that namespace.

So you have two options. Refresh the namespace or, alternatively, cider-undef the var.