Checking Your Reddit Karma with Common Lisp

Curious to learn more about programming in Common Lisp I decided to write a quick little reddit karma checker. Having just recently learned of quicklisp (a script that greatly simplifies using 3rd party libraries with Lisp), I was ready to wing it and see what I could come up with.  I’m going to share with you the process I went through go from just a vague idea of what I wanted to do to the complete Lisp script.

The Reddit API

The first step is to check out reddit’s API to see if and how I can use to to get my current karma. Looking this over I noticed under the section about “Fetching Information” it should be a simple matter of requesting a specially crafted URL to get the info I was looking for.

My first attempt looked something like this:

[zach@server]$ curl http://www.reddit.com/user/cliffwarden/about.json

{"kind": "t2", "data": {"has_mail": null, "name": "cliffwarden", "created": 1151777163.0, "created_utc": 1151777163.0, "link_karma": 294, "comment_karma": 330, "is_gold": false, "is_mod": false, "id": "9440", "has_mod_mail": null}}

[zach@server]$

Quicklisp

I knew that creating http connections to obtain json and then parsing it was a bit out of my abilities (for now) I started looking to see if any of this was built into the flavor of Lisp I was currently using (sbcl) or if there was a third party library somewhere.

After some looking I came across Quicklisp which is Lisp’s answer to Perl’s CPAN or Python’s “pypi” or Ruby’s “gems”.  This is seriously cool stuff!  After taking approximately output 10 seconds to install quicklisp I was able to find two packages that would help me out using the (ql:system-apropos _string_)

My session went something like this:

		sbcl --load quicklisp.lisp

This is SBCL 1.0.45.0.debian, an implementation of ANSI Common Lisp.
More information about SBCL is available at .

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.

  ==== quicklisp quickstart loaded ====

    To continue, evaluate: (quicklisp-quickstart:install)

* (quicklisp-quickstart:install)
; Fetching #
; 159.59KB
==================================================
163,424 bytes in 1.34 seconds (119.19KB/sec)
; Fetching #
; 180.00KB
==================================================
184,320 bytes in 1.65 seconds (108.96KB/sec)
; Fetching #
; 4.88KB
==================================================
4,995 bytes in 0.001 seconds (4877.93KB/sec)
; Fetching #
; 0.40KB
==================================================
408 bytes in 0.003 seconds (132.81KB/sec)

  ==== quicklisp installed ====

    To load a system, use: (ql:quickload "system-name")

    To find systems, use: (ql:system-apropos "term")

    To load Quicklisp every time you start Lisp, use: (ql:add-to-init-file)

    For more information, see http://www.quicklisp.org/beta/

NIL
* (quit)

Install Quicklisp itself

To make sure Quicklisp is loaded every time you run Lisp (under the current user) use the (ql:add-to-init-file) to add the appropriate code to your init file.

Drakma

Basically what I need is a Curl or Wget for Lisp. After some looking around Drakma looked like it would fit the bill - more power than I needed initially, but certainly usable.

So the first order of business is getting it installed. Since we have quicklisp loaded it is a simple matter of finding the appropriate package name:

SBCL  Port: 35999  Pid: 6430
; SLIME 2010-07-21
CL-USER> (quicklisp:system-apropos "drakma")
#<SYSTEM drakma / drakma-1.2.3 / quicklisp 2011-06-19>
CL-USER>

Yep, just as we suspected, it’s called “drakma”

and install it

(quicklisp-quickload "drakma")

starting the library install

The really cool thing about this is you just use these “quickload” statements at the top of your lisp programs - if it is already downloaded it will load the library, if it needs the code it will automatically download it and continue with the program when its done!

So let’s experiment with Drakma. I’m a big fan of exploratory programming so I usually “learn by tab completion”, meaning i just hit tab and see what is available. I know I’m probably looking for a “get” or “request”. “http-request” that is probably what I’m looking for. And the beauty of the Slime interface comes to the surface. Simply by typing (drakma:http-request **** I see the arguments that this function takes in emacs status line. URL, this seems pretty straight-forward.

CL-USER> (drakma:http-request "http://reddit.com/user/cliffwarden/about.json")
#(123 34 107 105 110 100 34 58 32 34 116 50 34 44 32 34 100 97 116 97 34 58 32
  123 34 104 97 115 95 109 97 105 108 34 58 32 110 117 108 108 44 32 34 110 97
  109 101 34 58 32 34 99 108 105 102 102 119 97 114 100 101 110 34 44 32 34 99
  114 101 97 116 101 100 34 58 32 49 49 53 49 55 55 55 49 54 51 46 48 44 32 34
  99 114 101 97 116 101 100 95 117 116 99 34 58 32 49 49 53 49 55 55 55 49 54
  51 46 48 44 32 34 108 105 110 107 95 107 97 114 109 97 34 58 32 50 57 52 44
  32 34 99 111 109 109 101 110 116 95 107 97 114 109 97 34 58 32 51 51 48 44 32
  34 105 115 95 103 111 108 100 34 58 32 102 97 108 115 101 44 32 34 105 115 95
  109 111 100 34 58 32 102 97 108 115 101 44 32 34 105 100 34 58 32 34 57 52 52
  48 34 44 32 34 104 97 115 95 109 111 100 95 109 97 105 108 34 58 32 110 117
  108 108 125 125)
200
((:CONTENT-TYPE . "application/json; charset=UTF-8")
 (:ACCESS-CONTROL . "allow ")
 (:SET-COOKIE
  . "reddit_first=%7B%22firsttime%22%3A%20%22first%22%7D; Domain=reddit.com; expires=Thu, 31 Dec 2037 23:59:59 GMT; Path=/")
 (:SERVER . "'; DROP TABLE servertypes; --")
 (:DATE . "Sun, 10 Jul 2011 17:05:35 GMT") (:CONTENT-LENGTH . "231")
 (:EXPIRES . "Sun, 10 Jul 2011 19:05:35 GMT") (:CONNECTION . "close"))
#
#
T
"OK"
CL-USER>

Well it did *something*

Drakma certainly made an http connection and got something but this isn’t the JSON we were looking for. What’s the deal?

After some search It looks like this is drakma’s expected response when dealing with a content type it isn’t aware of. So well fix this up by telling it JSON is “just text” and try this again.

; SLIME 2010-07-21
CL-USER> (ql:quickload "drakma")
To load "drakma":
  Load 1 ASDF system:
    drakma
; Loading "drakma"
............
("drakma")
CL-USER> (setq drakma:*text-content-types* (cons '("application" . "json")
					drakma:*text-content-types*))

(("application" . "json") ("text"))
CL-USER> (drakma:http-request "http://www.reddit.com/user/cliffwarden/about.json")
"{\"kind\": \"t2\", \"data\": {\"has_mail\": null, \"name\": \"cliffwarden\", \"created\": 1151777163.0, \"created_utc\": 1151777163.0, \"link_karma\": 294, \"comment_karma\": 330, \"is_gold\": false, \"is_mod\": false, \"id\": \"9440\", \"has_mod_mail\": null}}"
200
((:CONTENT-TYPE . "application/json; charset=UTF-8")
 (:ACCESS-CONTROL . "allow ")
 (:SET-COOKIE
  . "reddit_first=%7B%22firsttime%22%3A%20%22first%22%7D; Domain=reddit.com; expires=Thu, 31 Dec 2037 23:59:59 GMT; Path=/")
 (:SERVER . "'; DROP TABLE servertypes; --")
 (:DATE . "Sun, 10 Jul 2011 17:18:42 GMT") (:CONTENT-LENGTH . "231")
 (:EXPIRES . "Sun, 10 Jul 2011 19:18:42 GMT") (:CONNECTION . "close"))
#
#
T
"OK"
CL-USER>

Nearly there

We are nearly there, but there is still some unnecessary output that we don’t really need. Another quick check of the drakma docs reveals that we just need to turn off this setting to turn off header debugging.

Let’s see what our code looks like so far:

Gist at: https://gist.github.com/zpeters/1074729

cl-json

Now we have the JSON response from reddit in a nice little string, what are we going to do with it?

Looking through quicklisp again I found a very promising library with a pretty predictable name “cl-json”. This library will take our string representation of json data and decode it into Lisp objects.

This library is so simple to use (as far as decoding goes) you could almost guess the syntax. I fiddled around in Slime repl a bit and here is what I came up with:

CL-USER> (ql:quickload "cl-json")
[install messages]

CL-USER> (setq drakma:*header-stream* nil)
(setq drakma:*text-content-types* (cons '("application" . "json")
					drakma:*text-content-types*))

(("application" . "json") ("text"))

CL-USER> (setq out (drakma:http-request "http://www.reddit.com/user/cliffwarden/about.json"))

CL-USER> (json:decode-json-from-string out)
((:KIND . "t2")
 (:DATA (:HAS--MAIL) (:NAME . "cliffwarden") (:CREATED . 1.1517772e9)
  (:CREATED--UTC . 1.1517772e9) (:LINK--KARMA . 294) (:COMMENT--KARMA . 330)
  (:IS--GOLD) (:IS--MOD) (:ID . "9440") (:HAS--MOD--MAIL)))

From here it’s pretty much downhill. We are going to pass this object off to a few functions to extract the data we are interested in and wrap it all up into a nice format statement to print it out.

Finished code below:

Gist at: https://gist.github.com/zpeters/1073810

Hopefully you’ve seen through the course of this article it’s pretty easy with the Lisp environment (Emacs, Slime and Quicklisp) to go from small “sketches” and playing around in the repl prompt to incorporating these into a formal program. This is a very powerful platform for learning and exploration.

Update: I’ve fixed the script up a bit.  The use of assoc lists and pushing (rather than the ugly _setq) _was pointed out in the comments

Gist at: https://gist.github.com/zpeters/1079632