Programming Clojure Chapter 04

I got done with chapter 4 of Programming Clojure. This chapter was about functional programming.

Interestingly, they did not spend much time in this chapter on the concept of higher-order functions. They did use them, but it was not a major part of their definition of “functional programming”. They talked more about purity, referential transparency, immutability, persistent data structures, recursion, and laziness and eagerness (for both collections/sequences and transformations).

I have noticed that when Lisp/Scheme programmers talk about “functional programming”, they tend to talk about functions: higher-order, purity, immutability, laziness. When Haskell or Scala programmers talk about “functional programming”, they talk about types as much as they talk about functions.

The section on lazy sequences blew my mind. They worked with Fibonacci numbers, and I made a lazy sequence of factorials. It is amazing that you can calculate some very large numbers very quickly without blowing the stack.

In case you were wondering, here is 200 factorial (split across a few lines so there is no horizontal scroll):


That’s 7.88657 times 10 to the 374th. My answer matches what is on Calculator Soup.

Here is my sequence:

(defn lazy-seq-fact
   (concat [1N 1N 2N] (lazy-seq-fact 3N 2N)))
  ([a b]                                                                                                     
   (let [n (* a b)]
     (lazy-seq (cons n (lazy-seq-fact (inc a) n))))))

(nth (lazy-seq-fact) 200)

I am not quite sure I understand transducers or eduction as well as I would like. (I think they may be changing the meanings of the words “eduction” and “transducer” slightly.) I think one of the ideas behind transducers is that a transducer can be more efficient and do things in fewer steps. A threading macro with multiple functions will make a sequence or collection at each step. A transducer does not. They had a couple of examples that they converted from using the threading macro to using transducers and “into”. Only the first step in ->> (the starting collection, where it comes from) and the last step in ->> (the final collection, where to put it, which was a call to “vec”) are flipped when we go to “into”; the transformation steps are pretty much the same. In their example, they combined the steps into a function with “comp”. Also: transducers go right-to-left, which is unintuitive for me.

I think you use “eduction” when you want to transduce part of a collection and not have dangling resources.

I did not understand the threading macro until I had a need for it. Perhaps transducers and eduction will be the same.

I did find a way to use the partition function to work with moving averages. You need the members of your collection to be in order from most recent to oldest. The you can get a collection of moving averages with this:

(map #(/ (reduce + %1) 
         (count %1)) 
     (partition 4 

Chapter 5 is about spec. Clojure seems to have a lot of concepts that are new to me, even more than the little Scheme I have worked with.

You’re welcome.

Image from the Liuthar Gospels, a tenth century manuscript created in southern Germany, currently housed in the cathedral of Aachen, assumed allowed under public domain. I have read there are 30 full-page miniatures in the manuscript, but I have not been able to find a pdf or any other pictures than the one on this page.

Clojure Functions Dealing With Sequences

I am finally through Chapter 3 of Programming Clojure. That chapter deals with sequences and collections (list, map, vector, set, and the Clojure “seq” abstraction), functions common to many types of sequences, and functions that are specific to specific collection types.

I was going to add some of the samples that I went through, but I cannot figure out how to get decent looking Clojure code in WordPress anymore. I do not like the Gutenberg editor at all, and it does now work as well with the classic editor as it used to. I found one called “Enlighter” which sounded good, but it does not do Clojure or any kind of Lisp. Given that WordPress really wants people to use the Gutenberg editor, at some point I might look for something other than WordPress.

The notes are on github (link here). I will update my page listing the functions in the Clojure API (some are in the Set API); I will just point to this page, and if you want to see the examples, then go to the page on Github.

Here are the functions discussed in the notes:

  • assoc
  • cons
  • dissoc
  • drop-while
  • even?
  • every?
  • file-seq (not in notes, in a Clojure file on github
  • filter
  • first
  • for
  • get
  • hash-set
  • interleave
  • line-seq
  • list
  • map
  • next
  • peek
  • pop
  • rest
  • select-keys
  • seq
  • set
  • some
  • split-at
  • split-with
  • subvec
  • take-while
  • vector
  • set/difference
  • set/intersection
  • set/join
  • set/project
  • set/select
  • set/union

Going through this book is taking a while (I admit I don’t do it every day). But I have tried learning languages from koans and puzzles, and I just wind up trying random functions from API and not being very productive.

You’re welcome.

Image from “Grec 74”, a 12th century manuscript housed at the Bibliothèque nationale de France. Source / BnF; image assumed allowed under Public Domain.

Climbing Mount SICP

I have decided to focus more on Clojure, and start working on How To Design Programs (aka “HtDP”), and then onto Structure and Interpretation of Computer Programs (aka “SICP”). HtDP is written by the founders of the Racket project, and SICP uses Scheme (one of the authors, Gerald Sussman, invented Scheme along with Guy L. Steele; I think Scheme was Steele’s PhD thesis).

I looked one last time into The Reasoned Schemer. I changed to a different KanRen library. More of the examples worked, but this book is really not coming together for me as quickly as the other ones did. I think I will put aside the “Little” books, at least for the time being. I think it might be time to get ready to climb Mount SICP.

Many people have said that SICP is a good way to “level up” (as the gamer kids put it). Many people have said it gave them something like the famous “Lisp enlightenment”, that it altered their perception of how to make software and made them better. As the Teach Yourself CS site put it: Because SICP is unique in its ability—at least potentially—to alter your fundamental beliefs about computers and programming. Not everybody will experience this. Some will hate the book, others won’t get past the first few pages. But the potential reward makes it worth trying.

SICP is an introductory CS text used at MIT. The general consensus is that you will get more out of it if you have been in software development for a few years. It is a bit much for most people right off the bat. I have been through Simply Scheme, which states in the last chapter that it is intended to prepare you for SICP. HtDP was made as an alternative to SICP; the authors felt that SICP requires a lot of higher math, and if they made a less math-heavy alternative they could introduce more people to CS and help them become better programmers.

One of the authors of Simply Scheme wrote an essay entitled “Why Structure and Interpretation of Computer Programs matters”. He states that unlike other introductory textbooks, SICP does not spend a lot of time on the syntax of a specific language, but instead teaches broader concepts. Someone at the local Clojure meetup once said to me that if you can work through the problems in SICP, “you can solve any problem with parentheses”. In other words, you learn how to design programs.

A few years back, MIT (and a few other schools using SICP) replaced Scheme and SICP with other texts that use Python. Sussman said there were a few reasons for this. One is that he and Harold Abelson (one of the other co-authors of SICP) were tired of teaching the course after more than 15 or so years. Another reason is that SICP was designed for programmers who would make systems from the ground of using smaller components. Now a lot of programming is just linking or wiring together components that are black boxes. Sussman called it “programming by poking”. You could contrast this with the SICP style of “analysis-by-synthesis”. The site Lambda The Ultimate has a post with a link to a now-extinct blog post, but there is a quote that summarizes the ideas nicely. There is also a link to a thread on Hacker News. This have come up several times on Hacker News and Reddit.

There is a section in the preface to Simply Scheme that covers this:

There are two schools of thought about teaching computer science. We might caricature the two views this way:

  • The conservative view: Computer programs have become too large and complex to encompass in a human mind. Therefore, the job of computer science education is to teach people how to discipline their work in such a way that 500 mediocre programmers can join together and produce a program that correctly meets its specification.
  • The radical view: Computer programs have become too large and complex to encompass in a human mind. Therefore, the job of computer science education is to teach people how to expand their minds so that the programs can fit, by learning to think in a vocabulary of larger, more powerful, more flexible ideas than the obvious ones. Each unit of programming thought must have a big payoff in the capabilities of the program.

(I think you could also call these the “business mindset”, and the “engineering mindset”: Do things in a way that the suits think they understand, or actually knowing what you are doing.)

Per a comment on the Lambda The Ultimate post, one of the authors of HtDP said that other departments sent students to their CS intro course to learn how to think. That is where I want to go. I feel that “programming by poking” is what I have been doing. I feel like I should be better at this than I am, and that there has to be a better way of doing things. Where “progress” isn’t just learning yet another vendor’s product.

Why not level up? Why settle for poor understanding? As one of the commenters on Hacker News put it: What about the people who make the black boxes? How do you know they know what they are doing? If you could learn to solve any problem with parentheses, why wouldn’t you?

The obligatory links (I have not gone through most of these):

You’re welcome.

Image from “Evangelia quattuor, pars”, an 11th century manuscript housed at the Bibliothèque nationale de France. Source / BnF; image assumed allowed under Public Domain.

2021-04-11 Update

I know I said I was going to focus on Clojure, but I looked at some other things in the past week.

I am looking into some Amazon certification for AWS. There is a big push to get certified at my current employer, and I would like to do something different.

I started the org-mode tutorial from Udemy.

I also looked at The Seasoned Schemer. I think that having already gone through Simply Scheme I may have gotten some of the concepts. A lot of it deals with recursion.

Chapter 12 dealt with “letrec”. letrec is like let, but instead of binding a name to a variable or data, you can bind a name to a function, and you can call that function recursively.

One thing this can allow you to do is it can relieve you of the need to call a helper function. In Simply Scheme, I made all the functions tail-recursive. To do this, you usually need a parameter that is the “accumulator” or the output (see this answer on Stack Overflow or my post here).

In The Seasoned Schemer, they use letrec for functions in which one argument does not change and one does, such as checking if an atom is a member of a list, or performing a union of two lists:

(define (union-letrec s1 s2)
      ([U (lambda (set-1)
            (cond [(null? set-1) s2]
                  [(ss-sc:member? (car set-1) s2) (U (cdr set-1))]
                  [else (cons (car set-1) (U (cdr set-1)))]))])
    (U s1)))

I know I think the whole argument about Lisp/Scheme being hard to read because of all the parentheses is usually exaggerated, here I think for these letrec examples there might be something to it. These were hard to type correctly from the book. For example, the internal function has to be after the “(set-1)” which is the argument to the “lambda”, but before the closing parens for “lambda”. And the call to the internal function has to be before the closing call to “letrec”. You can have multiple functions defined in a “letrec”, and your call to the internal functions can be in a cond.

A helper function to do the actual work with tail-recursion might be more lines of code, but I think it is easier to understand.

Chapter 13 is about let/cc, combining let with “call-with-current-continuation”. I think some people consider this Scheme’s defining feature.

They have a function rember-upto-last which takes an atom and a list of atoms and removes everything in the list before the last instance of the atom as well as the atom.

Here it is:

(define (rember-upto-last a lat)
  (let/cc skip
          ([R (lambda (lat)
                (cond [(null? lat) '()]
                      [(ss-sc:equal5? (car lat) a) 
                       (skip (R (cdr lat)))
                       ; (R (cdr lat))
                      [else (cons (car lat) (R (cdr lat)))]))])
          (R lat))))

Here is a tail-recursive version:

(define (rember-upto-last-r a lat)
  (let/cc skip
          ([R (lambda (lat outp)
                (ss-sc:display-all "Calling R w/lat: " lat 
                                   " and outp: " outp)
                (cond [(null? lat) (reverse outp)]
                      [(ss-sc:equal5? (car lat) a) 
                         (skip (R (cdr lat) '()))
                         ;(R (cdr lat) '())
                      [else (R (cdr lat) (cons (car lat) outp))]))])
        (R lat '()))))

Both of these have a commented out call to “(R (cdr lat)” (with a second arg in the tail-recursive version) below the call to “skip”.[Note 1]. In the tail-recursive version, either  call gives the proper result. In the regular version, we are building our answer through a chain of calls to “cons”. The “skip” eliminates all the pending “cons” calls, and the function is called anew with whatever the then-current values are. In the tail-recursive version, we can get rid of those “pending” values ourselves.

I think Java might get continuations soon.

You’re welcome.

[Note 1]: For some reason, the code formatter I am using does not always color comments correctly. In all variants of Lisp that I have seen, comments start with a semi-colon.

Image from Psalterium Gallicanum with Cantica, a 9th century manuscript from the monastery of St. Gall, housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.

2021-03-28 Update

There is not a whole lot to report this month. I have not had a lot of time to look at Clojure.

You’re welcome.

Image from Collected Fragments Volume II from the Abbey Library of St. Gall, written around the 8th century from the monastery of St. Gall. Image from e-Codices. This image is assumed to be allowed under Fair Use.

Every and Any in Clojure

Today we look at a few functions in Clojure that take predicates and look at collections: any?, every?, not-every? and not-any?

any? seems to return true for any argument. According to the docs, it is used for clojure.spec. Maybe when I learn more about spec, that will make sense. You would think a function called “any?” would be useful if you had a collection of numbers, and wanted to know if any of them are odd, or greater than 10, or something like that. Perhaps they could have given it a better name than “any?”

every?, not-any? and not-every? behave as you would expect them to. You can always simulate “intuituve-any?” by calling “not-any?” and then calling “not”.

user> (def evens [2 4 6 8])
user> (def evens-with-9 [2 3 4 6 8 9])
user> (every? even? evens)
user> (every? even? evens-with-9)
user> (not-every? even? evens)
user> (not-every? even? evens-with-9)
user> (not-every? odd? evens)
user> (not-every? odd? evens-with-9)
user> (not-any? odd? evens)
user> (not-any? odd? evens-with-9)
user> (any? 'g)
user> (any? nil)
user> (not (not-any? odd? evens))
user> (not (not-any? odd? evens-with-9))

You’re welcome.

Image from the Rheinau Psalter, a 13th century manuscript housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.

Reduce In Clojure

The last of the Big Three Higher Order Functions in Clojure is reduce. In other languages it is called fold, accumulate, aggregate, compress or inject.

Sometimes reduce is compared to aggregate functions in databases, like average, count, sum, min or max. These functions take multiple values, and produce one value. Some functions in Clojure, like the four main math functions (+, -, * and /) are processed through reduce if they are called with more than two values.

I would like to point out that unlike the other aggregation functions that databases have, “average” cannot be done with reduce. To do an average, you sum the amounts, and then divide by the number of elements. No implementation of reduce (or fold or whatever you call it) keeps track of the number of elements and allows for an additional operation at the end. Maybe some languages or implementations have a reduce and an enhanced-reduce to cover this case.

The first parameter to Clojure’s reduce is a function that takes two parameters. There is an optional parameter that is an initial value, and the last argument is the collection you want to process. Reduce takes either the option initial value and the first element of the collection, or the first two elements of the collection, and passes them to the function. The output of the function is then used along with the next element in the collection. Reduce keeps using the output from the previous element and the next element as arguments to the function until the collection is exhausted.

I cannot find it now, but I remember reading somewhere that reduce can be hard to understand because “reduce” is frankly not a good name for it, and there are many functions that do what reduce does. As stated, some of the functions in Clojure (like max, min, and some math functions) call reduce. So you are actually using reduce a lot under the covers.

There is a cartoon about this. A teacher gives a student a few chances to say what is “2 + 3” equal to. The student says it equal to “2 + 3” and it is also equal to “3 + 2”. The student gets an F. The cartoon ends with: “Sad fact: many math teachers do not know the difference between equality and reduction.” I admit I do not either. A lot of the cartoons are about Haskell, which I have no desire to learn.

;; call + on a bunch of numbers
user> (+ 1 2 3 4 5 6)
;; winning with "the duce"
user> (reduce + [1 2 3 4 5 6])
;; using 0 for initial value
user> (reduce + 0 [1 2 3 4 5 6])
;; use 0 for +, 1 for *, or you will get incorrect results
user> (reduce + 1 [1 2 3 4 5 6])
;; define a function to explain reduce w/print statements
user> (defn add-with-print [x y]          
  (println "x is " x ", y is " y  ", (+ x y) is " (+ x y))
  (+ x y))
;; use w/just collection
user> (reduce add-with-print [1 2 3 4 5 6])
x is  1 , y is  2 , (+ x y) is  3           
x is  3 , y is  3 , (+ x y) is  6           
x is  6 , y is  4 , (+ x y) is  10          
x is  10 , y is  5 , (+ x y) is  15         
x is  15 , y is  6 , (+ x y) is  21         
;; use w/0 as initial value
user> (reduce add-with-print 0 [1 2 3 4 5 6])
x is  0 , y is  1 , (+ x y) is  1           
x is  1 , y is  2 , (+ x y) is  3           
x is  3 , y is  3 , (+ x y) is  6           
x is  6 , y is  4 , (+ x y) is  10          
x is  10 , y is  5 , (+ x y) is  15         
x is  15 , y is  6 , (+ x y) is  21         
;; use with 1 as initial value
user> (reduce add-with-print 1 [1 2 3 4 5 6])
x is  1 , y is  1 , (+ x y) is  2           
x is  2 , y is  2 , (+ x y) is  4           
x is  4 , y is  3 , (+ x y) is  7           
x is  7 , y is  4 , (+ x y) is  11          
x is  11 , y is  5 , (+ x y) is  16         
x is  16 , y is  6 , (+ x y) is  22         

You’re welcome.

Update 2022-02-10_22:33:14: I just read a post by Rich Hickey about transducers in which he talks a bit about reduce. The function you send to reduce takes two arguments: the result-so-far and the next item in the collection, and returns the next result-so-far.

Image from  Aurora Consurgens, a 15th century manuscript housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.

Focusing On Clojure

I have decided to focus on Clojure going forward.

I think of all the Lisps, Clojure right now has the most momentum. And I got a couple of inquiries about Clojure positions.

I have been going through the Clojure API, inspired by an article called “One Weird Trick To Become a Clojure Expert“. The article advocates going through the API alphabetically. Maybe I should have done that, but I think I might be better off going through a few books.

I know at some point I have to stop learning and start making something. But I think that going through the API might not be the best way. And the two are not mutually exclusive.

One of the books I will go through is the third edition of Programming Clojure. One of the co-authors is Alex Miller, who helps make Clojure at Cognitect. Clojure does things differently than other languages, and if anyone can explain it, it is the people at Cognitect. The same publishing company has a book called Getting Clojure, which is written by Russ Olsen, who also works at Cognitect.

Reading books by Cognitect employees is one way you can get Clojure.

You’re welcome.

Image from Ms DF III 3, a 10th century manuscript housed in the Strahov Monastery in the Czech Republic; image from Wikimedia, assumed allowed under Public Domain.

The Map Function In Clojure

The next member of the Big Three Higher Order Functions is “map“.

I don’t think “map” is a good name for it because 1. There is a data structure named “map” (which has different names in different languages) and 2. I don’t think it is the most descriptive name for it. “apply-to-every” or “apply-to-all” might be better.

The “map” function takes a function and one or more collections, and applies the function to each member in the collection. In Clojure, if you send multiple collections, they will be concatenated into one output collection.

I think the reason it is called “map” is that you are using a function to map a collection onto another collection.

Clojure also has “mapv” which does the same thing as “map” except that it returns a vector, and “mapcat” which takes functions that take collections as arguments. The functions sent to map and mapv take single arguments (if you send one collection) or multiple args, but not collections.

; a function that prints out what it's doing to better understand "map"
user> (defn inc-p [x]
  (println "x is " x " (inc x) is " (inc x))
  (inc x))

user> (map inc-p [1 2 3 4])
x is  1  (inc x) is  2
x is  2  (inc x) is  3
x is  3  (inc x) is  4
x is  4  (inc x) is  5
(2 3 4 5)
user> (mapv inc-p [1 2 3 4])
x is  1  (inc x) is  2
x is  2  (inc x) is  3
x is  3  (inc x) is  4
x is  4  (inc x) is  5
[2 3 4 5]

;; if you send multiple collections, your function has to be able to take multiple args
user> (map inc-p [1 2 3 4] [2 3 4 5])
Error printing return value (ArityException) at clojure.lang.AFn/throwArity (
Wrong number of args (2) passed to: user/inc-p
user> (map + [1 2 3 4] [2 3 4 5])
(3 5 7 9)
user> (mapc + [1 2 3 4] [2 3 4 5])
Syntax error compiling at (on-demand-repl:localhost:42447(clj)*:112:7).
Unable to resolve symbol: mapc in this context
user> (mapv + [1 2 3 4] [2 3 4 5])
[3 5 7 9]
user> (mapv + [1 2 3 4] [2 3 4 ])
[3 5 7]

user> (mapcat rest [1 2 3 4])
Execution error (IllegalArgumentException) at user/eval5576 (form-init2164788925227159491.clj:74).
Don't know how to create ISeq from: java.lang.Long
user> (mapcat rest [[1 2 3 4]])
(2 3 4)
user> (mapcat rest [[1 2 3 4] [2 3 4 5]])
(2 3 4 3 4 5)

You’re welcome.

Maps In Clojure

I was planning on looking into Clojure’s map function, the second of the Big Three Higher-Order Functions. But looking over my Clojure API page, I noticed there were a few functions with the word “map” in it that I had not gone over yet that deal with the map data structure.

Just as “map” is a noun and a verb in Clojure, the map data structure goes by different names in different programming languages: associative array, map, dictionary, symbol table, hash, hash table.

Anyway, I have looked at some of these functions earlier, but here are some more functions dealing with maps:

user> (sorted-map :a "value a" :b "value b" :c 33)
{:a "value a", :b "value b", :c 33}
user> (sorted-map :a "value a" :c "value c" :b 33)
{:a "value a", :b 33, :c "value c"}
user> (hash-map :a "value a" :c "value c" :b 33)
{:c "value c", :b 33, :a "value a"}
user> (array-map :a "value a" :c "value c" :b 33)
{:a "value a", :c "value c", :b 33}
user> (array-map :a "value a" :c "value c" :b 33 :c "more c")
{:a "value a", :c "more c", :b 33}
; honestly, I am not clear what he order is
user> (reverse (array-map :a "value a" :c "value c" :b 33 :c "more c"))
([:b 33] [:c "more c"] [:a "value a"])
user> (first (array-map :a "value a" :c "value c" :b 33 :c "more c"))
[:a "value a"]
user> (first (reverse (array-map :a "value a" :c "value c" :b 33 :c "more c")))
[:b 33]

user> (hash-map :a "value a" :d "what is d?" :c "value c" :b 33)
{:c "value c", :b 33, :d "what is d?", :a "value a"}
user> (hash-map :a "value a" :c "value c" :b 33 :b "again")
{:c "value c", :b "again", :a "value a"}
user> (hash-map :a "value a" :c "value c" :b 33 :b "again" :b "Third b")
{:c "value c", :b "Third b", :a "value a"}
user> (sorted-map-by > :a "value a" :c "value c" :b 33)
Execution error (ClassCastException) at user/eval5630 (form-init12885185727845486110.clj:148).                                                                           
class clojure.lang.Keyword cannot be cast to class java.lang.Number (clojure.lang.Keyword is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')   
user> (sorted-map-by compare :a "value a" :c "value c" :b 33)
{:a "value a", :b 33, :c "value c"}
user> (sorted-map-by compare :a "value a" :c "value c" :b 33 :c "more c")
{:a "value a", :b 33, :c "more c"}

user> (reverse (sorted-map-by compare :a "value a" :c "value c" :b 33))
([:c "value c"] [:b 33] [:a "value a"])
user> (sorted-map-by < 2 "value a" 33 "value c" 4 "v4")
{2 "value a", 4 "v4", 33 "value c"}
user> (sorted-map-by > 2 "value a" 33 "value c" 4 "v4")
{33 "value c", 4 "v4", 2 "value a"}
user> (sorted-map-by > 2 "value a" 33 "value c" 4 "v4" 33 "more 33")
{33 "more 33", 4 "v4", 2 "value a"}

user> (def empty-hm (hash-map))
user> (def empty-am (array-map))
user> (def empty-sm (sorted-map))
; zipmap will create a map from a collection of keys and a collection of values
user> (zipmap [:a :b :c] [1 2 3])
{:a 1, :b 2, :c 3}
user> (zipmap  '(:a :b :c) '(1 2 3))
{:a 1, :b 2, :c 3}
; does not work with sets
user> (zipmap  #{:a :b :c} #{1 2 3})
{:c 1, :b 3, :a 2}
user> (map? (zipmap  '(:a :b :c) '(1 2 3)))
user> (map? (zipmap  #{:a :b :c} #{1 2 3}))
user> (map? #{1 2 3})

You’re welcome.