Testing Groovy Actors

One thing I did not quite get when I was looking at multithreading technologies was how to unit test them.

I have not looked at Akka for a while, but a few years ago it seemed like the consensus was to use a logger like Log4J, and then parse the logging messages. That is kind of unwieldy and feels wrong. (Akka seems to have more ways to test now.)

I emailed the GPars list about putting private fields in an actor, changing them in the onMessage method, and then checking them afterwards; something like this:

class BadActor extends DynamicDispatchActor {
    private def someField
    private def someOtherField
    
    void onMessage( ObjectType message ) {
        someField = message.doStuff()
        someOtherField = message.doOtherStuff()
    }
}

You could do that. There is nothing wrong with having private fields in an actor. You might want to keep some data between requests (like an encryption key). But making a field just for testing seems wrong.

Then I had an idea: Have your actor call another class that does the actual work, and unit test the worker class:

class BetterActor extends DynamicDispatchActor {
    
    void onMessage( BetterObjectType message ) {
        def worker = new WorkerClass( message )
        def result = worker.doWork()
    }
}

I suppose that will not help with functional testing. It might violate some SOLID principle, or DRY, or GRASP or the Law of Demeter or Principle of Least Astonishment or Poe’s Law, but it seems like a pretty good idea to me.

You’re welcome.

Image from the “Imperial Menologion“, an 11th-century Byzantine manuscript, not to be confused with the Menologion of Basil II. Manuscript  housed at The Walters Art Museum, manuscript information here, image from World Document Library, image assumed allowed under Fair Use.

Groovy Concurrency With Closure Locks

I have played around a bit more with closures and locks in Groovy.

In one of Venkat S’s talks, he says that the java.util.concurrent.locks package solves a lot of the issues with the synchronized keyword: You have to have try/catch/finally blocks to get the lock and release it. I made a class that will hold a Lock object, and can take code as a closure and run it. It seems to work pretty well. At least I think so.

But I realized there is one possible issue: The ClosureLock class only imports from java.util.concurrent.locks. What if some of the code in the closure is from another package that is not imported in the ClosureLock class?

I made a class that imports groovy.time.Duration, and creates a couple of objects and invokes methods in a closure. And it ran without blowing up. I also made a class called DebugClosureLock that prints out some stuff to the console checking if the lock is held by the current thread.

Here is the DebugClosureLock class:

package info.shelfunit.concurrency.locks

import java.util.concurrent.locks.Lock
import java.util.concurrent.locks.ReentrantLock

class DebugClosureLock { 
    final Lock lock = new ReentrantLock()
    
    // prevent lock from being changed
    def setLock( arg ) {}
    
    def lockSomeCode( block ) {
        
        try { 
            println "Is lock ${lock.hashCode()} held? ${lock.isHeldByCurrentThread()}"
            println( "about to try to get the lock" )
            if ( lock.tryLock() || lock.tryLock( 500, TimeUnit.MILLISECONDS ) ) {
                println " About to call block. Is lock ${lock.hashCode()} held? ${lock.isHeldByCurrentThread()}"
                block.call()
            }
        } finally {
            println( "About to try to unlock" )
            lock.unlock()
        }
        println "Done with code. Is lock ${lock.hashCode()} held? ${lock.isHeldByCurrentThread()}"
    }
}

Here is the class that uses it:

package info.shelfunit.concurrency.locks

import groovy.time.Duration 

class SecondLockRunner { 

  def doStuff() { 
    DebugClosureLock cLock = new DebugClosureLock()
    def dur = new Duration(  30, // days 
        2, // hours 
        45, // minutes 
        32, // seconds 
        123 // millis 
    )
    println( "about to call dur.toString() " )
    cLock.lockSomeCode( { println "${dur.toString()}" } )
    def dur2 = new Duration(  3, // days 
        20, // hours 
        43, // minutes 
        31, // seconds 
        522 // millis 
    )
    println( "\nabout to call dur2.toString() " )
    cLock.lockSomeCode { 
        println "${dur2.toString()}"
        println "dur2 as long: ${dur2.toMilliseconds()}"
    }
    
  }

  static void main( String[] args ) { 
    SecondLockRunner slr = new SecondLockRunner()
    slr.doStuff()
  }
} // end class

So, to review: My question was would DebugClosureLock blow up when it was called since it does not import groovy.time.Duration? And the answer is it did not.

Here is the console output:

about to call dur.toString() 
Is lock 1740808647 held? false
about to try to get the lock
 About to call block. Is lock 1740808647 held? true
30 days, 2 hours, 45 minutes, 32.123 seconds
About to try to unlock
Done with code. Is lock 1740808647 held? false

about to call dur2.toString() 
Is lock 1740808647 held? false
about to try to get the lock
 About to call block. Is lock 1740808647 held? true
3 days, 20 hours, 43 minutes, 31.522 seconds
dur2 as long: 333811522
About to try to unlock
Done with code. Is lock 1740808647 held? false

Paul King shows how to do this on slide 36 of his talk Groovy and Concurrency using metaprogramming:

import java.util.concurrent.locks.ReentrantLock

ReentrantLock.metaClass.withLock = { critical ->
    lock()
    try { critical() }
    finally { unlock() }
}

Then you can use it like this:

ReentranLock lock = new ReentrantLock()
lock.withLock{ someCode() }

You’re welcome.

Image from “Evangeliarium [Evangéliaire dit de Charlemagne ou de Godescalc]”, an 8th century manuscript housed at the Bibliothèque nationale de France. Source gallica.bnf.fr / BnF; image assumed allowed under Fair Use. It also has a Wikipedia page.

Some Groovy Threading

There is an example of using threads with Groovy in Venkat’s book.

He uses the GDK extension to Thread and calls Thread.start. I wondered if the main thread would wait until the started thread completed, and what would happen if you started another thread.

I made some changes, and ran it.

Here is my class:

package info.shelfunit.venkat.ch07

class BetterThreadExample {
    
    def printThreadInfo( msg ) {
        println "$msg" 
    }
    
    def doStuff() {
        sleep( 5000 )
        printThreadInfo 'Main'
        println "Main is ${Thread.currentThread().name}"
        Thread.start {
            def name = Thread.currentThread().name
            printThreadInfo "AA Started: ${name}"
            sleep( 3000 ) 
            println "AA Interrupted 1" 
            sleep( 1000 )
            println "AA IN started"
            sleep( 3000 )
            println "AA still in started"
            println "AA Finished Started"
        }
        
        Thread.start( 'Second' ) {
            def name = Thread.currentThread().name
            printThreadInfo "BB Started Daemon: ${name}"
            sleep( 5000 ) 
            println "BB Interrupted 2" 
            sleep( 1000 )
            println "BB in second thread"
            sleep( 1000 )
            println "BB Finished Started Daemon"
        }
        println "At the end in main"
    }
      static void main( String[] args ) { 
          def bte = new BetterThreadExample()
          bte.doStuff()
      } // end method main

}

Here is the result:

Main
Main is main
At the end in main
AA Started: Thread-1
BB Started Daemon: Second
AA Interrupted 1
AA IN started
BB Interrupted 2
BB in second thread
BB Finished Started Daemon
AA still in started
AA Finished Started

 

So the main thread gets to its last statement pretty quickly. The two threads run simultaneously. I already knew this, but it’s good to review threads every now and then.

I was always intimidated by threading and concurrent  programming. Maybe it’s not too bad in Groovy.

You’re welcome.

Image from Aurora Consurgens, a 15th century manuscript housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.

Changing Immutable Objects

Recently someone at work someone sent me a link to an article saying that getters and setters are evil by a guy named Yego256. He has a few other interesting posts about immutable objects here and here. They can be added to the list I made of pages about immutable objects here.

Hopefully Grails 3 will utilize immutability.

It takes a shift in mindset to start using immutable objects. Plus, in Java (and Groovy) a lot of web frameworks don’t utilize them, so people are not too familiar with them.

One issue some developers have is: Many things in the world are mutable. How can you make “changes” to immutable objects?

One answer is that you can use the values you don’t want to change to populate a new immutable object, and your variable/field name points to a new immutable object.

(If you can think of a better way to explain this, let me know. It’s kind of confusing to say that an immutable object is referenced by a “variable name”.)

I will demonstrate this with Groovy (using the annotations EqualsAndHashCode, Immutable and ToString). I made a class to model a state in the USA. States do not change their names, abbreviations or year they joined the union. But many of them have changed their capital cities. So we will make an object, and change its capital city while keeping everything else the same.

Here is our class:

package info.shelfunit.states

import groovy.transform.EqualsAndHashCode
import groovy.transform.Immutable 
import groovy.transform.ToString

@Immutable
@ToString( includeNames = true )
@EqualsAndHashCode
class USState {
    String name
    String abbrev
    String capital
    int yearJoined
}

Now let’s look at using it in a Groovy shell. The first capital of Illinois was a town called Kaskaskia, which no longer exists. Then it moved to Vandalia (which does still exist) and finally to Springfield.

I will try to change a few fields to prove the objects are immutable, and I will call hashCode() to show that we are dealing with different objects.

groovy:000> import info.shelfunit.states.*
===> [import info.shelfunit.states.*]
groovy:000> il = new USState( name: 'Illinois', abbrev: 'IL', capital: 'Kaskaskia', yearJoined: 1818)
===> info.shelfunit.states.USState(name:Illinois, abbrev:IL, capital:Kaskaskia, yearJoined:1818)
groovy:000> il.name
===> Illinois
groovy:000> il.name = "Indiana"
ERROR groovy.lang.ReadOnlyPropertyException:
Cannot set readonly property: name for class: info.shelfunit.states.USState
        at info.shelfunit.states.USState.setProperty (USState.groovy)
        at groovysh_evaluate.run (groovysh_evaluate:3)
        ...
groovy:000> il.toString()
===> info.shelfunit.states.USState(name:Illinois, abbrev:IL, capital:Kaskaskia, yearJoined:1818)
groovy:000> il.hashCode()
===> 1673606286
groovy:000> il = new USState( name: il.name, abbrev: il.abbrev, capital: 'Vandalia', yearJoined: il.yearJoined )
===> info.shelfunit.states.USState(name:Illinois, abbrev:IL, capital:Vandalia, yearJoined:1818)
groovy:000> il.abbrev
===> IL
groovy:000> il.abbrev = "WI"
ERROR groovy.lang.ReadOnlyPropertyException:
Cannot set readonly property: abbrev for class: info.shelfunit.states.USState
        at info.shelfunit.states.USState.setProperty (USState.groovy)
        at groovysh_evaluate.run (groovysh_evaluate:3)
        ...
groovy:000> il.toString()
===> info.shelfunit.states.USState(name:Illinois, abbrev:IL, capital:Vandalia, yearJoined:1818)
groovy:000> il.hashCode()
===> -934597459
groovy:000> il = new USState( name: il.name, abbrev: il.abbrev, capital: 'Springfield', yearJoined: il.yearJoined )
===> info.shelfunit.states.USState(name:Illinois, abbrev:IL, capital:Springfield, yearJoined:1818)
groovy:000> il.capital
===> Springfield
groovy:000> il.capital = "Homerville"
ERROR groovy.lang.ReadOnlyPropertyException:
Cannot set readonly property: capital for class: info.shelfunit.states.USState
        at info.shelfunit.states.USState.setProperty (USState.groovy)
        at groovysh_evaluate.run (groovysh_evaluate:3)
        ...
groovy:000> il.toString()
===> info.shelfunit.states.USState(name:Illinois, abbrev:IL, capital:Springfield, yearJoined:1818)
groovy:000> il.hashCode()
===> 213225124
groovy:000>

So we used the existing values to make new objects. The JVM used the same variable name (“il”) to refer to different immutable objects.

Here is the code without the output:

import info.shelfunit.states.*
il = new USState( name: 'Illinois', abbrev: 'IL', capital: 'Kaskaskia', yearJoined: 1818)
il.name // returns "Illinois"
il.name = "Indiana" // will cause groovy.lang.ReadOnlyPropertyException
il.toString()
il.hashCode()

// Change capital to Vandalia
il = new USState( name: il.name, abbrev: il.abbrev, capital: 'Vandalia', yearJoined: il.yearJoined )
il.abbrev // returns "IL"
il.abbrev = "WI" // causes groovy.lang.ReadOnlyPropertyException:
il.toString()
il.hashCode()

// Change capital to Springfield
il = new USState( name: il.name, abbrev: il.abbrev, capital: 'Springfield', yearJoined: il.yearJoined )
il.capital // returns "Springfield"
il.capital = "Homerville" // causes groovy.lang.ReadOnlyPropertyException:
il.toString()
il.hashCode()

2015-06-06_14.23.56 update: I now realize that in order to prove the objects were truly different objects, I should have called System.identityHashCode(il) instead.

You’re welcome.

Image from the Rheinau Psalter, a 13th century manuscript housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.

4Clojure

I started working on the 4Clojure problems. Functional is certainly a different way of doing things.

I follow a couple of people. Sometimes I will solve a problem with a complex solution: if, loop, recur, let, etc, etc. And then I find out one of the people I follow solved it with one line. Then sometimes the opposite happens.

I am also looking at Luminus web framework. It would be nice if the Clojure community would get behind a web framework, or if there was a default choice for beginners/people new to the language. Any time this topic comes up, there is always a lecture on how Clojure/Lisp people like to put together libraries instead of use frameworks. Ruby has other frameworks besides Rails, and plenty of libraries at various levels of abstraction. But if you ask how to make a web app in Ruby, people will point you towards tutorials, not start preaching. Sometimes I wonder if Clojure will be the Lisp that takes off, or if it will just be another Lisp that never gained escape velocity.

Image from  Aurora Consurgens, a 15th century manuscript housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.

 

Locks And Closures In Groovy

In a previous post, I wrote about using lambdas in Java to simplify the use of locks. I was able to get this up and running in Groovy.

I have the code for it at Github. I made a class called ClosureLock which has a method called “lockSomeCode” which takes a closure as the argument. It creates an instance of ReentrantLock, and calls the closure in a try/finally block. Now that I look at it, i suppose I should refactor it so the ReentrantLock is a private variable and can be re-used, since ClosureLock can be reused.

I also created a class called FirstLockRunner, which instantiates ClosureLock and uses it a few times.

After I figured this out, I did some googling, and a few other people have done this before. A guy named David Clark has a class called DemoLockEnhancements in a project called GroovyThreading.

A guy named Johannes Link also did something similar.

There are a couple of new Groovy books out. I think I will look them over in order to understand closures/lambdas better. I have been reading that lambdas can be used to improve concurrency in Java and are thread safe.

I know that the consensus is to move away from locks and instead use other concurrency techniques. But I think that if locks can be done cleanly and safely, then their use should be considered.

Image from Rudolf von Ems’ Chronicle of the World, a 14th century manuscript housed at Central Library of Zurich (Wikipedia page here). image from e-Codices, assumed to be allowed under Fair Use.

Done With Venkat S’s Book

I am finally all through Programming Concurrency on the JVM by Venkat Subramaniam.

I am looking at some code he wrote to test classes in the java.util.concurrent.lock package. I might go through that.

I have a feeling that a lot of what I went through might be obsolete soon. There was a lot of code that could be done more concisely with lambdas, which are coming soon in Java 8. I think you might be able to do extend the ReadWriteLock class and have it take some code that you want to make thread-safe. In one of the times I saw him, he said that locks take care of any issues you would have with the synchronized keyword.

The problem is that you really need to use try/catch/finally blocks, and you should use the tryLock() method, and use a timeout. It can be a lot of code.  You might want to change one variable, but instead of having one line in your setter, you might need about 6. But if you could send that one line to a LockLambda (or whatever you want to call it), it would be better. You might need two, which granted is still more than one, but the LockLambda could handle all the try/catch/finally/get the lock/set the timeout/release the lock work for you, and you might be able to re-use the same LockLambda class.

I might blog about Akka, although I am not too thrilled with it. I do not like Scala, and some of the Akka APIs call Scala APIs. I liked GPars better. Although there were some things in GPars that are not worth the effort to use in Java, you can pretty much use just the GPars APIs. You do not need to look at any Groovy if you do not want to. I think some things I did not like with Akka will be easier once lambdas come out. But I still do not like Scala.

I may still look at the Play Framework.

 

Image from an 11th century manuscript made at the Abbey Library of Saint Gall  (Wikipedia page here), image from e-Codices, assumed allowed under Fair Use.

 

2013-07-14 Concurrency Update

I have been going through Venkat’s book, but I think I might bail on going any further.

The book came out in 2011, and there have been a LOT of changes to Akka in that time. They stopped using Multiverse for STM, and started using ScalaSTM. There were a lot of changes from version 1 to version 2 (starting actor systems, creating actors, sending messages, pretty much everything). There was a change in the API as I was going through the book.

Some of the method calls require inline Runnables or Callables, and that gets kind of awkward. I think I will move on to other things and come back when there is another edition of the book. Or I might wait until Java 8 comes out. I think the closures will make things easier.

Plus, since Akka is written in Scala, a lot of things are unwieldy in Java. I really do not like Scala. It has the syntactical elegance of Perl, with the simplicity and focus of C++. It is like religion or cigarettes: Its advocates tell me that if I gave it a chance I would see how wonderful it is. I did, and it’s not.

That said, I have no issue with the idea of using something written in Scala. I might take a look at the Play Framework. That is not as odd as it seems. I do not like working with C code. But a LOT of important stuff is written in C: The JVM, MRI, databases, the Linux kernel. If I refused to work with anything written in C, I would need to find a new line of work.

One language I think I will take a closer look at is Clojure. At first I was skeptical about it: It’s just another Lisp, and Lisp has never caught on, so why bother? But I have kept an eye on it, and I think this might be The Lisp That Sticks.

Plus, I have noticed something interesting about Lisp. A lot of people will start out developing in one language, and then they will become dissatisfied, and move to another. And then another. And then another. But it seems like when people find Lisp, they stop moving around. They seem to find what they are looking for.

In the next few months there are some Clojure meetups that will go over some Clojure web frameworks. I might look at them before that.

When Ken Kousen was here for the Groovy meetup, he said that he had a boss who kept saying, “Oh, we could do that in Lisp 20 years ago.” He thought it was really annoying. He said it was more annoying when he realized his boss was right.

Image from the Rheinau Psalter, a 13th century manuscript housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.

Concurrency Without Pain in Pure Java

Here are the notes I took when Venkat Subramaniam was here for Austin JUG. I will mostly present them as I typed them.

Why Concurrency? Why do we care?

We now have mutli-core processors. Things that work well on one core may not work well on multiple cores.

The Java memory model controls how info goes back and forth from main memory and working memory.

His arguments in his classes are all final.

In the account class. Should the getBalance() be synchronized?

“synchronized” gets a lock and crosses the memory barrier, working and main memory.

Accessing a volatile variable, wait, join, a few other calls where you cross the memory barrier. We could compromise the invariant, like during a constructor.

So if you do not make getBalance synchronized, you might get an old value.

The code could have deadlock, or livelock (you wait, but nothing waits for you).

When you call synchronized, you might wait forever. There is no time out. So you use the Lock interface.

So for ReentrantLock you could call tryLock, or throw the Exception, then you have to unlock it.

You can order the accounts by ID to removed threat of dead lock.

95% of Java code is broken for concurrency. (Is he including code for Groovy, Scala, Clojure?)

Synchronize and suffer model.

Solutions: Software Transactional Memory (from Clojure) and Actors (Scala)

STM: We modify a variable (data) in our code. Shared mutability is bad. A few solutions: Avoid mutability. STM says let’s have a special type of variable: Managed mutable variable. It will only change in certain circumstances: a transaction.

(def balance (ref 0))

(println @balance) ; dereference the reference

Identity: Values are immutable. The stock price at 1:00 is different than at 1:10. But the 1:00 price will never change.

The identity is “google stock price”, points to a different value at different times.

We must protect the identity.

The problem in Java is that many times the problems happen without warning or error message. In STM, you get an error message.

(dosync(ref-set balance 100))

This changes it in a transaction. If there are conflicts, the transaction is aborted and retried. Up to 100,000 times.

Can we do it in Java? He used ref and dosync, so can we use them in Java?

clojure.lang.Ref. Look at his code, see what class he uses for Ref.

private Ref balance; // this was an int

The set method will use Lambda expression in Java 8. For now, we could do a Callable. LockingTransaction.runInTransaction

He is using an inline Callable.call()

What if you try to put more in the to account than is the from amount? It will throw an exception.

When to use STM? When you have multiple readers, multiple writers, and infrequent write collision. Not too many things are good for high-write collision.

You should not modify unmanaged variables in a transaction. Do not write to a file, print out messages, do not send email.

What about a long-running task

Another solution to shared mutability: isolated mutability.

Problem with encapsulation: Many threads can call the methods that change your private variables.

It is like voice mail: Many people can leave him a message, he gets a lot of messages, he goes through them one at a time. Actors have built-in message queue. Asynchronous messages.

ActorSystem.create();

actorSystem.actorOf(new Props(HollywoodActor.class));

pitt.tell("Wonka", null);

actorSystem.shutdown();

The Actor runs on a separate thread.

When do actors make sense?

final String PATH = "/usr/bin";

 private static int countFiles(final String path) {

    int count;
    File file = new File(path);
       if (file.listFiles() != null)
}

ActorSystem actorSystem = ActorSystem.create();
MasterCollection master = actorSystem.actorOf(Prop(MasterCollector.class))
master.tell()

The fields are immutable. The messages are not, so they should be immutable.

public void onReceive(final Object message) {}

You can have variables in an Actor, since only one thread will touch an Actor.

When does Actor make sense: Single threaded tasks sequentially

It punishes readers as much as writers. In STM, readers are never blocked. Writers are punished.

Purpose of Actor is not to deal with shared mutability, you want to prevent concurrent access, you want to serialize.

New concurrency API in Java 8.

You need Java 8 with Lambdas.

Look at java.util.streams.*

public static void printHighestPrice( Stream< String > symbols ) {

    symbols
        .map( StockUtil::getPrice ) // look that up
        .filter( stockInfo -> StockUtil.isPriceLessThat( 500 ) )
        .reduce( new StockInfo( "", 0.0 ), StockUtil::pickHigh )

}

http://download.java.net/lambda/b78/docs/api/java/util/stream/package-summary.html
:: is a method reference

map(StockUtil::getPrice)

can be done as

map(stock -> StockUtil.getPrice(stock))

Java is getting more functional

Synchronize and shared mutability can be avoided

JDK 5 concurrency is the assembly code of concurrency

Image from the Rheinau Psalter, a 13th century manuscript housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.

 

Still Going Through Venkat’s Book

I am still going through Venkat S’s book.

I had gotten through the chapter on STM. A lot of the examples for STM dealt with some old versions of APIs. The first time I just read through them pretty slowly. I was able to follow. But then I did a bit more googling, and I was able to find later versions of the STM libraries he was using.

I had already started the chapter on actors. I decided to go back and work on the STM examples. It is taking a bit of time.

Image from  Aurora Consurgens, a 15th century manuscript housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.