Java

My CJUG History

by

As part of my site changes, I am getting rid of my pages about CJUG, the Chicago Java Users Group. I will post about my time with CJUG, as well as list some of the technologies/projects/frameworks that were interesting, with an emphasis on the ones that are still maintained and used.

I also had posts from the old CJUG blog. But now many of the links are dead, and many people are doing completely different things, so I will just put it all on disk and send those posts to the void.

This post will give my history with CJUG, and later I will post about the technologies that were presented which look interesting and I think are still maintained. There is also a big chunk about why I do not like Scala; I have been thinking about writing a post about why I do not like Scala, and this felt like a good a time as any.

I lived in Chicago for 14 years. For about 12 of those I was going to CJUG, as an attendee, sometimes as a presenter. For two years I was CJUG president.

The location changed over time. Before I ran things, some were held in the AT & T building (which is now the Franklin Center), the Bank of America building at 231 South LaSalle (which is now the Central Standard Building), and for a time various buildings at Loyola University. Loyola is about seven miles north of the Loop in Rogers Park and Edgewater. Attendance went down when the group moved there.

I am not too clear why the location changed up to Loyola. I know that one of the board members was on the CS faculty at Loyola, and he reserved rooms for us. One difference between Chicago and Austin is that Chicago feels like it has a center in the Loop; not just geographically, but culturally and financially as well. The Capitol is in the center of Austin, yet there is not as much concentration of businesses and jobs in that area as there is in Chicago in the Loop (granted, COVID may have changed things). A lot of people who worked in the Loop live outside the city and take the train to and from work. Going a few miles north and then coming back is just not convenient. (See Note 1 below.)

After a while I started to lose respect for our professor. He said we should drop Java and focus on Scala. I never really liked Scala, either the language or the community. It relies way too much on symbols; it looks like a cat walked across your keyboard. Scala is proof that while most things are more pleasant to do in Java than they are in C, Perl is not one of them. It has the syntactical elegance of Perl, with the simplicity and focus of C++ (that statement is both literal and sarcastic at the same time). I think that like C++, Scala is a multi-paradigm language that most of its users should not be using (or better yet, never invented). The people that seem to use it well are those who stick to a small subset of features; most developers do not do this and instead use Scala as a chance to prove they are smarter than everyone else, and in the end making life more difficult.

I never liked the Scala community in general. I have gotten along with a couple of Scala advocates. I like and respect Dean Wampler (he’s not just a developer, he’s a photographer and has a PhD in theoretical nuclear physics). And how can anyone hate Daniel Spiewak? (If you have Daniel Spiewak on speed-dial, none of these criticisms apply to you.)

The overall Scala community always struck me as condescending, and generally wrong. Scala people love to go around and tell us they are sooooo much smarter than everybody else, and Scala was going to replace Java, and we were all just too dumb to see it, and using all the symbols was so much better, and if all the punctuation looked like line noise, well, again we were all just not smart enough to get it. And did they fail to mention that it is functional? And that anything functional in Java could only have come from Scala, because until our lord and savior Martin Odersky walked the earth nobody ever understood functional programming.

(I never thought Dean Wampler was talking down to anyone, and Daniel Spiewak is too busy moving objects with his mind to say an unkind word about anyone.)

The Scala community can be split into three groups (or kinds, if you will):

  1. People who realize that using all of it is a bad idea, limit themselves to a subset of Scala, and generally have success.
  2. People who try to use as much of it as they can to show how smart they are, and look down upon the rest of the world, including Scala devs in group number 1.
  3. People who started in group number 2 a few years ago, and found out the hard way that not only was group number 1 using Scala correctly, but maybe the Scala critics were right and it really is a mess.

Pro tip for life: Never say that people WILL do something whether they like it or not, or whether they realize it or not. You can say you think people SHOULD do something, or you wish they would. But then, most Scala people are not that bright. Scala never went anywhere. And sometime around Scala 3, Odersky said he was going to simplify Scala. The Scala community again congratulated themselves for Odersky’s brilliance, oblivious to the fact that this move meant the critics were correct. And as far as functional programming goes, while I don’t know about the relationships between the big names in the Java community, I know that Guy L. Steele, Jr has been working on Java for a couple of decades. He was there before Odersky, and is still there. Scheme was basically his PhD thesis. If there was any smack being laid down, Steele explained reality to Odersky, not the other way around. (If there are going to be Chuck Norris facts about anyone in CS, Steele is a top contender.)

Scala is like the first two chapters of the Book of Job. It’s like God said, “What if we made a language as cluttered as Perl, with a community as arrogant as Haskell?”, and Satan said, “Hold my beer.”

So this bozo professor said we should ditch a language everybody uses for one that nobody uses. Like a small child, he thought it was the future because HE liked it. There were a couple of issues. One was that I was getting emails from people (admittedly, a lot of companies pitching products) asking if they could speak at CJUG. Meanwhile, the Scala group was struggling to find material. I was on the mailing list for a while, and most months Deal Wampler would send a message a week before the meeting and say if nobody had anything to present, then he would cancel the meeting. So the second issue is why steal what little Scala oxygen there was from the pre-existing Scala group?

The day I was appointed president we had a presentation at Loyola. For some reason, our reservation was not for one room for the whole period, as it usually was, but split between two rooms. In two different buildings. People were not happy about having to go to a different building halfway through. The final straw was when we had a room reserved, and while this room had one of the largest stand-alone projector screens I had ever seen, it had no projector. I did not have the professor’s number, and there were no students attending. Since neither I nor the speaker were affiliated with Loyola, we could not get a projector. The presenter was Damodar Chetty, and he did very well under the circumstances. Maybe the guy has been cursing me ever since, but at the time he was very gracious.

I posted to the JUG Leaders list for advice on how to handle this in the future. Someone said one thing to do is get the speaker’s notes ahead of time and be ready to print out copies or copy the file from a thumb drive. Another one of those brilliant and practical ideas that never occurs to you when you really need it. My other solution was to find a location in the Loop.

We had a few presentations at the headquarters of Thoughtworks in the Aon Center. This improved both attendance and morale. At least one presenter hosted their talk at the Merchandise Mart. At some point someone put me in touch with a guy who worked at the Chicago Mercantile Exchange, and we started having presentations at The Merc. That was the best location. Close enough to the Loop for the hipsters, and close enough to Union Station for the suburban dads. We had meetings on the third Tuesday of the month. All I had to do was see if the presenter could be there on that day, and call my contact at CME. He would reserve the room and order food once I had a count of the people. Some months I had multiple people wanting to speak, and some months I had to scrape something together myself. With regards to content, running a group is like living paycheck to paycheck.

After the Great Recession, I was let go from my job. Bank Of America lost billions after they bought Countywipe, and they decided to try to make it back by getting rid of me and a few thousand other people who had done everything they asked us to do. (Life pro-tip number two: Never buy anything from a guy who looks like a bizarre comic book character; see here and here for comparison.) I got a job with a startup that only lasted a year. Then I learned Rails. I tried to get a job with a particular firm in Chicago, but after a few months, the CEO said no. I needed to find work, and I got a job in Austin. I had to resign from CJUG rather suddenly. But from what I can tell it is doing pretty well these days.

There was also a CJUG West that met in Schaumburg. I never attended those meetings. As far as I know, CJUG West stopped around 2008.

You’re welcome.

Note 1: I am not clear on what the office use stats are for Chicago post-COVID. I still get a few emails from organizations in Chicago. One mentioned that tech companies are starting to move into the Old Post Office building.

Image from the Theodore Psalter, a 9th-century manuscript housed in the British Library, assumed allowed under public domain.

 

2021-02-07 Update

by

I completed an introductory Kotlin tutorial on Pluralsight.

Kotlin seems like an okay language. The one tutorial I went through did not touch on Kotlin Native, or Android, or web programming. I get the impression that Kotlin is mostly used for Android. I have little interest in mobile apps. And from what I read from a reliable source, Android apps are getting more complex (see here and here).

So I might put off learning more Kotlin for a while. Kotlin Native looks very interesting, but it is not covered at Pluralsight. Kotlin Native might be a less-painful way to do embedded programming. It would be really interesting if it could be used in robotics. I think that would open up robotics programming to a lot of developers.

WRT the language itself, I am kind of neutral. Every time the instructor said, “Here is something you can do in Kotlin that you cannot do in Java”, it was always something you could do in Groovy that you could not do in Java. Java is adding more features all the time.

I have started going through some of the “What’s new in Java X”, since honestly my Java skills are out of date.

There was a conference at CME about JVM languages about 10 years ago. One Groovy programmer said that as Java got more features, Groovy might become superfluous. To a certain degree, that seems to be happening.

Granted, it might appear that way to me since the Austin Groovy/Grails Meetup stopped meeting long before everything else stopped. And Ken Kousen does not mention Groovy a lot in his newsletter, and there has not been a new episode of the Groovy Podcast since 2020-08. I like Groovy, but why fight the tide? If Ken Kousen is spending time on other things, perhaps that is a sign of the times. I am already shaking one fist at the clouds by looking into Lisp.

One thing that I think held Groovy back is that you really have to know Java in order to use it. Knowing some C will probably help you with Ruby, but I think Ruby shields newbies from C better than Groovy does with Java. I think this is also true of Kotlin, but not as much.

Granted, from what I heard about the origins of Groovy, making Java easier without necessarily replacing seems like it was the point. But for someone who is new to JVM languages (or programming altogther), they kind of have to learn two languages at once. Plus when a lot of people get tired of Java, they tend to stay away from JVM languages altogether.

Maybe I should have found a way to keep the Austin Groovy group going.

I was hoping to get through as much Kotlin and Java on Pluralsight as I could before my subscription expires. But I might just renew it for a year so I do not put as much pressure on myself, and I can also spend time on other things.

Like Clojure. I got a few feelers about Clojure jobs, so I think I will get back to going through the functions in the core Clojure API. I made a list of functions that seem to take functions as arguments or return functions, or take predicates. I think I will look at some of those, since those are the functional parts of Clojure.

I found a tutorial on Clojure Spec (article here, Hacker News comments here), and some of it was new to me. It looks like Clojure is starting to encourage keywords to have namespaces.

You’re welcome.

Image from an 15th century manuscript housed at Stiftsarchiv, image from e-Codices, assumed allowed under Fair Use.

July 2020 Update

by

This past month, I spent more time going through tutorials on Pluralsight. At least, when I have time. There was a release for the app I work on at my job, and there were a lot of defects.

Some of the tutorials take a while. I like to type in the code as they do them. But I might still be going too fast for the material to sink in. Right now I am going for breadth rather than depth. I don’t think I want to do Java forever.

Right now I am going through some Spring Boot tutorials. So far it looks interesting. I am a bit leery of doing anything Spring related because of the database abstractions. Sometimes it can be hard to find the SQL that populates a list. There are all the implementations that call controllers that call DAOs, some of which call other DAOs, and then the SQL is being pulled out of some XML file. I get tired of the whole “the toe bone is connected to the foot bone is connected to the ankle bone is connected to the shin bone” routine. I just want the SQL that populates some list.

Frankly, the app I work with is not using the most current practices. I am grateful to have a job, but I feel like the world is passing me by. But sometimes I wonder if the app I work with can use a lot of the new Java features. You can’t use the Functional Big Three (map, filter, reduce) while reading or writing from a database, because that is a side effect. And with the application I work with, there are a lot of loops within loops. And then sometimes even more loops inside those. Can the big three handle nested data structures? I haven’t done as much Clojure as I would like, but in the twitter-retriever app I did some calls to doseq and a LOT of loop-recur.

On the other hand, putting logic in the views has been a bad idea for a while, and does not require anything from Java 8 to change.

Other languages I am thinking about looking into are Go, Elixir and Kotlin.

Image from the Beatus of Burgo of Osma, a 10th century manuscript of ‘Commentary on the Apocalypse‘, written in the 8th century by Beatus of Liébana; image from Arachne, licensed under Creative Commons License (BY-NC-ND 3.0).

May 2020 Update

by

I got a Pluralsight membership for a year. It was 30% off.

Right now I am going through some Java 8, Spring and microservices tutorials. My Java skills are out of date. After that, I think I will look at some Go again. There is one on microservices in Go that looks interesting.

I still would like to get through all the books in “The Little Schemer” series and then go through SICP. But sometimes I wonder if spending all that time on Scheme may have been a mistake. Perhaps I should have just been building some web apps in Clojure. Maybe going for “enlightenment” was not the best idea.

I have also been struggling with updating my laptop to the latest version of Ubuntu. I think the issue might be that I have snap disabled. There were a few stories on Hacker News that convinced me to disable snap (see here and here), and now I cannot upgrade.

For my next laptop, I plan on buying one with Linux pre-installed. I might get a Dell, but since they only offer Ubuntu I might go with another vendor. I am looking at System76 (submissions to Hacker News at this page) or Purism (Hacker News submissions at this page). They both offer systems with either Ubuntu or their own Linux. I think in both cases their OSes are derived from Ubuntu. Purism’s is PureOS, and System76’s is PopOS.

One thing I would really like is to be able to disable the mousepad. If I don’t attach a USB keyboard, I always hit the mousepad with my thumb. In all seriousness, who came up with that gesture anyway? It’s not like anyone ever lifts up their mouse and puts it down.

You’re welcome.

Image from the Morgan Beatus, a 10th century manuscript of ‘Commentary on the Apocalypse‘, written in the 8th century by Beatus of Liébana; image from Arachne, licensed under Creative Commons License (BY-NC-ND 3.0).

2019-09-08: Thoughts On August Austin JUG Meeting

by

The most recent Austin JUG meeting was on August 27: “The Evolution of Java and the Developer Community with Heather VanCura“.

A big chunk of the talk was about the JCP, how it has been streamlined, and how to get involved. She did talk about the future of Java, as well as mention some of the sub-projects withing OpenJDK. She mentioned Valhalla, which is about developing general language features, Panama, which will make connecting with native code easier, and Loom, which will make concurrency easier to use.

In addition to lighter threads, Loom will have continuations and tail-call elimination. I think this is another data point in support of my thesis that languages are becoming more like Lisp, and the industry has been expending a lot of time re-inventing the wheel.

I think she mentioned that they are moving a lot of projects from Mercurial to git.

A lot of the questions afterward were about the support model for Oracle JDK vs OpenKDK and binary compatibility between code compiled with different versions of the JDK. There were some questions about getting involved in the JCP and contributing to a JSR, but not as many as more technical questions. From what I remember, she had more information about JCP/JSR questions than other topics. Maybe Oracle needs to send someone out to talk about support and binary compatibility.

She said there were some organizations running on JDK 4 and earlier, which only made me feel slight less depressed that I did before.

Someone asked if other JVM languages were involved in the JCP. She said yes, but the only one that popped into her mind was JRuby, which was also depressing. I still think JRuby is a solution looking for a problem. If you want a dynamic, agile scripting language on the JVM, use Groovy. That is what it was invented for. Java devs using JRuby are jumping though a bunch of hoops they do not have to jump through just to impress their Ruby friends. Sort of like CNN hiring conservative whack jobs to grab some of Fox’s audience; it’s just not worth it. Besides, mixing chocolate and peanut butter might sound great in a Reese’s Pieces commercial. That does not mean it is a good idea all the time.

I have noticed a trend that when people give presentations about why they picked a particular JVM language, Groovy is rarely on the list. I have seen Ceylon get more attention. When I ask people why Groovy was not on their list, they say they have never heard of if. I always think, “If that is any indication how good you are at due diligence, it’s a good think you’re not a lawyer.”

But I digress. Tomorrow is the Austin Clojure meetup. A few of the Clojure people were there. They might show up tomorrow and offer some opinions. She also gave this talk to the Houston JUG the next day. I might email a few people and get their opinion.

Bottom line: Good talk, but I don’t think the things she spent the most time covering are the questions that Java developers and organizations are most concerned with. If you are a Java developer, things like binary compatibility and support will determine what JDK you use for development and deployment. But you do not need to be involved in the JCP.

You’re welcome.

Image from the Urgell Beatus, a 10th century manuscript of ‘Commentary on the Apocalypse‘, written in the 8th century by Beatus of Liébana; image from Arachne, licensed under Creative Commons License (BY-NC-ND 3.0).

Venkat On Improving Code

by

The speaker at the Austin JUG last week was Venkat Subramaniam (https://twitter.com/venkat_s, http://www.agiledeveloper.com/). He is to Java what Sandi Metz is to Ruby: If he is speaking in your town, you don’t care what the specific topic is. You know you need to go because you will learn something.

The topic was Twelve Ways to Make Code Suck Less. I will post some notes I took at the presentation. He gave the list counting down from twelve to one.

Why should we care about code quality? You cannot be agile if your code sucks. Code we write is how we tell our colleagues how we feel about them. Quality code is code that can be understood quickly.

  • 12. Schedule time to lower technical debt. Technical debt can take the form of not upgrading a framework (or JDK) to a more current version. Bad code is not technical debt; it is sabotage. You should make a list of your technical debt and ask yourself if you are adding to it, or reducing it. Do not schedule 100% of your developers’ time for adding new features. You need some slack time; you need to spend some time learning.
  • 11. Favor high cohesion. Highly cohesive code does one thing and does it well. It reduces the frequency/reasons for change. High cohesion leads to low complexity. He mentioned Cyclomatic complexity. TL/DR: Be a dessert topping OR a floor wax.
    (I once again point you to the github page for Boot, with this little gem: “The Modern CLJS tutorials are an excellent introduction to Boot and ClojureScript. Pretty much everything you need to know about Boot to get a project off the ground is covered there. Check it out!” What if I want to learn about Boot but do not want to spend any time on ClojureScript? Why not have a tutorial that is just about Boot?)
  • 10. Favor loose coupling. Remove coupling if possible. Code with high coupling is hard to test, hard to extend. Having a lot of mocks in a test is a bad sign.
  • 9. Program with intention. He mentioned Kent Beck’s Design Rules:
    – Passes the tests
    – Reveals intention
    – No duplication
    – Fewest elements
    See also http://wiki.c2.com/?XpSimplicityRules They are in order, so passing the tests takes priority over intention. (So why does Dr Subramaniam go in reverse order?) We should program deliberately.
  • 8. Avoid primitive obsession. Do not write general code. Imperative code has accidental complexity.
    If you write a for-loop, you might wonder if you need less-than, or less-than-or-equal-to. That is a sign. We do imperative because it is familiar. He gave an example of a method with a for-loop. He noted you would need extra variables to hold the result and the count. He contrasted it with an example of functional code using Java lambdas. Functional code is about a series of transformations, not mutations. You can read the functional example in one pass, while with the imperative you need to go up and down while reading it.
    Imperative code is easy to write, but harder to read, while functional is easier to read. Good code is a story, not a puzzle. Lambdas are glue code, and sometimes two lines might be too many.
    Lambdas still take some work to understand. You need to learn the semantics. The functional style is a combination of declarative code and higher-order functions.
  • 7. Prefer clear code to clever code. Clever code makes you feel smart. Do not focus on performance too early.
  • 6. Apply William Zinsser‘s principles on writing: Simplicity, clarity, brevity, humanity.
    He recommended “On Writing Well” by William Zinsser. He said when his son asked to borrow his copy, he refused. Instead he bought his some another copy, and his son has re-read it at least three times.
  • 5. Comment the why, not the what. A lot of developers make a lot of comments because a lot of companies and professors mandate comments. Do not comment to cover up bad code. Write expressive code that is self-documenting. Many times a comment is like explaining a joke. He recommended the book The Design of Everyday Things.
  • 4. Avoid long methods – Apply SLAP (Single Level of Abstraction Principle) Long methods are hard to understand, lack cohesion, tend to become longer, and are harder to test. How to break up methods? Think about what layer of abstraction you are at.
    He said it could be a bad idea to have a hard and fast rule about how many lines a method should be. I know Sandi Metz has a rule that methods should not be any longer than five lines. But is putting a rigid number such a good idea? I am probably not as smart as Sandi Metz, and frankly, you probably are not either. If you dogmatically insist on a hard number, you will probably just get a lot of methods that call methods that call methods, etc. Maybe we should stop using the word “Rule” when “Guideline” would be more appropriate.
    One example he gave is that if someone asks you what you did over the weekend, you might say, “I went to such-and-such city on Saturday, and stayed home and did laundry on Sunday”. If they ask for more details, you could give them. Or you would talk about how you woke up, had breakfast, got in the car, got gas, with route you took, etc, etc, etc. They might not want that level of detail off the bat.
    You might have a method that gets info from a database table, gets info from a web service, combines/transforms them somehow, and writes the result to another table. And has all the try/catch/finally blocks for all the exceptions. That would be a good candidate for breaking it up.
    He related a story about someone who attended one of his lectures who was hired by a company to refactor a method that was 40,000 lines long. After a year, this person got it down to 30,000 lines.
    I stopped to talk to someone after the presentation, and then walked to my car. I think I overheard at least three conversations about this particular point.
  • 3. Give good, meaningful names. Your variable names represent your level of abstraction. Don’t use “x”, “y”, “someVar”, etc.
    He gave an example of bad variable names from a company that he could not name. It was software used to control oil refineries. One license is $1 million. He would not name the company, but if one license is $1 million, that narrows it down.
  • 2. Do tactical code reviews. Do code reviews often, and make them incremental. If you put them off, people will just allow code to pass without understanding it or really looking at it.
  • 1. Reduce state and state mutation. Do not make a field in a class (or method) unless you really need it. He related a story about an exhibit of Picasso with twenty versions of a painting, instead of just showing the final result. If Picasso could iterate, so can we. (I have a note asking how do functional languages deal with databases.)

Here are some pages on his site that I found to be informative:

 

You’re welcome.

Image from Queen Mlké Gospel, a 9th century manuscript housed at the Mekhitarist Monastery on San Lazzaro degli Armeni; image from Wikimedia, assumed allowed under Fair Use.

Debugging A Java App

by

Here are the steps to debugging a Serious Java Application.

  1. The ToeBoneBean is connected to the
  2. FootBoneImpl.isConnectedTo()
  3. AbstractAnkleBone.checkConnection()
  4. ShinBoneFactory.makeConnection()
  5. CompositeKneeBone.connect()
  6. StaticThighBoneBuilder.buildConnection()
  7. HipBoneService.comeUpWithAnotherWayToSayConnect()
  8. BackBoneQueryManager.manageConnection()
  9. NeckBoneDao.executeConnection()
  10. Which leads you to the SQL you were looking for in the first place.

 

In all seriousness, I am not one of those people who hates Java. It’s just frustrating going to a meetup and hearing about all these great things in Java 8 like lambdas (and yes, I know Java 8 has been out a while) or seeing all the wonderful things that you can do with Spring Boot, and then I go to my job and it’s like the past ten years never happened.

And before any Scala people feel superior, something like _{}\+= is not much better. We did Perl way back when. Not too interested in going back.

You’re welcome.

Image from a 10th-century Armenian manuscript housed at The Walters Art Museum, manuscript information here, image from World Document Library, image assumed allowed under Fair Use.

2017-11-06 Update: Deeplearning4J

by

One thing I did this weekend was get permalinks working again on my blog. I got an SSL certificate from Let’s Encrypt several months ago, and I was not able to get permalinks to work. It bothered me for the longest time. Problem solved.

Not much on the Clojure API front.

I did look at Deeplearning4J, a deep learning framework written in Java. I cloned their repo of examples, and I got some of them working with gradle files, and learned a bit more about Gradle in the process. You can use a GPU, or a CPU. No messing with CUDA if you don’t want to.

There is even a Clojure wrapper for Deeplearning4J as well.

I had a post about using CPUs and GPUs from JVM languages for math/science/AI a while back. Since then, I have gotten both Neanderthal and Deeplearning4J to work on my system. I did mention Bytedeco, which is used by Deeplearning4J and a few other libraries. I looked at some of the Javadoc, and they use a LOT of static methods (see here for an example). I may leave Bytedeco for others and use some of the higher-level libraries, like Deeplearning4J. I think a lot of static methods is not idiomatic Java. I might consult “Effective Java” later for a second opinion.

I am going to include below one of the build.gradle files that I used to get some of the Deeplearning4J examples to work.

You’re welcome.

apply plugin: 'java'
apply plugin: 'maven'
// apply plugin: 'groovy'
apply plugin: 'codenarc'
// apply plugin: 'application'

repositories {
  mavenCentral()
  jcenter()
}

// this has to be at the top so runJava can read mainClassProp
ext {
    dl4jVersion        = '0.9.1'
    nd4jVersion        = '0.9.1'
    jacksonVersion     = '2.6.6'
    logbackVersion     = '1.1.7'
    nd4jBackend        = 'nd4j-native-platform'
    scalaBinaryVersion = '2.11'
    dl4jSparkVersion   = '0.9.1_spark_1'
    awsSdkVersion      = '1.11.109'
    jcommonVersion     = '1.0.23'
    mainClassProp      = 'org.deeplearning4j.transferlearning.vgg16.dataHelpers.FeaturizedPreSave'
}

/*
'org.deeplearning4j.emr.EmrSparkExample'
'org.deeplearning4j.transferlearning.vgg16.FitFromFeaturized'
'org.deeplearning4j.transferlearning/vgg16.dataHelpers.FeaturizedPreSave'
'org.deeplearning4j.rnn.SparkLSTMCharacterExample'
'org.deeplearning4j.stats.TrainingStatsExample'
'org.deeplearning4j.mlp.MnistMLPExample'
'org.deeplearning4j.mlp.MnistMLPDistributedExample'
*/

allprojects {
    dependencies {
        // testCompile 'junit:junit:4.10'
        // testCompile 'org.mockito:mockito-all:[1.8.5,)'
        // From Maven, it's GroupId:ArtifactId:Version 
    
        // logging
        // Using sfl4j allows us to replace different loggers later, if desired
        compile group: 'org.slf4j', name:'slf4j-api', version: '1.7.2'
        // sfl4j is just a facade for loggers. To get logging to work, tie it in with an implementation, e.g., logback
        compile group: 'ch.qos.logback', name:'logback-classic', version: "${logbackVersion}"
        compile group: 'ch.qos.logback', name:'logback-core',    version: "${logbackVersion}"
        
        compile "org.deeplearning4j:deeplearning4j-zoo:${dl4jVersion}"
        compile "com.fasterxml.jackson.core:jackson-databind:${jacksonVersion}"
        compile "com.fasterxml.jackson.core:jackson-annotations:${jacksonVersion}"
        compile "com.fasterxml.jackson.module:jackson-module-scala_2.11:${jacksonVersion}"

        compile "org.nd4j:${nd4jBackend}:${nd4jVersion}"

        compile "org.deeplearning4j:dl4j-spark_${scalaBinaryVersion}:${dl4jSparkVersion}"

        compile "org.deeplearning4j:dl4j-spark-parameterserver_${scalaBinaryVersion}:${dl4jSparkVersion}"

        compile "com.beust:jcommander:${jcommonVersion}"

		compile "com.amazonaws:aws-java-sdk-emr:${awsSdkVersion}"
		// scope: provided
		
		compile "com.amazonaws:aws-java-sdk-s3:${awsSdkVersion}"
		// scope provided
        // compile "org.nd4j:${nd4jBackend}:${nd4jVersion}"
        // for CUDA
        // compile "org.nd4j:nd4j-cuda-7.5-platform:${nd4jVersion}"
    }
}

task runJava( type: JavaExec ) {
    println( "-- Groovy version: " + groovy.lang.GroovySystem.getVersion() )
    dependsOn classes
    description = 'Run gradle.sample.SampleApp'
    // Java main class to execute.
    if ( rootProject.hasProperty( "mainClass" ) ) {
        main = mainClass
    } else {
        main = mainClassProp
    }
    // We need to set the classpath.
    classpath sourceSets.main.runtimeClasspath
    // Extra options can be set.
    systemProperty 'sysProp', 'notUsed'
    println System.properties['iterations'] 
    // systemProperty 'iterations', System.getProperty( 'iterations' )
    systemProperty 'iterations', System.properties[ 'iterations' ] 
    // you can use this if nothing is set via command line
    // systemProperties = System.getProperties()
    jvmArgs '-server'
    // We can pass arguments to the main() method
    // of gradle.sample.SampleApp.
    if ( rootProject.hasProperty( "mainArgs" ) ) {     
        args = mainArgs.tokenize() // args wants a List, so split by spaces if need be
    } 
}

Image from the Trebizond Gospels, a 12th-century Byzantine manuscript housed at The Walters Art Museum, manuscript information here, image from World Document Library, image assumed allowed under Fair Use.

2017-06-25 Update

by

Not a whole lot to report this week. Still working on Simply Scheme. Perhaps I should call it Slowly Scheme.

I decided that this website could use a bit sprucing up, perhaps some pictures. So I am looking at a Java app called Processing that does visual stuff. I think some people use it as a CAD tool, while other people use it to make generative art (abstract art, patterns, fractals, etc).

You’re welcome.

Thoughts On Native, GPU, Groovy, Java and Clojure

by

One interest of mine is math and scientific computing, although I admit right now my interest is far greater than my knowledge.

It seems like for a lot of math and scientific computing, you either need to do stuff in C, C++ or even Fortran, or use a library in a higher-level language that lets you interface with libraries in C/C++/Fortran. Another new trend is to use code that runs on the GPU, like OpenCL.

Lately I have been looking at a few Java libraries that interface with lower level code. I think it is a good idea to keep learning new things. I think AI and machine learning will become more important over time.

I contributed briefly to the SciRuby project a few years ago. One of their main subprojects is a Ruby wrapper around BLAS, ATLAS and LAPACK. I think some of the instructions for compiling BLAS, ATLAS and LAPACK might still have a few sentences written by yours truly. (It’s a very involved process.) Another one of their projects is a Ruby wrapper/interface around the GNU Scientific Library, or GSL.

One huge language in AI and data science these days is Python, with the NumPy and SciPy libraries. Again, some of the functionality is provided by linking to ATLAS and LAPACK. If it worked for them, maybe it will work for the JVM family of languages.

There is at least one project in Clojure that does this: Uncomplicate. It has sub-projects for OpenCL and BLAS and ATLAS. Along with Java and Groovy, Clojure is a language I am interested in. But for the time being I cannot use these projects. I think my graphics card is a bit too old and the drivers are too old for many of these libraries.

Uncomplicate is part of the reason I am looking at this area again. Somewhere on the site, the author points at the for development, it is okay to use pre-baked packages for ATLAS and LAPACK. One of these projects that I mention says for the best performance, you may want to hire someone to install ATLAS and LAPACK and other native libraries optimized for your production hardware. I did not know people made a living doing just that.

Another reason I am looking at this is GPU computing was mentioned in one of the side talks at the most recent ClojureConj. This is second- or third-hand, but apparently Goldman Sachs runs a nightly report analyzing the market (or some part of it, I am not too clear). It used to take nine hours to run. It was refactored to run on the GPU, and now it takes 20 minutes. I think all developers should become familiar with some of these technologies.

Will any of this gain traction in the Groovy space? I know things have been rough for Groovy for the past couple of years, especially after getting dropped by Pivotal. It seems like a lot of Groovy developers are happy making web apps with Grails. I like Grails, and it has worked out well for a lot of people. I will go through the Groovy Podcast backlog; perhaps Groovy developers are looking into this and I am just not aware of it.

Groovy developers can use the Java wrappers for native and GPU computing. No need to re-invent the wheel. Maybe going from Groovy to Java to native is not quite as fast as going from Python or Ruby to native, but I don’t think so. Even Ruby developers joke that Ruby is pretty slow. They say that sometimes the speed of the programmer is more important than the speed of the program. (But for some reason the Ruby devs who are moving to Elixir are now saying performance and concurrency are important after years of saying they were not. Whatevs.)

I do not have much experience in native or GPU programming. I am not too clear how NumPy and SciPy are used. I don’t want to swallow the Python ecosystem to get the answers to just a few questions, but: Are NumPy and SciPy programs (or bare-metal math/science apps in general) rewritten frequently? Or do they stay the same for years at a time? It seems to me if a GPU program does not change, you might be better off biting the bullet and doing things in C or C++.

But given the success of NumPy and SciPy, perhaps Groovy and Java can be used in this space. Maybe it’s not the end of the world if a bank’s nightly report takes 22 minutes. (How successful is SciPy? Not only are there conventions for general Python, there are also conferences for just SciPy). I doubt any technology will dethrone Python anytime soon, but it is an area to look at.

Having looked a small bit at some of these libraries, I do wonder how much can the unique features of Groovy be used. A lot of the tests deal with arrays of primitive types.  Someone on the Groovy mailing list wrote: I’ve found that you can really shrink down the lines of code just by using the basics that Groovy provides around collections, operator overloading, Groovy truthiness, etc.. Maybe that is enough reason to use Groovy for high performance computing (or at least higher performance). Can we make code calling native and GPU libraries better with features from Groovy, or will we just get Java without semicolons? Some of these functions do not want an ArrayList of objects, they want a fixed-length array of integers. Granted, you can make primitive arrays in Groovy:

def someArray = [[1, 2], [3, 4]] as int[][]

but I think that is more keystrokes than in Java.

Just looking at a small part of TensorFlow, here is a line from the original Java test:

final Tensor matrix = Tensor.create(new int[][] {{1, 2}, {3, 4}})

Here are two Groovy versions that both work (meaning the tests still pass):

final Tensor matrix = Tensor.create([[1, 2], [3, 4]] as int[][])
final Tensor matrix = Tensor.create([[1, 2], [3, 4]])

Granted, it’s one line, but I’d say that’s idiomatic Groovy. Perhaps this could be a (small) market opportunity for the Groovy community. Or maybe I’m just crazy.

If you google “groovy jni” the first hit is this page from Object Partners. It mentions JNA and BridJ which I think are used by a lot of the libraries mentioned below. Frankly, it sounds like a lot of work making this sort of thing happen.

Regardless, even if I have to stick with Java and Clojure, I will still keep an eye on these libraries and this space. I don’t think I can become an expert in all of these, but as I said, I think it is important for developers to keep an eye on this spae. I might start some pages about the different libraries, and perhaps share my thoughts on them. Below are a few notes on what I have found so far.

I started by googling “Java BLAS” or “Java GPU”, and at first I only got a few libraries. But as I kept looking, I kept finding more.

The first two I found where jblas and Aparapi. From the jblas website, jblas “is essentially a light-wight wrapper around the BLAS and LAPACK routines.” (And from what I can gather, the correct capitalization is “jblas”.)

Aparapi is a wrapper around GPU and OpenCL code. If you cannot get your GPU drivers installed properly, it will do everything on the CPU. Sometimes I get core dumps running the tests on my Ubuntu laptop. It is about six years old, and has 6 GB of memory, so perhaps that is the issue. But on my 8 GB Windows laptop two of the tests fail. I plan on getting a new laptop soon, so I will try this again when I do. I bet getting all this set up on an Apple laptop is really easy; for the first time in my life I am thinking about buying an Apple product.

TensorFlow has some Java APIs, although they warn that there may be API changes and breakages for languages other than Python and C. This can use either the CPU or the GPU, but for GPUs it can only work with NVidia cards.

The Lightweight Java Game Library works with OpenCL, OpenGL, Vulkan, EGL and a lot of other stuff I have never heard of. As the name suggests, it is primarily for game development, and the wrappers around the underlying native libraries and standards require more knowledge of those underlying native libraries and standards than most libraries.

There is ND4J (N-Dimensional Arrays for Java) and Deeplearning4j. I am not sure if they are related somehow, but they were both started by developers at a company called Skymind.

And then there is the granddaddy: ByteDeco. There is a lot of stuff here. Some of the contributors work on ND4J and Deeplearning4j, and also work at Skymind. This project provides Java interfaces to 21 different C/C++ libraries: video, math, science, AI, robotics, facial recognition, lots of good stuff here. And after looking at their list, yes, I think “Skymind” is a bit too close to “Skynet”. But at least they’re giving you a hint.

You’re welcome.

Image from an 11th-century manuscript housed at the Bamberg State Library (Wikipedia page here), manuscript description here, citation link here, image assumed allowed under Fair Use.