Trying Out GPT4All

I tried out GPT4All. It is an app that can run an LLM on your desktop. Some of the models it can use allow the output to be used for commercial purposes. The only downside was it is not very fast, and makes my CPU run hot. The project provides installers for Linux, Apple, and Windows (or, as I like to call it: Win-Don’t).

I found out about it while looking into alternatives to ChatGPT and OpenAI. There are a few videos about it on Matthew Berman’s channel, including an interview with the CEO of Nomic AI, the company that started the GPT4All project. I think it is great that people are making alternatives to ChatGPT and OpenAI.

I first downloaded it a couple of weeks ago. The installer itself worked fine, but I had to change the permissions to get it to run: chmod 755 gpt4all-installer-linux.run. I got an error when I tried to run the program: bin/chat: error while loading shared libraries: libxcb-cursor.so.0: cannot open shared object file: No such file or directory. I installed the library as root: apt install libxcb-cursor0. After that it worked fine.

I downloaded the installer again today and re-installed to see if anything was different. I might have still gotten the libxcb-cursor error if I had not installed it earlier. Now the program lets you choose a model when you first run it. Before the model file had to be in the same directory as the executable, and if there was more than one model file it would pick the default. When I tried to run it the first time, I could download a couple of models, but I could not get past that screen. I had to exit and restart the app to actually use the models. There is a list of the models available here (historical commit here). The output of the default model can be used in commercial applications. Most of the other models are based on LLaMA, and commercial use of the output is prohibited.

The new version also gives you the option of sending feedback on the responses back to the GPT4All project.

The model files are from 3.5GB to about 7.5 GB. I have downloaded several model files over the past few weeks, and I have used up as much hard drive space in two weeks as I did in five years on my recently deceased laptop.

While the application is idle it does not use a lot of resources, but when it is generating output the CPU usage and temperature both spike. I do not know if I will be running this on a regular basis. My Meerkat has an Intel i5-10210U CPU, 4.20 GHz, 6 MB Cache, 4 Cores and 8 Threads with 64 GB of memory. The GPU is Intel, so I do not think I will be running a lot of Pytorch.

When I was using the version of the software from two weeks ago and using the ggml-gpt4all-j-v1.2-jazzy model file, I asked it to write some sample sales emails for companies in different industries. It kept writing emails asking for sample sales email instead of an actual sales email. I tried the new ggml-gpt4all-j-v1.3-groovy model (“Current best commercially licensable model based on GPT-J and trained by Nomic AI on the latest curated GPT4All dataset”), and it did better, but the results were pretty similar for each one. They all started with “Exciting news!” and offered a “wide range of” something. I read “Exciting news!” so many times, I started hearing the voice of Professor Farnsworth in my head. Here is the output from the prompt “Write an email from a consulting company whose software developers use Emacs”:

Subject: Exciting news! Our software developers are now using Emacs.

Dear [Recipient’s Name],
I hope this email finds you well. My name is [Your Name] and I am the owner of [Company Name]. We are excited to announce that our software developers are now using Emacs as their primary text editor.

As a company that specializes in software development, we understand the importance of having the best tools for the job. That’s why we offer a wide range of software development tools, including Emacs. Our developers are trained to use the best tools available, and we are always looking for new clients to hire.

If you are interested in learning more about our services, please don’t hesitate to contact us. We would love to discuss your project and see how we can help you achieve your goals.

Thank you for considering [Company Name]. We look forward to hearing from you soon.

I have to agree, Emacs is the best tool for just about any job.

I tried the same prompt with the same model on the new version, and I got this:

Subject: Help with Emacs – Our Developers Need Assistance

Dear [Name],

I hope this email finds you well. We are writing to inquire about your experience using the popular text editor, Emacs. As we understand that you work as a developer for our esteemed client, we would like to offer some assistance in resolving any issues you may be facing while working on their projects.

We have noticed that many of our clients often struggle with certain aspects of using Emacs, such as navigating through files efficiently, finding specific lines of code, or even understanding basic syntax. These challenges can lead to frustration when trying to accomplish tasks within the program.

To help alleviate these frustrations, we would love to invite you to participate in a webinar where we will provide step-by-step instructions on how to effectively navigate and utilize the features of Emacs. This webinar will cover topics such as file navigation, search functions, and programming language support.

If you are interested in participating, please let us know by responding to this email. If not, no need to worry, we completely understand if you prefer to continue using another tool. However, we believe that our webinar could greatly benefit you and your team members who also work on projects involving Emacs.

Thank you for considering our invitation. We look forward to hearing back from you soon.

Best regards,
[Your Name]

Not as positive. I selected the option to regenerate output, and it was just one sentence that did not even mention Emacs.

I tried the prompt with the model ggml-gpt4all-l13b-snoozy, which is “Current best non-commercially licensable model based on Llama 13b and trained by Nomic AI on the latest curated GPT4All dataset”, and I got this:

Subject: Emacs Software Development at [Consulting Company]

Dear [Client],

I hope this email finds you well. My name is [Your Name], and I am the Director of Software Development at [Consulting Company]. I am writing to express our excitement about the opportunity to work with you on your upcoming project.

At [Consulting Company], we pride ourselves on our use of Emacs as our primary text editor. Our developers have found that Emacs provides a powerful and flexible platform for software development, and we believe that it will be an asset to your project as well.

We understand that your project is of the utmost importance, and we are committed to delivering a high-quality solution that meets your needs. Our team of experienced developers will work closely with your team to ensure that your project is completed on time and within budget.

If you have any questions or concerns, please do not hesitate to reach out to us. We look forward to working with you and delivering a successful project.

Best regards,

[Your Name]
Director of Software Development
[Consulting Company]

I agree that “Emacs provides a powerful and flexible platform for software development”, and that it is as asset to any project.

Some of the responses interpreted the prompt to produce what I want (“We are amazing and Emacs is amazing and we can make amazing software for you with Emacs”) and one was offering to teach developers how to use Emacs. I tried a couple of more times, and again the results were different with the same prompt.

There is a project that provides a command line interface to the models called LlamaGPTJ-chat, which I might get to in a few more days. I have tried a couple of other CLI programs, and they also use a LOT of CPU.

Michael Finney wrote about his experience with GPT4All here. Per his article, GPT4All will soon have an API that can be called and run on a local server. I joined the GPT4All discord, but I have not spent too much time there.

I do not know if I will be using this too much since it puts a lot of stress on my system. I will see if anyone is running models on the cloud and find out how much it costs. But I support anyone who will give the world alternatives to OpenAI.

You’re welcome.

I am not using an image of someone writing because while this post mentions Emacs, it is not about Emacs.

Image from Madrid Skylitzes, a 12th century manuscript of Synopsis of Histories by Greek historian John Skylitzes; manuscript housed at National Library of Spain; manuscript information here. Image from World Document Library, licensed under CC BY-NC-SA 4.0.

Reactions To Recent AI Developments

There are a variety of reactions to what has been happening in AI lately.

Some of them are depressing. All of the people who will not say the sun comes up until Microsoft Teams or Office says it does are doing the same thing with AI. They think it is all about OpenAI and ChapGPT. MS owns part of OpenAI. LLMs are one of the biggest changes to come along in technology in a decade. We should not pass up this chance to prevent the company that has made technology suck for the past three decades have any more influence in our lives or society [Note 1].

Maybe I should not be shocked by the immensity of human stupidity, but I do not understand why people are not grabbing this chance to get Microsoft out of our lives with both hands.

I see this a lot at my employer [Note 2]. There the unholy trinity is OpenAI, ChatGPT, and Microsoft Azure. There are a lot of higher-ups who just push whatever garbage vendors are trying to sell, regardless of whether or not any of the companies that we are supposed to be helping (aka “paying clients”) want any of it. I do not understand why these people are paid lots of money to look at “new technology”, only to turn around and say “More Oracle! More SAP! More Microsoft!” A few years ago, a lot of them were pushing blockchain, which has gone nowhere. And none of these block-chumps admit that they were wrong about that. [Note 3]

Now these shysters are pushing the metaverse, even though only literally the only person on the planet who wants it is Mark Snakerberg. One reason I think he wants it is if they can get you to go to their site with their gear, then they own the whole experience. Right now you have to use a browser on a PC or an app on a phone to use Facehook, and Meta does not own an OS or a browswer. But that is not anybody else’s problem. I think the other reason is that he is so stiff and robotic he is the only person who has an avatar that looks more human than he does, even without legs. Here is an article and discussion on Slashdot about retailers dumping the metaverse. I can see retailers using the metaverse: people might want to try out new clothes without actually changing several times. If retailers and Disney do not want the metaverse, then it really is dead. I have heard the headsets are too heavy. Do you want something wireless surrounding your entire head? (Hey, maybe that’s why Snakerberg keeps throwing money at something nobody else wants.)

I wonder if he is regretting giving Sandberg the boot.

Even the Emacs community is gung-ho about OpenAI and their products. I noticed that Sacha Chua started a section for AI in her weekly Emacs news posts. The first AI mention I could find was for a GPT package on 2022-11-21 (as of 2023-04-24, it looks like that package is for OpenAI models only). The section started showing up on 2022-12-12, on and off until February, and then consistently since then. There are a few packages that say they will incorporate other LLMs as more are available. Most of the AI packages are just more wrappers around ChatGPT. There are a few posts on the subreddit asking about open source alternatives, and one about OpenAssistant (which I mention below). The posts are here, here (with a response from Irreal here) and here. I know that OpenAI’s products are more mature than others at the moment, but it seems like even the open source crowd is going all-in on the billion-ai-res’ shiny object. [Note 4]

It will be interesting how the corrupt-o-currency crowd reacts to AI. A lot of people insisted they were in corrupt-o-currency “for the technology”. I think AI will show us if those people are really interested in technology, or if they are stupid, or grifters. Now we have something that really IS interesting. Corrupt-o-currency tech was not that interesting. Increment a number going into a hash function until your output starts with a particular number of zeroes. That is really it. Immutability is not the default in a lot of programming languages, and it can make data easier to manage and reason about, but it is not unique to corrupt-o-currency. And it’s a pretty dumb reason to fool yourself.

Blockchain was a stalking horse for bitcon that promised a glorious future that never arrived. Every time I watched or watched anything about blockchain, it was always vendors pushing products, and there were never any user testimonials. Contrast that with AI: with AI we see users actually trying it out. There are a lot of people talking about AI who are not trying to sell you something, while blockchain/NFT/bitcon was nothing but grifting. We do not see vendors going on about “someday”, or AI bros saying, “Have fun staying dumb.” We do see people implementing models to compete with ChatGPT (more on that below).

A lot of corrupt-o-currency advocates say it’s still early days for their magic beans. They have been saying that for ten years, and there have been a LOT of people pushing it and trying to find a use for it. They might counter this is not the first time people thought AI was going to change the world, and there were a couple of AI winters: 1974–1980 and 1987–1993, so could there be a corrupt-o-currency winter as well? One difference is that nobody went to jail for AI fraud. There were no “rug pulls” in AI back in the day. AI has been attempts at solutions that until recently never solved the problems they were trying to solve. Block-coin was a technology looking for a problem to solve (besides money laundering and selling drugs). The whole concept of “digital assets” makes no sense at all. They do not have income streams like companies do, and they have no intrinsic use beyond financial transactions like commodities do. It is like someone found a way to combine Beanie Babies and coin flips and people decided to start gambling on them. Aside from helping pedophiles launder North Korean drug money, none of this serves any purpose.

AI has been around for a while, but I think that it was the release of Chap-GPT 3.5 in November that really changed things. The first time we talked about ChatGPT at the EmacsATX meetings was in December.

Right now AI and LLMs are en fuego. Not everyone is just pushing incumbent vendors and their products. There are a lot of projects working on open source LLMs. I wrote about the idea that people might try to run LLMs locally, and there are projects working on that.

Some of them are based on Facebook’s LLaMA model, so they cannot be used for commercial purposes. One is llama.cpp. There is one in Golang called llama.go. Another in Golang called LocalAI. One goal of these projects is to be able to be run on a CPU, instead of needing a large cluster of GPUs. There is also a subreddit dedicated to running LLaMA locally. Some experts think that making models with more parameters is not the way to make progress, and that algorithms are where progress will occur. Will this mean that the need for GPUs will plateau, and CPUs will become more important in AI? I don’t know. I tried using some model files from Hugging Face that are based on LLaMA, like Stanford’s Alpaca, but so far I have not been able to get anything working with that one. I think it only works with the original LLaMA files.

There is a project called Red Pajama from Together.xyz to build a completely open source counterpart to the LLaMA model. Their plan is to have something that can be run on a consumer grade GPU.

An interesting project is GPT4All. This project can be run locally. It is made by a company called Nomic.AI; their main product makes visual representations of AI datasets. I found out about it from Matthew Berman’s Youtube channel. It is uses an older model from EleutherAI called GPT-J (Hugging Face page here, Wikipedia page here). I am part of the Discord for this project, so I will keep an eye on this. The CEO of Nomic was interviewed by Matthew Berman, and he talked about how they went through a lot of effort to get a dataset that is completely open source and can be used for research and/or commercial purposes. He said that he thinks OpenAI has a lot of proprietary data in their dataset, partially due to how they created it, partially due to people uploading their own internal data into it. He predicts there will be a lot of lawsuits over AI data for years.

I got GPT4All to work locally, but I did get an error the first time: /bin/chat: error while loading shared libraries: libxcb-cursor.so.0: cannot open shared object file: No such file or directory. A Google search led me to an issue on their Github repo. I did not need to run all those commands, I only needed to run this one: apt install libxcb-cursor0.

Another project is OpenAssistant, made by a non-profit in Germany called LAION (Large-scale Artificial Intelligence Open Network, pronounced like the large feline “lion”) (Github page here, Hugging Face page here). I think the goal is not only to use their dataset to respond to prompts, but to be able to retrieve external data (like Google search results) in real-time. They want to release something that can be run locally, but they want the main interface to be their website. One of the ways the developed their dataset is by having humans ask questions, other people answer questions, and still more people rate the questions and the answers; someone could perform all three roles for different questions.

I found out about this from the YouTube channel of one of the leaders of the project named Yannic Kilcher. At first I did not think he was a real person but an AI-generated avatar because there are some jumps and jerks in his speech. Then I looked at a few live streams, and I think that for his videos he edits out all the filler words and pauses. I am in their Discord and will keep an eye on this.

Another open source project is StableLM from Stable.AI. They made the image generator Stable Diffusion. I thought they were all about images until their LLM came out. Right now I do not know too much about this one.

One possible clue to what might happen is a science fiction project called Orion’s Arm (site here, Wikipedia page here). Unlike most sci-fi, there are no humanoid aliens. There are baseline humans, humans augmented with various levels of technology, and AIs of various levels, up to almost god-like omniscience. Some people have put some thought into how humans will live with AI beyond just “AI will kill us all.” Interestingly, some of the images on that site are made with the Midjourney AI.

One thing about all of this that depresses me is I never got the chance to work with technologies that really interest me. When I read about or talk to Lisp and Smalltalk developers of yore, it seems like there was an age of heroes when gods and men strode the earth together. Now the world is full of pinheads pushing vendor nonsense, and people too stupid and too lazy to use something not made by Microsoft. Let’s get this company out of our lives forever.

Another thing that depresses me is I bet all the alternative medicine woo-woo people are not worried about their jobs.

You’re welcome.


Note 1: If you think OpenAI is actually open because it has the word “open” in it, you are probably the same sort of person who thinks SharePoint is good for sharing because it has the word “share” in it. Do us all a favor: go sit in a rocking chair right now and get all your nutrients through a straw. You are about as useful to society as someone already in that state.

Note 2: I do not speak for my employer, nor do they endorse this blog, as per the About page. Per the Disclaimer page: if any of this bothers you, go jump off a cliff.

Note 3: I don’t want anybody to think I have nothing good to say about my employer. Granted, I am currently making a living with a proprietary software tool that I really do not like. When I joined I knew nothing about it, and they paid me while they trained me how to use it. A lot of companies would never do that. And there were a couple of times in the past decade when someone found something else for me to do while things were slow. It is a big company. I think it has more than half a million employees. There are some smart people there, and frankly some dumb ones. It seems like too many decisions are made for the wrong reasons: inertia, the sunk cost fallacy, and this-is-what-the-vendor-told-us-to-do. Why can’t these companies founded by billionaires do their own marketing?

The company does actually do some useful things. They do a lot of work for various governments and large corporations around the world. Entities that actually make products and services people use. Maybe startups have more interesting technology, but I never bought into the whole startup religion here in Austin. There are a lot of people who seem to want to work for startups just because they are startups. Asking them what their company does or why anyone should care is a question a lot of them do not seem to like and sometimes do not have an answer for. There were a few people that I would see at some meetups maybe once a year. Every time I saw them they were at a different company that I had never heard of. They all just seemed to fade away. Maybe they made more money than me, but what are they really doing for the world? If any of those companies are still around, they are just leaving messes for someone else to clean up.

Note 4: I do not want any of this to be interpreted as disparaging Sacha Chua or all the things she does for the Emacs community. I think her posts simply reflect what is happening in the Emacs community. There are some people in the Emacs community who are as leery of AI consolidation as I am.

Image from The Codex of Fernando I and Doña Sancha, aka Beatus Facundus, an 11th century manuscript manuscript of ‘Commentary on the Apocalypse‘, written in the 8th century by Beatus of Liébana; manuscript created at monastery of Santo Toribio de Liébana (Wikipedia page here), currently housed at the National Library of Spain; manuscript information here. Image from World Document Library, licensed under CC BY-NC-SA 4.0.

Notes on LLMs and AI

There has been a lot of press lately about AI, OpenAI, GPT-${X}, and how AI will affect the world. For the time being, I plan on not looking further into AI (unless my current or a future employer compels me to). I think that right now there is not enough diversity in the vendor population. I also have some thoughts on how it will affect things going forward.

I do not like to get too meta in my posts, but I have been writing this on and off for over a week, and I want to get it out before too much more time passes. I am still learning about this stuff, like what is the difference between models, weights, and datasets; some articles use a project name to refer to all three components. The LLaMA debacle is the textbook case: some stuff was released, some was leaked, there are projects that are based on Meta’s work, some that seem to be clean-room implementations, so how it all fits together is murky to me.

GPT-${X} by OpenAI is taking the world by storm, particularly Chat-GPT. It was the focus of a recent EmacsATX meeting. It is disruptive in the sense that it has capabilities beyond prior AI technology, and will probably have a profound affect on society going forward. But in another sense, it is the opposite of disruptive; it consolidates power and influence in OpenAI. One of the owners of OpenAI is Microsoft, and for me that makes anything by OpenAI something to avoid. They are not doing this for you.

I think a lot of people do not realize that when they play around with the OpenAI prompts in ChapGPT, they are training the OpenAI models and making them better and more powerful. Power that can be used by other users of tool. Not only the vendors, but also your competitors. There have been reports of confidential data and PII being put into ChatGPT, and then extracted by other users later. People need to be more careful. And stop making the rich and powerful richer and more powerful. A lot of people in corporate America might work at companies that are independent on paper, yet they all act like they want to be subsidiaries of Microsoft. Start looking out for your own company and your own career and your own life.

The GPT-${X} products were used in making GitHub Copilot. I mentioned Copilot when I posted I was moving from Github to Codeberg. It does not respect licenses, which could put a company at legal risk, and sometimes it “solves” a problem while violating stated constraints. GPT-${X} has the same issues: Who owns the training data? Who owns the output?

It is good to automate things, but could relying on AI too much make people stupider? A good point was brought up in the discussion about why MIT dropped SICP: When you rely on a black box, how do you know you can rely on the black box? I think we might be coming close to fulfilling a prophecy from Dune:

Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.

I think we should collectively make an effort to avoid anything by OpenAI, and anything Microsoft. I do not know how long Microsoft has been involved with OpenAI, but there are a few MS hallmarks: it is called “OpenAI” even though it is not open (they have been tight-lipped about how they trained their data), and when it is wrong it insists you are wrong. And when it is incorporated into MS products it has started pushing ads.

There are a few alternatives out there. There is a company called Hugging Face that I think provides datasets, different models and hosting for AI. I think you can provide your own data. There is a company called Lambda Labs which provides all your AI/GPU needs: cloud hosting, colocation, servers, workstations with terrabytes of memory, and a very expensive and very nice looking laptop with Ubuntu pre-installed (a LOT more than System76, but it is nice to see more Linux laptops out there).

WRT software, there are some implementations of AI that are open source. NanoGPT can be run on a local system, although it might take a while. You can find the Github link here, and a link to what might be a fork on Codeberg here. It was started by Andrej Karpathy, who worked on autonomous driving at Tesla and worked at OpenAI.

GPT is a type of artificial neural network known as a large language model, or LLM. Then Facehook/Meta released an LLM called Large Language Model Meta AI, or LLaMA, so now there are a few projects with names referring to South American camelids: llama.cpp (Github link here, Hacker News discussion here), and a fork of llama.cpp called alpaca.cpp (Github link here, Codeberg link here). Once they saw money going to someone else’s pockets, Stanford decided to get in on the act with their own LLaMA implementation, also called Alpaca. There is one caleld Vicuna (intro page here, Github link here). And, last but not least, Guanaco, which look like a fork of Stanford’s Alpaca (Github repos here, page here). You would think AI researchers would come up with more original names rather than run a theme into the ground.

Note: I think Facebook/Meta did release some papers about LLaMA, and then some parts of it were leaked. The status of these projects is a bit unclear to me at the moment. Some of the projects mentioned cannot be used for commercial purposes. IANAL, but I think that llama.cpp and alpaca.cpp can since they are clean-room implementations and were not created with any assistance or collaboration with Meta. Stanford got some early access to LLaMA, so its project and Vicuna cannot be used for commercial purposes.

You can find some more info about open source AI here on Medium (archive here), and here on HN. I think the group EleutherAI is trying to be an open source counter to OpenAI.

There are a LOT of other AI projects out there, but a lot of them are just interfaces to Chat-GPT or DALL-E or something else from OpenAI, as opposed to a program you can run for yourself. A lot of the forks and clean-room/non-OpenAI models require a LOT of memory. Some need at least 60 GB. The mini I got from System76 can have up to 64GB. They have desktops that can go up to 1TB of memory, and servers up to 8TB. Granted, maybe something local will never catch up to OpenAI, but as a few comments in the HN discussion on llama.cpp pointed out: the open source models are becoming very efficient very quickly. Granted, some of the commenters said that AI might be out-of-reach for the hobbyist. But then all this stuff is doing is simulating a human.

So where does all this go next? Honestly, who knows, but I will share my thoughts anyway.

First off: I dismiss the doomsday scenario that AI will kill us all. Like the Wikipedia page on “pessimism porn” states: A lot of people like to predict disaster because it makes them feel smart, even if years go by and their predictions never come to pass. There are lot of people with blogs and YouTube channels that are always predicting a stock market collapse, or who think we are about to become Weimar Germany if the price of a gallon of milk goes up one cent. They dismiss you if you cannot offer irrefutable proof that the world will NOT end, yet they insist their predictions are to be regarded as self-evident. Granted, maybe those are not the best arguments against Skynet, but I have dealt with a lot of people who confuse the strength of their convictions for logic. Sometimes the best prediction is that things will mostly continue as they are, just with more of something you do (or do not) like.

Since this will be a major change, there will be an effect on jobs. Some jobs will be lost. But there might actually be more jobs due to AI. Scott McNeally pointed out that making a system used to be a masters thesis, and systems were pretty limited. Now we have powerful software that is easy to install. We have packages (like JDK, Golang, Elixir) that are powerful compilers and runtimes, far beyond what people thought was possible a few decades ago, yet they can be downloaded as tar or zip files that once expanded let people create robust, powerful software. Linux and these VMs have created a lot of technology jobs. I think AI might wind up on net creating more than we have now.

Granted, it is possible that the jobs that get created are more soul-sucking than what we have. I joked on Mastodon that AI will not take your job; it will just take away the parts you like, leaving you with the parts you do not like.

There is one group of people that I do hope loses their jobs: all the More Bad Advice pinheads [Note 1] who all sound the same and think the answer to everything is to cut costs. I have had good and bad bosses, but honestly, a lot of people in Corporate America sound the same: asking when things will be done, going on and on about how important some arbitrary deadline they pulled out of thin air is, harping on about innovation yet only having the same tired ideas (piling on more work during the so-called “good times”, then cutting staff when things start looking shaky).

And there will be more people thinking the same. One thing that really grates on me is that we are told in technology that we have to be constantly learning new things. Yet the world is full of business pinheads who cannot conceive of not using Excel. It bugs me that there are lot of people in corporate America And there are plenty of software developers who cannot conceive of doing something in a language that is not Javascript. I have a bad feeling that OpenAI will become the Third Pillar of Technology Stupidity.

Sadly, maybe that will be the way to stay employed. Be a Microsoft drone, a Javascript drone, or an OpenAI drone. I have met tech people older than me who said they could do things decades ago with Lisp and Smalltalk that most languages and runtimes still cannot match. I feel like we took a wrong turn somewhere.

That said, even if AI leads to more jobs, there could still be downsides. We are already seeing this: Generative AI is already being used to craft more effective phishing emails. ChatGPT accused a law professor of sexual harassment (article here, HN discussion here). The HN comments have examples of AI making stuff up, but the professor gave a good summary: “ChatGPT falsely reported on a claim of sexual harassment that was never made against me on a trip that never occurred while I was on a faculty where I never taught. ChapGPT relied on a cited Post article that was never written and quotes a statement that was never made by the newspaper.” What if this is used for background checks and nobody verifies what the AI tells them? This could cause a lot of damage to people. Per the quote misattributed to Mark Twain, a lie can travel halfway around the world before the truth can get its boots on.

We should call AI “artificial inference”, because it mostly makes up stuff that sounds true. It just makes guesses about what seems logical. For a long time it was logical to think the earth is flat. Yet for some reason people think the output of AI is always true. Perhaps they are assuming that it must be true since it is based on large data and advanced technology. But sometimes the machine learning is just machine lying. Marissa Mayer said Google search results seem worse because web is worse (articles here and here). People used to put content on the web to tell you things, and now they just want to sell you things. There is lots of junk on the web. I predict here will be a lot of junk in AI.

Microsoft is putting ads in Bing AI chat which is already fostering distrust in AI (article here and here). Unlike Google search ads, the ads in the chat are hard to distinguish from the rest of the results. If companies need to put ads in AI, then make it like Google ads. People realize that things need to be paid for. Intermingling ads with AI just ruins the AI. You do not need advanced AI to say something you are getting paid to say. Google has been able to serve ads based on user input since 2004.

I think AI will lead to a lot of artificial and misleading content. Not just text, but also audio and video. People might not be able to believe what they read, see or hear online. It could cause more cynicism and distrust in our society. Perhaps we will not get Skynet, just a slow decay and further fracturing of society.

AI could, of course, lead to massive job losses. A lot of people care more about cost than quality. And it is possible that after a time some of those jobs might come back. There is a post on Reddit (link here, HN discussion here) about a freelance writer who lost a gig to ChatGPT. (Another writer wrote an “AI survival guide“.) A few comments gave anecdotes of multiple applications to jobs that all sounded the same that the HR people realized were all done with AI. If more companies start using AI, a lot of websites will all start to be the same. A lot of people hate it when an article “sounds/feels like it was written by AI”. Perhaps the human touch will make a comeback. There is a joke I read somewhere:

An LLM walks into a bar.
The bartender asks, "What will you have?"
The LLM says, "That depends. What is everyone else having?"

Granted, it might be a while before jobs lost to AI come back, assuming they ever do. And not all of the jobs might not come back.

I think that people who understand concepts will do better in the long run than people who just know a tool. At least, that is how things have been. It could be different this time. On the other hand, could an AI come up with “Artisanal Bitcoin“?

Software used to be done in binary or assembly, and over time the languages became more powerful, and the number of jobs increased. Software was always about automation, and there was always something to automate. Has that cycle stopped?

I am worried, but I cannot just yet get on board the Doom Train. I remember working at Bank of America in the 00s/Aughts/Whatever that decade is called, and we all thought that all our jobs would go to India and there would be No More Software Made In ‘Merica. That did not happen.

Or maybe it is all a bubble that will burst.

Maybe the AI is not as advanced as the companies are telling us. OpenAI does not publicize it, but they used people in Kenya to filter the bad stuff (Reddit discussions here, here and here, Time article here with archive here, Vice article here with archive here, another Vice article here with archive here). One major focus of the artcles is that looking at all the toxic content was so traumatic for the workers that the company that got the contract ended it several months early. Looking at toxic content can wear on people. But isn’t the point of an AI to figure this stuff out?

My employer had us watch some videos on up and coming technology, and one of them was on AI. One of the people on the panel kept taking about how important it is to “train” and “curate” your data. They kept saying that over and over. And I had the same thought: isn’t that what the AI is supposed to do? They made it sound like AI was just a big fancy grep or SQL query.

Per the Vice articles, tech and social media companies have been using people in low-wage countries to flag content for years, while letting people think that their technology was so amazing. Perhaps ChatGPT is no different. I do not know if they have to re-flag everything for each version of GPT. I get the impression the data is trained when the AI is started up, and from there it is just repeating what it figured out. Does it actually learn in real-time the way a human can? Can an AI make new inferences and be an old dog learning new tricks the way a human can, or does it just keep re-inforcing the same ideas and positions the longer it runs? What if you train your data and the world changes? What if neo-nazis stop using triple parentheses as an anti-Jewish code today, and your training data is from two years ago? I guess you are just back to web search.

I think part of what is going on is hype. As Charlie Stross pointed out, it does seem interesting that we see the AI hype just starting as the corrupt-o-currency hype is winding down. The vulture capitalists need something new to sell.

Another issue is: will this scale going forward? Technology does not always progress at the same rate. We could be headed for another AI winter. Research into AI for autonomous driving has hit a wall (no pun intended).

And how will this scale? The human brain still have 1000 times the number of connections as GPT-4 has parameters. There is already a shortage forming for the chips used for AI. Is it worth it to burn the planet and use all that metal and electricity to chew through a lot of data…to do what? Simulate a human brain in a world with 8 billion people? Especially when a lot of the humans’ intelligence is not being used efficiently (see penetration of Lisp vs Windows).

That said, I don’t think AI will go away. If I could have one thing, I would like to see alternatives to OpenAI, particularly open source. It might be possible to run LLMs locally. Do you really need an AI that knows about oceanography? Most of us do not. I do not think that AI will kill us all (it is not clear to me how we go from chatbot to Terminator). But corporate consolidation in AI would be a tragedy.

I just need a job where I can use Emacs and call people stupid.

You’re welcome.


Note 1: Not all More Bad Advice pinheads actually have a More Bad Advice degree. However, the ideas pushed by the More Bad Advice crowd are pretty much everywhere.

Image from an 11th-century manuscript housed in the Topkapi Palace in Istanbul, image from The Gabriel Millet Collection (collection page here), assumed allowed under public domain.

2018-05-28 Update: Clojure AI

I am still working through Clojure For the Brave and True.

I am on the exercises at the end of chapter 5, but I might skip a couple of them. I don’t do too well with these “re-implement function x” exercises. I will also start going through the videos on Purely Functional. I think he is raising the price, and my membership will renew at the end of the year.

I am looking into some Big Data/Deep Learning libraries for Clojure. This was inspired by the last Austin Clojure meetup: There were only four of us and we had a hard time thinking of topics. I tried searching for Clojure meetups around the country for topic ideas, and frankly the pickings were kind of slim.

Frankly, sometimes the technology industry in Austin and in general is kind of a drag. If you don’t want to do out-of-date Java or spend your life on the JavaScript-no-CoffeeScript-no-TypeScript-no-now-its-something-else-Script roller coaster, the pickings can be pretty slim. Sometimes I think about taking a year or so off and going though Structure And Interpretation of Computer Programs or How To Design Programs and become a smarter person (not just better with some particular language or API). The issue is that while a lot of people will say things like “we aren’t looking for people with experience in the technology we use; we just want to hire smart people”, they turn around and only hire people with experience in the technology they use.

Anyway, the consensus in the Clojure community is that Clojure needs to be a bigger player in these spaces. There is a LOT of Python in AI. Being a JVM language, Clojure will have wrappers around a lot of the Java libraries I wrote about in Thoughts On Native, GPU, Groovy, Java and Clojure (even though there was not a lot of Clojure in that post).

I know that Big Data and AI are different things. I was thinking about looking at Sparkling to work with Spark (which I hope I can do on my laptop; do you need big machines to work with Big Data libraries?). This weekend I started looking at some of the videos on the Clojure TV channel on YouTube from Clojure Conj 2017. I did not go, but there seemed to be a LOT of videos about AI/Deep Learning (yes, I am using those terms interchangeably even though a lot of people do not).

There was Deep Learning Needs Clojure by Carin Meier, author of Living Clojure. She wasted the first seven minutes on some stupid joke about Skynet, which is a lot for a thirty minute presentation. I am glad I did not pay to see that. After that it gets better. The talk was pretty general. She mentioned some Clojure libraries, Graal VM, and at about 25:00 talks about how to get into Deep Learning.

Declarative Deep Learning In Clojure by Will Hoyt talked about Deeplearning4j. He says that matrices and linear algebra pretty much IS deep learning. There is some neuroscience in this presentation. He also talks about how the Clojure code is easier to deal with than Java and builders. I do not think he ever posts a link to dl4clj, which according to the Deeplearning4j site is the official port.

The Tensors Must Flow by William Piel is about his library Guildsman, which is a new Clojure interface to Google’s TensorFlow. There are already two Clojure projects that allow access to TensorFlow (clojure-tensorflow and tensorflow-clj). They do some Java interop around the TensorFlow Java API. (You can see Google’s not-quite-Javadoc here.) He wanted something that was more idiomatic for Clojure programmers. TensorFlow is written in Python (what else?), which Google then ports to C++ and other languages. But like most Python AI libs, it seems like it is just a wrapper around CPU or GPU code.

I understood the talk when I watched it. Really, I did. From what I remember, TensorFlow uses something called gradients in its process. I think a gradient implementation is an operation in the process. bpiel says the best way to contribute to Guildsman is to actually contribute C++ gradients to TensorFlow itself. He said in the talk he wanted to be done with Guildsman before the Conj in October. It is almost June, and he is still working on it.

The last one about Deep Learning was Building Machine Learning Models with Clojure and Cortex by Joyce Xu. She talked about a Clojure library called Cortex. This library is closer to Uncomplicate, in that while it does interface with a GPU  library, it is not a wrapper around a Java library in the middle.

The traffic on the Cortex mailing list seems to have dropped off, it’s not at version 1 and there seems to be a drop-off in contributions since January.

I do wish the speakers spent a bit more time talking about the implementation details of these libraries. Hosted languages (like Java or Python) do not do a lot of their AI crunching directly. They usually call a native library to calculate on either the CPU or the GPU. And for the GPU, some can do either CUDA (in other words, NVidia) or OpenCL (every other video card). Some libraries have multiple options, like Uncomplicate or Deeplearning4j. TensorFlow can use a CPU (they have instructions on getting the JNI file here) or GPU withy NVidia only. I have not tried Guildsman, so I do not know how he handles things or if he requires an NVidia GPU. I also have no idea how Cortex handles it. Their instructions tell you to get some SDK from NVidia. Perhaps they default to a CPU if there is no NVidia GPU.

I bought my laptop used, and the one I used before this one is at least six years old. I think the older one had an Intel video card, but I could not find any SDK for that version of the video chip. I think my current laptop may also be too old. (Dealing with the Intel Math Kernel is a LOT easier than wading through their OpenCL pages.) The only reason I can think of to buy an Apple laptop is to not deal with this. It is a bit frustrating. The whole point of using a language like Java or Ruby or Python is that I do not want to deal with hardware details.

Anyway, besides all that, I still have a few ideas for a few web apps to do in Clojure.


2018-05-29_22.14.49 update:

I looked at the Cortex mailing list, and apparently you can run the tests just using the CPU:

lein with-profile cpu-only test

It would be great if they put that in the README.

You’re welcome.

Image from M.p.th.f.66, Quattuor Evangelia, a 9th century manuscript on Franconica, the online repository for the Würzburg University Library; image covered under CC BY-NC-SA 4.0  and CC BY-NC-ND 4.0; Imagemagick was used to enhance the colors, shrink the image, and rotate it.

2017-11-06 Update: Deeplearning4J

Not much on the Clojure API front.

I did look at Deeplearning4J, a deep learning framework written in Java. I cloned their repo of examples, and I got some of them working with gradle files, and learned a bit more about Gradle in the process. You can use a GPU, or a CPU. No messing with CUDA if you don’t want to.

There is even a Clojure wrapper for Deeplearning4J as well.

I had a post about using CPUs and GPUs from JVM languages for math/science/AI a while back. Since then, I have gotten both Neanderthal and Deeplearning4J to work on my system. I did mention Bytedeco, which is used by Deeplearning4J and a few other libraries. I looked at some of the Javadoc, and they use a LOT of static methods (see here for an example). I may leave Bytedeco for others and use some of the higher-level libraries, like Deeplearning4J. I think a lot of static methods is not idiomatic Java. I might consult “Effective Java” later for a second opinion.

I am going to include below one of the build.gradle files that I used to get some of the Deeplearning4J examples to work.

You’re welcome.

apply plugin: 'java'
apply plugin: 'maven'
// apply plugin: 'groovy'
apply plugin: 'codenarc'
// apply plugin: 'application'

repositories {
  mavenCentral()
  jcenter()
}

// this has to be at the top so runJava can read mainClassProp
ext {
    dl4jVersion        = '0.9.1'
    nd4jVersion        = '0.9.1'
    jacksonVersion     = '2.6.6'
    logbackVersion     = '1.1.7'
    nd4jBackend        = 'nd4j-native-platform'
    scalaBinaryVersion = '2.11'
    dl4jSparkVersion   = '0.9.1_spark_1'
    awsSdkVersion      = '1.11.109'
    jcommonVersion     = '1.0.23'
    mainClassProp      = 'org.deeplearning4j.transferlearning.vgg16.dataHelpers.FeaturizedPreSave'
}

/*
'org.deeplearning4j.emr.EmrSparkExample'
'org.deeplearning4j.transferlearning.vgg16.FitFromFeaturized'
'org.deeplearning4j.transferlearning/vgg16.dataHelpers.FeaturizedPreSave'
'org.deeplearning4j.rnn.SparkLSTMCharacterExample'
'org.deeplearning4j.stats.TrainingStatsExample'
'org.deeplearning4j.mlp.MnistMLPExample'
'org.deeplearning4j.mlp.MnistMLPDistributedExample'
*/

allprojects {
    dependencies {
        // testCompile 'junit:junit:4.10'
        // testCompile 'org.mockito:mockito-all:[1.8.5,)'
        // From Maven, it's GroupId:ArtifactId:Version 
    
        // logging
        // Using sfl4j allows us to replace different loggers later, if desired
        compile group: 'org.slf4j', name:'slf4j-api', version: '1.7.2'
        // sfl4j is just a facade for loggers. To get logging to work, tie it in with an implementation, e.g., logback
        compile group: 'ch.qos.logback', name:'logback-classic', version: "${logbackVersion}"
        compile group: 'ch.qos.logback', name:'logback-core',    version: "${logbackVersion}"
        
        compile "org.deeplearning4j:deeplearning4j-zoo:${dl4jVersion}"
        compile "com.fasterxml.jackson.core:jackson-databind:${jacksonVersion}"
        compile "com.fasterxml.jackson.core:jackson-annotations:${jacksonVersion}"
        compile "com.fasterxml.jackson.module:jackson-module-scala_2.11:${jacksonVersion}"

        compile "org.nd4j:${nd4jBackend}:${nd4jVersion}"

        compile "org.deeplearning4j:dl4j-spark_${scalaBinaryVersion}:${dl4jSparkVersion}"

        compile "org.deeplearning4j:dl4j-spark-parameterserver_${scalaBinaryVersion}:${dl4jSparkVersion}"

        compile "com.beust:jcommander:${jcommonVersion}"

		compile "com.amazonaws:aws-java-sdk-emr:${awsSdkVersion}"
		// scope: provided
		
		compile "com.amazonaws:aws-java-sdk-s3:${awsSdkVersion}"
		// scope provided
        // compile "org.nd4j:${nd4jBackend}:${nd4jVersion}"
        // for CUDA
        // compile "org.nd4j:nd4j-cuda-7.5-platform:${nd4jVersion}"
    }
}

task runJava( type: JavaExec ) {
    println( "-- Groovy version: " + groovy.lang.GroovySystem.getVersion() )
    dependsOn classes
    description = 'Run gradle.sample.SampleApp'
    // Java main class to execute.
    if ( rootProject.hasProperty( "mainClass" ) ) {
        main = mainClass
    } else {
        main = mainClassProp
    }
    // We need to set the classpath.
    classpath sourceSets.main.runtimeClasspath
    // Extra options can be set.
    systemProperty 'sysProp', 'notUsed'
    println System.properties['iterations'] 
    // systemProperty 'iterations', System.getProperty( 'iterations' )
    systemProperty 'iterations', System.properties[ 'iterations' ] 
    // you can use this if nothing is set via command line
    // systemProperties = System.getProperties()
    jvmArgs '-server'
    // We can pass arguments to the main() method
    // of gradle.sample.SampleApp.
    if ( rootProject.hasProperty( "mainArgs" ) ) {     
        args = mainArgs.tokenize() // args wants a List, so split by spaces if need be
    } 
}

Image from the Trebizond Gospels, a 12th-century Byzantine manuscript housed at The Walters Art Museum, manuscript information here, image from World Document Library, image assumed allowed under Fair Use.

Thoughts On Native, GPU, Groovy, Java and Clojure

One interest of mine is math and scientific computing, although I admit right now my interest is far greater than my knowledge.

It seems like for a lot of math and scientific computing, you either need to do stuff in C, C++ or even Fortran, or use a library in a higher-level language that lets you interface with libraries in C/C++/Fortran. Another new trend is to use code that runs on the GPU, like OpenCL.

Lately I have been looking at a few Java libraries that interface with lower level code. I think it is a good idea to keep learning new things. I think AI and machine learning will become more important over time.

I contributed briefly to the SciRuby project a few years ago. One of their main subprojects is a Ruby wrapper around BLAS, ATLAS and LAPACK. I think some of the instructions for compiling BLAS, ATLAS and LAPACK might still have a few sentences written by yours truly. (It’s a very involved process.) Another one of their projects is a Ruby wrapper/interface around the GNU Scientific Library, or GSL.

One huge language in AI and data science these days is Python, with the NumPy and SciPy libraries. Again, some of the functionality is provided by linking to ATLAS and LAPACK. If it worked for them, maybe it will work for the JVM family of languages.

There is at least one project in Clojure that does this: Uncomplicate. It has sub-projects for OpenCL and BLAS and ATLAS. Along with Java and Groovy, Clojure is a language I am interested in. But for the time being I cannot use these projects. I think my graphics card is a bit too old and the drivers are too old for many of these libraries.

Uncomplicate is part of the reason I am looking at this area again. Somewhere on the site, the author points at the for development, it is okay to use pre-baked packages for ATLAS and LAPACK. One of these projects that I mention says for the best performance, you may want to hire someone to install ATLAS and LAPACK and other native libraries optimized for your production hardware. I did not know people made a living doing just that.

Another reason I am looking at this is GPU computing was mentioned in one of the side talks at the most recent ClojureConj. This is second- or third-hand, but apparently Goldman Sachs runs a nightly report analyzing the market (or some part of it, I am not too clear). It used to take nine hours to run. It was refactored to run on the GPU, and now it takes 20 minutes. I think all developers should become familiar with some of these technologies.

Will any of this gain traction in the Groovy space? I know things have been rough for Groovy for the past couple of years, especially after getting dropped by Pivotal. It seems like a lot of Groovy developers are happy making web apps with Grails. I like Grails, and it has worked out well for a lot of people. I will go through the Groovy Podcast backlog; perhaps Groovy developers are looking into this and I am just not aware of it.

Groovy developers can use the Java wrappers for native and GPU computing. No need to re-invent the wheel. Maybe going from Groovy to Java to native is not quite as fast as going from Python or Ruby to native, but I don’t think so. Even Ruby developers joke that Ruby is pretty slow. They say that sometimes the speed of the programmer is more important than the speed of the program. (But for some reason the Ruby devs who are moving to Elixir are now saying performance and concurrency are important after years of saying they were not. Whatevs.)

I do not have much experience in native or GPU programming. I am not too clear how NumPy and SciPy are used. I don’t want to swallow the Python ecosystem to get the answers to just a few questions, but: Are NumPy and SciPy programs (or bare-metal math/science apps in general) rewritten frequently? Or do they stay the same for years at a time? It seems to me if a GPU program does not change, you might be better off biting the bullet and doing things in C or C++.

But given the success of NumPy and SciPy, perhaps Groovy and Java can be used in this space. Maybe it’s not the end of the world if a bank’s nightly report takes 22 minutes. (How successful is SciPy? Not only are there conventions for general Python, there are also conferences for just SciPy). I doubt any technology will dethrone Python anytime soon, but it is an area to look at.

Having looked a small bit at some of these libraries, I do wonder how much can the unique features of Groovy be used. A lot of the tests deal with arrays of primitive types.  Someone on the Groovy mailing list wrote: I’ve found that you can really shrink down the lines of code just by using the basics that Groovy provides around collections, operator overloading, Groovy truthiness, etc.. Maybe that is enough reason to use Groovy for high performance computing (or at least higher performance). Can we make code calling native and GPU libraries better with features from Groovy, or will we just get Java without semicolons? Some of these functions do not want an ArrayList of objects, they want a fixed-length array of integers. Granted, you can make primitive arrays in Groovy:

def someArray = [[1, 2], [3, 4]] as int[][]

but I think that is more keystrokes than in Java.

Just looking at a small part of TensorFlow, here is a line from the original Java test:

final Tensor matrix = Tensor.create(new int[][] {{1, 2}, {3, 4}})

Here are two Groovy versions that both work (meaning the tests still pass):

final Tensor matrix = Tensor.create([[1, 2], [3, 4]] as int[][])
final Tensor matrix = Tensor.create([[1, 2], [3, 4]])

Granted, it’s one line, but I’d say that’s idiomatic Groovy. Perhaps this could be a (small) market opportunity for the Groovy community. Or maybe I’m just crazy.

If you google “groovy jni” the first hit is this page from Object Partners. It mentions JNA and BridJ which I think are used by a lot of the libraries mentioned below. Frankly, it sounds like a lot of work making this sort of thing happen.

Regardless, even if I have to stick with Java and Clojure, I will still keep an eye on these libraries and this space. I don’t think I can become an expert in all of these, but as I said, I think it is important for developers to keep an eye on this spae. I might start some pages about the different libraries, and perhaps share my thoughts on them. Below are a few notes on what I have found so far.

I started by googling “Java BLAS” or “Java GPU”, and at first I only got a few libraries. But as I kept looking, I kept finding more.

The first two I found where jblas and Aparapi. From the jblas website, jblas “is essentially a light-wight wrapper around the BLAS and LAPACK routines.” (And from what I can gather, the correct capitalization is “jblas”.)

Aparapi is a wrapper around GPU and OpenCL code. If you cannot get your GPU drivers installed properly, it will do everything on the CPU. Sometimes I get core dumps running the tests on my Ubuntu laptop. It is about six years old, and has 6 GB of memory, so perhaps that is the issue. But on my 8 GB Windows laptop two of the tests fail. I plan on getting a new laptop soon, so I will try this again when I do. I bet getting all this set up on an Apple laptop is really easy; for the first time in my life I am thinking about buying an Apple product.

TensorFlow has some Java APIs, although they warn that there may be API changes and breakages for languages other than Python and C. This can use either the CPU or the GPU, but for GPUs it can only work with NVidia cards.

The Lightweight Java Game Library works with OpenCL, OpenGL, Vulkan, EGL and a lot of other stuff I have never heard of. As the name suggests, it is primarily for game development, and the wrappers around the underlying native libraries and standards require more knowledge of those underlying native libraries and standards than most libraries.

There is ND4J (N-Dimensional Arrays for Java) and Deeplearning4j. I am not sure if they are related somehow, but they were both started by developers at a company called Skymind.

And then there is the granddaddy: ByteDeco. There is a lot of stuff here. Some of the contributors work on ND4J and Deeplearning4j, and also work at Skymind. This project provides Java interfaces to 21 different C/C++ libraries: video, math, science, AI, robotics, facial recognition, lots of good stuff here. And after looking at their list, yes, I think “Skymind” is a bit too close to “Skynet”. But at least they’re giving you a hint.

You’re welcome.

Image from an 11th-century manuscript housed at the Bamberg State Library (Wikipedia page here), manuscript description here, citation link here, image assumed allowed under Fair Use.