Rainmakers and Neural Networks

How do we talk to complexity? How do we make complexity talk? How do we force ourselves to hear? How do we make sure we’re not only listening to things we want to hear?

How do we save ourselves?

I’m so sorry. This is an incredibly pretentious way of starting an essay. Our CTO Roger asked me to write something about the interpretability of neural networks. He’s going to be annoyed by everything that follows. The style, the content. Everything. I hope you aren’t though. 

Interpretability is a subject of artificial intelligence that wants to understand neural networks. Networks are how we construct intelligence. There was a time we thought we were smarter and could hand craft intelligence. We’ve mostly surrendered these pretensions. I thought so too when I was introduced to the field. This is what I was taught works:

•  Take small pieces of logic (“logic units”, “neurons”) 
• Arrange them in some networked fashion that you think is useful (“architecture”)
• Throw as much data as you can find on the “neural network” 
• Punish or reward it when it messes up on some task based on the data, or does it well (train). Repeat. Repeat. Repeat
• Voila! You have created a neural network based artificial intelligence

It really is as simple as it sounds. Yes, the details are complex, but the details of anything are complex. We’re avoiding complexity for now, we’ll have a conversation with her later on.

We have gotten very very good at making these things over the past few decades. Researchers and research engineers have thrown increasingly large amounts of data at increasingly large networks called transformers and came up with generative pretrained transformers. Generative meaning they can be set to generate data, pretrained meaning they are trained on massive diverse datasets before being specialised for different tasks. Transformers because they were initially architectured to transform one set of dataset to another (which they still do).

Enough engineering. Back to science. Or pre-science

We’re nearing the top of the intelligence hill having created these massive structures. They can answer any question, try out almost any desktop based task, write code and “empathise”. We’ve made sand sing. I know the voice of sceptics rings loud in your head (“they just predict the next token!”, “AI is not really intelligent”). For this story I’m going to ask you to silence the voices. To help you along,  consider that scepticism is mostly a status flex nowadays. I’d thus ask you to treat AI sceptics with scepticism.  They genuinely don’t understand what’s happening. Neither do I. I’m only partially aware of the complexity of the hill we’re running on.

Neural networks have almost all the hallmarks of complex adaptive systems. A complex system is made out of subcomponents that may themselves be simple, but behave in a manner that is difficult to predict purely by understanding the components. We call this emergence. Adaptivity here means that there’s a dynamic system with some top down, bottom up coherence at play. Examples are your brain, your city, your country and your planet. Neural networks have more in common with the weather than a hammer. So no, they’re not (just) tools. We actually have to sweat to make them act like tools.

Okay. Now we’re on this complex adaptive hill rushing to the top. But we’re getting insecure. “Wait, this thing writes better code than I do!” “Wait a minute, is it actually doing black hole astrophysics?” “Has it solved an Erdos problem? Impressive! What’s an Erdos problem?” “Huh, ati it’s scheming? ”and other singsong questions as we fall to the top. We want to know how and whether we can control them -the alignment problem. To do that it would be helpful to understand what they do, how they do what they do and why they do what they do. That’s interpretability. 

Now what you can’t do is decree and demand. We cannot demand of neural networks that they make themselves explainable. We cannot demand of the engineers to tell us exactly why they said this as opposed to that. We decree and demand when we feel our position weak or weakening.  I decree that the southern border be closed. I decree that the Reich will last a thousand years. I decree that Africa will be one united continent. I decree that string theory is the only useful theory of quantum gravity. Complex adaptive systems are beyond our decrees and demands. Our D & Ds haven’t worked for the nation states we created, why would they work for AI? Neural networks are successful because they’re complex. Because they’re complex they’re not going to be trivially explainable. You can’t have your neural network and grok it. This is a wicked problem, likely permanently out of our understanding and control.

When faced with such a nasty problem, there’s toolbags we’ve developed other than D&Ds. Usually, we first statistics the hell out of it. We take as much data as we can, plot histograms, error bars, significance tests and try to tell stories. Those of us who are particularly brave will confuse statistics for structure, calling these networks  “stochastic parrots”, “statistical look up tables”, and “prediction engines”.  This shadow chasing is insightful, but only for a while. Soon enough the statistics will stop running, look at you blankly and say “I wasn’t structure, only a shadow of structure.”  Some try to tell stories using the shadows. Let’s call them taxonomists.  

Taxonomists are more cautious. They know some structure exists, but they know they don’t know enough, or are not enough, to say what it is. So they collect data assuming structure. “This is like those.” “This is not like the others.” “Let’s put these together.” “This group is actually two.” And so on. I’d put the geographers here too.  The honest explorers whose calling is to make maps. Then hopefully their children will find meaning in the maps, see that all the continents were once one. It’s possible to be careless here too. Remember that you’re on a boat seeing only one side of the continent. Don’t confuse Africa’s kisogo for Africa. Also definitely don’t confuse the map for the territory. You’ll paint your country’s colours over a territory you don’t understand and can’t control. We’ve been horribly wrong about complex adaptive societies before. It’s possible for us to be existentially wrong about neural networks.

The people who most knock my socks off are the general theorists. I’ve always fantasised of joining this 80,000 light-hours club but I’ve neither the ability nor the opportunity. These ones bravely start from the story. They make a set of minimum assumptions, then use logic to construct a system which behaves on the whiteboard exactly like the system in reality. When they do their work very well they find genes, not shadows. As a bonus they tell you how the genes generate generations. Darwin, Einstein,  Noether, Mandelbrot, Allotey. From the names, it’s incredibly hard to be a general theorist. The physical world resists simple and powerful stories. And yet it is simple. This heresy is seductive and addictive and draws so many. Too many.

For neural networks like any other complex system, this is doubly difficult. You don’t hear of a general undisputed theory of the economy, law, sociology and anthropology for a reason. Any such pretenders are likely to be totalising narratives with violent exception handling. While I learn my ropes around general theories of superintelligent agents and theoretical alignment and interpretability, I also recognise that this is likely to be a huge waste of time. Even catastrophically counterproductive. The Machine Intelligence Research Institute was once chock full of these aspirational general theorists. It has given up on the hopes of developing a general theory of intelligence and is now doing technical governance and advocacy, to some D&D. And yet, it is simple.

Now come the much promised rainmakers. I received an African education, and we learned of them. I thought, and still do, that most of it was superstition, guesswork and knowledge control. There’s a germ of rainmaking that I now find particularly interesting. Growing up in the western part of Kenya, my late elder brother Evans told me that winged termites (or in binomial nomenclature: kumbe kumbe)  are a sign of rain. I found this deeply mystifying. Yes I grew up later. Yes, I learnt that termites are sensitive to slight humidity rises. But I imagine myself in my Bantu ancestors’ minds humbly but carefully observing the environment. Making connections between two interacting complex adaptive systems: the weather and the ecosystem. They would then adjust their complex system societies to this knowledge.  It is these connections that are deeply pleasant to the soul. I don’t know what I’m seeing. But yet it can inform.

Owain Evans and the team are the ones closest to rainmaking. They finetuned an otherwise safe neural network to write unsafe code. They then observed that the network also began behaving unsafely on normal chat based requests and responses. Breaking code generation safety breaks general behaviour. I find this  work exhilaratingly beautiful. These nontrivial connections between components of complex systems are in fact the most informative. Another seductive example is the observation by Anthropic interpretability researchers that a version of Claude had formed a connection between nuclear reactors and compost heaps. Ie that heating them up leads them to further heating up, called “positive feedback” by us stuffy ones. Ah!

If you read their work, you will see that the rainmakers pay homage to the statisticians, taxonomists and general theorists. Like rainmakers of old, they’re not averse to pragmatism, their work demands it. So they use what they can to get what they can. However there’s not a complete story, an addiction to structure or a chasing of numbers. There’s a partial story coming from semirigorous numbers telling of an incomplete structure. Maybe that’s all there is, or will be. So the rainmakers should be careful lest their program degenerates to superstition and guesswork. I wish them the best and hope to learn their methods. They’re reaching for the future. They know best how to speak in complexese.

The monsoon is coming. Where are our rainmakers?