Compression: Why Haki Cares About Knowledge Representation

Representative democracies are fascinating ideas. In a way, they are a recognition of the successes and failures of scale. Want to go to the moon? Your village is not the place to be. Want to build a billion dollar telcom company that doubles up as a bank? Get out of Kiondoo.  You want peace, community, and franchisement? Stay in your village, or in acephalous societies like the Igbo, !Kung or Hadzabe. Large societies will likely compress your agency, give it to the communist party, the cult of Ra or Safaricom. It’s not about you.

What the state, religion and company claim to do is represent your personal political, religious and economic interests. Not only can they do this for you, but for hundreds of thousands of people like you. Representative models claim to do this particularly well. This is from four or five year updates which allow iteration. Is the member of the national assembly not a good representative? Get rid of him. This way your voice scales, your wants are heard, and you get to build a powerful nation state, apparently.

I would like to pose it to you that your brain as a learning system does something similar. There’s no hope of your brain absorbing the universe, let alone its immediate environment. What our brain then does is build internal signals which correspond to the external environment. We can call these internal signals, representations. It is not necessary that the representations have any physical similarity with the externally real objects they represent. In fact this is severely wasteful and may not even work. The physical space and energy you need to build a chair is not the same energy you need to build an idea of a chair in your head.

It’s not enough for brains to build representations. You must also build a story for what the giraffe will do to the tree as opposed to you, rapacious carpenter. This needs a system of interpretations. The system specifies how certain representations interact, or are organised into a (hopefully, but almost never) coherent whole. This system is called a model. A model is a compression of an external environment as representation is a compression of its components. Remember that this is completely fractal. A representation at one level can be itself a system on another scale. 

A forest can be a representation when considering climatic interactions, but a system when talking about tree biology. Similarly for a tree when considering tree anatomy, and a plant cell when considering cell biology and so on. This is important, because the ability to form, enrich and unpack representations is central to learning. This is done dynamically by the brain throughout its life. 

A deep fact about models is that they’re never perfect. General relativity is not a perfect model of gravitation.  Evolution by natural selection does not exhaustively explain all biological phenomena. Keynesian economics won’t help you predict the almighty NVDA stock price. Models compress environments by choosing what is important and what isn’t. This is another important notion called abstraction. You have to get at a certain level of representations and stop zooming in and use that as your “fundamental”. At some point the sand-counting must stop, and focus shifted to the bricks, and the house made from the bricks, and the estate made from the house. It doesn’t matter that the MP has a few thousand more neurons (probably, less) than you. It matters that they’re representing you effectively. All models are wrong because all representations are incomplete.

Models are also incomplete for a more fundamental reason, they can’t model (forgive the pun) any interaction out here in the real world. Part of this is that the representations of any object are incomplete and can’t cover all the possible interactions of the object. There’s a much deeper limitation though. Remember the fractality of representations, which can themselves be considered systems? Well the model is just itself an imperfect representation of a system in the universe and therefore is certainly imperfect. There’s no hope for a perfect model.

Okay we have a system of representations abstracted into a model which itself is a representation which compresses the world. This is getting exhausting, but it’s also getting somewhere. If you would like your continent, country, and brain to be a competent actor in the world, building models of the world is your only route. Of course, representations are the bricks with which you build this house. 

AI companies and researchers are seeking effective representations of the world to put them into computers. They have developed two traditions of thinking: symbolism and connectionism. A symbolist believes that representations should be rich enough to carry out the functions we have described: 

1. Close enough to the actual object to convey meaning to the cognitive agent merely by the consideration of the representation
2. Combined into a model which effectively represents the real system

A representation becomes a symbol if it satisfies I and II. Symbolists believe in the explicit construction of symbols, there are entire books dedicated to this process. Of particular interest to Haki is knowledge graphs. Knowledge graphs are networks of symbols promoted into concepts by the mere fact of their existence in a network, the knowledge graph. Problem-solving in a particular domain reduces to searching over these symbols. Intelligence can thus be read as compression through symbolic representations, and search over the model built using the symbolic representations. Because of the primacy of these symbols, symbolists focus on crafting symbols and systems of symbols. 

In fact the primary difference between symbolists and connectionists is in how to construct these representations. While symbolists focus on handcrafting (the fancy-schmancy term for one who does this is a “knowledge engineer”), connectionists craft the architecture and learning signal, leaving the system to learn its own representations and models. Symbolists and connectionists often rarely talk to each other, sadly. When they do, they hold their noses.

Building usable AI software for the continent, Haki has no such tribalisms. We’re the trader selling Igbo-Ukwu art in Kiondoo. We have to use what we can get, and get what we can use.  Learning how symbolists and connectionists think about representations and models is central to our research and development programs. We are exploring neurosymbolic-in-architecture processes in addition to mature neurosymbolic-in-pipeline architectures and research. 

Thinking about representations here is particularly tricky. Representations in connectionist systems can be sub-symbolic, in the sense that they need to be combined with other representations to become soft symbolic. In fact, this is the basis of the power of connectionist systems, that they are parallel distributed processors. Nonetheless, we’re running experiments on injecting symbolic systems into neural network layers. This is a little easier on deeper layers, but we need to think carefully about how to inject or propagate symbols, and symbolic processes into ever shallower layers.

Nonetheless, the proverb springs eternal: “Seek ye effective representations and all these shall be given unto you.”

Stay tuned!