Knowledge Graphs, Language Models, and Knowledge Representation

Schema Light, Ontology Free Knowledge Graphs

TL;DR

• Turning long, messy legal text into usable structured knowledge is still brittle and ontology-heavy classic name entity recognition and relation extraction often misses what actually matters in a case and doesn’t scale cleanly across domains and languages.

• We use attention rollout to identify salient entities and their token-to-token influence, then convert those signals into a weighted influence network where graph centrality helps select candidate nodes and relations; optionally letting an LLM propose human-readable relation labels.

• On an initial small sample, the approach produces a graph that surfaces key actors and suggests relations, pointing toward a path to schema-light / ontology-free KG construction that we plan to scale to the full corpus.

Karibu! 

This will be our new research blog, summarising the things we’re interested in, working on and yes, abandoning.

A significant fraction of our research and development is dedicated to the problem of structuring knowledge for use in large language models (LLMs) and other forms of artificial intelligence. We believe that long term capabilities of AI will be downstream of our ability to develop expansive, multimodal knowledge representation systems.  We are investigating multiple schemes for ingesting, injecting and retrieving knowledge, ranging from traditional scaffolding approaches such as retrieval-augmented generation (RAG) to more explicitly structured representations based on knowledge graphs[1]. Across the field, structured knowledge retrieval has become a central concern, reflecting growing recognition that unstructured text alone is an insufficient substrate for robust reasoning.

Building on earlier knowledge-graph designs, most notably those associated with the semantic web, a major technological development has been the synthesis of retrieval-augmented generation with graph-based representations. These so-called Graph-RAG architectures aim to combine the precision of symbolic knowledge representation with the broad semantic coverage enabled by similarity search in high-dimensional embedding spaces[2].

There are several ways to achieve this synthesis. A common approach embeds selected features of a knowledge graph into a vectorisation scheme and performs semantic similarity search over graph nodes, allowing controlled jumps to neighbouring nodes, or “hops”, to incorporate contextual structure. One example of this approach is Spanner Graph, developed by Google[3]. Alternative schemes avoid explicit embeddings altogether, instead relying on an LLM to perform structured search over a predefined knowledge graph. Collectively, these approaches fall under a broader research program often described as knowledge graphs for LLMs[4].

A complementary, and in some sense reverse, research direction can be described as LLMs for

knowledge graphs[5]. Here, the representational and generative capacities of language models are used to construct, refine, and extend graph-structured knowledge. Because LLMs encode rich models of linguistic and semantic structure, they can be used to specify ontologies, prune or grow graphs, and extract compressed representations of knowledge from raw textual data. Most uses of LLMs 

At Haki Research, we have developed an in-house approach that exploits the attention

mechanisms of modern LLMs to extract knowledge graphs from unstructured text using

schema-free methods. This approach is motivated by three working hypotheses:

• LLMs possess sufficiently rich internal representations [6] that can be mapped to arbitrary chunks of text observed at inference time.
•  These representations, when combined with attention scores [7], can be used to generate saliency measures for entities appearing in the text.
• Saliency scores are related to graph-theoretic notions of centrality for the corresponding entities[8].

Methodologically, we make use of attention rollout techniques from mechanistic interpretability[9]. We developed our pipeline using data obtained from the National Council of Law Reporting, a repository of 250,000 legal case documentation files in Kenya. For this pipeline, we only worked with 15 documents. Chunks of text consisting of a few hundred tokens each are then piped to the language model. Attention rollout allows us to compute attention scores for all tokens within a given text chunk, aggregating attention across layers successively to derive mutual influence scores.  This can be summarised in an influence matrix shown below.