03. Large Language Models (LLMs), GPTs, and the Architecture

Published 2026-09-14

Core Theme: A large language model is not an encyclopedia in which truth is stored. It is a powerful probabilistic engine that has captured the patterns of human language in mathematical vectors and matrices — and predicts the most plausible next word.

AI, generative AI, LLM, GPT — what is the difference?

In ordinary conversation these four terms are used as though they were synonyms. They are not. They sit inside one another like nested boxes, and knowing which box you are talking about prevents a surprising number of expensive misunderstandings — including the common one of buying a chatbot when what the organisation needed was a forecasting system.

Figure 3.1: The conceptual hierarchy from AI to GPT — Artificial Intelligence (AI) → Generative AI → Large Language Models (LLMs) → Generative Pre-trained Transformers (GPT).

Artificial intelligence is the outermost box — the entire field of computers imitating human intelligence. It is roughly eighty years old and includes a great deal of work that has nothing to do with chatbots: chess engines, credit-scoring systems, the software that reads a bank cheque.

Generative AI is a subset of that field. Its defining property is that it produces new content — text, images, code, sound — rather than classifying or scoring content that already exists. The distinction between analysing and generating is the one that matters most in practice, and Chapter 02 sets it out in full.

Large language models are one kind of generative AI, specialised in language. Such a model has consumed effectively all the digitised writing available to it and extracted, mathematically, the patterns by which human language holds together. It is worth being precise about what was extracted: not the facts in those documents, but the statistical shape of how words follow one another. That distinction returns later in this chapter, and it explains almost everything people find puzzling about these systems.

GPT is an architecture, not a product — a Generative Pre-trained Transformer. It is the particular neural network design that made modern language models work, and it traces to one research paper published by Google researchers in 2017: Vaswani et al. (2017) — "Attention Is All You Need". The name that paper gave to its central mechanism is the reason these systems can handle context at all.

The practical point is that this is one family, not several competing inventions. OpenAI's ChatGPT, Google Gemini, Anthropic Claude and DeepSeek all run on the same Transformer principle. They differ in scale, in training data and in the values instilled during fine-tuning — which, as the section on Sinhala below shows, has substantial consequences for an officer working in this country. The full technical evolution is charted in Idasara Academy's published article The Gen AI Evolutionary Journey: 2017 to 2035.

Why did engineers copy the brain?

Before going inside these models, a question worth pausing on: why neural networks? Of all the ways one might build a machine that handles language, why did computer scientists settle on a mathematical imitation of brain tissue?

The answer is a principle that runs through the whole history of engineering: biomimicry — solving a hard technical problem by borrowing a design nature has already spent millions of years refining. And the clearest demonstration of it has nothing to do with computers.

Japan's Shinkansen Bullet Train and the Kingfisher's Lesson

The finest practical case of biomimicry on record comes from Japan in the 1990s.

As Japan's high-speed Shinkansen bullet trains exceeded 300 kilometers per hour, they ran into a serious aerodynamic and environmental crisis. When a train entered a narrow tunnel at extreme speed, its front face compressed the air inside the tunnel like a piston. That compressed air wave burst out of the far end of the tunnel with a deafening, thunderclap-like explosion — the sonic "tunnel boom." The noise rattled the windows of village homes a kilometer away and tormented residents. Because legal noise limits were being breached, the engineers were forced to slow the trains down.

Notice the shape of that problem, because it recurs everywhere: a technology that works beautifully in the open fails at a boundary, and the constraint that stops it is not technical but social. The engineers were not defeated by physics. They were defeated by the people living beside the tunnel, and rightly so.

The solution came from Eiji Nakatsu, a chief technical engineer at the JR-West railway company — and a devoted birdwatcher.

Observing nature, Nakatsu posed one extraordinary question: "Which creature plunges at high speed from low-density air into high-density water — without a splash, without a wave, without a sound?"

That question is the whole method, and it is worth studying as a piece of thinking. Nakatsu did not ask how to make a quieter train. He abstracted his problem until it stopped being about trains at all — a body moving at speed from a thin medium into a dense one — and then searched for anything in nature that had already solved it. The abstraction is what made the search possible.

The answer was the kingfisher.

When a kingfisher dives from the sky toward a fish, its long, conical, streamlined beak parts the water's resistance smoothly to either side. The result: no splash, no shock of resistance. Nakatsu also noted how the fine serrations on an owl's wing feathers absorb aerodynamic noise.

The 15-meter nose of the Shinkansen 500 Series, unveiled in 1997, was designed by computing the exact aerodynamic geometry of the kingfisher's beak. The results were astonishing:

1. The tunnel boom was eliminated entirely — the train could run within legal environmental noise limits.
2. Aerodynamic drag fell by 30%.
3. Electricity consumption fell by 15%.
4. Speed rose by 10%, sustaining continuous operation at 300 km/h.

Read those four outcomes together and note something important: only the first was the goal. The engineers set out to fix a noise complaint and incidentally gained thirty per cent less drag, fifteen per cent less electricity and ten per cent more speed. Designs borrowed from evolution tend to behave this way, because natural selection was never optimising for one variable in isolation.

Figure 3.2: Biomimicry in practice — the aerodynamic nose of Japan's Shinkansen 500 Series bullet train, modeled on the conical beak of the diving kingfisher. It eliminated the tunnel boom entirely and cut aerodynamic drag by 30%.

From Mechanical Biomimicry to Cognitive Biomimicry: Neural Networks

Japanese engineers borrowed a bird's form to solve a train's aerodynamic problem. Computer scientists did something structurally identical one level up: they borrowed the human brain's biological neural network to solve the problem of recognising the patterns of language and cognition. That is cognitive biomimicry — the same method applied to thinking rather than to movement.

The thing being copied deserves a moment's respect. The human brain holds roughly 86 to 100 billion biological neurons, connected by hundreds of trillions of synapses. When a small child learns words, no rulebook is involved. Electrochemical signals travel through the brain, and connections between neurons that activate together grow stronger — "neurons that fire together, wire together." Nobody teaches a three-year-old the rule for forming plurals; the child absorbs the pattern from exposure and then over-applies it, saying "mouses" before saying "mice."

That observation is precisely what pushed engineers away from the obvious approach. The natural instinct would be to teach a computer language the way traditional software does everything: explicit rules, if-else logic, a grammar book in code. Every serious attempt to do that failed, and failed for a reason worth understanding — language is too flexible and too dependent on context to be captured in rules. The word "bank" is not ambiguous when you know whether the sentence is about money or a river, and no finite rulebook covers how you knew.

So the engineers reconstructed, in mathematics, the way neurons and synapses behave: the artificial neural network. The models we call GPTs and LLMs are enormous instances of that idea, with billions — now trillions — of parameters. They did not learn language from a grammar. They learned it the way the child did, from exposure, at a scale no child could survive.

Figure 3.3: Cognitive biomimicry — a direct comparison of the brain's biological synapse and electrical signaling (action potential) with the mathematical construction of an artificial neural network: input layer, hidden layers 1–3, output layer, and synaptic weightings.

The parallel is close enough to be tabulated component by component, and reading it this way makes the vocabulary of AI considerably less intimidating — every strange technical term below has a familiar biological counterpart.

The 1:1 Engineering Parallel Between the Biological Brain and the Artificial Neural Network (LLM):

The row to dwell on is the second. When you read that a model has "405 billion parameters", the table tells you what that number means: 405 billion synaptic strengths, each a number adjusted during training. Nothing else is stored. There is no database of facts inside a language model — only the weights.

Evolutionary Optimization and the Critical Boundary Line

Two lessons sit inside this analogy, and the second is the more important of the two.

First, why nature was worth copying. Humanity did not need to invent network architecture from first principles. Across hundreds of millions of years of natural selection, evolution produced in the brain an unmatched biological computer — one that captures a staggering range of patterns on roughly 20 watts of power, making it among the most energy-efficient machines in the known universe. Twenty watts is a dim light bulb. The data centres attempting to approximate a fraction of that capability consume megawatts, which is a useful measure of how far the imitation still has to travel.

Second, and decisively, where the analogy stops. Any leader of public administration needs this boundary drawn sharply:

Shaping the bullet train's nose like a kingfisher's beak does not turn that iron train into a living kingfisher that flaps its wings, flies, and lays eggs. It remains a mechanical train running on rails.

In exactly the same way, building a large language model mathematically on the blueprint of the brain's neural structure does not grant that computer model biological consciousness, subjective feeling, intention, or understanding. What it does, still, is a statistical computation — estimating what the next word is likely to be, through the mathematical weightings of a neural network (P(w_t | w_<t)).

The reason this matters is not philosophical fastidiousness. It is that a system which sounds like a person invites you to treat it like one, and for an officer that leads directly to two errors: believing what it says because it sounds confident, and treating it as answerable for a decision that remains entirely yours. Anthropomorphism is not a harmless indulgence. Getting this boundary right is what lets you take the enormous efficiency on offer without inheriting the delusion that comes attached to it.

How is an LLM actually built?

A familiar comparison makes the construction easier to picture. Consider the Google search engine.

Google Search reads essentially every website on the internet and converts the words into an enormous table of contents — an index. When you search for a term, it consults that index and hands you the most relevant link. Crucially, what it returns is a pointer to a document someone else wrote. Google stores locations, not answers.

A large language model is built on a comparable scale but on a mathematical plane hundreds of thousands of times more complex, and the difference in what gets stored is the whole story.

First, the model reads the internet, along with every digitised book, research paper, dictionary, law report and conversation available to it. Second, it breaks all of that text into small fragments called tokens. Third — and this is where it parts company with a search index — because the machine understands neither English letters nor Sinhala ones, it converts every token into numerical values and builds an immense mathematical model of how the words of a language fit together.

What survives that process is a structure of numbers describing the relationships between fragments of language. The original documents are not in there. This is why a language model cannot give you a citation the way a search engine gives you a link, and why demanding one produces an invented reference rather than a refusal. The model has no location to point at. It kept the pattern and discarded the page.

What is inside the Transformer brain?

The question that follows naturally: what is doing the work? The answer is the multi-layer Transformer architecture — and its raw material is mathematics that Sri Lankan students already study.

Ask a student sitting G.C.E. Advanced Level Combined Mathematics where the matrices and vectors on their syllabus are used in the real world, and most cannot say. The honest answer is that the technology currently reshaping the global economy runs on precisely those two objects. Vectors position words in meaning-space; matrix multiplication is how the relationships between them get computed, billions of times a second. There is no more advanced mathematics hiding behind it — only the A/L syllabus, executed at industrial scale.

Figure 3.4: The fundamental architecture of a large language model (LLM) — the massive dataset, tokenization and embedding, the Transformer brain, the training process (pre-training & RLHF), and inference output.

Stage one: the massive dataset

The raw material is digitised books, encyclopedias, text documents, the open web and code repositories. Before any of it reaches the model it is cleaned — duplicates removed, quality filtered, false and toxic content stripped out.

That cleaning step deserves more attention than it usually gets, because it is a series of human editorial decisions that shape everything the finished model believes. Someone decided which sources counted as quality and which did not. The model inherits those judgments permanently and invisibly, and the section on Sinhala below is a direct consequence of exactly this stage.

Stage two: tokenization and embedding

Tokenization breaks sentences and paragraphs into small fragments. A token is usually somewhat less than a word — common words are single tokens, rarer ones split into pieces. This is also, incidentally, the unit you are billed in.

Vector embedding then gives every token coordinates in a mathematical space of thousands of dimensions, positioned so that words close in meaning sit close together. "Divisional Secretary" and "District Secretary" end up as near neighbours. That single property is what allows a model to handle a question it has never seen: it does not need the exact phrase, only something nearby in the space. It is also why a model can confuse two things that are similar in usage but critically different in law — proximity in meaning-space is not identity.

Stage three: the Transformer architecture

This is the brain, and it has four working parts.

Self-attention computes the contextual relationship between each word in a sentence and every other word. In "He signed the permit and gave it to the Secretary," the mechanism is what determines that "it" refers to the permit and not the Secretary. This is the mechanism the 2017 paper was named for, and the reason these models handle long documents in a way their predecessors could not: every word is weighed against every other word, simultaneously, rather than in sequence.

Encoder and decoder layers perceive the input and produce the output words in order.

Add and norm layers keep the mathematics stable as information passes through hundreds of neural layers, preventing the signal from fading away — the vanishing-gradient problem that limited neural networks for decades and whose solution is a large part of why depth became possible at all.

Feed-forward networks process and store the patterns, capturing the complex regularities that self-attention surfaces.

Stage four: training and learning

Pre-training is unsupervised: the model reads trillions of words and repeatedly predicts what comes next. Nobody labels the data, because the text is its own answer key — hide the next word and you have a question with a known correct response. From this single exercise, repeated at unimaginable scale, the model absorbs grammar, logic and a great deal of world knowledge as a side effect.

Fine-tuning with human feedback, known as RLHF, comes after. Human experts rank the model's answers and the model is adjusted toward responses that are appropriate, safe, courteous and truthful — aligned with human intent, including the standards expected in public service. Pre-training produces capability; RLHF produces a system that behaves like an assistant rather than an autocomplete. It is also, being human, the stage where cultural assumptions enter most directly.

Stage five: inference and generation

Finally, the trained model meets a real user. Given a prompt, it generates output using probabilistic decoding strategies — temperature and top-p sampling — which control how adventurous its word choices are. This is why the same question asked twice produces two different answers: the process is deliberately probabilistic rather than deterministic. Useful for drafting. Something to be aware of before relying on a single run for anything consequential.

How does a model learn?

The training process above is worth restating on its own, because "the model learns" hides a mechanism that is entirely unmysterious.

Pre-training works by prediction and penalty. The model is shown billions of sentences with a word hidden and made to guess what comes next. When it guesses wrong it is mathematically penalised — the backpropagation step — and its weights are adjusted so that the same mistake becomes marginally less likely. When it guesses right, the pattern is reinforced. Run that loop trillions of times and what accumulates is an extraordinarily detailed statistical map of human grammar, logic and knowledge.

There is no instruction anywhere in this process, and no comprehension. The model is not told what a verb is, that Colombo is in Sri Lanka, or that a pension has eligibility rules. All of it arrives as a by-product of getting better at predicting the next word — which is genuinely remarkable, and also the reason the knowledge is uneven in ways that are impossible to predict from the outside.

Reinforcement learning from human feedback then shapes behaviour rather than knowledge. Human reviewers grade the model's answers, steering it toward responses that are more courteous, more useful and more accurate. This is the stage that turns raw capability into something usable — and the stage where a model acquires its characteristic manner, including the relentless helpfulness that makes it so reluctant to say "I don't know."

Why is an LLM a plausibility engine, not a truth engine?

For a public officer, this is the single engineering fact to engrave permanently in mind. Everything else in this chapter is background; this is the part that changes how you work.

The model has no internal sense of true and false. When you ask a question, it does not consult a store of verified facts. It constructs, on mathematical probability, the most convincing and fluent sequence of words that could plausibly follow your question.

Follow that mechanism to its conclusion and the most dangerous property of these systems becomes obvious rather than surprising. Fluency and accuracy are produced by the same process, so they arrive together and cannot be separated by inspection. A correct answer and a fabricated one are generated identically, expressed with identical confidence, and formatted in identically beautiful official prose. There is no tremor in the voice on the invented circular number. The hallucination is not a malfunction in which the machine has failed to work properly; it is the machine working exactly as designed, on a question where the plausible answer and the true answer happened to diverge.

This reverses the ordinary human heuristic, and that is precisely why experienced officers are caught by it. We are trained from childhood to read fluency and confidence as evidence of competence, because in humans it usually is — a person who speaks precisely about pension regulations has generally studied them. That inference is sound for people and worthless for models. An officer must never be seduced by the model's form, and must always verify its truth. Chapter 08 sets out how to do that verification systematically.

Why do models differ in Sinhala and Tamil?

A complaint surfaces in every session with officers, and it is entirely fair: ask ChatGPT something in Sinhala and the answer is sometimes nonsense; ask the same thing in English and it is superb. Why?

The reason is not that the model is careless in Sinhala. It follows directly from stage one of Figure 3.4 — the training data — and once that connection is clear, the differences between models stop being mysterious and become predictable.

Western knowledge bias

American models such as ChatGPT and Claude are trained overwhelmingly on the Western world's English literature, Western law and Western data. What they hold about Eastern cultures, Sri Lankan administrative circulars, or the texture of our rural life is a sliver by comparison.

This produces two distinct failures, and the second is the dangerous one for an officer. The model may write awkward Sinhala, which is obvious and therefore harmless. Or it may write fluent, confident English about a Sri Lankan circular or administrative rule it has almost no data on — and that failure is invisible, because the prose quality gives no signal about the knowledge behind it.

Google Gemini's Sinhala advantage

Relative to the other American models, Google Gemini has a marked advantage in Sinhala, and the reason is straightforwardly historical. Google has spent more than twenty years indexing Sri Lanka's Sinhala and Tamil websites, news and search queries. That accumulated corpus feeds the model, and the result is noticeably better Sinhala grammar and word usage. Twenty years of quiet indexing turns out to be an asset no amount of recent investment can quickly replicate.

The surprise of China's DeepSeek

Work in Sinhala with China's DeepSeek and it understands surprisingly well — which puzzles people until the reason is named. Chinese is not a simple alphabetic language like English; it is a system of intricate pictographic characters and tonal patterns. A model trained deeply on Chinese and on broad non-English multilingual data develops a genuine capacity for complex, non-European sentence structure, and that capacity transfers to Sinhala.

The general lesson is worth extracting, because it is useful well beyond this one example: a model's strength in your language depends less on how much of your language it saw than on how much structurally similar difficulty it was trained to handle.

Why Tamil fares better than Sinhala

Officers working in Tamil have a comparatively easier time with these tools, and the arithmetic explains it. Tens of millions of Tamil speakers worldwide, including in India's Tamil Nadu, have contributed an immense body of Tamil literature and digital material to the internet. Sinhala is spoken by roughly twenty million people, almost all within one country, and the volume of Sinhala content published online is very limited.

That is not a statement about the languages. It is a statement about digital footprint, and it has a direct operational consequence: special vigilance is required when using AI in Sinhala, particularly for anything factual. It also points at something constructive. Every well-written Sinhala page published online becomes training data for the next generation of models. The scarcity is not permanent, and it is partly within our power to fix.

What does multimodal actually mean?

These models are no longer confined to text. Frontier systems now operate as fully multimodal architectures, and the engineering behind that word is more interesting than the marketing suggests.

Figure 3.5: The architecture of a multimodal generative AI system — multimodal data input (text, photos, videos, audio, code), the unified multidimensional vector space, the multi-headed Transformer, and multimodal output.

Ingestion without a text bottleneck

The training material now spans text documents, photographs, video, audio recordings and code. The significant change is in how non-text material enters the system.

Earlier approaches converted an image to a text description first and reasoned over the words — which meant everything the describer failed to mention was permanently lost. Modern systems skip that intermediate step. Photographs and video frames are broken into small pixel patches, typically 16×16, and become visual tokens directly. Audio waves become audio tokens by way of frequency spectrograms. Each modality enters on its own terms, and nothing is discarded in translation.

The unified vector space

This is the crowning achievement of the architecture, and the idea worth understanding even if you skip everything else in this section.

A photograph of a Sri Lankan National Identity Card, the spoken Sinhala phrase "ජාතික හැඳුනුම්පත", and the written English text "National Identity Card" all land in the same coordinate region of one multidimensional mathematical space. Three completely different physical signals — light, sound, text — reduced to positions in a shared space, close together because they mean the same thing.

Once that is true, comparison across modalities requires no special machinery. The model is not translating between image and text; it is measuring distance between two points that were always in the same space. That is why you can show a system a photograph and ask a question about it in Sinhala.

Cross-attention between modalities

The multi-headed cross-attention architecture then lets attention operate between modalities rather than only within one. Examining the signature on a scanned deed, the model can simultaneously weigh it against the landowner's name written in the text of the same document. Two sources of evidence, considered together, in one pass.

Multimodal training and safety alignment

Training proceeds on mixed video, audio and text through next-data-type prediction, and the fine-tuning stage carries an additional burden here: safety guidelines and protective filters against forged media. This is not an optional extra. A system that can generate a convincing human voice from a sample is the same system that can be used to impersonate a citizen on a telephone call, and the defences against that misuse have to be trained in rather than added afterwards.

Generation in any medium

The result is a system to which the user can pose a query in any medium — an audio recording of a spoken complaint together with a photograph of disaster damage — and which can produce, through its decoding and sampling strategies, whatever the task requires: an official Sinhala report, a damage-annotated map, or working code.

What this gives the public officer in practice

The capability becomes concrete very quickly, and these three examples are worth trying rather than reading.

Photograph an old handwritten land deed or letter and have its contents read back in a second — vision-based optical character recognition, working on handwriting that defeats ordinary scanning software. On a public day, record a citizen's spoken grievance and produce an official inquiry note immediately, while the citizen is still in front of you. Feed in the map or chart of a complex development plan and have its inconsistencies identified, where a reader would take an afternoon to find them.

Each of these replaces a specific, tedious, error-prone task that currently consumes hours of an officer's day. None of them removes the need for a human to check the result.

In the next chapter, we discuss how this natural language revolution is bringing the humanities back to center stage — and how AI closes the digital divide facing the ordinary citizen.