03. Large Language Models (LLMs), GPTs, and the Architecture

Core Theme: A large language model is not an encyclopedia in which truth is stored. It is a powerful probabilistic engine that has captured the patterns of human language in mathematical vectors and matrices — and predicts the most plausible next word.

1. The Conceptual Hierarchy: AI, Generative AI, LLM, and GPT

In conversation, many people use AI, Generative AI, LLM, and ChatGPT interchangeably. In engineering terms, however, they form a clear hierarchy:

Figure 3.1: The conceptual hierarchy from AI to GPT — Artificial Intelligence (AI) → Generative AI → Large Language Models (LLMs) → Generative Pre-trained Transformers (GPT).

• Artificial Intelligence (AI): the entire broad field of computers imitating human intelligence — the umbrella.
• Generative AI: the subset of AI that goes beyond classifying existing data to produce new content — text, images, code, sound.
• Large Language Models (LLMs): enormous models that have taken a whole language, studied essentially the entire literature and writing of the world, and captured the patterns of human language mathematically.
• GPT (Generative Pre-trained Transformer): the most successful and revolutionary neural network architecture for building an LLM. The Transformer revolution began with the historic 2017 research paper by Google researchers, Vaswani et al. (2017) — "Attention Is All You Need". Today OpenAI's ChatGPT, Google Gemini, Anthropic Claude, and DeepSeek all run on this same Transformer principle. (The full arc of this technical evolution can be studied further in Idasara Academy's article, The Gen AI Evolutionary Journey: 2017 to 2035.)

2. Nature's Blueprint, Japan's Bullet Train, and Neural Networks (Biomimicry to Neural Networks)

Here we should ask a fundamental engineering question: why did computer scientists choose "neural networks" to understand and generate language?

The answer lies in one of engineering's most revolutionary ideas: biomimicry — solving humanity's hardest technical challenges by borrowing the natural forms, shapes, and mechanisms that nature has perfected over millions of years of evolution.

Japan's Shinkansen Bullet Train and the Kingfisher's Lesson

The finest practical example of biomimicry on record comes from Japan in the 1990s.

Japan's high-speed Shinkansen bullet train, running at over 300 kilometers per hour, ran into a serious aerodynamic and environmental crisis. When the train entered narrow tunnels at great speed, its front face compressed the air inside the tunnel like a piston. That compressed air burst from the far end of the tunnel with a deafening explosive crack — a sonic "tunnel boom" — loud enough to rattle the windows of homes a kilometer away and torment the residents. Because legal noise limits were being exceeded, the engineers were forced to slow the trains down.

The solution came from Eiji Nakatsu, a chief technical engineer at the JR-West railway company — and a devoted birdwatcher.

Observing nature, Nakatsu posed one remarkable question: "Which creature dives at high speed from low-density air into high-density water — without a splash, without a wave, without a sound?"

The answer was the kingfisher.

When a kingfisher plunges from the sky toward a fish in the water, its long, conical, streamlined beak parts the water's resistance smoothly to either side. The result: no splash, no shock of resistance. Nakatsu also noted how the fine serrations on an owl's wings absorb aerodynamic noise.

The nose of the Shinkansen 500 Series, unveiled in 1997, was engineered as a 15-meter-long computerized reproduction of the exact aerodynamic geometry of the kingfisher's beak. The results were astonishing:

1. The tunnel boom was eliminated entirely: the train could run within the legal environmental noise limits.
2. Aerodynamic drag fell by 30%.
3. Electricity consumption fell by 15%.
4. Speed rose by 10%, sustaining continuous operation at 300 kilometers per hour.

Figure 3.2: Biomimicry in practice — the aerodynamic nose of Japan's Shinkansen 500 Series bullet train, modeled on the conical beak of the diving kingfisher. It eliminated the tunnel boom entirely and cut aerodynamic drag by 30%.

From Mechanical Biomimicry to Cognitive Biomimicry: Neural Networks

Just as Japanese engineers borrowed the bird's form to solve the train's aerodynamics, computer scientists and AI architects borrowed the biological neural network of the human brain to capture the patterns of human language and cognition. This is cognitive biomimicry.

The human brain holds roughly 86 to 100 billion biological neurons, interconnected by hundreds of trillions of synapses. When a small child learns words, no rulebook is consulted; electrochemical signals travel the brain, and the connections between neurons that activate together grow stronger ("Neurons that fire together, wire together").

Computer engineers understood that language cannot be taught to a machine with the rigid rules of conventional software (if-else logic). Language is a supremely flexible, contextual system. So they recreated, in mathematical equations on a computer, the way the brain's neurons and synapses work. That is the Artificial Neural Network (ANN). The GPTs and LLMs we speak of today are gigantic artificial neural networks with billions — and now trillions — of parameters.

Figure 3.3: Cognitive biomimicry — a direct comparison of the brain's biological synapse and electrical signaling (action potential) with the mathematical construction of an artificial neural network: input layer, hidden layers 1–3, output layer, and synaptic weightings.

The 1:1 Engineering Parallel between the Biological Brain and the Artificial Neural Network (LLM):

Evolutionary Optimization and the Critical Boundary Line

Two of the deepest technical lessons live here:

1. Nature's engineering optimization (Evolutionary Optimization): Humanity did not need to invent the wheel of network architecture from scratch. Through hundreds of millions of years of natural selection, nature built our brain as an unmatched biological computer — capturing maximal patterns on a mere 20 watts of power. AI scientists chose neural networks precisely because of the brain's consummate efficiency.

2. The critical engineering boundary: And yet, as a leader with engineering discernment, you must see the true limit with equal clarity:

   Shaping the bullet train's nose like a kingfisher's beak does not make that iron train a living kingfisher that flaps its wings, flies, and lays eggs! It remains a mechanical train running on rails.

In exactly the same way: modeling large language models mathematically on the brain's neural architecture does not give the computer a biological consciousness, subjective feeling, intention, or understanding. What it does, still, is a statistical computation — matrix multiplications through a network of weights, estimating what the next word is likely to be (P(w_t | w_<t)).

Grasping this reality is critical for leaders of public administration: it lets us seize the enormous efficiency of AI while refusing to be fooled into treating it as a person (anthropomorphism).

3. How Is an LLM Actually Built? (The Google Search Analogy)

To understand this, consider how the Google search engine works.

Google Search reads every website on the world's internet and converts the words it finds into a vast index. When you search for something, it consults that index and returns the most relevant link.

Large language models operate on a similar principle — but on a mathematical plane hundreds of thousands of times more complex:

1. The model reads essentially the entire internet, and every digitized book, research paper, dictionary, law book, and conversation in the world.
2. It breaks all that text into small pieces called tokens.
3. As noted earlier, the computer does not understand English or Sinhala letters. So it converts every token into numerical values and builds an immense mathematical model of language — of how the words of a language fit together.

4. A/L Combined Mathematics and the Transformer Brain

A question many people ask: "How does the brain of these models actually work?"

The brain here is the multi-layer Transformer architecture.

Ask a Sri Lankan student studying Combined Mathematics for the G.C.E. A/L (the university-entrance examination) where the matrices and vectors they learn are used in the real world, and most cannot say. Yet the artificial intelligence shaking the entire world today runs on exactly those mathematical principles!

Figure 3.4: The fundamental architecture of a large language model (LLM) — the massive dataset, tokenization and embedding, the Transformer brain, the training process (pre-training & RLHF), and inference output.

The Deep Engineering Anatomy of Figure 3.4's Five Stages:

1. Stage 1: The Massive Dataset (the raw material): * Digitized books, encyclopedias, text documents, the open web, and code repositories. * This enormous corpus is cleaned (deduplication and quality filtering), stripped of false or toxic content, and fed to the model.
2. Stage 2: Data Input, Tokenization & Embedding: * The tokenizer: breaking sentences and paragraphs into small character chunks — tokens. * Vector embedding: assigning every token coordinates in a mathematical space of thousands of dimensions. Words with kindred meanings — "Divisional Secretary" and "District Secretary" — sit close together in that space.
3. Stage 3: The Transformer Architecture (the "brain"): * The self-attention mechanism: computing the contextual relationship of each word in a sentence to every other word. (Example: in "He signed the permit and gave it to the Secretary," identifying what "it" refers to.) * Encoder & decoder layers: perceiving the input and producing the next words in sequence. * Add & norm layers: preventing mathematical signals from fading (vanishing gradients) as information flows through hundreds of neural layers, preserving stability. * Feed-forward networks (FFN): depositing the processed patterns into memory, capturing complex structure.
4. Stage 4: The Training & Learning Process: * Pre-training (unsupervised learning): reading trillions of words while predicting "what comes next" (next-word prediction). Here the model absorbs the entire grammar and logic of language. * Fine-tuning with human feedback (RLHF): human experts grade the model's answers, shaping it toward responses that are appropriate, safe, courteous, and truthful — aligned with human intent, including the standards of public service.
5. Stage 5: Inference & Text Generation: * Given the user's prompt, the model instantly generates the most suitable output using probabilistic decoding and sampling strategies (temperature, top-p).

5. Pre-training and Reinforcement Learning (RLHF)

A model does not become intelligent in an instant. It is built through a colossal, step-by-step training process:

1. Pre-training: * The model is given billions of sentences with a word masked, and made to guess the missing or next word. * When it guesses wrong, it is mathematically penalized (backpropagation) and its weights adjust. When it is right, the pattern is reinforced. * Trained this way trillions of times, the model acquires an extraordinary grasp of the grammar, logic, and knowledge embedded in human language.
2. Reinforcement Learning from Human Feedback (RLHF): * Human reviewers grade the model's answers, steering it toward responses that are more courteous, more useful, and more accurate.

6. A Plausibility Engine, Not a Truth Engine

As a public officer, this is the number-one engineering truth to engrave permanently in your mind:

The model has no inner sense of what is true and what is false. When you ask it a question, it does not verify facts against a store of truth; it constructs the most plausible, most fluent sequence of words that could serve as an answer to your question — on mathematical probability alone.

This is why, even when a model does not know a fact, it can present — without a flicker of hesitation, in the most elegant official prose — a completely fabricated falsehood (a hallucination). And it is why an officer must never be seduced by the model's form, but must always verify its truth.

7. Language Bias and Geopolitics: Why Do the Models Differ?

In the sessions, officers raise a fair complaint: "Sir, when we ask ChatGPT something in Sinhala, we sometimes get nonsense. Ask in English, and the answer is superb. Why?"

The reason is the enormous linguistic and cultural bias in the data on which these models were trained:

1. The Western English dominance (Western Knowledge Bias): * American models such as ChatGPT and Claude are trained overwhelmingly on the English literature, law, and data of the Western world. Of Eastern culture, Sri Lankan circulars, or our rural way of life, they hold only the thinnest slice of data.
2. Google Gemini's Sinhala advantage: * Relative to the other American models, Google Gemini holds a marked advantage in Sinhala. For more than twenty years, Google has been indexing Sri Lanka's Sinhala and Tamil websites, news reports, and search terms. Gemini's Sinhala grammar and word usage are correspondingly more accurate.
3. The secret of China's DeepSeek: * Work with China's DeepSeek in Sinhala and you will find it surprisingly fluent. The reason: Chinese is not a simple alphabet like English but a supremely complex system of ideographic characters and tonal patterns. Deeply trained on Chinese and on broad non-English multilingual data, DeepSeek has a natural capacity to adapt to the complex sentence patterns of our Sinhala.
4. The difference between Tamil and Sinhala: * For Sri Lankan officers working in Tamil, AI tools are comparatively easier. Tens of millions of Tamil speakers — in Tamil Nadu and across the world — have built a vast body of Tamil literature and digital data on the internet. * But Sinhala is spoken by only about twenty million people, nearly all in our own country. The digital Sinhala content we have contributed to the internet is very limited. When using AI in Sinhala, therefore, we must be especially vigilant.

8. Multimodal Integration and the Universal Vector Space (Multimodal AI Systems)

Today these models are no longer confined to text. Modern frontier models operate as fully multimodal systems.

Figure 3.5: The architecture of a multimodal generative AI system — multimodal data input (text, photos, videos, audio, code), the unified multidimensional vector space, the multi-headed Transformer, and multimodal output.

The Multimodal System Anatomy of Figure 3.5:

1. The Massive Multimodal Dataset: * Text documents, photographs, videos, audio recordings, and computer code.
2. Multimodal Ingestion & Tokenization: * Unlike older systems, an image is not first converted into text (no text-intermediate bottleneck). * Photographs and video frames are broken directly into small pixel patches (16×16) as visual tokens; audio waves become audio tokens via frequency spectrograms.
3. The Unified Multimodal Vector Space: * This is the system's crowning engineering achievement! * The photograph of a Sri Lankan national identity card (an image patch), the spoken Sinhala phrase "ජාතික හැඳුනුම්පත" (jathika hendunumpatha — "national identity card") as an audio wave, and the English text "National Identity Card" all land in the same coordinate region of this multidimensional mathematical space. The machine can therefore compare across modalities without barriers.
4. The Multi-Headed Cross-Attention Architecture: * Cross-attention operates between modalities: while examining the signature on a scanned deed, the model can simultaneously compare the landowner's name written in the text.
5. Multimodal Pre-training & RLHF: * Training on mixed video, audio, and text via next-data-type prediction. * Safety guardrails and filters against forged media (multimodal safety alignment against deepfakes).
6. Universal Generation & Output: * The user may pose a query in any medium (say, an audio recording of a complaint plus a photograph of disaster damage). * The system generates, via decoding and sampling strategies, an official Sinhala report, a damage-annotated map, or working code — whichever the situation demands.

The Practical Value for the Public Officer:

• Photograph an old handwritten land deed or letter and have its contents read in a second (Vision OCR).
• On a public day, record a citizen's spoken grievance and produce an instant official inquiry note.
• Feed in the map or chart of a complex development plan and have its inconsistencies identified.

In the next chapter, we discuss how this natural language revolution is bringing the humanities back to center stage — and how AI closes the digital divide facing the ordinary citizen.