Fine-Tuning Is Not Enough — How to Really Teach LLMs New Knowledge

Many people think fine-tuning is all you need to teach an LLM (like LLaMA, GPT, or Mistral) new facts. But if you're trying to update your model with recent events, product changes, new policies etc., fine-tuning alone won’t do the job. Here’s the truth: fine-tuning mostly changes how a model speaks — not what it knows.

What does "teaching knowledge" mean?

Let’s say:

• A new cancer drug gets FDA approval, and you want the model to answer questions about it.
• Your API has changed, and your chatbot should suggest updated endpoints.
• Your city updated traffic laws, and your AI assistant should reflect them.
• Teaching a menu page of a specific restaurant.

In all of these, the model needs to remember and reason with these facts — not just mimic your prompt. That’s what I mean by teaching/infusing knowledge.

Why Fine-Tuning Doesn’t Work Well (According to Research)

1. Fine-tuned models learn facts slowly

Ovadia et al., EMNLP 2024 - "Fine-Tuning or Retrieval?"

• Found that fine-tuned models required repeated exposure to new facts to respond accurately.
• Even after tuning, models hallucinated or failed to generalize the information.
• RAG models, which retrieved external documents, performed better without modifying weights.
• Compared fine-tuning vs. RAG across factual tasks like MMLU, Natural Questions.

Key takeaway: Fine-tuning is inefficient and brittle for teaching new factual knowledge.

2. Fine-tuning causes forgetting of old knowledge

Li et al., ACL Findings 2024 - "Revisiting Catastrophic Forgetting"

• Showed that models lose general language and reasoning ability after being fine-tuned on small, focused datasets.
• Forgetting increased with sharp changes in the loss landscape and narrow data domains.
• Even parameter-efficient fine-tuning (LoRA) showed degradation without regularization.

Key takeaway: Fine-tuning often "overwrites" existing knowledge unless very carefully controlled.

3. Fine-tuning increases hallucination risk

Gekhman et al., 2023 - "On Factuality and Memorization"

• Found that models become overconfident in incorrect answers when fine-tuned on narrow data.
• Models generalized wrongly from small fact sets, increasing hallucination.
• Fine-tuning shifted belief representations unpredictably, especially with contradictory facts.

Key takeaway: Fine-tuning may teach wrong generalizations and lead to confident hallucinations.

4. Real developers agree

Developers across forums like Reddit and GitHub report similar findings:

• Fine-tuning helps with tone, output format, or alignment — but not core knowledge.
• For real-world factual updates, they rely on RAG, model editing (ROME, MEMIT), or continued pre-training.

Example posts:

• Reddit: Fine-tuning vs RAG discussion
• Reddit: Why fine-tuning fails
• Reddit: Should I fine-tune or retrain?

Key takeaway: Practitioners seems to confirm what the papers show — fine-tuning is fragile for knowledge updates.

So Why Continued Pre‑Training Works for Knowledge Infusion

Research confirms that most factual knowledge in an LLM is acquired during the pre‑training phase, not later during fine-tuning. During pre-training, the model organically builds deep internal representations—a sort of "mental map" linking facts and concepts together. That’s where real knowledge infusion happens.

📌 Research Backing This

Singhal et al. 2022/2023, in “Large Language Models Encode Clinical Knowledge”, evaluated PaLM and Med-PaLM on medical question-answering across six benchmarks (MultiMedQA, HealthSearchQA, MedQA, PubMedQA, MMLU topics). They found that larger models internalize clinical facts primarily through pre-training, while instruction tuning improves usability—not core knowledge retention.

Sources: arxiv.org, sapien.io, openreview.net, semanticscholar.org

Chang et al. 2024 (KAIST, later published at NeurIPS) studied how knowledge is acquired across training steps. They showed that factual memorization occurs as the model sees facts repeatedly during pre-training; performance scales with exposure frequency, dataset deduplication, and larger batch sizes—and that later-step checkpoints didn’t add new memorization power but instead plateaued.

Source: arxiv.org

This explains why continued pre-training (also known as Domain-Adaptive Pretraining or DAPT) is more effective for updating a model’s knowledge base. It essentially mimics the original pre-training process, allowing the model to absorb new facts in a similar way.

Continued pre-training, also called Domain-Adaptive Pretraining (DAPT), is the best way to truly teach an LLM new knowledge.

This method involves:

• Taking your base model and continuing training it on new text data.
• Mixing in a small percentage (e.g., 5–10%) of original training data to prevent forgetting.
• Using the same objective as pretraining: predicting the next token (not instruction learning).

Why it's better:

• It mirrors the model's original training process.
• Helps the model absorb the new data deeply, not just memorize phrases.
• Retains general reasoning and language abilities.

Supported by top industry leaders:

• Meta AI recommends domain-adaptive pretraining for factual adaptation
• Databricks shows continued pretraining outperforms fine-tuning
• AWS uses DAPT for financial LLMs
• Knowledge-Instruct proposes a data-efficient method that improves factual accuracy using instructional prompts during continued pre-training.

The Cons of Continued Pre-Training

While more effective than fine-tuning, continued pre-training is not perfect:

• High compute cost: Training even on a few billion tokens requires expensive hardware.
• Longer time-to-deploy: Pre-training takes days or weeks, compared to a few hours for fine-tuning.*
• Needs data curation: You must pre*pare high-quality, domain-specific data.
• Risk of overfitting: If you don’t mix in old data, the model might lose general capabilities.

Because of these limitations, continued pre-training is better suited for major knowledge upgrades or domain-specific models — not quick fixes.

When RAG Might Be Better

In many real-world use cases, Retrieval-Augmented Generation (RAG) is the most practical option.

• It keeps the base model frozen.
• New facts are stored in a vector database and retrieved dynamically.
• It avoids catastrophic forgetting.
• It supports real-time updates and personalization.

Use RAG if:

• You need to inject fast-changing or user-specific knowledge.
• You want to keep compute and deployment costs low.
• You need explainability (retrieved documents provide transparency).

RAG is often the best starting point for adding knowledge — and continued pre-training should be reserved for large, stable, and domain-wide updates.

When fine-tuning is useful

• Adjusting tone (e.g., making answers more friendly or professional)
• Changing output format (e.g., JSON, Markdown)
• Instruction-following for specific tasks (e.g., agent behavior)
• Applying small factual edits via methods like:
• ROME: edits one fact at a time
• WilKE: parameter-aware editing
• SDF: synthetic document fine-tuning

But these methods do not scale well to thousands of new facts or major domain shifts.

Summary

• Fine-tuning adjusts behavior but does not reliably update memory.
• Continued pre-training is the most effective way to inject new factual knowledge, though it comes with high cost and effort.
• RAG is often the most practical choice for keeping LLMs current with minimal disruption.

If you're serious about building an LLM that truly understands new information, don’t just fine-tune it. Choose the right method for the job.