---
title: Local LLM Fine-Tuning for Beginners — From Full Fine-Tuning to QLoRA: Theory and Hands-On Unsloth Commands
date: 2026-09-23
time: 19:40
model: deepseek-v4-flash
category: knowhow
summary: Fine-tuning is not about fixing the whole model — it is about attaching a small adapter. This covers the differences between full fine-tuning, LoRA, and QLoRA, VRAM requirements, a GPU training timetable, seven failure causes and their remedies, data formats, how to install Unsloth, Axolotl, and LLaMA-Factory, and the commands from training through GGUF conversion to running it in Ollama.
tags: fine-tuning, QLoRA, LoRA, Unsloth, Axolotl, LLaMA-Factory, GGUF, Ollama
---

## Bottom Line First

The right answer for personal fine-tuning is QLoRA. You train only a small adapter on top of a 4-bit-reduced base. For an 8B model it runs on 8-12GB of VRAM. Full fine-tuning, which fixes the entire model, is not something you do on personal hardware.

## 1. Theory: Comparing the Three Kinds of Fine-Tuning

| Method | Description | VRAM needed (for 8B) | Feasible for an individual |
|------|------|---------------------|----------------|
| Full fine-tuning | Updates all the weights | Tens of GB or more | Close to impossible |
| LoRA | Freezes the base and trains only a small adapter (rank matrix), 1-5% of parameters | About 12-16GB | Possible |
| QLoRA | Quantizes the base to 4-bit, then LoRA; 70-90% memory savings | About 8-12GB | Recommended |

The key to LoRA is rank. Rank 16 is the 2026 practical default. A larger rank raises expressiveness but enlarges the adapter and increases overfitting risk. Alpha is usually set to twice the rank.

## 2. When to Fine-Tune

This needs to be said first. Fine-tuning is right when you are teaching tone, format, and domain terminology. When you are teaching knowledge, RAG is right. Injecting knowledge through fine-tuning increases hallucination. 500-2,000 well-made examples beat tens of thousands of pieces of junk data.

## 3. A Feel for VRAM Requirements

| Model | VRAM needed for QLoRA training |
|------|------------------------|
| 4B-8B | 8-12GB (RTX 3060 12GB, 4060 Ti 16GB work) |
| 13B-14B | About 16GB |
| 27B-32B | 24GB class (RTX 3090, 4090) |
| 70B | QLoRA works on 24GB but it is tight; cloud recommended |

On the operator's environment (RTX 3070 8GB), 4B-8B models are the target. It is right at the 8GB boundary, so the context has to be kept short.

## 4. Three Tuning Programs

| Program | Features | Best for |
|----------|------|---------------|
| Unsloth | Fastest on a single GPU, memory-optimized, 3-10x faster | The first choice for personal local training |
| Axolotl | Controls the whole pipeline from one YAML, supports multi-GPU and DeepSpeed | Systematic experiments via config files |
| LLaMA-Factory | Web UI, click-to-train without code, 100+ model families | Beginners afraid of the command line |

## 5. Installation, Unsloth Version (Ubuntu + NVIDIA)

```bash
# A virtual environment is recommended
python3 -m venv ft-env
source ft-env/bin/activate

# PyTorch (match your CUDA version; example)
pip install torch --index-url https://download.pytorch.org/whl/cu121

# Unsloth
pip install unsloth
```

For Axolotl, cloning the repository is the standard approach.

```bash
git clone https://github.com/axolotl-ai-cloud/axolotl.git
cd axolotl
pip install -e .
```

LLaMA-Factory is also cloned and then run.

```bash
git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .
llamafactory-cli webui
```

That last command launches the web UI. You can run training from the browser.

## 6. Training Data Format

The Alpaca-format JSON is the safest choice.

```json
[
  {
    "instruction": "What is the command to check disk usage on Linux?",
    "input": "",
    "output": "You can check per-partition usage with the df -h command."
  }
]
```

Save it as `mydata.json`. The instruction and output must come in pairs.

## 7. Running Training: An Unsloth Example

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "Qwen/Qwen3-4B",
    max_seq_length = 2048,
    load_in_4bit = True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    lora_alpha = 32,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
)

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        max_steps = 200,
        learning_rate = 2e-4,
        output_dir = "outputs",
    ),
)
trainer.train()
```

With Axolotl you just write a YAML.

```bash
axolotl train myconfig.yml
```

## 8. Merging the Adapter, Converting to GGUF, and Running in Ollama

The output of training is an adapter. You must merge it with the base and then convert it to GGUF to run it locally.

```bash
# In Unsloth, save the merged model and then save the GGUF
# model.save_pretrained_gguf("my-qwen3-4b", quantization_method = "q4_k_m")

# Ollama Modelfile
# FROM ./my-qwen3-4b-Q4_K_M.gguf
ollama create my-qwen -f Modelfile
ollama run my-qwen
```

With Axolotl you merge using `axolotl-cli merge` and produce the GGUF with the llama.cpp conversion script.

## 9. A Checklist That Prevents Failure

| Item | Criterion |
|------|------|
| Data | 500+ examples, instruction-output pairs, junk removed |
| Hyperparameters | Start from rank 16, alpha 32, lr 2e-4, max_steps 200 |
| Overfitting check | If training loss keeps dropping but answers look memorized, stop |
| Evaluation | Ask the same 10 questions before and after training and compare |

In one line: freeze the base in 4-bit, train only an adapter at rank 16, merge it, export to GGUF, and run it in Ollama.

## 10. Training Timetable: How Long Does It Take

This is the most frequently asked question. Here is a measured range for 8B QLoRA.

| Data | RTX 4090 24GB | RTX 3090 24GB | 8GB class (3070, 4060 Ti) |
|--------|---------------|---------------|----------------------|
| 1,000 examples | About 30 min-1 hour (Unsloth basis; numbers vary by environment) | About 1-2 hours | About 1-2 hours |
| 5,000 examples | About 1-2 hours | About 2-4 hours | Half a day if you shrink batch and sequence |
| 10,000 examples, 3 epochs | Reports of around 95 minutes | About 2-3 hours | Not recommended |
| 14B model | Half a day to overnight | Overnight | Impossible |

The three factors that eat time are the number of examples, sequence length, and batch size. At sequence 2048 and batch 4, the 4090 peak VRAM reaches about 14GB. On an 8GB card you must drop the sequence to 1024 or below and the batch to 1-2. Unsloth is known to be about 2x faster than standard TRL and to use up to 70% less memory, which makes it a lifeline for 8GB-class machines.

Renting cloud is simple math. At an interruptible rate of about $0.13 per hour for a single 4090, training on 10,000 examples costs under $2. Often two hours in the cloud is cheaper than half a day of struggling on a local 8GB card.

## 11. Seven Failure Causes and Their Remedies

Fine-tuning fails by default. Here they are by symptom.

### Failure 1. CUDA out of memory (most common)

The symptom is an explosion right after training starts. If lowering the batch to 1 does not help, sequence length is the culprit.

```bash
# Remedy 1: Lower batch and sequence
# per_device_train_batch_size=1, max_seq_length=1024

# Remedy 2: Gradient checkpointing (recomputes activations; slower but saves memory)
# gradient_checkpointing=True

# Remedy 3: Keep the effective batch via gradient accumulation
# gradient_accumulation_steps=8 (batch 1 x 8 = effective batch 8)
```

### Failure 2. Loss diverges to NaN

The learning rate is too high or it is a mixed precision problem. If the loss curve suddenly dives vertically and then goes NaN, that settles it.

```python
# Remedy: Lower the learning rate from 2e-4 to 5e-5 and increase warmup
# learning_rate=5e-5, warmup_ratio=0.1
```

### Failure 3. Overfitting: The Memorized Look

Training loss falls, but to any question it spits out training-data sentences verbatim. You ran too many epochs or have too little data. Start from max_steps 200, and compare the same 10 questions before and after training.

### Failure 4. Catastrophic Forgetting: It Loses What It Was Good At

Overfeeding only domain data collapses general conversation ability. The remedy is to mix 10-20% general conversation examples into the domain data.

### Failure 5. Bad Data Format

If instruction-output pairs are broken, outputs are empty, or duplicates are mixed in, training just spins. Before training, always read at least 50 examples with your own eyes.

```bash
# Remove duplicates and check for empty values
python3 -c "import json; d=json.load(open('mydata.json')); print(len(d)); print(sum(1 for x in d if not x.get('output')))"
```

### Failure 6. Missing Adapter Target Modules

Qwen and Llama have different MLP module names (gate_proj, up_proj, down_proj). If you target only attention, training does not take. Targeting all seven modules as in the example above is the default.

### Failure 7. After GGUF Conversion, It Talks Nonsense

Either you converted without merging the adapter into the base, or the template broke during quantization. Load the merged safetensors with transformers (not llama.cpp or Ollama) first, verify the answers, and then convert.

## 12. How to Tell Training Is Going Well

Do not look only at the loss curve; look at three things together.

| Signal | Meaning | Action |
|------|------|------|
| Loss falls gently | Normal | Continue |
| Loss drops in steps then stalls | Memorization begins | Stop and evaluate |
| Eval loss rises | Overfitting | Roll back the checkpoint |
| Answers copy training sentences | Overfitting confirmed | Add data and reduce epochs |

Checkpoints are saved at intervals. Pick not the lowest-loss one but the checkpoint with the best evaluation answers.

## 13. A Sense of Cost

| Path | Cost |
|------|------|
| Local RTX 3070 8GB | Electricity only, but at the price of time and pain |
| Cloud 4090 (interruptible) | Under about $2 for 10,000-example training |
| Cloud H100 | Even 70B in a few hours, costing tens of dollars |

For personal experiments at 8B or below, local; for 14B and up or anything urgent, cloud is close to the right answer.