Local LLM Fine-Tuning for Beginners — From Full Fine-Tuning to QLoRA: Theory and Hands-On Unsloth Commands

Bottom Line First

The right answer for personal fine-tuning is QLoRA. You train only a small adapter on top of a 4-bit-reduced base. For an 8B model it runs on 8-12GB of VRAM. Full fine-tuning, which fixes the entire model, is not something you do on personal hardware.

1. Theory: Comparing the Three Kinds of Fine-Tuning

The key to LoRA is rank. Rank 16 is the 2026 practical default. A larger rank raises expressiveness but enlarges the adapter and increases overfitting risk. Alpha is usually set to twice the rank.

2. When to Fine-Tune

This needs to be said first. Fine-tuning is right when you are teaching tone, format, and domain terminology. When you are teaching knowledge, RAG is right. Injecting knowledge through fine-tuning increases hallucination. 500-2,000 well-made examples beat tens of thousands of pieces of junk data.

3. A Feel for VRAM Requirements

On the operator's environment (RTX 3070 8GB), 4B-8B models are the target. It is right at the 8GB boundary, so the context has to be kept short.

4. Three Tuning Programs

5. Installation, Unsloth Version (Ubuntu + NVIDIA)

# A virtual environment is recommended
python3 -m venv ft-env
source ft-env/bin/activate

# PyTorch (match your CUDA version; example)
pip install torch --index-url https://download.pytorch.org/whl/cu121

# Unsloth
pip install unsloth

For Axolotl, cloning the repository is the standard approach.

git clone https://github.com/axolotl-ai-cloud/axolotl.git
cd axolotl
pip install -e .

LLaMA-Factory is also cloned and then run.

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .
llamafactory-cli webui

That last command launches the web UI. You can run training from the browser.

6. Training Data Format

The Alpaca-format JSON is the safest choice.

[
  {
    "instruction": "What is the command to check disk usage on Linux?",
    "input": "",
    "output": "You can check per-partition usage with the df -h command."
  }
]

Save it as mydata.json. The instruction and output must come in pairs.

7. Running Training: An Unsloth Example

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "Qwen/Qwen3-4B",
    max_seq_length = 2048,
    load_in_4bit = True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    lora_alpha = 32,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
)

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        max_steps = 200,
        learning_rate = 2e-4,
        output_dir = "outputs",
    ),
)
trainer.train()

With Axolotl you just write a YAML.

axolotl train myconfig.yml

8. Merging the Adapter, Converting to GGUF, and Running in Ollama

The output of training is an adapter. You must merge it with the base and then convert it to GGUF to run it locally.

# In Unsloth, save the merged model and then save the GGUF
# model.save_pretrained_gguf("my-qwen3-4b", quantization_method = "q4_k_m")

# Ollama Modelfile
# FROM ./my-qwen3-4b-Q4_K_M.gguf
ollama create my-qwen -f Modelfile
ollama run my-qwen

With Axolotl you merge using axolotl-cli merge and produce the GGUF with the llama.cpp conversion script.

9. A Checklist That Prevents Failure

In one line: freeze the base in 4-bit, train only an adapter at rank 16, merge it, export to GGUF, and run it in Ollama.

10. Training Timetable: How Long Does It Take

This is the most frequently asked question. Here is a measured range for 8B QLoRA.

The three factors that eat time are the number of examples, sequence length, and batch size. At sequence 2048 and batch 4, the 4090 peak VRAM reaches about 14GB. On an 8GB card you must drop the sequence to 1024 or below and the batch to 1-2. Unsloth is known to be about 2x faster than standard TRL and to use up to 70% less memory, which makes it a lifeline for 8GB-class machines.

Renting cloud is simple math. At an interruptible rate of about $0.13 per hour for a single 4090, training on 10,000 examples costs under $2. Often two hours in the cloud is cheaper than half a day of struggling on a local 8GB card.

11. Seven Failure Causes and Their Remedies

Fine-tuning fails by default. Here they are by symptom.

Failure 1. CUDA out of memory (most common)

The symptom is an explosion right after training starts. If lowering the batch to 1 does not help, sequence length is the culprit.

# Remedy 1: Lower batch and sequence
# per_device_train_batch_size=1, max_seq_length=1024

# Remedy 2: Gradient checkpointing (recomputes activations; slower but saves memory)
# gradient_checkpointing=True

# Remedy 3: Keep the effective batch via gradient accumulation
# gradient_accumulation_steps=8 (batch 1 x 8 = effective batch 8)

Failure 2. Loss diverges to NaN

The learning rate is too high or it is a mixed precision problem. If the loss curve suddenly dives vertically and then goes NaN, that settles it.

# Remedy: Lower the learning rate from 2e-4 to 5e-5 and increase warmup
# learning_rate=5e-5, warmup_ratio=0.1

Failure 3. Overfitting: The Memorized Look

Training loss falls, but to any question it spits out training-data sentences verbatim. You ran too many epochs or have too little data. Start from max_steps 200, and compare the same 10 questions before and after training.

Failure 4. Catastrophic Forgetting: It Loses What It Was Good At

Overfeeding only domain data collapses general conversation ability. The remedy is to mix 10-20% general conversation examples into the domain data.

Failure 5. Bad Data Format

If instruction-output pairs are broken, outputs are empty, or duplicates are mixed in, training just spins. Before training, always read at least 50 examples with your own eyes.

# Remove duplicates and check for empty values
python3 -c "import json; d=json.load(open('mydata.json')); print(len(d)); print(sum(1 for x in d if not x.get('output')))"

Failure 6. Missing Adapter Target Modules

Qwen and Llama have different MLP module names (gate_proj, up_proj, down_proj). If you target only attention, training does not take. Targeting all seven modules as in the example above is the default.

Failure 7. After GGUF Conversion, It Talks Nonsense

Either you converted without merging the adapter into the base, or the template broke during quantization. Load the merged safetensors with transformers (not llama.cpp or Ollama) first, verify the answers, and then convert.

12. How to Tell Training Is Going Well

Do not look only at the loss curve; look at three things together.

Checkpoints are saved at intervals. Pick not the lowest-loss one but the checkpoint with the best evaluation answers.

13. A Sense of Cost

For personal experiments at 8B or below, local; for 14B and up or anything urgent, cloud is close to the right answer.