Turkish Gemma-4 Fine-tune
Parameter-efficient fine-tuning of Gemma-4-E4B for Turkish RAG answers, published in three formats.
Problem
In RAG, the generator should answer from the retrieved context, not from what it memorised, and ignore irrelevant background in that context.
The goal was a small model that, given a Turkish question and a context passage, answers from that context. It should be cheap to train and easy to serve on vLLM or locally.
Architecture
- Data: the first 3,000 rows of Metin/WikiRAG-TR, formatted with the Gemma-4 chat template as a single turn: "Soru:\n<question>\n\nBağlam:\n<context>" → answer.
- Model: unsloth/gemma-4-E4B-it loaded in 4-bit, with LoRA on the language attention and MLP modules. Vision layers stay frozen.
- Training: TRL SFTTrainer with train_on_responses_only, so prompt tokens are masked and only the answer contributes to the loss.
- Export: LoRA adapters, a merged 16-bit model for vLLM, and a GGUF q8_0 build for llama.cpp / Ollama, all pushed to Hugging Face.
Design decisions
LoRA on a 4-bit base
r = 8, alpha = 8, dropout 0, on attention + MLP of the language layers only.
Trade-off: Very few trainable parameters and a single-GPU run, but less capacity to change behaviour than a full fine-tune.
Loss on the answer only
train_on_responses_only masks the question and context tokens, and the notebook decodes the labels to verify the mask.
Trade-off: Gradient goes to what the model should produce, not to reproducing long contexts. The model gets no signal on the input side.
Long sequences, conservative LR
max_seq_length 4096 so long contexts are not truncated, batch 4 × 4 gradient accumulation, adamw_8bit, lr 2e-5 with cosine decay, 2 epochs.
Trade-off: Fits long RAG passages in memory on one A100. A low learning rate is safer but may under-fit with only 3,000 examples.
Ship three formats
Adapters for further training, merged 16-bit for vLLM, and GGUF q8_0 for local inference.
Trade-off: Easy to consume anywhere, at the cost of extra export time and storage.
Stack
What I'd improve next
- Hold out an evaluation split. The current run trains with no eval set, so there is no quality number yet.
- Add unanswerable examples (the context lacks the answer) to teach refusals.
- Compare r = 8 against larger ranks on the same held-out set.
Results
- Turkish RAG training examples
- 3,000
- of parameters trained (LoRA)
- 0.23%
- training time on one A100
- ~42 min
- published versions: LoRA, 16-bit, GGUF
- 3