Error in fine-tuning

[b5y]

Hello,

Tensor assertion error is raised if you try to train the model. It starts with the following:

0%|          | 0/10 [00:00<?, ?it/s]../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [56,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

It's possible to avoid this error by resizing embeddings:

model.resize_token_embeddings(len(tokenizer))

Unfortunately, it might affect the performance. Could you please review the finetune.py script and fix this problem?

Thank you in advance.

BR,
Mehti

[Kaka23333]

Hi, thanks for your attention and sorry for not reply in time.

Our finetune.py is used for fine-tuning CodeLlama model and there were no issues in our reproduction.

If you hope to apply this script to other models, you may have to slightly modify the script. For example, we encountered the same problem when using finetune.py on llama3. Our solution is to replace the tokenizer.pad_token = "[PAD]" in line 87 with tokenizer.pad_token = tokenizer.eos_token.

Hope this solution is helpful for you. And If you have any other questions or suggestions, feel free to communicate with us.

[b5y]

Hello,

Thank you for your response.

Could you please create adapters and quantized models as well, if possible? I requested this on hugging faces and would highly appreciate it if you published different variations of models. I was able to run the training using a quantized 8bit CodeLlama model and fit it in memory on my home computer with 2 GPUs, each with 49 GB.

Unfortunately, requesting resources on cloud platforms in region-based locations is sometimes challenging. Besides, training the whole LLM model is quite expensive.

Looking forward to hearing from you.

BR,
Mehti