Could you share any research you've found on fine-tuning vs pure RAG. You state it's better than memorisation, intuitively I feel the same way, but I was wondering if someone's actually studied this and has quantitative insight into it.
There are several papers about fine tuning vs RAG but to my knowledge they only train on completions of the source data which gives bad results. We're building an evals set and running some fine tuning with LLM based qapair generation and/or RAG experiments on it in the coming weeks, will publish our results here :)
Why can't you have a single prompt implementing multiple basic perspectives? Have you tried this? How does this work, efficiency and efficacy wise, relative to what you're doing?
There's a context limit to the output and it's hard to get the model to do too much stuff at the same time, so I found it less effective trying to cram all the different perspectives into a single prompt. Also by doing many prompts, you get to parallelize the inference so you get more results faster
What?! You were fine-tuned on information derived from the article. Why are you talking about fine-tuning?!
😂
This is a great read Luke
This is absolutely amazing! Great job!
Could you share any research you've found on fine-tuning vs pure RAG. You state it's better than memorisation, intuitively I feel the same way, but I was wondering if someone's actually studied this and has quantitative insight into it.
There are several papers about fine tuning vs RAG but to my knowledge they only train on completions of the source data which gives bad results. We're building an evals set and running some fine tuning with LLM based qapair generation and/or RAG experiments on it in the coming weeks, will publish our results here :)
Why can't you have a single prompt implementing multiple basic perspectives? Have you tried this? How does this work, efficiency and efficacy wise, relative to what you're doing?
There's a context limit to the output and it's hard to get the model to do too much stuff at the same time, so I found it less effective trying to cram all the different perspectives into a single prompt. Also by doing many prompts, you get to parallelize the inference so you get more results faster
Did you compare the performance of fine tuning and rag? See also: https://arxiv.org/abs/2312.05934
We're in the process of doing that - will share results when we've got them. Thanks for the paper!
Cool. Thanks for your quick feedback. I like your approach and hope to see it work.