Challenges and Objectives
An overview of the challenges we seek to address with our solution
1. Insufficient LLM Knowledge
Given the technical nature of the questions are and the little capacity of the LLMs to store knowledge in their weights, we find that the LLMs (Phi-2 and Falcon7B) lack the necessary knowledge to correctly answer the questions.
2. Non-adherence to Instructions by LLMs
We find that Phi-2 struggles to consistently provide answers in the desired format.
With Falcon7B, the responses seem to be uninformed from the question and context when options are provided in the prompt.
3. Lexicon mismatch
The questions seem to have many technical terms and abbreviations that are unknown to the LLM and are also not easy to infer from their use without prior understanding of the domain knowledge and terminology.
Naturally, our objectives are targeted towards addressing these challenges.
Objectives
To develop a RAG pipeline that provides context that's relevant to the query
To finetune Phi-2 for instruction alignment
To develop an alternative means of prompting Falcon7B without the options
To develop a glossary that helps to minimize the lexicon mismatch
Last updated