💡Introduction
Documentation for the 3musketeers in the "Specializing Large Language Models for Telecom Networks by ITU AI/ML in 5G" challenge
Authors:
This documentation describes our solutions for both tracks in the "Specializing Large Language Models for Telecom Networks by ITU AI/ML in 5G" challenge.
A brief summary of each solution:
Phi-2
We develop a RAG pipeline based on ColBERT V2 (https://arxiv.org/abs/2004.12832 and https://arxiv.org/abs/2112.01488) and provide as many retrieved chunks as the context window can support.
We also append the full forms of abbreviations that appear in the question and options.
We finetune the LLM with context to align the LLM's generated to our desired output format, and to improve the "usable" context window.
Falcon7B
We adopt the same RAG pipeline and abbreviations expansion as for Phi-2.
Following the challenge guidelines, we could not finetune the model. We find that it responds very poorly when options are provided in the prompt. So, we simply do not provide the options in the prompt.
We allow the LLM to freely generate a response conditioned on the question and context, but not the options. We then develop a scoring system that attempts to find the most likely option given the response.
(Optional) Jump to reproducing results
If you'd like, you can quickly jump to reproducing results: REPRODUCING RESULTS
Last updated