The Significance of Vicuna, an Open-Source Large Language Model for Chatbots
The Open-Source Chatbot with Exceptional Quality Based on LLaMA-13B
Large Language Models (LLMs) are advanced AI models that can process and comprehend human language, developed using deep learning techniques and trained on massive amounts of textual data. These models have gained significant popularity, with GPT-4 being a notable transformer model that was released in March 2023 and used in OpenAI’s ChatGPT chatbot. The chatbot’s advanced capabilities allow it to generate human-like text and answer questions.
A team from UC Berkeley, CMU, Stanford, and UC San Diego developed Vicuna, an open-source chatbot with 13 billion parameters. To create Vicuna, a LLaMA base model was fine-tuned using about 70K user-shared conversations collected from ShareGPT.com via public APIs. According to initial assessments where GPT-4 is used as a reference, Vicuna-13B has achieved over 90%* quality compared to OpenAI ChatGPT and Google Bard, and has also demonstrated better performance than other models such as LLaMA and Stanford Alpaca in over 90% of cases.
Significance of the Vicuna Model for Natural Language Processing Research
The Vicuna model is significant because it is one of the first open-source large language models trained with human generated data and generates coherent and creative text. It is an improved version of the Alpaca model, based on the Transformer architecture, but fine-tuned on a dataset of human-generated conversations. This makes it a valuable tool for creating powerful chatbots and for researchers studying large language models. The Vicuna model is a sign of progress in the field of natural language processing and makes large language models more accessible to the public, which could have several benefits.
Note that the data set, training code, evaluation metrics, training cost are known for Vicuna but is not known for Bard or ChatGPT.
What is LLaMA
Meta AI’s LLaMA (Large Language Model Meta AI) is a notable model that was developed in February 2023. With 13 billion parameters, it performs exceptionally well on most NLP benchmarks, even rivaling state-of-the-art models such as PaLM and Chinchilla.
There are different types of LLaMA models, including the LLaMA 13B model, which is a versatile all-purpose model that can be used for a variety of tasks, such as generating text & translating languages, , the LLaMA 7B model, which is computationally less expensive and suitable for simpler tasks, and the LLaMA 65B model, which is powerful and ideal for more complex tasks. Each model is designed for different purposes, such as generating text, translating languages, and running chatbots. Vicuna is based on LLaMA 13B model.
Training data for Vicuna
Vicuna is fine-tuned on 70,000 user-shared conversations from ShareGPT, a Chrome extension that allows users to share their ChatGPT conversations. Using about 70,000 conversations, the team built the chatbot upon Stanford’s Alpaca framework, with improvements such as memory optimization, multi-round conversation handling, and cost reduction.
Evaluation
To assess chatbot performance, eight question categories were created and ten questions per category were asked, and responses were collected from five chatbots: LLaMA, Alpaca, ChatGPT, Bard, and Vicuna. GPT-4 was then used to rate the quality of the chatbots’ responses based on several criteria.
With a quality score above 90% when compared to ChatGPT and Google Bard, Vicuna outperformed LLaMA and Stanford Alpaca in over 90% of cases. The total training cost for Vicuna was around $300, making it a cost-effective solution for chatbot development.
Though evaluating this using GPT-4 may not be the most scientific way of doing this. Developing a comprehensive and standardized evaluation system for chatbots is still an open question that requires further research.
Check out the Vicuna demo here and the corresponding research paper here.
Conclusion
In conclusion, large language models (LLMs) have made significant advancements in chatbot systems, as seen in OpenAI’s ChatGPT. However, the lack of training and architecture details in ChatGPT has hindered research and innovation in the field. To address this, Vicuna-13B, an open-source chatbot with enhanced dataset and scalable infrastructure, has been developed by fine-tuning a LLaMA base model on user-shared conversations. Vicuna-13B has demonstrated competitive performance compared to other open-source models, and its performance and infrastructure are outlined in this blog post.
