The Explosion of Open Source LLMs (Vicuna LLM)
From ChatGPT to Vicuna: The Explosion of Open Source LLMs
In the wake of Meta's release of Alpaca and the subsequent leak of its weights, we've witnessed an unprecedented surge in open source large language models. First came Alpaca from the Stanford group, followed by Databricks' Dolly, numerous models from Cerebras, and most recently, GPT4All from Nomic AI. However, a new contender has emerged claiming to match ChatGPT's capabilities - Vicuna.
What is Vicuna?
Vicuna is an open-source chatbot that claims to achieve 90% of ChatGPT's quality. What makes Vicuna particularly interesting is its innovative training dataset and evaluation methodology.
How Vicuna Was Trained
Vicuna has 13 billion parameters and was created by fine-tuning the LLaMA model on user-shared conversations collected from ShareGPT. For those unfamiliar with ShareGPT, it's a platform where users can share their ChatGPT conversations, offering a wealth of examples for prompt engineering study.
The research team collected approximately 70,000 conversations from ShareGPT and fine-tuned the LLaMA model on this dataset. This approach is likely why Vicuna performs so well compared to ChatGPT - it's specifically trained on data collected from actual ChatGPT interactions.
Training Specifications
- 70,000 conversations from ShareGPT
- Trained on eight A100 GPUs for one day
- Maximum context length increased from 512 (Alpaca) to 2048
- Training cost: approximately $300 for the 13B parameter model
Evaluation Strategy
To compare Vicuna with other models, the team:
1. Selected 80 diverse questions across eight categories
2. Used GPT-4 to judge the outputs of different models
3. Rated answers based on helpfulness, relevance, accuracy, and detail
The categories included:
- Fermi problems
- Role-play scenarios
- Writing tasks
- Coding and math problems
- And more
According to their results, GPT-4 assessed Vicuna to be 92% as good as ChatGPT responses - remarkably close in quality.
Hands-On Comparison
Testing Vicuna against ChatGPT on various tasks revealed impressive capabilities:
Creative Writing
When asked to compose a travel blog about Hawaii, Vicuna produced engaging, well-structured content comparable to ChatGPT's output:
*"Aloha fellow Travelers! If you're looking for a Tropical Paradise with rich culture and breathtaking scenery, look no further than Hawaii. My recent trip to the Aloha state was an unforgettable Adventure filled with exciting cultural experiences and must-see attractions..."*
Role-Playing Scenarios
GPT-4 evaluated Vicuna as being just as good as ChatGPT in role-playing scenarios.
Coding Tasks
Vicuna demonstrated strong coding abilities, successfully writing and explaining Python programs similar to how ChatGPT would. When asked to create a website with three buttons that display random jokes, Vicuna produced functional code (though it initially had a small error that was easily fixed).
Philosophical Questions
When asked about the meaning of life, Vicuna provided a thoughtful, nuanced response discussing various perspectives and acknowledging the subjective nature of the question.
Future Availability
The team plans to release the model weights by providing delta weights that build on the original LLaMA weights, though the exact release timeline is still being determined. Training Vicuna yourself would be challenging, requiring approximately eight A100 GPUs.
Looking Ahead
Another similar model called Cabana is expected to be released soon. The rapid progress in open-source large language models over just a few weeks since Alpaca's release is truly remarkable, showcasing the accelerating pace of AI development and democratization.
Links related to this post:
BONUS Links:
Arcee AI Mergekit GitHub:
Comments
Post a Comment