Meta has released their Llama3 model, an AI chatbot assistant similar to GPT-4. The initial results have been quite impressive, with Llama3 performing well on various benchmarks. The 70 billion parameter version of Llama3 can be run on a beefy desktop machine and has achieved 82% on the HumanEval coding task benchmark, which is close to the performance of modern proprietary systems.

Llama3 has also performed well on the GPQA scientific test, scoring close to 40%, which is very impressive. However, its performance on math-focused tasks is not as strong, with a 50% score, compared to OpenAI's GPT-4 which can achieve over 70% on similar math problems.

Additionally, Meta is working on an even larger 400-ish billion parameter model, which is expected to have significant improvements over the 70 billion parameter version across various tests. This larger model is still in development but is expected to be released sometime before the end of the year.

The impressive performance of Llama3 is particularly noteworthy because it is an open-source model, available for free for everyone to use. This is in contrast to proprietary systems like GPT-4, which are not freely accessible. The fact that Llama3 is comparable to earlier versions of GPT-4 in some aspects is a testament to the progress of open-source AI models.

When it comes to evaluating the performance of AI models, the choice of benchmarks is crucial. The ideal benchmark should strike a balance between being challenging enough to differentiate models, yet not so difficult that the models struggle to perform well.

Tests with a success rate below 10% are generally not considered great for testing, as it becomes harder to draw statistically significant conclusions. On the other hand, tests with an 80-85% success rate or higher may start to lose meaning, as they could potentially contain errors or have answers that have been leaked, allowing models to simply replicate the correct responses.

The GPQA test, which covers a range of scientific disciplines like organic chemistry, molecular biology, and physics, is cited as a good example of a challenging yet meaningful benchmark. Achieving close to 40% on this test is considered very impressive.

When it comes to evaluating the performance of AI models for real-world use cases, the Arena leaderboard provides a more relevant assessment. This test measures the models' abilities to assist with tasks that normal people would typically perform, rather than just academic or technical challenges.

The key insights are that finding the right balance in benchmark difficulty is crucial, and that open-source models like Llama3 are now performing remarkably well, often rivaling or even surpassing proprietary systems like GPT-4 on various tests.

Llama3, Meta's open-source AI chatbot, has shown impressive performance on various benchmarks, rivaling proprietary systems like GPT-4. With a 70 billion parameter version available for use on beefy desktop machines, Llama3 has achieved 82% on the HumanEval coding task, and an impressive 40% on the challenging GPQA scientific test. While its math performance at 50% lags behind GPT-4's 70%, the upcoming 400 billion parameter model is expected to further improve on these results.

Equally impressive is Google DeepMind's Gemini 1.5 Pro, which has demonstrated strong performance on the Arena leaderboard, even surpassing earlier versions of GPT-4 in some aspects. Gemini 1.5 Pro's ability to learn from entire movies and codebases sets it apart, and it has achieved second place overall, as well as first place for longer prompts.

The availability of these powerful AI assistants, both open-source and proprietary, marks a significant advancement in the field. Users can now access Llama3 and Gemini 1.5 Pro for free, outside of the EU, through various platforms, allowing them to experience the capabilities of these cutting-edge models firsthand.

You can try the Llama3 and Gemini 1.5 Pro AI models for free in the following ways:

1. Llama3:

   • To the best of the author's knowledge, you can try Llama3 on Meta's website, if you are located outside of Europe.
   • You can also download and run the Llama3 model anywhere.
   • There are websites that are hosting Llama3, and you can try it through these platforms. The links are provided in the video description.

2. Gemini 1.5 Pro:

   • Gemini 1.5 Pro, developed by Google DeepMind, is also available for free to try, outside of the EU.
   • The link to access Gemini 1.5 Pro is provided in the video description.

The author emphasizes that both Llama3 and Gemini 1.5 Pro are impressive AI models that are nearly as good as paid proprietary systems, but you can use them free of charge.

AI assistants are improving at a stunning pace, and open-source models like Llama3 and Gemini 1.5 Pro are now nearly as capable as the paid proprietary systems, such as GPT-4. These open-source models are available for free for everyone to use, which is an incredible development.

Llama3, the 70 billion parameter model from Meta, is performing remarkably well on various benchmarks, including the challenging GPQA test in scientific domains. While it may not be as strong in math as GPT-4, it still delivers impressive results. Moreover, the upcoming 400 billion parameter Llama3 model is expected to further improve upon these capabilities.

Similarly, Google DeepMind's Gemini 1.5 Pro is also a highly capable AI assistant, rivaling the performance of earlier versions of GPT-4. It excels particularly in tasks involving longer prompts and complex contexts, such as watching entire movies or learning codebases.

The availability of these open-source AI models, which can be tried for free, is a significant milestone. It democratizes access to advanced AI technology and allows everyone to explore and benefit from these powerful tools. This is a wonderful time to be alive, as we witness the rapid progress of AI and the increasing accessibility of these remarkable capabilities.