Gemma 3 is the latest open-weight model released by Google, and it has been making waves in the AI community. This multimodal model is particularly impressive for its size, as it is the first open-weight model of its size to crack the top 10 positions on the ChatBot Arena leaderboard, with an impressive score of 1339.

This places Gemma 3 right between DeepSEQ V3 and DeepSEQ R1 on the leaderboard, which is a remarkable achievement. The Gemma 3 family consists of four different models, ranging from 1 billion to 27 billion parameters, allowing Google to cater to a wide range of devices and use cases.

The larger models in the Gemma 3 family, with their 128,000 token context window, are particularly versatile, as they can process inputs in the form of text, images, and short videos, making them highly capable in a variety of applications.

One of the key focuses in the development of Gemma 3 was improving the model's performance in terms of human preferences, which is reflected in its impressive ChatBot Arena leaderboard score. This was achieved through a multi-stage post-training strategy, including distillation from larger models, reinforcement learning from human feedback, and reinforcement learning from machine feedback to improve mathematical reasoning and coding capabilities.

The availability of official quantized versions of the Gemma 3 models, ranging from 32-bit to 4-bit precision, further enhances the model's accessibility and usability, as users can choose the appropriate version based on their hardware constraints.

Overall, the impressive performance of Gemma 3 on the ChatBot Arena leaderboard, combined with its multimodal capabilities and the availability of optimized versions, make it a highly compelling open-weight model for a wide range of applications.

Google has recently released the Gemma 3 model family, which includes four different models ranging from 1 billion to 27 billion parameters. This latest open-weight model from Google is not only the largest in the Gemma family but also a multimodal model, capable of processing text, images, and short videos.

The Gemma 3 models have shown impressive performance, with the 27 billion parameter model ranking among the top 10 on the Chatbot Arena leaderboard, scoring 1339. This places it between the DeepSEQ V3 and DeepSEQ R1 models, a remarkable feat for a model of its size.

The Gemma 3 family was trained using a new tokenizer, enabling multilingual support for over 140 languages. The smaller 1 billion parameter model is English-only, while the larger models have a much larger context window of 128,000 tokens.

The training process for the Gemma 3 models involved a four-stage post-training strategy. First, the models were distilled from larger InstructGPT models. Then, reinforcement learning from human feedback was used to align the model's predictions with human preferences. This is reflected in the model's strong performance on the Chatbot Arena leaderboard. Additionally, reinforcement learning from machine feedback was used to improve the models' mathematical reasoning capabilities, and reinforcement learning from execution feedback was employed to enhance their coding abilities.

Google has made the weights for all four Gemma 3 models available on Hugging Face, providing both base and InstructGPT versions. The company has also released official quantized versions of the models, ranging from 32-bit to 4-bit precision, making them accessible to a wider range of hardware, including consumer-grade GPUs like the NVIDIA 3090, 1490, or 1590.

While the Gemma 3 models show promise in areas like creative writing, the smaller models may not be suitable for coding tasks, where larger models like Sonet 3.5 may be more appropriate. The multimodal capabilities of the Gemma 3 models are also worth exploring, but the current implementation in AI Studio does not seem to support image input.

Overall, the Gemma 3 model family represents a significant advancement in Google's open-weight model offerings, providing a range of options for various applications and hardware requirements.

The Gemma 3 family of models was trained using several innovative techniques to improve their performance and capabilities:

1. New Tokenizer for Multilingual Support: The models were trained using a new tokenizer that enables multilingual support, allowing them to process inputs in more than 140 languages.

2. Varying Training Data Sizes: The different Gemma 3 models were trained on varying amounts of pre-training data, ranging from 2 trillion tokens for the smaller 1 billion model to 14 trillion tokens for the larger 27 billion model.

3. Multi-Stage Post-Training: The models underwent a four-stage post-training process:

   • Distillation from larger InstructGPT models into the Gemma 3 pre-training checkpoints.
   • Reinforcement learning from human feedback to align the model's predictions with human preferences.
   • Reinforcement learning from machine feedback to improve mathematical reasoning capabilities.
   • Reinforcement learning from execution feedback to enhance the models' coding abilities.

4. Multimodal Capabilities: The larger Gemma 3 models can process not only text, but also images and short videos as inputs, making them truly multimodal.

These innovations and improvements have resulted in the Gemma 3 models achieving impressive performance, particularly on the ChatGPT Arena leaderboard, where the 27 billion parameter model ranks among the top 10 models in terms of human preference scores.

The weights for all four Gemma 3 models are available on Hugging Face. Google has released both the base and instruct versions of these models. The base versions are particularly useful if you want to fine-tune your own models.

Google has also provided official quantized versions of all the Gemma 3 models, which is great news. These models are available in 32-bit, 16-bit, 8-bit, and 4-bit precision. This gives you flexibility in choosing the right model and precision based on your hardware capabilities.

To run the 27 billion parameter Gemma 3 model at full precision, you would likely need a couple of NVIDIA H100 GPUs. However, if you use the 16-bit quantized version, a single H100 with 80GB of VRAM should be sufficient. For even more constrained hardware, such as consumer GPUs like the RTX 3090, 3080, or 3070, the 4-bit quantized versions can be used.

Google's commitment to releasing these powerful open-source models with a relatively permissive license is commendable. While the license is not as permissive as Apache 2.0 or MIT, it is still very accessible, making these models available for a wide range of use cases.

Google's Gemma 3 is the latest open-weight multimodal model, boasting impressive performance on the ChatGPT Arena leaderboard. This family of models, ranging from 1 billion to 27 billion parameters, aims to cater to a wide range of applications, from smaller edge devices to more powerful systems.

The 27 billion model, in particular, stands out, often outperforming even larger models like Chinchilla 2.5 on various benchmarks. This suggests it could be a strong candidate for creative writing tasks.

To explore the capabilities of Gemma 3, I tested the model on a few tasks:

1. Text Generation: When prompted to "tell me about yourself," the model generated a verbose and engaging response, showcasing its strong language generation abilities.

2. Coding: However, when asked to generate an HTML script for a bouncing red ball within a hexagon, the model struggled, taking several minutes to produce code that ultimately lacked the requested functionality. This suggests that while Gemma 3 can handle coding tasks, it may not be the best choice for complex coding assistance, especially when compared to more specialized models like Codex.

3. Multimodal Capabilities: I also attempted to test the model's multimodal capabilities by uploading an image, but encountered issues, as the current implementation does not seem to support image input. This is a bit disappointing, as the model is touted as being multimodal.

Overall, Gemma 3 appears to be a strong performer for language-based tasks, particularly in the realm of creative writing. However, for more specialized tasks like coding, it may be prudent to consider alternative models that are better suited for those use cases. Further exploration of the model's multimodal capabilities would be needed to fully assess its potential in that area.

While the Gemma 3 family of models from Google showcases impressive performance on various benchmarks, including the ChatGPT Arena leaderboard, there are some limitations and recommendations to consider when using these models:

1. Coding Capabilities: The smaller Gemma 3 models (1 billion and 6 billion parameters) may not be suitable for complex coding tasks. The presenter recommends using larger, proprietary models like Codex or Sonet 3.5 for coding assistance, as the open-source Gemma 3 models may not have the necessary capabilities.

2. Multimodal Support: While the Gemma 3 models are advertised as multimodal, supporting text, images, and short videos, the presenter encountered issues when trying to upload and process images. This suggests that the multimodal capabilities may not be fully reliable or consistent across the different model sizes.

3. Inference Time: The presenter noted that generating the HTML code for a bouncing red ball within a hexagon took close to 3 minutes, which may be too slow for some real-time applications.

4. Quantization and Hardware Requirements: The presenter highlighted the availability of quantized versions of the Gemma 3 models, which can reduce the hardware requirements for running these models. However, the hardware needed to run the larger 27 billion parameter model in full precision is still significant, requiring multiple H100 GPUs with 80GB of VRAM.

5. Safety and Content Moderation: The presenter encountered some issues with the model triggering warnings for unsafe content, suggesting that the safety and content moderation aspects of the Gemma 3 models may need further improvement.

In summary, the Gemma 3 models can be a valuable resource, especially for creative writing tasks, but users should carefully consider the limitations and hardware requirements when choosing to deploy these models in their applications. The presenter recommends using larger, proprietary models for more complex tasks like coding, and further exploring the multimodal capabilities of the Gemma 3 models in future projects.