Indian artificial intelligence lab Sarvam has launched a new, multi-lingual generation of large language models. This launch is an important milestone in the development of open-source AI. Sarvam came onto the generative AI scene in 2023 and immediately turned a lot of heads. The firm has since raised more than $50 million in funding from big names such as Lightspeed Venture Partners, Khosla Ventures, and Peak XV Partners.
This record-breaking new generation includes the first-ever advanced models with 30 billion and 105 billion parameters. These vehicles are fully-loaded with advanced features like text-to-speech, speech-to-text and vision. The 30-billion parameter model is even crazier, with a 32k-token context window. The 105-billion parameter model goes even further with an astonishingly wide 128,000-token context window.
Sarvam’s models use a mixture-of-experts architecture to ensure efficiency and adaptability to different tasks. The 30B model was initially pre-trained on about 16 trillion tokens of text. The 105B model was trained on trillions of tokens which covered a plethora of Indian languages. This rigorous training prepares them to tackle time-sensitive applications, such as call center automation, conversational agents designed for Indian languages and voice based assistants running on the cloud.
As Sarvam’s work progresses, it intends to open-source the 30B and 105B models. This new innovation represents the next step in the company’s commitment to developing tools that encourage collaboration and accessibility in the AI community.
Pratyush Kumar, a leader at Sarvam, told us that in their building of the model, being deliberate about scaling was crucial.
“We want to be mindful in how we do the scaling,” – Pratyush Kumar
It’s this careful approach that sets Sarvam apart. Beyond encouraging broader, more equitable innovation, it hopes to produce models that are powerful and useful enough to address user needs.
“We don’t want to do the scaling mindlessly. We want to understand the tasks which really matter at scale and go and build for them.”
This deliberate methodology sets Sarvam apart as it seeks to create models that are not only powerful but also relevant to users’ needs.