In a groundbreaking leak, a revealing Google memo exposes the truth: open source AI is taking the lead, leaving Google in its wake. The memo outlines Google’s acknowledgment of their inability to compete against the open source community. As shocking as it may be, Google admits defeat in the race for AI dominance and proposes a plan to regain control of the platform.
The Inevitability of Open Source
The memo starts by admitting that OpenAI was never the real competitor. The true contender was always open source. Google now recognizes that they are ill-equipped to compete against the open source AI ecosystem. Bluntly put, open source is outpacing Google at an alarming rate. Problems that Google considers significant obstacles are already solved and in the hands of the public.
“While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months.”
The Downfall of Size
Google realizes that their size is no longer an advantage. The massive scale of their models, which was once seen as an insurmountable advantage, is now a disadvantage. The leaked memo highlights key events signaling the rapid decline of Google’s and OpenAI’s control over AI.
The open source community obtained Meta’s leaked LLaMA model, and within a matter of days and weeks, they replicated and developed clones like Bard and ChatGPT. Techniques such as instruction tuning and reinforcement learning from human feedback (RLHF) were swiftly adopted by the open-source community at a fraction of the cost.
Scaling Woes
What particularly alarms Google is how effectively the open source community scales their projects. For instance, the open source ChatGPT clone, Dolly 2.0, had a question and answer dataset entirely created by thousands of employee volunteers. In comparison, Google and OpenAI relied on scraped content from sites like Reddit. The open source community’s dataset is deemed higher in quality due to contributions from professionals with longer and more substantial answers.
“Here we are, barely a month later, and there are variants with instruction tuning, quantization, quality improvements, human evals, multimodality, RLHF, etc. etc. many of which build on each other. Most importantly, they have solved the scaling problem to the extent that anyone can tinker. Many of the new ideas are from ordinary people. The barrier to entry for training and experimentation has dropped from the total output of a major research organization to one person, an evening, and a beefy laptop.”
Open Source’s Triumph
This development mirrors past experiences where open source surpassed closed source models. OpenAI’s DALL-E and the open source Stable Diffusion serve as a clear example. Stable Diffusion overtook DALL-E in popularity just three weeks after its release, despite DALL-E being available for a year and a half. The memo exposes the existential threat of similar events overtaking Bard and OpenAI, causing Google to fear for its future.
The Open Source Advantage
Another factor that raises concern among Google engineers is the speed and cost-effectiveness of creating and improving open source models. Techniques like LoRA allow for fast and inexpensive fine-tuning of language models. Rather than starting from scratch, open source engineers can build upon previous work, resulting in a rapid pace of improvement that outstrips Google’s efforts.
“By contrast, training giant models from scratch not only throws away the pretraining but also any iterative improvements that have been made on top. In the open source world, it doesn’t take long before these improvements dominate, making a full retrain extremely costly. We should be thoughtful about whether each new application or idea really needs a whole new model…Indeed, in terms of engineer-hours, the pace of improvement from these models vastly outstrips what we can do with our largest variants, and the best are already largely indistinguishable from ChatGPT.”
Embracing Open Source
The memo concludes with a surprising realization: Google must embrace open source. They acknowledge that open source innovations are freely available and propose owning the platform, similar to their domination of Chrome and Android. They point to Meta’s success in releasing their LLaMA model for research, leveraging thousands of contributors for free. The memo highlights that going open source is not only the most viable option but also the path to innovation and control.
“Google should establish itself as a leader in the open source community, taking the lead by cooperating with, rather than ignoring, the broader conversation. This probably means taking some uncomfortable steps, like publishing the model weights for small ULM variants. This necessarily means relinquishing some control over our models. But this compromise is inevitable. We cannot hope to both drive innovation and control it.”
Open source AI is revolutionizing the landscape, and it seems Google intends to adapt and join the movement. The memo confirms the dire situation Google faces and the need for a strategic shift in order to secure their future. With open source walking away with the AI fire, Google’s next move may determine its fate.
Read the leaked Google memo here.
