\n<\/p>\n
Elon Musk\u2019s xAI released<\/a> its Grok<\/a> large language model as \u201copen source\u201d over the weekend. The billionaire clearly hopes to set his company at odds with rival OpenAI<\/a>, which despite its name is not particularly open. But does releasing the code for something like Grok actually contribute to the AI development community? Yes and no.<\/p>\n Grok<\/a> is a chatbot trained by xAI to fill the same vaguely defined role as something like ChatGPT or Claude: you ask it, it answers. This LLM, however, was given a sassy tone and extra access to Twitter data as a way of differentiating it from the rest.<\/p>\n As always, these systems are nearly impossible to evaluate, but the general consensus seems to be that it\u2019s competitive with last-generation medium-size models like GPT-3.5. (Whether you decide this is impressive given the short development time frame or disappointing given the budget and bombast surrounding xAI is entirely up to you.)<\/p>\n At any rate, Grok is a modern and functional LLM of significant size and capability, and the more access the dev community has to the guts of such things, the better. The problem is in defining \u201copen\u201d in a way that does more than let a company (or billionaire) claim the moral high ground.<\/p>\n This isn\u2019t the first time the terms \u201copen\u201d and \u201copen source\u201d have been questioned or abused in the AI world. And we aren\u2019t just talking about a technical quibble, such as picking a usage license that\u2019s not as open as another (Grok is Apache 2.0, if you\u2019re wondering).<\/p>\n To begin with, AI models are unlike other software when it comes to making them \u201copen source.\u201d<\/p>\n If you\u2019re making, say, a word processor, it\u2019s relatively simple to make it open source: you publish all your code publicly and let community to propose improvements or make their own version. Part of what makes open source as a concept valuable is that every aspect of the application is original or credited to its original creator \u2014 this transparency and adherence to correct attribution is not just a byproduct, but is core to the very concept of openness.<\/p>\n With AI, this is arguably not possible at all, because the way machine learning models are created involves a largely unknowable process whereby a tremendous amount of training data is distilled into a complex statistical representation the structure of which no human really directed, or even understands. This process cannot be inspected, audited, and improved the way traditional code can \u2014 so while it still has immense value in one sense, it can\u2019t ever really be open. (The standards community hasn\u2019t even defined what open will be<\/a> in this context, but are actively discussing it<\/a>.)<\/p>\n That hasn\u2019t stopped AI developers and companies from designing and claiming their models as \u201copen,\u201d a term that has lost much of its meaning in this context. Some call their model \u201copen\u201d if there is a public-facing interface or API. Some call it \u201copen\u201d if they release a paper describing the development process.<\/p>\n Arguably the closest to \u201copen source\u201d an AI model can be is when its developers release its weights<\/em>, which is to say the exact attributes of the countless nodes of its neural networks, which perform vector mathematics operations in precise order to complete the pattern started by a user\u2019s input. But even \u201copen-weights\u201d models like LLaMa-2 exclude other important data, like the training dataset and process \u2014 which would be necessary to recreate it from scratch. (Some projects go further<\/a>, of course.)<\/p>\n All this is before even mentioning the fact that it takes millions of dollars in computing and engineering resources to create or replicate these models, effectively restricting who can create and replicate them to companies with considerable resources.<\/p>\n So where does xAI\u2019s Grok release fall on this spectrum?<\/p>\n As an open-weights model, it\u2019s ready for anyone to download, use, modify, fine tine, or distill<\/a>. That\u2019s good! It appears to be among the largest models anyone can access freely this way, in terms of parameters \u2014 314 billion \u2014 which gives curious engineers a lot to work with if they want to test how it performs after various modifications.<\/p>\n The size of the model comes with serious drawbacks, though: you\u2019ll need hundreds of gigabytes of high-speed RAM to use it in this raw form. If you\u2019re not already in possession of, say, a dozen Nvidia H100s in a six-figure AI inference rig, don\u2019t bother clicking that download link.<\/p>\n And although Grok is arguably competitive with some other modern models, it\u2019s also far, far larger than them, meaning it requires more resources to accomplish the same thing. There\u2019s always a hierarchy of size, efficiency, and other metrics, and it\u2019s still valuable, but this is more raw material than final product. It\u2019s also not clear whether this is the latest and best version of Grok, like the clearly tuned version some have access to via X.<\/p>\n Overall, it\u2019s a good thing to release this data, but it\u2019s not a game-changer the way some hoped it might be.<\/p>\n It\u2019s also hard not to wonder why Musk is doing this. Is his nascent AI company really dedicated to open source development? Or is this just mud in the eye of OpenAI, with which Musk is currently pursuing a billionaire-level beef<\/a>?<\/p>\n If they are really dedicated to open source development, this will be the first of many releases, and they will hopefully take the feedback of the community into account, release other crucial information, characterize the training data process, and further explain their approach. If they aren\u2019t, and this is only done so Musk can point to it in online arguments, it\u2019s still valuable \u2014 just not something anyone in the AI world will rely on or pay much attention to after the next few months as they play with the model.<\/p>\n<\/p><\/div>\n