Open-Access AI: Lessons From Open-Source Software

In light of the explosive growth of generative AI, which the general public has adopted at a faster rate than personal computers or the Internet, it is natural to worry about who controls this technology. Most of the major industry players—including leading AI labs such as OpenAI (makers of ChatGPT), Anthropic (Claude), and Google (Gemini)—rely on closed models whose details are kept private and whose operation is entirely dependent on the whims of these (increasingly profit-hungry) private companies.

A notable exception to this trend of closed AI models is Meta, whose advanced foundation model, Llama, has publicly available parameters. Meta has gone all-in on what it calls “open-source AI”—what we, for reasons explained below, call “open-access AI”—going so far as to argue that such models are not only as good as but are indeed superior to closed models. Indeed, while open-access models have traditionally lagged behind their closed counterparts in performance, Meta, along with other makers of open-access foundational models such as IBM and AI chip giant Nvidia, has largely closed the gap. Although competition between closed and open-access models will no doubt continue, it is plausible that, going forward, there will not be a meaningful capability gap between open and closed models.

Meta’s argument in favor of openness in AI models relies heavily on analogy to open-source software. In a statement accompanying one of the Llama releases, Meta founder and CEO Mark Zuckerberg made a direct analogy to what is arguably open-source software’s greatest triumph: the development of the Linux operating system that runs a majority of the world’s computers, including most phones (Android being a fork of Linux) and web servers. He is not alone. 

But the comparison of open-access AI to open-source software is less compelling than it might first seem, because open-access models are substantially less open than the “open-source” moniker would suggest. While there are indeed many analogies between these two types of software, there are also important disanalogies, ranging from the nature of the technology itself and the access outside developers have over purportedly open models, to development costs, to the degree of centralization in a handful of companies. Some of the positive history of open-source software will likely carry over to open-access AI. But not all. Everyone—from industry techno-optimists releasing open-access models to serve open-source software’s ideals to policymakers wary of the threats introduced by such models—should be attentive to these differences to maximize the benefits and minimize the harms of open-access AI.

What Does “Open” AI Actually Mean?

Before analyzing how the lessons of open-source software might (or might not) apply to open-access AI, we need to define our terms and explain why we use the term “open-access AI” to describe models like Llama rather than the more commonly used “open-source AI.” We join many others in arguing that “open-source AI” is a misnomer for such models. It’s misleading to fully import the definitional elements and assumptions that apply to open-source software when talking about AI. Rhetoric matters, and the distinction isn’t just semantic; it’s about acknowledging the meaningful differences in access, control, and development. 

The software industry definition of “open source” grew out of the free software movement, which makes the point that “users have the freedom to run, copy, distribute, study, change and improve” software. As the movement emphasizes, one should “think of ‘free’ as in ‘free speech,’ not as in ‘free beer.’” What’s “free” about open-source software is that users can do what they want with it, not that they initially get it for free (though much open-source software is indeed distributed free of charge). This concept is codified by the Open Source Initiative as the Open Source Definition (OSD), many aspects of which directly apply to Llama 3.2. Llama 3.2’s license makes it freely redistributable by license holders (Clause 1 of the OSD) and allows the distribution of the original models, their parts, and derived works (Clauses 3, 7, and 8). 

But Llama 3.2’s license fails to be open source on many of its other prongs. For example, Llama 3.2 prohibits any company with more than 700 million monthly users at the time of the updated model’s release from using, modifying, or redistributing the model, its documentation, and its code—a term that violates Clause 5 of the OSD, which guarantees that open-source projects are universally accessible, without different restrictions or conditions based on user identity or purpose. Also, Llama 3.2’s enumerated list of disallowed uses and purposes for which the model, documentation, and code can be used violates Clause 6, which prohibits discrimination based on a person’s or group’s intentions vis-a-vis the project. And the change in license terms if Llama 3.2 is integrated into a product violates Clause 10, which requires the license governing use of the code to be technology neutral or consistent in provisions regardless of where the project is incorporated.

Most of these violations could be cured by small changes in Meta’s licensing without any technical changes. However, Llama 3.2’s violation of Clause 2, which requires that source code be in the preferred form for modification, is a nontrivial departure from the OSD. The availability and format of the source code for open-access models like Llama facilitates some modifications, such as tinkering with weights after the fact, but the absence of public access to the training data wholly precludes other forms of modifications, such as retraining the underlying model itself to generate new weights.

Open Weights: Tools to Tinker

At minimum, open-access AI models are open weight, meaning that they publicly disclose the model parameters. These weights and parameters are the numerical values that determine how inputs are processed to produce outputs—in other words, they determine how the model should transform a user prompt into a model response. Thus, access to weights and parameters is generally necessary to deploy a model. Open weights are also useful for some types of modification, since model weights can be “fine-tuned” to specialize a foundation model for a specific task. For example, AI engineers could fine-tune a general-purpose large language model to write lyrics in the style of Stephen Sondheim simply by retraining it on just his musicals’ lyrics, thereby tinkering with the model weights.

Open weights have enabled the proliferation of specialized and customized generative pretrained transformers (GPTs) based on Llama, including versions such as “badLlama,” which strip away all of the original model’s safeguards (and plausibly violates Meta’s license). In contrast, while OpenAI allows for the fine-tuning of ChatGPT, it does so through an application programming interface (API) that facilitates customization without making the model weights open. While APIs facilitate controlled customization, they also impose restrictions on what models can be fine-tuned with, such as prohibiting child sexual abuse material (CSAM) or copyrighted material, and require the data to be hosted on OpenAI’s servers, which implicates privacy. Put simply, users are not free to do with ChatGPT what they please—they are freer when it comes to modifying Llama, because it makes its model weights openly accessible. 

Open Data: Tools to Remodel

Open-weight models are not necessarily open data—a truer form of open-source AI, unlike open-weight models, which we characterize as merely open-access AI. Open data refers to not just the dataset used for training the model but also the source code and configuration details—such as hyperparameters—which dictate how the model learns from the data. Meta has not disclosed the data that went into, nor the architecture used for training, the Llama models. 

This is important because having access to the training data would maximize the ability to modify a model. While weights are sufficient for fine-tuning, the underlying training data is necessarily for training a potentially larger model or changing core architecture. Fine-tuning open weights allows for some modifications, such as specializing a model for specific tasks or adjusting its responses, but it’s limited to tweaking what’s already there. In contrast, retraining the underlying model to regenerate new weights requires access to the original training data and architecture, allowing for more fundamental changes and the creation of entirely new models and potentially new breakthroughs in model design. 

This distinction matters because while open weights enable incremental adjustments, only access to the underlying data allows for more comprehensive modifications such as rebuilding the model from the ground up or changing its core design. In particular, sophisticated developers, from hobbyists to professionals, need this deeper access to training data to build powerful new competitor models or novel solutions to complex problems. Without the training data, these users remain dependent on the original developers for any substantial updates or architectural changes.

Take Llama 3 as an example. It is a massive improvement on Llama 2. Could a particularly brilliant engineer outside Meta have used Llama 2 to build the next best thing? No, because access to the weights without the underlying data Llama 2 was trained on isn’t enough to go beyond tinkering. By withholding the underlying training data, Meta forecloses the possibility of a third party using Llama 2 to build a competitor to Llama 3. In open-source software, competing successor libraries are commonplace and are able to grow in their own direction. For instance, Apple was able to build off the open-source software community’s Berkeley Software Distribution and GNU Project to develop the open-source operating system Darwin that underlies their closed source macOS, iOS, and watchOS precisely because Apple had access to the original source code, not just the compiled programs. By contrast, so long as Llama 3.2’s training data is not publicly available, Meta has a monopoly on the terms and conditions of Llama’s successor models, as well as the engineering choices that define that model. 

Similarly, while current large language models are built off a “transformer” architecture, changing to a new model architecture—the high-level structure of how the model’s parameters interact—is impossible with just the weights. This is an important limitation because using a different architecture can significantly improve a model’s performance, efficiency, or suitability for a specific task. Different architectures allow for optimizations such as reducing computational requirements, handling larger or more complex data, or enhancing certain capabilities like reasoning, memory, or sequence processing. 

For instance, if developers wanted to adapt a general-purpose chat model for medical diagnosis, they might choose a different architecture, such as one that incorporates memory-augmented networks or a recurrent neural network (RNN) layer. These architectures can better handle sequential information and retain long-term context, which is essential for tracking a patient’s medical history, lab results, and treatment responses over time, paying heed to the chronology of inputs. By switching to an architecture designed to integrate and reason across multiple data points, the model could provide more accurate and context-aware diagnostic suggestions, rather than just generating responses based on the last input.

Open Development 

Another important difference between open-source software and open-access AI is in how they are developed. The key innovation of open-source software development is a social one: The development is decentralized among individuals and organizations rather than in one company. Two core social tools that define the open-source software ecosystem are pull requests (also known as merge requests and in some cases patches or contributions) and forks.

A pull request is a change to an open-source software project that is intended to be merged back into the original project. For example, chipmakers Intel and AMD both maintain the software inside the Linux kernel necessary to support their hardware. They regularly make contributions to the main open-source project specialized to their hardware in order to ensure that Linux has high-quality support for their hardware products. They are making incremental adjustments to an open-source project they use to better serve their needs, but in doing so they offer free labor that benefits other users of the same project. 

Creating pull requests through democratic and transparent processes maintained by most open-source projects is not possible for an open-weight AI. One cannot confidently contribute an improvement to a project without first being able to interpret and predict the model’s behavior, and how the proposed change would impact it—for that, one would need to understand and be able to experiment with the underlying training data. 

Unlike pull requests, which seek to modify an existing open-source project, forks are built off an existing project to make something new in a totally separate direction from the original project.

Forks, however, are incompatible with AI models that are merely open weight rather than also open data. Fine-tuning a model allows for adjustments to its behavior within the constraints of the original architecture and training data, but it does not create a truly independent version that can evolve in different directions. A fork, by contrast, involves branching off a project to pursue a new path with complete control over its core development, which, for the reasons described above, would require access to the original training data and the ability to modify the architecture. Fitting a neural network, or generating model weights, is a form of lossy compression—or the permanent trimming of unnecessary information to reduce size—of the training data; and once information is lost, it cannot be reliably recovered. One cannot deconstruct a cake to confidently identify all of its ingredients and how exactly it was baked. And so, one cannot recreate it (to eventually improve on it) perfectly. An open-weight model is much the same. And even if one could make minor changes to an open-weight model, those changes would have to be updated when the underlying model advances, imposing a heavy and ongoing development burden.

What Does Open-Source Software Reveal About Open-Access AI?

The open-source software movement has made many achievements in innovation, safety, and competition. To what extent should developers, scholars, and policymakers alike expect those achievements to occur with open-access AI? The answer is uncertain.

Innovation

Many supporters of open-access AI argue that it will increase innovation. For example, Zuckerberg draws on the success story of open-source software, particularly the rise of Linux, to justify his belief that open-access AI models will follow a similar trajectory. He asserts that, just as open-source Linux eventually outperformed and replaced many closed-source Unix systems, open-access AI will soon surpass proprietary models in both capability and utility. 

This comparison has much to recommend it; there are indeed parallels between open-source software and open-access AI, particularly in how transparency and community involvement can drive innovation. Just as Linux is the gold standard for cloud computing and supercomputer operating systems, so too might “future Llama models become the most advanced in the industry,” a bold but plausible prediction, given the monumental resources that Meta is investing in AI and the fact that Llama leads open-access AI models in popularity with downloads increasing 10-fold each year. Indeed, researchers have found that Llama 3.1 is either as good as or better than competitors on many benchmarks. 

Foundation models such as Llama are not on their own useful for much. To create even a chatbot (what they’re most easily deployed for) requires engineering a system around them and many other applications may require some amount of fine-tuning of the model. This work designing and engineering products on top of and with Llama can be developed with lower engineering costs for open models, and without the restrictions inherent to OpenAI’s APIs and the uncertainty of how OpenAI’s rates and availability may change.

But the ability of the community to contribute to and improve the performance of open-weight, rather than open-data, models is limited: flaws in the original model—whether they be bias, inaccuracies, or harmful behavior—cannot have architectural (and therefore, hopefully, robust) solutions developed by other parties. After-the-fact safety guardrails have had some limited success but are famously easy to dismantle by a dedicated bad actor for less than $200. Structural limitations of the model that are inherent to its architecture or training data are fully the responsibility of the party creating the open-weight AI in the first place. Downstream deployers, if they discover flaws, are limited in how they mitigate them and may be stuck going back and asking for the original developer to address them, not dissimilar from how they would handle issues with a proprietary model.

With companies such as Goldman Sachs, AT&T, and DoorDash using Llama for business functions like customer service, document review, and computer code generation, open-access AI is already lowering barriers to entry for AI customization. However, Llama’s terms require that products built on it have a license from Meta if they have over 700 million active monthly users. Meta thus maintains its chokehold on the market for viable competitors, which invites the question: Can something be truly innovative if it has to ask for permission? 

Safety

One of the most intriguing—and controversial—claims Zuckerberg made in defending open-access AI models is that they are not only as safe as closed models but potentially even safer. Zuckerberg’s statement flies in the face of the conventional wisdom that broadly distributed AI models pose serious and potentially even existential risks to public safety and national security, since they are free, public, customizable, and impossible to control the use and distribution of. Zuckerberg could have simply argued that the marginal safety risk of open models over their closed alternatives is outweighed by the benefits of open models. But the Meta CEO went much further, arguing, once again based on the analogy to open-source software, that open models are not only as safe as, but safer than, their closed counterparts.

To some extent the analogy holds true. Open-source software thrives on the principle that “many eyes make all bugs shallow”: the idea that open-source software, by its very nature, benefits from continuous scrutiny and improvement by a global community of developers, which in turn reduces the risk of unintentional harm. Similarly, open-access models invite everyone—researchers, developers, and tech enthusiasts alike—to perform relentless, worldwide red-teaming, the practice of actively searching for vulnerabilities by simulating attacks or trying to exploit weaknesses. Security vulnerabilities are often quickly identified and patched by the community. Independent researchers can dive into the model’s responses, checking for unfair or prejudiced outputs. Safety experts can explore potential pitfalls, like the model accidentally recommending harmful actions or spreading misinformation. And academics can dissect the model, publish their findings, and share insights that don’t just benefit the maker of the open model but can be applied across the entire AI landscape.

In stark contrast, closed models are locked down. The companies behind them decide who gets access, how much access they get, and what they can do with it. Internal red-teaming might catch some issues, but with fewer people involved, and with findings often kept under wraps, the process is slower and less thorough. This lack of transparency also means external experts can’t independently verify a model’s safety, which leaves only one option: simply trust the company’s word—a risky bet in any scenario.

But, as explained above, open-access AI is not as open as open-source software is, which weakens the safety analogy. Open-weight AI developers can only rely on the community security testing that doesn’t require understanding how the model was created. As the field of AI security continues to grow and transform, it is not clear yet what the cost trade-offs are to this limitation. Similarly, in the open-source software space, discoverers of security bugs are often able to get quite far in understanding why they occur and how to prevent them. It is not clear how this translates to open-weight AI, where, for example, a bias toward the use of the word “delve” may originate from extensive use of Nigerian-high-register English in the training set, but outside participants are limited in their ability to identify where or why (beyond educated guessing) this bias formed.

Of course, model design is just one aspect of AI safety. An even bigger danger is model misuse. While an open-access model’s ability to crowdsource red-teaming can help types of third-party misuse, which can in turn inform the design of safety guardrails, AI openness also presents novel safety risks. By withholding model weights, closed models such as ChatGPT significantly reduce the odds of third parties dismantling the safety guardrails, preventing harmful model uses. However, open-access AI with widely available model weights gives bad actors the information they need to more efficiently jailbreak the AI to serve ethically deplorable purposes, such as the generation of exploitative deepfakes and CSAM, to existentially nefarious purposes, such as the production of bioweapons. Admittedly, this is an area in which the gap between open-access and truly open-source AI may redound to the benefit of the former, since the difficulty (explained above) of innovating on open-access AI may lower the risk of model misuse relative to open-source AI. 

Competition

Open-source software has historically lowered barriers to entry and enabled a more dynamic, competitive environment. The hope is that open-access AI holds out the similar promise of enabling AI access for smaller players without having to play exclusively by a tech giant’s rules. Competition is possible with open-access AI because it democratizes access to advanced technology. By making the core models and tools freely available, open-access AI allows smaller players—including start-ups, academic institutions, and individual developers—to innovate on top of existing work without needing to reinvent the wheel. While this does not remove all barriers to smaller players—models, whether open or closed, are incredibly expensive to train and run—openness creates a foundation for a more vibrant and diverse ecosystem where new ideas can flourish. At least those who can surmount the scaling challenge will have the tools to do so, without being locked out by proprietary barriers.

Interoperability is another critical advantage of open-access AI, just as it has been for open-source software. Openness fosters a common framework that different tools and systems can build upon, making it easier for diverse AI applications to work together. This interoperability is essential in creating a competitive market—rather than build the entire castle again and again, open models provide the scaffolding or template for others to build compatible components, leading to more comprehensive and powerful solutions. Meta’s own growth is a testament to the power of open ecosystems. By embracing open-access AI, Meta has positioned itself not just as a tech leader but as a facilitator of broader innovation. In Europe, companies such as Mistral are already taking advantage of open-access AI to carve out their own space in the market, demonstrating that even in a landscape dominated by giants, the risks of innovation can be pushed to smaller players participating in meaningful competition. (However, Mistral recently announced an enormous partnership with Microsoft, casting some doubt on its continued commitment to openness.)

But, as outlined above, the differences between open-access AI and open-source software limit the competition benefits of the current generation of open models. The inability for smaller players to rebuild or form next generations of these open-weight models strengthens Meta’s control of the evolution of the ecosystem and preserves Meta’s presumably preferred structure to the industry: a handful of tech giants making foundational models and a litany of client companies taking risks to identify how to turn AI into profit. This problem has already played out in other, non-AI, software contexts: Google’s development of the open-source Blink and Chromium projects has allowed it to exert outsized control of how the web is developed and shaped, especially as other browsers such as Microsoft’s Edge and Brave have become repackaged versions of Chromium.

***

While Meta’s promotion of open-access AI as the natural successor to open-source software highlights significant potential for democratizing AI, boosting innovation, and enhancing safety, it risks oversimplifying the unique challenges that AI presents. While open-source principles of transparency and community-driven customization and testing can help identify areas for improvement of flaws, the inability to retrain models without the original data restricts comprehensive contributions. While AI regulation is outside the scope of this piece, it’s clear that whatever actions the government takes to maximize AI’s benefits and minimize its harms, it will have to account for these differences and not simply take the analogy between open-source software and open-access AI at face value.

– Parth Nobel, Alan Z. Rozenshtein, Chinmayi Sharma, Published courtesy of Lawfare.