Concept 2 of 2

Open weights and release decisions

3 questions test this

Releasing a model is not a choice between secret and public but a spectrum, and a model's position on it decides who can inspect it, who can change it and who can take it away. The argument conducted in public is about what a laboratory should publish. The decision most product people make is whether to build on somebody else's open weights.

The spectrum, from hosted to downloadable

At one end is a model reachable only through an interface its provider controls. Nobody outside sees the weights, and the provider can filter inputs and outputs, watch how it is used, change it and cut off an account.

In the middle sit weights released under conditions. A licence restricting fields of use, an application process, a research release, or a staged release with a smaller model published and a larger one held back.

At the far end are weights anybody may download, run locally, modify and pass on. That step cannot be undone, because copies exist beyond the publisher's reach the moment it happens.

Open weights is not open source

The two phrases are used interchangeably and are not the same thing. What gets published is normally the weights plus enough code to run them. The training data, the filtering decisions, the training code and the tuning that shaped the model's behaviour usually stay private.

That matters because weights alone do not let you reproduce a model or explain where a behaviour came from. You can test it, adapt it and run it privately, which is a great deal, and it is not the transparency open source normally promises. Several definitions of open source AI now exclude weights only releases.

The case for releasing

Scrutiny. A model anybody can run can be examined by anybody. Much of what is known about bias, memorisation and jailbreaking came from researchers working on downloadable models, and that is far harder through an interface that logs and rate limits.

Access. Research, teaching and work where there is no budget for hosted inference depend on models that run on ordinary hardware.

Competition and control. If capable models can only be rented from a handful of providers, a small number of organisations decide which uses are allowed and what a model may say. Open weights remove that single point of control, and for many people that is the strongest argument.

The case against

Safety training can be removed. Refusal behaviour lives in the weights, and fine tuning it away has been demonstrated repeatedly, cheaply and with little data. The safeguards a publisher ships are advisory for anyone who wants them gone.

Release cannot be reversed. A hosted model found to have a problem can be patched or withdrawn. Published weights cannot be recalled, so the decision is made against capabilities nobody has discovered the model has yet.

Misuse becomes invisible. A provider can see abuse patterns across its customers and intervene. Nobody sees what a downloaded copy is doing, which removes the monitoring and the ability to stop anyone.

Marginal risk, the test both sides use

The question that decides most releases is not whether a model can be misused. It is how much this release adds to what somebody could already do with existing open models and ordinary tools.

Publishers use it to argue for release, on the grounds that a capability already available adds little risk while the benefits are real. Critics use the same test against release, pointing out that the baseline moves with every publication, so a run of individually marginal steps arrives somewhere nobody assessed. Both ask the right question and disagree about the baseline.

What you are actually deciding

Building on open weights means running the model in your own environment, keeping data off a third party's systems, adapting it to your domain, and knowing it will not change underneath you.

It also moves responsibility onto you. The provider's filters, abuse monitoring, usage policy and incident process no longer sit between your users and the model, so evaluation, guardrails, logging and refusal behaviour become work your team owns. For many products that is a sound trade, and it is a trade rather than a saving.

Common misconceptions

Open weights means the model is open source.

Weights are the numbers a training run produced. The training data, the filtering decisions and the training code are usually not published, so you can run and adapt the model without being able to reproduce or audit how it came to behave as it does.

The safety training travels with the model, so an open model behaves the same wherever it runs.

Refusal behaviour is a property of the weights, and anybody able to fine tune can remove it cheaply. The safeguards you can rely on are the ones you run around the model, not the ones baked into it.

Release decisions are a laboratory problem with nothing in them for a product team.

Choosing to build on open weights is the same decision seen from the other end. You gain control and you also inherit the evaluation, monitoring and abuse handling the hosted provider was doing on your behalf.

3 questions test this concept

A vendor describes its model as open source on the grounds that the weights can be downloaded. What is wrong with the description?

  • ANothing, since publishing the weights is what open source means when the artefact is a model.
  • BThe training data, the filtering decisions and the training code usually stay private, so you can run and adapt the model without being able to reproduce or audit how it came to behave as it does.
  • CWeights are not software, so the term cannot apply to them at all.
  • DOpen source requires a permissive licence, and weights are always released under terms that restrict the field of use.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
AI Engineering, On hosting, adapting and evaluating a model you run yourself.