As investors weigh OpenAI’s valuation, they might consider the humble paperclip. A cautionary tale about corporate profit maximizers building a robot that so excels in producing the office supply that it wipes out humanity might seem far-fetched. But a single-minded capitalist could make the economically rational decision to bear such a risk. As OpenAI races towards a fundraising that could value it at $150 billion, the implicit promise is that gains enormous enough to make that danger thinkable are on the horizon. That itself underscores the barriers to growth.

The paperclip story goes like this. One day, engineers at ACME Office Supplies unveil a hyper-sophisticated AI machine with one goal: produce as many paperclips as possible. The incomparable silicon intellect chases this task to the furthest extreme, converting every molecule on Earth into paperclips and promptly ending all life.

Profit-hungry OpenAI investors like Microsoft might be assumed, like ACME, to only value short-term gains, inviting the risk that they build their own Paperclip Maximizer. Sam Altman, OpenAI’s CEO, says that he is mindful of the risk. His company’s structure is meant to limit bad incentives, capping profit available to investors. Such protections are worth an asterisk now: a ceiling on profit was set in 2019 at a 100 times return for initial investors. OpenAI initially expected to lower it over time. Instead, the company’s latest fundraising now hinges on changing that structure, including by removing the cap, Reuters reported.