Over the past few years, the conversation around artificial intelligence has often centred on safety:

Safety research.

Safety pledges.

Safety frameworks.

For a time, it appeared that the frontier AI companies might converge around voluntary restraint.

The idea was simple: if models became too capable relative to available safeguards, development would pause.

That premise assumed something fragile.

Alignment of incentives.

And incentives rarely remain aligned for long.

In high-velocity technological races, safety does not remain purely an ethical consideration.

It becomes a ‘strategic variable’.

Share Something For The Weekend

The Original Premise

Many AI developers began with a relatively straightforward principle:

If model capabilities outpaced the ability to control them safely, training would pause.

This was framed as responsible scaling.

In theory, the approach had logic: slow the system before it outruns governance.

But voluntary brakes only work if everyone applies them simultaneously.

Once one actor accelerates, restraint becomes asymmetrical.

And asymmetry becomes competitive disadvantage.

The Competitive Reality

AI development has become one of the most competitive industrial races of the modern era:

Venture capital.

Private credit.

Hyperscale compute.

Government contracts.

Enterprise deployment.

The incentive gradient is steep.

Companies that pause risk losing technological leadership.

Companies that accelerate capture market share and infrastructure control.

Under those conditions, unilateral constraint becomes fragile.

Safety frameworks evolve.

What begins as a hard stop often becomes a softer mechanism:

Not a pause.

A goal.

Not a rule.

A guideline.

That evolution is rarely driven by ideology.

It is driven by incentives.

A Familiar Pattern

If this dynamic feels familiar, it should.

Cybercrime has been operating on similar incentive structures for years.

Attackers study constraints.

They analyse insurance policies discovered during breaches.

They calibrate ransom demands below coverage limits.

They target organisations where recovery economics favour payment.

This is optimisation.

The attacker’s objective function is simple: maximise payout while minimising resistance.

AI development operates within a different domain but under the same economic logic.

Optimisation against constraints.

Incentives Over Intent

This is not a story about good actors and bad actors.

It is a story about systems responding to incentives:

When incentives reward speed, systems accelerate.

When incentives reward scale, systems expand.

When incentives reward safety, systems slow down.

The difficulty is that incentives rarely align perfectly:

Markets reward capability.

Governments reward strategic advantage.

Investors reward growth.

Safety frameworks must operate inside that reality.

Not outside it.

The Structural Problem

Voluntary restraint works best in stable systems.

Frontier technologies are not stable systems.

They are competitive races with global participants.

In those conditions, safety becomes a coordination problem.

Each actor faces a dilemma:

If everyone slows, safety improves.

If one accelerates, advantage shifts.

This is the competitive trap.

No individual participant wants to be the one who brakes first.

The Parallel with Cyber Risk

Cyber attackers understand incentive systems well:

They analyse organisational structures.

They model insurance thresholds.

They monitor incident response patterns.

In many cases, ransom demands are calibrated deliberately.

High enough to be profitable.

Low enough to be payable.

The objective is not destruction.

It is optimisation.

AI development is governed by the same economic gravity.

Different actors.

Different objectives.

Same logic.

The Board Question

For boards, the issue is not whether frontier technologies will continue accelerating.

They will.

The question is how governance frameworks respond when safety becomes strategically negotiable and technology velocity is all but assured.

Boards should now be asking:

• Are we relying on voluntary safety commitments from vendors whose incentives may change?

• Do we understand the competitive pressures shaping AI deployment decisions?

• Are our internal AI adoption decisions influenced more by capability race than risk architecture?

• Do our risk models assume adversaries who optimise economically, not just technically?

• Are we prepared for safety frameworks to evolve under competitive pressure?

The governance challenge is subtle.

It is not about mistrust.

It is about recognising incentive structures.

What This Means for Strategy

Strategy in frontier technologies cannot rely on static assumptions:

Safety frameworks will evolve.

Competitive pressures will intensify.

Capability races rarely slow down.

Organisations therefore need to design resilience under conditions where:

Technological acceleration is structural.

Incentives will shift.

Safety commitments may adapt.

That means building architectures that assume volatility rather than stability:

Vendor risk models must include incentive drift.

Cyber defence must assume economically rational adversaries.

Governance must anticipate that competitive dynamics reshape risk boundaries.

In short:

Do not assume the rules will hold.

Design systems that remain resilient when they move.

The Deeper Strategic Question

There is a temptation to frame the safety debate as ethical.

That is incomplete.

The real issue is economic.

If those incentives remain misaligned, voluntary restraint will struggle to persist.

History offers many examples:

Financial markets.

Arms races.

Industrial competition.

Frontier technologies follow the same pattern.

The Weekend Takeaway

In high-speed technological races, safety cannot rely solely on intention.

It must be engineered into systems that remain stable even when incentives shift.

Because when competition intensifies, voluntary brakes rarely hold.

The real challenge is not preventing acceleration.

It is ensuring that resilience scales alongside it.

Leave a comment

📚 Further Reading

Governance of AI Program (Centre for the Governance of AI): Uncertainty, Information, and Risk in International Technology Races (2023)
Explores how competition between AI developers can create incentives to cut corners on safety, potentially triggering a “race to the bottom” in responsible deployment.

ScienceDirect / Computers & Security: Between a rock and a hard(ening) place: Cyber insurance in the ransomware era (2023)
Examines how cyber-insurance interacts with ransomware incentives and whether insurance coverage can inadvertently influence attacker behaviour and ransom dynamics.

UCL / Royal United Services Institute: How Cyber-Insurance Influences the Ransomware Payment Decision (2023)
Academic study analysing whether organisations with cyber-insurance are more likely to pay ransomware and how insurance alters attacker-victim incentives.

Oxford University Press - Industrial and Corporate Change: Competition Between AI Foundation Models (2024)
Analyses the emerging competitive dynamics between large AI model developers and the regulatory implications of rapid capability races.

ScienceDirect: Voluntary safety commitments provide an escape from over-regulation in AI development (2022)
Game-theoretical analysis of whether voluntary commitments by AI developers can meaningfully improve safety outcomes without enforceable coordination.

SSRN: Racing to Safety - Tax Policy for AI Safety-by-Design (2025)
Proposes economic and policy incentives to align competitive AI development with stronger safety outcomes through taxation and market mechanisms.

arXiv: Vigilant Incentives Help Regulatory Markets Improve AI Safety (2023)
Uses evolutionary game theory to examine how regulatory incentive structures can influence AI developers’ safety behaviour in competitive environments.

SSRN: Insurance Against Ransomware (2022)
Economic modelling of how cyber-insurance availability can unintentionally increase ransomware frequency and severity by altering attacker incentives.

ScienceDirect: Strategic Insights from Simulation Gaming of AI Race Dynamics (2025)
Simulation-based research showing how competitive dynamics between states and firms can increase the probability of AI safety failures.

RUSI - Royal United Services Institute: Insurance as Crime Governance: Comparing Kidnap for Ransom and Ransomware (2023)
Explores how insurance markets shape incentives in extortion-based crime and how attackers adapt to the financial structures of victims.

🧠 Techie Corner for the Non-Techies

Strip away the jargon.

When people talk about “AI safety,” they often mean two technical things.

First: capability control

AI models can perform increasingly complex tasks. Safety mechanisms attempt to ensure those capabilities cannot be misused.

This includes techniques such as:

• Reinforcement learning with human feedback (training models to follow certain behaviours).

• Guardrails that prevent models from generating harmful outputs.

• Monitoring systems that detect misuse.

Second: deployment governance

Even safe models can create risk if deployed incorrectly.

That is why companies implement:

• Usage policies.

• Access controls.

• Monitoring systems that track how models are used.

But here is the technical challenge.

AI models are general-purpose systems.

Unlike traditional software, they can perform many tasks depending on the instructions they receive.

That flexibility is powerful.

It is also difficult to constrain perfectly.

In competitive markets, companies face pressure to release more capable models quickly.

Safety research therefore becomes a balancing act:

Improve safeguards without slowing development too much.

This tension between capability and control is not unique to AI.

It appears in many frontier technologies.

AI simply compresses the timeline.

Last Week on Something for the Weekend

Last week, we explored how artificial intelligence acts as an aggression multiplier.

AI amplifies four forces simultaneously:

Speed, scale, strategic calibration and democratisation.

Attacks move faster.

Operations scale across thousands of targets.

Extortion becomes economically optimised.

And the expertise barrier for launching sophisticated operations falls dramatically.

When both attackers and defenders use AI, the system does not stabilise.

It becomes more volatile.

Boards therefore need to assume adversaries who are not only technically capable, but economically rational and AI-augmented.

The strategic implication was clear:

Cyber risk is no longer episodic.

It is a continuous campaign.

Read here: 👇

When the elusive Higgs boson was finally confirmed at CERN in 2012 — nearly half a century after he had proposed its existence — Professor Peter Higgs gave a characteristically understated response:

“It’s very nice to be right sometimes.”

That remark captured the man as much as the science: quiet, modest, almost diffident, yet with an intellect that reshaped our understanding of the Universe. With his passing, it feels important to pause and reflect not just on his scientific legacy, but on the human qualities that made it possible.

A scientist’s respect

Before becoming a CEO in the insurance and risk world, I was a scientist. That training never leaves you: the discipline of asking questions, testing assumptions, and trusting evidence over noise.

That is why the life and work of Peter Higgs strike such a chord with me. There is something profoundly moving about the way a single idea, quietly developed against the grain, can eventually shift the foundations of what we know about reality.

I remember following the Higgs boson discovery in 2012 with a mix of awe and recognition. Awe at the sheer scale of the experiment - a $10bn machine buried under the Franco-Swiss border, built to test one man’s half-century-old theory. Recognition of the patience, persistence, and faith it takes to pursue a question when the answers may only come decades later, perhaps even beyond your own lifetime.

Subscribe now

The question that changed physics

Born in Newcastle in 1929, Higgs was a brilliant student who first shone in chemistry before turning to physics. After completing his PhD at King’s College London, he took up a post at the University of Edinburgh. There, in the early 1960s, he began to wrestle with a fundamental question: why do some particles have mass while others do not?

The answer he proposed was radical: a field that permeates space, interacting with particles to give them mass. Attached to that field would be a particle - later nicknamed the Higgs boson.

His paper was initially rejected, partly because few understood it. Other groups published similar theories around the same time, but it was Higgs’s name that stuck. And with that, the scientific community embarked on one of the most ambitious hunts in history: a fifty-year search for proof.

The long road to discovery

Science is often portrayed as a series of “Eureka!” moments. Higgs himself dismissed that myth. His idea was not a flash of inspiration but the product of years of careful work.

For decades, experimental physicists searched. Each generation of particle accelerators edged closer to the energies required, but none quite reached it. That changed with CERN’s Large Hadron Collider - the most ambitious scientific experiment ever built.

In July 2012, the announcement finally came: two separate detectors had found evidence of the Higgs boson. For many, it was champagne and celebration.

For Higgs, it was simply the quiet confirmation of what he had long believed: “It’s very nice to be right sometimes.”

A year later, he was awarded the Nobel Prize in Physics alongside François Englert. True to form, he wasn’t there to take the call. He didn’t own a mobile phone. A neighbour stopped him in the street to break the news.

Humility in an age of noise

The boson was nicknamed the “God particle” by the media, a label that made for headlines but obscured the science. Higgs himself never sought fame. Friends described him as shy, retiring, even too modest for his own career.

And yet, perhaps that humility was his strength. In a world that often confuses visibility with value, Higgs proved that real breakthroughs don’t come from constant noise. They come from rigour, persistence, and conviction.

That resonates with me not just as a scientist, but as a CEO. In business and in science, the temptation is always to chase attention.

But Higgs’s legacy is a reminder: contribution matters more than noise.

A lasting legacy

Peter Higgs passed away having seen his life’s work vindicated, celebrated, and written into the foundations of physics. He leaves behind not just the boson that bears his name, but an example of how deep, transformative work is done: with patience, humility, and dedication to truth.

For me, as someone who has lived both the life of a scientist and the life of a business leader, his story is more than physics. It is a reminder of why we persist with questions, why we value integrity over hype, and why the most lasting contributions are often the quietest.

As we remember him, perhaps the most fitting tribute is not the nickname “God particle,” but his own words:

“It’s very nice to be right sometimes.”

✨ Reflection for readers: In your own life and work: whether in science, business, or elsewhere, do you value persistence and substance over visibility and noise? What might change if more of us followed Higgs’s example?

Subscribe now