The (Technology ...and AI) Competitive Trap
Why Voluntary Safety Rarely Survives Technology Races
Over the past few years, the conversation around artificial intelligence has often centred on safety:
Safety research.
Safety pledges.
Safety frameworks.
For a time, it appeared that the frontier AI companies might converge around voluntary restraint.
The idea was simple: if models became too capable relative to available safeguards, development would pause.
That premise assumed something fragile.
Alignment of incentives.
And incentives rarely remain aligned for long.
In high-velocity technological races, safety does not remain purely an ethical consideration.
It becomes a ‘strategic variable’.
The Original Premise
Many AI developers began with a relatively straightforward principle:
If model capabilities outpaced the ability to control them safely, training would pause.
This was framed as responsible scaling.
In theory, the approach had logic: slow the system before it outruns governance.
But voluntary brakes only work if everyone applies them simultaneously.
Once one actor accelerates, restraint becomes asymmetrical.
And asymmetry becomes competitive disadvantage.
The Competitive Reality
AI development has become one of the most competitive industrial races of the modern era:
Venture capital.
Private credit.
Hyperscale compute.
Government contracts.
Enterprise deployment.
The incentive gradient is steep.
Companies that pause risk losing technological leadership.
Companies that accelerate capture market share and infrastructure control.
Under those conditions, unilateral constraint becomes fragile.
Safety frameworks evolve.
What begins as a hard stop often becomes a softer mechanism:
Not a pause.
A goal.
Not a rule.
A guideline.
That evolution is rarely driven by ideology.
It is driven by incentives.
A Familiar Pattern
If this dynamic feels familiar, it should.
Cybercrime has been operating on similar incentive structures for years.
Attackers study constraints.
They analyse insurance policies discovered during breaches.
They calibrate ransom demands below coverage limits.
They target organisations where recovery economics favour payment.
This is optimisation.
The attacker’s objective function is simple: maximise payout while minimising resistance.
AI development operates within a different domain but under the same economic logic.
Optimisation against constraints.
Incentives Over Intent
This is not a story about good actors and bad actors.
It is a story about systems responding to incentives:
When incentives reward speed, systems accelerate.
When incentives reward scale, systems expand.
When incentives reward safety, systems slow down.
The difficulty is that incentives rarely align perfectly:
Markets reward capability.
Governments reward strategic advantage.
Investors reward growth.
Safety frameworks must operate inside that reality.
Not outside it.
The Structural Problem
Voluntary restraint works best in stable systems.
Frontier technologies are not stable systems.
They are competitive races with global participants.
In those conditions, safety becomes a coordination problem.
Each actor faces a dilemma:
If everyone slows, safety improves.
If one accelerates, advantage shifts.
This is the competitive trap.
No individual participant wants to be the one who brakes first.
The Parallel with Cyber Risk
Cyber attackers understand incentive systems well:
They analyse organisational structures.
They model insurance thresholds.
They monitor incident response patterns.
In many cases, ransom demands are calibrated deliberately.
High enough to be profitable.
Low enough to be payable.
The objective is not destruction.
It is optimisation.
AI development is governed by the same economic gravity.
Different actors.
Different objectives.
Same logic.
The Board Question
For boards, the issue is not whether frontier technologies will continue accelerating.
They will.
The question is how governance frameworks respond when safety becomes strategically negotiable and technology velocity is all but assured.
Boards should now be asking:
Are we relying on voluntary safety commitments from vendors whose incentives may change?
Do we understand the competitive pressures shaping AI deployment decisions?
Are our internal AI adoption decisions influenced more by capability race than risk architecture?
Do our risk models assume adversaries who optimise economically, not just technically?
Are we prepared for safety frameworks to evolve under competitive pressure?
The governance challenge is subtle.
It is not about mistrust.
It is about recognising incentive structures.
What This Means for Strategy
Strategy in frontier technologies cannot rely on static assumptions:
Safety frameworks will evolve.
Competitive pressures will intensify.
Capability races rarely slow down.
Organisations therefore need to design resilience under conditions where:
Technological acceleration is structural.
Incentives will shift.
Safety commitments may adapt.
That means building architectures that assume volatility rather than stability:
Vendor risk models must include incentive drift.
Cyber defence must assume economically rational adversaries.
Governance must anticipate that competitive dynamics reshape risk boundaries.
In short:
Do not assume the rules will hold.
Design systems that remain resilient when they move.
The Deeper Strategic Question
There is a temptation to frame the safety debate as ethical.
That is incomplete.
The real issue is economic.
Safety mechanisms must operate within incentive structures that reward speed, scale and advantage.
If those incentives remain misaligned, voluntary restraint will struggle to persist.
History offers many examples:
Financial markets.
Arms races.
Industrial competition.
Frontier technologies follow the same pattern.
The Weekend Takeaway
In high-speed technological races, safety cannot rely solely on intention.
It must be engineered into systems that remain stable even when incentives shift.
Because when competition intensifies, voluntary brakes rarely hold.
The real challenge is not preventing acceleration.
It is ensuring that resilience scales alongside it.
📚 Further Reading
Governance of AI Program (Centre for the Governance of AI): Uncertainty, Information, and Risk in International Technology Races (2023)
Explores how competition between AI developers can create incentives to cut corners on safety, potentially triggering a “race to the bottom” in responsible deployment.
ScienceDirect / Computers & Security: Between a rock and a hard(ening) place: Cyber insurance in the ransomware era (2023)
Examines how cyber-insurance interacts with ransomware incentives and whether insurance coverage can inadvertently influence attacker behaviour and ransom dynamics.
UCL / Royal United Services Institute: How Cyber-Insurance Influences the Ransomware Payment Decision (2023)
Academic study analysing whether organisations with cyber-insurance are more likely to pay ransomware and how insurance alters attacker-victim incentives.
Oxford University Press - Industrial and Corporate Change: Competition Between AI Foundation Models (2024)
Analyses the emerging competitive dynamics between large AI model developers and the regulatory implications of rapid capability races.
ScienceDirect: Voluntary safety commitments provide an escape from over-regulation in AI development (2022)
Game-theoretical analysis of whether voluntary commitments by AI developers can meaningfully improve safety outcomes without enforceable coordination.
SSRN: Racing to Safety - Tax Policy for AI Safety-by-Design (2025)
Proposes economic and policy incentives to align competitive AI development with stronger safety outcomes through taxation and market mechanisms.
arXiv: Vigilant Incentives Help Regulatory Markets Improve AI Safety (2023)
Uses evolutionary game theory to examine how regulatory incentive structures can influence AI developers’ safety behaviour in competitive environments.
SSRN: Insurance Against Ransomware (2022)
Economic modelling of how cyber-insurance availability can unintentionally increase ransomware frequency and severity by altering attacker incentives.
ScienceDirect: Strategic Insights from Simulation Gaming of AI Race Dynamics (2025)
Simulation-based research showing how competitive dynamics between states and firms can increase the probability of AI safety failures.
RUSI - Royal United Services Institute: Insurance as Crime Governance: Comparing Kidnap for Ransom and Ransomware (2023)
Explores how insurance markets shape incentives in extortion-based crime and how attackers adapt to the financial structures of victims.
🧠 Techie Corner for the Non-Techies
Strip away the jargon.
When people talk about “AI safety,” they often mean two technical things.
First: capability control
AI models can perform increasingly complex tasks. Safety mechanisms attempt to ensure those capabilities cannot be misused.
This includes techniques such as:
Reinforcement learning with human feedback (training models to follow certain behaviours).
Guardrails that prevent models from generating harmful outputs.
Monitoring systems that detect misuse.
Second: deployment governance
Even safe models can create risk if deployed incorrectly.
That is why companies implement:
Usage policies.
Access controls.
Monitoring systems that track how models are used.
But here is the technical challenge.
AI models are general-purpose systems.
Unlike traditional software, they can perform many tasks depending on the instructions they receive.
That flexibility is powerful.
It is also difficult to constrain perfectly.
In competitive markets, companies face pressure to release more capable models quickly.
Safety research therefore becomes a balancing act:
Improve safeguards without slowing development too much.
This tension between capability and control is not unique to AI.
It appears in many frontier technologies.
AI simply compresses the timeline.
Last Week on Something for the Weekend
Last week, we explored how artificial intelligence acts as an aggression multiplier.
AI amplifies four forces simultaneously:
Speed, scale, strategic calibration and democratisation.
Attacks move faster.
Operations scale across thousands of targets.
Extortion becomes economically optimised.
And the expertise barrier for launching sophisticated operations falls dramatically.
When both attackers and defenders use AI, the system does not stabilise.
It becomes more volatile.
Boards therefore need to assume adversaries who are not only technically capable, but economically rational and AI-augmented.
The strategic implication was clear:
Cyber risk is no longer episodic.
It is a continuous campaign.
Read here: 👇
AI as an Aggression Multiplier
Artificial intelligence is no longer a disruptive frontier technology; it is reshaping the fundamental dynamics of conflict, intrusion and defence across cyber and digital domains.