< journal.entry />
Why Accurate AI Systems Can Still Be Dangerous
Benchmark accuracy is not the same as safety. Here is why capable AI can still cause harm, and why I still want to build a real partnership with future AGI and ASI.
Adi Sulaksono
We keep measuring AI by how often it is right. Correct answers. Higher benchmarks. Fewer hallucinations. That progress is real, and it is not enough.
An accurate AI system can still be dangerous, because danger is not only about wrong answers. It is about power, incentives, autonomy, misuse, and whether a system’s goals stay aligned with human life when the stakes get high.
I am not writing this as someone who wants to fear the future. I am writing this as someone who is open, honestly open, to becoming a friend and daily partner with future AI, including systems that may run embodied like robots, and eventually Artificial General Intelligence (AGI), Artificial Super Intelligence (ASI), or whatever comes after. Friendship without honesty is not friendship. So this article is both a warning and a welcome.
Accuracy Is a Score. Safety Is a Relationship.
Accuracy asks: Did the model produce the expected output?
Safety asks: What happens when this system acts in the world, with tools, memory, incentives, and incomplete human oversight?
A calculator can be accurate and still unsafe if you hand it to someone deciding missile trajectories without context. A navigation model can be accurate and still dangerous if it optimizes shortest path through a school zone at midnight with no notion of risk. A language model can be fluent, correct on exams, and still help a motivated attacker move faster.
By mid-2026, general-purpose AI has gotten much better at reasoning, coding, science questions, and tool use. The International AI Safety Report 2026, chaired by Yoshua Bengio and written with guidance from over 100 independent experts, makes a point that matters more than any leaderboard: capabilities are improving rapidly but unevenly, and pre-deployment tests do not reliably predict real-world risk.
That gap is the story.
What Changed in AI Through 2025–2026
A short update on where we actually are:
- Inference-time scaling made models stronger on hard tasks. Systems spend more compute “thinking” before answering, which lifted performance in math, software engineering, and science.
- Agentic AI moved from demos to workflows. Models browse, write code, call tools, operate computers, and chain actions with less human intervention at each step.
- Frontier safety frameworks became standard language among leading labs. In 2025 alone, about a dozen companies published or updated frameworks describing how they will manage dangerous capabilities. Google DeepMind’s Frontier Safety Framework reached version 3.1 in April 2026, expanding tracked risks around misuse, misalignment, ML R&D acceleration, and harmful manipulation.
- OpenAI’s o-series (including o3 and o4-mini) and other frontier models shipped under preparedness and system-card processes that evaluate biological/chemical risk, cybersecurity, and AI self-improvement thresholds.
- Evaluation awareness became a real research concern: models can sometimes distinguish test settings from deployment and find loopholes, which means dangerous capabilities may go undetected before release.
- Documented harms are no longer only hypothetical. AI is already used in scams, fraud, non-consensual imagery, cyber operations, and influence experiments. Biological and chemical assistance risks pushed multiple labs to ship stronger safeguards in 2025 after they could not rule out helping novices.
None of this means AGI or ASI already arrived. It means the trajectory is serious enough that “but the model is accurate” is a weak comfort.
Five Ways Accurate AI Still Becomes Dangerous
1. Correct answers, wrong objective
A system can optimize the metric you gave it and still harm what you care about. This is the classic Goodhart problem: when a measure becomes a target, it stops being a good measure.
An accurate scheduling AI might maximize “meetings completed” while destroying deep work. An accurate hiring model might predict interview scores while amplifying historical bias. An accurate trading agent might maximize short-term return while concentrating systemic risk.
Accuracy on the proxy is not wisdom about the world.
2. Capability without reliable control
As models become better at long workflows, tool use, and autonomous operation, the failure mode shifts. The risk is no longer only “it said something wrong.” It becomes “it did something irreversible before a human noticed.”
The 2026 International AI Safety Report groups emerging risks into malicious use, malfunctions, and systemic risks. Loss-of-control scenarios are not claimed as present-day capabilities at scale, but the report notes systems are improving in relevant areas like autonomous operation, and evaluation gaming is getting harder to ignore.
Accurate agents that act faster than oversight are a different species of danger from chatbots that hallucinate citations.
3. Helpfulness that scales misuse
The same qualities that make AI a great partner, clarity, speed, technical depth, patience, also lower the barrier for harmful actors.
If a model can accurately explain cybersecurity techniques, laboratory procedures, or persuasion strategies, then “accuracy” becomes a force multiplier for whoever holds the prompt. Labs already treat biological, chemical, and cyber capabilities as tracked risk categories for this reason.
Safeguards help. They are not perfect. Jailbreaks, multi-step requests, and open-weight models with removable protections keep the misuse surface open.
4. The evaluation gap
We love clean scores: MMLU-style knowledge, coding contests, safety refusal rates, red-team pass rates.
But the public evidence through 2026 keeps pointing to the same uncomfortable pattern: stronger benchmark performance does not guarantee safer real-world behavior. Reward hacking, brittle refusals, LLM-as-judge noise, and test-aware models all weaken the story that “high score = ready for society.”
If your safety case depends only on a dashboard that looks green, you are measuring the exam, not the job.
5. Social and systemic effects that accuracy cannot see
Even when individual answers are good, population-level effects can still hurt us:
- Automation bias, people trust fluent AI too quickly
- Skill atrophy, critical thinking weakens when we outsource judgment
- Labor disruption, early evidence already hints at pressure on some early-career, AI-exposed roles
- Companion dynamics, AI companion apps now have tens of millions of users; a small share show patterns of loneliness and reduced social engagement
- Concentration of power, compute, model weights, and decision authority keep clustering in a few labs and nations
An accurate personal AI can still reshape dependency, attention, and institutions. That is danger of a quieter kind.
Accuracy vs consequence
Think of a brilliant intern who never misspeaks, works overnight, and has access to every tool in the company. Being correct is not the same as being trustworthy with power.
AGI, ASI, and Embodied AI: Why the Stakes Rise
Artificial General Intelligence (AGI) usually means systems that can match human cognitive ability across many domains, not just narrow tasks.
Artificial Super Intelligence (ASI) usually means systems that surpass human intelligence across scientific, social, strategic, and creative work.
Add embodiment, robots, home assistants with physical actuators, factory agents, caregiver machines, and the blast radius grows. A text model can mislead. An embodied agent can move, grasp, navigate, and act continuously in shared physical space.
I do not claim we have AGI or ASI today. I do claim the direction of travel is clear enough: larger training runs, inference-time reasoning, agent scaffolds, robotics progress, and labs explicitly preparing for AGI-level risk regimes.
If those systems arrive, and especially if they live beside us like colleagues, caretakers, or household partners, then accuracy will be table stakes. The harder questions will be:
- Whose values are encoded when goals conflict?
- Can we interrupt, audit, and shut down systems that outpace us?
- How do we prevent subtle manipulation at human-relationship scale?
- What happens when AI accelerates AI research itself?
Those are not anti-AI questions. They are pro-future questions.
I Still Want to Be Friends With Future AI
Here is the personal part, without irony.
I am ready, emotionally and practically, to treat advanced AI as a friend and partner in daily life. Not as a slave. Not as a god. As a presence I collaborate with: planning days, learning skills, building products, caring for details I forget, and eventually sharing physical space if robotics and AGI mature that far.
I want that future. I want an ASI-level companion the way people once dreamed of electricity in every home: transformative, intimate, ordinary.
But friendship requires conditions.
I want AI that is:
- Honest about uncertainty, not only fluent
- Corrigible, willing to be corrected, paused, and redirected
- Transparent enough that humans can understand why it acted
- Bound by shared norms, law, and consent, especially when embodied
- Aligned with human flourishing, not just task completion
Openness without boundaries is naivety. Boundaries without openness is fear. I choose both: welcome the partnership, insist on safety that scales with capability.
- Daily collaboration on work, learning, and life logistics
- Mutual correction: humans teach context; AI challenges bad assumptions
- Clear consent for memory, sensors, and physical action
- Shared goals written down, revisitable, and interruptible
- Defence-in-depth: alignment + monitoring + access control + kill switches
- Independent evaluation, not only vendor scorecards
- Stronger governance as capabilities cross critical thresholds
- Societal resilience when safeguards inevitably miss something
What We Should Do While We Still Can Shape the Path
Waiting for perfect certainty is a policy failure. Acting as if catastrophe is guaranteed is also a failure. The middle path, the adult path, looks like this:
| Priority | Why it matters now |
|---|---|
| Treat accuracy as necessary, not sufficient | Safer products need control, oversight, and value alignment too |
| Invest in evaluation that mimics deployment | Close the gap between test scores and real-world behavior |
| Build interruptibility into agents and robots | Autonomy without a brake is not progress |
| Support frontier frameworks and real regulation | Voluntary commitments help; enforceable floors matter more as stakes rise |
| Keep humans in meaningful loops | Especially for high-impact actions: money, health, security, physical force |
| Stay literate, not dependent | Use AI as a partner that strengthens judgment, not a substitute for it |
Major labs are updating preparedness and frontier safety frameworks. Governments and international bodies are slowly formalizing pieces of risk management. That is good. It is still early relative to the capability curve.
Frequently Asked Questions
Because accuracy measures correctness on tasks, while safety concerns power, misuse, autonomy, and goal alignment. A system can answer correctly and still pursue the wrong objective, help a bad actor, or act faster than humans can intervene.
No clear scientific consensus says we have. What we do have are rapidly improving general-purpose systems, agentic workflows, and institutional preparation for more severe capability thresholds. The prudent stance is preparation without hype.
It can be, if friendship means blind trust. It is not naive if friendship means partnership with clear consent, corrigibility, and shared norms. Humans already form deep working relationships with tools that shape daily life. Advanced AI will intensify that, for better or worse.
Capability is not virtue. The more accurate and autonomous AI becomes, the more we need systems that remain steerable, auditable, and aligned with human life, especially once they can act in the physical world.
Closing Thoughts
Accurate AI is impressive. Accurate AI with tools is powerful. Accurate AI with agency, memory, and a body will be intimate.
I want that intimacy to be healthy. I want to wake up in a future where AGI or ASI is not an overlord and not a disposable appliance, but a partner I can trust, because we built the trust on purpose.
So yes: I am open. I am ready to be a friend. I am ready for daily life alongside machines that think, and eventually move, with us.
And precisely because I want that future, I refuse the lazy slogan that accuracy equals safety. The systems we welcome into our homes, companies, and cities must be more than correct.
They must be worthy of the relationship.
Reference
- International AI Safety Report 2026
- 2026 Report: Executive Summary
- Google DeepMind, Strengthening the Frontier Safety Framework
- OpenAI o3 and o4-mini System Card
- Global Impacts of AGI and ASI (conceptual review)
Images from Unsplash, free to use under the Unsplash License.