AI that won’t fail you: Insights on securing critical systems

Welcome back to Cohere’s monthly newsletter on Enterprise AI.

Hubspot Email Header-Mar-09-2023-02-44-46-8477-AM

Greetings vv,

This month, we announced agreements with both the U.K. and Canadian governments to leverage AI to enhance public services and national sovereignty. We also partnered with U.S. government software provider Second Front to deliver AI solutions to public services.

These partnerships aren’t just about bringing AI to critical infrastructure; they’re centered on secure AI that’s built from the ground up with security, governance, and reliability at its core.

This reflects our belief that AI security needs to be tightly woven into the solutions that organizations adopt and the way they adopt them, rather than being treated as an add-on.

The state of AI security

As organizations race to adopt AI, they’re opening up novel vulnerabilities and an expanded attack surface — particularly as more agentic AI systems are deployed.

In one recent example, a cyber security company said in June it had discovered a critical vulnerability in Microsoft’s 365 Copilot AI assistant. The flaw, dubbed “EchoLeak,” enabled attackers to obtain sensitive internal data without any action by the user — a so-called “zero-click” attack.

AI agents are a growing source of anxiety for CTOs and CISOs. To do their work, agents need access to a broad range of internal data and tools. If they have access to private and sensitive data, exposure to untrusted content (such as that sent by an attacker), and the ability to communicate externally, it’s a recipe for potential breaches. Even with humans in the loop, their wide reach and autonomy could create serious concerns for security.

“While the enormous potential of agentic AI is exciting, it’s important to highlight an uncompromising caveat: we can’t move forward unless these systems are built on a foundation of trust.”

– Aaron Shilts, CEO, NetSPI

To earn trust with AI, organizations must tackle security challenges, from data governance and model behavior to controlling how users and agents access systems and tools.

If control is the answer, what are the questions?

Here are three critical questions CTOs and CISOs must ask to stay in control of AI.

1. How will you protect data privacy?

AI systems process vast amounts of sensitive data, from customer information to internal financial figures. Without strong privacy and governance policies, that data can be exposed by attackers, or even by well-intentioned employees.

Some 38% of employees report sharing sensitive company data with publicly available AI tools — a risky practice known as “shadow AI.” In addition, prompts can inadvertently leak sensitive information, and models can be deliberately coached into revealing private and proprietary data. AI's ability to infer personal information from seemingly anonymous data sets creates new vulnerabilities.

“Our research shows that employees using AI applications without proper guardrails are unwittingly exposing sensitive company data at an alarming rate.”

– Ben van Enckevort, Co-Founder and CTO, Metomic

To mitigate data-leak risks with confidence, it’s crucial to embed privacy in AI solutions throughout their life cycle. That means integrating privacy-by-design principles at every stage, from model training and inference to deployment and decommissioning.

Running models privately, and particularly on companies’ own premises, is an effective way to insulate data, and it's growing in popularity: 21% of workloads and data are being moved back to private environments. This trend highlights a shift towards the security and control offered by private infrastructure.

2. How will you prevent misuse and adversarial attacks?

AI systems can be manipulated in ways that are hard to detect using traditional cybersecurity methods.

The No. 1 LLM security threat for 2025 is considered to be prompt injection attacks. A big challenge to AI security, harmful commands can be embedded in seemingly innocent inputs, or indirectly in external resources that are pulled into the model.

Security researchers have demonstrated how prompt injections could be used to compromise data (as with EchoLeak), steal or manipulate code, or take control of user accounts.

“The fundamental risk … is that whoever provides input to the LLM has a high degree of influence over the output.”

– Rich Harang, AI Systems Security Architect, Nvidia

At Cohere, we approach these threats by combining red-teaming, vulnerability scanning, and adversarial testing to stress-test our models against a range of scenarios. For more, check out the Cohere Secure AI Model Framework or download our guide on how to securely deploy LLMs in private environments.

3. How can you ensure the integrity of AI supply chains?

AI models aren’t built in isolation. They rely on complex supply chains, including external data sources, open-source libraries, and third-party APIs, any of which can introduce security risks.

Bad actors can insert malicious elements into a model’s training data through ”backdoors,” or tamper with the weights that models use to provide responses. Even widely trusted repositories have been shown to contain backdoors that could be activated post-deployment to compromise an LLM and steal sensitive data.

“AI supply chains are a growing target, with attackers manipulating data, training models, and software libraries.”

– Michael Adjei, Director of Systems Engineering, Illumio

At Cohere, we implement various security measures to protect the supply chain, including verifying the integrity of external components, ensuring secure data transmission, and maintaining a trusted network of partners and suppliers.

For AI to be truly enterprise-ready, it must be secure. We aim to help our customers move fast and innovate with confidence.

For more, catch up on our latest articles, or read on for this month’s highlights and upcoming events.

Read more

Product

Robust security and governance measures are essential for accelerating the adoption of AI and ensuring highly sensitive data remains protected. We are proud to announce Cohere has achieved ISO 42001 and ISO 27001 certifications.

For Business

This month, we are thrilled to announce our partnership with Ensemble Health Partners to bring secure agentic AI to some of healthcare’s most complex workflows using a customized version of North.

Developers

Did you know we have several AI agent cookbooks available for you to explore? Check them out and start building today! For more, download our latest guide: Building enterprise AI agents.

Research

This July, join the Cohere Labs Open Science Community for ML Summer School. You'll be part of a global community exploring the future of ML and hear from speakers across the industry. Don’t miss out! Register today.

Company

Cohere President and COO Martin Kon joined global leaders and partners at Viva Technology in Paris to discuss practical AI deployment strategies and the future of AI's impact on global businesses. And don’t miss our CEO Aidan Gomez on The MAD Podcast / Data Driven NYC discussing how we're innovating to build secure, private, and customizable enterprise AI, without chasing the AGI hype.

Upcoming events with Cohere

July 3 [Online]: Join Cohere Labs and Antonio (Tony) Silveti-Falls, Associate Professor of AI at CentraleSupélec, discussing how to train neural networks at any scale.
July 8 [Paris, France]: Join us for RAISE Summit and visit our booth to experience the latest demos. Plus, be sure to check out the fireside chat on July 9th with Cohere CTO Saurabh Baji.
July 17 [Online] Register for our latest webinar with leaders from Microsoft and DraftWise highlighting benefits of AI for the legal industry.
July 27 [Vienna, Austria]: Join us at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), where we will share our latest insights in NLP technologies.

For all upcoming events, explore cohere.com/events or watch our past events on-demand.

Spotlight

AI in financial services: An executive roadmap. Download our latest guide and learn how enterprises are using AI to help with common tasks like claim processing, fraud detection, and customer service inquiries.

Enterprise AI security challenges and solutions. Watch the latest webinar featuring Cohere experts Diane Chang, AI Safety Product Manager, and Prutha Parikh, Head of Security, who discuss the challenges of ensuring the security and safety of AI deployments.

Five ways to modernize manufacturing with AI. From ensuring compliance to working with legacy systems, manufacturers can overcome AI adoption challenges in five ways.

Request a demo