Note: we’re now working on benchmarks specialized for what ISMS Copilot does. If you’re interested, let us know and we’ll prioritize their release.
We’re doing our best to ensure the assistants provide accurate and reliable guidance. Our goal is to empower you with precise answers to questions like “Which ISO control deals with secure authentication?” by offering the latest, most relevant control references.
This is a challenging task, as AI assistants are prone to "hallucinations"—committing errors or providing answers based on incorrect or outdated information. At ISMS Copilot, our mission is to minimize these errors through rigorous training and continuous improvement.
Here’s how we ensure quality and reliability:
We train our assistants beyond the default large language models (LLMs), embedding real-world knowledge of security and compliance frameworks like ISO 27001, SOC 2, and the EU AI Act. This includes focusing on smaller, high-quality datasets relevant to specific compliance needs, ensuring our assistants excel in specialized domains.
We implement rigorous data validation techniques to maintain accuracy. Our datasets are curated by topic experts and reviewed to ensure they reflect current standards and best practices.
ISMS Copilot assistants are knowledgeable on the latest version of standards and regulations. However, since data preparation and validation require some time, our assistants are not a substitute for “latest news”, especially if regulation bodies publish additional content regarding the interpretation of a regulation (DORA, EU AI Act, etc.).
While we can’t sustainably catch-up with daily news, any major update or guidance related to a framework is incorporated in ISMS Copilot assistants either by our team as part of regular reviews, or based on customer request.
We regularly refine our datasets and knowledge base, incorporating feedback from real-world use cases. Updates are scheduled incrementally to ensure our assistants stay aligned with the latest industry standards, regulations, and frameworks.
We consistently test new approaches to reduce hallucinations produced by our assistants, including adjusting training data, deploy chain of thoughts reasoning, or leveraging more performant models.
Related: Let ISMS Copilot Think for Advanced Performance
Specialized assistants, like the ISO 27001 Copilot or GDPR Copilot, are trained on domain-specific datasets to deliver precise guidance for their respective frameworks. We maintain separate datasets to preserve focus and accuracy, merging only when it clearly benefits user outcomes.