Skip to content

Understanding Security Validation

Security validation uses controlled tests and evidence to check whether security controls work as expected, not merely whether they are installed or enabled.

What you will learn

By the end, you should be able to explain how a validation question becomes a measurable test, distinguish BAS from related approaches, and identify what evidence is needed before calling a control effective.

What is security validation?

Core concept

Security validation compares an expected defensive result with evidence from an authorised test. It checks whether relevant activity is visible, detected, investigated, and handled as intended across people, processes, and technology.

QuestionTestTelemetryControl resultImprovement

Keep in mind: Successful test execution does not prove that a defence worked. Validation must compare the expected control behaviour with recorded evidence and account for missing or unexpected results.

ControlA safeguard intended to prevent, detect, or respond EmulationControlled reproduction of selected adversary behaviour EvidenceA traceable record used to assess the result

What security validation is not

Security validationMeasure whether selected controls produce the expected result
Vulnerability scanningIdentify known weaknesses and configuration problems
Penetration testingFind and demonstrate exploitable paths within an agreed scope
Red teamingPursue an objective while testing people, processes, and technology
Threat modellingReason about likely threats and weaknesses before testing

Connected, but not interchangeable

These approaches can inform one another. A vulnerability may become a validation target, while a penetration test or red-team exercise may reveal controls that need repeatable testing. Their scope, method, and evidence are still different.

Where BAS and adversary emulation fit

01Security validationThe broad objective: measure whether defences work
02BASAutomated or partly automated repeatable testing
03Adversary emulationReproduce selected threat behaviours in a controlled exercise

Authorisation defines the boundary

Testing can interrupt services, change systems, or trigger automatic response. Approve the targets, actions, timing, owners, stop conditions, evidence handling, and recovery plan before execution.

Why security validation matters

Prove visibility

Confirm that endpoint, network, identity, or cloud activity reaches the tools expected to record it.

Test detections

Verify that useful alerts appear with enough context for an analyst to understand and investigate them.

Check response

Measure whether the expected manual or automated action occurs safely, within its approval and recovery boundaries.

Verify improvement

Repeat the same bounded behaviour after a change to demonstrate whether the defensive outcome improved.

How security validation works

  1. Define the questionSelect one control or defensive assumption and state the result that should be observed.
  2. Set the scopeApprove the systems, identities, network paths, actions, owners, time window, and evidence handling.
  3. Check safetyUnderstand each action, dependency, possible effect, stop condition, cleanup step, and recovery method.
  4. Prepare observationConfirm that relevant security tools and owners are ready to preserve telemetry, alerts, decisions, and response results.
  5. Run the testExecute only the approved behaviour while monitoring the target and remaining within the documented boundary.
  6. Compare evidenceCompare expected visibility, detection, investigation, and response with the actual result.
  7. Improve and retestAssign the gap, change the relevant control or process, then repeat the same test to verify the outcome.

Validate a defensive control

Work through a simulated validation of a PowerShell detection on the authorised endpoint WIN-LAB01. Inspect the evidence and correct the first control gap before deciding whether the defence has been validated.

Try it: follow the complete evidence chain

Follow the six stages in order. The simulated command is harmless and the destination is reserved for documentation, but the same action must still be explicitly scoped in a real environment.

Exercise debrief

The first run proves execution and basic process visibility, but the expected detection is absent because its required PowerShell telemetry is not collected. Enabling the relevant logging and repeating the same test validates the complete scoped chain. This does not prove that every PowerShell technique or endpoint is covered.

A useful validation record includes the question, approved scope, ability or action, expected outcome, actual timestamps, telemetry, alert details, analyst decision, response result, unexpected impact, and cleanup status. It should also name the control owner and retest date. This makes a gap actionable and allows another analyst to distinguish a product limitation from missing data, configuration, timing, or process failure.

What makes security validation useful

Start with a control question

Can we detect and investigate a controlled scripting event on this endpoint?

  1. 1Define successState the telemetry, alert, analyst context, and response expected from the control.
  2. 2Bound the testApprove one safe behaviour, target, time window, owner, and recovery path.
  3. 3Preserve evidenceRecord execution, visibility, detection, investigation, and response results.
  4. 4Retest improvementCorrect the first useful gap and repeat the same controlled behaviour.
Plan for operational friction
Unsafe abilitiesUnclear scopeMissing telemetryClock differencesAutomatic responseNo retest owner

Automation improves repeatability, but every exercise still needs authorisation, monitoring, evidence review, cleanup, and an owner for remediation.

Security validation platform in this kit

CALDERA is the current practical platform in this section. It supports security validation through automated adversary emulation, but it is not synonymous with the wider security-validation discipline.

Automated adversary emulation

MITRE CALDERA

Choose it when: You want repeatable, ATT&CK-aligned exercises and can review every action before execution.

Model: Agents execute approved abilities as part of an operation against explicitly scoped systems.

Strength: Reproducible emulation, operation planning, evidence collection, and retesting after defensive changes.

Plan isolation, permissions, plugins, agent control, network access, logging, cleanup, and recovery. Default content is not automatically safe for every environment.

Choose one control question first

Begin with a low-impact behaviour on a dedicated endpoint. Validate one telemetry and detection path before adding abilities, targets, or automated response.

Continue learning