Comparing Investigations¶

Comparison answers two different questions, and it is worth being clear about which one you are asking:

"Did anything change between these two runs?" — diff two investigations.
"Is this run still acceptable?" — check one investigation against tolerance rules.

Both go through compare_investigations, and both read the report — nothing is re-derived.

Diffing two investigations¶

from cyvest import Cyvest, compare_investigations

expected = Cyvest(investigation_name="baseline")
url = expected.observable_create("url", "https://evil.test")
expected.finding_create("phishing-page", weight=4.0)

actual = Cyvest(investigation_name="candidate")
actual.observable_create("url", "https://evil.test")
actual.finding_create("phishing-page", weight=7.0)

diffs = compare_investigations(actual, expected)
for diff in diffs:
    print(diff.status.value, diff.rule_id, diff.expected_score, "→", diff.actual_score)

✗ phishing-page 4.0 → 7.0

Because a finding's identity is its rule_id and is stable across investigations, a changed score shows up as one mismatch rather than an addition plus a removal.

Status	Meaning
`+` (`ADDED`)	present in actual, absent from expected
`-` (`REMOVED`)	present in expected, absent from actual
`✗` (`MISMATCH`)	present in both, with a different score or verdict

A mismatch also carries the observables that explain it, so you can see that a finding moved because one of its observables did:

for observable_diff in diffs[0].observable_diffs:
    print(observable_diff.value, observable_diff.expected_score, "→", observable_diff.actual_score)

Tolerance rules¶

Pinning an exact score makes a test that breaks every time you tune a weight. An ExpectedResult expresses a band instead:

from cyvest import ExpectedResult, Verdict, compare_investigations

diffs = compare_investigations(
    actual,
    result_expected=[
        ExpectedResult(rule_id="phishing-page", score=">= 1.0", verdict=Verdict.SUSPICIOUS),
        ExpectedResult(key="fnd:domain-reputation", score="< 2.0"),
    ],
)

A rule targets a finding either by rule_id or by full key; one of the two is required.

Supported operators¶

>=, <=, >, <, ==, != — for example ">= 0.01", "< 3", "== 1.0".

`ExpectedResult` fields¶

Field	Meaning
`rule_id`	match the finding by rule
`key`	match the finding by exact key
`verdict`	the conclusion expected
`score`	a band, as a rule string
`effect`	how the finding enters the total
`ignore`	statuses to tolerate, e.g. `{DiffStatus.REMOVED}`

ignore is how you say "this finding may or may not fire, and that is fine":

ExpectedResult(rule_id="optional-enrichment", score="> 0", ignore={DiffStatus.REMOVED})

Pinning the effect¶

A conclusion bounds the total instead of adding a term to it, so it carries no score. A rule that states only a verdict therefore vouches for it silently becoming a term of the sum — the one change that turns a rule capping the case into a rule inflating it. Say so explicitly:

from cyvest import Effect

ExpectedResult(rule_id="analyst-call", verdict=Verdict.MALICIOUS, effect=Effect.FLOOR)

conclusion() derives the direction from the verdict, so on a finding it created the two travel together. finding_create does not: it takes effect and verdict independently, and a SUSPICIOUS conclusion may be either a floor or a ceiling. Pinning effect is the only way to state which — and the only way to tell a conclusion from an ADDITIVE finding.

Comparing across engines is refused¶

Two engines do not produce scores on the same scale, so a diff between them would be arithmetic without meaning:

compare_investigations(actual, expected)
# EngineMismatchError: Cannot compare a basic-v1 report with a bayesian-v1 one;
# scores from different engines are not on the same scale.

Either re-evaluate both with the same engine, or take responsibility explicitly:

actual.reevaluate(engine="basic-v1")
expected.reevaluate(engine="basic-v1")

# ... or, if you know what you are doing:
compare_investigations(actual, expected, allow_engine_mismatch=True)

Displaying a diff¶

actual.display_diff(expected, title="Nightly regression")

from cyvest.io.render import display_diff

display_diff(diffs, title="Nightly regression")
display_diff(diffs, lambda renderable: logger.rich("INFO", renderable, width=150))

An empty diff still renders a table, saying so — silence is ambiguous.

Each row is a tree: the finding, the observables it links, and the signals that asserted them, with the expected and actual band side by side at every level. Branches are listed whether they moved or not — a score tells you that something changed, the tree tells you what changed it.

│ domain-reputation         │   NOTABLE 0.50    │    NOTABLE 1.00    │   ✗    │
│ └── example.com           │     INFO 0.00     │    NOTABLE 0.50    │        │
│     ├── MISP Warning List │     INFO 0.00     │     INFO 0.00      │        │
│     └── VirusTotal        │     INFO 0.00     │    NOTABLE 0.50    │        │

Pass a printer when you also log

Without one, the table goes to a rich.Console on stdout while your logger writes to stderr. The two streams are not synchronised, so tables surface before their own headers. Every display_* method takes a printer for exactly this reason:

def to_logger(renderable: object) -> None:
    logger.rich("INFO", renderable, width=150)

actual.display_diff(expected, printer=to_logger)

From the shell:

cyvest diff actual.json expected.json
cyvest diff actual.json expected.json --rules tolerances.json --engine basic-v1

[
  {"rule_id": "domain-reputation", "score": ">= 1.0"},
  {"key": "fnd:ai-analysis", "verdict": "SUSPICIOUS", "score": "< 3.0"}
]

Use cases¶

Regression testing a rule change¶

baseline = Cyvest.io_load_json("baseline.json")
candidate = run_pipeline_with_new_rules(sample)

diffs = compare_investigations(candidate, baseline)
assert not diffs, f"{len(diffs)} unexpected differences"

Validating a sample corpus¶

tolerances = [
    ExpectedResult(rule_id="phishing-page", score=">= 3.0"),
    ExpectedResult(rule_id="benign-domain", verdict=Verdict.SAFE),
]

for sample in corpus:
    diffs = compare_investigations(analyze(sample), result_expected=tolerances)
    assert not diffs, f"{sample.name}: {diffs}"

This is the form that survives policy tuning: it asserts conclusions, not magnitudes.