Every incident conversation reaches the same moment. Someone asks how many customers were affected, and the honest answer is that nobody can tell, because the events that would answer it were never recorded.
You cannot add logging retroactively. Whatever exists at the moment of the incident is the entirety of what you will ever know — and the difference between a two-hour investigation and a month of speculation is decided months earlier, cheaply.
The events that answer the questions you will be asked
- Authentication: every success and failure, with actor, source address, user agent and method. Failures matter more than successes.
- Authorisation failures: every 403. A burst of them from one account is the clearest signal of enumeration you will ever get.
- Administrative actions: role changes, permission grants, impersonation, configuration changes, feature-flag flips. Who, what, before and after.
- Data access at scale: exports, bulk reads, report generation, anything returning more than a handful of records.
- Sensitive record access: for health, financial or identity data, log the read itself, not only the write.
- Credential lifecycle: password changes, MFA enrolment and removal, API key creation, session revocations.
- Payment and refund events, with the gateway reference, so your record can be reconciled against theirs.
What each event needs on it
An event without context is a line that proves something happened and nothing else.
- Actor: the authenticated principal, not just an IP address. For impersonation, both — the operator and the account acted upon.
- Target: the specific record or resource, by identifier.
- Action and outcome: what was attempted, and whether it succeeded.
- Source: address, user agent, and the request or trace identifier so an event can be tied to everything else in that request.
- Timestamp in UTC, from a synchronised clock. Correlating across systems with drifting clocks is its own small nightmare.
Where they live matters as much as what they contain
- Ship them off the machine that generates them. An attacker with access to the host will edit or delete local logs, and their absence is itself a finding you would rather notice.
- Make the destination append-only for the shipping identity. Write, never delete.
- Retain at least ninety days hot, longer cold. Most breaches are discovered well after they begin, and a thirty-day window frequently starts after the interesting part.
- Restrict who can read them, and log that access too.
Alert on a handful of things, not everything
A dashboard nobody opens is not detection. Pick the few events that are almost always worth a human looking.
- A spike in authorisation failures from a single account or address.
- Any change to administrative roles or permissions.
- Bulk export above a normal threshold, especially outside working hours.
- MFA removal, or a password change followed immediately by an email change.
- Access from a new region for an account that has never travelled.
The test that tells you if this is real
Pick a customer at random and answer, with evidence: who has accessed this account’s data in the last thirty days, from where, and what did they do? If that takes more than fifteen minutes, the gap is in your logging, and it is much cheaper to close today than during the incident where you first need it.