Collecting and using log data

Logging refers to the storage and utilisation of log data. Log data is used to determine what, why and when something happened. Log data is collected by operators and devices connected to the internet. This guideline is intended for organisations that have in-house information security expertise.

A log is a chronological recording of events, or transactions, and their causes. Transactions and changes in information systems, applications, data networks and data content are recorded in the log, that is, logged.

Log data is generated all the time, everywhere. Your computer maintains an access log, wireless network base stations and wired routers store transaction logs, your phone operator keeps a communication log, your daily software have access control logs, error logs and so on.

An adequate log contains

1: Timestamp

The system logs the time of the transaction.

2: Transaction and user

The system logs what was done or attempted and the user.

3: Access right

The system logs which authorisation or access rights enabled the transaction.

4: Transaction source

The system logs where the transaction was performed and where the change data originated.

5: Transaction status

The system logs whether the user succeeded in the attempted action. If the action failed, it records the cause of the failure.

ENSURE DATA PROTECTION

If a log contains personal data, its processing must comply with the EU General Data Protection Regulation (GDPR). According to the GDPR data minimisation principle, personal data must be limited to what is necessary for the purposes of processing the data. A privacy statement must be drawn up concerning the personal data in the system.

Excessively detailed monitoring of transactions combined to the identities of individuals may violate the individuals’ data protection.

As a rule, the following data should not be stored in a log:

  • Personal identity codes
  • Special categories of personal data within the meaning of the GDPR (‘sensitive data’)
  • Credit card numbers
  • Passwords (not even in encrypted form)
  • Intersystem access keys and secrets
  • Authorization data
  • Interpersonal communication content.

 

Log customisation

1: How much is appropriate?

The easiest way to determine the appropriate amount of log data is to try it out. Start carefully and increase the detail of data collection as required. If excessive logging is enabled at the deployment stage, the system will quickly be overwhelmed, and filtering useful data among the vast mass becomes difficult. With regard to personal data, the GDPR minimisation principle requires that only such data whose processing is necessary is processed.

2: Log retention period

Depending on the protected data, the sufficient retention period varies between 6 and 24 months. The GDPR does not specify precise retention periods for personal data. The controller must assess the retention period and necessity of personal data for the purpose of use in question. Personal data may only be retained for as long as it is necessary for its purpose of use. The controller must assess and be able to justify the retention periods.

3: Archiving and erasure

By archiving log data, access is ensured to older observations that cannot be kept in an active log for reasons pertaining to the use of storage space, for example.

In principle, data must be erased or anonymised when their processing is no longer required. Log data usually has a life cycle at the end of which it is no longer of use, and it is not necessary to retain the data. A procedure should be in place for erasing log data.

4: Responsibilities related to log processing

The main responsibility for the processing of logs falls with appointed administrators. To ensure traceability, individual administrators should not be authorised to edit the centralised log management system.

The log description specifies for which purposes log data is collected and for which purposes it may be used. The collection of each piece of log data must be justified. The security and appropriateness of logs must be audited on a regular basis. Administrators monitor the necessary logs, data security officers monitor logs related to data security, and ultimately, the organisation’s senior management is responsible for the log data.

5: Passwords recorded at incorrect locations remain visible

A login log typically records both successful and failed login attempts. Login identification data includes, for example, the user ID, the time of the login attempt and the address used for the login attempt. Failed login attempts also leave an entry which can be viewed in the log. How often have you accidentally entered your password in the username bar and pressed enter? It is now stored in a log in a readable format, and it can be accessed by the administrator. You should always change your password if you accidentally entered it to a visible location.

6: Ensure data protection

If a log contains personal data, its processing must comply with the GDPR. According to the GDPR data minimisation principle, personal data must be limited to what is necessary for the purposes of processing the data. A privacy statement must be drawn up concerning the personal data in the system.

Excessively detailed monitoring of transactions combined to the identities of individuals may violate the individuals’ data protection.

7: Access control

The processing of logs requires thoroughly planned access control. Logs and their login services can best be protected against unauthorised use by using a log environment whose integrity has been secured and which is separated from the rest of the production environment.

In particular, the separation of the system from the rest of the environment should be ensured so that the integrity of log transactions cannot be affected by a security breach. In addition, log data should be backed up, entries should be recorded on the processing of log data, and alerts should be set up for unauthorised editing attempts.

FURTHER READING ON LOGGING

Logging made easy (External link)’ is an open source project maintained by the UK National Cyber Security Centre, and it is publicly distributed in the GitHub service. The project presents a practical way to create basic monitoring and logging in a Windows environment. The project is not perfect, and it is not intended for professional use, but it enables basic functionality. The situation is not ideal, but basic-level monitoring and logging is better than no monitoring or logging at all.