Collecting and using log data

Logs can be classified based on their form as well as purposes and methods of use.

A maintenance log is used to maintain data on changes to the system’s operation and access rights as well as to manage error situations. This type of log is necessary for version management and for monitoring the overall architecture of the operating environment.

An access log, or an event log, is probably the most common and indispensable log type. It registers user logins and logouts as well as data on other normal processes performed by the system. The system’s modules leave a trace in the access log when calling other modules. Entries on printing and reading data content in a database are recorded in the access log.

Entries on any additions and changes to and deletion of data are recorded in a change log. When the source of the changed data appears in the log entries, its validity can be traced and verified, if necessary.

An error log is particularly necessary when solving problem situations. When the cause of an error is logged as accurately as possible, it is easier to fix the cause.

There are also other commonly used log formats. A communication log can contain data about conveyed communications: message source, endpoint and other data, such as time, quantity, unique identifier and status. Identification data is an example of communication log data. The most common e-mail servers are set up to maintain a communication log.

A holder log tells you who was the holder of an internet address, a telephone number, a domain, or a rental car at a specific point of time. Holder data can be connected directly to a person, an organisation or a system.

Log processing must have a legal basis.

The legislation sets requirements for the content of logs, the retention period, the methods of ensuring the integrity of data and the purpose of the log.

Logs are needed, for example, to ensure the functionality of information systems, for compiling statistics on system use and to ensure data security. The collection and processing of the log must have a legal basis.

As there are various types of logs, their processing methods and rights depend on what type of data the log contains and for what purpose the log was originally collected.

Log requirements based on the legislation relate to the content, retention period, the methods of ensuring the integrity of data and the purposes for which the log data is used. In addition, logs must be processed in accordance with the organisation’s internal guidelines, if any.

Personal data makes the log a personal data file

If the log data contains personal data, the log is considered a personal data file. Personal data refers to any information relating to an identified or identifiable natural person (data subject); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person. In this context, particular attention must be paid to the obligations imposed by the GDPR and, for example, a statement must be drawn up on the processing of personal data.

When connected to log data, a name or an e-mail address, for example, are considered personal data. In some cases, an IP address may also be considered personal data.

If processing a log that contains electronic communication transfer data, the log data must be processed in compliance with Chapter 17 of the Act on Electronic Communications Services (917/2014). Transfer data includes, for example, data about the sender and recipient of an e-mail message, network addresses, the duration of the connection, routing, point of time, and the amount of data transferred.

If the log or the technical system that produces it is intended to be used for the supervision of personnel in order to, for example, protect business secrets or investigate cases of misuse, the so-called Lex Nokia sections of Chapter 18 of the Act on Electronic Communications Services shall apply to the use of the log. In that case, users must be informed of the procedure. In addition, the employer must organise a co-operation procedure.

Log management must be restricted so that all logs or the data contained in a single log are not accessible to everyone. Thus, processing authorisations should be limited on the basis of, for example, an employee’s information requirements based on their duties. This necessity requirement, or the so-called “need to know basis”, is recorded both in the principles of the GDPR and in the Act on Electronic Communications Services.

Logger’s check list

The correct construction of a log and the processing of log data require thorough planning, a clear process and unambiguous instructions. This is a short check list for loggers to facilitate successful log projects:

• Consider the purpose of logging.
• Is the data stored in the log necessary for the purpose of use?
• Remember the legal obligations for different types of logs.
• Process log data in compliance with predefined systems and procedures.
• When the data is no longer needed, erase it.
• Set access rights based on information requirements.
• The data protection and legal protection of data subjects and the administration must be ensured.
• Provide users with sufficient information, especially if the log is used for technical monitoring. Remember the co-operation procedure.

Log processing requirements are based on, for example, the following legislation

 General Data Protection Regulation, GDPR

What is SIEM (Security Information and Event Management)?

In most cases SIEM is referred to as a data security product that combines information produced by other systems. However, this concept does not add as much value to the introduction of SIEM as SIEM can actually offer. Log management and SIEM should primarily be considered as a process or a method of managing data security that enables data collection from several systems that are not directly dependent on each other. The collected data is either stored or processed in a centralised location. This creates the opportunity to detect larger events consisting of several smaller transactions. The greatest strength of SIEM lies precisely in the opportunity to display and correlate data at a centralised hub.

The greatest misunderstanding concerning SIEM is the assumption that it is a finished product that provides added value through its mere existence. However, this rarely results in any remarkable added value or development of the situational picture.

Deployment of centralised log management

The most important thing is to first define a policy for managing logs. What is logged, how it is logged, and in particular, what the collected log is used for? One of the key issues in log management is the enormous number of transactions. It can quickly lead to a situation where system administrators can no longer efficiently find the required data among the logged mass. When introducing centralised log management, it should be specified precisely why the procedure is performed.

For example, the reason for deploying a centralised log management system can relate to requirements set by a regulation. In that case, it is not expedient to build an excessively heavy system if the organisation as a whole is not prepared to start using the SIEM process. In addition, it is advisable to exercise caution if the marketing material states that the organisation would meet certain requirements immediately after the SIEM product is deployed. In practice, this is always just a marketing phrase and should not be trusted as such. When implementing centralised log management, especially SIEM, it should be remembered that Rome was not built in a day. In practice, this means that the number of monitored log sources should be increased gradually and that the organisation’s operating methods should be developed simultaneously as enabled by the available resources. It is often advisable to set up a single log server and try command line-based queries before considering the procurement of a SIEM system.

Log management in practice

Centralised log management and SIEM do not revolutionise the security of organisations as such. Even if the organisation invests resources (procurement, installation, and connection to source systems) in the SIEM system, it will not generate alarms or observations on its own. Commercial SIEM products, in particular, do contain pre-defined identifying data that enable the detection of harmful events. However, each organisation has a unique IT environment, and generic built-in identifiers are not sufficient to provide significant added value in addition to antivirus software. A specific sequence of events may look like an attack in one environment while in another one, it may be part of normal operation.