Monitoring Linux Security with the Linux Sensor Pack

Any IT engineer has war stories on how hackers tried or even succeeded to break into their systems. It is an ongoing battle that happens in the background, not really visible until one day you can find your systems "owned" by a script kiddie or by state-sponsored criminals.

We did a small experiment with our cloud provider and placed an internet-facing Linux machine, without any firewalls, using a plain-vanilla configuration. Ping was disabled though and root password modified.

Our Linux server is under attack

We deployed the AutoMonX Linux sensor pack to watch after the Linux server performance and security. Lo and behold, in a matter of a few hours the machine metrics, as reflected in our Linux sensor pack clearly showed it was under attack, as can be seen in the graph below.

The graph shows constant failed password and authentication failures as recorded by the Linux security log. These messages were automatically collected, parsed and displayed via the Linux sensor pack.

Analyzing the data collected during the passing month, it is obvious someone is really keen on breaking into our poor Linux and this guy took only short breaks which leads to the conclusion that the attack is automated and some hacking tools were deployed.

Why other performance indicators are not good enough?

Looking at other indicators such as CPU Load, Memory and even the sshd daemon specific performance indicators, couldn't reveal that the system is under attack. The attacker is clever enough to try to break-in without making too much noise.

Deploying the Security log sensor

The deployment of the Linux sensor pack is fairly straight-forward and involves downloading it from the AutoMonX web downloads page, extracting it to the Custom EXEXML directory of the PRTG Probe and requesting an evaluation license.

Following the deployment and licensing, use our UI is to discover the Linux server resources and on-board them to PRTG

Discovery results - now all you have to do is choose the Linux log security sensor and add it to PRTG

Configuration and customization of the Security Log sensor

The security log sensor of the AutoMonX Linux Sensor pack is quite flexible and allows configuring enterprise-wide or server-specific log parsing rules that can facilitate identification of security issues. Below is the rules file shipped with the sensor pack that covers the Security and Messages logs sensors.

It is very easy to add more text patterns - edit the default_patterns.ini file and add four colons (:) between each pattern.

Watching Processes, Services and Open Files

Additional metrics to watch are the number of running processes, vital services and open files (per specific user).

Overall monitoring of how many processes and threads are used by the system can be utilized to notify if a certain system opens too many threads or spawns too many processes than usual (for instance Crypto-mining activity).

The number of Open files and processes per specific user such as root can also indicate an abnormal system behavior that needs to be investigated.

Conclusion

Any server or device exposed to the Internet can be attacked in a matter of hours if not minutes. The menace exists even in networks behind firewalls and it is strongly suggested to deploy monitoring tools such as the AutoMonX Linux Sensor pack to complement the active defense means and identify abnormal system behavior.