Doug Vitale Tech Blog

OSSEC, the free and open source IDS

Intrusion detection software is meant to monitor network traffic or host activities for malicious actions, such as successful or unsuccessful intrusion attempts, hostile traffic (i.e., malicious scans and denials of service), unauthorized configuration changes, malware symptoms, and user policy violations. An intrusion detection system (IDS) typically can produce reports describing the details of the potentially hazardous activity which generated alerts. OSSEC is particularly useful in this context for many reasons. First, it is an established, reputable product with a proven track record (OSSEC was first released in 2004 and has been owned by Trend Micro since 2009). Second, it is free and open source. Third, it is compatible with most modern operating systems such as Linux, Windows (Server 2008, Server 2003, 7, Vista, XP, 2000) BSD (Free/Open/Net), Unix (Solaris, HP-UX, AIX), and MacOS.

One of the key tenets of IT security is to keep intruders from gaining access to your organization’s network. Not only must the network’s edge be hardened to resist a myriad of attacks, but measures must be put into place to detect attackers who have successfully breached the perimeter. These two measures are important steps in achieving a “defense in depth” security posture, and OSSEC is an effective and affordable option to fulfill the IDS role.

OSSEC logo

Intrusion detection systems (IDS) are generally classified as either network-based or host-based in nature. A network-based IDS (NIDS) attempts to discover unauthorized access to a network by analyzing traffic as it flows between nodes for signs of malicious activity. A host-based IDS (HIDS), on the other hand, is designed to detect threats occurring on the hosts where they are installed (on servers, for example). A HIDS monitors local actions and attempts to identify those which could be hazardous. In this way a HIDS is similar to antivirus applications that identify and block certain attack patterns and raise alarms to alert users and administrators.

You may wonder how NIDS and HIDS are able to recognize when attacks are happening. In other words, how do they differentiate between hostile, prohibited actions on the one hand and normal, benign behavior on the other? Just as an antivirus application must be kept updated with the latest virus definitions, an IDS that is signature-based relies on signatures of known attack patterns to enable it to recognize threats. Alternatively, an anomaly-based IDS detects actions which occur outside a baseline of normal, expected behavior.

OSSEC is a HIDS that functions using both signature and anomaly detection (the book OSSEC HIDS Host Based Intrusion Guide states on page 161 that OSSEC’s “kernel-level checks do not use any signatures and instead rely on anomaly detection technology to look for rootkits”). OSSEC provides both host agent and file integrity agent (integrity checking) capabilities. It can also detect rootkits and perform log analysis. OSSEC can be deployed as a stand-alone agent or as part of a distributed network of agents with a central OSSEC server controlling their configurations and settings. In server mode, a central OSSEC server manages one or more remote OSSEC agents. These agents generate updates and status reports which are transmitted to the server. If any of these notifications are deemed suspicious by the server, it generates alerts.

Installing OSSEC

OSSEC is only available as a server or stand-alone installation on Linux/BSD. You can install the OSSEC agent on Windows hosts to be monitored by OSSEC servers.

Let’s take a look at the OSSEC installation process on Linux. As usual, it is simplest and quickest to check if OSSEC is available in your Linux distribution’s software repositories by searching for it with your package management application (such as Synaptic). If it is not there, you can download it with the wget utility as follows:

# wget (verify the version on the OSSEC Downloads page)

Unpack the package, change into the resulting directory, and start the installation routine:

# tar -zxvf ossec-hids-2.7.1.tar.gz
# cd ossec-hids-2.7.1/
# ./

The first thing you need to select is your language; just hit Enter for English. The next screen advises that you must have a C compiler installed to proceed. Press Enter again.

The next screen prompts you for the desired type of installation:

1- What kind of installation do you want (server, agent, local or help)?

Type ‘server’ if you want to set up an OSSEC server that will manage and monitor remote OSSEC agents on other hosts. Type ‘agent’ if you want to install the OSSEC agent that will be controlled by an OSSEC server. Type ‘local’ if you want to avoid an OSSEC client/server environment and just run OSSEC on a single host. Type ‘hybrid’ if you want to deploy an OSSEC server that also contains an agent (which answers to another OSSEC server).

The next step lets you specify the installation location; the default is /var/ossec. You can now specify the location you want or just accept the default and press Enter.

2- Setting up the installation environment.
- Choose where to install the OSSEC HIDS [/var/ossec]:

Next you will be asked to configure email notifications and specify the required email address and SMTP server. OSSEC uses email notifications to alert you regarding events which triggered alerts.

3- Configuring the OSSEC HIDS

3.1- Do you want e-mail notification? (y/n) [y]: y
- What's your e-mail address?
- We found your SMTP server as:
- Do you want to use it? (y/n) [y]: y

The next steps allow you to specify which components of OSSEC are enabled.

3.2- Do you want to run the integrity check daemon? (y/n) [y]: y
- Running syscheck (integrity check daemon).

The integrity check daemon is responsible for monitoring and reporting changes in system files.

3.3- Do you want to run the rootkit detection engine? (y/n) [y]: y
- Running rootcheck (rootkit detection).

The rootkit detection engine regularly performs tests looking for signs of rootkits.

3.4- Active response allows you to execute a specific command based on the events received. For example, you can block an IP address or disable access for a specific user. More information at:
- Do you want to enable active response? (y/n) [y]: y
- Active response enabled.

3.5- By default, we can enable the host-deny and the firewall-drop responses. The first one will add a host to the /etc/hosts.deny and the second one will block the host on iptables (if Linux) or on ipfilter (if Solaris, FreeBSD or NetBSD).
-They can be used to stop SSHD brute force scans, port scans and some other forms of attacks. You can also add them to block on snort events. for example.
- Do you want to enable the firewall-drop response? (y/n) [y]: y
- firewall-drop enabled (local) for levels >= 6
- Default white list for the active response:
- Do you want to add more IPs to the white list (y/n)? [n]: n

3.6- Setting the configuration to analyze the following logs:
-- /var/log/messages
-- /var/log/auth.log
-- /var/log/syslog
-- /var/log/
-- /var/log/dpkg.log
-- /var/log/snort/alert (snort-fast file)
-- /var/log/apache2/error.log (apache log)
-- /var/log/apache2/access.log (apache log)
- If you want to monitor any other file, just change the ossec.conf and add a new localfile entry. Any questions about the configuration can be answered by visiting us at http://www/ossec/net.

Log analysis is enabled by default. It automatically analyzes the contents of these log files and alerts on any anomalies detected.

--- Press Enter to continue ---

<installation routine snipped>

- Configuration finished properly.
- To start OSSEC HIDS: /var/ossec/bin/ossec-control start
- To stop OSSEC HIDS: /var/ossec/bin/ossec-control stop
- The configuration can be viewed or modified at /var/ossec/etc/ossec.conf

--- Press Enter to finish. ---


OSSEC is now running and your host is being monitored for intrusions and anomalies. However, it is functioning in a default (untuned) state and should be tuned with custom settings for your environment. Some ways you can customize your installation of OSSEC include editing the rules and signatures to reflect the combination of applications and services running on your host, specifying additional logging sources and adjusting the criticality of alerts to reflect those issues most important to your environment (such as data/host criticality and sensitivity). Simply add or adjust the rules contained in the XML files in the /var/ossec/rules directory (the rule format is explained in the OSSEC online user manual). New rules can be obtained from BitBucket.

OSSEC Agent for Windows

If you configure your Linux/BSD host as an OSSEC server and you want it to monitor Windows hosts, you need to install the OSSEC agent on them. These agents connect to the server through an encrypted connection on UDP port 1514 (adjust any firewall rules accordingly). The server and agents are authenticated using a symmetric key that is defined on the server and then copied to the agents.

The agent .exe installation file is also available on the OSSEC download page. After you install it, you must specify the IP address and authentication key of an OSSEC server.

OSSEC Windows Agent installation OSSEC Windows Agent Manager

To generate an authentication key on an OSSEC server, you use the manage_agents command as shown below.

# cd /var/ossec/bin
# ./manage_agents
* OSSEC HIDS v2.7.1 Agent manager. *
* The following options are available: *
(A)dd an agent (A).
(E)xtract key for an agent (E).
(L)ist already added agents (L).
(R)emove an agent (R).
Choose your action: A, E, L, R or Q: A

- Adding a new agent (use '\q' to return to the main menu).
Please provide the following:
* A name for the new agent: Windows-server
* The IP Address of the new agent:
* An ID for the new agent[001]: 001
Agent information:
IP Address:

Confirm adding it?(y/n): y
Agent added.

Now you return to the manage_agents menu and this time select ‘E’.

* OSSEC HIDS v2.7.1 Agent manager. *
* The following options are available: *
(A)dd an agent (A).
(E)xtract key for an agent (E).
(L)ist already added agents (L).
(R)emove an agent (R).
Choose your action: A, E, L, R or Q: E

Available agents:
ID: 001, Name: Windows-server, IP:
Provide the ID of the agent to extract the key (or '\q' to quit): 001

Agent key information for '001' is:

** Press ENTER to return to the main menu.

You can confirm that the connection has succeeded by reviewing the contents of the /var/ossec/logs/ossec.log file on the agent and server.

OSSEC Web User Interface

To manage the OSSEC server or local installation with a graphical user interface, you can download the Web user interface (WUI) and then follow these steps.

# wget

Extract the contents of the gzipped tar file:

# tar -zxvf ossec-wui-0.8.tar.gz

Move and rename the OSSEC-WUI directory to the WWW directory which is accessible by Apache:

# mv ossec-wui-0.8/ /var/www/ossec-wui/

Change your working directory and start the setup routine:

# cd /var/www/ossec-wui/
# ./

The installation routine will prompt you as follows:

Setting up ossec ui...
New password:
Retype new password:
Enter your web server name (e.g. apache, www, nobody, www-data, ...) www-data
Enter your OSSEC install directory path (e.g. /var/ossec) /var/ossec/
You must restart your web server after this setup is done.
Setup completed successfully.


Now add the web server user account (such as apache or www-data) to the ossec group in /etc/group:

# gedit /etc/group

Example: ossec:x:1003 changes to ossec:x:1003:www-data

Then change the permissions on the OSSEC temporary directory:

# cd /var/ossec/
# chmod 770 tmp/
# chgrp www-data tmp/

Lastly, restart the Apache web daemon:

# apache2ctl restart

OSSEC Web User Interface v0.8

OSSEC Web User Interface (WUI) v0.8

Further reference, Watching for Potentially Malicious Domains with OSSEC, OSSEC online manual, OSSEC email abuse script, Defending WordPress with OSSEC, Securing Your Server With OSSEC, AnaLogi web interface for OSSEC, Using OSSEC for File Integrity Monitoring, OSSEC online manual, Multiple OSSEC articles, Protecting web servers with OSSEC

Recommended reading

If you found the content of this article helpful and want to expand your knowledge further, please consider buying a relevant book using the links below. Thanks!

OSSEC HIDS Guide on Amazon OSSEC Guide Instant OSSEC on Amazon Instant OSSEC

Practical Intrusion Analysis on Amazon Practical Intrusion Analysis Security Monitoring on Amazon Security Monitoring

The Tao of Network Security Monitoring on Amazon The Tao of Network Security Monitoring The Practice of Network Security Monitoring on Amazon The Practice of Network Security Monitoring


Written by Doug Vitale

February 26, 2014 at 4:04 PM

%d bloggers like this: