The main function of OSSIM Agent is to collect all the data sent by various devices that exist on the network, and then send it to OSSIM Server in a standard way. After collecting the data, Agent should normalize the data before sending it to Server. This paper mainly focuses on how to send the data in an orderly way and how to complete normalization.Discuss.
OSSIM sensors convert the communication protocol and data format between OSSIM proxy and OSSIM server through GET framework.Let's start with a brief look at the ossim-agent script:
#!/usr/bin/python -OOt import sys sys.path.append('/usr/share/ossim-agent/') sys.path.append('/usr/local/share/ossim-agent/') from ossim_agent.Agent import Agent agent = Agent() agent.main()
Here, GET is required to transport data to the OSSIM server as an OSSIM proxy.The two main operations required for tight integration are "Mapping" for generating (or) OSSIM-compliant events and "Transferring" servers for such data to OSSIM.The two components of the GET framework that it is responsible for such operations are EventHandler and Ender Agent, as shown in Figure 1.
Figure 1 integrates Get framework content into OSSIM
The primary task of Event Handler is to map events collected by the data source plug-in to the OSSIM standardized event format for SIEM instance alerts.To perform this process, the original message undergoes a transition from RAW LOG to the existing normalized data field format; in the figure above, these mechanisms are represented as Normalization and OSSIM messages.Partial log normalization code:
from Logger import Logger from time import mktime, strptime logger = Logger.logger class Event: EVENT_TYPE = 'event' EVENT_ATTRS = [ "type", "date", "sensor", "interface", "plugin_id", "plugin_sid", "priority", "protocol", "src_ip", "src_port", "dst_ip", "dst_port", "username", "password", "filename", "userdata1", "userdata2", "userdata3", "userdata4", "userdata5", "userdata6", "userdata7", "userdata8", "userdata9", "occurrences", "log", "data", "snort_sid", # snort specific "snort_cid", # snort specific "fdate", "tzone" ] def __init__(self): self.event = {} self.event["event_type"] = self.EVENT_TYPE def __setitem__(self, key, value): if key in self.EVENT_ATTRS: self.event[key] = self.sanitize_value(value) if key == "date": # In seconds self.event["fdate"]=self.event[key] try: self.event["date"]=int(mktime(strptime(self.event[key],"%Y-%m-%d %H:%M:%S"))) except: logger.warning("There was an error parsing date (%s)" %\ (self.event[key])) elif key != 'event_type': logger.warning("Bad event attribute: %s" % (key)) def __getitem__(self, key): return self.event.get(key, None) # Event Representation def __repr__(self): event = self.EVENT_TYPE for attr in self.EVENT_ATTRS: if self[attr]: event += ' %s="%s"' % (attr, self[attr]) return event + "\n" # Return internal hash value def dict(self): return self.event def sanitize_value(self, string): return str(string).strip().replace("\"", "\\\"").replace("'", "") class EventOS(Event): EVENT_TYPE = 'host-os-event' EVENT_ATTRS = [ "host", "os", "sensor", "interface", "date", "plugin_id", "plugin_sid", "occurrences", "log", "fdate", ] class EventMac(Event): EVENT_TYPE = 'host-mac-event' EVENT_ATTRS = [ "host", "mac", "vendor", "sensor", "interface", "date", "plugin_id", "plugin_sid", "occurrences", "log", "fdate", ] class EventService(Event): EVENT_TYPE = 'host-service-event' EVENT_ATTRS = [ "host", "sensor", "interface", "port", "protocol", "service", "application", "date", "plugin_id", "plugin_sid", "occurrences", "log", "fdate", ] class EventHids(Event): EVENT_TYPE = 'host-ids-event' EVENT_ATTRS = [ "host", "hostname", "hids_event_type", "target", "what", "extra_data", "sensor", "date", "plugin_id", "plugin_sid", "username", "password", "filename", "userdata1", "userdata2", "userdata3", "userdata4", "userdata5", "userdata6", "userdata7", "userdata8", "userdata9", "occurrences", "log", "fdate", ] class WatchRule(Event): EVENT_TYPE = 'event' EVENT_ATTRS = [ "type", "date", "fdate", "sensor", "interface", "src_ip", "dst_ip", "protocol", "plugin_id", "plugin_sid", "condition", "value", "port_from", "src_port", "port_to", "dst_port", "interval", "from", "to", "absolute", "log", "userdata1", "userdata2", "userdata3", "userdata4", "userdata5", "userdata6", "userdata7", "userdata8", "userdata9", "filename", "username", ] class Snort(Event): EVENT_TYPE = 'snort-event' EVENT_ATTRS = [ "sensor", "interface", "gzipdata", "unziplen", "event_type", "plugin_id", "type", "occurrences" ]
Log code: Zhengzhou Infertility Hospital: http://yyk.39.net/zz3/zonghe/1d427.html
import threading, time from Logger import Logger logger = Logger.logger from Output import Output import Config import Event from Threshold import EventConsolidation from Stats import Stats from ConnPro import ServerConnPro class Detector(threading.Thread): def __init__(self, conf, plugin, conn): self._conf = conf self._plugin = plugin self.os_hash = {} self.conn = conn self.consolidation = EventConsolidation(self._conf) logger.info("Starting detector %s (%s).." % \ (self._plugin.get("config", "name"), self._plugin.get("config", "plugin_id"))) threading.Thread.__init__(self) def _event_os_cached(self, event): if isinstance(event, Event.EventOS): import string current_os = string.join(string.split(event["os"]), ' ') previous_os = self.os_hash.get(event["host"], '') if current_os == previous_os: return True else: # Failed and added to cache self.os_hash[event["host"]] = \ string.join(string.split(event["os"]), ' ') return False def _exclude_event(self, event): if self._plugin.has_option("config", "exclude_sids"): exclude_sids = self._plugin.get("config", "exclude_sids") if event["plugin_sid"] in Config.split_sids(exclude_sids): logger.debug("Excluding event with " +\ "plugin_id=%s and plugin_sid=%s" %\ (event["plugin_id"], event["plugin_sid"])) return True return False def _thresholding(self): self.consolidation.process() def _plugin_defaults(self, event): # Get default parameters from configuration file if self._conf.has_section("plugin-defaults"): # 1) Date default_date_format = self._conf.get("plugin-defaults", "date_format") if event["date"] is None and default_date_format and \ 'date' in event.EVENT_ATTRS: event["date"] = time.strftime(default_date_format, time.localtime(time.time())) # 2) Sensors default_sensor = self._conf.get("plugin-defaults", "sensor") if event["sensor"] is None and default_sensor and \ 'sensor' in event.EVENT_ATTRS: event["sensor"] = default_sensor # 3) Network Interface default_iface = self._conf.get("plugin-defaults", "interface") if event["interface"] is None and default_iface and \ 'interface' in event.EVENT_ATTRS: event["interface"] = default_iface # 4) Source IP if event["src_ip"] is None and 'src_ip' in event.EVENT_ATTRS: event["src_ip"] = event["sensor"] # 5) Time Zone default_tzone = self._conf.get("plugin-defaults", "tzone") if event["tzone"] is None and 'tzone' in event.EVENT_ATTRS: event["tzone"] = default_tzone # 6) sensor,source ip and dest != localhost if event["sensor"] in ('127.0.0.1', '127.0.1.1'): event["sensor"] = default_sensor if event["dst_ip"] in ('127.0.0.1', '127.0.1.1'): event["dst_ip"] = default_sensor if event["src_ip"] in ('127.0.0.1', '127.0.1.1'): event["src_ip"] = default_sensor # Type of detection log if event["type"] is None and 'type' in event.EVENT_ATTRS: event["type"] = 'detector' return event def send_message(self, event): if self._event_os_cached(event): return if self._exclude_event(event): return #Use default values for some empty attributes. event = self._plugin_defaults(event) # Check before merge if self.conn is not None: try: self.conn.send(str(event)) except: id = self._plugin.get("config", "plugin_id") c = ServerConnPro(self._conf, id) self.conn = c.connect(0, 10) try: self.conn.send(str(event)) except: return logger.info(str(event).rstrip()) elif not self.consolidation.insert(event): Output.event(event) Stats.new_event(event) def stop(self): #self.consolidation.clear() pass #Override in subclasses def process(self): pass def run(self): self.process() class ParserSocket(Detector): def process(self): self.process() class ParserDatabase(Detector): def process(self): self.process()
... ...
As you can see from the above, the normalization of the sensor is mainly responsible for re-encoding the data fields within each LOG to generate a completely new event that may be used to send to the OSSIM server.In order to achieve this goal, the GET framework contains specific functions for converting all functions to fields requiring BASE64 conversion.The OSSIM Message is responsible for populating fields that do not exist in the original event generated by the GET.So the plugin_id, plugin_sid mentioned above are used to represent the log message source type and subtype, which are also required fields to generate SIEM events.For event format integrity, sometimes the system will use 0.0.0.0 to populate the field by default when the source or destination IP cannot be confirmed.
Note: This required field allows us to view the MySQL database for OSSIM using the phpmyadmin tool.
Sender Agent is responsible for the following two tasks:
Send events collected by GET and formatted by event to OSSIM server. This task is accomplished by Event Hander creating message queues to send to message middleware, as shown in Figure 2.
Figure 2 Sequence diagram: Conversion from security detector log to OSSIM server events
2) Manage the communication between the GET framework and the OSSIM server, with TCP 40001 as the communication port through a two-way handshake.Normalizing the original logs is an important part of the normalization process. OSSIM preserves the original logs while normalizing the logs, which can be used for log archiving and provides a means to extract the original logs from normalized events.
The normalized EVENTS is stored in the MySQL database, as shown in Figure 3.Then the correlation engine makes cross-correlation analysis according to the parameters such as rule, priority, reliability, etc. to get the risk value and send out various alert information.Zhengzhou Infertility Infertility Hospital: http://yyk.39.net/zz3/zonghe/1d427.html
Figure 3 OSSIM platform log storage mechanism
Next, let's look at an example. Here is a raw log of Apache, CiscoASA, and SSH, as shown in Figures 4, 5, and 6.
- Regular expressions in the Apache plug-in:
[0001 - apache-access] Access logevent_type=event regexp=((?P\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})(:(?P\d{1,5}))? )?(?P\S+) (?P\S+) (?P\S+) \[(?P\d{2}\/\w{3}\/\d{4}:\d{2}:\d{2}:\d{2})\s+[+-]\d{4}\] \"(?P[^\"]*)\" (?P\d{3}) ((?P\d+)|-)( \"(?P[^\"]*)\" \"(?P[^\"]*)\")?$ src_ip={resolv($src)} dst_ip={resolv($dst)} dst_port={$port} date={normalize_date($date)} plugin_sid={$code} username={$user} userdata1={$request} userdata2={$size} userdata3={$referer_uri} userdata4={$useragent} filename={$id}
[0002 - apache-error] error log
event_type=event regexp=\[(?P\w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \d{4})\] \[(?P(emerg|alert|crit|error|warn|notice|info|debug))\] (\[client (?P\S+)\] )?(?P.*) date={normalize_date($date)} plugin_sid={translate($type)} src_ip={resolv($src)} userdata1={$data}
Figure 4 Apache original log
Figure 51 Cisco ASA raw log
Figure 6 Cisco ASA event classification
The actual format after normalization by OSSIM is presented to you through the Web front end for easy reading.Comparing the normalized events with the original logs will be explained in the book OSSIM Troubleshooting for Open Source Security Operations and Maintenance Platform: Introduction.In the example shown in Figure 7, only Userdata1 and Userdata2 are used, and the extensions Userdata3-Userdata9 are not used, mainly to reserve for other devices or services, where the destination address is marked as an IP address, for example, Host192.168.11.160.In fact, normalization occurs after the system collects and stores events, before Association and data analysis, in SIEM tools, the data is converted into a readable format during collection, and the formatted data is easier to understand.