Device Normalizer Paper – ACM-formatted journal

Hello!

Today, I will try to share my research and implementation of one of the modules of our undergraduate thesis, the Adaptable Software-based Log Consolidation and Incident Management (AdLCIM), called the Log Normalizer Module. This module is very important for standardization of logs gathered for network monitoring.   The module is adaptable to other projects that need normalization or topics related to that.

Some prerequisites

There are some lessons required to learn to be able to understand this article although level of knowledge doesn’t have to be very advanced. These include: object oriented programming,  basic syntax in C#, dynamic loading. In order to follow the article per se, C# should be installed in a Windows platform.

Brief background re AdLCIM

Our thesis AdLCIM is basically a networking monitoring tool that accepts logs coming from different network devices in a LAN. The system collects the logs and analyzes them. But before the logs are analyzed, the logs must be put in a standardized form. Although the logs that are sent to the system have a format (Syslog format thru RFC 3164), the message still differs from one device to another due to device type or proprietary differences. This makes the role of the Log Normalizer Module very essential. If the system can standardize the log collected in a way that the user can easily understand then it will be very helpful to monitor the whole network. And since the system is adaptable to new devices, the system can be useful in future deployments/updates/versions by just creating new normalizers. To cut the story short, the standardized logs are summarized and correlated. The logs are classified as normal, attack or alerts logs. The attack logs are handled by an incident manager while alert logs are given recommendations to be resolved.

Building Device Normalizers

The research paper we made regarding the normalization of logs can be seen before the start of this article although it’s very boring to read. I just put it for reference. Anyway, a very important characteristic of the AdLCIM is that it is adaptable to new devices. This means that it should also be adaptable to standardization of new devices. But of course, handling new devices should not require the system to be rebuilt again for the sake of the new device. The program for the new device should be the only one compiled so it can connect to the system. To solve the issue, the use of dynamic loading is necessary. In this sense, dynamic loading is the ability to run a program together with a newly deployed program (connected together) without recompiling the original system.

In the Visual C# environment, some necessary configurations must be done in order to invoke dynamic loading. Here are some necessary codes:

using System.Reflection;

public static Object FindNormalizer(string AssemblyName,
string ClassName, string MethodName, Object[] args)

{
// Load Assembly
Assembly assembly = Assembly.LoadFrom(AssemblyName);

// Get Class
foreach (Type type in assembly.GetTypes())
{
if (type.IsClass == true)
{
if (type.FullName.EndsWith(“.” + ClassName))
{
// Activate Class
object ClassObj = Activator.CreateInstance(type);

// Dynamically Invoke the method
object Result = type.InvokeMember(MethodName,
BindingFlags.Default | BindingFlags.InvokeMethod,
null,
ClassObj,
args);
return (Result);
}
}
}
throw (new System.Exception(“Could not invoke method”));
}

The code shown above is the code use to invoke dynamic loading. In the first line you will see the library used so that assemblies can be called- System.Reflection. This library is capable of invoking and manipulating dynamically linked classes although we won’t be digging deeper on its other functionalities but its more or less its major function.

The FindNormalizer method has 4 parameters: Assembly name, Class name, Method and Parameters. This is a very easy to analyze since any source code has a file name (Assembly name), class, method and parameters. These are the main considerations that you have supply in order to invoke something dynamically. Once it is supplied with the necessary contents, the dynamically loaded program will be run when the method FindNormalizer is called with its fields filled.

Classifying Normalizer Type

When the AdLCIM is run, it accepts logs from network devices but from recognized ones. Meaning if a device tries to send logs but it is not recognized in the database then no log is recorded. A recognized device is a device given with proper identification such as a hostname, device type, device specification and device details. An example can be the laptop I’m using right now. Its hostname is Plato-DaAcademy, device type is COMPUTER, device specification is WINDOWS and device detail is 7. This process is required to other devices to be able to have its logs recorded to the database.

The importance of identifying devices is for you to know what devices you’re monitoring. Getting logs from unrecognized devices may bring confusion to log analysis. Thus, if there’s no device identification then all devices can just send logs though not necessary for monitoring. Another reason is for normalization process. It will be explained in the next paragraph.

As seen in the first code above, the FindNormalizer method has 4 parameters. The first parameter is the assembly name or the file name of the normalizer to be opened. The assembly name is classified based on the device type and device specification of the hostname that wants to send a log. So for example, the hostname Plato-DaAcademy (from previous example) tries to send a log. The system will look for the device type and device specification of Plato-DaAcademy and after that, its assembly name can be classified. For this example the device type and device specification are COMPUTER and WINDOWS. Therefore the assembly name that will be opened is “COMPUTER_WINDOWS_NORMALIZER.dll.”

The second parameter is the class name, which is the same as the format for the assembly name. Therefore the class name for the previous example is still COMPUTER_WINDOWS.  The third parameter is the method which in the system it is standardize as “Normalize.” All devices have a method “Normalize” which literally normalizes a log based on the standard imposed on them. The fourth parameter is the parameter of the method which is an object because there can be multiple parameters of different data types.

Below is a normalizer for an Intrusion Detection System called Snort.

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using LogNormalizer;

namespace LogNormalizer
{
class IDS_SNORT_NORMALIZER : SuperNormalizer
{

static string searchNormalizeAttack(string attackMessage)
{
int bCheck = 0;
string readLine, attackNotFound = “OTHERS”;
int checkComma, checkCompare;
string checkOriginal, normalizedAttackMessage;
StreamReader sr = new StreamReader(“c:\\AdLCIM_SnortNormalizedAttacks.txt”);
while (bCheck == 0)
{
readLine = sr.ReadLine();
Console.WriteLine(readLine);
checkComma = readLine.IndexOf(‘,’);
checkOriginal = readLine.Substring(0, checkComma);
Console.WriteLine(checkOriginal);
checkCompare = String.Compare(attackMessage, checkOriginal);
//Console.WriteLine(checkCompare);
if (checkCompare == 0)
{
normalizedAttackMessage = readLine.Remove(0, checkComma + 2);
Console.WriteLine(normalizedAttackMessage);
Console.WriteLine(“END”);
bCheck = 1;
sr.Close();
return normalizedAttackMessage;
}
else if (sr.EndOfStream)
{
bCheck = 1;
sr.Close();
return attackNotFound;
}

}
sr.Close();
return attackNotFound;
}

public static string parseSyslog(string SnortMessage)
{
try
{
string normalizedSnortMessage;
string getNormalizeAttackNow;
int result, result2, result3, result4, result5, result6, result7, result8, result9;
string message, message2, message3, message4, message5, message6;
string attackMessage, protocol, srcIP, srcPort, destIP, destPort;
string returnMe = “OTHERS”;

result = SnortMessage.IndexOf(‘]’);
// Console.WriteLine(result);
message = SnortMessage.Remove(0, result);
//Console.WriteLine(message);
result2 = message.IndexOf(‘ ‘);
message2 = message.Remove(0, result2 + 1);
//Console.WriteLine(message2);
result3 = message2.IndexOf(‘[‘);
attackMessage = message2.Substring(0, result3);
attackMessage = attackMessage.Substring(0, attackMessage.Length – 1);
Console.WriteLine(attackMessage);
// Console.WriteLine(message2);
result9 = message2.IndexOf(‘{‘);
message3 = message2.Remove(0, result9);
Console.WriteLine(message3);
result4 = message3.IndexOf(‘}’);
protocol = message3.Substring(1, result4 – 1);
Console.WriteLine(protocol);

if (protocol == “TCP” || protocol == “UDP”)
{

message4 = message3.Remove(0, result4 + 2);
//Console.WriteLine(message4);
result5 = message4.IndexOf(‘:’);
srcIP = message4.Substring(0, result5);
//Console.WriteLine(srcIP);
message5 = message4.Remove(0, result5);
//Console.WriteLine(message5);
result6 = message5.IndexOf(‘ ‘);
result7 = message5.IndexOf(‘>’);
srcPort = message5.Substring(1, result6 – 1);
//Console.WriteLine(srcPort);
message6 = message5.Remove(0, result7 + 2);
//Console.WriteLine(message6);
result8 = message6.IndexOf(‘:’);
destIP = message6.Substring(0, result8);
//Console.WriteLine(destIP);
destPort = message6.Remove(0, result8 + 1);
//Console.WriteLine(destPort);
getNormalizeAttackNow = searchNormalizeAttack(attackMessage);
normalizedSnortMessage = getNormalizeAttackNow + “, ” + protocol + “, ” + srcIP + “, ” + srcPort + “, ” + destIP + “, ” + destPort;
Console.WriteLine(“Attack found: ” + normalizedSnortMessage);

EventSnipe(destIP, destPort);
return normalizedSnortMessage;
}
else if (protocol == “ICMP”)
{
message4 = message3.Remove(0, result4 + 2);
Console.WriteLine(message4);
result5 = message4.IndexOf(‘-‘);
srcIP = message4.Substring(0, result5 – 1);
Console.WriteLine(srcIP);
result6 = message4.IndexOf(‘>’);
destIP = message4.Remove(0, result6 + 2);
Console.WriteLine(destIP);
getNormalizeAttackNow = searchNormalizeAttack(attackMessage);
normalizedSnortMessage = getNormalizeAttackNow + “, ” + protocol + “, ” + srcIP + “, X, ” + destIP + “, X”;
Console.WriteLine(normalizedSnortMessage);
return normalizedSnortMessage;
}
return SnortMessage;
}
catch (Exception er)
{
Console.WriteLine(“Cannot parse logs. ” + er.Message);
return SnortMessage;
}

}
public static void Normalize(DateTime timestamp, string hostname, int facility, int severity, string message)
{
string DeviceType;
string DeviceSpec;
string Facility;
string Severity;
string Message;
string Impact;
string Priority;
string ParseMessage;

Console.WriteLine(“\nNormalizing Syslog Message now…”);
Facility = getFacility(facility);
Severity = getSeverity(severity);
Priority = getPriority(facility, severity);

Impact = getImpact(facility, severity);
DeviceType = GetDeviceTypeFromHostname(hostname);
DeviceSpec = GetDeviceSpecFromHostname(hostname);
ParseMessage = parseSyslog(message);

Console.WriteLine(“\nLog successfully normalized. Sending to consolidator…”);
checkNormalizedMessageIfAlreadyExists(timestamp, hostname, DeviceType, DeviceSpec, Facility, Severity, ParseMessage, Priority);

}

} // END OF IDS_SNORT_NORMALIZER CLASS
}

The second code you see is a normalizer for a device with a device type IDS and device specification SNORT. The program is dynamically invoked once a hostname with a type IDS and specification SNORT tries to send a log.  The Normalize method has 5 parameters which will all be used for normalization. The ParseSyslog method is responsible for fixing the logs into a standardized format. The searchNormalizeAttack method on the other hand has a text file where in all possible attacks are seen and its standardized naming convention is found. An example log coming from a SNORT IDS is:

snort: [1:100000:0] THE BACKDOOR TINI HAS BEEN DETECTED [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} 172.16.4.106:1050 -> 172.16.4.101:7777

After passing through the IDS_SNORT_NORMALIZER.dll, the log becomes:

TINI, TCP, 172.16.4.106, 1050, 172.16.4.101, 7777

The log is now standardized into a format with- Attack Name, Protocol Type, Src IP, Src Port, Dest IP, Dest Port. This is the standardized log format for all IDS may it be SNORT or other IDS. So for instance, there’s a new IDS called “YATCO IDS” and it has different format, then an IDS_YATCO_NORMALIZER.dll must be created in order to make the log look like the standardized log. For other device types, there are different standardization methods made so it’s up to the administrator to follow it.

I hope this article is able to help you understand how to create a device normalizer and more importantly, make you realize how important it is to normalize logs. And of course, dynamically load assemblies without recompiling the whole program.