Why Centralize Logs?
Centralizing your logs saves time and increases the reliability of your log data. When Windows log files are stored locally on each server, you have to individually log into each one to go through them and look for any errors or warnings. If the server is unresponsive you might be out of luck. If you aren’t sure which servers are affected, you have to hunt through each one, which could take a long time on larger networks. The log files are also safer in a centralized location because even when your instances are terminated or your files are deleted (intentionally or unintentionally), the centralized backup copies of your logs are unaffected.
Windows Event Subscription
It’s possible for a Windows server to forward its events to a “subscribing” server. In this scenario the collector server can become a central repository for Windows logs from other servers in the network.
We will now see how that’s done in seven steps described below. These steps work on both Windows Server 2008 R2 and Windows Server 2012.
Example System
First, let’s describe our example system. We are using two Active Directory Domain–joined Windows Server 2012 systems. The domain name is MYTESTDOMAIN.COM and as shown in the figure below, the machines are both registered with the domain.
One of the servers, called MYTESTSQL, hosts a SQL Server 2014 instance. The other machine, called MYTESTSERVER, will work as an event log subscriber. It will be used to centralize all SQL Server related logs from MYTESTSQL. We will call MYTESTSQL the “source” and MYTESTSERVER the “collector.”
Setup Steps
1. Enable the Windows Remote Management Service
To begin with, remotely log into the source computer (MYTESTSQL) as a local or domain administrator and open a command prompt. Execute the following command:
1 | winrm quickconfig |
This enables Windows Remote Management service in the source server. If it has already been running, you will receive a message like this:
2. Configure the Windows Event Collector Service
Next, remotely access the collector machine (the centralized log collector, MYTESTSERVER) as a local or domain administrator. Again, open a command prompt and execute the following command:wecutil qcIn the prompt that comes up, as shown below, press y:
3. Configure the Event Log Readers Group
Once the commands have been run successfully, go back to the event source computer and open the Computer Management applet from the Server Manager. Once that starts, expand the Local Users and Groups node from the navigation pane and select the Groups folder.Double click on the “Event Log Readers” group. Once the dialog box appears, click Add.
In the “Select Users, Computers, Service Accounts or Groups” dialog box, click on the “Object Type” button and select the checkbox for “Computers” and click OK (shown below):
Type “MYTESTSERVER” and click on the “Check Name” button. If the computer account is found, it will be confirmed with an underline. Click OK and then OK again. This will close the dialog box shown below:
4. Configure Windows Firewall
If the source computer is running Windows Firewall, make sure it’s allowing Remote Event Log Management and Remote Event Monitor traffic, as shown below:
5. Create a Subscription
Next, start the Event Viewer application in the collector server (MYTESTSERVER) and select the Subscription node. From the Action menu in the right pane, choose the “Create Subscription…” link.
The following images show the options to choose in the Subscription Properties dialog box that appears:
a. Provide a name and description for the subscription.
b. Leave the “Destination log” field set to default value of Forwarded Events.
c. Choose the first option (“Collector initiated”) for subscription type and then click on the “Select Computers…” button.
d. Click on the “Add Domain Computers…” button in the dialog box that pops up.
e. Type the name of the source server (MYTESTSQL) and verify the name. Click OK twice to come back to the Subscription Properties main dialog box.
f. In the Events to collect section, click on the “Select Events…” button to bring up the Query Filter window.
g. Select Last 24 hours from the “Logged” drop-down list.
h. Select all the types of events (Warning, Error, Critical, Information, and Verbose).
i. Choose the option “By log” and in the drop-down list, select the tick box beside Application.Basically we want the remote server to forward all application events that happened in the last 24 hours.
j. Click OK to come back to the Subscription Properties main dialog box again.
k. Click on the “Advanced…” button and then in the Advanced Subscription Settings dialog box select the option for “Machine Account” if it’s not already selected.
l. Change the “Event Delivery Optimization” option to “Minimize Latency.”
m. Leave the Protocol to default value of HTTP and the Port to 5985.
n. Click OK to go back to the Subscription Properties dialog box and then click OK to close it.
The Subscription node in the collector computer’s event viewer should now show the subscription we just created.
6. Verify Events on Collector Computer
If we now select the Forwarded Events node, we will see this:
The column “Computer” now shows the events are from the remote computer MYTESTSQL.MYTESTDOMAIN.COM.You can enable or disable the collector subscription by right-clicking on the subscription and choosing “Disable” from the pop-up menu. When disabled, the status of the subscription is shown as disabled in the main window.An active collector subscription doesn’t mean it’s succeeding. To see if the collector can connect to the source, right click on the subscription and choose “Runtime Status” from the pop-up menu. In the image below, we can see the collector can’t connect to the source. By default it retries every five minutes:
If all is OK, subscription run-time status shows a green tick with an active status:
7. Create a Custom View (Optional)
Now that we have all the forwarded events, we can create custom views on top of those. For example, we might create one custom view for error events only. In this particular case, we can create a custom view for SQL Server–related messages only. A collector computer may host thousands of records from dozens of servers. Custom views allow us to create some order in it. For detailed steps, see the section Creating a Custom View.
There are several Windows services that you can use to centralize all your logging data to an external logging service. These services help you send logs over syslog to a cross-platform log server or cloud-based logging service like Loggly.
We recommend NXLog, which is a popular, freely downloadable service that runs as a background service. Alternately, you might like to use syslog-ng or Snare, which are also services that can collect your log files. All three services provide additional professional support for a fee.
To package your log files with NXlog:
- DOWNLOAD THE CURRENT VERSION OF NXLOG AND INSTALL IT. THE DOWNLOAD WILL INCLUDE AN INSTALLER, WHICH IS VERY SIMPLE AND INTUITIVE.
- ONCE THE INSTALLATION IS COMPLETE, OPEN NXLOG’S CONFIGURATION FILE AND MODIFY IT TO FIT YOUR NEEDS. WE WILL DESCRIBE HOW TO DO THIS IN THE CONFIGURING NXLOG SECTION NEXT.
- RESTART THE NXLOG SERVICE.
Configuring NXLog
By default, the NXLog configuration file is located at C:/Program Files (x86)/nxlog/conf/nxlog.conf. It allows you to create several types of configuration modules:
- INPUTS FOR THE SOURCE OF YOUR LOGS
- OUTPUTS FOR WHERE YOU WANT TO SEND THE LOGS
- ROUTES TO MAP YOUR INPUTS TO YOUR OUTPUTS
We will describe several modules for common log types below.
Windows Event Logs
Third-Party Logging Services
The following example shows how to modify the NXLog configuration file to centralize your Windows event logs. The im_msvistalog input module will send new entries to the Windows Event Log, including system, hardware, application, and security-related events.
You can use the same Output and Route configuration when you centralize your other logs, including application log files.
1 2 3 4 5 6 7 8 9 | # Windows Event Log <Input eventlog> # Uncomment im_msvistalog for Windows Vista/2008 and later Module im_msvistalog # Uncomment im_mseventlog for Windows XP/2000/2003 # Module im_mseventlog # If you prefer to send events as JSON data Exec $Message = to_json(); </Input> |
Application Log Files
Application log files are usually generated by programs or background services about their state or any error conditions encountered. The specifics of what information will be output is determined by the individual application’s developer. We have a great set of Best Practices for Application Logging.
File Logs
NXLog can be used to centralize logs from different applications. In this case you need to know the module file name of the application. In the following example, the file name is FILE1. The file name can also include directories or wild cards. You can use the same Output and Route configuration that you use for event logs.
1 2 3 4 5 6 | <Input FILE1> Module im_file File "FILE1" SavePos TRUE Exec $Message = $raw_event; </Input> |
IIS Logs
IIS logs contain access logs which are stored in W3C format. We recommend you convert them to JSON format for easy processing by a log management tool. NXLog can do this conversion using the W3C extension. Make sure that you use the proper format in the configuration file so the parsing happens correctly, and that you are including log files from all your sites. You can use the same Output and Route configuration that you use for event logs.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | <Extension w3c> Module xm_csv Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> # Convert the IIS logs to JSON and use the original event time <Input IIS_Site1> Module im_file File "C:/inetpub/logs/LogFiles/W3SVC1/u_ex*" SavePos TRUE Exec if $raw_event =~ /^#/ drop(); else w3c->parse_csv(); $SourceName = "IIS"; $Message = to_json(); </Input> |
SQL Server Error Logs
Microsoft SQL Server is Microsoft’s enterprise-class flagship database platform. It comes in a suite of database and data warehouse tools. SQL Server typically has its own logs saved in the application’s installation directory in the Windows file system. The default location for SQL Server 2012 is C:/Program Files/Microsoft SQL Server/MSSQL11.MSSQLSERVER/MSSQL/Log.The log entries are also sent to the Windows Application Event Log.
SQL Server operations like backup and restore, query timeouts, or slow I/Os are therefore easy to find from Windows Application Event Log while security-related messages like failed login attempts are captured in Windows Security Event Log.
Forwarding Logs to a Server
NXLog can forward logs from any of the input described above to an external destination such as a log server or cloud-based log management service.
To forward logs, add an output module in your nxlog.conf configuration file. Then add a Route module to send logs from your chosen inputs to your chosen outputs. In the example below, we are sending logs as syslog over TCP to the host HOSTNAME over the default syslog port 514. We are routing the eventlog input to this external destination.
1 2 3 4 5 6 7 8 9 | <Output out> Module om_tcp Host HOSTNAME Port 514 </Output> <strong><strong> </strong></strong> <Route 1> Path eventlog => out </Route> |
Several log management solutions offer specific setup instructions for Windows logging. Loggly is an example of one provider and has more detailed information about setting up NXLog to gather your log files in their guide, Logging from Windows.
Encrypting Logs with TLS
Sniffers can look at your log data if you transmit it over the Internet in clear text. Therefore, it’s a good idea to encrypt your log data when it’s in transit, especially if it contains sensitive information like personal identification details, government regulated data, or financial information. The most common protocol for encrypting syslog communication is TLS, or Transport Layer Security.
TLS encrypts your logs, preventing anyone from snooping on sensitive data in your logs, like personal information and customer details. While you don’t want to log information like passwords, poorly written applications might log passwords anyway. TLS encryption helps keep this kind of data safer. Malicious users subscribing to hosting companies (Platform-as-a-Service) can write custom scripts and applications to read user log data. Encryption guards against such malicious scripts.
Here’s an example NXLog configuration with TLS encryption. First copy the digital certificate file to your NXLog cert directory.
1 | copy loggly_full.crt C:/Program Files*/nxlog/cert<strong><strong> </strong></strong> |
Then configure your output module with om_ssl and the certificate location. The default syslog port for encrypted logs is 6514.
1 2 3 4 5 6 7 | <Output out> Module om_ssl Host server.example.com Port 6514 CAFile %CERTDIR%/example.crt AllowUntrusted FALSE <Output> |
How do you centralize your logs? Please add a comment to let us know!
Source: http://docphy.com/technology/computers/software/centralizing-windows-logs.html
No comments:
Post a Comment