Azure Monitor: Collect Logs and Metrics from On-Premises

Reading Time: 5 minutes

In this blog we are going to discuss how we can collect logs and metrics from the Azure resource and on-prem infrastructure to the azure monitor.

Azure Monitor
data flow between different components

Suppose we have an application running on on-premises so we need to collect the logs and metrics from it and send it to Azure log analytics for analysis of the logs and metrics and create dashboards for the same.

Azure Monitor can collect data directly from your physical or virtual Linux computers in your environment into a Log Analytics workspace for detailed analysis and correlation using the azure log analytics agents.

Installing the Log Analytics agent allows Azure Monitor to collect data from a data center.

Before analyzing and acting on collected data, you first need to install log analytics agents on all of the machines that you want monitor.

Log analytics agent sends

  • System logs,
  • Performance metrics
  • Custom logs from any location

Prerequisite:

  • Log analytics workspace 
  • Log analytics workspace ID and Primary key

The agent communicates outbound to the Azure Monitor service over TCP port 443

Supported Linux operating systems:

  • Amazon Linux 2017.09 (x64)
  • CentOS Linux 6 (x86/x64) and 7 (x64)
  • Oracle Linux 6 and 7 (x86/x64)
  • Red Hat Enterprise Linux Server 6 (x86/x64) and 7 (x64)
  • Debian GNU/Linux 8 and 9 (x86/x64)
  • Ubuntu 14.04 LTS (x86/x64), 16.04 LTS (x86/x64), and 18.04 LTS (x64)
  • SUSE Linux Enterprise Server 12 (x64) and 15 (x64)

Network firewall requirements

The information below lists the proxy and firewall configuration information required for the Linux and Windows agents to communicate with Azure Monitor logs.

Agent ResourcePortsDirectionBypass HTTPS inspection
.ods.opinsights.azure.comPort 443OutboundYes
.oms.opinsights.azure.comPort 443OutboundYes
.blob.core.windows.netPort 443OutboundYes
Firewall Rules

Install the log analytics agent for Linux

To configure the Linux computer to connect to a Log Analytics workspace, run the following command providing the workspace ID and primary key copied earlier. 

Step-1: The following command downloads the agent validates its checksum and installs it.

Enter the loganalytics workspace id and primary key.

wget https://raw.githubusercontent.com/Microsoft/OMS-Agent-for-Linux/master/installer/scripts/onboard_agent.sh && sh onboard_agent.sh -w <YOUR WORKSPACE ID> -s <YOUR WORKSPACE PRIMARY KEY>

Step-2: Restart the agent by running the following command:

Enter the loganalytics workspace id.

sudo /opt/microsoft/omsagent/bin/service_control restart [<workspace id>]

Collect event and performance data

Azure Monitor can collect events from the Linux Syslog and performance counters that you specify for longer-term analysis and reporting. It can also take action when it detects a particular condition.

Follow these steps to configure the collection of events from the Linux Syslog, and performance counters.

  1. Go to Azure Portel
  2. Search log analytics
  3. Select your log analytics workspace
  4. Click on advance setting
  5. Select Data, and then select Syslog.
  6. You add Syslog by typing in the name of the log. Enter Syslog and then select the plus sign +.
  7. In the table, uncheck the severities Info, Notice and Debug.
  8. Select Save at the top of the page to save the configuration.
  9. Select Linux Performance Data to enable collection of performance counters on a Linux computer.
  10. When you first configure Linux Performance counters for a new Log Analytics workspace, you are given the option to quickly create several common counters. They are listed with a checkbox next to each.
  11. Select Apply below configuration to my machines and then select Add the selected performance counters. They are added and preset with a ten-second collection sample interval.
  12. Select Save at the top of the page to save the configuration.
Azure Monitor

View collected data

Now that you have enabled data collection, let’s run a simple log search example to see some data from the target computer.

  1. Go to your log analytics workspace
  2. In the selected workspace, from the left-hand pane, select Logs.
  3. On the Logs query page, type Perf in the query editor and select Run.For example, the query in the following image returned 10,000 Performance records. Your results will be significantly less.
Azure Monitor
Azure Monitor

Log Analytics Agent data sources:

These are the data source of the agent:

Azure Monitor

Performance Metrics to Monitor the Instance:

These are the performance metrics which we are getting by the azure log analytics agent.

Object NameCounter Name
Logical Disk% Free Inodes
Logical Disk% Free Space
Logical Disk% Used Inodes
Logical Disk% Used Space
Logical DiskDisk Read Bytes/sec
Logical DiskDisk Reads/sec
Logical DiskDisk Transfers/sec
Logical DiskDisk Write Bytes/sec
Logical DiskDisk Writes/sec
Logical DiskFree Megabytes
Logical DiskLogical Disk Bytes/sec
Memory% Available Memory
Memory% Available Swap Space
Memory% Used Memory
Memory% Used Swap Space
MemoryAvailable MBytes Memory
MemoryAvailable MBytes Swap
MemoryPage Reads/sec
MemoryPage Writes/sec
MemoryPages/sec
MemoryUsed MBytes Swap Space
MemoryUsed Memory MBytes
NetworkTotal Bytes Transmitted
NetworkTotal Bytes Received
NetworkTotal Bytes
NetworkTotal Packets Transmitted
NetworkTotal Packets Received
NetworkTotal Rx Errors
NetworkTotal Tx Errors
NetworkTotal Collisions
Physical DiskAvg. Disk sec/Read
Physical DiskAvg. Disk sec/Transfer
Physical DiskAvg. Disk sec/Write
Physical DiskPhysical Disk Bytes/sec
ProcessPct Privileged Time
ProcessPct User Time
ProcessUsed Memory kBytes
ProcessVirtual Shared Memory
Processor% DPC Time
Processor% Idle Time
Processor% Interrupt Time
Processor% IO Wait Time
Processor% Nice Time
Processor% Privileged Time
Processor% Processor Time
Processor% User Time
SystemFree Physical Memory
SystemFree Space in Paging Files
SystemFree Virtual Memory
SystemProcesses
SystemSize Stored In Paging Files
SystemUptime
SystemUsers

Data Retention in azure log analytics workspace:

The retention period of the collected data stored in the database depends on the selected pricing plan. Collected data is available for 31 days by default but can be extended to 730 days. Data is stored encrypted at rest in Azure storage, to ensure data confidentiality, and the data is replicated within the local region using locally redundant storage (LRS). The last two weeks of data are also stored in SSD-based cache and this cache is encrypted.

Conclusion:

We can collect logs(System or application logs) and performance metrics from the log analytics agent and It is totally secure. But this agent can’t send the application level metrics like java application. For the application-level metrics, we should go for the azure application insight which is part of the azure monitor.

Thank you for sticking to the end. If you like this blog, please do show your appreciation by giving thumbs ups and share this blog and give me suggestions on how I can improve my future posts to suit your needs. Follow me to get updates on different technologies

References:

  1. https://docs.microsoft.com/en-gb/azure/azure-monitor/learn/quick-collect-linux-computer
  2. https://docs.microsoft.com/en-gb/azure/azure-monitor/platform/log-analytics-agent#network-firewall-requirements
  3. https://docs.microsoft.com/en-gb/azure/azure-monitor/overview
  4. https://docs.microsoft.com/en-gb/azure/azure-monitor/platform/data-security
  5. https://docs.microsoft.com/en-gb/azure/azure-monitor/platform/log-analytics-agent
  6. https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-performance-counters
Azure Monitor

Written by 

Azmat Hasan is a Software Consultant at Knoldus Software LLP. He has done MCA from CDAC Noida in 2019. He has good knowledge of DevOps technologies i.e docker, Ansible, CI/CD(Jenkins, Bamboo), Kubernetes, Monitoring(Prometheus, Grafana), Logging(ELK Stack), etc. He is a self-motivated, enthusiastic person who believes in striving to achieve what we can sustain over a longer period of time, instead of working for short term benefits. He believe in working together to create synergy.