Introduction:
It an ideal and a best practice to centralize logging and monitoring by integrating various services and Azure resources/workloads into their Log Analytics Workspace. These resources may include security systems, network devices, application servers and many more. In cloud world It’s always crucial to identify the top contributors to log ingestion to better manage costs, enhance performance, and streamline troubleshooting.
In this blog, we will explore the top 10 resources that inject logs into an Azure Log Analytics Workspace. We’ll analyze their log ingestion patterns, the volume of data they contribute, and the potential costs associated with handling their data. Understanding these contributors will help you fine-tune your logging strategy for better operational and cost management.
Kusto Query to retrive expensive log- generating resources
let startTime = ago(1d); // Adjust the time range as needed
let endTime = now();
let costPerGB = 2.35; // Replace with your actual cost per GB
let logs = union *
| where TimeGenerated between (startTime .. endTime)
| summarize LogCount = count(), TotalSizeInBytes = sum(_BilledSize) by ResourceId
| extend TotalSizeInGB = round(TotalSizeInBytes / (1024 * 1024 * 1024))
| extend DataIngestionCost =strcat("$", round(TotalSizeInGB * costPerGB))
| top 10 by LogCount desc
| project-away TotalSizeInBytes;
logsstartTime – Defines the time range of log extraction.
costPerGB– Define the rate/GB (It may vary) .
union * – union with wildcard(*) combines all the tables in the workspace.
where TimeGenerated between (startTime .. endTime) – Filter the logs to include only those that were generated between startTime (1 day ago) and endTime (now).
summarize LogCount = count(), TotalSizeInBytes = sum(_BilledSize) by ResourceId
Group the logs by ResourceId by calculating the total number of logs (LogCount) for each resource and sums the size of the logs (TotalSizeInBytes) based on the _BilledSize field (which represents the amount of data being billed).
extend TotalSizeInGB = round(TotalSizeInBytes / (1024 * 1024 * 1024)) – This Converts the total size from bytes to gigabytes by dividing TotalSizeInBytes by (1024 * 1024 * 1024) and rounds the result to make the size more readable .
top 10 by LogCount desc – Sorts the LogCount by desc and retrives top 10 .
DataIngestionCost – Will give you the exact cost based on the log size per day.
project-away TotalSizeInBytes – Removes the TotalSizeInBytes column from the final result, leaving only the ResourceId, LogCount, and TotalSizeInGB
Finally, the result shows the top 10 resources (by ResourceId) that injected the most logs into the workspace over the last day.
Summary:
The Kusto query analyzes log data from all resources in the Azure Log Analytics Workspace over the past 24 hours. It combines logs from multiple sources, filters them within a specified time range, and summarizes the data based on the number of logs (LogCount) and the total size of the logs in gigabytes (TotalSizeInGB). The query then identifies the top 10 resources (by ResourceId) generating the highest volume of logs. By showing both the number of logs and the data size, the query provides a clear view of the resources contributing most to log ingestion, allowing for better monitoring, performance optimization, and cost management.
