Amazon CloudWatch Logs integration
Amazon CloudWatch Logs is a monitoring service that enables you to collect, store, and analyze log data from AWS resources, applications, and services in near real-time. Site24x7’s integration with CloudWatch Logs enables seamless monitoring and analysis of your AWS log data in a centralized dashboard.
By integrating CloudWatch Logs with Site24x7, you gain real-time visibility into application logs, system logs, and AWS service logs, allowing you to detect anomalies, troubleshoot issues faster, and improve operational efficiency.
Overview
Site24x7's integration with CloudWatch Logs helps you aggregate, search, and analyze log data from multiple AWS services, set up alerts based on log patterns, and correlate logs with performance metrics for comprehensive observability. This integration also provides the following child monitors in addition to the CloudWatch Logs monitor:
- CloudWatch Logs Region monitor: Monitors the health and availability of all CloudWatch log groups within a specific AWS region.
- CloudWatch Log Group monitor: Tracks log ingestion metrics and custom filters for a specific CloudWatch log group to identify critical events and trends.
Use case
With Site24x7’s integration with Amazon CloudWatch Logs, customers gain deep visibility into the performance, security, and reliability of their AWS workloads by monitoring key metrics at the log group level.
For instance, using the CloudWatch Log Group monitor, users can track metrics related to subscription filters, log transformers, and anomaly detectors, along with general log group-level metrics. Subscription filters allow users to set patterns and route filtered log events in real-time to other AWS services, such as Amazon Kinesis, AWS Lambda, or Amazon Data Firehose, for downstream analysis or action.
Site24x7 enhances this by alerting users to delivery errors and throttling issues in these filters, helping them resolve data ingestion problems promptly. Additionally, transformer metrics—including transformed bytes, transformation errors, and processed event counts—are monitored to ensure that logs are normalized efficiently before storage or analysis. This is particularly useful when dealing with logs from heterogeneous sources. Moreover, Site24x7 also notifies users when critical configurations like data protection or anomaly detection are disabled for any log group to maintain compliance and security posture. Overall, the integration empowers users to proactively manage log health, troubleshoot issues faster, and optimize log processing pipelines across their AWS environment.
Benefits of Site24x7's Amazon CloudWatch Logs integration
Site24x7's integration with CloudWatch Logs provides you with the following benefits:
- Status propagation and proactive alerting: When thresholds are configured, any status change in child monitors automatically updates the parent monitor’s status, ensuring timely alerts and quicker incident response.
- Region-level visibility: The CloudWatch Logs Region monitor provides account-level insights, including subscription filters, transformation metrics, total log group count, and the overall storage used by log groups, helping you monitor logging activity across AWS regions.
- Log group-level insights: The CloudWatch Log Group monitor gives detailed log-group-level visibility, including metrics on log group subscription filters and transformation settings, allowing you to track the health and configuration of individual log groups.
- Focus on active log groups: Only log groups updated in the last month are actively monitored, ensuring that the data collected is current and relevant, improving performance and reducing noise.
- Improved operational intelligence: Gain deeper insights into active log groups and logging infrastructure health, enabling more efficient troubleshooting, auditing, and performance optimization.
Setup and configuration
- Log in to your Site24x7 account.
- Go to Cloud > AWS > Integrate AWS Account and create a cross-account IAM role to enable Site24x7 to access your AWS resources.
- On the Integrate AWS Account page, select CloudWatch Logs from the Services to be discovered list based on your requirements.
Permissions
Ensure that Site24x7 receives the following permissions to monitor Amazon CloudWatch Logs:
- "logs:GetDataProtectionPolicy",
- "logs:DescribeLogGroups",
- "logs:ListLogAnomalyDetectors",
- "logs:DescribeLogStreams",
- "logs:DescribeSubscriptionFilters",
- "logs:GetLogEvents",
- "logs:DescribeAccountPolicies",
- "logs:DescribeMetricFilters",
- "logs:DescribeFieldIndexes",
- "logs:ListTagsForResource"
Polling frequency
Site24x7 queries AWS service-level APIs to the set polling frequency (from once a minute to once a day) to collect metrics from Amazon CloudWatch Logs monitors.
Supported metrics
The supported metrics for Amazon CloudWatch Logs monitors are given below.
CloudWatch Logs Region monitor
| Metric name | Description | Statistics | Unit |
|---|---|---|---|
| Total Logs Groups | The total number of log groups in the region. | Maximum | Count |
| Total Storage Consumed | The total amount of storage consumed by the log groups in the region. | Sum | MB |
| Number of Standard Class Logs Groups | The total number of standard class log groups. | Maximum | Count |
| Number of Infrequent Access Class Logs Groups | The total number of infrequent access class log groups. | Maximum | Count |
| Number of Logs Groups in the last one hour | The number of log groups created in the last hour. | Maximum | Count |
| Number of Metric Filters | The total number of metric filters in the region. | Maximum | Count |
| Call Count | The number of specified API operations performed on your account. | Sum | Count |
| Error Count | The number of API operations performed in your account that resulted in errors. | Sum | Count |
| Throttle Count | The number of API operations performed in your account that were throttled because of usage quotas. | Sum | Count |
| Account Level Delivery Errors | The number of log events for which CloudWatch Logs received an error when forwarding data to the subscription destination. This is for the account-level subscription filter. | Sum | Count |
| Account Level Delivery Throttling | The number of log events for which CloudWatch Logs was throttled when forwarding data to the account-level subscription destination. | Sum | Count |
| Account Level Forwarded Bytes | The volume of log events in compressed bytes forwarded to the account-level subscription destination. | Sum | Bytes |
| Account Level Forwarded Log Events | The number of log events forwarded to the account-level subscription destination. | Sum | Count |
| Account Level Transformation Errors | The number of errors encountered while transforming log events with the account-level transformer. | Sum | Count |
| Account Level Transformed Bytes | The volume of the output of transformed log events, in uncompressed bytes. | Sum | Bytes |
| Account Level Transformed Log Events | The number of transformed log events by the account-level transformer. | Sum | Count |
CloudWatch Log Group monitor
| Metric name | Description | Statistics | Unit |
|---|---|---|---|
| Total Storage Consumed | The amount of storage consumed by the log group. | Sum | MB |
| EMF Parsing Errors | The number of parsing errors encountered while processing an embedded metric format log. These errors occur when logs are identified as embedded metric format but do not follow the correct format. | Sum | Count |
| Incoming Bytes | The volume of log events in uncompressed bytes uploaded to the log group. | Sum | Bytes |
| EMF Validation Errors | The number of validation errors encountered while processing embedded metric format logs. These errors occur when metric definitions within embedded metric format logs do not adhere to the embedded metric format and MetricDatum specifications. | Sum | Count |
| Incoming Log Events | The number of log events uploaded to the log group. | Sum | Count |
| Log Group Level Transformation Errors | The number of errors encountered while transforming log events with the log-group level transformer. | Sum | Count |
| Log Group Level Transformed Bytes | The volume of the output of transformed log events, in uncompressed bytes. | Sum | Bytes |
| Log Group Level Transformed Log Events | The number of transformed log events. | Sum | Count |
| Medium Priority Anomaly Count | The number of anomalies with medium priority. | Sum | Count |
| High Priority Anomaly Count | The number of anomalies with high priority. | Sum | Count |
| Low Priority Anomaly Count | The number of anomalies with low priority. | Sum | Count |
| Number of Metric Filters | The number of metric filters in the log group. | Maximum | Count |
| Log-Group Level Delivery Errors | The number of log events for which CloudWatch Logs received an error when forwarding data to the subscription destination. | Sum | Count |
| Log-Group Level Forwarded Bytes | The volume of log events in compressed bytes forwarded to the subscription destination. | Sum | Count |
| Log-Group Level Forwarded Log Events | The number of log events forwarded to the subscription destination. | Sum | Count |
Threshold configuration
To configure thresholds for a CloudWatch Logs monitor:
- Log in to your Site24x7 account and navigate to Admin > Configuration Profiles > Threshold and Availability.
- Click Add Threshold Profile.
- Select the applicable monitor type from the Monitor Type drop-down menu. The available monitor types are CloudWatch Logs, CloudWatch Logs Region, and CloudWatch Log Group.
- Provide an appropriate name in the Display Name field.
- The supported metrics are displayed in the Threshold Configuration section. You can set threshold values for all the metrics mentioned above.
- Click Save.
Status propagation in CloudWatch Logs monitors
Site24x7's CloudWatch Logs integration includes the status propagation feature, which ensures that alerts from child monitors are propagated to the parent CloudWatch Logs monitor. By enabling the Notify for Any Child Monitor Status Changes option on the Edit Threshold Profile page, you can streamline how alerts are handled across your CloudWatch Logs resources.
Child monitors, such as CloudWatch Logs Region and CloudWatch Log Group monitors, will notify the parent CloudWatch Logs monitor of any status changes. This propagation helps you maintain a centralized view of all alerts and ensures timely responses to potential issues.
For instance, if a CloudWatch Log Group monitor detects a high error rate and changes to a Trouble state, this status is automatically propagated to the parent CloudWatch Logs monitor, triggering an alert—provided the Notify for Any Child Monitor Status Changes option is enabled. This ensures you’re alerted at the top level without needing to check each individual log group monitor.
Licensing
- Every 100 CloudWatch Log Group monitors will consume one advanced monitor license. This means rather than counting each CloudWatch Log Group monitor separately, Site24x7 considers a set of 100 CloudWatch Log Group monitors as one advanced monitor unit.
- The CloudWatch Logs and CloudWatch Logs Region monitors are free monitors.
Viewing Amazon CloudWatch Logs data
To monitor your CloudWatch Logs environment, log in to your Site24x7 account and navigate to Cloud > AWS > CloudWatch Logs.
Monitor data
The monitor data for each Amazon CloudWatch Logs monitor is given below.
CloudWatch Logs
Summary
The Summary tab provides an overview of the event timeline and metrics in the form of charts.
Regions
View the Regions Availability and the list of Region monitors along with their status, monitor type, and action options. You can configure thresholds using the Action button of the preferred source server monitor, and you can set bulk thresholds using the Threshold Configuration button. To view the CloudWatch Logs Region monitor details, click the desired monitor name hyperlink.
Log Groups
View the Log Groups Availability and the list of Log Groups monitors along with their status, monitor type, and action options. You can configure thresholds using the Action button of the preferred source server monitor, and you can set bulk thresholds using the Threshold Configuration button. To view the CloudWatch Log Group monitor details, click the desired monitor name hyperlink.
Outages
The Outages tab provides details on an outage's start time, end time, duration, and comments (if any).
Inventory
Obtain details like the Resource Name, Check Frequency, and Monitor Licensing Category on the Inventory tab. The Threshold and Availability Profile and the Notification Profile can be set according to the user and viewed on this tab.
Log Report
This tab offers a consolidated report of each CloudWatch Logs monitor's log status, which can be downloaded as a CSV file.
Alert Logs
This tab displays a chronological list of all triggered alerts related to CloudWatch Logs. This tab helps you trace alert history and severity to assess issues and validate threshold settings.
CloudWatch Logs Region
Summary
The Summary tab provides an overview of the event timeline and metrics in the form of charts.
Log Groups
View the Log Groups Availability and the list of Log Groups monitors along with their status, monitor type, and action options. You can configure thresholds using the Action button of the preferred source server monitor, and you can set bulk thresholds using the Threshold Configuration button. To view the CloudWatch Log Group monitor details, click the desired monitor name hyperlink.
Configuration
View the configuration information of CloudWatch Logs Region monitors, such as Account Level Transformer and Account Level Subscription Filter details, in this tab.
Outages
The Outages tab provides details on an outage's start time, end time, duration, and comments (if any).
Inventory
Obtain details like the Resource Name, Check Frequency, and Monitor Licensing Category on the Inventory tab. The Threshold and Availability Profile and the Notification Profile can be set according to the user and viewed on this tab.
Log Report
This tab offers a consolidated report of each CloudWatch Logs monitor's log status, which can be downloaded as a CSV file.
Alert Logs
This tab displays a chronological list of all triggered alerts related to CloudWatch Logs. This tab helps you trace alert history and severity to assess issues and validate threshold settings.
CloudWatch Log Group
Summary
The Summary tab provides an overview of the event timeline and metrics in the form of charts.
Log Streams
View the list of Log Streams associated with the CloudWatch Log Group monitor. Click the desired log stream name to view the Log Events dialog box. The Log Events dialog box provides details such as Log Stream Name, Creation Time, and Last Events Time along, with the list of log events for the selected time period, allowing you to view each event's message and timestamp.
Subscription Filters
This tab is displayed only when subscription filters are present for the monitor. You can view subscription details such as Filter Name, Filter Pattern, Destination ARN, and Destination Type from this tab.
Metric Filters
This tab displays all the CloudWatch metric filters configured for the selected log group. Metric filters enable you to extract metric data from log events for visualization or alerting. The tab shows key filter attributes such as Filter Name, Creation Time, and Filter Pattern.
Monitored Resources
The Monitored Resources tab lists all the KMS monitor resources that are managed and monitored by Site24x7 if the log group is associated with the KMS service.
Data Protection Policy
This tab shows the log group’s data protection policies if any are configured. These policies help secure sensitive data in your log streams by specifying how to identify and redact PII or confidential data.
Configuration
View the configuration information such as Region, Log Group Name, and Log Group Class in this tab.
Outages
The Outages tab provides details on an outage's start time, end time, duration, and comments (if any).
Inventory
Obtain details like the Log Group Name, Check Frequency, and Monitor Licensing Category on the Inventory tab. The Threshold and Availability Profile and the Notification Profile can be set according to the user and viewed on this tab.
Log Report
This tab offers a consolidated report of each CloudWatch Log Group monitor's log status, which can be downloaded as a CSV file.
Alert Logs
This tab displays a chronological list of all triggered alerts related to the CloudWatch Log Group monitor. This tab helps you trace alert history and severity to assess issues and validate threshold settings.