Businesses generate a staggering amount of log data that contains rich information on systems, applications, user requests, and administrative actions. When managed effectively, this treasure trove of data can help you investigate and debug system issues, gain operational and business insights and meet security and compliance needs.

But log management is challenging. You need to manage very high volumes of streaming data, provision resources to handle peak loads, scale fast and efficiently and have the capability to analyze data in real-time.

Starting today, Google Cloud Logging is available in beta to help you manage all of your Google Compute Engine and Google App Engine logs in one place, and collect, view, analyze and export them. By combining Google Cloud Monitoring with Cloud Logging, you gain a powerful set of tools for managing operations and increasing business insights.

The Cloud Logging service allows you to:

  • Ingest and view the log data, so that you can see all your logs in one place
  • Search the log data in real-time, so that you can resolve operational issues
  • Analyze the log data in real-time, so that you can glean actionable insights
  • Archive logs data for longer periods, to meet backup and compliance requirements

Several customers are already using the features for logs viewing and analysis. Here’s what Wix has to say about Cloud Logging.
At Wix we use BigQuery to analyze logs of Compute Engine auto-scaled deployments. We get a large volume of syslog data that we send to BigQuery to get insights on system health state and error rates. We generate time series data and integrate it with Google Cloud Monitoring to monitor system performance and business metrics. This provides us with essential insight for the running of our operations.  - Dmitry Shestak, Engineer@Infrastructure team, Wix

Ingest and view the log data

We understand that it’s important for you to keep all your logs in one place so that you can easily analyze and correlate the data. Cloud Logging solves this problem in several ways:

  • Compute Engine VM logs can be automatically collected for about two dozen log types through the Google packaged fluentd agent, with additional logs possible through custom configuration.
  • Compute Engine Activity logs record all system actions and API calls are enabled by default, with no agent installation required.
  • App Engine logs that include syslog, request logs and application logs are automatically enabled for all App Engine projects, including applications using Managed VM runtimes.

You can view the logs in the Logs Viewer (shown below) in the Google Developers Console by clicking on the “Logs” link under “Monitoring.”
When viewing logs in the Logs Viewer, you can filter results using filter text or drop-downs

Search the log data in real-time

The Logs Viewer lets you quickly investigate and debug issues, correlate logs between different services and find the root cause of an outage. You can filter logs using the drop-down menu and the filter bar, stream logs in real-time ("tail -f") and navigate through your log timeline without awkward next/previous page buttons.

Here’s an example that shows how you can filter Compute Engine logs to see only Compute Engine “Firewall” service logs, pick a particular firewall resource to see the logs and do this for a particular log level.
A filtered view of logs data using the Logs Viewer


Analyze the log data in real-time

Many scenarios will require complex querying of the logs data in real-time. Cloud Logging allows you to easily stream logs to Google BigQuery as they arrive, letting you search, aggregate and view your data using SQL-like queries. To learn how to configure BigQuery export, visit the Exports tab of the Logs Viewer, or see the detailed documentation.

Once you enable BigQuery export, you can stream logs to BigQuery in real-time, and view them there in seconds.
Log data in the BigQuery tables
Let’s explore a couple of examples of how this data and the analysis capability can be really useful to you.

  • Monitoring Code Performance: There are situations when something unexpected happened or is indicative of an imminent problem e.g. “disk space low.” With Compute Engine log data in BigQuery, you can generate a time series and monitor logs with a particular severity. It’s simple, you just query metadata.severity = “WARNING” in the relevant tables. E.g.
     SELECT Count (*) AS total, Date(metadata.timestamp) AS time FROM (Table_date_range(TABLE ID, Timestamp('2015-03-01'), Timestamp('2015-03-12'))) WHERE metadata.severity = "warning" GROUP BY time ORDER BY total;
     
  • Monitoring Request Latency: High latency leads to poor user experience and failed requests, which can lead to frustrated users and lost revenue. With App Engine log data in BigQuery, you can create time series of latency data by aggregating and charting the “protoPayload.latency” field. You can see unusual latencies in real-time and take steps to resolve the issue.

Archive logs data for longer period

Cloud Logging retains logs in the Logs Viewer for 30 days, but in some scenarios, you need to store log data for a longer period. With the click of a button, you can configure export to Google Cloud Storage. It’s another channel for you to take data to BigQuery, Google Cloud Dataflow or any Hadoop solution for further processing and analysis of data. This makes it easier to meet your business or compliance requirements. And with the recent launch of Google Cloud Storage Nearline, long-term log storage becomes even more affordable.

Getting Started

If you’re a current Google Cloud Platform user, Cloud Logging is available to you at no additional charge. Applicable charges for using Google Cloud Platform services (such as BigQuery and Cloud Storage) will still apply. For more information, visit the the Cloud Logging documentation page and share your feedback.

- Posted by Deepak Tiwari, Product Manager