We recently made available the Beta release of Google Cloud Monitoring, a new service based on Stackdriver that allows customers to gain insight into the performance, capacity and uptime of their applications. Combined with Google Cloud Logging, Google Cloud Platform provides our customers with a powerful set of operational tools.

One of our core goals is to make operations easy and enable insights for our customers. Google Cloud Logging targets three very important customer challenges:

  • Being able to analyze log data in real-time
  • Monitoring errors and error rates to proactively manage issues
  • Archiving log data for longer periods in a cost-effective manner

Today, we're making a new export feature available in beta, that allows customers to export their Google App Engine logs to a Google BigQuery dataset as well as to a Google Cloud Storage bucket. These features will help customers store and analyze log data.

The export can be configured in the Cloud Logs Viewer. The export configuration UI can be accessed by clicking on the gear icon, next to the filter dropdowns.

Key Features

Export to BigQuery – in the export configuration page, you can chose to stream logs to BigQuery by using a predefined dataset (or adding a new dataset) to export logs.

Once you check the box to “Stream App Engine Logs to BigQuery” and hit “Done” you have enabled the export. You can now visit BigQuery under “Big Data” in the developers console to view, manage, and analyze logs in BigQuery.
Now you can use the full power of BigQuery to analyze the logs and gain insights on things like traffic, errors, latency, cost, etc. For example, you can run a query on your syslog files to get the count of logs by cluster zones or errors in a particular time range.

Export to Google Cloud Storage – in the export configuration page, you can chose to export logs to Cloud Storage by using a pre-defined storage bucket (or adding a new bucket) to store logs.
Once you check the box to “Stream App Engine Logs to Cloud Storage” and hit “Done”, you have enabled the export. You can now visit Cloud Storage under “Storage” in the developers console to manage the archived logs.

Cloud Storage allows users to archive logs based on custom archival needs to help meet business or compliance requirements. It also provides the flexibility to transfer the logs to BigQuery, Cloud Dataflow, or any Hadoop solution for further processing and analysis of data.

Customers are already gaining benefit from using BigQuery for logs analysis, one of whom is Wix.
At Wix we use BigQuery to analyze logs of Compute Engine auto-scaled deployments. We get a large volume of syslog data that we send to BigQuery to get insights on system health state and error rates. We generate time series data and integrate it with Google Cloud Monitoring to monitor system performance and business metrics. This provides us with essential insight for the running of our operations.
        - Dmitry Shestak, Engineer@Infrastructure team, Wix

Getting Started

If you’re a current Google Cloud Platform user, this Cloud Logging export feature is available to you at no additional charge. Please note that applicable charges for your use of Google Cloud Platform services will still apply. For more information, please visit the the Cloud Logging documentation page; and we would love to hear your feedback.

-Posted by Deepak Tiwari, Product Manager