The promise of big data is faster and better insight into your business. Yet it often turns into an infrastructure project. Why? For example, you might be collecting a deluge of information and then correlating, enriching and attempting to extract real-time insights. Should you expect such feats, by their very nature, to involve a large amount of resource management and system administration? You shouldn’t. Not in the cloud. Not if you’re using big data the cloud way.

Big data the cloud way means being more productive when building applications, with faster and better insights, without having to worry about the underlying infrastructure. More specifically, it includes:

  • NoOps: Your cloud provider should worry about deploying, managing and upgrading infrastructure to make it scalable and reliable. “NoOps” means the platform handles such tasks and optimizations for you, freeing you up to focus on understanding and exploiting the value in your data.
  • Cost effectiveness: In addition to increased ease of use and agility, a “NoOps” solution provides clear cost benefits via the removal of operations work; but the cost benefits of big data the cloud way go even further – the platform auto-scales and optimizes your infrastructure consumption, and eliminates unused resources like idle clusters. You manage your costs by dialing up or down the number of queries and the latency of your processing based on your cost/benefit analysis. You should never have to re-architect your system to adjust your costs.
  • Safe and easy collaboration: You can share datasets from files in Google Cloud Storage or tables in Google BigQuery with collaborators inside or outside of your organization without the need to make copies or grant database access. There’s one version of the data – which you control – and authorized users can access it (at no cost to you) without affecting the performance of your jobs.

Google has been blazing the big data trail for the rest of the industry – so when you use Google Cloud Platform, big data the cloud way also means:

  • Cutting-edge features: Google Cloud Dataflow provides reliable, event-time-based stream processing, available by default with no extra work. But making stream processing easy and reliable doesn’t mean removing the option of running in batch. The same pipeline can execute in batch mode, which you can use to lower costs or analyze historical data. Now, consistently processing streaming data at large scale doesn’t have to be a complex and brittle endeavor that’s reserved for the most critical scenarios.

Google Cloud Platform delivers these characteristic by making data analysis quick, affordable and easy. Today, at the Hadoop Summit in Brussels, we announced that our big data services are taking a big step forward – allowing everyone to use big data the cloud way.

Google Cloud Dataflow now available in beta

Today, nothing stands between you and the satisfaction of seeing your processing logic, applied in your choice of streaming or batch mode, executed via a fully managed processing service. Just write a program, submit it, and Cloud Dataflow will do the rest. No clusters to manage – Cloud Dataflow will start the needed resources, autoscale them (within the bounds you choose), and terminate them as soon as the work is done. You can get started right now.

Google BigQuery has many new features and is now available in European zones

BigQuery, the quintessential cloud-native, API-driven service for SQL analytics, has new security and performance features. For example, the introduction of row-level permissions makes data sharing even easier and more flexible. With its ease of ingestion (we’ve raised the default ingestion limit to 100,000 rows per second per table), virtually unlimited storage, and fantastic query performance even for huge datasets, BigQuery is the ideal platform for storing, analyzing and sharing structured data. It also supports repeated records and querying inside JSON objects for loosely structured data. In addition, starting today, BigQuery now offers the option to store your data in Google Cloud Platform European zones. You can contact Google technical support today to use this option.

A comprehensive set of big data services

Google Cloud Pub/Sub is designed to provide scalable, reliable and fast event delivery as a fully managed service. Along with BigQuery streaming ingestion and Cloud Dataflow stream processing, it completes the platform’s end-to-end support for low-latency data processing. Whether you’re processing customer actions, application logs or IoT events, Google Cloud Platform allows you to handle them in real time, the cloud way. Leave Google Cloud Platform in charge of all the scaling and administration tasks so you can focus on what needs to happen, not how.

Using big data the cloud way doesn’t mean that Hadoop, Spark, Flink and other open source tools originally created for on-premises can’t be used in the cloud. We’ve ensured that you can benefit from the richness of the open source big data ecosystem via native connectors to Google Cloud Storage and BigQuery along with an automated Hadoop/Spark cluster deployment.

Google BigQuery customer zulily joined us recently for a big data webinar to share their experience using big data the cloud way and how it helped them increase revenue and overall business visibility while decreasing their operating costs. If you’re interested in exploring these types of benefits for your own company, you can easily get started today by running your first query on a public dataset or uploading your own data.

Here’s a simplified illustration of how Google Cloud Platform data processing services relate to each other and support all stages of the data lifecycle:



Scuba equipment helps humans operate under water, but divers still fall hopelessly short of the efficiency and agility of marine creatures. When it comes to big data in the cloud, be a dolphin, not a scuba diver. Google Cloud Platform offers a set of powerful, scalable, easy to use and efficient big data services built for the cloud. Embrace big data, the cloud way, by taking advantage of them today.

Learn more about Google Cloud Platform’s big data solutions or get started with Dataflow and BigQuery today. We can’t wait to see what you achieve when you use big data the cloud way.

-Posted by William Vambenepe, Product Manager