Google Cloud Dataflow Makes Big Data Offerings Even Bigger

In the latest of a string of cloud upgrades, Google on Thursday revealed automated data-processing features that the company hopes will encourage users to adopt its cloud for their big data and analytics needs.

From the Hadoop Summit in Brussels, Google launched a beta version of Google Cloud Dataflow, a managed logic-processing service. The Internet giant also unveiled upgrades to its popular BigQuery analytic platform, including new European zones.

The big data capabilities allow users to gain insights from their data without having to worry about managing hardware infrastructure and system administration, wrote William Vambenepe, a product manager at Google, in a blog post.

[Related: Google Bulks Up Cloud Networking Capabilities]

"The promise of big data is faster and better insight into your business. Yet it often turns into an infrastructure project, as you’re collecting a deluge of information and then correlating, enriching and attempting to extract real-time insights," Vambenepe said.

The big data functionality comes to Google's cloud just days after the release of several new networking features. In the past month, Google also released a smartphone app for remote cloud management, and a cold-storage service with a remarkably fast data-access time.

Google is touting a "NoOps" philosophy for cloud-centric big data processing that abstracts IT away from infrastructure through an auto-scaling platform that optimizes infrastructure consumption. Cloud Dataflow allows users to just submit their code, and the service provisions resources, auto-scales and releases those resources when the job is finished.

Dj Das, founder and CEO of Third Eye Consulting, a Google partner based in San Francisco, told CRN that Cloud Dataflow and Big Query "are actually some of the most required products on the market."

Third Eye has been focused on providing big data solutions since before the term was coined, Das said, and Google's cloud delivers unrivaled power, performance and security for those applications, especially for projects not burdened by legacy architectures.

Third Eye has already been using a preliminary version for three months. The service enables a single programmer to complete a job that used to require five people to administer Hadoop, he told CRN.

"With Google Dataflow, all I need to do is create the structure in Dataflow and the Google service takes care of the processes, how to manage them, scale up and scale down," Das said.

"It's one of the best technologies that we've seen yet," he said. "We don’t need so many administrators, people to make sure things are running properly, to manage the systems, just making it functional. Right now, I can write my code and deploy it on Google Dataflow and it takes care of the rest. How cool is that?"

Google upgraded Big Query with new security and performance features like row-level permissions and a higher default ingestion limit.

Thiag Loganathan, president of the big data insights division at DMI, a Google partner based in Bethesda, Md., told CRN via email that Google's ’new features excite us because they open the door to many new possibilities of value creation for our customers."

"More and more of our customers want to do things the 'cloud way.' This means frictionless project ramp-up, effective use of resources, little operational overhead and on-demand scale. With a public cloud platform like Google's, our customers can invest where it matters, in mobile app development and data science," Loganathan said.

Dataflow's ability to handle real-time or batch data with automatic scaling is a huge benefit for developers, he added. "Gone are the days of handling hardware exceptions, network failures or tuning execution to accommodate petabytes of data and processing pipeline complexity."

BigQuery's availability in European zones will address ’safe harbor’ requirements from European customers, Loganathan told CRN, and row-level permissions will be welcome features for companies with complex security requirements.

Chris Murphy, DMI's director of big data solutions, told CRN the company is seeing a trend of more and more clients moving to doing -- as Google likes to say -- ’Big Data the Cloud Way."

"We feel that our partnership with Google strengthens both of our offerings and creates a competitive market that provides clients with high-value, scalable solutions without heavy and sometimes unnecessary infrastructure investment,’ Murphy told CRN.

PUBLISHED APRIL 16, 2015