5 Things We Learned From Amazon CTO Werner Vogels At Re:Invent

Vogels Vast Vision

Amazon CTO Werner Vogels took the stage Thursday at AWS re:Invent to deliver his traditional final keynote of the conference.

The cloud computing guru, as usual, focused on the needs of developers during his ebullient keynote, sharing a vision for how applications would be built in the future on the industry's leading public cloud platform.

In addition to his unique insights into the future of cloud development, Vogels introduced some new AWS capabilities largely influenced by Amazon's own efforts to deliver highly scaled services to customers of its massive e-commerce platform.

Vogels also discussed the best and worst days of his career at Amazon—both of which had to do with the company's tortured relationship with Oracle databases.

Serverless State Of Mind

"Serverless is still on my mind as kind of the next generation of how we're going to build systems," Vogels told re:Invent attendees.

Amazon Web Services pioneered the approach to running workloads sans servers with Lambda. But the cloud leader is still working hard to advance the serverless paradigm, Vogels said, before introducing two new capabilities in that arena.

Lambda Layers allows developers to centrally manage code and data shared across multiple Lambda functions.

And Lambda Runtime API delivers a simple interface developers can take advantage of to use their favorite programming languages or specific language versions in developing their Lambda functions.

Runtime API can be used with Layers to enable code written in any programming language to be shared across Lambda functions.

Worst Day at Amazon

Dec 12, 2004 was Werner Vogels worst day at Amazon, he said.

Back then, the e-commerce giant was running its entire business on a drastically different database architecture—three massive databases storing all customer, product, and order information on Oracle RAC servers.

As customers put in orders that year on the last day they could get items to ship in time for Christmas, a bug took down one of those databases for 12 hours.

"We realized we had somewhat ourselves to blame," Vogels said. "We had no control over the reliability, scalability, performance."

What followed was a comprehensive project to rearchitect Amazon's database infrastructure to allow rapid scaling with only single unit of failure. Amazon ultimately implemented a thin layer of management over its traditional databases, taking control of its database sharding and partitioning functionality.

By breaking up the system into smaller pieces, Amazon reduced the blast radius of future failures.

"Remember, everything fails, all the time," Vogels told re:Invent attendees.

Happiest Day at Amazon

Less than a month ago, Vogels enjoyed his happiest day at the e-commerce giant.

That day was Nov. 1, when Amazon switched off perhaps the world's largest data warehouse and migrated it from Oracle to its own data warehouse product, RedShift.

RedShift was able to assume the load because of substantial improvements made over the last year to the native AWS data warehouse service, Vogels said.

AWS took advantage of its ability to "observe how hundreds of thousands of businesses make use of RedShift" to drive upgrades to that service. Since then, AWS has "seen enormous improvements in real-world workloads" running on RedShift, Vogels said.

Streaming Simplifications

The burgeoning number of sensors being deployed nearly everywhere, from cars to manufacturing facilities, are streaming massive amounts of data into the cloud for constant analysis.

Kafka is a popular platform for ingesting those data streams, including audio and video, Vogels said. But that open source technology is still very hard to use.

Amazon will make it easier with a new service Vogels introduced to re:Invent attendees: Amazon Managed Streaming for Kafka.

Through that service, the cloud leader will deliver fully managed Kafka clusters, replicated over multiple availability zones and secured with the full AWS portfolio of data-protection tools.

A Well-Architected World

Vogels has focused a lot of his energies in recent years on evangelizing Amazon's vision for optimal cloud application architecture.

To that end, Amazon has published best practices for developers to build resilient, highly available and secure applications.

To help developers better adhere to that prescriptive architecture, Vogels introduced in his re:Invent keynote the AWS Well-Architected Tool.

The new tool makes it easier for developers to access AWS knowledge resources when reviewing the architecture of their applications.

"It enables you to dive deep into what are really the best practices on AWS," Vogels said.