The Rise of AWS – How Amazon Tops the Game of Cloud Computing

Many people love watching their favorite shows on Netflix on long car rides, but a few of them know the behind powerful mechanism that allows this happen when you can stream anywhere, whenever you want. It’s Amazon Web Services (AWS), the most profitable product of Amazon.

Whether it’s watching a Prime, Netflix, Hulu, or other movies, the content is often streamed through Amazon Web Services. Notably, major events like the Super Bowl, Major League Baseball, NASCAR, and Formula One racing all rely on AWS for streaming.

Yet streaming is just a part of how AWS serves, it plays a pivotal role in much of the technology we use daily, from calling a Lyft and checking video doorbells. Even services like Intuit, used for tax preparation, run on AWS.

Amazon’s leaders has been growing AWS into the world’s most successful second act. However, little do people now that the driven point of the amazing take rooted from an internal struggle. let’s rewind a little.

From an Internal Dilemma to the Industry Troubleshooter

It was the late 1990s, when sales were soaring between 1997 and 1998. There were a bunch of different databases that grew with every new customer, product category, and market as Amazon expanded, that’s when the system started to struggle. The situation was chaotic, leading to significant redundancy.

These architectural limitations posed tangible business constraints for Amazon. This problem reached its peak when Amazon was hired by retailers like Walmart and Target, to set up their e-commerce platforms.

Looking at the existing monolithic architecture, Jeff Bezos decided that was the right time to embrace SOA (Service-Oriented Architecture). One thing to understand is that when Bezos mentioned SOA, he was referring to APIs (Application Programming Interfaces).

Jeff Bezos created a team within Amazon whose sole purpose was creating them to share with third-party developers. This directly resulted in the integration of APIs into merchant.com.

This not only made developing Merchant.com smoother but also benefited internal developers. It set the stage for a more organized and disciplined way of creating tools internally in the future.

“We expected all the teams internally from that point on to build in a decoupled, API-access fashion, and then all of the internal teams inside of Amazon expected to be able to consume their peer internal development team services in that way. So very quietly around 2000, we became a services company with really no fanfare,” Andy Jassy said.

At the time, Amazon was focused on rapid growth, and that led them to counter new challenges. Dive into this example, Amazon wanted to start selling clothes, but clothes aren’t the same as books.

While books are almost perfect commodities, clothes aren’t. They come in different sizes and colors. In order to sell clothes, Amazon’s engineers, people like Allan Vermeulen, had to build all kinds of new features.

A bigger product catalog and more website traffic meant amazon.com required additional community features such as username verification and review ratings. Unfortunately, these features weren’t implemented as rapidly as needed.

When Jassy, who was the chief of staff for Amazon CEO Jeff Bezos, looked into the problem, he found a common issue. The executive team expected a project to be done in three months, but it ended up taking the entire three months just to build the database, compute, or storage part.

Andy realized that the reason projects were taking so long was that software engineers’ time was constantly being consumed by doing work that wasn’t directly related to the feature they were building.

Rather than focusing on essential tasks, they were dedicating as much as two-thirds of their time to constructing databases, storage, and compute infrastructure. It’s easy to see that this was a big problem. Amazon needed a solution. If it didn’t find one, it risked squandering the advantage it worked so hard to build. And it came in a classic memo.

Amazon and Problem Hitters – The Birth of AWS

The author of the memo was a guy named Matt Round, recognized the setbacks, he set out some principles for maximizing the amount of time that Amazon’s Engineers spent building cool things, maximizing their autonomy, adoption of rest architectures, standardization of infrastructure, removal of bureaucracy, and continuous deployment.

Matt’s memo was likely discussed during an offsite meeting at Jeff Bezos’s house in Medina, where Amazon’s senior leadership team gathered. All the company’s biggest hitters were there: Jeff Bezos, Andy Jassy, Alan Vermeulen, Rick Delzell, and others.

That’s when the people in the room began to be mindful of a key point. The Distributed Computing Manifesto, Amazon’s migration to SOA, Amazon launching web services in 2002, and third-party developers building more than 100 applications within 2 years—Matt Round’s memo— were not separate endeavors. Instead, they were all on the same road.

Amazon had started providing basic software tools with Amazon Web Services 1.0 while building the digital infrastructure for its own needs. The idea arose: maybe it was time to share this expertise externally, helping others improve their products.

The idea for AWS didn’t fully form during the offsite in 2003, but it was the driving force behind the six-page vision statement for AWS later on. Andy realized that Amazon should build and sell digital infrastructure to third parties. He knew that if Amazon didn’t take the leap, the company would miss out on a massive market.

To shape his perspective, he gathered a tiger team comprising the top 10 technical minds at Amazon. They dissected major web services like Google and eBay, contemplating Amazon’s principles, target audience, competitors, and a vision emerged.

The team pinpointed a set of primitives—fundamental, highly adaptable services like storage, compute, databases, and a content distribution network. These served as the building blocks for what eventually became AWS.

One thing has to be said that AWS just didn’t happen without Jeff Bezos and Andy Jassy. But they didn’t invent the concept of AWS.

The reality is that it wasn’t the effort of a single person. It was the result of countless memos, numerous meetings, and countless random conversations involving dozens, if not hundreds, of individuals to create the first core component of AWS.

The Cure – Ease the Burden of the Ecommerce Arena

Today, AWS is a set of so many distinct cloud computing services and products. Servers, storage, networking, email, mobile development, remote computing, and security are all offered by the lucrative Amazon division.

AWS can be divided into 3 key products: EC2, Amazon’s virtual machine service, Glacier, an affordable cloud storage service, and S3, Amazon’s storage system. AWS is so powerful and prominent in the world of computing that it’s much beat out its rivals.

And even now, AWS still maintains lead in the global cloud market with 32 percent in the second quarter of 2023. So, the question here is, what makes AWS stands out? We can find the answers in the subheadings below.

Create Opportunities – Eat your Own Dog Food

If you’re starting a new industry, being a pioneer is really helpful. If your business becomes successful, it’s likely that you’ll create new opportunities and challenges for others. You’ll also need to solve new problems just to keep your main business running smoothly.

“You have to consider the relationship between AWS and Amazon the e-commerce company,” said Ed Anderson, distinguished VP analyst at Gartner — which has AWS as its clear leader in its latest Magic Quadrant for Cloud Infrastructure and Platform Services.

Just like current Google Cloud customers aim to emulate the success of Google, early users of AWS sought to harness the technology that had propelled Amazon’s rapid growth into a major e-commerce player.

“A hallmark of AWS has been how technical and capable it has been,” Anderson notes. “And being really oriented around that ‘builder’ audience of developers, implementers, and architects,” he adds. “As a consequence, the sales team is very technical and capable in having those conversations, which means the experience customers have is really smooth.”

Customer Obsession – Understand what Customers Need

Before the release of S3, both Amazon and other businesses were facing high expenses for data storage. For businesses handling large amounts of data, the outdated business model presented a difficult choice: invest heavily in fixed server capacity up front or choose a less expensive solution that would limit future growth.

Data storage costs prevented many businesses from reaching their full potential or even starting in the first place. That was where S3 totally flipped things on its head.

All of a sudden, anyone could access data stored in the cloud and pay for the services they actually used. There were no large upfront charges associated with purchasing space in S3. Anyone could sign up using just their email address and credit card, and they could begin using the space the same day. According to Andy Jassy, the flexibility was tailored to the user’s requirements based on input from Amazon customers.

In addition to being highly cost-effective, the S3 pricing model was optimistic because when you offer a solution that lets people focus on their expertise or what they’re good at without worrying if the backend infrastructure is going to stop them from thriving.

And when they thrive, these winning entities are going to need more storage, and they will continue to spend more money on AWS.

In fact, AWS is wonderful for building a company from the ground up as it offers every asset necessary for a company to start up with the cloud. If your company is already up and running, Amazon offers reasonably priced migration services to make sure your current infrastructure can be seamlessly transferred to AWS.

As a company grows, AWS provides tools to help in expansion and as the business model allows flexible usage, clients won’t ever have to spend time contemplating whether or not they need to review their computing usage. Apart from budgetary factors, companies might genuinely “established and fail to remember” all their computing needs.

Never Rested on the Laurels – Day in Day out Executing

AWS kept rolling out new features ever since the first launch of AWS in 2006, and they just kept on going. In late 2008, they released CloudFront, a service that speeds up the delivery of web content, Virtual Private Cloud, a serverless database service, and a massive data warehouse platform.

S3 originally launched with eight microservices. When Amazon’s CTO Werner Vogel took to the stage to deliver the keynote address at the 2022 re:Invent conference, it had more than 235.

AWS remains operational during maintenance activities, and except for a service disruption in 2011, it has consistently maintained uptime. AWS customers don’t need to switch to new versions because the service undergoes continuous updates.

The crucial point here is that conceiving the idea of AWS was a remarkable achievement worthy of study. However, the ongoing day-to-day execution is what has empowered Amazon’s leaders to transform it into the most successful internal startup globally.

The Hardware to Protect Data in any Hostile Environment

Many companies are shifting from their own data centers to cloud providers like Amazon. However, transferring large amounts of data online is costly and time-consuming, especially with massive volumes like petabytes or yottabytes.

To address the challenges of cost and time, Amazon introduced robust hardware called Snowballs. Data center operators can link their infrastructure to Snowballs, make data copies, and then send these devices to AWS data centers, enabling faster and more efficient data migration.

One interesting thing about the Snowball is it even passed an explosives test and meets the military’s requirements for being airdropped. It’s also designed for the most extreme environments.

“They can operate at really high temperatures, like 140 degrees ambient temperature and really cold temperatures, like -20. And it can have unconditioned power from a generator, and it’ll continue to operate,” said Bill Vass, VP Engineering at Amazon Web Services.

While AWS was the initial public cloud provider to introduce hardware like this for data transfer, competitors have since developed similar products. However, AWS is the only company that felt like it needed to go even bigger. That’s why Snowmobile popped up.

AWS Snowmobile is a data transfer service designed to move very large amounts of data, at the Exabyte scale, to Amazon storage services.

According to Andy Jassy, the truck will help transferring data more quickly.

“”We’ll drive the truck up to your data center, providing power and network fiber connections. Once you fill ‘er up, the truck will return, the trailer will be reattached, and we’ll move it back to AWS. To put the idea of moving an exabyte of data into perspective, assigning a 10 gigabit per second line would take approximately 26 years. However, with 10 Snowmobiles, the process could be completed in just under six months,” he said.

Snowmobile employs Amazon’s Zero-G racks, which suspend the system from both the top and bottom of the truck to absorb impacts during transportation. Additionally, Snowmobile is equipped with its own power and cooling systems.

Once the transfer finishes, the truck goes into transport mode. An armed guard and an escort accompany the truck as it returns to Amazon’s data centers for the upload. Throughout the entire trip, satellite and cellphone communications are used to track its location.

Amazon keeps the Snowmobile truck unbranded for privacy. When it connects to the AWS center, data is decrypted, checked, and validated. After unloading and confirming everything is in order, the Snowball is ready for the next customer.