AWS 2021 re:Invent Recap
This year AWS re:Invent 2021 ran in-person from November 29th to December 3rd. As always, there was a torrent of new service announcements, existing service updates, and tons of additional AWS re:Invent content! We've had a bit of time to digest the entirety of the conference, so we thought we would discuss the highlights and things that interested us.
We asked our internal cloud thought leaders, AWS architects and engineers to give us their impressions on the events and point out what excited them the most in the announcements. Let's take a look at the new services, exciting new features added and our thoughts on the annual event.
Long Le, AWS Practice Manager & reluctant Funko Pop! collector on digital transformation:
My biggest takeaway from this year's Re:invent was Adam Selipsky's messaging around creating capabilities that allow for "transformations". Faster processors, better networking, new services are all great and that's the cycle we've all grown accustomed to after decades in IT. How we use those legos to deliver value and drive innovation is equally, if not more important, to our companies and customers. I want to expand our portfolio with those solutions in the coming year, to directly address problems or provide new innovations for industries we support. Moving beyond foundational and into the disruptive. Certain verticals are slow to adopt new technologies, for a myriad of reasons (mostly not having to do with tech), having worked in a few of them, I know cloud can change the way they work for the better.
One of the more interesting things I noticed at reinvent was AWS Fleetwise: a new IoT service designed to aggregate data from multiple vehicle types. Before Fleetwise, generating useful data from vehicle fleets was difficult. Each OEM has unique data encoding methodology, specific but similar sensor types and uses multiple protocols. If this wasn't complicated enough, determining what information to send is difficult as well.
AWS Fleetwise empowers OEMs and vehicle fleet management teams to easily collect and organize data from multiple vehicle types by modeling the vehicle in Fleetwise. During this step, the vehicle is defined (protocols, sensor types and locations). Next, AWS IoT Fleetwise edge agent is loaded on the vehicle's hardware. After the vehicle is talking to Fleetwise, the final step is to define 'what' is important. During this step the user defines a 'scheme'. A scheme defines what data to collect and why. (example: Collect sensor A for 500 ms if pressure B is over 100 kpa for 300 ms for example). Schemes can be simple or complex. After the scheme is defined, Fleetwise initiates the collection process by pushing down the scheme to the IoT agent on the vehicle. Any data that machines the scheme is sent to AWS Fleetwise.
As the internet of things evolves into the internet of everything, data is king. For data to be useful, it must be filtered, collected, aligned and analyzed. AWS Fleetwise empowers OEMs and vehicle fleet management teams to build a picture of a 'what' their fleet is doing; reducing disruption and driving understanding behind warrantee claims and unexpected failures. AWS Fleetwise is driving the future forward, today.
IP addresses can be hard to manage across more than a handful of VPCs in a single account, let alone a large multi-account AWS environment. AWS announced a new way to manage and audit network addresses at scale by introducing Amazon VPC IP Address Manager (IPAM). This service brings a centralized operational dashboard for IP addresses across all of your accounts, regions and VPCs. Integration with AWS Organizations allows you to share your IPAM address pools to member accounts within your organization.
Not only can you centrally manage your IP addresses, but IPAM allows customers to monitor and audit their network addresses. It keeps track of important information, such as routing, account, VPC, etc. and can alert you to critical errors like network over provisioning or overlapping addresses. CloudWatch can be used to proactively alert you about potential issues, rather than dealing with problems as things break.
IPAM can be deployed net new and VPCs can be configured to use network pools within IPAM. For existing VPCs, IPAM will proactively monitor your environment to bring in the current inventory. Once you create pools, IPAM will begin import your existing inventory based on the business rules for that pool. That means you can get started right away with auditing and monitoring for your current infrastructure, and can later configure your VPCs to use IPAM for automated provisioning should you choose to do so.
AWS announced Amazon MSK Serverless in public preview during AWS re:Invent. This adds and additional type of Amazon MSK cluster that provides the ability to deploy Apache Kafka without having to manage its capacity. MSK Serverless automatically provisions and scales compute and storage resources.
With a few clicks in the AWS management console, you can setup secure and highly available clusters that automatically scale as your application I/O scales. MSK serverless is fully compatible with Apache Kafka, so you can run existing applications without having to create new apps to migrate to or perform any code changes. MSK Serverless supports native AWS integrations that provide capabilities such as private connectivity with PrivateLink, secure access with IAM, performance metrics analysis with CloudWatch Metrics, and schema evolution control with Glue Schema Registry.
With Serverless MSK being in preview the current Cluster throughput is limited to 200 MiB/second write and 400MiB/second read. Cluster storage is set to 250 GiB per partition with a limit of 120 partitions.
MSK Serverless on demand pricing is based around Cluster Hours, Partition Hours, Storage, Data In and Data Out.
The continuing evolution of Serverless solutions in AWS is critical to enabling business to focus more on the important things and less about managing infrastructure.
AWS Control Tower Account Factory for Terraform (AFT) is a Terraform module that makes it easy to create and customize new accounts that comply with your organization's security guidelines. AFT defines a pipeline for automated and consistent creation of AWS Control Tower accounts, giving you the benefits of Terraform's workflow and Control Tower's governance features. AWS maintains this module.
What is the basic architecture?
AFT provides you with a single Terraform infrastructure as code (IaC) pipeline to provision an AWS Control Tower managed account that helps meet your business and security policies before handing the accounts to end-users. AFT provides automated account creation that includes monitoring when account provisioning is complete and then triggers additional Terraform modules to enhance the account with any customizations necessary as a part of the same IaC pipeline. As part of the customization process, you can configure the pipeline to install your own custom Terraform modules, or you can choose to use one or more of the AFT Feature Options that are AWS provided options for common customizations.
- Terraform v0.15+ (Open Source - OSS, Terraform Cloud - TFC or Terraform Enterprise - TFE)
- an AWS account with AWS Control Tower enabled
- the AWS CLI
- a Github account
What are the main benefits?
AWS Control Tower deployment and provisioning an AWS account is a relatively easy task, but fine-tuning these items for a new AWS Account is a highly daunting task for customers who are new to Cloud
- Security aspects like enabling Guard Duty, IAM Access Analyzer, Security Hub and much more
- Creating Networking constructs like VPCs, Subnets, AZs, Security groups, NALCs, IGW, Nat Gateway, TGW attachments, VPC Flow Log collection and more
- Enabling Monitoring and Alerting (native or 3rd party) tools
- Enforcing custom CFTs or TF modules
AFT provides out-of-the-box Terraform modules for customers to apply on new AWS accounts to automate the customizations in a most consistent and reliable fashion.
When do you need AFT?
AFT helps you to automate the customization process (Organization-wide or Account Specific) by leveraging Terraform Modules as well as AWS native services like AWS Steps Functions, Lambda, SNS, SQS, DDB, CodeBuild, CodeCommit, and CodePipeline (In addition to traditional services like CT, Service Catalog Account Factory, CFTs, AWS Org and etc.
Exceptions: If you provision new AWS Accounts without AWS Control Tower, this solution will not work.
AWS Cloud WAN and AWS Direct Connect SiteLink were easily the two standout AWS infrastructure darlings of re:Invent 2021. AWS customers have long lamented about the struggles related to connectivity among VPC's, data centers and on-premises networks in addition to associated monitoring and visibility of each. Incremental solutions to these challenges have been introduced over the years such as AWS Site-to-Site VPN, VPC peering, Direct Connect, Direct Connect Gateway and Transit Gateway. While each of these has worked well to address a seemingly smaller scale or more narrowly scoped networking challenge, none have quite managed to address the holistic, all-encompassing network connectivity management, monitoring and visibility needs that customers have been hoping for. AWS may finally managed to have cracked that nut with the preview release of AWS Cloud WAN which enables customers to easily build, manage and monitor global wide area networks within a single service offering. While Transit Gateways provide a regional connectivity solution (and a "not quite there yet" native inter-region capability), AWS Cloud WAN attempts to solve this and more on a global scale with far less fuss. Customers may want to consider evaluating this solution as part of their enterprise WAN architecture as a primary, secondary or replacement backbone WAN.
AWS Direct Connect SiteLink on the other hand may form the basis of one's data center WAN as it is able to interlink supported AWS Direct Connect locations without routing through AWS regions. For instance if I'm an Equinix customer with a presence in both their US East and US West colo facilities, I may simply want to use their Equinix Fabric service to link my two regions. Though if I merge with an organization that is a CoreSite customer, it might behoove me to link all my Equinix and CoreSite facilities through a single AWS Direct Connect SiteLink cloud (which is based on AWS Direct Connect Gateway technology). While this is still completely possible to accomplish by linking various Direct Connect locations through one or more Transit Gateways and Transit VPC's designs, SiteLink allows us to completely bypass any latency and data transfer charges associated with routing our traffic through regions, TGW's and VPC's. Example workloads might include traffic from VoIP, VDI, data migrations and data backups. While neither the AWS Cloud WAN nor AWS Direct Connect SiteLink solutions are intended to replace one's existing legacy WAN or SD-WAN solutions entirely by themselves (as these are not "last-mile" technologies), they do appear to offer an intriguing alternative as a backbone connectivity and network operating model
The Network Access Analyzer can help our customers identify configurations in their network that may lead to unintended network access. It is designed to help point out ways to improve security posture in an agile and flexible manner. We all agree that manual efforts are prone to error so the fact that the Network Access Analyzer is an automated tool makes it very cool. Customers can analyze their networks regardless of size and complexity. It examines a wide range of resources including VPC endpoints, security groups, EC2 instances, subnets, TGW, NGW, peering connections and a lot more. I like the fact that the findings are based on your own configurations and evaluations are against a set of scope that you determine. Organizations and their security teams can take advantage of this great addition to the list of network security tools like the VPC flow log.
For past Re:Invents, I could always be found attending the security sessions. This year, I took a different approach, hoping to get a better perspective on other services offered by AWS. I ended up in a session that focused on AWS Connect and improving the experiences for Contact Centers. A little background on me, I was part of the initial VOIP deployment for a major cable company, and I have worked on Cisco's Contact Center solutions, so I came into the AWS Connect with a good understand of traditional call centers.
What I discovered was a true digital transformation. AWS's cloud-based Contact centers has taken the challenges of the traditional contact centers. The main goal is to improve the user experience, and AWS has done this by integrating AWS services like Lex and Polly, to improve speech quality through machine learning and personalize a contacts experience.
AWS AI/ML solutions boost a customers engagement while expanding the conversational experiences for interactive voice recognition. This also aids the agents in organizations to understand the customers intent and thereby improving the interactions between the contact and the agent.
Building out solutions using AWS AI/ML once again shows the power of DevOps and how it is changing the game for organizations and their customers.
RDS Custom for SQL Server makes the benefits and usage of RDS open to many more scenarios now. Not only can CLR, elevated permissions, and custom drivers be included but the previous 100 database limit is no longer an issue. These restrictions have previously been a stumbling block to utilizing RDS for non-refactored legacy applications. Having these options available will now remove another roadblock to allow 'lift and shift' of these workloads so even these older applications can glean the benefits of RDS.
Keeping with the database theme, DynamoDB Standard-IA will allow developers to keep data a millisecond away while not burning through the budget for normal storage costs. Once the code is updated to store the appropriate data to the IA tables there's no need to worry about pulling in 'archived' data from S3/other and rehydrating. This should simplify development and operations by keeping the data housed where it can be immediately used.
Margo Beck, Sr. Consultant and conference neophyte on the re:Invent experience and community inclusivity:
AWS Re:Invent 2021 was my first ever convention. I had been working with our AWS Cloud team for about 7 months on an AWS MAP program drowning in unidentifiable acronyms and just trying to keep up. When a coworker pushed me to ask about going to Re:Invent I initially just laughed. Me? A consultant who barely understands what an S3 bucket is? Who in their right mind would send someone like me to a 60,000 person master class in all things AWS? Not only did I have concerns around my technical prowess, thoughts of being in the minority in others ways started to make me anxious. Being a Women in Tech has always and inevitably put me in countless rooms dominated by men, meaning I am not a stranger to advocating for myself, my thoughts, and my leadership. However, I was concerned that lacking an AWS background made me a stranger among men and women alike.
Upon stepping into the Venetian on Tuesday November 30th I was immediately overwhelmed by all of the sights, sounds, and smells that was Vegas. As I made my way through the conference my cellphone began to ring and it was Chris Williams, the co-worker and friend I was meant to meet up with. Chris is what is known in the AWS community as an AWS Hero, and until last week I had absolutely zero idea what that meant. To me, Chris was someone that came to work, did AWS magic, was super nice, helpful, and a genuinely good person. But to my surprise, being an AWS Hero is a lot more than just being really smart. Chris immediately introduced me to dozens of Heroes, friends, and coworkers alike. It was an experience I assume akin to your wedding day, being shepherded around and saying hi to people you may or may not actually know. Through the droves of introductions and conversations, not one person was interested in seeing my AWS qualifications or asked me what I was doing at the convention. The environment was immediately collaborative, inclusive, and clearly everyone there was just happy to be there.
As the days continued on, hanging around the Heroes lounge, I found myself bewildered by the amount of genuine connections happening between Heroes and AWS enthusiasts alike. The AWS Heroes lounge is an area at the convention that allows anyone to stop by and ask the Heroes anything on their mind. Chris and another Hero Calvin Hendrix-Parker set up a "person on the street" interview spot where they would invite passerby's to talk about what they were most excited to learn at the convention. As I got my sea legs underneath me I began wandering around the endless maze of booths picking up t-shirts, pens, and other paraphernalia from the mass of vendors. I found myself stopped in my tracks by an Aviatrix demo of their newly released Threat IQ product. I was so engrossed by the presentation I didn't even realize it had ended until they started their promotional raffle. While I am not an expert in all things Cloud I consider myself an avid learner and someone who enjoys the organized chaos that is IT. AWS Re:Invent reminded me of just how cool being a nerd is.
After completing my "person on the street" interview I felt the buzz of my phone yet again. Upon checking, it was my customer asking where I was located. Initially, a sense of dread fell over me wondering if I had missed a meeting by accident. I promptly responded and was quickly met by a tap on the shoulder. My customer was also attending the conference and was interested in meeting up! After coercing him into completing a "person on the street" interview we broke off from the group and spent an hour floating around the vendor booths chatting about user groups, work, future work, and which vendors we thought had the most interesting things to offer this year. It was such a surprising and humanizing experience meeting with my customer in a face to face way that had absolutely nothing to do with our upcoming deliverables. Being a big AWS advocate himself, my customer promptly walked me to the AWS community booth and watched as I completed my enrollment into the Denver AWS User Group on meetup. He handed me a "She Builds" AWS sticker (which now sits proudly on the back of my WWT laptop) and we said our goodbyes. In the lobby of the Venetian, my customer and I traded thoughts on privilege at these types of events and how he makes an active effort to ensure he is not taking up more space than necessary. It was a refreshing take from a customer especially who are sometimes known to be very focused on their own needs. It was evident that my customer supported me and my Cloud journey, something that gave me a sense of relief knowing my customer could run circles around me in regards to my AWS knowledge.
Between all of the handshakes, salutations, cab rides, and chaos at Re:Invent I found myself astounded by the inclusive and supportive nature of the community. On my last day, I was introduced to some of the most prolific female AWS Heroes, engineers, and community builders finding myself fitting in with ease. AWS Re:Invent is not only about obtaining certifications and flexing your technical prowess, it is truly a meeting of minds who are interested in learning, building community, and supporting one another. I was impressed and taken aback by how well set-up the event was, making me feel safe, comfortable, and free to express myself and my interests. AWS Re:Invent is definitely a special conference and one I hope to return to year after year. Being a Women in Tech is an awesome accolade and AWS Re:Invent is an environment that supports folks like me by encouraging safe, respectful, and thoughtful collaboration.
Chris Williams, Enterprise Consultant, AWS Hero, and eternal gadfly on the AWS Well-Architected Framework Sustainability Pillar:
The AWS Well-Architected Framework is basically a decision matrix that enables one to create architectures that take several "Pillars" of knowledge into consideration. Historically these have been:
- Operational Excellence
- Performance Efficiency
- Cost Optimization
I'm very excited that AWS has added a sixth pillar to the framework: Sustainability!
This means that going forward, AWS considers sustainability to be as important as the other 5 pillars. This is a huge step forward in getting AWS customers to think about consuming less energy in AWS datacenters by improving app design, storage, code efficiency and utilization:
Coincidentally, all of these measures will also lower a customers monthly bill!
Another announcement made during Dr. Vogel's keynote was around a newly announce AWS Customer Carbon Footprint Tool. There isn't any good information on this yet so I won't speculate as to how comprehensive this new tool will be… but I have many questions already 😊
I'm excited to use these new tools with our existing customers to do my part to both lower their bills and help them reach their sustainability goals!
One other thing that I really enjoyed was The Hallway Track! Meeting folks I'd only seen online, discussing new services, & sharing ideas is the best part of the conference for me. I like it so much I recorded several interviews!
The Battle of the Belt Award goes to…
The WWT AWS cloud practice has a fun internal competition each year. Each employee that sits and passes a certification gets a number of points that are tallied during re:Invent and a winner is announced. This year the winner is Roger White who sat and passed:
- ACE Aviatrix Professional
- CHIP Terraform HashiCorp
- AWS Solutions Architect Professional
- AWS Database Specialty
- HashiCorp Vault
- AWS Ambassador
Congratulations Roger, fantastic job!