How to Implement Continuous Optimization with AWS

With organizations growing and evolving so quickly, continuous optimization (CO) is very important to implement. There are many ways to accomplish this, but carrying it out in practice can be tricky. There are four phases to continuous optimization: visibility, awareness, management, and the actual optimization. Let’s dive into how you can do this with AWS.

This phase is all about collecting data. This would include monitoring your existing architecture, workloads, and resources. This might involve using services like CloudWatch, Security Hub, QuickSight, and Cost Explorer. You can utilize service features like S3 Lens to get some deeper visibility into your bucket usage as well.

CloudWatch can be used to monitor your resource usage. View how each resource is being utilized. Are there random spikes in usage? Is there a schedule that the resource usage follows? Do you see too many under-utilized resources? This is the phase where you take note of all of these issues.

(Not familiar with CloudWatch? Check out this hands-on lab).

AWS Security Hub gathers information from several other security services like AWS Firewall Manager, Amazon GuardDuty, Amazon Inspector, and AWS Config. When you want a view of your environment's security stance, this is your go-to service. Look for any findings that need to be addressed. AWS Security Hub assigns a Low, Medium, or High status to each finding to help you prioritize those findings. 

In this part of optimization, you should be looking at your findings and getting a good game plan in mind. Plan to address these findings and find who all needs to be included in correcting them. I've written a more in-depth article on Security Hub you can read.

Amazon QuickSight is a business intelligence service. Use it for a way to visualize all the data from your environment and resources including data from services like Amazon S3, Amazon Redshift, and Amazon Athena. It can be used to visualize everything from costs to usage. Amazon QuickSight also has some pretty amazing features like shareable dashboards. You can create a dashboard and provide a URL that can be shared AND embedded, in case you want to implement it into an application. 

Amazon QuickSight Q allows you to ask natural language questions about your data and receive your answers WITH visualizations. Amazon QuickSight Paginated Reports can help you create, schedule, and share reports as well as data exports. For this phase in optimization, use Amazon QuickSight to pull together your data and visualize it with a dashboard. Use the dashboards to help get the information to the right people. For more details, read our QuickSight explainer article.

AWS Cost Explorer is a cloud financial management service. Use it to visualize, understand, forecast, and manage your costs and usage over time. AWS Cost Explorer can be used to produce cost reports by service, account, specified time, and savings plan or reservation. Reports can be customized and shared as well. 

Get as granular as you need to in these reports and make sure that the information you are pulling together matches up with what needs to be included. Get on the same page with the right group of people who most need to view the reports. I've done a deep dive course on AWS Cost Optimization which you can check out, which includes eight hands-on labs.)

Now that we've seen where we are with our existing architecture, workloads, and resources, we are ready to spread awareness. Take all the information we gathered and put it in the hands of those who make the decisions. You can also take note of how many services are being used that are managed services. This will help you tackle the next step easier.

This is also when we visit our old buddy Trusted Advisor. Trusted Advisor can help us see where the architecture is not meeting best practices. It gathers information from other services to produce its findings. These findings span across several categories: cost optimization, performance, security, fault tolerance, and service limits. Those findings are also prioritized by action recommended, investigation recommended, and checks with excluded items. Take those findings and pass them along to the correct teams. 

Our next step is taking action. All of those under-utilized resources need to be axed or downsized. Snapshots should be cleaned out. Unused elastic IPs should be released. Schedules of usage need to be implemented. Autoscaling should be put in place for your instances. There are several services like Amazon RDS that will autoscale automatically as it is a managed service. Utilizing managed services where you can will help you curb your costs ahead of time and save your team effort in the long run.

During this phase, holes in security need to be patched and that might include implementing a new service or using AWS Systems Manager Patch Manager to get your instances updated. This would also include updating your AMIs to make sure that as you move forward, any instances using those AMIs are NOT a security risk. Take those findings from Security Hub and Trusted Advisor and get them corrected.

If there is no budget alarm in place, get it in place and set it to alert the right folks. Taking control of your cloud budget is important to be able to quantify just how the organization is benefiting from using the cloud in general.

This is our "last" phase. You've got all the things corrected and now you are fine tuning existing architecture, workloads, and resources. Tighten up where you can by using services like AWS Compute Optimizer or get more granular on your initial visualizing.

Though this is a final phase, that does NOT mean you are done. Not by a long shot. The idea of optimization is a continuous process. Keep monitoring, visualizing, sharing, correcting, and start all over again as often as you can. As your organization evolves and grows, so too does your architecture. To quote the great Werner, "Evolve or die."

Implement architecture that can easily evolve and grow using event-driven architecture. This will help you move along the optimization process smoother and not hit walls of inability. 

Thank you so much for joining me today, Gurus. Remember: deep breaths, baby steps, and as always, keep being awesome!

Start building your cloud skills with these 10 fun hands-on projects to learn AWS.