AWS re:Invent 2023: Data, generative AI, and human intelligence

We’re halfway through AWS re:Invent 2023, and the momentum hasn’t slowed down. This morning, Dr. Swami Sivasubramanian, AWS Vice President of Data and AI, took the keynote stage to speak about the relationship between data, generative AI, and human intelligence—and how to use all three together to get ahead. 

He dove deeper into some of yesterday’s announcements while sharing exciting new updates that will help orgs build generative AI applications more effectively. Keep reading to learn more.

We know AWS re:Invent can be overwhelming. That’s why we created a guide to the top sessions and speakers you’ll want to keep an eye on. 

• Data and generative AI technology fuel human intelligence
• The essentials of building a generative AI application
• Creating responsible AI-generated images with Titan Image Generator
• Additional Amazon Q integrations
• Reduce model training time with Amazon SageMaker Hyperpod
• The reskilling revolution: Soft skills in the age of generative AI
• More AWS data and AI announcements
• Preparing for the future of AI and cloud computing
• Get your hands cloudy with hands-on AWS labs

Dr. Sivasubramanian began by explaining the connection between data, generative AI, and human intelligence. He likened it to symbiotic relationships in nature with the example of whale sharks and remora fish.

The remora fish cleans the whale shark and keeps it healthy, and the whale shark, in turn, keeps the fish safe from predators. In the same way, data, generative AI, and humans have a mutually beneficial relationship.

“Generative AI…is fueling human intelligence and creativity,” said Dr. Sivasubramanian. 

Michael Cassidy, DevOps professional and Pluralsight Author, shared his perspective with me. “The synergy of a symbiotic relationship allows humans to leverage AI for various advanced applications, from accelerating the development of customized software solutions to improving critical areas like healthcare,” he said.

“AI's computational power and data processing capabilities complement human ingenuity, leading to groundbreaking advancements and efficiencies in diverse fields.”

Dr. Sivasubramanian laid out the four essentials of building generative AI applications: 

1. Access to a variety of foundation models (FMs)

2. A private environment to leverage your data

3. Easy-to-use tools to build and deploy applications

4. Purpose-built ML infrastructure

Data is also important. “A strong data foundation is critical to generative AI,” he said. That foundation includes a set of comprehensive, integrated, and governed services.

One of the most exciting announcements to come out of Dr. Sivasubramanian’s keynote was Amazon Titan Image Generator for Amazon Bedrock. You can generate and customize high-quality, realistic images using natural language prompts. Dr. Sivasubramanian did a quick walkthrough, and the customization options will make it a valuable asset for content creation.

Any image generated from Titan Image Generator also contains an invisible watermark to identify it as AI-generated content, reduce misinformation, and support responsible use of AI.

Amazon Q is a secure AI chatbot for business use that can be tailored to specific organizational data. Today, Dr. Sivasubramanian announced a few additional ways it will integrate with other Amazon services.

• Amazon Q Generative SQL in Amazon Redshift: Produce SQL query recommendations from plain English 

• Amazon Q data integration in AWS Glue: Simplify custom ETL jobs and integrate data using natural language

• Amazon Q in Quicksight: Generate stories from data and quickly pull out insights

“This really opens the door for so many different people to leverage their data warehouses and their data that’s stored within things like S3 buckets without having to be data experts,” shared Andru Estes, architect, engineer, and Pluralsight Author.

“I’m certain that the ability to leverage Q and the zero-ETL integrations they talked about in order to build data pipelines is going to be a game changer.”

Training foundation models can be challenging. You need to collect the data, create clusters, and distribute the model training. Then you need to checkpoint the model and remediate any issues. And if one cluster fails? The entire training process halts.

Amazon SageMaker HyperPod addresses these pain points by providing a purpose-built infrastructure for distributed training at scale. It can automatically check clusters, replace faulty nodes, and resume training from checkpoints. According to Dr. Sivasubramanian, it reduces model training time by up to 40%.

AI technology can, and will, enhance everything from data management to content creation and customer support. But due to the AI skills gap, leveraging AI technology effectively is a time-consuming and resource-intensive process for many organizations. 

One of the themes that emerged from Dr. Sivasubramanian’s keynote was the importance of making AI tools and technologies accessible to everyone—from AI and ML experts to novices new to the scene. 

The products and features he unveiled supported this. Depending on the tool and its use cases, AI engineers, developers, data analysts, and customer service reps alike would be able to use AI to streamline processes and boost productivity.

Because everyone will interact with AI, technical skills aren’t the only skills to develop. Dr. Sivasubramanian explained soft skills like creativity, ethics, and adaptability will become even more important with the emergence of AI.

Learn more about the importance of critical thinking and soft skills in the age of generative AI.

Dr. Sivasubramanian touched on a number of new products and services that will make it easier to manage your data, build generative AI applications, and deploy machine learning models.

• Amazon Titan Multimodal Embeddings: Easily build more accurate, relevant multimodal search results and recommendations for users

• Model Evaluation on Amazon Bedrock: Quickly evaluate, compare, and select the best foundation model for your use case

• Custom Model Program for Anthropic Claude: Work with AWS experts to customize Claude for your needs

• AWS Clean Rooms ML: Collaborate with partners to build, train, and deploy ML models without needing to share your entire dataset

• Amazon OpenSearch Service zero-ETL integration with Amazon S3: Seamlessly search, analyze, and visualize data in one place without creating an ETL pipeline

• Vector engine for Amazon OpenSearch Serverless: Store, update, and search billions of vector embeddings 

• Amazon Neptune Analytics: Store graph and vector data together with this analytics database engine for Amazon Neptune

Dr. Sivasubramanian’s keynote wasn’t the only exciting event on Wednesday. Several other experts led sessions about preparing for the future of AI. 

• Building a generative AI architecture: Francessca Vasquez, AWS Vice President of Professional Services, explained how to create architecture to scale generative AI securely, economically, and responsibly. Ultimately, the right architecture allows organizations to fully leverage the advantages of generative AI technology.

• Innovate faster with generative AI: The AI insights kept coming with a presentation by Dr. Bratin Saha, AWS Vice President of AI & ML. He spoke with customers and uncovered the strategies and approaches they’re using to transform their businesses with AI and ML.

• Future-proofing your applications with AWS databases: As the IT landscape rapidly evolves, preparing for the future is a key concern for most organizations. Jeff Carter, AWS Vice President of Databases & Migration Services, and Rahul Pathak, AWS Vice President of Relational Database Engines, explained how to design optimal databases for change, agility, and adaptability so you can adopt the next new tech, whether that’s generative AI or something else.

Dr. Sivasubramanian ended by reiterating the way data, generative AI, and humans can develop a symbiotic relationship. They'll only strengthen each other as time goes on, and they're all imperative to creating a flywheel of success and accelerating the AI journey.

Ready to turn your insights into action? Start a free trial and get hands-on experience in AWS with our labs: 

• Hands-on labs for learning AWS essentials

• AWS security essentials

• AWS developer essentials

Looking to develop cloud and AI skills in your organization? Learn more about the Pluralsight cloud transformation solution and resources to close the AI skills gap.