Search Images Using Azure AI Search and Azure AI Vision

You should already be logged into the Azure portal, using the credentials provided with the lab. When you first log in to the Azure Portal, you will land on the overview page for the resource group already deployed for you. Note the location/region for the resource group and note the Azure Storage account already deployed for you, with a name that starts with, "labstorage...".

Using the Azure portal, create an Azure AI Search service with the following configuration:

• Create the service in the existing resource group, and in the same location as that resource group.
• Use any valid name you choose.
• Create the service in the Basic pricing tier. Do not choose the Free tier, as only one is allowed per subscription, and your lab environment is on a shared subscription with other students.

In this objective, you will complete the tasks required to import sample data and build an index on your search service, based on that data. The indexer that populates the index will also include an AI Vision set of skills in a skillset tag and caption images to make them searchable.

Retrieve and store sample data

• You should have already created an Azure AI Search service. Navigate to the newly deployed service, and on the "Overview" select the Resource Group to navigate back to the resource group that was set up for you and that was the landing page when you first logged into the lab.
• Navigate to the Azure Storage resource already created for you and navigate to the blob container called lab-container.
• In a separate browser window, navigate to the GitHub repository link provided in the "Additional Information and Resources" section of this lab.
• You should see two files in the AI_Vision folder. Download both of those files to your local environment, and then return to the storage account in the Azure portal and upload both files to the blob container called lab-container.
• Use the breadcrumb trail to return to the resource group Overview page. Navigate to the newly deployed Azure AI Search service.

Import data and configure index and skillset

On the "Overview" page for the search service, select "Import Data" to kick off the wizard. Let the wizard guide you through the process, including the following specific details and properties.

• Choose to import data from Azure Blob Storage. Set up the new Data Source with any name you prefer, include all content and metadata, and set the parsing mode to Default.
• Select the pre-existing storage account and lab-container. You will not need to specifically reference files in the container; data will be pulled from all valid sources of data in the selected blob storage container.
• Under Add cognitive skills, Attach AI Services, confirm that the AI Services Resource Name is "Free (Limited enrichments)." This setting determines the compute resource that will be used to power the AI enrichment(s) you select. The free option allows you to use non-billed resources instead of setting up and paying for compute and storage — on an Azure AI multi-service resource.
• Under Add Enrichments you should use the content field as your source data field. Do not use the default source data field. It must be set to content for the enrichment to work.
• Check the box to apply Enable OCR and merge all text into merged_content field. Even though OCR is not the focus of this lab, checking this box enables two more, which are the focus of this lab: Generate tags from images and Generate captions from images. Check both of those boxes.
• All other features on the Add cognitive skills tab can remain at their defaults.
• Under Customize target index, familiarize yourself with the fields extracted from the source data. For simplicity, make all fields Retrieveable and Searchable, including the newly added fields from the skillset.
• Move on to the Create an indexer and Submit to create the index, the skillset, and the indexer, which will kick off the first run of the indexer.
• When the indexer run is complete, navigate to the index populated by the indexer and note the number of JSON documents created and the storage size. There should be 2 documents, each containing the data for each image in the source data.

Tip: If the UI Indicates 0 Documents and 0 Bytes

If the UI in the index screen appears to indicate that there are no documents, make sure the indexer has completed running. However, there may also be a quirk in the UI. You can also perform a quick query by putting an asterisk (*) in the search bar and running the query. If the query returns documents, the UI just hasn't caught up with the underlying data statistics; you can proceed with the next objective.

Context: For this objective, you should already be on the index page for your search index, which defaults to the Search Explorer pane.

Browse the index data with Search Explorer

• Using the Search Explorer, browse the index to examine the output of the data source data — especially the imageTags and imageCaption fields

  Tips: To see both documents, simply execute a search with nothing or a * in the search bar. Use the bottom, horizontal scroll bar to view the simple caption generated for each image.

• Open the images on your desktop or in the GitHub user interface to compare the objects in the pictures to the tags and the captions. Keeping in mind that most search applications would return the image — and not necessarily the AI-generated tags and captions — think about what other AI-driven enhancements you could add to the skillset to make the data even more useful for Azure AI Search use cases.