Introducing: Azure Cognitive Search – Part 1

With this article series, you will learn what Azure Cognitive Search is, and how to use it. See the essential steps and make your application's data searchable, improve search performance and the quality of search results. As this is the first article of the series, it provides you with an introduction to Azure Cognitive Search and demonstrates how to create your first search service.

In this article:

Many projects come with the requirement for some integrated search capability. With current JavaScript frameworks like Angular, it is very easy to search all application data directly on the client.

To achieve search capabilities on the client, all data must be transferred to the client from a server-side API. Especially when dealing with lots of data, this may lead to long-running operations. Additionally, handling a vast amount of data in the client’s memory may result in serious performance issues. All this happens at the expense of the user’s experience with the application.

Another approach to search data is to place the search logic within the remote API. In this case, the search operation is performed on the server, which obviously leads to better response times (servers can scale to perform faster), especially when there is a lot of data to be searched. However, executing search queries against a vast amount of data is compute-intensive, so pushing search capabilities to the server is just an intermediate solution. Extracting the search functionality and offloading it to a dedicated search service seems even more promising.

Search services are tools or frameworks that help you prepare your data for optimized searching and provide programmatic means to search this data efficiently. One of these services is Azure Cognitive Search. With Azure Cognitive Search Microsoft offers a search service that could be used to search different types of data with minimal effort.

Azure Cognitive Search as a Managed Search Service

Azure Cognitive Search is a hosted search service (SaaS), which is used to consolidate and search through heterogeneous content. To manage and interact with the service, any of the existing Azure management interfaces like Azure Portal, Azure REST API, Azure CLI, Azure PowerShell, and a wide variety SDKs can be used.

Azure Cognitive Search is using a subset of Apache Lucene for full-text search. When the search service receives search requests, the query parser deconstructs the search query into separate parts and builds a query tree. The query tree is sent to the underlying search engine, which gets the results from the search index and returns them to the client.

Extend Your Search

Azure Cognitive Search offers more than just a simple API to search through individual data. Using Azure Cognitive Search, you can create tailored search experiences and present correct search results to users in the given context. Well-known and proven additions like search suggestions are also exposed by the service and can improve the overall user-experience of your application. Some highlights are:

  • Artificial intelligence (AI) & Machine Learning (ML): With AI and ML you are able to extract information from unstructured data or images and documents. The cognitive skills will find the required information in any document and extract it for you.
  • Auto Complete & Suggestion Search: With those features the Azure Search Service offers you results for autocomplete or suggestions that are similar/close to the search term.
  • Scoring profiles: Boost the search results by manually adding some scoring profiles to affect the weight of your search results.
  • Hit highlighting: Highlight parts of search results that match the actual query to get users’ attention at the right place.

In this article series, I will walk you through the basics of Azure Cognitive Search. However, knowing about the advanced capabilities is important. Microsoft offers great documentation on those features, which you may want to take your search to the next level.

Pricing

If you want to use Azure Cognitive Search, you have to choose a service tier. The service is available in different tiers (including but not limited to freebasic, and standard). The required tier depends on actual service capabilities and integration features to ensure seamless integration with pre-existing cloud infrastructures.

The service tier depends on multiple factors, just as:

  • Region: Dependent on the region where your Azure Service is hosted, the pricing will change.
  • Performance: Metrics like the maximum number of queries per second (QPS)
  • Capacity: Number of indexers or storage that is used for your indexes.
  • Elasticity: Dynamic scaling capabilities
  • Security: Private endpoints, IP firewall, Customer-managed encryption keys

Obviously, the free tier is great to get started and to determine the actual service capability requirements. However, due to its limitations and the missing service level agreement (SLA), you should never use a free plan for production workloads.

Consult the official documentation to find the proper tier for your requirements, and use the Azure pricing calculator to estimate service fees.

The Index as the Center of the Search Universe

Your data is reflected in an index that can be searched. The data will be prepared (indexed) for searching, so it can be queried as efficient as possible. The index only contains the fields that are available and necessary to be searched. Before you can use the index, data must be pushed to it. This can be done in different ways:

  • Usage of an API or SDK to feed the index with data that is not hosted on Azure (REST API, .NET SDK, Node SDK, …)
  • Autofill with an indexer that pushes data from various Azure data source (like Azure SQL databases or Azure Storage Accounts) into the index

When the index has been created and is holding data, the index can be queried in different ways:

Every query can be sent using different parameters like search term, count (indicates that the result should contain search result count), search result field list, or the order-by field. To prevent unauthorized access to your search service, every query has to be sent with an API key. You get a query API key using Azure Portal, Azure CLI, or any other management interface.

When querying Azure Cognitive Search using the REST interface, you will receive a JSON result with all your requested data. Here is a simple example that is optimized to increase readability:

				
					{
    @odata.context: "https://sj-search.search.windows.net/indexes('pokeapi-index')/$metadata#docs(*)",…}
    @odata.context: "https://sj-search.search.windows.net/indexes('pokeapi-index')/$metadata#docs(*)"
    @odata.count: 6
    @search.facets: {weight: [{count: 0, to: 0}, {count: 97, from: 0, to: 9999}, {count: 0, from: 9999}]}
    value: [
        {@search.score: 0.78965807, height: "22", id: "460", name: "abomasnow", weight: 1355,…}
        {@search.score: 0.78965807, height: "6", id: "591", name: "amoonguss", weight: 105,…}
        {@search.score: 1.2634529, height: "12", id: "153", name: "bayleef", weight: 158,…}
        {@search.score: 1.2634529, height: "4", id: "182", name: "bellossom", weight: 58,…}
        {@search.score: 0.78965807, height: "7", id: "69", name: "bellsprout", weight: 40,…}
        {@search.score: 1.2634529, height: "3", id: "761", name: "bounsweet", weight: 32,…}
    ]
				
			

Search result example

Important to know

Azure Cognitive Search Services are only available on Azure and not available for on-prem installations.

Demo Application

To demonstrate Azure Cognitive Search’s integration into a custom application, I created a small demo application. In addition to Azure Cognitive Search, other, fundamental Azure Services are used to create this end-to-end example. As a data source, I use the PokeApi to index and search Pokemon characters.

A sample search query with its corresponding result can be seen here:

Instead of implementing a full backend API using .NET, I used Azure Functions to do all the work I need for my purposes. It prepares all the data that has to be indexed, creates the index itself, and creates and runs an indexer to feed the index with data from the PokeApi. To display search results, I will use the preview site that can be created by Azure itself. It uses the REST API to query my search index and show the results of it.

As seen in the picture above, the preview can be created in the Azure Portal in your search service. All the code for this article series, including the client page, is available on GitHub.

Before we Start: Create a Search Service

Before we can continue working with Azure Cognitive Search, we need to create a Search Service in Azure. There are different ways to create it. You can create it in the Azure Portal, or you can create it with the Azure CLI. I will create the Azure Search Service with the CLI. You should take a look at the official installation instructions, if you haven’t installed Azure CLI yet.

Once Azure CLI is installed on your system, open a terminal, and use the following commands to authenticate and to create your Azure Cognitive Search instance:

				
					# login to Azure
az login

# list and select a proper Azure Subscription
az account list -o table
az account set --subscription <SUBSCRIPTION_ID>

# create an Azure Resource Group
az group create --name rg-poke-search \
    --location westeurope

# create the Azure Cognitive Search
az search service create --name poke-search \
    --resource-group rg-poke-search \
    --location westeurope \
    --sku PricingTierName
				
			

Azure CLI will present some basic information about the newly created Azure Cognitive Search service once provisioning has finished. For demonstration purposes, I only specified the most important arguments as part of az search service create. You can ask Azure CLI for all available arguments using az search service create --help.

Upcoming

In the next article of this series, you will see the main part of the search service: the index. You will not only learn how to create and modify it but also see different ways of configuration.

More articles about Azure, Cloud Native
Free
Newsletter

Current articles, screencasts and interviews by our experts

Don’t miss any content on Angular, .NET Core, Blazor, Azure, and Kubernetes and sign up for our free monthly dev newsletter.

EN Newsletter Anmeldung (#7)
Related Articles
AI
sg
One of the more pragmatic ways to get going on the current AI hype, and to get some value out of it, is by leveraging semantic search. This is, in itself, a relatively simple concept: You have a bunch of documents and want to find the correct one based on a given query. The semantic part now allows you to find the correct document based on the meaning of its contents, in contrast to simply finding words or parts of words in it like we usually do with lexical search. In our last projects, we gathered some experience with search bots, and with this article, I'd love to share our insights with you.
17.05.2024
Angular
SL-rund
If you previously wanted to integrate view transitions into your Angular application, this was only possible in a very cumbersome way that needed a lot of detailed knowledge about Angular internals. Now, Angular 17 introduced a feature to integrate the View Transition API with the router. In this two-part series, we will look at how to leverage the feature for route transitions and how we could use it for single-page animations.
15.04.2024
.NET
KP-round
.NET 8 brings Native AOT to ASP.NET Core, but many frameworks and libraries rely on unbound reflection internally and thus cannot support this scenario yet. This is true for ORMs, too: EF Core and Dapper will only bring full support for Native AOT in later releases. In this post, we will implement a database access layer with Sessions using the Humble Object pattern to get a similar developer experience. We will use Npgsql as a plain ADO.NET provider targeting PostgreSQL.
15.11.2023