Introducing: Azure Cognitive Search – Part 1

With this article series, you will learn what Azure Cognitive Search is, and how to use it. See the essential steps and make your application's data searchable, improve search performance and the quality of search results. As this is the first article of the series, it provides you with an introduction to Azure Cognitive Search and demonstrates how to create your first search service.

In diesem Artikel:

Many projects come with the requirement for some integrated search capability. With current JavaScript frameworks like Angular, it is very easy to search all application data directly on the client.

To achieve search capabilities on the client, all data must be transferred to the client from a server-side API. Especially when dealing with lots of data, this may lead to long-running operations. Additionally, handling a vast amount of data in the client’s memory may result in serious performance issues. All this happens at the expense of the user’s experience with the application.

Another approach to search data is to place the search logic within the remote API. In this case, the search operation is performed on the server, which obviously leads to better response times (servers can scale to perform faster), especially when there is a lot of data to be searched. However, executing search queries against a vast amount of data is compute-intensive, so pushing search capabilities to the server is just an intermediate solution. Extracting the search functionality and offloading it to a dedicated search service seems even more promising.

Search services are tools or frameworks that help you prepare your data for optimized searching and provide programmatic means to search this data efficiently. One of these services is Azure Cognitive Search. With Azure Cognitive Search Microsoft offers a search service that could be used to search different types of data with minimal effort.

Azure Cognitive Search as a Managed Search Service

Azure Cognitive Search is a hosted search service (SaaS), which is used to consolidate and search through heterogeneous content. To manage and interact with the service, any of the existing Azure management interfaces like Azure Portal, Azure REST API, Azure CLI, Azure PowerShell, and a wide variety SDKs can be used.

Azure Cognitive Search is using a subset of Apache Lucene for full-text search. When the search service receives search requests, the query parser deconstructs the search query into separate parts and builds a query tree. The query tree is sent to the underlying search engine, which gets the results from the search index and returns them to the client.

Extend Your Search

Azure Cognitive Search offers more than just a simple API to search through individual data. Using Azure Cognitive Search, you can create tailored search experiences and present correct search results to users in the given context. Well-known and proven additions like search suggestions are also exposed by the service and can improve the overall user-experience of your application. Some highlights are:

  • Artificial intelligence (AI) & Machine Learning (ML): With AI and ML you are able to extract information from unstructured data or images and documents. The cognitive skills will find the required information in any document and extract it for you.
  • Auto Complete & Suggestion Search: With those features the Azure Search Service offers you results for autocomplete or suggestions that are similar/close to the search term.
  • Scoring profiles: Boost the search results by manually adding some scoring profiles to affect the weight of your search results.
  • Hit highlighting: Highlight parts of search results that match the actual query to get users‘ attention at the right place.

In this article series, I will walk you through the basics of Azure Cognitive Search. However, knowing about the advanced capabilities is important. Microsoft offers great documentation on those features, which you may want to take your search to the next level.

Pricing

If you want to use Azure Cognitive Search, you have to choose a service tier. The service is available in different tiers (including but not limited to freebasic, and standard). The required tier depends on actual service capabilities and integration features to ensure seamless integration with pre-existing cloud infrastructures.

The service tier depends on multiple factors, just as:

  • Region: Dependent on the region where your Azure Service is hosted, the pricing will change.
  • Performance: Metrics like the maximum number of queries per second (QPS)
  • Capacity: Number of indexers or storage that is used for your indexes.
  • Elasticity: Dynamic scaling capabilities
  • Security: Private endpoints, IP firewall, Customer-managed encryption keys

Obviously, the free tier is great to get started and to determine the actual service capability requirements. However, due to its limitations and the missing service level agreement (SLA), you should never use a free plan for production workloads.

Consult the official documentation to find the proper tier for your requirements, and use the Azure pricing calculator to estimate service fees.

The Index as the Center of the Search Universe

Your data is reflected in an index that can be searched. The data will be prepared (indexed) for searching, so it can be queried as efficient as possible. The index only contains the fields that are available and necessary to be searched. Before you can use the index, data must be pushed to it. This can be done in different ways:

  • Usage of an API or SDK to feed the index with data that is not hosted on Azure (REST API, .NET SDK, Node SDK, …)
  • Autofill with an indexer that pushes data from various Azure data source (like Azure SQL databases or Azure Storage Accounts) into the index

When the index has been created and is holding data, the index can be queried in different ways:

Every query can be sent using different parameters like search term, count (indicates that the result should contain search result count), search result field list, or the order-by field. To prevent unauthorized access to your search service, every query has to be sent with an API key. You get a query API key using Azure Portal, Azure CLI, or any other management interface.

When querying Azure Cognitive Search using the REST interface, you will receive a JSON result with all your requested data. Here is a simple example that is optimized to increase readability:

				
					{
    @odata.context: "https://sj-search.search.windows.net/indexes('pokeapi-index')/$metadata#docs(*)",…}
    @odata.context: "https://sj-search.search.windows.net/indexes('pokeapi-index')/$metadata#docs(*)"
    @odata.count: 6
    @search.facets: {weight: [{count: 0, to: 0}, {count: 97, from: 0, to: 9999}, {count: 0, from: 9999}]}
    value: [
        {@search.score: 0.78965807, height: "22", id: "460", name: "abomasnow", weight: 1355,…}
        {@search.score: 0.78965807, height: "6", id: "591", name: "amoonguss", weight: 105,…}
        {@search.score: 1.2634529, height: "12", id: "153", name: "bayleef", weight: 158,…}
        {@search.score: 1.2634529, height: "4", id: "182", name: "bellossom", weight: 58,…}
        {@search.score: 0.78965807, height: "7", id: "69", name: "bellsprout", weight: 40,…}
        {@search.score: 1.2634529, height: "3", id: "761", name: "bounsweet", weight: 32,…}
    ]
				
			

Search result example

Important to know

Azure Cognitive Search Services are only available on Azure and not available for on-prem installations.

Demo Application

To demonstrate Azure Cognitive Search’s integration into a custom application, I created a small demo application. In addition to Azure Cognitive Search, other, fundamental Azure Services are used to create this end-to-end example. As a data source, I use the PokeApi to index and search Pokemon characters.

A sample search query with its corresponding result can be seen here:

Instead of implementing a full backend API using .NET, I used Azure Functions to do all the work I need for my purposes. It prepares all the data that has to be indexed, creates the index itself, and creates and runs an indexer to feed the index with data from the PokeApi. To display search results, I will use the preview site that can be created by Azure itself. It uses the REST API to query my search index and show the results of it.

As seen in the picture above, the preview can be created in the Azure Portal in your search service. All the code for this article series, including the client page, is available on GitHub.

Before we Start: Create a Search Service

Before we can continue working with Azure Cognitive Search, we need to create a Search Service in Azure. There are different ways to create it. You can create it in the Azure Portal, or you can create it with the Azure CLI. I will create the Azure Search Service with the CLI. You should take a look at the official installation instructions, if you haven’t installed Azure CLI yet.

Once Azure CLI is installed on your system, open a terminal, and use the following commands to authenticate and to create your Azure Cognitive Search instance:

				
					# login to Azure
az login

# list and select a proper Azure Subscription
az account list -o table
az account set --subscription <SUBSCRIPTION_ID>

# create an Azure Resource Group
az group create --name rg-poke-search \
    --location westeurope

# create the Azure Cognitive Search
az search service create --name poke-search \
    --resource-group rg-poke-search \
    --location westeurope \
    --sku PricingTierName
				
			

Azure CLI will present some basic information about the newly created Azure Cognitive Search service once provisioning has finished. For demonstration purposes, I only specified the most important arguments as part of az search service create. You can ask Azure CLI for all available arguments using az search service create --help.

Upcoming

In the next article of this series, you will see the main part of the search service: the index. You will not only learn how to create and modify it but also see different ways of configuration.

Mehr Artikel zu Azure, Cloud Native
Kostenloser
Newsletter

Aktuelle Artikel, Screencasts, Webinare und Interviews unserer Experten für Sie

Verpassen Sie keine Inhalte zu Angular, .NET Core, Blazor, Azure und Kubernetes und melden Sie sich zu unserem kostenlosen monatlichen Dev-Newsletter an.

Newsletter Anmeldung
Diese Artikel könnten Sie interessieren
Cloud Native
favicon

Cloud Security: The 4C’s Of Cloud-Native Security – Part 5

In the last part of the article series on the 4C's of cloud-native security, we will look at securing the cloud environment itself. Although actual steps to harden your cloud infrastructure differ based on the cloud vendor and the services used to architecture your cloud-native application, many concepts and methodologies are quite similar.
14.04.2022
Cloud Native
favicon

Code Security: The 4C’s Of Cloud-Native Security – Part 4

In this part of the article series on the 4C's of cloud-native security, we will take a closer look at code security. We will briefly inspect why code security is essential, why it should be addressed from the beginning, and why trends such as shift-left security are important aspects of overall security considerations.
18.03.2022
Cloud Native
favicon

Cluster Security: The 4C’s Of Cloud-Native Security – Part 3

Securing the Kubernetes cluster (which may act as a runtime platform for several components of typical cloud-native applications) addresses one of the 4C's in cloud-native security. If you haven't heard about the 4C's of cloud-native security yet or want a quick refresher, you should read my corresponding introduction article.
04.03.2022
Cloud Native
favicon

Container Security: The 4C’s Of Cloud-Native Security – Part 2

Securing the container images of your cloud-native application building blocks addresses one of the 4C's in cloud-native security. If you haven't heard about the 4C's of cloud-native security yet or want a quick refresher, you should read my corresponding introduction post.
24.02.2022
Cloud Native
favicon

Code, Container, Cluster, Cloud: The 4C’s Of Cloud-Native Security – Part 1

Containers and cloud-native design patterns gained popularity over the past years. More and more organizations use containers in production and adopt cloud-native practices and methodologies to get even more value from existing containerized applications and underlying technologies such as container orchestrators like Kubernetes.
15.02.2022
Cloud Native
favicon

Ausführen von Containern in Azure Container Instances (ACI) direkt über die Docker CLI

Vor einigen Monaten haben sowohl Microsoft als auch Docker eine nahtlose Integration von Azure Container Instances (ACI) und Docker CLI angekündigt. Als Container-Enthusiast musste ich mich natürlich mit dieser Integration befassen. Ich habe es in verschiedenen Szenarien getestet, um zu sehen, wie es mich unterstützen und meine Produktivität steigern könnte.
05.11.2021