Bing Search API  

Introduction: This article is a continuation of the series of articles starting with the description of SignalR service. We followed up with a discussion of Azure Gateway service, Azure Private Link, and Azure Private Endpoint and the benefit of diverting traffic to the Azure Backbone network. Then we started reviewing a more public internet-facing service such as the Bing API.  

Description:  

Azure Bing API is developer-friendly and provides a robust way of searching custom and scoped content with the same standard as the public internet. A customized search instance can be created using the Bing custom search portal. Bing Search API maintains an index for the internet using web crawlers.  The Bing Search API is an Azure Resource available via the Azure Marketplace. After logging in to the Azure portal, one can select either the Bing API resource or the Bing Custom Search resource.  

In this article, we continue with the discussion on advanced features of Bing API. Bing supports deep learning models. Bing uses the Fire Tokenizer which is fast and enables sub-second response time even with the execution of complex data models. The tokenization is compatible with BERT algorithm. BERT just helps with transfer learning where knowledge gained from earlier training is used with novel cases not encountered during training. Transfer learning is a machine learning technique where a model trained on one task is re-purposed on a second related task. It only works in deep learning where the model features learned from the first task are general and the features are suitable to both the base and the target tasks. This form of transfer is also called inductive transfer and the scope for the allowed hypothesis is narrowed down in a beneficial way because the model is fit on a different but related task. When the data is sparse or expensive, this focus becomes especially efficient. Bert is quite popular for Natural language processing, but Bing’s Fire tokenizers allows customizations to models. The original BERT tokenization can handle say 0.4MB/second but this tokenizer can handle up to 15 MB/second. Some of the well-known tokenization algorithms include Moore Machine that finds token boundaries and piece boundaries for Word piece and Mealy machine maps a token into ID and an ID into token such that the ids go from 0 to Number of Tokens - 1

With a subscription key, the Bing Web Search API allows searching billions of documents for content that is relevant to the user’s search string with a simple HTTP GET method. The headers allow continuity across api-calls and allow the responses to be more refined based on the history of searches. Language is can also be specified in the header.   

One specific example of text analytics with Bing Search APIs is its application to track sentiments. Major brands recognize the importance of public sentiment. It has a dual purpose. How the search results show up a brand shows how the world perceives it.  And customers are recognized by brand based on their reputation.  Building trust and maintaining positive control over the brand’s reputation can be achieved with tools such as reputation management from the Bing Search toolkit.  A brand can choose to have these insights gained from the Bing news data provider which is quite large and representative for drawing analysis. News search can be pointed outward to track a competitor’s latest release or announcements. It can also be pointed at the brand name to populate an internal dashboard with new or trending news content regarding that brand. The Bing News Search API allows extracting headlines and excerpts from articles and return them in the response. Articles, social media mentions and opinion columns can be analyzed to detect their sentiments. The Text Analytics API available from Cognitive services can work with this datastream to emit a score between 0 and 100% to show negative or positive sentiments.  Tracking the sentiments help manage the perception.

Other kind of analysis are also possible. Quantitative Analytics on usages for the Bing Search API is provided by ‘enabling the Bing Statistics’ on the Bing Azure Resource via the Azure Portal. Predefined dashboards show analytics.  

Conclusion: These are the ways in which the Bing Search Web API can be used, and it is available as just another cloud resource along with the benefits that come with a cloud service.  

 

 

 

Bing Search API

Introduction: This article is a continuation of the series of articles starting with the description of SignalR service. We followed up with a discussion of Azure Gateway service, Azure Private Link, and Azure Private Endpoint and the benefit of diverting traffic to the Azure Backbone network. Then we started reviewing a more public internet-facing service such as the Bing API.

Description:

Azure Bing API is developer-friendly and provides a robust way of searching custom and scoped content with the same standard as the public internet. A customized search instance can be created using the Bing custom search portal. Bing Search API maintains an index for the internet using web crawlers.

The Bing Search API is an Azure Resource available via the Azure Marketplace. After logging in to the Azure portal, one can select either the Bing API resource or the Bing Custom Search resource.

With a subscription key, the Bing Web Search API allows searching billions of documents for content that is relevant to the user’s search string with a simple HTTP GET method. The headers allow continuity across api-calls and allow the responses to be more refined based on the history of searches. Language is can also be specified in the header. 

The query parameters include market to restrict the search results. It can additionally include safe search preferences. The text-decoration and text format can also be specified to enhance the search experience. Count and offset can enable paging of the results.

Bing provides a response filter that is independent of the REST-based parameters. The responseFilter can be used to specify entity types that are to be searched. For example, it can include comma-separated entities like webpages, news, images etc. Exclusions of entities can be specified with a minus sign in the response filter. This is not a substitute to call entity-specific search API if there is only one entity to be searched.

A request to the WebSearch API results in a SearchResponse object in the response body. This object encapsulates plenty of information in the form of ‘webPages’, ‘deepLinks’, malware notice, ‘contractual rules’, ‘images’, ‘videos’, ‘news’, ‘computation’, ‘timeZone’, ‘spellSuggestions’, ‘queryContext’, ‘places’, ‘entity’, ‘translations’, and  ‘rankingResponse’

Each search response includes a ranking response which groups content by mainline content and sidebar content.  Within each group, the items array lists the order for the content to be displayed. This lets us refer to the results via conventions like WebPages 0 through 9 or Entity 0. 

The search API cannot be called beyond the maximum number of queries per second otherwise the requests will fail with HTTP status code 429.

Analytics for the Bing Search API is provided by ‘enabling the Bing Statistics’ on the Bing Azure Resource via the Azure Portal. Predefined dashboards show analytics.

Conclusion: These are the ways in which the Bing Search Web API can be used, and it is available as just another cloud resource along with the benefits that come with a cloud service.

 

 Bing Search API

Introduction: This article is a continuation of the series of articles starting with the description of SignalR service. We followed up with a discussion of Azure Gateway service, Azure Private Link, and Azure Private Endpoint and the benefit of diverting traffic to Azure Backbone network. Then we started reviewing a more public internet-facing service such as the Bing API.

Description:

Azure Bing API is developer-friendly and provides a robust way of searching custom and scoped content with the same standard as the public internet. A customized search instance can be created using the Bing custom search portal. Bing Search API maintains an index for the internet using web crawlers.

The Bing Search API is an Azure Resource available via the Azure Marketplace. After logging in to the Azure portal, one can select either the Bing API resource or the Bing Custom Search resource.  The resource represents the service that handles the API requests for Web Search. It is created by specifying a name that contains alphanumeric characters and a hyphen. The subscription, pricing tier, resource group, and location are also required to create the resource. When the resource is provisioned, it can be called with a simple GET method to the https://api.bing.microsoft.com/v7.0/search API and the subscription key specified in the header with key Ocp-Apim-Subscription-Key.  The request headers also include a header for providing continuity of experience and another for user location in location-aware queries. The more information that can be provided to Bing, the better the search experience. The continuity behavior is introduced with the header for X-MSEdge-ClientID and this header is used for both the request and the response. Bing uses this header to provide users with consistent behavior across Bing API calls. New features and improvements are flighted with this header. The same value for this header must be specified in all requests across Bing API calls otherwise they won’t be in the same traffic flight. The clientID also helps with search history as the Bing API can improve query results with history. With the same client ID across multiple requests for the same end-user and device, the API can provide higher click-through rates via better quality of results from the Bing APIs. The only caveat is that this client ID must not be linkable to any authenticated user account information.

The response header includes the market used by the request. This is in the form of <languageCode>-<countryCode> as in en-US. It might be different from the request's mkt query parameter and the value for Accept-Language. BingAPIs-TraceID header holds the ID of the log entry that contains the details of the request.

The query-parameter is merely one parameter at the minimum. This parameter called q is set to the user's query string. The user's query string and all query parameter values must be URL encoded. Additional query parameters are optional and include the following: 1) count and offset to refer to the query results by page, 2) the mkt query parameter which is used to specify the market where the results come from, 3) safe search which is used to specify the user's safe search preference and 4) text decorations and text format which is used to turn on hit highlighting. 

 

 

Canned Canned Kusto queries and dashboards

 

Introduction:

The Azure portal allows a way to create resources, query them and display them on dashboards.  While the PowerShell automation is a great way to execute queries and leverage built-in cmdlets for Azure Resources, there are much more powerful tools with Kusto queries, Azure Resource graph explorer and dashboard elements. The portal offers several pre-written queries. This article talks about those canned queries and suggests similar treatment for dashboard panels.

Description:

Creating a dashboard for “all resources” view is as straightforward as pointing and displaying it on the Azure Portal. And the rendering of query results often involves one visualization chart that is most suitable for the results. For example, the aggregation results are best displayed by bar charts and the allocations are best depicted by pie charts. When the queries are rendered in pre-determined dashboard panels, it adds convenience to the user.

A list of queries and their corresponding dashboard panels are listed below:

1) Count Azure resources

Returns number of Azure resources that exist in the subscriptions that you have access to.

Bar chart

2) List resources sorted by name

Returns any type of resource, but only the name, type, and location properties.

List view

3) Show resources that contain storage

Returns any Azure resource that contains the word 'storage'.

Pie chart based with drill down

4) Show aliases for a virtual machine resource

Azure alias policies are used by Azure Policy to manage resource compliance. Show aliases for a virtual machine resource.

Prefix based trie view

5) Get all alerts on iot hub by type

Get all alerts on iot hub, filtered by alert type

Bar/block chart

Conclusion: Visualization of query results is an important task and one that is often thought again and again. Instead, it can be built-in and made it easy for the end-users.

 

 

Query and templates in Azure

 

Introduction. This is continuation of the article that describes Azure services. We review some of the querying, dashboard and templates conversions available from Azure.

Description. The Azure portal allows a way to create resources, query them and display them on dashboards.  While the PowerShell automation is a great way to execute queries and leverage built-in cmdlets for Azure Resources, there are much more powerful tools with Kusto queries, Azure Resource graph explorer and dashboard elements.

Creating a dashboard for “all resources” view is as straightforward as pointing and displaying it on the Azure Portal.  It would look something like this:

When the dashboard is shared, it can be exported as a template because it can be considered just another resource. Such a resource also can be expressed in the form of Azure Resource Manager Template. The above view is a list view but in the summary view, a map can be displayed with the resources and their locations.

One of the ways in which the Azure resources can be queried is in the form of a resource graph explorer.

A sample query for an Azure Resource Graph would look like this:

Resources

| where isnotnull(zones)

| order by type asc

Here the zones is a dynamic property and hence has restrictions to its usage but there are standard properties also available.

Finally, the Azure CLI offers another way to query resources but this time from a command line:

az graph query -q "Resources | where type =~ 'Microsoft.Storage/storageAccounts' | where tags['tag with a space']=='Custom value'"

These are some of the ways queries can be written and re-used to visualize via charts and graphs.

 

Sample dashboard as Azure resource:

{

    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",

    "contentVersion": "1.0.0.0",

    "parameters": {

        "dashboards_76e30d20_4a35_476f_b80f_799a0b0e8081_name": {

            "defaultValue": "76e30d20-4a35-476f-b80f-799a0b0e8081",

            "type": "String"

        }

    },

    "variables": {},

    "resources": [

        {

            "type": "Microsoft.Portal/dashboards",

            "apiVersion": "2020-09-01-preview",

            "name": "[parameters('dashboards_76e30d20_4a35_476f_b80f_799a0b0e8081_name')]",

            "location": "eastus2",

            "tags": {

                "hidden-title": "My Dashboard"

            },

            "properties": {

                "lenses": [

                    {

                        "order": 0,

                        "parts": [

                            {

                                "position": {

                                    "x": 0,

                                    "y": 0,

                                    "rowSpan": 4,

                                    "colSpan": 6

                                },

                                "metadata": {

                                    "inputs": [

                                        {},

                                        {},

                                        {},

                                        {}

                                    ],

                                    "type": "Extension/HubsExtension/PartType/BrowseAllResourcesPinnedPart"

                                }

                            }

                        ]

                    }

                ],

                "metadata": {

                    "model": {}

                }

            }

        }

    ]

}

 

 

 

Bing Search API

Introduction: This article is a continuation of the series of articles starting with the description of SignalR service. We followed up with a discussion of Azure Gateway service, Azure Private Link and Azure Private Endpoint and the benefit of diverting traffic to Azure Backbone network. Then we started reviewing a more public internet facing service such as the Bing API.

Description:

Azure Bing API is developer friendly and provides a robust way of searching custom and scoped content with the same standard as the public internet. A customized search instance can be created using the Bing custom search portal. Bing Search API maintains an index for the internet using web crawlers. There is an option for custom search where the same technology for different content types can be scoped to individual web sites. Cognitive search is geared for content from Azure data sources. It can index any json document that conforms to its service across clients and data sources. Unlike the Database search technology offered from database platforms that provide a built-in search capability for the content stored in their databases, the data is not proprietary for Bing Custom search, only the tokens are. Both database platforms and cognitive search can index data very well and the latter provides advanced features for deep learning but Bing Search API is most comprehensive in terms of web content that is indexed. If search and storage must be combined, SQL Server and Cosmos DB have out-of-box features to support this use case and can work well on both on-premises and in cloud. Many solutions use both but Cognitive service can perform advance text and natural language processing with its features for autocorrection of misspelled words, synonyms, suggestions, scoring controls, facets and custom tokenization. The Azure cognitive search persists data only in the form of an inverted index and it provides no solution for data storage. The use case where a data storage might be required to be independent from the search service could include the case where the database is targeted towards the online transaction processing and the cloud service is externalized to adjust elastically to query volume. Azure Bing API serves this space with a collection of APIs from different categories. The Bing Entity Search API returns search results containing entities such as people, place or things. The Image Search API lets users search for and find high quality static images with the inclusion or exclusion of attributes such as size, color, license and freshness. The internet provides a vast resource for news and the Bing Search API includes new search API in its portfolio. The video search API helps find videos across the web and the visual search API looks for similarity based on visual aspects. Local businesses search API narrows downs search based on contact and location information.

 

Conclusion:

With the large collection of resources available on the internet, the Bing Search API facilitates search by content on all public sites and with the flexibility to customize to specific sites.

 

Introduction:

This article is a continuation of the series of articles starting with the description of SignalR service. We followed up with a discussion of Azure Gateway service, Azure Private Link and Azure Private Endpoint and the benefit of diverting traffic to Azure Backbone network. Azure Gateway spanned private IP addresses, public IP addresses and availability zones with tremendous control over these backend pool members. We now review a more public internet facing service such as the Bing API.

Description:

The Bing Web Search API is a search engine API with tremendous ability to perform custom search. Indexes are built from HTML, XML and other web content on public sites. The web crawlers can be scoped to individual web sites.  While Azure Cognitive search enabled us to transfer our content and create our own indexes. If the JSON document conforms to indexers which has tremendous flexibility about JSON structure, the document can be indexed, and the search enabled with enhanced user experiences. Bing Search, on the other hand, can be used to perform Entity Search, Image Search, News Search, Video Search, Visual Search, Local business search and even perform Custom Search. In addition, it provides Autosuggest features and Statistics which include call volume, top query strings, and geographic distributions.

The difference between the suitability of Bing Custom Web Search and the Azure Cognitive Search is truly gray area but some points stand out very well. If we just want to integrate applications for custom search, the former is programmatically available with little or no maintenance ownership. The latter powers AI driven enhanced user experience where they matter. Azure Cognitive Search differs from the Do-It-Yourself techniques in that it is a fully managed search-as-a-service, but it is primarily a full-text search. It provides rich user experience with searching all types of content including vision, language and speech. It provides machine learning features to contextually rank search results. It is powered by deep learning models. It can extract and enrich content using AI-powered algorithms. Different content can be consolidated to build a single index. 

The search service supports primarily indexing and querying. Indexing is associated with the input data path to the search service. It processes the content and converts them to JSON documents. If the content includes mixed files, searchable text can be extracted from the files. Heterogeneous content can be consolidated into a private user-defined search index. Large amounts of data stored in external repositories including Blob storage, Cosmos DB or other storage can now be indexed.  The index can be protected against data loss, corruption and disasters via the same mechanisms that are used for the content.  Index is also independent from the service so if one goes down, another can read the same service.  

The querying service supports search experience from a variety of clients and occurs on the outbound path of the search service. The index and the querying service are separate. In this article, we will compare this service with other search services

Bing supports creating a customized search instance using the Bing Custom Search portal where the domain, websites and webpages to be indexed can be specified. Search experience can even be previewed and search rankings can be adjusted. A user interface widget is available to be embedded in websites and applications.

Conclusion: Azure Bing API is developer friendly and provides a robust way of searching custom and scoped content with the same standard as the public internet.