[ad_1]
Table of Contents
What Is a Recommender System?
A recommender system is a specialized algorithm or software designed to predict and suggest items or content that a user may find relevant, appealing, or useful based on their preferences, behavior, or interaction history. These systems are widely used in various industries, including eCommerce, social media, streaming services, and online advertising, to enhance user experience, increase engagement, and maximize revenue.

Cloud network technology – artistic visualization. Image credit: Rawpixel via freepik, free license
Recommender systems rely on multiple techniques, such as collaborative filtering, content-based filtering, and hybrid approaches. Collaborative filtering analyzes user-item interactions and similarities among users or items to generate recommendations. Content-based filtering, on the other hand, considers the features of items and user preferences to provide personalized suggestions. Hybrid approaches combine both methods to optimize the prediction accuracy and tackle challenges such as cold start and data sparsity.
What Are the Benefits of Building It on Azure?
Building recommender systems in the Azure cloud offers several benefits, including:
- Scalability: Azure cloud provides scalable computing resources, enabling businesses to build and deploy recommender systems that can handle large volumes of user data and provide real-time recommendations to users.
- Flexibility: Azure cloud offers a range of services and tools that can be used to build recommender systems, including machine learning services, data storage services, and analytics services. This allows businesses to choose the services and tools that best fit their needs and budget.
- Integration: Azure cloud integrates with a wide range of other Azure services, such as Azure Active Directory, Azure DevOps, and Azure Security Center. This allows businesses to build recommender systems that are integrated with other business applications and services, enabling a seamless and integrated user experience.
- Security: Azure cloud provides robust security and compliance features, including encryption, identity and access management, and threat detection. This helps businesses to protect their user data and ensure that their recommender systems are compliant with industry regulations and standards.
- Cost-effectiveness: Azure cloud offers a pay-as-you-go pricing model, allowing businesses to control their Azure costs and only pay for the resources they use. This can help businesses to reduce their IT costs and increase their return on investment (ROI) for building and deploying recommender systems.
Building a Real-Time Recommendation API on Azure
The following architecture explains how recommender models can be trained with Azure Databricks and deployed as an API with Azure ML (Machine Learning), Cosmos DB, and AKS (Azure Kubernetes Service).

Azure cloud structure. Image source: Azure
Here is an outline of the creation and deployment of a real-time recommendation API for offering the top ten movie recommendations to users.
Dataflow
- Monitor user behaviors, e.g., logging user interactions with movies, products, or articles.
- Import data from a data source into Databricks.
- Process data and divide it into a training set and a testing set.
- Apply Spark collaborative filtering model to the imported data.
- Assess the model’s performance based on ratings and rankings.
- Precompute top ten recommendations for each user, storing them in Cosmos DB.
- Use ML APIs to containerize an API service and deploy it to AKS.
- Upon receiving a user’s request, call the recommendation API in AKS to display the top ten recommendations.
Components
- Azure Databricks: A development environment for data preparation and recommender model training on a Spark cluster, offering an interactive workspace for collaboration.
- AKS (Azure Kubernetes Service): Hosts containerized models, enabling deployment and operationalization of ML model services on Kubernetes clusters. Provides scalability, identity and access management, and monitoring.
- Azure Cosmos DB: A globally distributed database service for storing top 10 movie recommendations per user, featuring low latency and high performance.
- Azure Machine Learning: Tracks and manages machine learning models, packaging and deploying them to an AKS environment.
- Microsoft Recommenders: An open-source repository with utility code and samples for developing, evaluating, and operationalizing recommender systems.
Performance
Performance efficiency refers to a workload’s ability to scale and meet user demands efficiently. For real-time recommendations, performance is crucial as it directly affects user experience on websites.
The combination of Azure Kubernetes Service (AKS) and Azure Cosmos DB in this architecture offers a solid foundation for medium-sized workloads with minimal overhead. In a load testing scenario with 200 users, it delivered recommendations with a 60 ms median latency and a 180 requests/second throughput. The test used a default configuration for a deployment of three D3 v2 AKS clusters with 42 GB memory, 12 vCPUs, and 11,000 RUs (request units) per second for Cosmos DB.

Image Source: Azure

Image Source: Azure
Image Source: Azure
Azure Cosmos DB is preferred for its globally distributed nature and versatility. One way to minimize latency is to use Azure Cache for Redis, which can enhance the performance of recommendation systems relying on backend data stores.
Scalability
For smaller workloads or if not using Spark, consider a DSVM (data science virtual machine) rather than relying on Azure Databricks. DSVMs are virtual machines equipped with deep learning (DL) frameworks and tools for data science and machine learning, and their models can be operationalized on AKS using Machine Learning.
While training the model, either use a fixed large-size Spark cluster or enable autoscaling in Databricks. Autoscaling adjusts the cluster size according to the workload. Adjust the AKS cluster’s scale to meet throughput and performance needs, utilizing the appropriate number of pods and nodes. AKS cluster autoscaling is also available.
To optimize the performance of Azure Cosmos DB, estimate the required reads per second to enable provisioning of the necessary throughput (in RUs per second).
Optimizing costs
Cost optimization involves minimizing unnecessary expenses and enhancing operational efficiencies. Key cost drivers in this scenario include the cluster sizes for Databricks and AKS, and the number of RUs provisioned for Cosmos DB. Manage costs by implementing less frequent retraining, turning off the idle Spark cluster, and adjusting Cosmos DB and AKS resources based on site traffic volume, scaling them accordingly to meet performance requirements.
Deployment
Deploying this architecture involves these steps:
- Create a workspace in Databricks.
- Set up a new cluster in Databricks with the specified settings.
- Generate a personal access token in the Databricks workspace.
- Copy the Microsoft Recommenders repo to an environment where scripts can be executed.
- Install the required libraries on Azure Databricks following the installation instructions.
- Prepare Azure Databricks for operationalization using the installation instructions.
- Import the relevant notebook (ALS Movie Operationalization) into the workspace by pasting the provided URL in the Import section of your Azure Databricks home directory.
- Open the notebook in Databricks, attach your configured cluster, and execute it.
- Upon completion, you will have the necessary Azure resources to create a recommender API that offers the top ten movie recommendations for a specific user.
Conclusion
In conclusion, the Azure Cloud provides a robust and scalable environment for building and deploying recommender systems. By leveraging Azure Databricks, Azure Kubernetes Service, and Azure Cosmos DB, developers can create an efficient architecture that serves personalized recommendations in real-time.
The Microsoft Recommenders repository offers valuable resources and tools to facilitate the development process, while the Azure platform ensures flexibility in scaling resources and optimizing costs.
By following the deployment steps outlined in this article, businesses can create a powerful recommender system that enhances user experience and drives engagement across various digital platforms.
Author Bio: Gilad David Maayan
Gilad David Maayan is a technology writer who has worked with over 150 technology companies including SAP, Imperva, Samsung NEXT, NetApp and Check Point, producing technical and thought leadership content that elucidates technical solutions for developers and IT leadership. Today he heads Agile SEO, the leading marketing agency in the technology industry.
LinkedIn: https://www.linkedin.com/in/giladdavidmaayan/
[ad_2]
Source link