Hyper Personalized Recommendation System through robust Data Driven Approaches
Overview
Today’s hyper-connected universe of 5G, Internet of Things (IoT), and digital transformation has led to the proliferation of digital consumers, whose differentiated expectations have plateaued revenue growth and consumer retention around the globe. In this complex ecosystem, increasing your brand’s share of wallet requires superior consumer engagement through personalized recommendations. Delivering highly relevant products, services, and recommendations at the most appropriate times, i.e. hyper personalization is a key strategy in enabling this type of interaction to improve consumer engagement significantly
Mu Sigma helped one of the largest telecom providers in the US weave hyper personalization into their consumer interactions. We developed a System of Intelligence, an AI powered framework that can push personalized recommendations by mining consumer engagement in real-time.
The Problem
With a consumer base the size of 95+ million, this telecom provider felt the need to transition towards a Digital Service Provider model. Mu Sigma inferred the need for a data management platform to anticipate consumer expectations and optimize their experience in real time. The next step in their personalization maturity was empowering them to deliver hyper personalized consumer experiences through highly tailored products, services, and offers.
We broke the problem down into component outcomes that would guide the solution:
• Move existing data to a faster, scalable cloud-based architecture
• Break data silos to enable complete consumer profiles
• Create a rule-based decision engine to determine the most relevant consumer recommendations
• Push hyper personalized promotions based on real-time triggers
• Shortlist reusable tools and platforms to scale the solution quickly across functions
These broad strokes were fed into our outcome-driven Art of Problem-Solving framework, which helped us pan out a systematic plan to create a highly functional prototype.
The Mu Sigma Approach
At Mu Sigma, we develop and deploy Decision Support ecosystems to drive meaningful decisions beyond the scope of traditional insight generation using:
• Data Engineering
• Data Science
• Decision Science
Our proposed solution involved defining and developing a System of Intelligence framework spanning all three layers to create a lasting competitive distinction for this telecom provider. This framework would mine consumer data in real time, enabling the brand to push highly relevant consumer communication with the right offer at the right time through the right channel.
This framework would work over and above the existing systems of record and engagement to drive deeply personalized consumer interactions.
The System of record collects data from multiple sources to the Systems of Intelligence layer, enabled by AI at every step, to generate unparalleled real-time actionable consumer insights at an individual consumer level.
The Solution
A System of Intelligence (SoI) shifts the focus from gathering consumer data to utilizing them to drive actionable decisions.
For this telecom giant, Mu Sigma operated as a neutral guiding force while discovering the right tools to implement the SoI. Since extreme experimentation is one of our core tenets, we experimented rapidly with multiple tools at low-cost to develop the final SoI based hyper personalized recommendation system on cloud.
We then designed the entire data architecture from scratch, infused with artificial intelligence wherever applicable, keeping in mind best practices for the future and client preferences. Since they were already utilizing AWS services, we tested complementary AWS platforms and open-source tools to create the baseline architecture for the recommendation system.
1.Data Ingestion
The omnichannel reach-out to consumers translates to high data inflow from multiple sources. Data ingestion eases the process by connecting various data sources, extracting structured and unstructured data, and preparing the data for further usage and storage.
In this case, data ingestion was enabled through:
• Amazon’s Kinesis services to benefit from its dual ingestion and processing capabilities
• StreamAnalytix to simplify the front-end for the users
2.Data Processing
Out of the two approaches to real-time big data processing architectures – lambda and kappa – we selected lambda because:
• It can handle low-latency reads and updates in a linearly scalable and fault-tolerant manner
• It can run ad-hoc queries against all data to get results
Running ad-hoc queries is quite expensive, so we precompute the results as a set of views and then query the views. The lambda architecture is implemented as three layers: batch, speed, and serving layer.
3. Data Storage
For such a massive operation, we require a data storage platform that can handle huge incoming data traffic. We picked Cassandra for its superior write performance compared to other NoSQL databases.
4. Model Operationalization
Model Operationalization enables this telecom giant to deploy and run models at scale. We evaluated various tools for this function based on – model development, model deployment, and model management.
The core upgrade we recommended here was creating a central model container. The teams in this organization deploy multiple models arising from various tools. We gathered all these models in a single container on cloud for one-stop deployment from this central location.
Based on our evaluation and client preferences, we chose Amazon SageMaker because:
• It provides the flexibility of development and deployment using one single service
• It helps deploy models from various tools (Skleam, Spark, TensorFlow, PMML) using a single container service
5. Logging and Monitoring
To ensure performance efficiency of this hyper personalized recommendation system, we need to constantly monitor the health of multiple services. Logging and monitoring tools are crucial to pinpointing the cause of errors and identifying performance bottlenecks.
We recommended AWS Cloudwatch for this purpose because:
• It’s designed to work collaboratively with the AWS services
• It’s fully managed and easily configured
6. API Management
We preferred Apigee for API management due to security and ease of deployment. It serves as a consistent platform for multiple services of APIs starting from data collection till model operationalization.
The Impact
The initial prototype we developed had clear tactical wins:
• This hyper personalized recommendation system could provide a consumer recommendation in less than 5 seconds with basic resources. Based on the input load expected, the allocation of optimal resources can reduce the time to recommend even further
• 5+ models ran on a single AI-powered architecture in real time with sub-second latency
• Benchmarking metrics which helped predict independent evaluation of future choices
This prototype is being implemented org-wide by this Telecom leader through the hyper personalized recommendation system that we created. Essentially, the System of Intelligence will propel them into the realm of hyper personalization from consumer segment-driven personalization. The implementation of SoI will revolutionize the way consumers engage with and experience their tailored services.