AgilOne: Machine Learning at Enterprise Scale

AgilOne operates complex Machine Learning (ML) models and stores vast quantities of data for its customers, including major brands like Lululemon, Travelzoo, and Tumi. AgilOne Cortex is a very robust and flexible machine learning framework built into a customer data platform. AgilOne Cortex uses supervised machine learning models to predict customer events such as purchase, subscription, and engagement. It also intelligently segments customers together based on interest and behavior using unsupervised learning techniques. AgilOne Cortex’s recommender models allow the orchestration of offers and messages to customers on a 1:1 basis.

AgilOne Cortex fully supports all the functionality above with the scalability, configurability, and performance requirements needed from an enterprise machine learning platform. These features include:

• Configurability – AgilOne Cortex models are highly configurable to each brand’s unique needs.
• Data – Cortex models seamlessly leverage any data sources integrated by the client, including online, offline and third-party data.
• Scale – Cortex runs on a full cloud modular architecture that is horizontally scalable to  support any models a client may envision.
• Recipes – Cortex provides several “recipes” of the same model, for out-of-the-box  flexibility based on brand, region or any other dimensions.
• Consumption – Cortex model outputs are directly available in all AgilOne applications. AgilOne Actions readily exposes these model outputs thus enabling marketers to segment and personalize based on these ML model outputs seamlessly.

AgilOne operates Cortex on both Amazon Web Services (AWS) and Google Cloud Platform (GCP) and performs close to one billion predictions every day, averaging dozens of millions of customer predictions for each client across all its models. Their CDP is deployed on Google Cloud Platform and leverages Google Cloud Platform to provide unified customer data, customer intelligence and insights, and data activation and orchestration.

In order to meet the challenges of such vast amounts of data and millions of predictions, AgilOne partnered with Qubole to better automate the provision of machine learning data-processing resources based on workload, while allowing for portability across cloud providers; eliminating prototyping bottlenecks, supporting the seamless orchestration of jobs, and automating cluster management.

AgilOne now runs a variety of workloads for querying data, running ML models, orchestrating ML workflows, and more on Qubole —all on a single platform with optimized versions of Apache Spark, Apache Airflow, Zeppelin Notebooks, and leveraging Qubole’s APIs to automate tasks.

AgileOne is a leading customer data platform (CDP) for enterprise companies. AgilOne’s vision is to restore the personal relationships companies once had with customers before channel-specific marketing silos got in the way. AgilOne enables a single customer view through a best of breed identity resolution engine used on first-party customer data. Machine learning and analytical algorithms enhance customer data leveraged through real-time APIs and orchestrated across all touchpoints. AgilOne helps brands build authentic omni-channel customer relationships that maximize lifetime value. The AgilOne Customer Data Platform supports more than 150 brands worldwide.

By enabling AgilOne’s data science team to make cluster management and cluster provisioning more self service, smarter, and less dependent on the Operations teams, Qubole is helping AgilOne delivery of ML models more efficiently.

Thanks to its advanced cluster lifecycle management and workload-aware auto-scaling capabilities, there’s now less reliance on the Operations team as infrastructure is provisioned automatically through Qubole. Indeed, provisioning of new and larger clusters with different sets of permissions, installing dependencies on VMs, maintaining stable prototyping environments, or upgrading software is made easy. The data science team’s variable infrastructure needs are now addressed by Qubole’s intelligent automation—spinning up and releasing clusters and different types of nodes as needed.

Qubole’s comprehensive quality assurance and support, coupled with zero-downtime software upgrades and roll-back capabilities, delivered the assurance and stability that AgilOne required. In Qubole’s managed Zeppelin Notebooks environment, AgilOne can prototype its Python/Pyspark/Scala applications.

Qubole is strategic for AgilOne. It has helped us innovate faster on our machine learning road-map and empower marketers with more models at scale.

Gangadhar Konduri, Chief Product Officer, AgilOne

AgilOne supports deployment on both Amazon Web Services and Google Cloud Platform. Hence it is imperative that the underlying infrastructure used in AgilOne Cortex is cloud-agnostic. With Qubole’s cloud-agnostic platform, AgilOne teams can interact with data and models with consistent user experience, build automations with the same set of APIs, and the same data-processing engines, regardless of the cloud provider.

The increased efficiency and portability across clouds are essential for AgilOne.

Gangadhar Konduri, Chief Product Officer, AgilOne

AgilOne Cortex requires a powerful orchestration system to run, monitor dozens of models for all of its clients, and to run each model across all of their users every day. Qubole provides Airflow as a service, which allowed AgilOne’s data science team, to leverage configuration-as-code workflow engine, Airflow. This allowed AgilOne to better manage the lifecycle of its ML workflows by providing easy maintenance, versioning, and testing.

Qubole also provides a comprehensive set of APIs critical for end-to-end automation. This includes automating such tasks as starting and stopping cluster usages, submitting a Spark job or changing the Spark configuration, generating reports, increasing the timeout, and more.

Execution is further enhanced by Qubole’s excellent support. “The Qubole enterprise support is of really good quality,” says Arnaud Prades, Director of Data Science. “Support engineers are really knowledgeable, and escalation to higher support tiers happens faster.” On a related note, setup was easy. “The setup is really easy and straightforward,” says Prades. “Qubole can be up in running in about an hour or two with the right people in the room.”