A wide range of possible applications exist for this technology, and the inventors have validated the system using examples from fields as disparate as education, healthcare and web forums. Any company using predictive analysis and machine learning can use this system.
While big data and data mining systems that can analyze data exist, a key bottleneck in the field of machine learning/data mining and enterprise model search is the inability of systems to predict or suggest algorithms for a new data set based on experiences with prior data sets. This provides a dataset-size invariant approach to accelerating model search that provides significant improvement over state of the art grid search and hyperparameter search methods.
Delphi is a distributed system that runs multiple machine learning algorithms on many machines across a cloud to find predictive models of optimal efficiency. It is designed to learn from its experience across a multitude of data sets and act as a recommender system that points out the most promising approach. It uses Bayesian optimization and Multi-Armed Bandit techniques for inter-model selection. It manages big data by breaking it down into smaller chunks to find optimal models on these, and then combining them into a higher performance meta-model while making use of the previously created models to enable knowledge transfer in modeling. The software has been tested with multiple data sets.
- Algorithms predicted by Delphi shown to be 3-5% more accurate, and up to 20% more in some cases
- Shortens the algorithm design process, which would usually take a human team up to a year.