It's 21st century, computational powers have boomed, GPUs and CPUs competing speed of light, trillions of Gigabytes of data being produced every second. Researchers have declared this as an Artificial Intelligence(AI) era. But long ago, similarly people claimed that one day there will be a computational machine which will be able solve every problem, which Alan Turing in 1936 proved wrong by the 'Halting problem' undecidable proof.
And now again, many researchers believe, AI can fetch solutions of all decidable problems. Individuals and organizations have similar expectations of being able to use Machine learning(ML) as a silver bullet to be able to solve all their business problems.
Several decision makers take a simplistic view of machine learning being a readymade set of tools and packages, which can be applied to any dataset, and it would throw up solutions to virtually any problem.
Well, Machine Learning helps us to build solutions without explicitly coding logic but unfortunately, it is still also highly dependent on human based decisions.
ML algorithms are data hungry and Deep Learning requires even more data than conventional ML. Also supervised ML algorithms requires outputs along data but there exist tons of data where output isn't available and can only be used for unsupervised or reinforcement models.
Experts are needed to analyze the data integrity before choosing it. Data generation sources provides us large amounts of data but larger quantity doesn't assure better results always. Data pre-processing is a highly under-rated activity but has significant importance in making or breaking a project.
Data anomalies are there, which lead to depreciate our ML results. On the other hand, most of available data is unstructured, requires great effort to sense out of it. Having the right data is the first hurdle. These days many researchers are working on this problem. There have been advances in Auto ML also, but it is still premature, and we still need the human element.
Features engineering is another important prerequisite before applying ML where engineers need to extract or generate those features, which may increase efficiency and avoid under-fitting or over-fitting. Highly skilled resources are required for feature engineering with similar domain expertise. One with similar domain experience will highlight the key deciding features to be extracted, in lesser time, which when passed to ML algorithm results better. But it is very difficult to find such expert resources.
Another problem with ML is 'model selection'. After understanding problem domain, ML engineer looks for the possible models on which they can train. And this selection of model is mostly done by combination of literature review of research done for similar problems and by 'hit and trial'.
It is not a simple solution of taking the whole set of models and tuning for all possible hyperparameters, throwing the highest available computing resources on the task. Like feature engineering, model selection also requires highly experienced ML engineers. Undoubtedly model selection is mostly research along with 'hit and trial' but still without the experienced supervisor and expert resources, there is high chance of selecting wrong model.
It is also found that if incompatible model has been chosen initially, we might have to restart everything again from scratch as structuring data, extraction of features and model selection are highly correlated with each other. Having ML engineers who understand the domain thoroughly is important to save time in these efforts.
But what if there's a new problem?
After all of these we might be thinking that if we get through these above mentioned stages, we are ready to go. But wait a minute!
These practices don't guarantee us that our ML algorithm will train itself to highest accuracy or train itself to the threshold we want to achieve.
• Maybe more features are required?
• Or maybe more training data?
• Or maybe incompatible model was selected?
Even stakeholders can raise their concerns about low accuracy achieved after months and years of hassle and investment.
So this is an ongoing process and people relating AI with some magical spell have been misinterpreting ML. No doubt ML has helped for resolving many problems, but like conventional programming here also exist limitations and even more complex problems which may lead to years and years of research, still a satisfactory solution is not always guaranteed.
So if you think that "Machine Learning is a silver bullet", you need to reconsider it!