Data science has evolved from being primarily an IT initiative into business applications where it actually adds value. Innovative software companies have disrupted the data science industry to give rise to more jobs and more competition in the data science, machine learning and even data visualization space. The barrier to entry for companies to get into analytics has been greatly reduced in the past decade as a result of software advances. With technical boot camps, MOOCs and a good statistics background, anyone can become a data scientist. One thing to emphasize is that data science is the furthest thing from a new concept; it’s fundamentally statistics. What has changed is that software has enabled engineers and business owners to leverage this knowledge into powerful applications. The data science field is growing rapidly with vast improvements in technology and software, and many of the tools like R and Python are free.
McKinsey Research estimates that there will be a severe job shortage of 190,000 by 2018 to meet this demand. Most data scientists can command an average salary of $122,000, according to KDnuggets. With technology always evolving, we must rely on software development to help grow the data science field to automate many tasks in a data scientist’s workflow.
To a certain extent, job demand will follow the current hype until software automates it. You don’t hear anyone claim there is going to be a job shortage of 200,000 steam engine mechanics to help aid the modern Industrial Revolution, do you? Of course not. But this example illustrates that job demand is largely driven by the current hype and demand. Even several years of software and hardware innovations can make current jobs obsolete.
The “severe job shortage” that is predicted in the near future will not become a reality because three reasons:
1) Software in the data science industry will gradually evolve to automate redundant or archaic processes now, thus relieving pressure to hire more personnel with data science talent in the future.
2) Training will lower the cost of entry for future potential candidates. Bootcamps, MOOCs and even the modern collegiate programs are embracing this skill set.
3) It’s popular to be a data scientist right now, but something else will pique organizations’ interest such as Machine Learning, Computer Vision, Artificial Intelligence, Internet of Things and Augmented Reality. Even though the workflow of these positions can be nearly identical to data science, the hype will transform it into something else.
In the near future, data science will largely be considered a field rather than a singular position. Acquiring software will be extremely valuable in addition to hiring the right positions in a data science team. A data designer is a new job title that will enter the market in the coming years. They are tasked with communicating complex data workflows closer to a graphic designer, rather than a data scientist, since it will likely require minimal statistics knowledge. The majority of data scientist initiatives just involve hiring someone that can communicate the value of advanced analytics. Data analyst, data engineer and data designer will all be positions that will gain traction and stem from the core data science field.
Here are several recommendations before embarking on a data science initiative:
1) Don’t reinvent the wheel. Many other organizations may have already created great software to solve complex data science problems. Not Invented Here (NIH) plagues modern businesses.
2) Subject matter expertise and communication are by far the most important factors in a successful data science hire. Hiring the right tech talent with a passion and knowledge of your industry will increase project success exponentially. Don’t hire a jack of all trades, as they will be a master of none.
3) Instead of hiring amateur data scientists, rent the software instead. Many disruptive companies are advancing data science that will enable business analysts to create self-service data science reports, a term called Citizen Data Science. Use your data scientist FTEs to solve complex problems and use software to automate redundant workflows.
These are exciting times for data nerds, but now the difficulty in this market will be for the right talent to stand out to employers. What will be difficult for an organization to determine is when to hire or buy software. As time goes on, more data science software vendors will evolve and automate the workflow of data scientists. Many companies such as Continuum Analytics (makers of Anaconda), Dato (makers of GraphLab) and Microsoft’s Azure Machine Learning are already creating great solutions to automate data science workflows. Software may be eating the world, but it’s creating greater job opportunities in data science, not job shortages.