The AI industry needs to stop peddling myth of skills shortages


Talk of a chronic and deepening skills shortage in AI has been endemic for several years, and apparently backed up by numerous reports from supposedly reputable analysts. Yet look beneath the bonnet and most of these reports are founded on hot air with little solid evidence for the numbers given.

There are of course shortages of skills in some specific sectors where AI is being applied, but look closely and these are mostly for engineers trained in a particular discipline. The belief in a severe shortage of AI is sustained partly by the hype sweeping the field itself and associated with vested interests on the part of the parties concerned. AI practitioners themselves naturally want to instill a belief their skills are rare, to boost their market value and that has worked extremely well given that average salaries are running at almost $200,000, far higher than those of people equally capable and well qualified in IT who lack the AI label.

This is reminiscent of a related myth of skills shortages among financial advisors, which continued to hike inflated salaries in that sector even after the credit crunch of 2008. The myth there was that financial advisers had to be highly intelligent and well qualified in the first place, given that it has become clear those skills actually make little difference to the quality or accuracy of the service. Now ironically, AI based systems are outperforming human advisors and will ultimately cost much less.

AI firms, on the other hand, want to promote the skills shortage myth for the opposite reason, to increase the supply of talent and so contain the costs. That could explain one of the most highly cited numbers from China’s tech conglomerate Tencent, with its assertion that there are just 300,000 AI researchers and practitioners worldwide while the “market demand” is for millions. Not a shred of evidence was furnished for this number, which appeared plucked out of hat.

Governments have also taken the bait, noting how French member of parliament and professor of mathematics Cédric Villani earlier this year published a report proposing to expand greatly education in AI to multiply number of students coming through by 3, citing a worsening talent shortage. It is odd then that applicants for posts in AI often report being turned away for lack of experience. This could be a rerun of the age-old adage that people cannot get jobs until they have the experience but can’t get the experience until they find a job.

But in the case of AI there seems to be something else at work, which is a misassumption over where the skills needed lie. This is for people who might be described as vertically integrated, having a deeper understanding of the problems they are trying to solve in a given sector or even related group of sectors, rather than for general AI practitioners. There may then be a skills shortage in some sectors but not at the level of basic recruitment, more at the multi-disciplinary industry level.

Some AI start-ups have recognized this and targeted specific sectors in which they are becoming expert. They are then destined to succeed or at least survive providing they have chosen sectors that are ripe for AI. But meanwhile it would help us all if this bleating about skills shortages subsided.