How machine learning is improving job transitions through skills matchmaking

Researchers have developed a machine learning-based system that acts as a skills matchmaker to assist with the process of effective job transitions

Restrictions and lockdowns associated with the COVID-19 pandemic have upended the livelihoods of many people around the world. In Australia, more than half a million casual workers alone lost their jobs in 2020 accounting for around two-thirds of the total of displaced workers by mid-2020. From hospitality workers and shop assistants, through to tourism operators, café owners and arts and entertainment workers – just to name a few – many have been forced to undertake the difficult and sometimes distressing job transition challenge of finding a new occupation quickly.

In reality, COVID-19 is a harsh crisis that has accelerated the inexorable effects of ongoing technological and economic changes which impact work in many ways. In historically normal times, the typical Australian will change careers five to seven times during their professional lifetime. And this job transitions process is likely to increase as new technologies automate labour, production is moved abroad, and economic and other crises unfold.

The job market has always adapted to changing needs, says Mary-Anne Williams, a Professor in the School of Management and Governance and the Michael J Crouch Chair in Innovation at UNSW Business School. “To be competitive today, workers need to manage their skill development to ensure it fits the current and future job markets – a challenging task requiring the ability to assess and predict the usefulness of skills with limited and patchy data,” she explains.

When it comes to job transitions and career planning, Prof. Williams says we tend to rely on our personal experience or stories shared by our trusted personal and professional networks. “Sometimes we pay for professional advice. Most of the information we gather to help guide us is generic and directs us toward well-trodden pathways that have been successful for others in the past,” she says.

However, there is limited scope for organisations to design new occupations in roles that allow us to pursue our interests while leveraging our unique capabilities. But just as education and technology drive change and disruption, Prof. Williams says they are also enabling new ways of acquiring skills, opening new pathways for transitioning to both traditional and emerging occupations, and shaping the job market.

Gaps and challenges in the job transition process

New research from the ABS showed almost 70 per cent of Australians between 18 and 24 years of age are willing to change their career path to ensure they get a job, and around six million Australians have considered a job or career shift during the past 12 months. However, Prof. Williams says the problem is there are no easily accessible tools to help understand the options and possibilities to lower the barriers.

“The ultimate goal is to remain employable throughout our entire working life and/or develop the skills that will allow us to create our own viable business. We can increase our chances of success if we understand how to leverage our existing skills and acquire valuable new skills,” she explains.

“The underlying issues are to identify the range of employment options within your range of skill development in the timeframe you must work with, that could provide sufficient remuneration to cover your reasonable costs in the foreseeable future. An ideal transition takes you to a desirable job with feasible training requirements and viable pay levels sufficient to support a reasonably anticipated lifestyle. There is no easy way to find this job transition sweet spot.”

Prof. Williams says current methods for identifying new job opportunities are complex, inefficient, crude, and tend to provide generic pathways. And more customised approaches are expensive and out-of-range for low wage workers: “this advice is not based on current job market data and workers feel like they are flying blind especially when forced to navigate unexpected job transitions,” she says.

Addressing job transition gaps and challenges

Lifelong learning is a proactive strategy that helps to enhance job transition opportunities and build resilience and adaptability in changing and uncertain jobs markets, Prof. Williams explains. The underlying challenge, though, is how to make the best skill acquisition choices in an increasingly crowded, complex and dynamic skills marketplace. “How can we optimise our learning experiences to follow the sweet spot as it morphs and changes throughout our careers?

Read more: The long-term impact of COVID-19 recession on human capital

“Navigating a job transition requires actionable insights and strategies based on data. We do not control the job market, but we can better prepare for challenges and opportunities by understanding our skill sets and how we build upon them to transition across fields and industries to desirable jobs,” she says.

Recent Atlassian and PwC research suggests that increasingly workers want to work to live, not live to work, and this means developing the ability to grow in your current job and learning to purposely transition efficiently and effectively as opportunities arise.

“When we want to transition to a new occupation, we typically experience five main pain points,” Prof. Williams explains. “The data we have access to is limited and out-of-date, the proposed solutions tend to be traditional based on limited human judgement, gaining customised advice is expensive, and it is slow and difficult to discover how best to leverage your current skills to optimise life and livelihood outcomes.”

The Job Transitions Recommender System

To make the job transition process easier, and increase the chances of success, researchers from UNSW Sydney and UTS have developed a machine learning-based method that can identify and recommend jobs with similar underlying skill sets to someone’s current occupation.

The “Job Transitions Recommender System” can respond in real-time to changes in job demand and provide recommendations of the precise skills needed to transition to a new occupation. Developed by Prof. Williams in collaboration with Dr Nikolas Dawson and Dr Marian-Andrei Rizoiu from the University of Technology Sydney (UTS), the system is based on findings from their new study, Skill-driven Recommendations for Job Transition Pathways, published in the international journal PLOS ONE.

The system is a working prototype that uses current Australian online job advertisements data, real worker job transition data and machine learning to build a model that can be used to recommend new skills for job transitions. Jobs and occupations are characterised as sets of skills. A new occupation can be created and modelled by bundling a new set of skills.

“The system addresses the five pain points I mentioned earlier by solving two deep technical problems. First, calculating the similarity between different skill sets (occupations), and second accurately predicting the likelihood of transitioning between specific occupations when new skills are added to a skill set,” says Prof. Williams, who adds the system makes job transitions faster, cheaper, and better by discovering and assessing the effort required to pursue specific training pathways.

How the system can better job transitions in practice

“It can be used to help workers develop the necessary actionable insights and strategies to develop an adaptable roadmap and to more safely navigate the highly complex process of determining the skills they need to optimise their short/medium/long term career outcomes,” she says.

“One of the exciting opportunities it opens up is the design of new and hybrid occupations that can maximise the likelihood of attracting candidates with the new mix of skills or that require the least amount of training.”

For example, an emerging new occupation is a “digital employee designer”, an occupation which brings chatbots to life giving them a face, a voice, a personality and the ability to converse with humans. “Our system can determine if it is more efficient or cost effective to train a digital designer or a cognitive scientist for this new occupation. If you are a quantum physicist, how might you transition to a quantum financial analyst?” says Prof. Williams.

The Job Transitions Recommender System can also help organisations and workers adapt and pivot. “It could also play an important role in helping low-wage workers navigate to higher economic ground, and it can recommend skills that workers need to develop to help businesses survive, thrive and drive disruption. Since there are time and resource constraints for developing new skills, workers need help to plan and prioritise which skills to develop,” she says.

For more information on the Job Transitions Recommender System, please contact Professor Mary-Anne Williams, Michael J Crouch Chair in Innovation at UNSW Business School.