Stanford University’s Institute for Human-Centered Artificial Intelligence (Stanford HAI) launched in 2019 with a mission to advance AI research, education, policy and practice to improve the human condition. Stanford HAI recently published the 7th edition of its AI Index Report. This year's report is the most expansive yet, covering a broad range of essential trends in AI, including technical advancements, public perceptions and geopolitical dynamics.
Top 10 Takeaways
At just over 500 pages, the report is not a casual read! But, rather helpfully, the authors set out their top 10 takeaways for those with less time on their hands.
- AI beats humans at some (but not all) tasks. AI has surpassed human performance on several benchmarks, including some in-image classification, visual reasoning and English understanding, yet it trails on more complex tasks like competition-level mathematics, visual common sense reasoning and planning.
- Industry continues to dominate frontier AI research. In 2023, industry produced 51 notable machine learning models, while academia contributed only 15. There were also 21 notable models resulting from industry-academia collaborations, a new high.
- Frontier models get way more expensive. Per AI Index estimates, the training costs of state-of-the-art AI models have reached unprecedented levels e.g. OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while compute for Google’s Gemini Ultra cost $191 million.
- US leads China, EU & UK as leading source of top AI models. In 2023, 61 notable AI models originated from US-based institutions, far outpacing the European Union’s 21 and China’s 15.
- Robust, standardised evaluations for LLM responsibility lacking. AI Index research reveals significant lack of standardisation in responsible AI reporting, which complicates efforts to systematically compare the risks and limitations of top AI models.
- Generative AI investment skyrockets. Despite a decline in overall AI private investment last year, funding for generative AI surged to $25.2 billion, with OpenAI, Anthropic, Hugging Face & Inflection reporting substantial fundraising rounds.
- AI makes workers more productive and leads to higher quality work. Several studies assessed AI’s impact on labour, suggesting that AI enables workers to complete tasks more quickly and improve the quality of their output, demonstrating AI’s potential to bridge the skill gap between low and high-skilled workers.
- Scientific progress accelerates even further, thanks to AI. 2023 saw the launch of a number of significant science-related AI applications, from AlphaDev (which makes algorithmic sorting more efficient) to GNoME (which facilitates process of materials discovery).
- Number of AI regulations in the US has sharply increased. The number of AI-related regulations in the US has risen significantly in the past year and over the last five years, with 25 AI-related regulations in 2023, up from just 1 in 2016. Last year alone, the total number of such regulations grew by 56.3%.
- People more cognizant of AI’s potential impact...and more nervous. Per an Ipsos survey, over the last year, the proportion who think AI will dramatically affect their lives in the next 3-5 years has increased from 60% to 66%, with 52% expressing nervousness toward AI products and services (a 13% rise from 2022).
Having read the AI Index’s report cover-to-cover, hear are some other highlights:
- There have been significant improvements in AI's natural language understanding, with models now achieving human parity on many benchmark tasks, enabling more natural and intuitive interactions with AI assistants. In the field of science and medicine, the performance of clinically knowledgeable AI systems has been improving at a remarkable rate.
- Computer vision systems have also seen major leaps, with AI now matching or exceeding human performance on a wide range of visual recognition and analysis tasks. This is unlocking new applications in areas like medical imaging and autonomous vehicles.
- The compute usage of notable AI models has been increasing exponentially in recent years with cost, power and environmental consequences. Research shows that tasks like image generation have a much higher classification than text classification.
- A question of whether AI models will run out of data has emerged. Foundation models have been trained on meaningful percentages of all the data which has ever existed on the internet, with concerns that future generations of computer scientists will run out of data to further scale and improve their AI systems.
- Whilst the use of synthetic data (data generated by AI models themselves) may offer a solution to data shortage, there is growing awareness of a phenomenon called “model collapse” where synthetic data is used for training purposes, resulting in a significant drop in the model’s output quality.
- The majority of foundation models released in 2023 were released with “open access” (i.e. like open source software code, open-source models can be modified and freely used by anyone) in line with the general trend over the past few years. However, on a range of benchmarks, the performance of the top closed models outperformed open models.
- Industry (Google, followed by Meta and Microsoft) dominates over academia in terms of the number of foundation models released. The growth of open source AI software development projects (as recorded in GitHub) also accelerated rapidly.
- China and the US lead the way with new AI patent applications and grants, although, relative to one another, China’s share has increased significantly whilst the US’ share is declining relative to China.
- LLMs remain susceptible to factual inaccuracies and content hallucination, creating seemingly realistic yet false information. Research using one benchmark, designed to assess hallucinations in LLMs, indicates that ChatGPT fabricates unverifiable information in approximately 19.5% of its responses. This is concerning given widespread deployment in critical fields such as law and medicine.
- However, according to some benchmarks, LLMs are becoming progressively better at providing truthful answers, although hallucination remains a significant ongoing issue, with widespread reports of LLM hallucinations being especially pervasive in legal tasks.
- And finally, it is intriguing to note that, as AI models scale, it appears that they are gradually becoming more aligned with human moral judgement.
Legal and regulatory issues abound
Whilst the number of AI-related regulations globally is growing fast, issues around failures to comply with privacy and data protection laws continue, as do concerns about copyright, transparency and explainability, fairness, bias and discrimination, deepfakes, cybersecurity and safety. For example, with LLMs reliant on massive amounts of data, obtaining genuine and informed consent for training data collection is a significant challenge. In many cases, data subjects are unaware of how their data is being used or the extent of its collection. Further, properly anonymising data to enhance privacy while retaining data usefulness can be technically challenging due to the risk that anonymised data can be re-identified.
Conclusion
For anyone with even a passing interest in AI, the 2024 AI Index Report paints a truly fascinating, comprehensive picture of the rapid evolution of artificial intelligence and its far-reaching implications. By tracking key technical, societal and geopolitical trends, the report aims to provide policymakers, researchers and the general public with the data and insights needed to navigate the complex landscape of AI development and its impact on the world.
If you would like to discuss anything in this article, please get in touch with Tim Wright.
