According to a comprehensive study conducted by Databricks, a leading data and AI company, the demand for large language models (LLMs) has experienced an astounding surge of 1,310 per cent.
The “2023 State of Data + AI” report analyses anonymised usage data from over 9,000 global Databricks customers, providing valuable insights into organisations’ data and AI initiatives.
The latest report by Databricks delves deep into the current state of digital transformation within enterprises, highlighting the specific platforms and tools they employ to harness their potential.
The findings of the Databricks study shed light on the rapidly evolving landscape of data and AI, with LLMs and data integration taking centre stage, open-source technologies gaining traction, and enterprises demonstrating increasing proficiency in AI implementation.
Key Findings from the Study:
LLMs are capturing attention: Between the end of November 2022 and the beginning of May 2023, the utilisation of Software-as-a-Service (SaaS) LLMs, such as OpenAI’s models, witnessed an exponential growth rate of 1,310 per cent among Lakehouse customers. Transformer-related libraries like HuggingFace, an NLP toolkit and model hub utilised for training custom LLMs, also experienced an impressive 82 per cent growth within the same timeframe, indicating substantial demand even prior to the introduction of ChatGPT.
Data transformation and integration take centre stage: The fastest-growing tools on the Databricks platform are dbt, a popular data transformation tool (206 per cent growth), and FiveTran, a data integration tool (181 per cent growth). Additionally, among the top 10 most popular data and AI products, six belong to the data integration category, including renowned names like Informatica and Qlik, highlighting the remarkable growth of this market within the Databricks Lakehouse ecosystem.
Embracing open source technologies: While Microsoft Power BI and Plotly currently dominate the landscape of data and AI products, there is a notable inclination toward open-source solutions. Eight out of the top 10 most popular data and AI products are based on open-source software, with notable examples being dbt, HuggingFace, and GeoPandas, showcasing organisations’ affinity for open technologies.
Enterprises drive increased AI adoption: The study reveals a significant rise in the number of AI projects undertaken by enterprises, with a staggering 411 per cent year-over-year growth in the deployment of models in production. Experimental projects also witnessed a substantial increase of 54 per cent.
Notably, the data indicates that, on average, one in three experimental models have the potential for real-world implementation, compared to one in five last year. This suggests that organisations are becoming more adept at successfully constructing and scaling these projects.
Balancing AI with traditional data analytics: Despite the growing prominence of AI, the report emphasises the ongoing importance of traditional data analytics. Microsoft Power BI held the top position as the most popular program running on the Databricks Lakehouse last year, while serverless data warehousing with Databricks SQL experienced remarkable growth of 146 per cent during the same period.
Full report here.
Keep up to date with our stories on LinkedIn, Twitter, Facebook and Instagram.
