Competitor Analysis

Lumi AI vs Snowflake CoPilot: Head-to-Head for Enterprise Data Analytics

October 1, 2024

.

14 mins

Blog

Competitor Analysis

Introduction

LLMs have redefined how users can interact with data, making it easier to ask questions and get answers without needing deep technical expertise. This has created a new category of applications known as text-to-SQL tools, which translate natural language into database queries. Beyond this, multi-agent systems are enabling deeper analysis by managing complex data workflows, interpreting business context, and understanding data schemas to generate more accurate and actionable insights.

In this blog, we explore the differences between text-to-SQL tools and multi-agent systems by comparing Snowflake Copilot and Lumi AI.

For a deeper understanding of the evolving landscape and the diverse solutions available, refer to AI Tools for Data & Analytics: Understanding the Landscape.

Head-to-Head Summary of Results

We connected the same dataset and asked the same questions. Here are the results.

Snowflake Copilot shows potential in aiding developers with SQL query generation but struggles significantly with queries requiring deep business context and advanced analytics. It often generates SQL queries with syntax errors, leading to delays and necessitating manual corrections. Additionally, its limited customization options—such as a single custom context box capped at 2,000 characters—hinder its ability to adapt to specific business needs. In contrast, Lumi AI leverages a multi-agent architecture that integrates business context and handles complex queries effectively. It excels in advanced analytics, understands business-specific contexts, and adapts over time through its "Memories" feature.

The article showcases Snowflake Copilot's ability to assist developers,  however lacks capabilities for organizations seeking advanced analytics and business relevant insights.

Overall Platform Assessment

In addition to the performance comparison, below is a comparison of Lumi AI and Snowflake Copilot across key categories. These categories reflect essential features such as the intended user, the depth of responses, and the ease of integrating business context.

Benchmarking Overview

This head-to-head comparison highlights how each tool performs in real-world conditions by connecting the same dataset, asking the same questions, and comparing the outputs.

Analysis & Benchmarking Process Summary

For this head-to-head comparison, we tested each platform with 21 questions of varying complexity, categorized as Easy, Medium, and Hard. These questions simulate typical business scenarios, from basic lookups to complex analytical calculations.

  • Easy: Simple tasks involving basic SQL queries and foundational schema knowledge.
  • Medium: Multi-step tasks requiring intermediate SQL or business context.
  • Hard: Complex tasks that demand advanced SQL skills and deep business expertise.

Please click here for more in-depth information on question evaluation methodology.

Scoring Methodology

Correctness of Answer (0-2 points) (Total 42)

Evaluation of how accurately the AI platform generates the correct SQL or answer.

  • 0 points: The platform did not provide a correct answer (e.g., missing key query logic or producing incorrect results).
  • 1 point: The platform delivered the correct answer on the second attempt after additional context or provided a partially correct answer.
  • 2 points: The platform delivered the correct answer on its first attempt, with the correct logic, structure, and expected results.

Dataset

The dataset used in this benchmarking was sourced from the dynamic retail and supply chain sector, with data spanning from 2020 to 2023. This real-world dataset includes information on inventory flows, sales trends, and vendor interactions, making it ideal for testing each platform’s ability to deliver actionable insights. Below is an overview of the simplified dataset used for the benchmark.

Head-to-Head Comparison Results Summary

The results of the Snowflake Copilot and Lumi AI benchmark test are summarized in the table below. The table scores each platform's performance across easy, medium, and hard queries using the scoring methodology described above.

Platform Score Accuracy
Snowflake Copilot 21/42 50%
Lumi AI 40/42 95.2%

Detailed Results

Difficulty Question Category Test Question Snowflake Co-Pilot Lumi AI
Easy Outstanding Invoices Do we have any invoices that are still outstanding? 1 2
Easy Brand Analysis What are all the distinct brands available? 2 2
Easy Promotion Check Are there any promotions that are currently active? 2 2
Easy Brand & Category What are the top 5 brands and categories that have the highest safety stock? 2 2
Easy Top Vendors Identify the top 5 vendors based on the total quantity of products supplied in 2023. 2 2
Easy Customer Lifetime Value For each customer, what is the count of purchases, average number of orders per month, and average order value based on revenue? 2 2
Easy Payment What is the histogram of days difference with buckets of 1 day for payment date to actual payment? 2 2
Medium Sales and Moving Average For item 1001, what is the total sales for each month in 2023, along with the moving average? Please also include the delta. 2 2
Medium Category Analysis What were the sales for every month in 2023 for premium bikes? 1 2
Medium Discrepancy Analysis Which items have the largest discrepancy when comparing the minimum order quantity to the average monthly sales in 2023? 0 1
Medium Product Performance Which products have shown the highest year-over-year growth in sales from 2022 to 2023? 1 2
Medium Vendor Analysis What items have we purchased from multiple suppliers? Only consider purchases made in 2023. 2 2
Medium Margin Which items do we yield the lowest margins on? Output should be: Item ID, Total Revenue, Total Profit, Total Cost, Margin %. 0 2
Medium Safety Stock Are there any issues with our inventory? 0 2
Hard Customer Activity For customer Globex Corp, for every month from 2022-2023, what is the number of orders for each month along with the cumulative count? 0 2
Hard Lagging Items What are items that have experienced the largest decline in gross profit when comparing the last 3 months to the previous 3 months (use July 1st, 2023 as current date). Output: Item, Item description, Previous 3 Months, Past 3 Months, Delta. 0 2
Hard Supply Analysis Which items have the highest months of supply? Assume the date is July 1st, 2023. 0 2
Hard Order Pattern Detection Which items are most frequently ordered together? 1 1
Hard In Stock Provide the number of in-stock and out-of-stock days in 2023 for each item, and add a column to show the in-stock percentage as well. 0 2
Hard Promotion Can you compare the average weekly sales of item 1016 while on promotion vs. not on promotion by summing the sales and dividing by the number of distinct selling weeks? 1 2
Hard Revenue What are the items that make up 80% of the total order volume in 2023? 0 2

Please click here for more in-depth information on platform results.

Snowflake Copilot: Strengths and Limitations

Snowflake Copilot is designed to assist developers in writing SQL queries within the Snowflake ecosystem. While it has potential in streamlining development tasks, our benchmarking revealed several limitations:

Strengths
  • Developer Assistance: Helps developers with basic SQL query generation, reducing the time spent on writing boilerplate code.
  • Integration with Snowflake: Native integration within the Snowflake environment for those already using the platform extensively.
Weaknesses
  • Lack of Business Context: Often lacks customizable business context, leading to inaccurate responses and query failures. It is limited to a single custom context box capped at 2,000 characters, which restricts the amount of business-specific information that can be provided.
  • Developer-Focused Design: Primarily designed for technical users, making it inaccessible to non-technical team members who could benefit from data insights.
  • Lower Quality Output: Frequently generates SQL queries with syntax errors or inconsistencies, causing delays and requiring manual fixes.

Lumi AI: Strengths and Limitations

Lumi AI distinguishes itself with a multi-agent architecture designed to cater to both technical and non-technical users, incorporating deep contextual understanding and advanced query generation.

Strengths
  • Business Context Integration: Demonstrates exceptional ability to understand and incorporate business-specific context without additional prompting, offering customizable options beyond character limitations.
  • Advanced Query Handling: Excels in complex tasks, generating accurate and efficient multi-CTE queries for advanced analytics.
  • User Accessibility: Designed for users across the organization, from data analysts to business leaders, facilitating broader data-driven decision-making.
  • Iterative Learning: The "Memories" feature allows Lumi AI to learn and adapt over time based on user interactions, providing a personalized and context-rich experience.
  • High Accuracy: Consistently delivers correct answers on the first attempt, significantly reducing the need for manual intervention.
Weaknesses
  • Setup Complexity: Initial configuration is manual for the moment, especially when setting up custom business metrics and KPIs.

Final Thoughts

When comparing Snowflake Copilot and Lumi AI head-to-head, it is evident that each platform serves different needs within the enterprise analytics space. Snowflake Copilot may offer some benefits for developers working within the Snowflake environment, aiding in basic SQL query generation. However, its limitations in understanding business context, handling complex queries, and accessibility to non-technical users significantly hinder its effectiveness as a comprehensive analytics tool.

In contrast, Lumi AI stands out with its ability to deliver advanced analytics and personalized insights across the organization. Its multi-agent architecture and deep integration of customizable business context make it a robust and future-proof solution for enterprises aiming to leverage AI for comprehensive data analysis.

For businesses focused on long-term scalability, advanced analytics, and fostering a data-driven culture accessible to all team members, Lumi AI emerges as the clear winner in this comparison.

Discover the Lumi AI Advantage

Experience how Lumi AI's multi-agent architecture can revolutionize your data and analytics processes with advanced AI-driven insights and automation.

Book your demo today.

Experience the Future of Data & Analytics Today

Make Better, Faster Decisions.
Request Demo