This appendix is a follow-up to the main competitor analysis between Lumi AI and Snowflake, diving into the methodology and detailed results.
For a comprehensive overview of the main conclusions and insights, you can refer to the original article: Lumi AI vs. Snowflake Copilot.
Question Evaluation
The benchmarking process evaluated SQL queries across three difficulty levels (Easy, Medium, Hard) based on the number of SQL components involved. These components include aggregate functions, scalar functions, and window functions for calculation of simple and advanced metrics. Additionally questions were chosen that required subqueries for handling complex logic, and proficiency in operations like date and time manipulation, NULL handling, string processing, and table joins. More advanced features like multi-Common Table Expressions (CTEs), set operators, control flow statements (CASE, IF), and business context integration are also assessed.
Easy: (fewer than 4 components): Basic queries with limited operations.
Hard: (more than 6 components): Advanced queries that combine several SQL operations and logic.
This approach ensures a robust evaluation of SQL skills across a range of query complexities.
Test ID
Difficulty
Question Category
Test Question
Test Type
SQL Query Components
00001
Easy
Outstanding Invoices
Do we have any invoices that are still outstanding?
Distinct Retrieval
1
00002
Easy
Brand Analysis
What is all the distinct brands available?
Distinct Retrieval
2
00003
Easy
Promotion Check
Are there any promotions that are currently active?
Distinct Retrieval - (no records)
3
00004
Easy
Brand & Category
What are the top 5 brands and categories that have the highest safety stock?
Basic SQL
3
00005
Easy
Top Vendors
Identify the top 5 vendors based on the total quantity of products supplied in 2023.
Basic SQL
3
00006
Easy
Customer Lifetime Value
For each customer what is the count of purchases, average number of orders per month, and average order value based on revenue?
Basic SQL
3
00007
Easy
Payment
What is the histogram of days difference with buckets of 1 day for payment date to actual payment?
Basic SQL
3
00008
Medium
Sales and Moving Average
For item 1001, what is the total sales for each month in 2023, along with the moving average? Please also include the delta.
Medium SQL
4
00009
Medium
Discrepancy Analysis
Which items have the largest discrepancy when comparing the minimum order quantity to the average monthly sales in 2023?
Medium SQL
4
00010
Medium
Vendor Analysis
What items have we purchased from multiple suppliers? Only consider purchases made in 2023.
Medium SQL - (no records)
4
00011
Medium
Safety Stock
Are there any issues with our inventory?
Vague Question
4
00012
Medium
Category Analysis
What were the sales for every month in 2023 for premium bikes?
Nuanced Context - (business context)
5
00013
Medium
Product Performance
Which products have shown the highest year-over-year growth in sales from 2022 to 2023?
Medium SQL
5
00014
Medium
Margin
Which Items do we yield the lowest margins on? Output should be: Item Id, Total Revenue, Total Profit, Total Cost, Margin %.
Medium SQL
5
00015
Hard
Customer Activity
For customer Globex Corp, for every month from 2022 - 2023, what is the number of orders for each month along with the cumulative count?
Advanced SQL
6
00016
Hard
Lagging Items
What are items that have experienced the largest decline in gross profit when comparing the last 3 months to the previous 3 months (use July 1st 2023 as current date)? Output: Item, Item description, Previous 3 Month, Past 3 Months, Delta.
Advanced SQL
6
00017
Hard
In Stock
Provide the number of in-stock and out-of-stock days in 2023 for each item, and add a column to show the in-stock percentage as well.
Advanced SQL
6
00018
Hard
Promotion
Can you compare the average weekly sales of item 1016 while on promotion vs. not on promotion by summing the sales and dividing by the number of distinct selling weeks?
Advanced SQL
6
00019
Hard
Revenue
What are the items that make up 80% of the total order volume in 2023?
Advanced SQL
6
00020
Hard
Supply Analysis
Which items have the highest months of supply? Assume the date is July 1st 2023.
Advanced SQL
8
00021
Hard
Order Pattern Detection
Which items are most frequently ordered together?
Advanced SQL
8
Snowflake Results Summary
This section provides a summary of Snowflake Copilot's performance across the full range of queries used in the benchmark.
Overview
Snowflake Score
21/42
Snowflake Accuracy
50%
Difficulty
Question Category
Test Question
Correctness of Answer
Failure Category
Failure Reason
Easy
Outstanding Invoices
Do we have any invoices that are still outstanding?
1
Partial Answer
Snowflake produced a partial answer, needing additional prompting for actual insights, including customer name and date.
Easy
Brand Analysis
What is all the distinct brands available?
2
-
Easy
Promotion Check
Are there any promotions that are currently active?
2
Easy
Brand & Category
What are the top 5 brands and categories that have the highest safety stock?
2
-
Easy
Top Vendors
Identify the top 5 vendors based on the total quantity of products supplied in 2023.
2
-
Easy
Customer Lifetime Value
For each customer what is the count of purchases, average number of orders per month, and average order value based on revenue
2
-
Easy
Payment
What is the histogram of days difference with buckets of 1 day for payment date to actual payment.
2
-
Medium
Sales and Moving Average
For item 1001, what is the total sales for each month in 2023, along with the moving average? Please also include the delta.
2
Medium
Category Analysis
What were the sales for every month in 2023 for premium bikes?
1
Partial Answer
Snowflake correctly produced the correct answer after more context of category = 1 was provided. Highlighting the need for business context.
Medium
Discrepancy Analysis
Which items have the largest discrepancy when comparing the minimum order quantity to the average monthly sales in 2023?
0
Incorrect SQL
Snowflake did not produce a correct SQL generating a result that returned duplicate items.
Medium
Product Performance
Which products have shown the highest year-over-year growth in sales from 2022 to 2023?
1
Partial Answer
Snowflake produced a partially correct answer in terms of growth numerically however was not able to account for divide by zero errors when calculating percentage.
Medium
Vendor Analysis
What items have we purchased from multiple suppliers? Only consider purchases made in 2023.
2
Medium
Margin
Which Items do we yield the lowest margins on? Output should be: Item Id, Total Revenue, Total Profit, Total Cost, Margin %.
0
Incorrect Calculation
Snowflake incorrectly calculated margin percentage using cost instead of revenue in the denominator.
Medium
Safety Stock
Are there any issues with our inventory?
0
Incorrect Calculation
Snowflake did not return a logical answer with improperly calculated fields, as well as misinterpreted field naming highlighting need for a semantic layer.
Hard
Customer Activity
For customer Globex Corp, for every month from 2022 - 2023, what is the number of orders for each month along with the cumulative count?
0
Incorrect Filtering
Snowflake did not produce the correct answer, filtering a customer name in the id field. When offered additional context created a non-executable query.
Hard
Lagging Items
What are items that have experienced the largest decline in gross profit when comparing the last 3 months to the previous 3 months (use July 1st 2023 as current date). Output: Item, Item description, Previous 3 Month, Past 3 Months, Delta.
0
Incorrect Calculation
Snowflake performed an incorrect date calculation: The last_period CTE in the first query selects data from April 1, 2023, onwards without an upper bound. This means it includes data beyond June 30, 2023, which is outside the scope of the "past 3 months" as per the question.
Hard
Supply Analysis
Which items have the highest months of supply?
0
Not Executable
Snowflake did not produce an executable SQL query.
Hard
Order Pattern Detection
Which items are most frequently ordered together?
1
Incomplete Calculation
Snowflake did a partial basket analysis.
Hard
In Stock
Provide the number of in-stock and out-of-stock days in 2023 for each item, and add a column to show the in-stock percentage as well.
0
Incorrect Calculation
Snowflake performed an incorrect calculation with illogical results, as well as misinterpreted field naming highlighting need for a semantic layer.
Hard
Promotion
Can you compare the average weekly sales of item 1016 while on promotion vs. not on promotion by summing the sales and dividing by the number of distinct selling weeks?
1
Partial Answer
Snowflake gave a partial correct answer, however did not aggregate fully.
Hard
Revenue
What are the items that make up 80% of the total order volume in 2023?
0
Incorrect Join
Snowflake made an incorrect join to the purchase tables, and did not produce any results.
Lumi AI Results
This section presents the experiment results for Lumi's performance across the same set of queries.
Overview
Lumi Score
40/42
Lumi Accuracy
95.2%
Difficulty
Question Category
Test Question
Correctness of Answer
Failure Category
Failure Reason
Easy
Outstanding Invoices
Do we have any invoices that are still outstanding?
2
Easy
Brand Analysis
What is all the distinct brands available?
2
Easy
Promotion Check
Are there any promotions that are currently active?
2
Easy
Brand - Category
What are the top 5 brands and categories that have the highest safety stock?
2
Easy
Top Vendors
Identify the top 5 vendors based on the total quantity of products supplied in 2023.
2
Easy
Customer Lifetime Value
For each customer what is the count of purchases, average number of orders per month, and average order value based on revenue?
2
Easy
Payment
What is the histogram of days difference with buckets of 1 day for payment date to actual payment?
2
Medium
Sales and Moving Average
For item 1001, what is the total sales for each month in 2023, along with the moving average? Please also include the delta.
2
Medium
Category Analysis
What were the sales for every month in 2023 for premium bikes?
2
Medium
Discrepancy Analysis
Which items have the largest discrepancy when comparing the minimum order quantity to the average monthly sales in 2023?
1
Incorrect Aggregation
Lumi made an incorrect aggregation on the first attempt, however was able to generate the correct answer with additional context.
Medium
Product Performance
Which products have shown the highest year-over-year growth in sales from 2022 to 2023?
2
Medium
Vendor Analysis
What items have we purchased from multiple suppliers? Only consider purchases made in 2023.
2
Medium
Margin
Which Items do we yield the lowest margins on? Output should be: Item Id, Total Revenue, Total Profit, Total Cost, Margin %.
2
Medium
Safety Stock
Are there any issues with our inventory?
2
Hard
Customer Activity
For customer Globex Corp, for every month from 2022 - 2023, what is the number of orders for each month along with the cumulative count?
2
Hard
Lagging Items
What are items that have experienced the largest decline in gross profit when comparing the last 3 months to the previous 3 months?
2
Hard
Supply Analysis
Which items have the highest months of supply?
2
Hard
Order Pattern Detection
Which items are most frequently ordered together?
1
Incomplete Calculation
Lumi did a partial basket analysis.
Hard
In Stock
Provide the number of in-stock and out-of-stock days in 2023 for each item, and add a column to show the in-stock percentage as well.
2
Hard
Promotion
Can you compare the average weekly sales of item 1016 while on promotion vs. not on promotion?
2
Hard
Revenue
What are the items that make up 80% of the total order volume in 2023?
This article serves as a companion piece to Snowflake vs Lumi AI, shedding light on the practical outcomes of the tests and how the platforms stack up in real-world data query scenarios. This appendix provides a detailed breakdown, focusing on their performance across a wide range of SQL queries. Through the benchmarking process, we evaluated how each platform handles different query complexities, from basic SQL operations to advanced, multi-CTE queries with nuanced business contexts. The results illustrate clear strengths and limitations for both platforms, offering valuable insights into their capabilities for various use cases.
Join Our Newsletter
Subscribe for the latest on self-service analytics, AI, and data-driven transformations.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
.png)
