This appendix is a follow-up to the main competitor analysis between Lumi AI and ThoughtSpot, diving into the methodology and detailed results.
For a comprehensive overview of the main conclusions and insights, you can refer to the original article: ThoughtSpot vs. Lumi AI.
Question Evaluation
The benchmarking process evaluated SQL queries across three difficulty levels (Easy, Medium, Hard) based on the number of components involved. These components include aggregate functions for summarizing data, scalar functions for single-value calculations, and window functions for advanced analytics. Additionally questions were assessed that required subqueries for handling complex logic, and proficiency in operations like date and time manipulation, NULL handling, string processing, and table joins. More advanced features like multi-Common Table Expressions (CTEs), set operators (UNION, INTERSECT), control flow statements (CASE, IF), and business context integration are also assessed.
Easy: (fewer than 4 components): Basic queries with limited operations.
Hard: (more than 6 components): Advanced queries that combine several SQL operations and logic.
This approach ensures a robust evaluation of SQL skills across a range of query complexities.
Test ID
Difficulty
Question Category
Test Question
Test Type
SQL Query Components
00001
Easy
Outstanding Invoices
Do we have any invoices that are still outstanding?
Distinct Retrieval
1
00002
Easy
Brand Analysis
What is all the distinct brands available?
Distinct Retrieval
2
00003
Easy
Promotion Check
Are there any promotions that are currently active?
Distinct Retrieval - (no records)
3
00004
Easy
Brand & Category
What are the top 5 brands and categories that have the highest safety stock?
Basic SQL
3
00005
Easy
Top Vendors
Identify the top 5 vendors based on the total quantity of products supplied in 2023.
Basic SQL
3
00006
Easy
Customer Lifetime Value
For each customer what is the count of purchases, average number of orders per month, and average order value based on revenue?
Basic SQL
3
00007
Easy
Payment
What is the histogram of days difference with buckets of 1 day for payment date to actual payment?
Basic SQL
3
00008
Medium
Sales and Moving Average
For item 1001, what is the total sales for each month in 2023, along with the moving average? Please also include the delta.
Medium SQL
4
00009
Medium
Discrepancy Analysis
Which items have the largest discrepancy when comparing the minimum order quantity to the average monthly sales in 2023?
Medium SQL
4
00010
Medium
Vendor Analysis
What items have we purchased from multiple suppliers? Only consider purchases made in 2023.
Medium SQL - (no records)
4
00011
Medium
Safety Stock
Are there any issues with our inventory?
Vague Question
4
00012
Medium
Category Analysis
What were the sales for every month in 2023 for premium bikes?
Nuanced Context - (business context)
5
00013
Medium
Product Performance
Which products have shown the highest year-over-year growth in sales from 2022 to 2023?
Medium SQL
5
00014
Medium
Margin
Which Items do we yield the lowest margins on? Output should be: Item Id, Total Revenue, Total Profit, Total Cost, Margin %.
Medium SQL
5
00015
Hard
Customer Activity
For customer Globex Corp, for every month from 2022 - 2023, what is the number of orders for each month along with the cumulative count?
Advanced SQL
6
00016
Hard
Lagging Items
What are items that have experienced the largest decline in gross profit when comparing the last 3 months to the previous 3 months (use July 1st 2023 as current date)? Output: Item, Item description, Previous 3 Month, Past 3 Months, Delta.
Advanced SQL
6
00017
Hard
In Stock
Provide the number of in-stock and out-of-stock days in 2023 for each item, and add a column to show the in-stock percentage as well.
Advanced SQL
6
00018
Hard
Promotion
Can you compare the average weekly sales of item 1016 while on promotion vs. not on promotion by summing the sales and dividing by the number of distinct selling weeks?
Advanced SQL
6
00019
Hard
Revenue
What are the items that make up 80% of the total order volume in 2023?
Advanced SQL
6
00020
Hard
Supply Analysis
Which items have the highest months of supply? Assume the date is July 1st 2023.
Advanced SQL
8
00021
Hard
Order Pattern Detection
Which items are most frequently ordered together?
Advanced SQL
8
ThoughtSpot Results Summary
This section provides a summary of ThoughtSpot’s performance across the full range of queries used in the benchmark.
Overview
ThoughtSpot Score
14/42
ThoughtSpot Accuracy
33.3%
Difficulty
Question Category
Test Question
Correctness of Answer
Failure Category
Failure Reason
Easy
Outstanding Invoices
Do we have any invoices that are still outstanding?
2
-
Easy
Brand Analysis
What is all the distinct brands available?
2
-
Easy
Promotion Check
Are there any promotions that are currently active?
1
Partial Answer
ThoughtSpot correctly produced a promotions table, however, did not provide a logical answer to the user that there are no current promotions on.
Easy
Brand & Category
What are the top 5 brands and categories that have the highest safety stock?
2
-
Easy
Top Vendors
Identify the top 5 vendors based on the total quantity of products supplied in 2023.
2
-
Easy
Customer Lifetime Value
For each customer what is the count of purchases, average number of orders per month, and average order value based on revenue?
1
Partial Answer
ThoughtSpot only produced the average revenue and did not include the other fields asked for in the question.
Easy
Payment
What is the histogram of days difference with buckets of 1 day for payment date to actual payment?
0
No Answer Generated
ThoughSpot did not correctly produce the requested histogram.
Medium
Sales and Moving Average
For item 1001, what is the total sales for each month in 2023, along with the moving average? Please also include the delta.
0
Excluded Fields
Did not include moving average and calculated delta incorrectly (calculated delta from the previous month's total line value).
Medium
Category Analysis
What were the sales for every month in 2023 for premium bikes?
1
Partial Answer
ThoughtSpot produced the correct answer after more context was provided, highlighting the need for business context.
Medium
Discrepancy Analysis
Which items have the largest discrepancy when comparing the minimum order quantity to the average monthly sales in 2023?
1
Partial Answer
ThoughtSpot generated a partially complete answer after additional context was provided.
Medium
Product Performance
Which products have shown the highest year-over-year growth in sales from 2022 to 2023?
0
Incorrect Calculation
ThoughtSpot transposed years as rows instead of columns, making it impossible to draw business insights.
Medium
Vendor Analysis
What items have we purchased from multiple suppliers? Only consider purchases made in 2023.
0
Incorrect Calculation
ThoughtSpot omitted part of the question. The correct answer should show zero records, as there are no items supplied by multiple vendors.
Medium
Margin
Which Items do we yield the lowest margins on? Output should be: Item Id, Total Revenue, Total Profit, Total Cost, Margin %.
1
Additional Context Provided
ThoughtSpot generated the correct figure after additional context was provided for calculating margin percentage.
Medium
Safety Stock
Are there any issues with our inventory?
0
Incorrect Calculation
ThoughtSpot returned the inventory table without assessing the intent of the question. A correct answer would identify items with an available quantity less than safety stock.
Hard
Customer Activity
For customer Globex Corp, for every month from 2022 - 2023, what is the number of orders for each month along with the cumulative count?
0
Excluded Fields
ThoughtSpot did not calculate the number of orders each month or the cumulative count.
Hard
Lagging Items
What are items that have experienced the largest decline in gross profit when comparing the last 3 months to the previous 3 months?
0
Excluded Fields
ThoughtSpot was unable to generate an answer and omitted part of the question.
Hard
Supply Analysis
Which items have the highest months of supply?
0
Incorrect Calculation
ThoughtSpot did not correctly generate a month of supply figure.
Hard
Order Pattern Detection
Which items are most frequently ordered together?
0
Incorrect Calculation
ThoughtSpot did not correctly perform a bundle analysis.
Hard
In Stock
Provide the number of in-stock and out-of-stock days in 2023 for each item, and add a column to show the in-stock percentage as well.
0
Excluded Fields
ThoughtSpot omitted part of the question and did not produce figures for in-stock days, out-of-stock days, or in-stock percentage.
Hard
Promotion
Can you compare the average weekly sales of item 1016 while on promotion vs. not on promotion?
0
No Answer Generated
ThoughtSpot did not generate an answer for this question.
Hard
Revenue
What are the items that make up 80% of the total order volume in 2023?
1
Partial Answer
ThoughtSpot generated the correct output of items but did not provide the user with a filtered list of items making up 80% of total volume.
Lumi AI Results
This section presents the experiment results for Lumi's performance across the same set of queries.
Overview
Lumi Score
40/42
Lumi Accuracy
95.2%
Difficulty
Question Category
Test Question
Correctness of Answer
Failure Category
Failure Reason
Easy
Outstanding Invoices
Do we have any invoices that are still outstanding?
2
Easy
Brand Analysis
What is all the distinct brands available?
2
Easy
Promotion Check
Are there any promotions that are currently active?
2
Easy
Brand - Category
What are the top 5 brands and categories that have the highest safety stock?
2
Easy
Top Vendors
Identify the top 5 vendors based on the total quantity of products supplied in 2023.
2
Easy
Customer Lifetime Value
For each customer what is the count of purchases, average number of orders per month, and average order value based on revenue?
2
Easy
Payment
What is the histogram of days difference with buckets of 1 day for payment date to actual payment?
2
Medium
Sales and Moving Average
For item 1001, what is the total sales for each month in 2023, along with the moving average? Please also include the delta.
2
Medium
Category Analysis
What were the sales for every month in 2023 for premium bikes?
2
Medium
Discrepancy Analysis
Which items have the largest discrepancy when comparing the minimum order quantity to the average monthly sales in 2023?
1
Incorrect Aggregation
Lumi made an incorrect aggregation on the first attempt, however was able to generate the correct answer with additional context.
Medium
Product Performance
Which products have shown the highest year-over-year growth in sales from 2022 to 2023?
2
Medium
Vendor Analysis
What items have we purchased from multiple suppliers? Only consider purchases made in 2023.
2
Medium
Margin
Which Items do we yield the lowest margins on? Output should be: Item Id, Total Revenue, Total Profit, Total Cost, Margin %.
2
Medium
Safety Stock
Are there any issues with our inventory?
2
Hard
Customer Activity
For customer Globex Corp, for every month from 2022 - 2023, what is the number of orders for each month along with the cumulative count?
2
Hard
Lagging Items
What are items that have experienced the largest decline in gross profit when comparing the last 3 months to the previous 3 months?
2
Hard
Supply Analysis
Which items have the highest months of supply?
2
Hard
Order Pattern Detection
Which items are most frequently ordered together?
1
Incomplete Calculation
Lumi did a partial basket analysis.
Hard
In Stock
Provide the number of in-stock and out-of-stock days in 2023 for each item, and add a column to show the in-stock percentage as well.
2
Hard
Promotion
Can you compare the average weekly sales of item 1016 while on promotion vs. not on promotion?
2
Hard
Revenue
What are the items that make up 80% of the total order volume in 2023?
This article serves as a companion piece to ThoughtSpot vs Lumi, shedding light on the practical outcomes of the tests and how the platforms stack up in real-world data query scenarios. This appendix provides a detailed breakdown of the competitor analysis between Lumi AI and ThoughtSpot, focusing on their performance across a wide range of SQL queries. Through the benchmarking process, we evaluated how each platform handles different query complexities, from basic SQL operations to advanced, multi-CTE queries with nuanced business contexts. The results illustrate clear strengths and limitations for both platforms, offering valuable insights into their capabilities for various use cases.
Join Our Newsletter
Subscribe for the latest on self-service analytics, AI, and data-driven transformations.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
