Introduction

The rise of LLMs has unlocked new possibilities to leverage generative AI in revolutionizing the efficiency of data analytics and business insights. These advancements, while promising, come with a variety of architectural challenges that different platforms address in their unique ways.

This blog provides a detailed comparison of two AI-driven analytics platforms: ThoughtSpot and Lumi AI.

For a deeper understanding of the evolving landscape and the diverse solutions available, refer to AI Tools for Data & Analytics: Understanding the Landscape.

Head-to-Head Summary of Results:

We connected the same dataset and asked the same questions. Here are the results.

ThoughtSpot excels in basic data retrieval and simple SQL tasks but struggles with queries requiring deep business context and advanced calculations. It has limited iterative capabilities in refining outputs through conversation. In contrast, Lumi AI leverages a multi-agent architecture that integrates business context and handles complex queries effectively. It excels in advanced analytics, understands business-specific contexts, and adapts over time through its "Memories" feature.

The article concludes that while ThoughtSpot is suitable for simple, search-driven analytics, Lumi AI is the superior choice for organizations seeking advanced analytics and personalized insights, making it a future-proof solution for comprehensive data analysis.

Overall Platform Assessment

In addition to the performance comparison, below is a comparison of Lumi AI and AI-powered BI tools like ThoughtSpot across several key categories. These categories reflect essential features such as the intended user, the depth of responses, and the ease of integrating business context.

Benchmarking Overview: 

This head-to-head comparison highlights how each tool performs in real-world conditions by connecting the same dataset, asking the same questions, and comparing the outputs.

Analysis & Benchmarking Process Summary:

For this head-to-head comparison, we connected the same dataset and tested each platform with 21 questions of varying complexity, categorized as Easy, Medium, and Hard. These questions simulate typical business scenarios, from basic lookups to complex analytical calculations.

• Easy: Simple tasks involving basic SQL queries and foundational schema knowledge.
• Medium: Multi-step tasks requiring intermediate SQL, or business context.
• Hard: Complex tasks that demand advanced SQL skills and deep business expertise.

_{Please click here for more in-depth information on question evaluation methodology}

Scoring Methodology

Correctness of Answer (0-2 points) (Total 42)

Evaluation of how accurately the AI platform generates the correct SQL or answer on the initial attempt.

• 0 points: The platform did not provide a correct answer on the first attempt (e.g., missing key query logic or producing incorrect results).
• 1 point: The platform delivered the correct answer on second attempt after expanded additional context, or partially correct answer.
• 2 points: The platform delivered the correct answer on its first attempt, with the correct logic, structure, and expected results.‍

Dataset

The dataset used in this benchmarking was sourced from the dynamic retail and supply chain sector, with data spanning from 2020 to 2023. This real-world dataset includes information on inventory flows, sales trends, and vendor interactions, making it ideal for testing each platform’s ability to deliver actionable insights. Below is an overview of the simplified data set used for the benchmark.

Head-to-Head Comparison Results Summary:

The results of the ThoughtSpot and Lumi AI benchmark test are summarized in the table below. The table scores each platform's performance across easy, medium, and hard queries using the scoring methodology described above.

‍

_{Please click here for more in-depth information on platform results}

ThoughtSpot: Strengths and Limitations

ThoughtSpot is known for its combination of AI and traditional visualization capabilities, it has the capability to allow users to search and interact with their data using natural language. However, during our benchmarking, ThoughtSpot demonstrated several limitations:

• Strengths:
  • Simple Data Search: Quick lookups and basic data retrieval were handled effectively.
  • Simple Aggregation: Basic SQL tasks, such as calculating simple moving averages, were executed correctly.
• Weaknesses:
  • Lack of Business Context: ThoughtSpot struggled with questions requiring deeper business context. For example, during the category analysis task, it failed to understand that "premium bikes" referred to a specific product category ("Ultra-Premium Road Bikes") without additional prompting.
  • Advanced Calculations: Complex tasks, such as calculating months of supply or performing basket analysis, either returned incomplete or incorrect results, requiring manual intervention.
  • Limited Iterative Capabilities: ThoughtSpot’s iterative abilities in chat-based workflows were limited, reducing its ability to refine outputs through conversation.

Lumi AI: Strengths and Limitations

Lumi AI sets itself apart with a multi-agent architecture designed to handle both technical and business user needs, incorporating deep contextual understanding and complex query generation:

• Strengths:
  • Business Context Integration: Lumi’s ability to understand and incorporate business-specific context is unparalleled. For example, in the premium bike category analysis, Lumi automatically referenced the correct item category without additional clarification.
  • Advanced Query Handling: Lumi excelled in complex tasks like months of supply analysis and basket analysis, where it generated complex multi-CTE queries and accurately calculated KPIs.
  • Iterative Learning: Lumi’s “Memories” feature allows it to adapt and improve over time based on user feedback, offering a personalized, context-rich experience that ThoughtSpot lacks.
• Weaknesses:
  • Setup Complexity: While Lumi outperforms ThoughtSpot in terms of advanced analytics, the initial setup can be somewhat involved, particularly for configuring custom business metrics and KPIs.

Final Thoughts

When comparing ThoughtSpot and Lumi AI head-to-head, it’s clear that each platform has its own strengths. ThoughtSpot is an excellent choice for organizations looking for simple, search-driven analytics and quick insights. However, when it comes to handling more complex data tasks that require deep business context and advanced calculations, Lumi AI is the superior choice. Its multi-agent architecture and adaptive learning capabilities make it a future-proof solution for enterprises seeking to leverage AI for comprehensive data analysis.

For businesses focused on long-term scalability, advanced analytics, and personalized insights, Lumi AI stands out as the clear winner in this head-to-head comparison.

Discover how Lumi AI's multi agent architecture can revolutionize your data and analytics processes with advanced AI-driven insights and automation. 

Book your demo today.