SQL (Structured Query Language) has been the standard language for managing and querying relational databases for decades. Relational databases store data in tables and use structured schemas to define relationships between these tables. However, with the increasing complexity of data relationships in modern applications, there has been a growing need to model and query data in ways that go beyond traditional tabular structures.

Graph databases, such as Neo4j, Amazon Neptune, and NebulaGraph, have emerged to address these needs by storing data as nodes (entities) and edges (relationships). They are optimized for traversing and querying complex relationships, making them ideal for applications like social networks, recommendation engines, fraud detection, and network analysis.

In response to the rising popularity of graph databases and the demand for more flexible data modeling, several major SQL database systems have introduced support for graph queries. This includes the ability to create, manage, and query graph structures directly within a SQL database, leveraging the familiar syntax and capabilities of SQL. 

This introduction does bring benefits to organizations looking for unified data management as both relational and graph data can be managed within the same database system, simplifying their data architecture and reducing the need for multiple specialized databases. However, it does not necessarily mean that graph databases will no longer be needed or that they will be replaced. 
Here are several key points to consider regarding the impact of this development on graph databases:

# Data Model Complexity and Flexibility
The graph database is a natural fit for graph data. If the data inherently represents complex relationships and networks (e.g., social networks, knowledge graphs), a graph database provides a more natural and intuitive way to model and query the data. What’s more, graph databases offer more flexibility in evolving the data model. In relational databases, schema changes can be complex and disruptive, whereas graph databases allow for more dynamic and flexible schema evolution.

# Query Language and Usability
Graph databases often provide specialized query languages (e.g., Cypher for Neo4j, Gremlin for TinkerPop) that are designed specifically for graph traversal and pattern matching, making it easier to express complex queries. Since the ISO-GQL was officially released in April this year, the graph database industry is expecting this standard graph query language to provide a unified and standard experience for developers familiar with graph concepts, as using a standard graph-specific query language, can be more intuitive and efficient compared to using SQL extensions for graph queries.

# Real-Time Analytics and Insights
For applications requiring real-time insights and analytics, such as fraud detection or recommendation engines, graph databases are optimized for low-latency query execution and can handle real-time data updates and queries more effectively. Some graph databases are designed to integrate seamlessly with streaming data sources, enabling real-time graph updates and analytics. Apparently, under situations requiring real-time analytics and insights, even though relational databases can perform graph queries, they may not be ideal options.

# Scalability and Distributed Processing
In terms of scalability and distributed processing, relational databases with graph query support might not be competitive to dedicated graph databases. Many graph databases are designed to scale horizontally, distributing data and queries across multiple nodes in a cluster, for instance NebulaGraph. This can be particularly important for large-scale graph applications where the data volume and query complexity require distributed processing. Besides, dedicated graph databases often come with built-in, optimized graph algorithms (e.g., shortest path, centrality, community detection) that can be executed efficiently at scale. 

# Complementary Use Cases
Graph databases are optimized specifically for graph data and queries, offering advanced features such as efficient graph traversals, pattern matching, and handling of complex relationships. While SQL databases with graph query support can handle basic graph operations, dedicated graph databases are likely to remain superior for use cases that require high performance and scalability in graph processing. Therefore, SQL databases with graph capabilities and dedicated graph databases can coexist, each serving different needs and use cases.

# Performance Considerations
While SQL databases with graph query support can handle graph data, they might not match the performance and efficiency of dedicated graph databases, especially for large-scale graph processing and real-time analytics. Graph databases are designed with specific optimizations for graph workloads, such as index-free adjacency, which can significantly improve query performance for certain types of graph operations.

# Conclusion
In summary, while the support for graph queries in SQL databases enhances their capabilities and provides more flexibility, it does not render dedicated graph databases obsolete. Each type of database has its strengths and is suited for different use cases. Organizations will continue to choose the best tool for their specific requirements, and dedicated graph databases will remain crucial for applications that demand high-performance graph processing and advanced graph analytics. The coexistence of these technologies provides organizations with a broader range of tools to meet their diverse data management needs.

While the support for graph queries in SQL databases enhances their capabilities and provides more flexibility, it does not render dedicated graph databases obsolete.

Graph Databases vs. SQL with Graph Queries: Friends or Foes?

In the digital age, recommendation systems have become an integral part of online experience, driving better user experience and engagement. A recommendation system, often integrated with ad serving systems, is designed to suggest highly relevant content or advertisements to users based on users’ profile and engagement history. However, the complexity of user interactions and preferences demands robust data management solutions. Enter graph databases—an innovative approach that enhances the efficiency and accuracy of recommendation systems. This blog explores how recommendation systems work and how graph databases empower recommendation systems, providing real-life examples and architectural insights.

# What are Recommendation Systems and How They Work
## Key Components of Recommendation Systems:
- User Data: Information about users, including their demographics, behavior, preferences, and past interactions. This data is crucial for understanding user interests and making personalized recommendations.
- Content Data: Information about the content or ads available for recommendation. This includes metadata, categories, keywords, and performance metrics.
- Recommendation Algorithms: The core of the system, these algorithms analyze user and content data to generate personalized recommendations. Common types include collaborative filtering, content-based filtering, and hybrid methods.
- Real-Time Processing: The ability to process data and generate recommendations in real-time, ensuring that suggestions are timely and relevant.
- Feedback Loop: Mechanisms to collect user feedback on recommendations (e.g., clicks, likes, shares) to continuously improve the accuracy and relevance of the system.

## Types of Recommendation Systems
Recommendation systems are powerful tools designed to analyze user behavior and preferences to deliver personalized content and suggestions. These systems operate by collecting extensive data on user interactions, such as browsing history, ratings, and purchase activities. Using this data, recommendation systems employ sophisticated algorithms to identify patterns and relationships within the data. 

Two common approaches are collaborative filtering and content-based filtering. Collaborative filtering analyzes the behavior and preferences of similar users to make recommendations; for example, if two users have a history of liking similar movies, the system might suggest a movie liked by one user to the other. Content-based filtering, on the other hand, focuses on the attributes of items and recommends items that are similar to those a user has previously enjoyed. By combining these methods, recommendation systems can generate highly personalized suggestions that cater to individual tastes and preferences. This tailored approach not only enhances the user experience by presenting relevant content but also drives higher engagement and satisfaction, ultimately benefiting both users and businesses.

# What are Graph Databases
## Definition and Structure
Graph databases are a type of NoSQL database designed to handle and represent complex relationships between data entities. Unlike traditional relational databases that use tables to store data, graph databases use nodes, edges, and properties to create a flexible and highly interconnected data structure. Nodes represent data entities, edges denote the relationships between these entities, and properties provide additional information about the nodes and edges. This structure allows graph databases to efficiently model, store, and query intricate relationships and dependencies within data. Graph databases are quite young compared with relational databases and well-known graph databases include Neo4j, NebulaGraph, Amazon Neptune, and TigerGraph.

They excel at managing complex relationships because their inherent design closely mirrors real-world networks, such as social connections, organizational hierarchies, and financial transactions. This enables rapid traversal and querying of relationships, providing powerful insights and enabling applications like fraud detection, recommendation systems, and network analysis to operate with high performance and scalability.

## Architectural Insights

1. Data Modeling:
    * Nodes: Represent users, items, categories, etc.
    * Edges: Capture interactions such as likes, views, purchases, etc.
    * Example: In a movie recommendation system, nodes could represent users and movies, while edges represent viewing history and ratings.
2. Querying:
    * Graph Traversal Queries: Efficiently find relevant items by traversing the graph.
    * Example: To recommend books, a query might traverse from a user node to books they’ve rated highly, then to other users who rated those books highly, and finally to other books those users have rated highly.
3. Performance Optimization:
    * Indexing and Caching: Improve query performance by indexing frequently accessed nodes and caching results.
    * Example: Caching the results of common recommendation queries can reduce latency and improve user experience.

# How Graph Databases Support Better Recommendation Efficiency and Accuracy

Graph databases, such as [NebulaGraph](https://www.nebula-graph.io/), are particularly well-suited for handling the complex relationships and interconnected data involved in both ad serving and recommendation systems. Here’s how they support efficiency:

1. Complex Relationships and Connections:
    * Graph Structure: Graph databases store data in nodes (entities) and edges (relationships), making it easy to model and query complex relationships.
    * Example: Users, ads, and their interactions can be represented as nodes, with edges capturing relationships like views, clicks, and purchases. [LinkedIn](https://www.linkedin.com/) uses graph databases to model professional connections, endorsements, and job recommendations. By understanding the intricate relationships between users, companies, and skills, LinkedIn can provide highly relevant job suggestions and networking opportunities.
2. Real-Time Recommendations:
    * Fast Traversal: Graph databases excel at traversing relationships quickly, enabling real-time generation of recommendations based on user behavior and preferences.
    * Example: Quickly finding similar users or content based on shared attributes or interactions. [Pinterest](https://www.pinterest.com/) leverages graph databases to recommend pins based on user interactions and the connections between pins, boards, and users. This allows Pinterest to deliver personalized content in real-time, enhancing user engagement.
3. Personalization and Targeting:
    * Rich User Profiles: Graph databases can store and query detailed user profiles, including their connections, behaviors, and preferences, allowing for highly personalized ad targeting.
    * Example: Identifying users with similar browsing patterns and recommending ads that performed well with similar users. [eBay](https://www.ebay.com/) uses graph databases to improve its recommendation engine by analyzing user behavior, purchase history, and item relationships. This enables eBay to provide personalized product suggestions, increasing the likelihood of purchases.
4. Scalability:
    * Handling Large Datasets: Graph databases are designed to handle large volumes of interconnected data, making them suitable for high-traffic environments.
    * Example: Managing millions of users and ads, and their interactions, without performance degradation. [Facebook](https://www.facebook.com/)’s social graph, which models the relationships between billions of users, posts, and interactions, relies on graph databases to provide scalable and efficient recommendations. This ensures that users receive timely and relevant content, even as the platform grows.
5. Data Integration and Enrichment:
    * Combining Data Sources: Graph databases can integrate data from various sources, enriching user profiles and improving recommendation accuracy.
    * Example: Combining browsing history, social media activity, and purchase history to create a comprehensive user profile.
6. Feedback Loop and Continuous Learning:
    * Adaptive Algorithms: Graph databases facilitate the continuous updating of user profiles and recommendation algorithms based on real-time feedback.
    * Example: Adjusting recommendations based on recent user interactions to improve relevance.

# Conclusion
In summary, graph databases enhance the efficiency of ad serving and recommendation systems by providing a powerful way to model, store, and query complex relationships. They enable real-time personalization, effective targeting, and scalability, ultimately leading to better user engagement and optimized ad performance. As demonstrated by industry leaders like LinkedIn, Pinterest, eBay, and Facebook, leveraging graph databases can significantly improve user engagement and satisfaction.


This blog explores how recommendation systems work and how graph databases empower recommendation systems, providing real-life examples and architectural insights.

From Data to Insight: Leveraging Graph Databases for Superior Recommendations

In the ever-evolving digital landscape, the choice of a database is not just a technical decision; it's a strategic one. The right database can be the cornerstone of a business's success, while the wrong choice can lead to inefficiencies, data loss, and even business failure. In this article, let’s explore the critical factors that should guide your decision when selecting a database for modern applications.
# Part 1: The Database Product
## Data Model
The first step in choosing a database is understanding the data model that aligns with your business needs. Whether you're dealing with structured, semi-structured, or unstructured data, the data model should reflect the nature of your data and how it will be used. Relational databases are suitable for structured data, document databases for flexible data models, and graph databases for scenarios that require analyzing complex relationships. Each type of database has its unique strengths and optimal use cases, and the choice should be made considering the specific requirements of the business.

Relational databases organize data into tables with relationships defined by primary and foreign keys. They are ideal for structured data that fits neatly into rows and columns, such as financial transactions, customer information, or inventory records. Graph database, like Neo4j or NebulaGraph, which uses nodes, edges, and properties to represent and store data. They are excellent for managing complex networks of connections, such as social networks, recommendation engines, or fraud detection systems. For instance, an e-commerce platform might store product information, user reviews, and purchase history in documents. The flexibility of document databases allows the platform to change the data structure, such as adding new fields for product ratings or social media links, without significant reworking of the database schema.

## Scalability
As your business grows, so will your data. Scalability is crucial for future-proofing your database. As businesses grow, the data needs will inevitably expand, and choosing a database that can scale efficiently and cost-effectively is crucial for maintaining performance, reliability, and overall user satisfaction. Netflix's migration from Oracle to Cassandra is a well-documented case that highlights the importance of scalability when selecting a database for business. The shift allowed them to handle the exponential growth of their user base and data volume without compromising performance. 
- **Handling Growth**:
Businesses need to anticipate growth in data volume and user load. A scalable database ensures that the system can handle increased demand without significant rearchitecting or downtime.

- **Performance**:
As the user base grows, the performance of the database can directly impact the user experience. A scalable database maintains performance levels by distributing load effectively.

- **Cost Management**:
Scalable solutions often provide more cost-effective ways to grow. Horizontal scaling with commodity hardware can be cheaper than vertical scaling with high-end servers.

- **Reliability and Availability**:
Scalable databases often come with built-in mechanisms for fault tolerance and high availability, which are crucial for maintaining service levels and user trust.

## Performance
Database performance is integral to a business’s success across various dimensions, including customer experience, operational efficiency, real-time analytics, resource optimization, and customer support. Investing in high-performance databases and regularly optimizing them can yield significant benefits and provide a competitive edge in the market. A database that can't deliver quick query responses will slow down your applications and frustrate your users. For example, e-commerce websites like Amazon or eBay rely heavily on database performance. When customers search for products, add items to their cart, or complete a purchase, these actions interact with the database. If the database is slow, customers may experience delays, leading to frustration and potentially abandoning their shopping carts. Financial institutions like JPMorgan Chase or Goldman Sachs rely on real-time data analytics for trading, risk management, and customer transactions. An underperforming database can delay these analytics, leading to slower decision-making and potential financial losses. For instance, in high-frequency trading, even milliseconds of delay can translate into significant financial impacts.

## Reliability
Reliability refers to the ability of a database system to perform its functions correctly and consistently over time, especially in the face of failures or unexpected conditions. It is about the database's durability and consistency. Consistency ensures that data is accurate and reliable across all nodes in a distributed system. Companies like Walmart use data analytics to track sales, manage inventory, and forecast demand. Reliable data is essential for these analytics to be accurate and actionable. Take the case of Uber, which switched from PostgreSQL to a custom MySQL setup to ensure data consistency across its global operations, despite the high write and read demands. 

​​Investing in a reliable database system is not just a technical necessity but a strategic business decision. By ensuring that their databases are reliable, businesses can not only avoid potential pitfalls but also gain a competitive edge in their respective industries.

## Security
The security of a database is paramount for any business, as it affects a wide range of operational, legal, and reputational aspects. Here are some key reasons why database security is crucial, along with real-life examples to illustrate each point:
- **Protection of Sensitive Information**
Databases often contain sensitive information such as personal data, financial records, and proprietary business information. Securing this data is essential to prevent unauthorized access and misuse.
Companies like Equifax store vast amounts of personal and financial data. In 2017, Equifax experienced a data breach that exposed sensitive information of 147 million people. The breach led to significant financial losses and reputation damage.
- **Compliance with Regulations**
Various industries are subject to stringent regulations regarding data protection. Non-compliance can result in hefty fines and legal repercussions.
Hospitals and clinics must comply with HIPAA regulations in the United States. A breach of patient data can lead to substantial fines and legal action. For example, Anthem Inc. faced a $16 million settlement in 2018 after a data breach exposed the personal information of nearly 79 million people.
- **Preventing Financial Loss**
Data breaches can lead to direct financial losses from theft, fraud, and the costs associated with addressing the breach. Additionally, businesses may face indirect losses from reputation damage.
- **Ensuring Business Continuity**
A security breach can disrupt business operations, leading to downtime, loss of productivity, and operational inefficiencies.

# Part 2: Cost, Community and Support
## Cost
When selecting a database solution, cost is often a primary concern for businesses. The choice between open-source and commercial editions can significantly impact the overall expense. Open-source databases, such as MySQL or PostgreSQL, are generally free to use, which can be appealing for startups and small businesses with limited budgets. However, the hidden costs associated with open-source solutions often lie in the need for skilled personnel. Companies may need to hire expensive experts to handle installation, customization, and ongoing maintenance, which can add up over time. On the other hand, commercial database solutions like Oracle or Microsoft SQL Server come with licensing fees that can be substantial. However, these costs are often offset by the comprehensive support and maintenance provided by the vendor, reducing the need for in-house expertise and allowing businesses to focus on their core operations.

## Community and Support
The availability of community and support can make or break a database solution for many businesses. Open-source databases benefit from vibrant communities of developers and users who contribute to forums, documentation, and code repositories. This can be an invaluable resource for troubleshooting and best practices. However, relying solely on community support can be unpredictable and time-consuming, especially for mission-critical applications. Commercial databases, on the other hand, come with dedicated support teams from the vendor, offering guaranteed response times and expert assistance. This level of support can be crucial for businesses that require high availability and cannot afford downtime. Additionally, commercial vendors often provide training and certification programs, ensuring that your team has the skills needed to effectively manage the database.

# Part 3: Pitfalls to Avoid

When selecting a database for modern applications, companies often encounter several pitfalls that can lead to inefficiencies, higher costs, and even data loss. Here are some common pitfalls and strategies to avoid them:

- **Ignoring Business Needs**: Choosing a database without aligning it with the specific needs of the business is a common mistake. For instance, not considering the nature of the data, the expected data volume, or the required query performance can lead to a system that doesn't meet the application's requirements.

- **Underestimating Scalability**: As businesses grow, so does their data. Failing to select a database that can scale horizontally or vertically to accommodate this growth can lead to performance bottlenecks and increased costs down the line.

- **Neglecting Performance Optimization**: Performance issues can arise if a database is not properly optimized. This includes not only the database configuration but also the queries, indexing strategies, and hardware resources.

- **Overlooking Data Quality**: Poor data quality can lead to inaccurate insights and decision-making. Companies must implement data validation rules and regularly cleanse their data to maintain accuracy and consistency.

- **Insufficient Security Measures**: Data breaches can be costly. It's crucial to choose a database that supports robust security features, including encryption, access controls, and regular security updates.
Lack of Disaster Recovery Planning: Without a solid backup and recovery plan, companies risk losing data due to hardware failures, human errors, or natural disasters. Regular backups and a clear recovery strategy are essential.

- **Inadequate Resource Allocation**: Databases require maintenance and management. Not allocating sufficient resources, including personnel and budget, can lead to poorly managed databases that don't perform as needed.

- **Rigid Data Models**: Some databases have rigid data models that don't accommodate changing business requirements well. It's important to select a database that can adapt to evolving data structures and relationships.

To avoid these pitfalls, companies should conduct a thorough analysis of their data requirements, projected growth, and compliance needs. By taking a comprehensive approach to database selection, companies can choose a solution that will support their business needs both now and in the future.

# Conclusion
Selecting a database is a complex decision that requires a deep understanding of your business needs and the capabilities of the database solutions available. By considering factors like data model, scalability, performance, consistency, reliability, security, cost, you can make an informed decision that will serve your business well into the future. Remember, the database is the backbone of your applications; choose wisely.

This article explores the critical factors that should guide your decision when selecting a database for modern applications.

Database Selection: Critical Factors for Modern Applications

NebulaGraph Enterprise v5.0 is **the first and now the only** distributed graph database that implements the ISO/IEC 39075: GQL, the first international graph query language standard. As standardized by GQL and promoted by NebulaGraph in all previous versions, the latest version adopts the property graph model for managing and storing user data mainly as vertices (nodes) or edges, each containing a set of labels and properties. Users get to use the market-proven MATCH (graph pattern matching) clauses in addition to a rich set of other SQL-like clauses to query such a property graph.

# GQL
Similar to previous versions of NebulaGraph, the schema of a property graph is to be created or altered by users and managed by the database. The following code snippet is an example of a DDL query written in GQL that tries to create a graph type containing a set of node types (i.e., City, Country, Continent, etc.) and edge types (i.e., CITY_IS_PART_OF_COUNTRY, COUNTRY_IS_PART_OF_CONTINENT, etc.). Each node or edge type has its type name (e.g., City), a set of optional labels (e.g., City, Place) and a set of optional properties (e.g., id, name, url).

```
CREATE GRAPH TYPE IF NOT EXISTS ldbc_type AS {
  NODE City (LABELS City&Place {id INT64 PRIMARY KEY, name STRING, url STRING}),
  NODE Country (LABELS Country&Place {id INT64 PRIMARY KEY, name STRING, url STRING}),
  NODE Continent (LABELS Continent&Place {id INT64 PRIMARY KEY, name STRING, url STRING}),
  ...
  EDGE CITY_IS_PART_OF_COUNTRY (City)-[:IS_PART_OF]->(Country),
  EDGE COUNTRY_IS_PART_OF_CONTINENT (Country)-[:IS_PART_OF]->(Continent),
  EDGE UNIVERSITY_IS_LOCATED_IN_CITY (University)-[:IS_LOCATED_IN]->(City),
  }
```

Once a graph type is created, we can use it to create as many graphs as we want, all of which would be confined to the same graph type. In the following, we create a graph with the name LDBC using the ldbc_type created above.

```
CREATE GRAPH IF NOT EXISTS LDBC TYPED ldbc_type
```

Assuming that we have loaded the LDBC graph with nodes and edges, we can query it using MATCH queries like the following that first declares two path patterns to be matched from the graph, order the results by the [person.id](http://person.id/) and [message.id](http://message.id/) properties, limit the cardinality of the result set to 20 and return a set of properties of interest.

```
MATCH
  (person:Person)-[:KNOWS]->(friend:Person),
  (message:Message)-[:HAS_CREATOR]->(friend)
ORDER BY
  person.id DESC,
  message.id ASC
LIMIT
  20
RETURN
  friend.id AS personId,
  friend.firstName AS personFirstName,
  friend.lastName AS personLastName,
  message.id AS postOrCommentId
```
For a concise description of GQL, please refer to [GQL Database Language](https://jtc1info.org/slug/gql-database-language/e9bddae) published by ISO and [Understanding GQL: A Comprehensive Overview of the Standard Graph Query Language](https://www.nebula-graph.io/posts/An_Comprehensive_Overview_of_the_Standard_Graph_Query_Language).

# Internals
Compared with the previous NebulaGraph series, we have made fundamental technical advancements in all components of Enterprise v5.0. The following diagram compares the two architectures. The internals of the v3.0 and the Enterprise new are placed on the left- and right-hand side, respectively. For the new version, the green boxes are new features and the red boxes are enhanced features over their counterparts in previous versions.

![image](https://www-cdn.nebula-graph.io/blogs/version%20comparison.png)

- Full-Fledged Support for GQL
  - **New GQL frontend** that parses and validates the GQL queries, conforming to the GQL standard.
  - Deprecate previous vendor-locked query languages.
- Distributed Architecture
  - In the market, NebulaGraph has been known for its **distributed architecture**, the computing and storage capacity of which scales out nicely to cover even 200 TB of data of huge graphs in a single cluster. The new version honors the same architecture with major enhancement on the implementation of **Raft**, a consensus protocol for distributed systems.
  - The Enterprise v5.0 **cluster** consists of a set of stateless **graphd** nodes and a set of **storaged** nodes organized in Raft groups. The metadata of each such cluster is managed by a **metad** service, consisting of metad nodes also organized in Raft groups. A single metad service can serve multiple clusters, if needed, in an isolated manner, all of which are referred to as a **supercluster**. This architecture is flexible and has been proven by the market to fit the needs of data management in enterprises.
- Planner & Optimizer
  - **Enterprise-grade RBO** (rule-based optimizations): The optimization of new version has a rich set of optimization rules including both the market-tested ones inherited from the previous series and new rules optimized for GQL. The new does not start itself from ground zero but from the shoulder of the old series.
  - Support **graph-native operators** that are specially designed and optimized for graphs in the query plans (e.g., breadth-first search in a graph) in addition to conventional relational algebra operators (e.g., joins).
- Data Structures
  - **Compact Vectorized Layout**: Within the memory, we store the data of graph elements (nodes and edges) in the newly designed Nebula Vector data structure that is compatible with Apache Arrow. Vector uses a compact layout to encode and store all data types such as strings, lists, records and so on, saving the memory overhead by up to 95% compared with the previous series.
  - **No serialization/deserialization RPC overhead**: The compact layout of vectors is RPC-friendly. The vectors are essentially a set of self-contained and sequentially stored data chunks in the main memory with no pointers. They can be passed via the RPC directly without any need to serialize or deserialize the data or allocating temporary memory spaces for this process.
  - **Memory Graph**: We now support storing graphs using adjacency lists directly in the memory. This graph-native structure allows fast graph traversals and property retrievals by accessing the graph topology and properties directly in the memory, supporting queries that are orders of magnitude faster than those constrained by network & disk I/Os and KV encoding/decoding. We have exciting features based on this memory graph in future versions.

- Executor
  - **Vectorized Executor**: Instead of processing each row one by one, the new version now fully implements all operators in the vectorized way that applies operations on all values in a vector simultaneously. This allows the executor to exploit the hardware benefits of modern multi-core processors and DRAMs. As an example, the vectorized hash join operator is now 2-3 times faster than that in the previous series.
  - **Fully concurrent and pipelined runtime**: Instead of processing each operator one by one using only one thread, the Enterprise v5.0 now splits a query plan into multiple pipelines that are executed concurrently. Within each pipeline, data in the form of vectors streams from the first source operator to the final pipeline breaker without being blocked. And, at each point during the execution, there can be as many concurrent pipelines as needed and tuned. This runtime improves the workload balancing and exploitation of multi-cores significantly and addresses the NUMA problem in modern multi-socket multi-core processors.
  - **Sub-query plan pushdown**: NebulaGraph Enterprise v5.0 can now push a complete sub-query plan from a graphd to multiple storaged for near-data processing (NDP). Previously, the v3.0 series benefited greatly from pushing some filters/predicates down to the storaged for performance improvements and network savings, despite that not all possible push-downs were implemented. We have greatly enhanced this feature from pushing down some filters/predicates to complete sub-query plans containing all possible operators, filters/predicates, etc. The push-downed sub-query plan is processed by the same pipelined and vectorized executors in the storaged.
  - **Fine-grained memory management**: To prevent OOM failures, we have implemented a fine-grained memory tracker that can track the memory usage of each query, each operator as well as small objects in the runtime. In the same way as in old series, users are free to set memory budgets and prevent the processes from being killed by OOM.
- Storage Engine
  - **Optimized KV Encoding**: We have optimized the KV encoding according to our latest understanding of how graph queries lookups, order and manipulate the underlying KV data. A new format is implemented to optimize prefix scans for nodes and edges.
  - **Bucket in partitions**: Like in the previous series, we continue to partition graph data in the shared-nothing architecture. In the new version, we split a partition into fixed-size buckets. All graph elements are mapped to corresponding buckets deterministically. The design allows faster data balancing and migration since data only needs to be moved at the bucket level. And, the new supports altering the number of partitions after creating a graph.

- Other Features
  - **Plugin**: We now support plugins into the database kernel, as in popular databases like PostgreSQL. Quite some important features have been implemented as built-in plugins such as functions, procedures, authentications, etc. Developers can implement their own plugins in C++ and load them into the database kernel.
  - **Procedure**: We support the development of procedures in C++. Users can call them in GQL queries. Procedures usually have better performance than plain queries. And, they can be reused once created. Users can also develop graph algorithms in procedures. We will support the development of GQL-based, Python-based procedures in future versions.
  - For a complete feature list, please wait for our upcoming release note.

# Performance
Now let’s look at how Enterprise v5.0 performs in comparison with the previous series.
The following chart shows the speedup of the new for the LDBC SNB Interactive benchmark, using the 99-th percentile of query latencies as the metric, on the same setup (a three-node cluster, 64-core Intel 8352Y CPU, 2.20GHz, 256 GB DRAM and 2 TB SSDs), with a scale factor (SF) of 100. **Enterprise v5.0 is 1.01 to 6.43 times faster, with an average speedup of 3.01. All the queries executed on the new are written in GQL**. For the old version, the queries are written in openCypher. Both languages are based on the same graph pattern matching theory, with GQL released as the international standard. It has always been our belief that users shall only need to express their queries using declarative query languages and enjoy a good level of performance with ease, rather than being forced to write queries as procedures in C++. C++ procedures are good patches to the database. But it would be a nightmare if some queries could only be achieved with it because they are hard to code, hard to debug, hard to maintain and hard to scale.

>Disclaimer: We used the LDBC SNB benchmark as a starting point. However, the test results aren’t audited, so we want to be clear that this is not an LDBC Benchmark test run, and these numbers are not LDBC Benchmark Results.

![image](https://www-cdn.nebula-graph.io/blogs/query%20latency.png)

If we zoom in on the Q1 for example (GQL query shown in the following) on which the new version is **3.82** times faster, v5.0’s peak memory footprint of all processes is **64.47%** smaller than that of previous one. So, in a nutshell, the Enterprise v5.0 can use around 1/3 of the memory resources to produce more than 3 times faster performance in this case.

```
USE ldbc
MATCH 
  (p:Person{id:2}),
  (friend:Person{firstName:"Kyle"})
WHERE 
  p<>friend
MATCH 
  path1 = ANY SHORTEST PATH (p)-[:KNOWS]-{1,3}(friend)
RETURN 
  min(length(path1)) AS distance,
  friend GROUP BY friend
NEXT 
USE ldbc
ORDER BY
  distance ASC,
  friend.lastName ASC,
  friend.id ASC
LIMIT
  20
MATCH
  (friend)-[:IS_LOCATED_IN]->(friendCity:City)
OPTIONAL MATCH
  (friend)-[studyAt:STUDY_AT]->(uni:University)-[:IS_LOCATED_IN]->(uniCity:City)
RETURN 
  collect(CASE WHEN NOT uni.name IS NULL THEN RECORD 
      {uniName:uni.name,
      studyAtClassYear:studyAt.classYear, 
      uniCityName:uniCity.name} END
    ) AS unis,
  friend,
  friendCity,
  distance GROUP BY friend,
  friendCity,
  distance
NEXT 
USE ldbc
OPTIONAL MATCH
  (friend)-[workAt:WORK_AT]->(company:Company)-[:IS_LOCATED_IN]->(companyCountry:Country)
RETURN
  collect(CASE WHEN NOT company.name is null THEN RECORD 
      {companyName:company.name,
      workAtWorkFrom:workAt.workFrom,
      companyCountryName:companyCountry.name} END
    ) AS companies,
  friend,
  unis,
  friendCity,
  distance GROUP BY friend,
  unis,
  friendCity,
  distance
NEXT
USE ldbc
ORDER BY
  distance ASC,
  friend.lastName ASC,
  friend.id ASC
LIMIT 20
RETURN
  friend.id AS friendId,
  friend.lastName AS friendLastName,
  distance AS distanceFromPerson,
  friend.birthday AS friendBirthday,
  friend.creationDate AS friendCreationDate,
  friend.gender AS friendGender,
  friend.browserUsed AS friendBrowserUsed,
  friend.locationIP AS friendLocationIp,
  friend.email AS friendEmails,
  friend.speaks AS friendLanguages,
  friendCity.name AS friendCityName,
  unis AS friendUniversities,
  companies AS friendCompanies
```

# Summary
**ALL IN GQL**:  The upcoming NebulaGraph Enterprise v5.0 is the first and now the only distributed GQL database. For the first time in the history of graph databases, we are thrilled to welcome the arrival of GQL. It not only allows users to develop applications in a standardized way without worrying about using vendor-locked languages but also enables us to develop a full-fledged graph database with both the latest database technologies and comprehensive grammar support for graph-native features.

**Faster with Less Resources**: Evaluated using the LDBC SNB benchmark, the latest NebulaGraph Enterprise version is on average 3 times faster and around 3 times more memory efficient than the previous state-of-the-art enterprise version. We will release more evaluation results in more scenarios as the GA of the new Enterprise version approaches.

**If you're interested in take a trial, feel free to [reach out](https://www.nebula-graph.io/contact) to us.**

**About the author**

[Dr. Xuntao Cheng](https://www.linkedin.com/in/chengxt/) is the R&D leader of Vesoft and a member of the ISO/IEC JTC1/SC 32/WG3 Database Language expert group.


This blog dives into the new features and functions provided by NebulaGraph Enterprise.

Technical Preview of NebulaGraph Enterprise v5.0

Following our previous discussion on [LDBC data generation](https://www.nebula-graph.io/posts/the_secret_fair_data_evaluation_and_repeatable_results_LDBC_SNB), this blog post primarily focuses on how the query within the SNB Workload is created, and how the Driver executes the workload during performance testing.

**Workload**

After the preparation phase of the benchmark test, three types of data are generated:
- Dataset: used for Data from initial import
- Update Streams: used for real-time updates by the Driver during the performance testing phase
- Parameters: used for the Driver's read requests during the performance testing phase

# Write Query
Given the complexity of the data generated by SNB, it is not feasible to generate update requests in real-time during performance testing through the SNB Driver. Therefore, the data for real-time updates is actually generated by DataGen. As we mentioned in the previous post, when using LDBC, DataGen needs to specify how long the data is to be generated for, and from which year social activities begin. DataGen will split all the data based on timestamps, using 90% of the data as the Dataset, and the remaining 10% as Update Streams for subsequent write requests.

For all write requests, DataGen will schedule the time of execution for the update query, and generate the related update parameters into a JSON file. We will explain later why it is necessary to specify the query scheduling time.

# Read Query
The read requests in SNB are divided into two main categories: 14 complex read-only queries and 7 short read-only queries.
Complex read queries start with a given person, querying the relevant information within a one-hop or two-hop subgraph. For example, the query pattern of Interactive Complex 9 (IC9) is as follows: given a starting person to retrieve the 20 most recent messages created by this person's friends or friends of friends, as shown below.
![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-17-2.webp.jpg)

Short read queries are divided into two categories: one is to query a person’s name, age, and other information, or the friends of this person and the messages sent by these friends. The other is to retrieve the message’s sender and the related information, similar to the diagram below.
![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/204-04-17-3.webp.jpg)

The complete request description can be found in Chapters 6.1 and 6.2 of "The LDBC Social Network Benchmark Specification".

**Request Parameters**

In addition to Dataset and Update Streams, DataGen also generates parameters for different read requests, such as the starting PersonId for IC9 above. To ensure the results of performance testing to be reliable, SNB requests that the parameters generated for read requests meet the following properties:

1. Query time has an upper limit: The data involved in Complex Read and Short Read requests will vary depending on the starting point, but from a macro perspective, the subgraphs they query are finite, so there will be a reasonable range for their query time.
2. Query execution time meets a stable distribution: The query data volume will vary for different starting points of the same read request, but the final query time distribution is stable.
3. Identical logical plan: Given a query, no matter how different the starting points are, the logical query plan generated by the optimizer is the same. This property ensures that each query can test some technical difficulties in the graph database, such as a large number of JOINs, cardinality estimation, etc..

The first two properties are relatively easy to understand, but what about the last one? We all know that an excellent optimizer will determine the JOIN order during actual execution based on query parameters. Even for the same query pattern, the final execution plan can vary greatly due to different query parameters. Let's see how DataGen selects parameters to enable the same query to ultimately adopt the same query plan.

Firstly, it is known that the query data volume and query time are strongly correlated. For each query pattern, LDBC has an ideal execution plan during its design, in which the data volume can be pre-calculated based on the starting point.

Take Complex 2 as an example.
![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-17-4.webp.jpg)

When retrieving the most recent 20 messages sent by a given person's friends, the ideal query plan is as follows.

![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-17-5.webp.jpg)

For this ideal execution plan, we can preprocess to get the number of rows of the JOIN intermediate result for each starting point. For example, for the starting point of PersonID=1542, the number of intermediate results of the two JOINs are 60 and 99 respectively. Ultimately, for each Query, a table of the starting point and the number of rows of the intermediate JOIN result, called the Parameter-Count table, will be generated.

The Parameter-Count table is generated by DataGen during the data generation process. With this table, selecting parameters actually turns into choosing the most similar rows in the table. The method used in this step is a greedy heuristic search: first, select the most similar rows based on the number of Join1 intermediate results (rows where Join1 value is 60), then select from these rows based on the number of Join2 intermediate results (resulting in rows where Join2 values are 99, 102, 103), finally filtering out PersonID 1542, 1673, 7511.

One concern that may raise is that, logically, the number of parameters to be selected, k, should be determined first, then the Parameter-Count table should be screened. Otherwise, as the paper suggests, this could lead to the final number of selected rows being less than k. This has not been investigated in depth here, but the overall approach is reasonable.

Through the above steps, for each Query type, several most similar parameters will be selected to make the query plan as similar as possible. However, a question remains, how to determine how many parameters each query needs, i.e., the quantity of different queries?

# Query Mix
First of all, the LDBC SNB Interactive Workload expects the execution time ratio of Update Query, Complex Query, and Short Query during the performance test to be 10%, 50%, and 40% respectively. This ratio is designed to simulate the real load of social networks, where read requests generally far outnumber write requests, and to realistically reflect the performance of the database.

Although in the previous step, DataGen ensures that the execution times of different parameters in the same query were similar. However, there are huge differences in the data volume and execution data of different queries. Therefore, in order to avoid a certain type of query playing a decisive role in the final test results, the quantity of different types of queries needs to be carefully designed.

The SNB Interactive Workload mainly believes that the Complex Read Query should dominate the performance test scores. The number of Complex Read Queries is determined by the number of Update Queries, as shown in the table below.

![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-17-6.webp.jpg)

For example, in SF100, a Complex Read Query 9 should be executed after every 527 Update Queries. As mentioned before, the scheduling time for each Update Query has been determined by DataGen, so the scheduling time for each Complex Read is also set. **Note that the scheduling time can be understood as a relative time compared to the start time of the performance test, which will actually be affected by the SNB Driver parameters.** The reason it is a scheduling time rather than an execution time is that the database being tested may already be fully loaded or unresponsive, causing a Query that has reached its scheduling time to be unable to execute in a timely manner.

This is where we can understand why SNB claims that its test results are repeatable. The key lies in the deterministic data generated by DataGen. Once the scheduling time of the Update Query is determined, it also sets the scheduling time of the Complex Read Query, thereby ensuring the repeatability of its performance test results. Understanding this is crucial for subsequent comprehension of performance test results and relevant parameters of the SNB Driver.

Meanwhile, Short Queries are interspersed after each Complex Query. As mentioned before, Short Queries fall into two major categories: querying person-related information or message-related information. To be precise, different types of Complex Queries may return different information. Based on the results of the Complex Query, a type will be selected from the Short Queries, and the result of the Complex Read, such as identifiers of Person or Message, will be used as parameters for the Short Query.

After a Short Query is scheduled, another Short Query may be scheduled subsequently, with the probability decreasing each time. That is, a string of Short Queries may be scheduled after a Complex Query, referred to as Short Read Random Walk in the paper. Different Queries may trigger different types of Short Queries, as shown in the table below. For instance, only IS1, IS2 and IS3 can be triggered after IC10.

![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-17-7.webp.jpg)

IC in the table stands for Interactive Complex query, and IS stands for Interactive Short query.

Unlike Complex Queries, the scheduling time of Short Queries is dynamically determined by the Driver based on the completion time of the Complex Read it depends on. However, the number of Short Queries triggered is determined by the aforementioned probability. Since all the random generators used during the performance testing process are the same, the proportion and order of all Queries in the entire benchmark testing process are fixed.

**Driver**
The Driver relies on the Dataset, Update Streams, and Parameters generated by DataGen, and it mainly accomplishes two tasks:

- Validate: to check the correctness of the database
- Benchmark: to test the performance of the database

Validation is relatively straightforward. Based on the Update Streams and Parameters, several parameters are generated for validation. To verify the correctness of the database results, validation depends on executing queries in a determined order, so it can only be performed in a single thread. The Driver will ultimately output whether the database has passed the validation.

Benchmarking, on the other hand, generates a determined Workload, including an Operation sequence that contains Query and corresponding scheduling times, based on the Dataset, Update Streams, Parameters, and relevant configurations. According to the scheduling times, Workload is sent in sequence to the database being tested, while also their scheduling times, start execution times, and end execution times are saved, ultimately outputting a performance report.

The Driver is mainly composed of the Runtime, Workload, Database Connector, and Report.

# Runtime & Workload
Regardless of the final performance test environment or the graph database being tested, the Workload it generates will be the same as long as the Driver's input (data and configurations) is the same. To achieve the highest throughput, the Workload is often broken down into multiple Streams. For graphs, multiple operations have dependencies, such as the existence of a friend before a friendship can be added. Therefore, the Driver supports execution of this sequence by multiple threads (controlled by the thread_count parameter), but it also needs to ensure that even with multiple threads executing, the dependencies between operations still satisfy the scheduling time relationships generated by DataGen. The principle of this part is not elaborated, but those interested can read the relevant section in the paper.

Each Operation in the Workload includes the following information:
- Query type
- Query parameters
- Scheduling time
- Expected result (used only in Validate)

As mentioned before, all the scheduling times for Update Query and Complex Query are predetermined by DataGen. The Driver sends the Query to the database for execution via the Database Connector according to the scheduling time. Obviously, if these scheduling times are all a fixed value, we cannot know the real performance of the database.

In the Driver, the parameter `time_compression_ratio` (default is 1) can stretch or compress the entire Workload in the time dimension. For example, a `time_compression_ratio` of 2.0 means that the running duration of this benchmark will become twice the default, that is, the interval time between Queries will become twice the original. Similarly, a`time_compression_ratio` of 0.1 means that the speed at which the Driver sends Queries is 10 times the default. In short:

- time_compression_ratio > 1, slower than default
- time_compression_ratio < 1, faster than default

Note that modifying the `time_compression_ratio` will not affect the ratio of different types of Queries in the Query sequence, the Query parameters, and the execution order.

**Database Connector**

To interface a database with SNB, vendors need to implement the corresponding Database Connector with the primary task to connect to the target database:
1. Convert the Query type and parameters of the Driver framework into the Query of the target graph database.
Send the Query to the target graph database through the Database Connector.
2. This part only needs to implement several interfaces of the Driver, referring to: https://github.com/ldbc/ldbc_snb_interactive_v1_driver and the related integration documentation.

# Driver Configuration
The main purpose of the benchmark is to achieve the `optimal time_compression_ratio` (the smaller, the higher the performance) while ensuring a timeliness rate of more than 95%. An Operation is considered timely if it is scheduled by the Driver within 1 second after the expected scheduling time:

actual_start_time − scheduled_start_time < 1 second

Moreover, a valid Benchmark takes at least two hours while the performance during the half-hour warm-up period is not included in the results.

![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-17-7.webp.jpg)

To run a valid Benchmark, the Driver has several important parameters to adjust:
- time_compression_ratio: Adjust the execution speed of the Workload, more precisely, the expected execution speed.
- operation_count: The total number of Operations that the Driver needs to generate, modify with - time_compression_ratio, to ensure the duration of the Benchmark.
- thread_count: The size of the thread pool for executing Operations. Depending on the performance of the database to be tested, it may also be necessary to adjust the number of concurrently executing Queries.

# Report
Eventually, the Driver will generate the following performance metrics:
- Start Time: When the first Operation begins.
- Finish Time: When the last Operation is completed.
- Total Run Time: The duration between Finish Time and Start Time.
- Total Operation Count: The total number of Operations, the total number of Operations executed.

The system's QPS can be obtained by Total Operation Count / Total Run Time.

For each type of Query, the Driver will also output the total number of each Query, the average, median, P95, P99 of the execution time, and the distribution of scheduling time latency.

![image](https://www-cdn.nebula-graph.io/blogs/LDBC_report.jpeg)

**Reference**

The original intention of these articles is to help readers understand the logics of LDBC SNB as we found SNB not easily understood when we were running the testing of our own system. We may write more articles around the LDBC design topics. Please stay tuned!
- ldbc-snb-interactive-sigmod-2015.pdf (ldbcouncil.org)
- The LDBC Social Network Benchmark Specification
- GitHub - ldbc/ldbc_snb_interactive_v1_driver: Driver for the LDBC SNB Interactive workload


This blog post primarily focuses on how the query within the SNB Workload is created, and how the Driver executes the workload during performance testing.

Understanding LDBC SNB: Benchmark’s Query Generation and Execution

The Linked Data Benchmark Council (LDBC) Social Network Benchmark, commonly known as LDBC SNB, is a benchmark test specifically designed to evaluate the performance of graph databases in the context of social networking scenarios.

# An Introduction to LDBC
Apart from the Social Network Benchmark, LDBC offers several other benchmark tests, including the Graphalytics Benchmark, Financial Benchmark, and Semantic Publish Benchmark, respectively tailored for graph analysis, finance, and RDF scenarios. The Social Network Benchmark was the first benchmark test proposed by LDBC and has since become the most popular graph database benchmark test worldwide. Notably, as a non-profit organization, LDBC merely provides official audits. The performance of different graph databases may vary due to the operating environment and relevant benchmark parameters. 

This article delves into the main components of LDBC SNB: the testing benchmark process, and the preparatory stage of SNB, namely Schema and data generation.

LDBC SNB primarily consists of three main components:
1. Data Generator: This is a tool for generating large-scale complex data with social network characteristics. The data includes typical social network entities and relationships such as people, posts, comments, geographical locations, organizations, and more.

2. Interactive Workload: This primarily caters to OLTP, simulating everyday user activities on social networks, such as posting, adding friends, liking, etc. Read requests mainly involve one-to-two-hop queries, possibly accompanied by some write requests.

3. Business Intelligence Workload: This is primarily for OLAP, simulating in-depth analysis of social network data with whole-graph queries. For example, it analyzes user social behavior, community formation and evolution, and other tasks requiring complex analyses and substantial data processing.

The LDBC SNB paper also refers to an SNB Algorithms component, which as the name suggests, primarily runs graph algorithms like PageRank, community discovery, breadth-first search, etc. However, since the paper was published in 2015 when the scenario was still in its drafting phase at the time, this component has now been moved to the Graphalytics Benchmark.

Furthermore, to run the LDBC SNB test, an official Driver provided by LDBC is required. Different databases need to implement the corresponding Connector based on the Driver's interface to connect the Driver and the database. Subsequently, the Driver generates a Workload, which can be understood as a series of query statements, based on the Benchmark's relevant parameters. It then drives the database under test to execute these query statements, ultimately yielding performance test results.

The entire LDBC SNB benchmark testing process is divided into three main stages: the preparation, the benchmark test, and the result output.

During the preparation stage, data generation takes place, which includes the initial import of full data, as well as subsequent real-time updates. In official audits, validation is also required on the SF10 Dataset, which is explained at the end of this article, so this stage also generates data for validation.

The benchmark testing phase begins with validation on the SF10 Dataset, followed by performance testing on the SF30 or SF100 Dataset. The validation process commences after data import, where the Driver, based on a series of queries and expected results prepared in the previous stage, verifies the database's query results to ensure their accuracy. There is no specific time requirement for this validation process. The subsequent performance testing is divided into import, warm-up, and performance testing. The database is allowed a 30-minute warm-up time, while the performance testing must last at least two hours. The test results are then compiled and output.

![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-23-2.jpg)

# Generation of LDBC SNB Schema
The data in SNB simulates a social network similar to Facebook, with all data built around the Person entity. A person joins a network. Each Person might have several Forums, within which they can post several Posts. Other people may like some of these Messages.
The volume of these elements is mainly influenced by the number of Persons and time:
- A Person with more friends will post more comments or likes.
- Over time, as one makes more friends, the number of comments or likes will also increase.

Some data does not change with the number of the Persons, mainly including some Organizations (mainly schools in this context) and Places (mainly residential cities, countries, and other geographical information). This data plays a role during data generation. For instance, people who attend the same school during the same period are more likely to become friends.

The complete SNB Schema is illustrated in the following diagram:
![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-23-3.jpg)

Most graph databases model entities as points and different relationships as edges during testing. However, this is just a convention; the SNB data modeling and the actual database Schema can differ, provided the database can perform the corresponding Workload queries.

## Data Generation for LDBC SNB
SNB is the Data Generator (referred to as DataGen), used to generate data that meets the above Schema. The data generated by the Generator is determined by the following three parameters:

- Number of Persons
- Number of years of data to simulate
- The year from which the simulation begins

According to the official documentation, the data generated by DataGen has the following properties:

- Reality: The generated data simulates a real social network. On one hand, the attributes, cardinalities, data correlations, and distributions in the generated data are carefully set to simulate real social networks like Facebook. On the other hand, its original data comes from DBpedia, ensuring the attribute values in the data are authentic and relevant.
- Scalability: DataGen can generate different sizes of datasets (from GB to TB level) for systems of various scales and budgets. Furthermore, DataGen can complete data generation on a single machine or within a cluster.
- Determinism: Regardless of the number of machines used to generate data or the machine configuration, the data generated by DataGen is deterministic and consistent. This essential function ensures that any data system can use the same dataset, guaranteeing fair comparisons between different system environments and repeatable benchmark test results.
- Usability: DataGen is designed to be as easy to use as possible.

The entire data generation process is illustrated in the following flowchart, which we will break down into several sections for explanation:

![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-23-4.jpg)

## Distribution of Attribute Generated

The first step is initialization. The original data used by DataGen comes from DBpedia. For each attribute, DataGen decides the attribute's distribution based on:
- How many possible attribute values there are
- Probability of occurrence of each attribute value

The final distribution of attributes is saved as resource files and DataGen parameters.

## Generating Persons and Friendships
As mentioned earlier, the core of the SNB Schema is the Person, which is also reflected in the data generation process. DataGen generates all Persons, and some of the information needed for subsequent operations in Person, such as how many Friendships each Person has (this value is crucial, its distribution follows a Power law), the university attended by the Person, the company where the Person works, etc.

Then, DataGen creates the Friendship relationships, i.e., the 'knows' in the flowchart, for each Person. Just like in real social networks, people with the same interests or behaviors are likely to connect. To simulate such a social network, SNB considers the following three dimensions when generating Friendships:

- The university attended by the Person, the time of attendance, and the city where the university is located
- The Person's interests
- Each Person generates a random value, and the closer the random values, the more similar they are (this is to simulate that not all friends are made through university and interests)

These three dimensions account for 45%, 45%, and 10% of the weight of each Person's Friendship relationships, respectively. This divides the process of building edges between Persons into three sub-steps.

DataGen sorts all Persons according to three dimensions (sorted by one dimension at a time) and then divides the sorted Persons into multiple disjoint parts, distributed to different Worker processes. Even after division, the Persons a Worker thread is responsible for might exceed the memory size. Therefore, the Worker thread maintains a sliding window, where the probability of establishing Friendship relationships between Persons within the sliding window follows a geometric distribution.

As shown in the figure below.

![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-23-5.jpg)

Suppose we've sorted by the university attendance dimension, resulting in an ordered sequence of Persons. The Worker then maintains a sliding window, generating Friendship relations each time for the person at the far left of the sliding window (P2 in the figure). The proportion of other people in the window establishing Friendship with the first person follows a geometric distribution. Once the first person in the sliding window has established enough Friendships, the starting point of the sliding window moves to the next person.

*Note: The specifics of the sliding window size, geometric distribution parameters, and even random generator parameters are not delved into here. It's unclear how DataGen handles situations where it's impossible to generate enough Friendships within the sliding window.*

After sorting, distributing, and establishing edges by the sliding window for all three dimensions, DataGen moves to the next stage.

## Generating Social Activities
After generating Persons and Friendships, DataGen starts to generate each Person's social activities, including Forums, Posts, and Comments. Some correlations exist in this data:
- Persons with more Friendships are more active on social networks.
- Each Person is more likely to Post or Comment in Forums related to their interests or the university they attended.
- Social activities are related to time; for example, discussions related to football surge near the World Cup.

## Final Output
After the above steps, DataGen completes data generation. The simulated social network graph is in two parts:
- Dataset: 90% of the data is for initial import.
- Update Streams: 10% of the data is for subsequent real-time updates.

In addition, it generates parameters (mainly starting points) for subsequent Workload requests. We'll explain parameter generation in detail in the next article, but here we'll briefly describe the SNB read request Workload. Interactive Workload queries primarily aim to get results within one second. All read queries start from a point in the graph, obtaining a small part of the subgraph information. Also, as different starting points have different degrees of input and output, this essentially determines the data volume accessed by this read request.

To test different systems and scenarios, SNB defines a scale factor (SF) to control the final data volume size. For instance, the original data size of SF1 is 1 GB, while the sizes of SF0.1 and SF300 are 100 MB and 300 GB, respectively. The amount of vertex-edge data of different scale factors is shown in the table below.

![image](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/2024-04-23-6.jpg)

The final generated Dataset is divided into two main categories: Static and Dynamic, both formatted in CSV. Depending on the number of threads configured in DataGen, the final generated data will also be divided into multiple fragments. Static includes Organizations, Places, Tags, etc., all of which are static data generated based on DBpedia. The number of these does not change with the scale factor. In other words, this data is independent of the number of Persons. The Dynamic part mainly includes Persons, knows (the Friendship described in the data generation section), Forums, Posts, and Comments, etc.

Update Streams contain all update operations, mainly simulating real-time actions such as registering new users, commenting, liking, adding friends, and more.

# Reference

We have now roughly completed the introduction to the preparation stage. We will elaborate on the parameters generated at the end of the preparation stage in the next article when we discuss Workload.
For further reading and detailed understanding, refer to 

- the LDBC Social Network Benchmark Specification
- ldbc-snb-interactive-sigmod-2015.pdf (ldbcouncil.org)







The Linked Data Benchmark Council (LDBC) Social Network Benchmark, commonly known as LDBC SNB, is a benchmark test specifically designed to evaluate the performance of graph databases in the context of social networking scenarios. This article delves into the main components of LDBC SNB: the testing benchmark process, and the preparatory stage of SNB, namely Schema and data generation.

Understanding LDBC SNB: Schema Generation and Data Generation

In the era of information overload, sifting through vast amounts of data to provide accurate search results in an engaging and comprehensible manner has become an uphill battle. Traditional search enhancement techniques often fall short when it comes to complex queries and the high demand brought by cutting-edge technologies like ChatGPT. This is where Graph RAG steps in – a game-changing concept crafted by NebulaGraph that promises to revolutionize the field.

**“Graph RAG” is a concept pioneered by NebulaGraph**. This technology harnesses the power of knowledge graphs in conjunction with Large Language Models (LLMs) to provide search engines with a more comprehensive contextual understanding. It assists users in obtaining smarter and more precise search results at a lower cost.

NebulaGraph's revolutionary Graph RAG (Retrieval-Augmented Generation) technique, which is based on knowledge graphs, is a breakthrough in this area. Graph RAG combines knowledge graphs with a large language model to provide more cost-effective, intelligent, and precise search results. **Capable of deep integration with large language models such as Llama Index and LangChain, NebulaGraph is the first in the industry to propose the concept of Graph RAG.** Besides, Graph RAG has also achieved excellent results in the field of vector database integration.

In this article, we will introduce the concept of Graph RAG and compare it with other RAG techniques. You are also welcome to visit the NebulaGraph‘s website to [Try Demos](https://www.nebula-graph.io/demo).



# Challenges Faced by Traditional Search Enhancement Techniques

Obtaining accurate search results is a challenge, especially with complex or long-tail queries when traditional search engines rely on keyword matching, which often does not meet the users' actual needs. The bottleneck of traditional search augmentation techniques is the lack of training data and text understanding, which makes them less effective when it comes to generating accurate, diverse and relevant information.

To address this problem, RAG came into being. **RAG (Retrieval-Augmented Generation) refers to the process of enhancing search results through RAG models.** Specifically, it is a technique that combines retrieval and language generation to enhance the generation process. By using RAG, traditional search engines can generate more accurate and relevant search results.

However, the RAG technique still faces challenges in terms of training data and text understanding. These challenges include:

- **Train data:** the RAG technique requires a large amount of data and computational resources for training and generating models, especially when handling multiple languages and complex tasks.
- **Text understanding:** RAG needs to understand the intent of the query, but for complex queries or polysemous queries, RAG may have ambiguity or uncertainty, which affects the quality of the generated content.

Therefore, the problem of how to find more powerful retrieval enhancement techniques to obtain search results that better meet the searcher's expectations more efficiently is even more urgent.

# What is Graph RAG?

**Graph RAG** is proposed by NebulaGraph, which is a retrieval enhancement technique based on knowledge graphs. It uses a knowledge graph to show the relationship between entities and relationships and then uses the large language model LLM (Large Language Model) for retrieval enhancement.

As mentioned in the previous [blog](https://www.nebula-graph.io/posts/graph-llm), the [graph database](https://www.nebula-graph.io) is naturally suitable for storing and expressing complex context information by organizing and connecting information in a graphical format. By using graph technology to build a knowledge graph to enhance in-context learning, users can provide more context information to help large language models (LLM) better understand the relationship between entities and improve their expression and reasoning abilities.

Graph RAG equates the knowledge graph to a large-scale vocabulary, and entities and relationships correspond to words. In this way, Graph RAG can jointly model entities and relationships as units during retrieval, which can more accurately understand the query intent and provide more accurate search results.

# Demo: Graph RAG vs. Vector RAG vs. Text2Cypher

In the following sections, we'll explore two demos that provide a visual comparison of Graph RAG with Vector RAG and Text2Cypher.

## Graph RAG vs. Graph + Vector RAG

The GIT below compares the results returned by Vector RAG (vector retrieval) with Graph and Vector RAG (graph-enhanced vector retrieval).

![Graph RAG vs. Vector RAG](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/1.gif)

- Results on the left: Vector RAG 
- Results on the right: Vector RAG with Graph

As an example, let's consider the data set of "Guardians of the Galaxy 3". When we search for "Peter Quill's related information," traditional retrieval techniques like vector retrieval engines only provide simple information on his identity, plot, and actors. However, with the Graph RAG-enhanced search results, we can obtain more information about the protagonist's skills, character goals, and identity changes.

This example highlights how Graph RAG effectively complements the shortcomings of traditional methods like embedding and vector retrieval and provides more in-depth knowledge and information relevant to the query. By combining knowledge graphs with large language models, Graph RAG can understand the relations between entities, equate knowledge graphs to large-scale vocabularies and better comprehend the intent of complex queries, leading to more accurate and relevant search results.

## Graph RAG vs. Text2Cypher 

Another interesting approach to knowledge graph-based LLM is Text2Cypher, which is a natural language generation graph query. This method does not rely on entity subgraph retrieval but translates tasks or questions into an answer-oriented graph query, which is essentially the same as what we commonly call Text2SQL.

Text2Cypher and Graph RAG differ mainly in their retrieval mechanisms. Text2Cypher generates graph pattern queries based on the knowledge graph schema and the given task, while (Sub)Graph RAG obtains relevant subgraphs to provide context. Both have their advantages, and you can get a more intuitive understanding of their characteristics through the following demo.

![Graph RAG vs. Text2Cypher](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/2.gif)

- Results on the left: Text2Cypher 
- Results on the right: Graph RAG



From the above demo, it's evident that the two graph query modes exhibit pronounced differences in [visualization](https://www.nebula-graph.io/posts/graph-database-visualization). Retrieval using Graph RAG presents more comprehensive results. Users not only receive fundamental introductory information but also gain access to a range of outcomes based on associative searches and contextual inferences, such as "Peter Quill is the leader of the Guardians of the Galaxy," "This role implies that he will return in the sequel," and insights into the character's personality.

# NebulaGraph  : Build Your Enterprise-Specific Knowledge Graph Applications in One Step

**NebulaGraph database has seamlessly integrated with large language model frameworks like Llama Index and LangChain.**Therefore, developers can focus on LLM orchestration logic and pipeline design without having to deal with many details of abstraction and implementation and generate high-quality and low-cost enterprise-level large language model applications in one stop.

The advent of Graph RAG presents a breakthrough for processing and retrieving information on a massive scale. By fusing knowledge graphs and graph storage within the LLM stack, Graph RAG offers an unparalleled level of context learning. With the utilization of NebulaGraph, building Graph RAG is highly simplified and requires minimal effort, based on just three lines of code. Even more complex RAG logic, such as Vector RAG with graph can be easily integrated.

![Graph+Vector RAG](https://www-cdn.yueshu.cn/yueshu-website/images/blogs/%E8%AF%AD%E8%A8%80%E6%A8%A1%E5%9E%8B/demo-3.png)

As graph technology and deep learning algorithms continue to advance, the adoption of Graph RAG in data processing and retrieval is set to become increasingly widespread. We invite you to [contact us](https://www.nebula-graph.io/contact) for a free trial of NebulaGraph database and to effortlessly create your knowledge graph application!



NebulaGraph database's revolutionary Graph RAG (Retrieval-Augmented Generation) technique, which combines knowledge graphs with a large language model to provide more cost-effective, intelligent, and precise search results.

NebulaGraph Launches Industry-First Graph RAG: Retrieval-Augmented Generation with LLM Based on Knowledge Graphs

This year's SIGMOD'22 was held in Philadelphia, and I was honored to attend as a speaker on behalf of the company. During this conference, I communicated a lot with students, professors, and companies. On the one hand, I publicized our products; on the other hand, I got many valuable suggestions and opinions. Next, I'll share some of these suggestions, as well as my overall thinking.

# Graph learning and storage technology have attracted much attention

‍

**There are two hotspots that most participants pay attention to at this conference, which reflects the new trend in the industry.&nbsp;**

**First, the importance of machine learning is further increasing, and graph learning is becoming a new hotspot**. In recent years, more and more conferences that focus on specific systems set sections about machine learning to study the use of machine learning in optimizing systems or algorithms. Among the sub-fields of machine learning, a new force, i.e., graph learning, which is machine learning on graphs, suddenly rises. Just like other sub-fields of machine learning, graph learning also aims at solving problems in the real world. **Through the method of embedding, graph learning extracts feature vectors from a graph, and then uses machine learning methods to solve complex problems in the graph, including classification, subgraph matching, an  
d link classification.**		

Judging by the tendency of conferences and guidance from the National Science Foundation (NSF), this trend is likely to continue. We have also received a lot of inquiries from students and professors about whether NebulaGraph supports graph learning. From a graph database provider's perspective, graph learning is one of the NebulaGraph application scenarios. We should provide not only the graph database core, but also libraries, interfaces, and even optimizations for upper-layer applications to ensure the smooth and efficient operation of upper-layer applications. On the other hand, if NebulaGraph can help academia make some breakthroughs in areas like graph learning, it'll be great for our company, and the industry as a whole.
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/thoughts-after-attending-the-sigmod22-conference/figure1.jpeg)

**In addition to machine learning, I think another hotspot is memory, including in-memory databases, persistent memory, etc**. Non-volatile memory, or persistent memory, has become more and more mature through the joint efforts of academia and industry for more than ten years. Research on how to use persistent memory is also very hot, and there are numerous articles on it at top events of almost all systems.

**In the database field specifically, research topics include how to design an in-memory database, how to mix durable memory with SSD, DRAM, and other types of storage, and how to solve the problem of memory segregation.** NebulaGraph, as a graph database provider, should be embracing the changes that new hardware brings, and exploring persistent memory-based storage systems as well.

# The advantages and development direction of the graph

**In addition to the new trends, my biggest takeaway and thought from the conference was the comparison between graph databases (GDBMS) and relational databases (RDMBS). **There were quite a few people at the conference who were wondering why the graph database was not implemented based on a relational database, because relational databases have been optimized over the years, and they are really armed to the teeth. For example, Professor Peter Boncz (one of the founders of LDBC) gave a keynote address at the EDBT/ICDT conference earlier this year on The (Sorry) State of Graph Database Systems [1]. He stated that compared with RDBMS, GDBMS still had a lot to improve in terms of computation and storage, especially regarding AP queries like subgraph matching. He proposed a performance benchmark for subgraph matching scenarios. In his experimental environment, none of the graph database software came close to the performance of relational databases Hyper and Umbra in the subgraph matching scenario (see the following figure).
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/thoughts-after-attending-the-sigmod22-conference/figure2.png)

Certainly, the setup of this experiment is worthy of discussion. Still, Peter, as a founder of LDBC, did not mean to disparage graph databases but to encourage graph database providers to improve their products by using subgraph matching, a query scenario in which graph databases are not well optimized.

From the broader picture, NebulaGraph, as a graph database provider,** must know the differences between graph databases and relational databases and grasp the advantages of graph databases**. For example,** GQL is easier to use and more efficient when executing graph queries than SQL**. Emil Eifrem, CEO of Neo4J, has a perfect example [3]: For a very common query in an AP system, if using SQL, it needs a very long statement consisting of 23 SELECTs, 21 WHEREs, 11 JOINs, and 9 UNIONs. But if you use a graph query language, you only need a MATCH and a WHERE. For SQL, the query task may be impossible to complete or easy to go wrong, but for a graph query language, the task is easy to complete, saving countless manpower and resources.

**In terms of performance**, in addition to the natural advantages of graph databases over relational databases in association relation queries, i**t is also necessary to gain an advantage over relational databases in a variety of other graph-related queries (not limited to subgraph matching mentioned above). **This is actually the core enlightenment of Peter's proposal of subgraph query as a benchmark for us.		

‍

# Thoughts about the underlying storage structure

![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/thoughts-after-attending-the-sigmod22-conference/figure3.png)


At last, we also gained a number of suggestions for the underlying storage. NebulaGraph currently applies RocksDB, which is based on LSM-Tree, as the underlying storage engine. However, it has been controversial whether LSM-Tree is suitable for the workload of the graph database, especially, whether it is the best choice with the lowest cost on the cloud. Possible alternatives are:

- Bε-tree File System, or betrFS.[ https://www.betrfs.org/](https://www.betrfs.org/)

- LiveGraph. [https://marcoserafini.github.io/papers/LiveGraph.pdf](https://marcoserafini.github.io/papers/LiveGraph.pdf)

- B+ Tree. We will follow up the research on this aspect.	

All in all, the SIGMOD'22 trip was fruitful. Look forward to SIGMOD next year. See you in Seattle!

[1] [https://conferences.inf.ed.ac.uk/edbticdt2022/?contents=invited_talks.html](https://conferences.inf.ed.ac.uk/edbticdt2022/?contents=invited_talks.html)

[2] [https://docs.google.com/presentation/d/1xUbooiL8rIMzkp9G9EXmN4tIMMWt2mC53Q0u4mslq5g/edit#slide=id.ge0f13c0edc_0_47](https://docs.google.com/presentation/d/1xUbooiL8rIMzkp9G9EXmN4tIMMWt2mC53Q0u4mslq5g/edit#slide=id.ge0f13c0edc_0_47)

[3] [https://people.eecs.berkeley.edu/~istoica/classes/cs294/15/notes/21-neo4j.pdf](https://people.eecs.berkeley.edu/~istoica/classes/cs294/15/notes/21-neo4j.pdf)

‍

This year's SIGMOD'22 was held in Philadelphia, and I was honored to attend as a speaker on behalf of the company. During this conference, I communicated a lot with students, professors, and companies. On the one hand, I publicized our products; on the other hand, I got many valuable suggestions and opinions. Next, I'll share some of these suggestions, as well as my overall thinking.

Thoughts after attending the SIGMOD'22 conference

This article is transcribed from NebulaGraph's recent webinar about data visualization in NebulaGraph, where three NebulaGraph engineers talked about how data visualization in implemented in [NebulaGraph](https://nebula-graph.io/), the world's most powerful graph database management system.

# What is visualization?

From my personal understanding, visualization is the use of certain data or text that we usually see in a relatively abstract way, such as icons or charts, for secondary display and more convenient for users to interact. Simply speaking, the common buttons and icons on web pages, such as the "Google" button in blue after typing in https://www.google.com/ and the auxiliary function icons in the lower right corner, are all simple and basic visual displays. Without these small icons, you may need to call the terminal to execute a certain command to complete the corresponding operation conveniently. The visualization can greatly reduce the operation cost as well as make it easier for users to understand the corresponding operation.

In the above example, we are talking about UI visualization. Speaking of UI, let's read the memory of more than ten years ago. As you know, in the early years, computer technology was not yet developed, and we all looked at the command line. And now, why computers can fly into ordinary people's homes? It is because the computer or computer has an interface, a UI, something that meets human intuition. So, visual things that improve efficiency are called visualization. If we were to give a definition, visualization would probably have two points.

1.  The above example and the GUI, such visualization is visible to everyone, but in my opinion is not the biggest highlight that reflects visualization.
2.  The design, or the art, which is the soul of the visualization. Because with this thing, it will give meaning to the visualization of life. In layman's terms, whether something is good or not, often not you can make it out it is good; but if it is good interaction, good experience, good visual effect, can meet the human intuition, dig out some information, it is considered a good visualization. So, I think visualization still needs two parts, one is in the physical sense, that is, the first part of the rendering, and the second part is its design experience, which involves software and hardware.

In fact, the concept of visualization can be viewed in two dimensions: broad and narrow.

In the broad sense, visualization can be understood as taking a complex piece of information and making it simpler and more intuitive to present to the user through certain processing, so that the user can receive some information more easily. This belongs to the category of visualization, like GUI, which is actually a kind of visualization, abstracting the underlying commands in a visual way, so that users can interact with the machine more easily.

In a narrow sense, it is the application of the scenario. For example, along with the development of the Internet, the amount of data we deal with will become larger and larger, we have to present these data through a certain way, the better the presentation, the better the data expression can be understood. How to display more data more smoothly on the basis of the current hardware with certain performance bottlenecks may be the meaning of the current visualization. In this way, visualization in a narrow sense, it may be charts, Canvas animations, or the rise of VR technology, browser rendering, 3D rendering in the meta-universe, these visualization researchable directions.

Specifically, because Nebula is a distributed graph database, the visualization we mention is more about data visualization. For example, Nebula Dashboard is a visual operation and maintenance monitoring product, the visualization practice in this area is not only icons and animations, but also how to make the complex operation and maintenance information easier for users to control the cluster through a simple way.

![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-in-nebula-graph/figure1.gif)

# Visualization technologies

At different stages and in different scenarios, we choose different visualization technologies to better serve our products and business.

As far as visualization technologies are concerned, the common visual rendering technologies for front-end display are SVG, Canvas, CSS and WebGL, and what to use to draw SVG and Canvas is another topic. D3, as often mentioned by the Nebula community, is not a visualization technology, but essentially a visualization algorithm tool that can be used on the web or in a terminal application. D3 itself has some peripheral tools to do simple SVG wrapping, which is more on the upper layer of application tools, such as the charting library [ECharts](https://echarts.apache.org/en/index.html) is an example.

Wang Yang, the product leader of Nebula Dashboard, said that the common visual rendering technologies such as CSS and SVG mentioned above will be customized according to the application scenarios and computing resources faced. For example, in order to facilitate third-party access, the designers of the visualization library will abstract the syntax and make the syntax sugar more scientific and standardized; for scenarios with high performance requirements, they will encapsulate the tools in order to obtain high performance. Here he mentioned Ant open source visualization engine: G2 https://github.com/antvis/g2/, G2 was inspired by the idea of the book "The Grammar of Graphics" to design this set of visualization based on the theory of graphics grammar underlying engine, G2's set of grammar implementation is also more recognized in the industry to the scientific specification. Compared with other visualization libraries, G2 is more like a complete product in terms of syntax design.

When it comes to G2, the core developer of the visual graph exploration tool Nebula Explorer, Miao Zhuang, said that although the syntax of G2 is standardized and scientific, it is not a very smooth visualization tool if you really want to do data visualization, because it is based on the theory of graph syntax and requires certain expertise, so there will be a certain threshold. More often than not, G2 may still be used as a visual charting library rather than a visualization library.

# Performance issues of visualization

Nebula Explorer is a graph exploration tool and it will face the problem of node data presentation. The community also often asks how many node data can be rendered by a single canvas page. In response to the visualization performance issue, Miao Zhuang, the core product developer, said: visualization performance is not an absolute performance, it is often accompanied by the user experience. The user experience and performance are somehow inversely proportional, the better the performance, the better the user experience, and the less information will be presented; while presenting more data, it will consume more hardware resources. This is a balance problem.

And there are two ways to improve the performance of visualization: one is to improve the hardware configuration, before the very popular terrain - distributed concurrency is not enough what to do? Add a machine, there is nothing that can not be solved by adding a machine. But the simple and brutal method of adding machines is not a good policy, although the physical limitations are solved but the cost of adding the problem, in addition to the cost of the machine and the corresponding system maintenance costs. A machine, or more than one machine to join the system will certainly bring the complexity of the system increases. In addition, adding machines is not a universal solution, some users may not be able to add machines as they wish. Another is to optimize the algorithm, compared to adding machines, method 2 is more difficult, the current algorithm is very good, and continue to optimize only the marginal decreasing effect.

# Nebula visualization tool optimization practice

Nebula Explorer optimization performance is also from the above two hardware and software direction to start. In terms of hardware, the hardware performance of different users is different, so Explorer will choose better hardware to achieve cool effects during the development process; while for users with poor hardware performance, it provides two modes: high quality and normal quality, adapting to the user's hardware to achieve a better rendering effect. Although Explorer's 3D graphics exploration mode looks cool and "eats" computing resources, the hardware requirements for this feature are not as high as you might think, and any phone's GPU configuration can run Explorer 3D mode.

In fact, in addition to the performance optimization of hardware and software, there is also a strategic optimization of product design. For example, some of the more difficult performance problems can be circumvented. In terms of performance optimization, there is a time-honored principle, which is to ensure that 60 frames are rendered in 1 second. Even in special scenarios where 60 frames a second is not possible, achieving more than 40 frames a second will keep the system smooth. This is like an optimized implementation of React, which breaks up a lot of big tasks into smaller ones, and then plugs in smaller ones when the system is free. Another way to think about this is that if the problem cannot be solved at the code level, you can modify the corresponding interaction. For example, a long time-consuming task is running, and if the page is not clickable, you will feel very laggy, right? However, adding a loading animation at this time will greatly reduce the user's waiting anxiety and make the user feel less laggy.

Here is a personal experience: when Nebula Dashboard first launched the visualization screen, we stepped through the performance pitfalls. When the visualization screen was first launched, I just thought it was a simple screen that might not involve too many performance problems. As a result, once the CPU was spinning like crazy, I was confused: Nebula Dashboard visualization screen only had 7 or 8 charts. Although the interface was cool, there were actually only a dozen network requests, so we ruled out the problem that the CPU was consuming too many resources in the request calculation. In addition, when the big screen was online, there were not many concurrent cases... After a round of checking, we finally located that the SVG was consuming too much resources. So we adjusted the animation implementation scheme: for animations with high CPU requirements, we downgraded them and reduced the number of rendering and rendering frequency. This is a bit like image compression, you throw a 1 MB image to the compression site, the compression may be 200 KB, at this time the user's naked eye is not aware of it.

# Visualization technology selection

As mentioned above, hardware and software can improve the performance of visualization, but a good choice of technology can also affect the performance of your system.

Before Nebula Explorer v2.2, due to the technical selection and business logic organization, once we encountered hundreds of nodes displaying problems, we would encounter lag and the animation was obviously not smooth. After that, Nebula Explorer was refactored and replaced by G6, a [graph visualization](https://www.nebula-graph.io/posts/graph-database-visualization) engine based on G2, which performed poorly in Nebula Explorer, probably due to G6's over-focus on business scenarios and lack of validation for high-performance scenarios. After switching to Force Graph, the performance has improved by about 10 times. Of course, it is not that the performance of G6 is not good, but the visualization requirements of Nebula Explorer itself are not MATCH with G6.

On the contrary to Nebula Explorer, Nebula Dashboard does not have thousands of points of data display, but a relatively fixed number of graphs and the corresponding operation execution. For monitoring charts, Grafana is currently the most recognized product in the market, and Grafana itself is an open source product. So, we researched Grafana's implementation and found that it uses the lightweight uPlot https://github.com/leeoniya/uPlot. Because uPlot is less than 40kb, it features time series charting and is easy to do custom development. In addition to monitoring, Nebula Dashboard also has an important function for cluster management and maintenance operations. This is a departure from Grafana's focus on monitoring, and the three criteria for selecting Nebula Dashboard are: first, easy to develop and start; second, active open source community; third, many base libraries to improve development efficiency. In the end, G2 was chosen as the underlying visualization tool for Nebula Dashboard in order to improve development efficiency.

# Visual Product Design

Nebula Explorer has two graph exploration modes, 2D and 3D, which involve how the pages are laid out. In the Nebula Explorer design, algorithms are used to achieve aggregation and hashing of specific groups of points; and the full graph bird's eye view mode is displayed in 3D mode, because the mainstream computer is 1920*1080, and a plane can display up to a million pixels, then we can use 3D mode to present more data than 2D when we change the angle.

‍

This article is transcribed from NebulaGraph's recent webinar about data visualization in NebulaGraph, where three NebulaGraph engineers talked about how data visualization in implemented in NebulaGraph, the world's most powerful graph database management system.

How data visualization is implemented in NebulaGraph


This article is based on a talk given by Dr. Min Wu, a senior expert at Vesoft Inc. Dr. Wu talked about the status quo of the global graph database market, the design and features of NebulaGraph as a [distributed graph database](https://www.nebula-graph.io), as well as NebulaGraph’s open-source community.

# The Global Graph Database Market

Let’s start with some numbers. Markets and Markets anticipates the graph database market will reach $2.4 billion by 2023 from $821.8 million in 2018. 

Graph database is still in a rising trend and it is one of Gartner’s top 10 data and analytics trends in 2021. This is because graph databases can be used in many more areas than traditional databases, including computing, processing, deep learning, and machine models.

![The Global Graph Database Market](https://user-images.githubusercontent.com/90186547/153827006-decd54de-76fa-4123-83e3-739512013fce.png)

Graph databases gained the most in popularity in the past 10 years, according to data compiled by DB-Engines. The data is based on social media mentions, Stack Overflow questions, and search trends.

![The Global Graph Database Market](https://user-images.githubusercontent.com/90186547/153827206-90b5b420-99b4-466c-904c-78250115ee1e.png)

# Advantages of Graph Databases

One of the most significant advantages of graph databases is that they are intuitive. If you want to express the character relationships of Game of Thrones, you can use both traditional tabular databases and graph databases. But as it is shown below, using graph data is much more intuitive, though they both express the same data model.

![Advantages of Graph Databases](https://user-images.githubusercontent.com/90186547/153827594-7c9f5e95-ae9d-499a-bb5b-f41cb48d3578.png)

(Tabular data)

![Advantages of Graph Databases](https://user-images.githubusercontent.com/90186547/153983950-58cd84d4-8bed-400e-8daa-ce1e04549172.png)

(Graph data)

In another example, we can compare how to search in SQL databases and graph databases. For example, here is how to find out how many posts and comments were created in a given time frame and rank the results in SQL databases and graph databases.

SQL:

```
--PostgreSQL 

WITH RECURSIVE post_all (psa_threadid

                      , psa_thread_creatorid, psa_messageid

                      , psa_creationdate, psa_messagetype

                       ) AS (

    SELECT m_messageid AS psa_threadid

         , m_creatorid AS psa_thread_creatorid

         , m_messageid AS psa_messageid

         , m_creationdate, 'Post'

      FROM message

     WHERE 1=1 AND m_c_replyof IS NULL -- post, not comment

       AND m_creationdate BETWEEN :startDate AND :endDate

  UNION ALL

    SELECT psa.psa_threadid AS psa_threadid

         , psa.psa_thread_creatorid AS psa_thread_creatorid

         , m_messageid, m_creationdate, 'Comment'

      FROM message p, post_all psa

     WHERE 1=1 AND p.m_c_replyof = psa.psa_messageid

     AND m_creationdate BETWEEN :startDate AND :endDate

)

SELECT p.p_personid AS "person.id"

     , p.p_firstname AS "person.firstName"

     , p.p_lastname AS "person.lastName"

     , count(DISTINCT psa.psa_threadid) AS threadCount

END) AS messageCount

     , count(DISTINCT psa.psa_messageid) AS messageCount

  FROM person p left join post_all psa on (

       1=1   AND p.p_personid = psa.psa_thread_creatorid

   AND psa_creationdate BETWEEN :startDate AND :endDate

   )

 GROUP BY p.p_personid, p.p_firstname, p.p_lastname

 ORDER BY messageCount DESC, p.p_personid

 LIMIT 100;
```

Here is how to realize the same query using the Cypher graph query language:

```
--Cypher
MATCH (person:Person)<-[:HAS_CREATOR]-(post:Post)<-[:REPLY_OF*0..]-(reply:Message)
WHERE  post.creationDate >= $startDate   AND  post.creationDate <= $endDate
  AND reply.creationDate >= $startDate   AND reply.creationDate <= $endDate
person. RETURN
id,   person.firstName,   person.lastName,   count(DISTINCT post) AS threadCount,
  count(DISTINCT reply) AS messageCount
ORDER BY
  messageCount DESC,  person.id ASC
LIMIT 100
```

In addition, the graph ecosystem is diversified. The following is the [graph technology landscape in 2020](https://graphaware.com/graphaware/2020/02/17/graph-technology-landscape-2020.html), and we can expect more graph related technologies coming along in 2021.

![Advantages of Graph Databases](https://user-images.githubusercontent.com/90186547/153828477-22d4c36c-63ea-438e-9ee6-ed3c618cd37a.png)

# NebulaGraph

Now, let's take NebulaGraph, a distributed graph database, as an example to talk about the evolution of graph technology. I will also share the challenges the team had faced when developing NebulaGraph and how we had solved them.

When we started designing the blueprint of NebulaGraph in late 2018, the team had set four goals for the database. They are scalability, production-ready, [OLTP](https://www.nebula-graph.io/posts/OLTP-vs-OLAP-what-are-the-differences)(Online Transaction Processing), and open source. The four goals are still influencing the roadmapping of NebulaGraph until today.

**Scalability**

Scalability is the No.1 design principle of NebulaGraph. This is because we believe the data that businesses will process in the future must be massive and there will be no way that single machines can handle them. That’s why we designed NebulaGraph in a way that it is capable of handling graph data with trillions of vertices and edges.

**Production-ready**

NebulaGraph is also designed to be production-ready on the first day, including the design of its query language, visualization, programmability, and DevOps.

**OLTP**

OLTP (online transactional processing) enables the real-time execution of large numbers of database transactions by large numbers of users, typically over the internet. One of the priorities of NebulaGraph’s design is OLTP. This makes NebulaGraph an online, high-concurrency, and low-latency graph database.

**Open source** 

NebulaGraph is also devoted to building an open-source community and integrating with the big data world, supporting graph computing and training frameworks like Tencent Plato and Spark GraphX.

![NebulaGraph](https://user-images.githubusercontent.com/90186547/153829025-447f8153-0154-443e-96b6-aaf54f620877.png)

# The Nebula Core

The above figure shows the ecosystem built around NebulaGraph. The section with red background is the NebulaGraph Core, which consists of three parts called Meta, Graph, and Storage.

NebulaGraph’s query language is our in-house nGQL, which is also compatible with openCypher. We have also developed clients in languages including Java, C++, Python, and Go. Then on the top we have a number of SDKs that can work with frameworks like Spark, Flink, GraphX, Tencent Plato. 

![The Nebula Core](https://user-images.githubusercontent.com/90186547/153829317-ad4a7330-8b3b-461c-b611-fe017856336c.png)

Let’s dive into the Nebula core, which, as I mentioned above, consists of the meta, graph, and storage services. The meta service deals with metadata, while the storage service stores the data, and the graph service is in charge of querying. The three modules run on their independent processes, ensuring the separation of compute and storage.

The meta service manages the schema. NebulaGraph is not a schema-free database, and it requires the properties of vertices and edges to be pre-configured. The meta service also manages storage spaces, long-duration tasks, and data cleaning.

NebulaGraph can handle data with trillions of vertices and edges. This means the system must segment the data in storage and handling. NebulaGraph uses the segmentation of edges and stores vertices in partitions. Each partition may have a few replicas and run on different machines. The query engine is stateless, meaning that all query data should either be retrieved from the meta service or the storage service and there is no communication between query services.

The above is about NebulaGraph’s separation of compute and storage. Now let’s talk about data characteristics. We have mentioned that NebulaGraph is not a schema-free database and that all the data stored is pre-defined by Data Definition Languages (DDLs). We call types of vertices Tag and types of edges EdgeType. Vertices are defined using a 2-tuple consisting of the vid and the tag. Edges are defined using a 4-tuple consisting of the endpoints, EdgeType, and rank.

NebulaGraph supports primitive data types like boolean, int, and double as well as composite types like list, set, or graph data types like path and subgraph. If there is long string data stored in the database, it is usually indexed by Elasticsearch.

![The Nebula Core](https://user-images.githubusercontent.com/90186547/153829515-9259bd0c-d127-43ee-9724-0d2561619296.png)

Here is some additional information about the storage engine. For the query engine graphd, the external interface exposed by the storage engine is a distributed graph service, but it can also be used as a distributed key-value (KV) service if necessary. In the storage engine, partitions adopt the Raft consensus protocol. NebulaGraph stores vertices and edges in separated partitions. The following figure is about how KV is implemented using the storage partitioning.

![The Nebula Core](https://user-images.githubusercontent.com/90186547/153829730-a73d0aa3-ab99-404e-b355-06e1b98111d6.png)

In NebulaGraph, each edge is stored as two pieces of data. As mentioned above, the storage layer relies on VID and guarantees strong consistency using the Raft protocol.

NebulaGraph uses ElasticSearch for full-text index. From NebulaGraph v2.x, our R&D team has optimized the write performance of Nebula's indexing capability. Since v2.5.0, NebulaGraph has started to support the combination of data expiration TTL and indexing. And from v2.6.0, NebulaGraph started to support the TOSS (Transaction on storage side) function to achieve the eventual consistency of edges. That is to say, edges are either successfully written or failed at the same time when they are inserted or modified. 

NebulaGraph uses its in-house nGQL as its query language. In June 2021, the International Organization for Standardization has drafted the standard for the syntax and semantics of GQL and there is a consensus between major graph database vendors.

From NebulaGraph v2.0, nGQL started to be compatible with openCypher, which was an open-source version of Neo4j’s Cypher query language. Now, nGQL supports the Doctrine Query Language (DQL) of openCypher and has developed its own vanilla nGQL syntax style in Data Manipulation Language (DML) and Data Definition Language (DDL).

We also mentioned that NebulaGraph is born to be production-ready. So it supports a wide range of operation features like data isolation, user permission and authentication, and replica configuration. Also, NebulaGraph supports clustering. Nebula Operator, which was released in April 2021, started to support Kubernetes.

As for the performance of NebulaGraph, most performance tests are carried out by users in the community, such as engineers from tech companies like Meituan, WeChat, 360 DigiTech, and WeBank, etc. The figures below are performance reports compiled by users.

As for the performance of NebulaGraph, most performance tests are carried out by users in the community, such as Meituan, WeChat, 360 DigiTech, WeBank. The figures below are performance reports compiled by users.

![The Nebula Core](https://user-images.githubusercontent.com/90186547/153830033-15cfdf2f-fa39-457f-96b4-b83bebdab759.png)

![The Nebula Core](https://user-images.githubusercontent.com/90186547/153830090-2b9d1624-00ea-42d0-a6b4-737719683b41.png)

We have mentioned that one of the priorities of NebulaGraph’s design is OLTP. But that doesn’t mean it neglects analytical processing (AP). NebulaGraph has integrated AP frameworks like Apache Spark’s GraphX and Plato developed by Tencent.

Generally speaking, NebulaGraph performance has improved significantly when the deep traversal is conducted.

NebulaGraph meets users' AP (Access Point) requirements of OLTP (Online Transactional Processing ) by docking with Spark's Graph X and supports Plato, the graph computing engine of Tencent's WeChat team. Plato docking is actually the data connection between the two engines, which needs to change the internal data format of NebulaGraph to be that of Plato and then Partition to map them one by one. 

# The NebulaGraph Community

![The NebulaGraph Community](https://user-images.githubusercontent.com/90186547/153830303-56568605-7547-4c00-a3ff-9d6e7b78e398.png)

NebulaGraph became open source in May 2019. Its v1.0 GA was released in June 2020, even though some companies had applied NebulaGraph in production before that. NebulaGraph first entered DB-Engines’ graph database management system ranking two years ago and now it ranks 15th on the list.

![DTCC2021](https://user-images.githubusercontent.com/90186547/153830477-11d7f607-aecb-468e-b7f5-f0ab360ff71b.png)

NOTE: The screenshot is the DB-Engine ranking in Apr. 2021 when this talk was given.

NebulaGraph is also one of the top open source players in China. This following report, which was published by the X-Lab of [East China Normal University](https://en.wikipedia.org/wiki/East_China_Normal_University), ranks the companies according to the community popularity of their open source products. Vesoft Inc., the maker of NebulaGraph, ranks the eighth, before TikTok parent Bytedance, and just one position after Huawei.

![The NebulaGraph Community](https://user-images.githubusercontent.com/90186547/153830897-9cc1d126-1735-48c4-a8c3-10db68eee805.png)

Here are some of my thoughts about open source graph databases. Open source is very common in the graph database industry because it is a relatively new area and only got traction in recent years. This is why NebulaGraph chose to be open source from day one. Open source software can also attract more developers to use and gain valuable feedback from adopters.

If you encounter any problems in the process of using NebulaGraph, please refer to [NebulaGraph Database Manual](https://docs.nebula-graph.io/2.6.0/pdf/NebulaGraph-EN.pdf) to troubleshoot the problem. It records in detail the knowledge points and specific usage of the graph database and the graph database NebulaGraph.

Join our [Slack channel](https://join.slack.com/t/nebulagraph/shared_invite/zt-7ybejuqa-NCZBroh~PCh66d9kOQj45g) if you want to discuss with the rest of the NebulaGraph community!


This article introduced the status quo of the global graph database market, the design and features of NebulaGraph as a distributed graph database, as well as NebulaGraph’s open-source community.