Storaged, the core storage component of NebulaGraph, is responsible for ensuring durability, availability, and scalability. It plays a critical role in handling reads and writes efficiently. This article explores the architecture of Storaged from the perspective of its underlying storage engine, RocksDB [1], and shares insights from real production use cases on diagnosing and resolving performance bottlenecks. 

One notable production example is Snapchat, which uses NebulaGraph to enhance its friend recommendation system at scale. (Read more [here](https://www.nebula-graph.io/posts/snapchat_enhances_friend_recommendations_with_nebulagraph).)

# How is data read and written in NebulaGraph

In NebulaGraph, both vertices and edges are stored as key-value pairs using RocksDB as the storage engine. In version 3.x, we use RocksDB 3.7.3. The data is partitioned and replicated, with each partition replica backed by a separate RocksDB instance.

![image](https://www-cdn.nebula-graph.io/blogs/storage.png)

To distribute data across multiple storage nodes, we use hash function:
- **Vertices** are partitioned based on their vertex ID.
- **Edges** are partitioned based on the source vertex ID.

This design ensures that a vertex and all its outgoing edges reside in the same partition, improving query performance. However, **super nodes** (e.g., highly connected entities such as celebrities or popular websites) can cause load imbalances. Hashing helps distribute these nodes more evenly across Storaged instances.

# Consistency and Replication via Raft

To maintain consistency between replicas, NebulaGraph employs the **Raft consensus algorithm** [2], which elects a **Leader** among the replicas. The Leader handles most read and write requests, while **Followers** serve as backups. If a Leader fails due to network issues or maintenance, a new Leader is elected.

The data write and replication process follows these steps:

1. The Leader writes data to the **Write-Ahead Log (WAL)**.
2. The write is replicated to all Followers, who also write to their own WAL.
3. Followers asynchronously persist the data to RocksDB and acknowledge the Leader.
4. Once the Leader receives acknowledgments from all Followers, it commits the write to RocksDB.

The diagram below illustrates this process.
![image](https://www-cdn.nebula-graph.io/blogs/storage_process.png)

# High Availability with Zones

For environments requiring high availability, we recommend distributing partition replicas across different zones. A zone refers to a distinct logical or physical grouping of infrastructure. In cloud environments, zones typically correspond to availability zones (AZs) [3] - geographically separate data centers within a region. These zones are close enough to support low-latency connections but far enough apart to minimize the risk of simultaneous failures due to localized disruptions (e.g., power outages or extreme weather). In this setup:

* Writes are always handled by the Leader, regardless of zone placement.
* Reads can be optimized with **zone affinity**, where queries prioritize a Storaged instance within the same zone as the compute node (Graphd) handling the query. 
* With zone affinity enabled, both Leaders and Followers can serve read requests, improving efficiency and reducing cross-zone data transfer costs. (E.g., In AWS, intra-zone data transfer is free.)

# Diagnosing High Storaged Latency: Common Causes and Solutions

In production environments, high P90 query latency and increased query failures often signal performance issues. While some problems originate from Graphd (our compute nodes) or Metad (our meta service nodes), when the root cause lies in Storaged, it always manifests as high storage latency. This section covers the most common causes of high storage latency and how to mitigate them.

## 1. Low Resource Utilization

If Storaged latency is high, but CPU usage, disk I/O, and disk throughput remain low, the software may not be fully utilizing available hardware resources. A key area to check is thread allocation for the Thrift server and RocksDB. 

Storaged relies on three main thread pools:
- Thrift IO thread pool – Handles incoming connections in an event loop. 
- Thrift worker thread pool – Handles service requests, including writes.
- Read handler thread pool – Manages disk reads via RocksDB.

The Thrift IO threads process the connections very fast and rarely cause latency issues. If latency persists, focus on the worker and read handler pools. In production, setting these pools to match the number of CPU cores on the machine or pod running Storaged typically yields optimal performance.

## 2. High CPU usage

If high CPU usage is observed across all Storaged instances, the root cause likely falls into one of the following categories.

### (1) CPU Spikes During Data Ingestion

If high CPU usage occurs primarily during data ingestion, the most likely cause is RocksDB compaction [4]. Compaction is an essential process that merges and reorganizes data in RocksDB’s Log-Structured Merge Tree (LSM tree) to optimize read performance. However, it can be resource-intensive.

To mitigate excessive CPU usage during compaction, consider the following optimizations:

* **Reduce compaction concurrency:**
    * Lower `max_subcompactions` and `max_background_jobs` to slow down compaction and distribute CPU load more efficiently.
* **Optimize custom compaction filtering**:
    * In NebulaGraph, expired data is garbage-collected using a **custom compaction filter** [5] when TTL is enabled [6]. This filter scans affected vertices and edges during compaction and verifies whether their TTL has expired. This additional check incurs CPU overhead.
    * The config `min_level_for_custom_filter` determines at which LSM level the custom compaction filter starts processing data.
    * A lower value of `min_level_for_custom_filter` will scan more levels of data and thus increase CPU load.
    * A higher value of `min_level_for_custom_filter` postpones TTL filtering to later stages, which reduces CPU usage but increases the temporary retention of expired data.

Since an excessively high value of `min_level_for_custom_filter` can result in bloated storage and potential availability issues, careful fine-tuning is necessary. The goal is to determine an optimal value that balances CPU efficiency with the retention of expired data. We will explore the impact of accumulated expired data later in this article.

### (2) Persistent High CPU Usage, Regardless of Writes

If high CPU usage persists even when there are no active writes, the cause is likely related to RocksDB read operations or thread contention within Storaged. To diagnose the issue more precisely, we recommend running `perf top` on a high-CPU Storaged instance.

* On physical or virtual machines, running `perf top` is straightforward.
* On Kubernetes deployments, additional privilege changes are required. Our Nebula Operator [7] includes the necessary configurations and can enable `perf top` with a single setting.

If `perf top` shows excessive thread context switching and preemption occurring within Thrift’s internal queues, this indicates over-threading - too many active threads competing for CPU resources. Reducing the number of worker and read handler threads can improve CPU efficiency.

## 3. High Disk I/O Utilization

If IOPS or I/O throughput reaches the disk's limits, query latency can degrade significantly. Identifying the underlying cause is crucial for maintaining performance.

### (1) High I/O During Data Ingestion

If excessive disk I/O occurs only during data ingestion, the likely causes are:
* High data ingestion speed
* High compaction concurrency

To alleviate this, consider:
* Reducing ingestion speed or adjusting the batch size
* Lowering the values of `max_subcompactions` and `max_background_jobs` to limit compaction concurrency

### (2) Constantly High I/O with Storage Errors

If disk I/O remains persistently high and approaches the disk's limit, and error logs such as `RPC failure in StorageClient` with `timeout` or `loadshedding` appear in Graphd, this likely indicates excessive amount of expired data..

In this case, Storaged is scanning the disk excessively to filter out expired data, which significantly increases I/O load. To verify this, check the statistics by running:

```
SUBMIT JOB STATS;
SHOW STATS;
```
If the amount of stale data is twice the normal level or higher, trigger a full compaction to clean up expired data:

```
SUBMIT JOB COMPACT;
```
# Conclusion

Storaged is a critical component of NebulaGraph, and its performance directly impacts query latency and system stability. This article provided insights into how data is stored, partitioned, replicated, and accessed in Storaged, along with best practices for diagnosing and resolving high storage latency.

#Reference

[1] Rocksdb. https://github.com/facebook/rocksdb 

[2] Raft. Ongaro, Diego, and John Ousterhout. "In search of an understandable consensus algorithm." 2014 USENIX annual technical conference (USENIX ATC 14). 2014.

[3] Availability Zone. https://learn.microsoft.com/en-us/azure/reliability/availability-zones-overview?tabs=azure-cli 

[4] Rocksdb Compaction. https://github.com/facebook/rocksdb/wiki/Compaction 

[5] Compaction Filter. https://github.com/facebook/rocksdb/wiki/Compaction-Filter 

[6] TTL in NebulaGraph. https://docs.nebula-graph.io/3.8.0/3.ngql-guide/8.clauses-and-options/ttl-options/

[7] Nebula Operator. https://github.com/vesoft-inc/nebula-operator 





This article explores the architecture of Storaged from the perspective of its underlying storage engine, RocksDB, and shares insights from real production use cases on diagnosing and resolving performance bottlenecks.

When NebulaGraph Slows Down: Diagnosing Storage-Related Performance Issues

Since the 80s, graph databases have been gaining popularity in various fields such as data analytics and other areas. However, in the last 20 years, we have seen a sudden increase in demand for graph databases due to the need for fast and powerful data-handling systems.

In 2023, the graph database market was valued at $2.9 billion. With more users adopting these databases; the market value is expected to grow to [USD 7.3 Billion](https://www.marketsandmarkets.com/Market-Reports/graph-database-market-126230231.html) by 2028. 

As you can expect, the need for these infrastructures has seen many players venture into the market. Among the many vendors, Amazon Neptune and NebulaGraph are some of the top considerations. A thorough comparison of the two will help you pick the most suitable for your task. 

# Amazon Neptune and NebulaGraph Overview 

These two databases are popular and boast great power and other amazing features. In this overview, we explore each in brief. 

## Amazon Neptune 

Amazon Neptune is designed and hosted by Amazon Web Services. Amazon Neptune is a managed database with high integration ability with other Amazon Web Services.  It uses the Amazon [Neptune Analytic engine](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/what-is-neptune-analytics.html) to enable super-fast data look-up and analysis with minimal latency. The built-in algorithm, in-memory computing and vector search ensure that the database can perform billions of processes per minute, hence quick results. 

* In terms of schema, Amazon Neptune is schema-free 
* For access management, you can define access rights through AWS IAM
* Neptune does not offer in-memory capabilities

## NebulaGraph 

NebulaGraph is another popular name on the market, a powerful [open source graph database](https://nebula-graph.io/) dedicated to storing, analyzing, and retrieving information from stored data. The database possesses amazing performance with super low latency, allowing it to achieve billions of queries per second. In addition to the open source option, NebulaGraph also offers an enterprise edition that comes with 24/7 support and professional consulting. [Compare the plans](https://www.nebula-graph.io/pricing). 

NebulaGraph’s database architecture allows linear scaling, meaning it can continuously replicate as you add more resources. Due to its scalability and flexibility, NebulaGraph is mostly used by Fintech companies and large internet firms, among others. 

* In terms of schema, NebulaGraph offers strong typed schema 
* NebulaGraph uses role-based access control
* NebulaGraph offers in-memory capabilities

NebulaGraph is the only graph database solution that can host graphs that contain dozens of billions of vertices and trillions of edges. All this while still offering millisecond latency, high availability and robust security for your data.

# Amazon Neptune Vs. NebulaGraph comparison: Development and Features Comparison

Although these databases are designed to deal with graphical data, they bear important differences in aspects such as the architecture, query language, and other qualities. 

Let’s compare them on the following key items that should help you pick the right one for your use case. 

In terms of release dates, Amazon  Neptune entered the market in 2017, while NebulaGraph came 2 years later in 2019.  

## 1. Database Developer 

Amazon Neptune, as the name suggests, is developed and maintained by Amazon through Amazon Web Services.
 
On the other hand, NebulaGraph is designed by Vesoft Inc. The company is founded by engineers from top tech giants such as IBM, Alibaba, and Facebook among others.

Being open source with a good name behind it, NebulaGraph is easy to tweak to suit your needs. Also, it allows for high transparency and flexibility.  

## 2. Querying Language 

Unlike some years back when querying a database was an uphill task, graph databases are much easier to use as developers are crafting derivative languages to ease accessibility. 

Both Amazon Neptune and NebulaGraph have their distinct query language(s). Amazon Neptune uses query languages like TinkerPop Gremlin and RDF 1.1 / SPARQL 1.1. This is because the database supports Property Graphs and W3C’s RDF graphs.
 
On the other hand, NebulaGraph uses the [NebulaGraph Query Language](https://docs.nebula-graph.io/3.0.0/3.ngql-guide/1.nGQL-overview/1.overview/) (nGQL), an SQL derivative. nGQL is a declarative language, meaning you don’t need to specify every step. Once you tell the database what you want, it takes you there. Thus, it's an intuitive and easy-to-learn language. The language is simplified to give users simple utilization; thus, [developers and operators](https://www.nebula-graph.io/posts/graph-query-languages-you-should-know) can use it with ease. The release of NebulaGraph 3.0 enhanced the query language to support OpenCypher. 

## 3. Licensing

NebulaGraph is an open-source graph database that also offers an enterprise version.  Based on the open-source edition, the [enterprise version](https://www.nebula-graph.io/pricing) has value-added features and support services that include  24/7 hours on-call. 

Even though not modifiable, it is designed to work seamlessly. Also, the professional support and high flexibility offered by the organizations behind the software make them a great choice. Amazon Neptune is a commercial graph database.  

## 4. Scalability

Well, when it comes to scalability, Amazon Neptune and NebulaGraph both offer this feature. Amazon Neptune has automatic scalability, where the storage and processing power increase automatically depending on your requirements. With the ability to grow your storage volume up to 128 TiB, it means more processing power even for complex tasks. 

On the other hand, NebulaGraph’s distributed architecture allows horizontal scalability. This means that whenever the demand rises, all that is required is to add more servers without compromising on performance and stability.  

# Amazon Neptune Vs. NebulaGraph:  Use Cases Comparison

Both graph databases have vast applications in different fields. While they are utilized in similar fields, the querying and general functionality are quite different. 

Let’s compare the popular use cases below.  

## 1. Fraud Detection

For both Neptune and NebulaGraph, relationships form the foundation for fraud detection. Since both can execute fast queries, it's easy to find relationships that point to fraud.

To create graph fraud detection solutions with Amazon Neptune, you start by loading data directly into Neptune. With the data now in Neptune, you can use Neptune alongside Amazon SageMaker to build machine learning models that can identify fraud.

Fraud detection with NebulaGraph is made easy with Nebula Explorer, which is a graph data exploration app that has been built upon the NebulaGraph database. You can load huge amounts of data and then query it with visualized graphs to visually identify fraudulent activities.

The key difference here is that while both graphs support loading and querying large-scale graph data, NebulaGraph goes a step further to enable visualization.

## 2. Social Recommendations

Both graphs can support recommendations of social networks. First, it’s always important to first understand the purpose of the social graph that you want to build so that you can build it right for the system that will eventually use it, such as CRM. 

For both you will load the data, build the social graph, then use it to achieve different social functions. 

For Amazon Neptune, you can expose the graph data to external systems using the Amazon API Gateway.    It's also capable of running interactive graph queries, which make it easy to equip applications with amazing social features. You can, for instance, build a social feed that prioritizes updates from people that users would be most interested in such as family members.

For NebulaGraph, again visualization is a distinct feature through the Nebula Explorer. You can visualize relationships and easily enable social actions such as new friend recommendations, finding common neighbors, feed generation, etc.

See how [WeChat](https://www.nebula-graph.io/posts/nebula-graph-for-large-social-network) customized NebulaGraph to handle their large social datasets.

**Also Read:** [Neo4j vs NebulaGraph vs JanusGraph](https://www.nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph)

## 3. Knowledge graph

As may know, knowledge graphs have become popular in use cases such as chat robots, risk control in big data, and medical treatment, among others. This makes them very important.

Both graphs enable the building of knowledge graph applications. It means that these applications will support the storage of data in a graph model. Queries are then used to navigate the graph datasets.

Neptune being a managed graph means that you can host the knowledge graph on their managed service, making it easier to use.

NebulaGraph makes use of property graphs to store relationships. Scaling is made easier through separation of storage and computing architecture.

Through collaboration with renowned LLM teams worldwide, NebulaGraph is integrating LLMs with graph databases. Thanks to these efforts, NebulaGraph now supports Text2Cypher in  a Graph + LLM first with the capability to translate natural language into graph queries. This revolutionary technology enables users to query knowledge graphs with the use of natural language via a dialogue interface. What this means is that companies will no longer need to invest in long learning curves to bring their developers or users to a level where they can extract meaningful insights from graph data. This creates the opportunity to create knowledge graphs from huge and complex data at a much lower cost. Get a first hand [demo](https://www.nebula-graph.io/posts/build-knowledgegraph-via-natural-language-demo) and see for yourself how to build and query a knowledge graph with the use of natural language.

## 4. Ecommerce recommendation engines

Recommendation systems have become popular in industries like eCommerce. It's all about recommending stuff to customers, be it products, services, games, content, etc.

Both Amazon Neptune and NebulaGraph support the building of recommendation engines by making it possible to store relationships. These relationships are then queried to provide real-time recommendations

Amazon Neptune relies on Graph Neural Network (GNN), which is a deep learning framework that can be used in graph data to perform prediction tasks at node level, edge level or even at graph-level. Algorithm implementation is achieved through Neptune ML.  

In NebulaGraph, developers use NebulaGraph's query language (nGQL) to implement recommendation algorithms. These algorithms will consist of queries that identify patterns as well as relationships between users and items. Once you finish building the recommendation engine, you can use Nebula's APIs and connectors to integrate it with the eCommerce platform. See [how to use NebulaGraph to design a recommendation engine](https://www.nebula-graph.io/posts/use-cases-of-graph-databases-in-real-time-recommendation). 

NebulaGraph is also big on data visualization, which is key in building efficient recommendation engines. [NebulaGraph Explorer](https://explorer.nebula-graph.io/login) is the tool that brings visualization to life. Developers use this tool in combination with NebulaGraph core to visualize massive interactions in graph data.

## 5. Network and IT Infrastructure Security 

With Neptune's fully managed service, it's easy to execute fast queries. This makes it possible to detect the wrong access or exposed resources in real-time. You can also export the security graph data to tools such as Splunk to help you dig deeper into insights. Still on insights, the native integration of Neptune with Amazon OpenSearch Service can also come in handy.

When it comes to network security, NebulaGraph excels in protecting sensitive information and detecting potential threats. Through native storage and the shared-nothing architecture, NebulaGraph supports tracking the network and device paths. Any potential threats found can be responded to instantly.

The key difference here is that while Amazon Neptune focuses on making it possible to build security graphs that perform fast queries and analyze the data for insights, NebulaGraph is big on real-time tracking, finding potential threats and responding promptly.


# Conclusion 

From the comparison above, it is evident that these databases are powerful and offer amazing functionality. 

Amazon Neptune is suited for use cases that lean more towards Amazon Web Services since it allows direct integrations with these services.
 
On the other hand, NebulaGraph provides amazing power in data analysis across the board. 


Are you torn between using Amazon Neptune and NebulaGraph? Which one should you go with? Check the latest Amazon Neptune vs. NebulaGraph comparison and be the judge. 

Amazon Neptune vs NebulaGraph

Many organizations are keen to harness the full potential of big data. Central to the intricacies of big data lies a seemingly straightforward yet profoundly complex element: relationships. 
When the relationships in a given data set are complex, making sense out of this data becomes a tough nut to crack. Luckily, graph databases have emerged as the ultimate solution to this challenge.
The graph database market has significant options, which is good for solution seekers. The only challenge comes in when you want to pick one over the other. If this is the kind of challenge you are facing in choosing between Memgraph and NebulaGraph, then this resource should hopefully answer your questions and give you a clarification to make the ideal decision.
Let's start by looking at each, in brief.

# NebulaGraph

NebulaGraph is an [open source graph database](https://nebula-graph.io/) that is scalable with lightning fast performance. The impressive scalability is derived from its shared-nothing architecture whereas its swift and real-time traversal is thanks to its native graph storage architecture. It’s suitable for real-time recommendation, risk control, knowledge graph and network security. 
NebulaGraph has been [recognized by Gartner](https://www.nebula-graph.io/posts/gartner-market-guide) in the Market Guide for Graph Database Management Systems.

# Memgraph

Memgraph is an in-memory open source graph database that can query large scale graph data. It's suitable for analytics needs. It's essentially a memory-first graph that is built in C/C++ and boasts a redundant architecture.
**Also Read:** [Neo4j vs TigerGraph vs Dgraph vs NebulaGraph](https://www.nebula-graph.io/posts/best-graph-database-for-enterprise)

## At a glance : A comparison of essential features between Memgraph and NebulaGraph
| Component |Memgraph | NebulaGraph|
| --- | --- | --- |
| Deployment | On-premise or Cloud | On-premise or Cloud |
| Programming Language | C++ | C++ |
| License | Apache 2.0, BSL, Commercial | Apache 2.0, Commercial  |
| Hosting | Memgraph Cloud | NebulaGraph Cloud |
| Query Language | Cypher | Nebula Query Language, compatible with OpenCypher |
| Data Schema | Flexible | Flexible |
| Current version | v2.12.1 | v3.6.0 |
| Stars in Github | 1.9k | 9.7k |
| Year of Release | 2017 | 2019 |


# Memgraph vs NebulaGraph: Detailed comparison 

At the onset, we would like to make it clear that both Memgraph and Nebulagraph have distinct strengths that make them great for graphical data for different needs. This comparison, therefore, is designed to help you refine your pick depending on what you want your database to achieve. 
The comparison is based on a series of aspects that we believe are critical when working with data. 

## Performance

NebulaGraph’s native graph engine unleashes extremely fast QPS and TPS with negligible latency whose range is in the milliseconds. With this, you can process extremely huge volumes of data in a snap! 
Memgraph also demonstrates strong read performance, particularly for small to medium-sized graphs. Its in-memory storage and parallel processing capabilities are instrumental in achieving efficiency in handling read requests.
When the [Tencent Cloud Security team tested NebulaGraph](https://www.nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph) alongside other graphs, specifically Neo4j and JanusGraph, NebulaGraph emerged the fastest for larger data sets.

## Ease of use

NebulaGraph provides a user-friendly web console for managing the graph database, creating and executing queries, and [visualizing](https://www.nebula-graph.io/products/studio) graph data. 
Memgraph similarly provides a user-friendly web interface for managing the graph database, executing Cypher queries, and visualizing graph data through [Memgraph Lab](https://memgraph.com/docs/data-visualization).

## Security

Both NebulaGraph and Memgraph prioritize data security with a comprehensive set of features to protect graph data from unauthorized access and maintain data integrity. Both databases implement [role-based access control](https://www.nebula-graph.io/posts/access-control-design-code-nebula-graph) (RBAC) to restrict data access based on user roles and permissions. 
They support multiple authentication methods. These include password and token-based authentication, as well as  LDAP integration. This is in addition to strong encryption to safeguard data at rest and in transit.
To ensure traceability and accountability in case of security incidents, both NebulaGraph and Memgraph provide comprehensive [auditing capabilities](https://docs.nebula-graph.io/3.1.0/5.configurations-and-logs/2.log-management/audit-log/). All user actions are logged, including data modifications. They also maintain [vulnerability disclosure](https://memgraph.com/policies/disclosure) programs and promptly release security updates to address identified vulnerabilities. 

## Scalability

NebulaGraph is horizontally scalable. It means you can add more nodes to the cluster to handle increasing workloads. 

Memgraph is vertically scalable, meaning that you can upgrade the hardware of your existing nodes to handle increasing workloads.

## Use cases


### NebulaGraph is well-suited for the following use cases

1. Real-time recommendation engines: NebulaGraph can capture user preferences that are useful for building recommendation engines that can personalize product suggestions, content recommendations, and more. This capability is fantastic for eCommerce platforms. 
2. Social networks: NebulaGraph's ability to efficiently store and query interconnected data makes it the best graph database for managing social networks. It powers efficient friend recommendations, community detection, and influencer analysis.
3. Financial risk control/fraud detection: NebulaGraph can effectively handle the highly complex data that comes from fraudulent transactions and return results immediately in the course of the transaction. 
4. Knowledge graphs: The ability to store and query semantic relationships makes NebulaGraph suitable for building knowledge graphs that can represent and organize information from various sources, enabling knowledge discovery and inference.
5. Cybersecurity:  Through native graph storage and shared-nothing architecture, NebulaGraph is capable of tracking  the paths of networks and devices to discover potential threats.


### Memgraph is suited for these use cases

1. Cybersecurity: Like NebulaGraph, Memgraph can also be used in cybersecurity to detect threats. 
2. Recommendation engine: Suitable for building recommendation engines aimed at enriching customer experience and engagement.
3. Identity and Access Management: Build IAM systems capable of monitoring complex permissions.
4. Network Resource Optimization: Memgraph's real-time, concurrent, write-heavy data capabilities, powered by its C++ in-memory architecture, make it ideal for optimizing network resource utilization.


## Data model

For NebulaGraph, six data structures are used to store data. These structures include:

* Graph spaces
* Vertices
* Edges
* Tags
* Edge types
* Properties

Read more about the [NebulaGraph’s data structures](https://docs.nebula-graph.io/3.6.0/1.introduction/2.data-model/)

Memgraph uses data structure types such as Linked List, Matrix, Queue, Stack and Tree Structure. Dig deeper into how these [data structures](https://memgraph.com/blog/data-structures-cheat-sheet) can be represented in Memgraph.

## Indexing 

Both NebulaGraph and Memgraph employ indexing and optimization techniques to enhance performance. 

In NebulaGraph, the indexes are stored together with the raw data (vertices and edges) in a partition. The Raft protocol is used to synchronize different partitions within a storage engine. Multiple storage engines can be contained in one storage server. Read more about [indexing in NebulaGraph](https://www.nebula-graph.io/posts/how-indexing-works-in-nebula-graph). 

Memgraph, on the other hand, leverages the concept of skip lists as the foundation of its indexing. Skip lists are like a smart way of organizing information so you can find things quickly in a list. Imagine you have a list of items, like numbers, in a specific order. Now, instead of going through each item one by one to find what you need, a skip list creates shortcuts or 'highways' to help you get there faster. Picture it like this: the list is like a regular road, and then there are faster roads built above it. The bottom road keeps everything in order, like the original list. The higher roads are like shortcuts that let you skip some items and get to your destination faster. It's a clever way to speed up searching in a list. Read more about [how indexing works in Memgraph](https://memgraph.com/blog/how-to-read-query-execution-plans-optimize-queries). 

## Consistency 

Both NebulaGraph and Memgraph offer consistency and availability guarantees. They ensure data integrity and accessibility even in the face of failures or network partitions. 
NebulaGraph provides eventual consistency, where all replicas are eventually synchronized. A feature known as TOSS was introduced from the start of v.2.6. to guarantee eventual consistency of edges upon INSERT, UPDATE, and UPSERT. Find out more about [NebulaGraph’s eventual consistency](https://www.nebula-graph.io/posts/nebula-graph-v2.6-toss). 
Memgraph also provides strong consistency, guaranteeing that all replicas are consistent at all times. It achieves this by using the RAFT algorithm for replication.  The RAFT algorithm acts like a voting system among replicas: one node becomes the "leader", proposing changes to the database. Other nodes vote on those proposals, and if a majority agrees, the change is applied to all replicas simultaneously, ensuring up-to-date information, thus achieving strong consistency. 

## Community
 
Both NebulaGraph and Memgraph have active and growing communities that provide support and resources for users. 
NebulaGraph's community is particularly large and well-established, with a strong presence on [GitHub](https://github.com/vesoft-inc/nebula) and a [forum](https://community-chat.nebula-graph.io) hosted within NebulaGraph . It offers extensive documentation, tutorials, and code examples to help users get started and troubleshoot issues. 
Memgraph's community is smaller but growing and especially active in [Discord](https://discord.com/invite/memgraph). It also provides comprehensive documentation, tutorials, and dedicated support channels.

# Conclusion

NebulaGraph and Memgraph are both powerful graph databases with distinct strengths. NebulaGraph excels in managing complex data, horizontal scalability, strong consistency guarantees, and support for time-series data. 
Memgraph excels in real-time processing and analytics.
To try out NebulaGraph, you're most welcome to the [NebulaGraph Demo](https://www.nebula-graph.io/demo) (No Installation). Feel free as well to explore the [NebulaGraph Manual](https://docs.nebula-graph.io/3.6.0/), and join the vibrant [Nebula community in Github](https://github.com/vesoft-inc/nebula/stargazers).




If you are choosing between Memgraph and NebulaGraph, then this resource should hopefully answer your questions and give you a clarification to make the ideal decisionю

Memgraph vs NebulaGraph Comparison

Generative A.I. (Gen A.I.)  is taking the world by storm. Before the advent of Gen A.I., machines could not produce content without human intervention. 
What makes Gen A.I. revolutionary is that this intelligence equips machines with the ability to produce varying content – music, images, art, etc. – that’s similar to what humans produce. To achieve this feat, Gen A.I. memorizes existing data and, using advanced machine learning models, it can near-perfectly mimic human behavior.
The ability of a generative A.I. model to produce content is influenced to a great extent by the training method used. Luckily, developers have various training methods at their disposal. 
In this article, we take a look at key training methods and show how the use of graph databases improves accuracy and enhances the training of generative models. 

# What are generative models?

Generative A.I. models are sets of algorithms that use training data to produce content that resembles human production. As a result of generative AI training, machines today can produce text, audio, and also create images that we could not dream of  in the not-too-distant past.
While it’s still at its infancy, Gen A.I. has taken a hold on various industries. This is down to the ability to accomplish tasks at speeds beyond human ability. Gen A.I. improves efficiency and since it requires little or no human intervention, it reduces labor costs. The technology is currently being applied in various industries including:
* Manufacturing
* Finance
* Entertainment
* Healthcare

This is an exploding technology that is projected to be worth [$110.8 billion](https://masterofcode.com/blog/generative-ai-for-enterprises-statistics-use-cases-business-examples#:~:text=Companies%20are%20using%20Generative%20AI,easier%20and%20insurance%20more%20efficient.) by the year 2030. A few factors make the adoption of Gen A.I. inevitable. Business owners particularly find it appealing for these reasons:
* **Cost cutting:** AI-powered  solutions outperform human labor and reduce HR costs..
* **Greater efficiency:** Machines and robots perform with greater efficiency than humans
* **Faster decision-making:** Company executives might take too long to arrive at a decision because of lack of credible information. Gen A.I. hastens this process by providing quick and precise data that humans might not be aware of.


# How to train a generative AI model with  graph database

So, what steps do you need to take to train generative AI models using graph databases? Two major steps come into play.  

## 1. Identify use cases

To decide where you want to implement graph A.I., you might want to identify patterns and trends that impact your business.
If your company is being mentioned in social media, for example, you might want to analyze the posts and the relationships between the users. Should you run a clothing eCommerce site, you might want to know how changes in the seasons affect your sales.

## 2. Choose the right graph database

* Define the entities, properties, and relationships relevant to your case
* Select an A.I. [graph database](https://nebula-graph.io/) platform. Leading graph databases include **NebulaGraph**, Amazon Neptune, Neo4j, Microsoft Azure, among others 
* Import data to the graph database. To make machine learning effective, ensure that the imported data is devoid of inconsistencies and is properly formatted.

Also Read: [How to import data from Neo4j to NebulaGraph](https://www.nebula-graph.io/posts/neo4j-nebula-graph-import-best-practice)

## 3. Algorithm building

Once a graph database is in place, next is to build algorithms and queries to extract insights from the data.
From the algorithms and queries created, the machine is taught by being presented with a host of tasks. The tasks that the machine will be required to perform will vary depending on whether the teaching method is supervised or unsupervised. 

### Supervised learning

For supervised learning, the machine could be required to:
* **Predict whether a relationship exists between nodes in the database.** Should a relationship exist, what properties can be deciphered about that relationship? This is the kind of querying used by eCommerce stores such as Amazon for their [recommendation engines.](https://www.nebula-graph.io/posts/recommendation-engine-for-ecommerce) Once a buyer purchases a particular item, for example, what else might they want to buy?
* **Predict the properties of a node.** Predicting the properties means that the machine is required to classify it. If, for instance, it’s presented with a blouse in a retail store, where is it going to classify it?

### Unsupervised learning

For unsupervised learning, the machine could be tasked to:
* **Find and measure similarity or dissimilarity between pairs of nodes.** By telling the differences between data points, the machine is being taught to understand relationships and patterns within data. This is the kind of technology used for image recognition and anomaly detection. If the machine was presented with two images, similarity algorithms will help it tell whether the two feature the same person or different people.
* **Conduct community detection.** In graph machine learning, community detection is used to locate groups with similar attributes. In a network, a community is identified by the presence of similar nodes. When analyzing social posts, the machine can detect a community that makes similar posts from the similarity of the nodes.
* **Carry out representation learning.**  Some data could have a lot of hidden features. Representation learning helps algorithms to discover such features. Moreover, this process helps algorithms extract patterns from data and make it easier to understand.


# Why graph databases are best for training generative models 

Unlike relational databases, graph databases contain features that make a huge difference in  machine learning.

Let’s look at the top ones. 

## 1. Graph databases can handle huge amounts of data, which enhances scalability

A generative model that’s trained on a huge amount of credible data will ultimately perform better than one trained on lesser data. When graph technology is applied, a generative model can be trained on a vast amount of data as this technology enables for faster data processing.
When compared to relational databases, for example, graph technology provides faster performance because the latter uses indexes and pointers to reach nodes and relationships without necessarily having to scan all the material collected. The availability of vast datasets simply provides ideal conditions for machine learning.
Should there be a reason to augment the training dataset, graph databases can be scaled up to meet the new requirement.

## 2. Graph databases offer exceptional visualization 

The aim of all generative AI models is to make machines operate the way humans do and hence create content similar to what human beings could create. [Data visualization](https://www.nebula-graph.io/posts/graph-database-visualization) is important here because it helps the machine (or person) looking at some content to see how several points are related to each other.
Data from relational databases which is presented in the form of chart bars, pies, and percentages does not help us see the relationship between the various points. But by using nodes and edges to present relationships, graph databases help the learning machine to quickly decipher the meaning behind the data.

## 3. Graph databases offer greater context

To develop intelligence that is as close to that of humans as possible, machines need to put the data they are trained on in context. In a structured Query Language (SQL) system, data is presented in rows and columns but, looking at it, it’s never easy to tell how the various points in the data are related.
Since graph databases are centered on the relationship between the various points, a machine trained using these graphs can see the context. The machine can notice the structures and patterns in the training data, then reproduce similar versions with near-human creativity.

## 4. Graph databases are analogous to the human brain

The structure of the human brain and that of graph databases share astounding similarities which make the latter ideal for training generative models.
As noted earlier, a graph database consists of vertices (or nodes) and dots which are connected by edges. A graph database, therefore, is a network of vertices. The human brain, on the other hand, is made up of billions of neurons and over a quadrillion connections. When it comes to the identification of relationships, a graph database, therefore, operates exactly like the human brain.
Other generative training models might falter or fail outright to see connections between training data. But graph science has the infrastructure in place to develop algorithms that develop problem-solving skills in a manner that is similar to the human brain.

## 5. Graph databases deliver greater analytical accuracy

Since graphs can clearly identify the relationship between vertices and edges, they allow for appreciation of the real-world situation. 
Compared to other methods of generative model training, graphs are better at ensuring that the end product is almost identical to what an intelligent human being could produce.

## 6. Graph database deliver easy to understand insights
 
In many organizations, a lot of information is only understood by a tiny fraction of nerds who might have neither the time nor the ability to explain complex concepts to their colleagues or customers. Unfortunately, some of the people who actually need to grasp such concepts could be executives and decision-makers.
In the world of academia, for instance, a lot of information is available in jargon which the average person can hardly comprehend. Training machines using graph databases could help LLMs produce data in a form that is devoid of jargon and which ordinary people can understand. 
Jim Webber of InfoWorld in [How knowledge graphs improve generative AI](https://www.infoworld.com/article/3707814/how-knowledge-graphs-improve-generative-ai.html) gives the example of a publishing company that is preparing an A.I. tool. The tool is trained using knowledge graphs to produce content that will be accessible to users in plain language.

## 7. Graph databases solve most of the problems that data scientists encounter 

Graph databases excel in addressing these common challenges that data scientists are constantly fighting when using traditional approaches. 

### Data enormity

The amount of data that an average organization might have to analyze to help executives make informed decisions could be overwhelming. 
For example, the chief decision makers need to understand how company operations are getting affected by the competition, gauge customer sentiments, and find out what social media commentators are saying about the company. In a nutshell, there’s so much data to analyze and, unfortunately, traditional data processing methods lack the ability to do so.
Graph databases provide an ideal solution for organizations that lack the capacity to handle such data. Where traditional analysis methods could take days or even months to provide insights from big data, the use of graph technology enables organizations to analyze such data in real time. This is possible because they store relationships using the connectivity between vertices. By utilizing this connectivity, queries can be answered in milliseconds.

### Poor quality data

Data quality has a direct influence on data analyses – if the quality is poor, so will the analyses. In many organizations, data is stored in messy and inconsistent conditions. Such data can hardly be relied upon for correct analyses.
When you are unsure of the quality of the company’s data, you must make  efforts to clean and preprocess it before using it.  Cleaning and other data validation techniques are time-consuming and expensive. But with graph databases, such challenges vanish.
Unlike traditional data storage models which store data in haphazard ways, the data in a graph database is stored in a state that shows clear relationships between various elements of data. This greatly enhances the quality of the data. 
If, for example, your company has been maintaining data manually and wishes to automate the process, a graph database ensures the availability of high-quality and immediately actionable data. 

### Difficulties communicating technical stuff to non-technical people

While data scientists are technical people, they have a responsibility to communicate to people who aren’t tech-savvy. A slight miscommunication can deny decision-makers the necessary actionable insights.
Luckily, graph databases make data storytelling a breeze. From the graphs, even the most tech-challenged person can decipher clear relationships between nodes and edges and visualize how such data can aid in effective decision-making.

# Conclusion

Given the ability of graph technology to overcome hurdles that have always hindered other training models, graph databases are gradually taking center stage in generative model training.
It's no wonder then that today, graph databases are used in many learning areas including natural language processing (NLP). Through NLP, machines are taught to understand and generate human language.
In NLP,  graph databases could be used to model the interlinkage between words and phrases. The more the generative model learns how certain words and phrases are related, the better it gets at tasks such as translation, text classification, and sentiment analysis. 
NebulaGraph unleashes mature graph database technology to power the development of powerful custom graph-driven [LLMs](https://www.nebula-graph.io/posts/graph-ai) for industrial applications, at a significantly lower cost. [Get Started](https://hub.docker.com/extensions/weygu/nebulagraph-dd-ext).

Graph databases have emerged as the go-to solution for training generative models. They address major shortcomings in traditional training techniques.

How to Train a Generative AI Model Using Graph Databases 


This sharing is mainly from the perspective of (potential) users and discusses about what considerations the technical team will have when selecting a database. At the very beginning, our RoyalFlush Knowledge Graph team chose Neo4j when the knowledge graph strategy was started. However, during the using process, it turned out that Neo4j could not handle our growing business well. Therefore, the technical team came up with a solution: adding a multi-level cache between the database and the application layer. Although caching can alleviate the problem to a certain extent and improve the operation performance, it actually causes more problems.

So, we began to think about whether there is a [graph database](https://www.nebula-graph.io) that can not only meet the current business needs, but also is capable of handling the future business growth. Thus, our Knowledge Graph team starts the journey of selecting an ideal graph database.

# How to select a graph database

![How to Select a Graph Database: Best Practices at RoyalFlush](https://user-images.githubusercontent.com/90186547/142156266-3654804f-ec19-4241-a8ef-80f365dbf5e0.jpg)

This speech can be divided into three parts: How to choose, What to choose, and Why. In addition, our RoyalFlush Knowledge Graph team and related businesses will be introduced at the end.

When selecting a Graph Database, we will consider the following aspects: 

* Inclusive functions with outstanding performance
* Commercially mature products
* Active community
* Large amount of data to handle the current business and future business growth
* Open-source preferred
* Technical support
* Domestic preferred

Selecting a graph database is similar to select a stock. We will take information analysis, fundamental analysis, and technical analysis into consideration. In the world of the graph database, they mean:

* Information analysis: Whether the graph database meets the above seven demands
* Fundamental analysis: Whether the graph database is stable and durable
* Technical analysis: Whether the graph database meets the technical requirements

## Information analysis

![Information analysis](https://user-images.githubusercontent.com/90186547/142158108-5b94ca33-1f96-450e-a6e7-5eea8578785d.png)

NOTE: The screenshot is the DB-Engine ranking in Sep. 2021 when this talk was given.

Generally, we will go to DB-Engine for the information about the graph database. You can see that Neo4j ranks first here. After all, it is the earliest graph database that is relatively mature. After a round of research, we found that Neo4j did a great job in marketing. If you search "graph database" on Baidu or Google, their ads will show up.

In this DB-Engine ranking, you can see other famous graph databases like JanusGraph (ranked seventh). So, among so many databases, how do we determine the database we will ultimately use?

![Information analysis](https://user-images.githubusercontent.com/90186547/142160038-9f889dbe-a35c-4716-8eb9-e7226bfcac3d.png)

As for the ranking rules of DB-Engine, it is based on the search results of Google and Bing, keyword trends in Google Trends, discussions on Stack Overflow, DBA Stack Exchange, the number of posts on recruitment websites like Indeed and Linkedin, and the amount of discussions on social networks like Twitter.

We can find a problem: the above websites are either inaccessible in China or relatively niche websites.

Therefore, the ranking on DB-Engine is actually biased for some domestic products.  At least its data certainly cannot reflect the true conditions. If some domestic products are also included, the rankings of HugeGraph and NebulaGraph in this DB-Engine ranking will definitely ascend a lot.

After knowing what is DB-Engine, let's filter the information on this DB-Engine ranking:

![Information analysis](https://user-images.githubusercontent.com/90186547/142160285-7deea852-f897-4d6d-acfd-c114505fed35.png)

![Information analysis](https://user-images.githubusercontent.com/90186547/142160472-c21dfe95-7f45-426f-8c4d-eb9adde62077.png)

From the view of the information analysis, these databases do not meet the needs of our RoyalFlush Knowledge Graph team. This result also proves that the DB-Engine ranking results are not so accurate. For example, some databases ranked before NebulaGraph are already inactive or no longer updated products. The above screenshots are all from DB-Engine.

## Fundamental analysis

![Fundamental analysis](https://user-images.githubusercontent.com/90186547/142160638-0bc6d130-144e-4b32-8ac6-f9baba294be6.png)

![Fundamental analysis](https://user-images.githubusercontent.com/90186547/142160840-e988a22c-ef6e-4a71-bbb7-638dc4193264.png)

After the information analysis, we filter out 6 graph database products including Neo4j, Orient DB, Dgraph, JanusGraph, HugeGraph, Nebula, etc. These 6 products are relatively active and support multiple languages.

According to the earliest release time of the product, Neo4j is ranked first, which was released in 2007, followed by OrientDB, Dgraph, JanusGraph, HugeGraph, and NebulaGraph which released in 2019. The higher the ranking of the database product, the higher its language support level, and the more complete its system functions. This is also where the new databases need to be perfected and improved.

![Fundamental analysis](https://user-images.githubusercontent.com/90186547/142160840-e988a22c-ef6e-4a71-bbb7-638dc4193264.png)

The functions of many products like HugeGraph and Neo4j are only supported in the Enterprise Edition. We also need to know the content of their Enterprise Edition and the corresponding pricing.

![Fundamental analysis](https://user-images.githubusercontent.com/90186547/142161062-be705952-f83e-4192-b977-2376f0e9bf0e.png)

The above picture is more interesting. It points out the order of magnitude supported by various graph databases in different scenarios. For example, Neo4j supports billions of nodes, trillions of relationships with millisecond response time and unlimited elasticity. OrientDB is more interesting. It supports unlimited data. I don't know how it achieves this value. Dgraph did not clarify the amount of data, but said that it provides Google production level scale and throughput, with low enough latency to be serving real-time user queries, over terabytes of structured data.

As for the rest two, one is HugeGraph, open sourced by Baidu, and the other is NebulaGraph. Nebula is still relatively dominant. It is capable of hosting graphs with hundreds of billions of vertices and trillions of edges, and serving queries with millisecond-latency.

## Technical analysis

![Technical analysis](https://user-images.githubusercontent.com/90186547/142164284-9c1fd737-bc25-4259-b390-1bf542f6f09c.png)

After the information analysis and fundamental analysis, there is actually no clear winner. Let's look at the technical analysis. Since HugeGraph and JanusGraph have similar architectures and the technical support of OrientDB is plain, we will only focus on the following three graph databases: Neo4j, HugeGraph, and NebulaGraph.

Neo4j only supports isomorphic deployment and does not support heterogeneous deployment, so it cannot take advantage of each machine. For example, some machines may be computational-oriented, and some machines have strong IO. If you follow Neo4j's deployment settings, it requires that each machine is in the same condition. The analysis and performance improvement of Neo4j mainly relies on the virtual machine composed of two fabrics to provide regional partitioning capabilities (the green part of the above figure).

At the architecture level, the separated architecture  of storage and computing of NebulaGraph and HugeGraph has more advantages. At least users can configure the machines according to the realities of the business department.

![Technical analysis](https://user-images.githubusercontent.com/90186547/142164746-cd271108-a0f6-4dbb-8bc8-9529d1c74145.png)

HugeGraph supports a variety of storage engines in the storage area, which can be connected to a variety of third-party engines, and then a computing framework is added on this basis. Above the computing framework is the interface layer, such as Huge Console, Huge Loader, etc.

In computing, the storage engine at the bottom mainly provides [OLTP and OLAP](https://www.nebula-graph.io/posts/OLTP-vs-OLAP-what-are-the-differences). Here the storage engine of HugeGraph is not self-provided, but mainly depends on other storage engines. It may not be as good as Nebula's own storage engine. For example, the third-party storage engine we use is HBase. The real-time performance of HBase is very unsatisfactory. At this time, using it as a storage engine will affect the performance of the upper layer.

![Technical analysis](https://user-images.githubusercontent.com/90186547/142167861-1aa1c3c1-e48e-4fe7-bb81-078becaf4cbd.png)

NebulaGraph also applies an architecture that separates storage and computing. Each layer can be expanded infinitely. You can expand the storage/computing layer if it is full. Relatively speaking, this architecture is relatively advanced.

After talking about the architecture, RoyalFlush also refers to the test results of other technical teams:

![Technical analysis](https://user-images.githubusercontent.com/90186547/142168255-8099bcf8-837a-48b5-a0a3-33fc9f81e485.png)

This is the performance test of v1.x carried out by the Tencent Cloud Security Team.

![Technical analysis](https://user-images.githubusercontent.com/90186547/142168458-5d451e57-5947-4ee3-8b22-d205ec60b8eb.png)

This is the performance comparison of Nebula, Dgraph, and HugeGraph carried out by the Meituan NLP team. Here, HugeGraph uses HBase as the storage engine, while Dgraph and NebulaGraph uses RocksDB. As mentioned before, HBase is at a disadvantage in real-time performance. If all three databases use the KV storage engine, there will be no such a large gap in the results.

![Technical analysis](https://user-images.githubusercontent.com/90186547/142168642-ac5b9dff-33e1-42d2-9db9-a1c8a01315fe.png)

The following is the performance test carried out by RoyalFlush. We not only compare different graph databases, but also compare two query languages (native nGQL and nGQL that is compatible with openCypher) of NebulaGraph.

From the data, we can see that the native query language nGQL has better performance (nebula-ngql and nebula-cypher in the figure above). In the multi-hop query, you can see that for queries that are more than 5 hops, Neo4j stays in a timeout state.

In terms of performance, Nebula still has a relatively large advantage over Neo4j.

![Technical analysis](https://user-images.githubusercontent.com/90186547/142173955-941d1da2-ed1e-45f1-87ca-5529927e5497.png)

Considering factors such as architecture, performance, community support, etc., we will still choose a graph database that is more suitable for us, namely NebulaGraph. Although some minor problems were found in the using process:

* Chinese identifiers are not supported.
* Syntax error prompts are not user-friendly.
* Parameterized queries are not supported in Java.
* The source vertex ID must be specified in GO queries.

In general, NebulaGraph still meets our business requirements.

# RoyalFlush · the Knowledge Graph team

![RoyalFlush · the Knowledge Graph team](https://user-images.githubusercontent.com/90186547/142174441-0c30845c-cd49-40f6-9cf3-4e3c6f8ea5c1.png)

Finally, let me introduce our RoyalFlush Knowledge Graph Team. Our main serving businesses are intelligent dialogue system, system recommendation, search engines, investment strategies, and some businesses such as multi-mode integration, stock tips, investment consulting and education, which may expand later.

![RoyalFlush · the Knowledge Graph team](https://user-images.githubusercontent.com/90186547/142174763-4ec054b2-fb6a-4d76-aa05-0e827e1dba83.png)

At present, we maintain several graphs. One is the graph of the industry chain, which means that the data are public. The graph of the industry chain is constructed through semantic extraction.

NOTE: The right part in the above figure is an example of Graph used in the industry chain, which shows leading enterprises rankings, qualified enterprises rankings, and non-listed benchmarking enterprises rankings.

![RoyalFlush · the Knowledge Graph team](https://user-images.githubusercontent.com/90186547/142174956-347116c4-3798-46f3-99aa-3ee8c7a6d6ce.png)

We also maintain the graph of the supply chain,

NOTE: The right part in the above figure is an example of graph used in the supply chain, which displays acquired clients and co-supplier of an enterprise. 

![RoyalFlush · the Knowledge Graph team](https://user-images.githubusercontent.com/90186547/142175115-24130468-0091-4c14-8090-5b0de14f7523.png)

the graph of personal relationships,

NOTE: The right part in the above figure is a true case of graph used in personal relationships. 

![RoyalFlush · the Knowledge Graph team](https://user-images.githubusercontent.com/90186547/142175262-16ec088d-498c-4a35-999b-d0592d6398c4.png)

and the graph of enterprises, so as to meet the financial investment needs of RoyalFlush. The above is all that I'd like to share today. Thank you!

NOTE: The right part in the above figure is an example of graph used in enterprises, which shows infinite upward and downward penetration of equity relations, he beneficiary and actual controller of an enterprise etc.

Welcome to join our [Slack channel](https://join.slack.com/t/nebulagraph/shared_invite/zt-7ybejuqa-NCZBroh~PCh66d9kOQj45g) if you want to discuss with the rest of the NebulaGraph community!





This sharing is mainly from the perspective of (potential) users and discusses about what considerations the technical team will have when selecting a database.

How to Select a Graph Database: Best Practices at RoyalFlush

The performance report runs against GO, LOOKUP and MATCH queries in NebulaGraph v2.6.0. Meanwhile, it has compared the performance of v2.6.0 with that of v2.5.0.

NebulaGraph v2.6.0 Performance Report

This article shares the benchmarking tests conducted by the NLP team at Meituan. The test included data import and throughput/latency performance.

Benchmarking the Mainstream Open Source Distributed Graph Databases at Meituan: NebulaGraph vs Dgraph vs JanusGraph


![How to Reduce Docker Image Size](https://user-images.githubusercontent.com/56643819/74205747-8dcb7680-4cb3-11ea-97c0-7c8bf36dd0e1.png)

If there are top ten buzzwords in the technology industry in the year of 2019, **container** is sure to be one of them. With the popularity of Docker, more and more scenarios are using Docker in the front-end field. This article shows how do we use Docker in the visualization interface of **NebulaGraph**, a distributed open source graph database.

## Why Using Docker

Docker is widely used in daily front-end development. NebulaGraph Studio (A visualization tool for **NebulaGraph**) uses Docker based on the following considerations:

- **Unified operating environment**: There are several services behind our tools, such as existing services from different technology stacks, and pure front-end static resources.
- **Low user cost**: Currently, cloud services are under development. We want a smooth experience of the combined services, and Docker makes this possible with the ability of starting and applying all the services locally just in one step.
- **Quick deployment**: This front-end project is inspired by the [NebulaGraph Docker Image](https://github.com/vesoft-inc/nebula-docker-compose).

## Building Docker Image

We need to build an image first before hosing our services with Docker. Here we need a configuration file named [Dockerfile](https://docs.docker.com/engine/reference/builder/) that contains descriptions of the building steps. To be brief, we need to copy the project into the image and set the startup method:

```shell
# Select base image
FROM node:10
# Set work directory
WORKDIR /nebula-web-console
# Copy the current project to the /nebula-web-console directory of the image
ADD . /nebula-web-console
# Download front-end dependency in the image
RUN npm install
# Run the building
RUN npm run build
EXPOSE 7001
# Deployment commands executed when the image starts
CMD ["npm", "run", "docker-start"]
```

## Reducing Docker Image

The above configuration file will build a Docker image with a size of about 1.3GB, which looks a bit scary because downloading is too time-consuming even with a fast network. That is totally unacceptable.

After some research, we learned some tips that help reduce Docker image size.

### Using Smaller Base Image

Docker base image (for example, the above mentioned `node:10`) is the basic image on which you add layers and create a final image containing your applications. There are **multiple versions** of the Node.js image on [DockerHub](https://hub.docker.com/_/node), and each of them share a different **internal environment**. For example, the [alpine](https://yeasy.gitbooks.io/docker_practice/cases/os/alpine.html) version is a more simplified Linux system image that removes some tools like bash, curl etc. to decrease size.

Based on our needs, we change the base image to [alpine](https://yeasy.gitbooks.io/docker_practice/cases/os/alpine.html) and rebuild to reduce the docker image from 1.3GB to 500MB+. So if the docker image you are building is too large, you can **replace the basic image**.

### Multi-stage Build

[Multi-stage](https://docs.docker.com/develop/develop-images/multistage-build/) build in docker is a new feature introduced in docker 17.05. It is a method to reduce the image size, create a better organization of docker commands, and improve the performance while keeping the dockerfile easy to read and understand.

#### Docker Building Principle

In short, multi-stage build is dividing the dockerfile into multiple stages to pass the required artifact from one stage to another and eventually deliver the final artifact in the last stage. This way, our final image won't have any unnecessary content except our required artifact. Let's consider an example:

```shell
# Set up the image generated in the first step and name it builder
FROM node:10-alpine as builder
WORKDIR /nebula-web-console
# Copy the current project to the image
ADD . /nebula-web-console
# Start building
RUN npm install
RUN npm run build
....

# Start the second step build
FROM node:10-alpine
WORKDIR /nebula-web-console
# Copy the product of the first step image to the current image. Only one image layer is used here, which saves the number of image layers in the previous building step.

COPY --from=builder . /nebula-web-console
CMD ["npm", "run", "docker-start"]
```

### .dockerignore

Similar to the well known `.gitignore` that ignores unnecessary (such as document files, git files, node_modules, etc) files when using `COPY` or `ADD` command to copy or add files, we can use [.dockerignore](https://docs.docker.com/engine/reference/builder/#dockerignore-file) to specify files to be ignored.

### Merging Multiple Layers Into One

When building a Docker image with a Dockerfile, each operation adds a new layer based on the previous step image. We can use `&` to merge multiple operations to reduce layers. For example:

```shell
# The two operations represent two layers
RUN npm install
RUN npm run build
```

Merge the above command to one:

```shell
# It becomes a single layer with &
RUN npm install && npm run build
```

### Regular Front-End Optimization

- Compress ugly code and remove source code
Finish this step when building image so that the image size is further reduced.
- Only downloading code needed for production with node_modules
Finish this step when deploying, be sure to download only third party dependence code for production: `npm install --production`
- Place public source on CDN
If the image is expected to run in network environment, place large public files ( pictures and third party libraries, etc.) on the the CDN server so that some resources are separated and the image size is further reduced.
- ...

The above suggestions are only for your reference. You can migrate more regular front-end optimizations to the image given that image building itself is an environment for running code.

## Summary

The above is our experience on reducing the Docker image of the NebulaGraph Studio. Please leave us a comment if you have any questions. Welcome try [NebulaGraph Studio](https://github.com/vesoft-inc/nebula-web-docker).

## References

- [How to reduce Docker Image sizes using multi-stage builds](https://blog.logrocket.com/reduce-docker-image-sizes-using-multi-stage-builds/)
- [NebulaGraph Docker](https://github.com/vesoft-inc/nebula-docker-compose)
- [NebulaGraph Studio](https://github.com/vesoft-inc/nebula-web-docker)

## You might also like

1. [NebulaGraph Architecture — A Bird’s View](https://nebula-graph.io/posts/nebula-graph-architecture-overview/)
1. [An Introduction to NebulaGraph's Storage Engine](https://nebula-graph.io/posts/nebula-graph-storage-engine-overview/)
1. [An Introduction to NebulaGraph’s Query Engine](https://nebula-graph.io/posts/nebula-graph-query-engine-overview/)

This post shows you several ways to reduce Docker image size. These tips helped NebulaGraph devs reduce the image from 1.3G to 0.3G.