NebulaGraph v2.0 GA Overview

We have seen more and more users trying to adopt the graph technology since the general availability of NebulaGraph 1.0 in June 2020. The development of NebulaGraph 2.0 was brought to the table not long after the initial release as the Nebula team have collected many problems and feature requests from our developers’ community. Following hard work for nearly one year and several beta versions, NebulaGraph 2.0 is now generally available and can be used in production as a stable version.

Version Planning

The Nebula team has taken into consideration the following factors when planning what should be included in version 2.0:

• Suggestions and feature requests from current adopters
• Trends in the graph database and big data industry
• Feedback from the community

Finally, the team has made the decision that the development of NebulaGraph 2.0 should focus on the following aspects:

Better query performance. High throughput and low latency in handling large data volume is the natural architectural advantage of NebulaGraph. Most users turned to NebulaGraph because other products cannot meet the challenges brought by the growth of their data volume. Therefore, NebulaGraph 2.0 continues to maintain its advantage in large data volume query scenarios.

Stability is guaranteed. As more and more users use NebulaGraph in production, any slight system jitter will have a huge impact on their businesses. NebulaGraph always ensures users’ clusters in stable operations as their businesses evolve and data volume increases.

Enhanced system openness. NebulaGraph 2.0 added several important features and its underlying logic was optimized to make the system more open. NebulaGraph is now compatible with openCypher so that Neo4j users can use NebulaGraph with a flat learning curve. It can also integrate with more graph analysis algorithms to meet users' needs for offline analysis and graph computing.

Easier operation and maintenance. The complexity of operation and maintenance of distributed databases is incomparable to that of standalone ones. NebulaGraph 2.0 now provides users with more operation and maintenance tools to simplify deployment, upgrade, backup, and enable them to monitor the system and locate errors more accurately and faster.

Based on the above-mentioned principles, the team restructured and optimized NebulaGraph, trying to make it a reliable and user-friendly graph database. While some of the changes are visible to users such as better performance and easier operation, some are under the hood and are the building blocks of a more reliable and performant distributed system.

Performance Improvement

In NebulaGraph 2.0, the data type has been unified across all subsystems such as Storage, Graph, and Client, which can significantly reduce the data type conversion between subsystems or submodules, as well as serialization and deserialization.

In the meantime, the query engine of NebulaGraph 2.0 has been restructured to introduce multiple modules such as syntax analysis, statement optimization and resource scheduling. With syntax analysis, illegal statements can be ended before being passed for further processing, which reduces unnecessary resource consumption and provides users with more readable error information. In NebulaGraph 1.0, the query statements are processed by an independent executor, which significantly limits us from optimizing query execution in the system. Therefore, in NebulaGraph 2.0, we made reasonable abstraction to computation actions in query execution and designed multiple operators. In this way, a query statement can be split into an execution plan consisting of multiple operators and can be optimized via RBO (Rule Based Optimization). We will continue optimizing the operators and the execution plan in the future for better performance.

We have also redesigned the Scheduler to better allocate resources for execution plans.

Stability Enhancement

We have done a huge amount of work to guarantee the stability of NebulaGraph 2.0. We have built a new testing framework at the query engine level, including unit testing, integration testing, and performance test. The integration testing framework based on BDD helps us verify integration use cases fast and significantly reduces the difficulty of use case writing. The performance testing framework based on Jmeter is now functioning. The current version can help us verify performance issues quickly. A sound testing framework ensures the stability of the entire system. We have enhanced the chaos testing framework in NebulaGraph 2.0 to ensure the reliability and stability of the storage engine.

System Openness

In NebulaGraph 2.0, the previous Executor module has been split into multiple independent modules for better encapsulation and testing. The current query engine consists of Parser, Validator, Optimizer, Executor, and Scheduler. The new architecture not only makes NebulaGraph easier for maintenance, but also allows more openness to external feature requests.

In the past year, we have re-written most of the query engine code, making it compatible with both nGQL 1.0 and openCypher. At the same time, we are trying our best to reuse the execution mechanism and code for both nGQL and openCypher compatibility. In Validator, we have enhanced syntax verification so that errors that are captured during execution can be caught during compiling. The Optimizer has partially realized RBO, which makes adding rules to the existing framework easier. Besides, we have re-written the Executor based on asynchronous calling mechanism so that it can focus on computing and aggregating. Tasks like error handling, status saving and task scheduling are all transferred to the execution framework. The query engine is more module based and more open to extension, further reducing the threshold of community participation.

In NebulaGraph 2.0, we also added some built-in algorithms such as shortest path, full path and subgraph. We have also added a lot of scalar functions, aggregate functions, and tabulated functions. NebulaGraph now supports the “string” type vertex ID and NULL, which allow users to express more conveniently. Besides, we have implemented lots of modification and optimization at the storage engine level. The full-text indexing feature has been realized by connecting Elasticsearch with the learner role in RAFT. The multi-site high availability feature has been realized by abstracting Zone and Group in partition. Redesigning the index implementation has significantly improved data import performance with indexes. The above work has laid a solid foundation for continuous improvement of NebulaGraph for the goal of "More stable, Faster, Stronger" in the next year.

Easy Operation & Maintenance

In terms of operation & maintenance, NebulaGraph 2.0 supports the deployment of Helm Charts. In this way, users can deploy a NebulaGraph cluster with a few simple Helm commands. Meanwhile, since inner components are able to communicate with each other via domain addresses, configuration can be simplified and users only need to provide enough nodes according to their business needs. We have provided a tool for automatic upgrade to help existing users easily migrate from other versions to NebulaGraph 2.0. We are also intensively developing Backup & Restore tools, and the Dashboard feature.

For a detailed release note, please check: https://nebula-graph.io/posts/nebula-graph-v2.0-ga-release-note/

Here is the GitHub repo for NebulaGraph 2.0: https://github.com/vesoft-inc/nebula-graph. Give it a try! And here is the comprehensive documentation for your reference: https://docs.nebula-graph.io/2.0/.

If you encounter any problem, head to our discussions for help: discussions. Or you can join the Slack channel for discussion.