System-testing
Adding a Test Case for NebulaGraph
Before We Start
NebulaGraph is a product under continuous development, so the format of the returned result may be updated as required, which may cause parse errors to your applications.
For example, after the release of NebulaGraph v2.0.0, the updated SHOW JOBS syntax caused crash of NebulaGraph Studio because it cannot parse the statement correctly.
To avoid such errors, the NebulaGraph team have developed a new testing framework to help its users add new test cases without too much effort.
Preparing Environment
According to Install NebulaGraph by compiling the source code, prepare the environment as follows:
a. Get a development machine ready. Here is the minimum requirement:
- Python 3.7 or later.
python3 --version
b. Follow the steps described in the documentation. Create a branch from the master branch.
git checkout -b dev tck
For further details, refer to the steps of How to Contribute in NebulaGraph Database Manual.
c. Install the environment for executing the test cases.
$ cd nebula-graph/tests/
$ make init-all
If an error occurs informing that the module version is too old, you should upgrade it according to the prompt. For example, run the command pip3 install --upgrade keyrings.alt.
d. Execute the existing cases.
make test
Adding New Test Cases
Learn from Existing Cases
There have been a lot of test cases in the tests/tck/features/ directory of the nebula-graph repository on GitHub. Each feature has its own subdirectory.
Take MATCH as an example. Look at the match/MatchById.feature file.
Feature: Match By Id
Background:
Given a graph with space named "basketballplayer" # Choose a ready graph space named basketballplayer for unit testing.
Scenario: single node # Give the new scenario (test case) a name.
When executing query: # The line surrounded by three double quotes is the statement sent to NebulaGraph.
""" # This statement is to obtain a vertex with the "James Harden" ID.
MATCH (n) WHERE id(n) == 'James Harden' RETURN n
""" # The "Then" line gives the expected result.
Then the result should be, in any order, with relax comparison:
| n | # The expected result should be composed of one column named "n".
| ("James Harden") | # A record with "James Harden" surrounded by parentheses, representing a vertex
When executing query: # The next statement.
"""
MATCH (n) WHERE id(n) == 'not_exist_vertex' RETURN n
"""
Then the result should be, in any order, with relax comparison:
| n |
nebula-test is a BDD-based testing framework. All these feature files are written in Gherkin. For more information about BDD, refer to BDD-Based Integration Testing Framework for NebulaGraph: Part I.
Here is an introduction to the sections of a feature file:
- Feature: Gives the subject of the file.
- Background: Gives the steps that are common to all the scenarios in the same feature file. In this example, a ready graph space named "basketballplayer", with schema, indexes, and data, is selected.
- Scenario: Composed of a pair of
WhenandThensteps. A scenario is the smallest unit of concurrent execution. - When executing query: Sends a series of commands to NebulaGraph.
- Then the result should be: Obtains the result from NebulaGraph and compares the result with the expected result.
The returned result can be compared with the expected result with one of these methods:
- the result should be: The results must be exactly the same, which means the content of the rows, columns, and records are exactly the same.
- the result should contain: If the returned result contain the mentioned content in the expected result, it is considered valid.
- the result should be, in any order: The order of rows in the returned result is not used to determine the legality of the result.
- the result should be, in any order, with relax comparison: The returned result should comply with the given format but the order of the rows does not matter. Besides, the result must contain the specified content. For example, each record must has one or more "Tiago Splitter" strings.
- the execution should be successful: The execution is successful.
- an ExecutionError should be raised at runtime: not existed! The execution failed. The ExecutionError exception is thrown, and the error code is
not existed!.
Writing a Test Case
If you want to do a check of the format of the returned result of SHOW JOBS, you can add a feature file named JobCommands.feature under the directory: nebula-graph/tests/tck/features/job/.
@mintest # A tag. This is for debugging.
Feature: Job compact, flush, rebuild_index
Background: # Choose a graph space with a dataset.
Given a graph with space named "basketballplayer"
Scenario: submit job
When executing query:
"""
SUBMIT JOB COMPACT;
"""
Then the result should contain:
| New Job Id |
| /\d+/ | # Write a regular expression for positive integers
When executing query:
"""
SUBMIT JOB FLUSH;
"""
When executing query:
"""
SUBMIT JOB STATS;
SHOW JOBS;
"""
Then the result should contain: # Do a check of the result format
| Job Id | Command | Status | Start Time | Stop Time |
| /\d+/ | "STATS" | /\w+/ | /\d+-\d+-\d+T\d+:\d+:\d+/ | /\d+-\d+-\d+T\d+:\d+:\d+/ |
Debugging a Test Case
nebula-graph/tests> make fmt
Do a check of the format and get the file format valid.
nebula-graph/tests> pytest -m "mintest"
Run the test case. In general, the case may need some changes. You can use # to comment out a line or use @skip to skip a scenario.
nebula-graph/tests> pytest -n4
If the memory capacity is not big enough, use -n4 to control the concurrency.
For more information, refer to the test manual.
Submitting Test Case via Git
If the test case passes, remove the @mintest tag and submit a pull request to the NebulaGraph repository on GitHub. For more information, refer to How to Contribute.
If you are interested in learning more about NebulaGraph, welcome to join the NebulaGraph Slack channel!
