In the previous article of this series, "[Azure Marketplace: How to list your cloud offering successfully](https://nebula-graph.io/posts/how-to-list-cloud-offer-azure-marketplace)", we generally introduced how to publish your SaaS offerings on the Azure Marketplace. In this article, I will dive deeper into Azure Marketplace’s billing system to help you understand how you can let Azure know end users’ usage and bill them properly.

# The billing model
As mentioned in the previous article, you need to configure your offering information in the Microsoft Partner Center. This applies to the billing information as well. Before you get started, here are some concepts related to billing that you must understand. 

* The offer: The SaaS offer that you are about to launch. The following picture is an example of all the offers that we have created.
   ![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/a-deep-dive-into-azure-marketplace-billing-system/figure1.png)
* Plans of the offer: A plan defines an offer’s scope and limits, and the associated pricing when applicable. One offer may have multiple plans, which will be shown to end-users upon their purchases. 
  ![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/a-deep-dive-into-azure-marketplace-billing-system/figure2.png)
* Pricing models: The pricing models of each plan can vary from business model to business model. You can choose recurring subscriptions or usage-based pricing. Please note that if you choose a usage-based pricing model, you can only configure up to 30 billing metrics, and you cannot modify them after the offer has been launched.
  ![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/a-deep-dive-into-azure-marketplace-billing-system/figure3.png)

Simply put, once you have configured the offer, offer plans, and pricing models, and published your offer, Azure will bill users based on the usage you have uploaded to it.

Basic billing settings should be enough for simple SaaS offers. For example, for a news service with a fixed monthly subscription fee of $20, the developer only needs to create a monthly plan in the Azure portal that charges every subscriber $20 per month. 

There are also slightly more complex situations. For instance, an instant-messaging service, where there is a fixed monthly setup fee of $10 and each new message exceeding the quota is charged $0.001. Under this circumstance, the setting is now as complicated, you simply need to set up the basic monthly fee and add a few more usage metrics.

But for vendors that want to offer infrastructure software in the Cloud the situation of a lot more complicated. Database as a service (DBaaS) is essentially part of SaaS, but the pricing model of DBaaS is more complicated, especially for distributed database systems like NebulaGraph.

Here are some of the challenges we had faced.
* Vendors often need to provide different types of instance options to meet the needs of users in a combined manner according to their different levels of data volume.  Different instance options also mean different usage metrics for us. For example, NebulaGraph adopts an architecture that is called “[the separation of storage and compute,](https://blog.starburst.io/technical-blog/separation-of-storage-and-compute)” and each storage or computing layer should be running on their according to the types of instance.
* As a software vendor running in the cloud, our cloud service will use a lot of cloud vendors’ resources, and the consumption of these resources will eventually need to be paid by end-users. For example, the size of the disks used will be different according to the amount of data, and then the amount of data stored will be different according to the frequency of access, and eventually, the use of network traffic will be different, similar to the usage metrics billed based on on-demand use. This leads to a problem: it is difficult for ISVs to charge these on-demand resources to a fixed amount per month for users to pay. And it is not a good idea to add more billing metrics based on that, because you can only add 30 billing metrics.

These are the real issues we encountered when launching NebulaGraph Cloud on the Azure Marketplace. As a partner, we also gave Microsoft feedback on some of the issues we encountered, but this will be a long-term process, so we have to look for workarounds. For example, regarding the limit of 30 billing metrics, what you can do is merge and abstract existing billing metrics. For example, different instance configurations, whether it is 4 cores 16GB, or 8 cores 32GB, if you see it from a perspective based on instances, you will need as many metrics as the number of your instance configurations. But if you abstract it, the metrics are simply the number of cores, and the size of RAM, and the storage, significantly reducing the billing metrics.

# Reporting user usage to Azure

After defining your product plans and the billing model of each plan, you need to find a way to tell Azure how much a billing user has used in each billing period. This step is done by Azure’s [Metering API](https://docs.microsoft.com/en-us/azure/marketplace/partner-center-portal/saas-metered-billing), using which you can send usage data of each user in a fixed timeframe and Azure will deal with the billing. This means that ISVs don’t need to deal with the calculation of the charge, and what they only need to do is to report users’ usage to Azure properly.

In the [previous article](https://nebula-graph.io/posts/how-to-list-cloud-offer-azure-marketplace) of this series, I mentioned that you have to update users’ usage data to Azure’s billing system within 24 hours and that one billing metric can only be updated once within one hour. You may encounter a duplicate error if you try to update twice within an hour. Other than that, the more important thing that ISVs have to pay attention to is the accuracy of users’ usage. Here are some of our tips.

*   Use Azure’s API directly to get the usage of its resource data, and do not store the usage data yourself. This sounds simple, but we need to know explicitly the current resources used by each user, and whether there is a ready-made cloud vendor interface for us to call to get to. Once we can't get it, we need to be prompted to modify the current way of allocating the use of resources based on users, which often requires modifying the service architecture.&nbsp;
*   Although it is technically feasible to store the usage data by yourself, the cost of just keeping records of the data used by users will take up a lot of overhead. Plus, compared to Azure, your own usage data will probably become inaccurate, resulting in inconsistent data with vendors on both sides.
*   Pay attention to those usage data reported on demand, such as traffic, blob object storage. Even in an hourly usage cycle, the actual usage interval of these data will have many cases. For example, if a user uploaded a file in a certain hour, and then deleted it immediately after finishing using it in a few minutes, you need to accurately record the usage of the user to avoid that they may be overcharged.

# Sales and discounts

For newly-released SaaS products, vendors usually want to reach the first users as early as possible, so they often adopt sales or discounts as a way of promotion. For example, selling services at a discount of 50%.

However, there is no way that you can configure discounts in the Microsoft Partner Center. The only workaround you can find is to add a new plan that has a price of 50% of the original plan. But this is not sustainable, because if you only want to offer the discount to users for the first year of their service, there is no way that you can change the pricing of their purchased plans afterwards.

Our solution to Azure’s inflexible billing system is adding a standalone billing metric called “service credit” on our end. This billing metric is different from the billing metrics we defined. It solely serves as a buffer metric that enables us to control users’ usage more flexibly. For example, if we want to give new users a 30% discount for one year, we will give them service credits that are equivalent to 30% of their billing amount so that they are essentially paying 70% of the amount of their bills.

This feature is achieved by taking advantage of the fact that we can report resource usage data to Azure. When our service credits can offset part of the resource consumption, we choose not to report the amount of resources offset, so that the user will not be charged for the amount.&nbsp;

# Conclusion

In summary, Azure Marketplace provides a backend configuration system that can be adapted for all types of SaaS products. But a general-purpose platform like Azure cannot take into account the more detailed and specific needs of each SaaS product beyond the most basic general requirements.&nbsp;

Ultimately the decision you need to make is either to simplify your SaaS product to better fit the backend configuration and billing model it currently provides, or to add a workaround logic layer between you and the Azure platform based on your needs. In my personal practice, I prefer the former, because the platform itself is being improved and enhanced, and adding workaround code too early is not very easy to maintain once the platform is updated.&nbsp;

But the reality is always less than ideal. Sometimes you have to use workarounds to meet the needs of your product. I would advise you to think ahead about how to maintain your workarounds when building them in case the platform's rules change.🤝

NebulaGraph Cloud is currently available in the Azure Marketplace and it is in a beta period that offers a generous 70% off for beta users. Sign up here to [get the offer](https://nebula-graph.io/nebula-graph-cloud/) if you are interested.
 

# About the author

Jerry Liang is a technical lead at NebulaGraph’s Cloud division. He and his team are behind a series of NebulaGraph’s visualization tools such as [Nebula Dashboard](https://nebula-graph.io/products/dashboard/), [Nebula Explorer](https://nebula-graph.io/products/explorer/), and [Nebula Studio](https://nebula-graph.io/products/studio/).

*   [Part I: How to list your cloud offer successfully?](https://nebula-graph.io/posts/how-to-list-cloud-offer-azure-marketplace)
*   [Part II: A deep dive into Azure's billing system](https://nebula-graph.io/posts/a-deep-dive-into-azure-marketplace-billing-system)
*   [Part III: My takes on three cloud service models for ISVs](http://nebula-graph.io/posts/azure-marketplace-three-cloud-service-models-for-isvs)

a-deep-dive-into-azure-marketplace-billing-system

Azure Marketplace series: A deep dive into the billing system


![Troubleshooting a Crash Triggered by Clang Compiler Optimization](https://user-images.githubusercontent.com/57335825/102327048-05081180-3f3a-11eb-8d39-752e3563c8d8.png)

If someone told you that the following C++ function would cause the program to crash, what would you think it is that caused the problem?

```cpp
std::string b2s(bool b) {
    return b ? "true" : "false";
}
```

If you were given more descriptions, for example:

- The crash reappears with a certain probability.
- The error message shows Program terminated with signal `SIGSEGV`.
- The backtrace information is usually incomplete or even missing.
- When the optimization level is higher than -O2, the crash is more likely to happen.
- The crash only happens while using Clang, but not GCC.

Well, some of the veterans may already have a clue. Here's the minimal program that can reproduce the crash:

```cpp
// file crash.cpp
#include <iostream>
#include <string>

std::string __attribute__((noinline)) b2s(bool b) {
    return b ? "true" : "false";
}

union {
    unsigned char c;
    bool b;
} volatile u;

int main() {
    u.c = 0x80;
    std::cout << b2s(u.b) << std::endl;
    return 0;
}
```

```cpp
$ clang++ -O2 crash.cpp
$ ./a.out
truefalse,d$x4DdzRx

Segmentation fault (core dumped)

$ gdb ./a.out core.3699
Core was generated by `./a.out'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000012cfffff0d4 in ?? ()
(gdb) bt
#0  0x0000012cfffff0d4 in ?? ()
#1  0x00000064fffff0f4 in ?? ()
#2  0x00000078fffff124 in ?? ()
#3  0x000000b4fffff1e4 in ?? ()
#4  0x000000fcfffff234 in ?? ()
#5  0x00000144fffff2f4 in ?? ()
#6  0x0000018cfffff364 in ?? ()
#7  0x0000000000000014 in ?? ()
#8  0x0110780100527a01 in ?? ()
#9  0x0000019008070c1b in ?? ()
#10 0x0000001c00000010 in ?? ()
#11 0x0000002ffffff088 in ?? ()
#12 0xe2ab001010074400 in ?? ()
#13 0x0000000000000000 in ?? ()
```

Though incomplete, the backtrace information still indicates that the program did not crash in the first place. In such a case, to quickly locate the first scene, let's try AddressSanitizer (ASan).

```cpp
$ clang++ -g -O2 -fno-omit-frame-pointer -fsanitize=address crash.cpp
$ ./a.out
=================================================================
==3699==ERROR: AddressSanitizer: global-buffer-overflow on address 0x000000552805 at pc 0x0000004ff83a bp 0x7ffd7610d240 sp 0x7ffd7610c9f0
READ of size 133 at 0x000000552805 thread T0
    #0 0x4ff839 in __asan_memcpy (a.out+0x4ff839)
    #1 0x5390a7 in b2s[abi:cxx11](bool) crash.cpp:6
    #2 0x5391be in main crash.cpp:16:18
    #3 0x7faed604df42 in __libc_start_main (/usr/lib64/libc.so.6+0x23f42)
    #4 0x41c43d in _start (a.out+0x41c43d)

0x000000552805 is located 59 bytes to the left of global variable '<string literal>' defined in 'crash.cpp:6:25' (0x552840) of size 6
  '<string literal>' is ascii string 'false'
0x000000552805 is located 0 bytes to the right of global variable '<string literal>' defined in 'crash.cpp:6:16' (0x552800) of size 5
  '<string literal>' is ascii string 'true'
SUMMARY: AddressSanitizer: global-buffer-overflow (/home/dutor.hou/Wdir/nebula-graph/build/bug/a.out+0x4ff839) in __asan_memcpy
Shadow bytes around the buggy address:
…
...
```

From the information provided by ASan, we can see that global buffer overflow happened when the `b2s(bool)` function was reading the string constant `true`. Alright, let's take a "God View" at the abnormal function and the program. It seems like we can get such a conclusion: Because the `bool` type parameter `b` of the `b2s` function was not initialized, the value stored in b is neither 0 nor 1[1], and that caused the problem. But here's another question, why would such a value cause buffer overflow? If that interests you, change the type of `b` from `bool` to `char` or `int` and the problem will be fixed.

To answer the new question, we have to check what instructions does Clang++ generate from `b2s`. Before we do that, we should know:

- In the sample program, the return value of `b2s` is a temporary `std::string` object, which is stored on the stack.
- After C++ 11, GCC applies SBO (Small Buffer Optimization) by default on the implementation of `std::string`. The definition is something like this: `std::string{ char *ptr; size_t size; union{ char buf[16]; size_t capacity}; }`. For strings shorter than 16 bytes, no additional memory is required.

OK. Now let's take a look at the disassembling of `b2s`. I added some key comments.

```cpp
(gdb) disas b2s
Dump of assembler code for function b2s[abi:cxx11](bool):
   0x00401200 <+0>:     push   %r14
   0x00401202 <+2>:     push   %rbx
   0x00401203 <+3>:     push   %rax
   0x00401204 <+4>:     mov    %rdi,%r14         # Save the starting address of the return value (a string) in R14.
   0x00401207 <+7>:     mov    $0x402010,%ecx    # Save the starting address of "true" in ecx.
   0x0040120c <+12>:    mov    $0x402015,%eax    # Save the starting address of "false" in eax.
   0x00401211 <+17>:    test   %esi,%esi         # Test whether parameter b has a non-zero value.
   0x00401213 <+19>:    cmovne %rcx,%rax         # If b is not zero, save the address of "true" in rax.
   0x00401217 <+23>:    lea    0x10(%rdi),%rdi   # Save the starting address of buf in the string in rdi.
                                                 #  It is also the first parameter of memcpy.
   0x0040121b <+27>:    mov    %rdi,(%r14)       # Save rdi in the string's PTR  field, i.e., SBO.
   0x0040121e <+30>:    mov    %esi,%ebx         # Save the value of b in ebx.
   0x00401220 <+32>:    xor    $0x5,%rbx         # Calculate the XOR value of 0x5 and save it in rbx (i.e., ebx).
                                                 # Note that if rbx is 0 or 1, then the value saved to rbx is 4 or 5,
                                                 # which is the length of "true" or "false". 
   0x00401224 <+36>:    mov    %rax,%rsi         # Save the starting address of the string in rsi, which is the second parameter of memcpY.
   0x00401227 <+39>:    mov    %rbx,%rdx         # Save the length of the string in rdx, which is the third parameter of memcpy.
   0x0040122a <+42>:    callq  <memcpy@plt>      # Call memcpy.
   0x0040122f <+47>:    mov    %rbx,0x8(%r14)    # Save the length of the string in string::size.
   0x00401233 <+51>:    movb   $0x0,0x10(%r14,%rbx,1)  # Add '\0' to the end of the string.
   0x00401239 <+57>:    mov    %r14,%rax         # Save the String address in rax, that is, the return value.
   0x0040123c <+60>:    add    $0x8,%rsp
   0x00401240 <+64>:    pop    %rbx
   0x00401241 <+65>:    pop    %r14
   0x00401243 <+67>:    retq
End of assembler dump.
```

At this point, the situation becomes crystal clear:

1. Clang++ assumes that the value of a Boolean is either 0 or 1.
2. During the compiling, the compiler has the length of `true` and `false`.
3. The compiler uses the XOR operations (`0x5 ^ false == 5`, `0x5 ^ true == 4`) to calculate the length of the string to be copied.
4. When the compiler finds that the `bool` type value does not conform to the preceding assumption, the length calculation goes wrong.
5. Because the target address of `memcpy` is on the stack (for this example only), the stack buffer may overflow, causing the program to run off the rail and the backtrace information to be missing.

NOTE:
[1]. The C++ standard requires a `bool` type value to represent at least two states: `true` and `false`, but does not specify the `sizeof(bool)`. But on almost all compilers, a `bool` type value takes up an addressing unit, i.e., a byte. Therefore, from the storage perspective, the value range is from 0x00 to 0xFF, which represents 256 states.

# You Might Also Like

- [An Introduction to Google's Memory Error Detect Tool AddressSanitizer](https://nebula-graph.io/posts/introduction-to-google-memory-detect-tool-addresssanitizer/)
- [Compiling Trouble Shooting: Segmentation Fault and GCC Illegal Instruction](https://nebula-graph.io/posts/segmentation-fault-gcc-illegal-instruction-trouble-shooting/)

This is an interesting debugging process of troubleshooting a crash appears in Clang but not in GCC.

Troubleshooting a Crash Triggered by Clang Compiler Optimization


![Debug the NebulaGraph processes inside the Docker container](https://user-images.githubusercontent.com/57335825/97400812-f9d72480-18ac-11eb-8761-e0781ba97401.png)

# Requirements

In the development or testing process, we often use the [vesoft-inc/nebula-docker-compose](https://github.com/vesoft-inc/nebula-docker-compose) repository to deploy NebulaGraph. But to compress the docker image of each NebulaGraph service as much as possible, all the tools commonly used for development are not installed in the Docker images, not even the editor Vim.

This makes it difficult to locate problems inside the containers because it was cumbersome to install essential toolkit every time to get things done. In fact, there is another way to debug a process inside a container, without breaking the content structure of the container or installing any toolkit in it.

Actually, this technique, the Sidecar mode, is commonly used in the K8S environment. The principle is quite simple: Start a container that shares the same namespaces of the PID/network with the container we are debugging. In this way, we can view the processes and network namespaces in the original container through the container we use to debug, which has everything we need installed in it.

# Demonstration

Here's how to do it.

Start by deploying a NebulaGraph cluster locally with Docker-Compose, as we mentioned above. For a detailed tutorial, see the [README](https://github.com/vesoft-inc/nebula-docker-compose) file in the repository. After the deployment, we start all the services and check their status as follows.

```shell
$ docker-compose up -d
Creating network "nebula-docker-compose_nebula-net" with the default driver
Creating nebula-docker-compose_metad1_1 ... done
Creating nebula-docker-compose_metad2_1 ... done
Creating nebula-docker-compose_metad0_1 ... done
Creating nebula-docker-compose_storaged2_1 ... done
Creating nebula-docker-compose_storaged1_1 ... done
Creating nebula-docker-compose_storaged0_1 ... done
Creating nebula-docker-compose_graphd_1    ... done
$ docker-compose ps
              Name                             Command                       State                                            Ports
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
nebula-docker-compose_graphd_1      ./bin/nebula-graphd --flag ...   Up (health: starting)   0.0.0.0:32907->13000/tcp,0.0.0.0:32906->13002/tcp, 0.0.0.0:3699->3699/tcp
nebula-docker-compose_metad0_1      ./bin/nebula-metad --flagf ...   Up (health: starting)   0.0.0.0:32898->11000/tcp,0.0.0.0:32896->11002/tcp, 45500/tcp,45501/tcp
nebula-docker-compose_metad1_1      ./bin/nebula-metad --flagf ...   Up (health: starting)   0.0.0.0:32895->11000/tcp,0.0.0.0:32894->11002/tcp, 45500/tcp,45501/tcp
nebula-docker-compose_metad2_1      ./bin/nebula-metad --flagf ...   Up (health: starting)   0.0.0.0:32899->11000/tcp,0.0.0.0:32897->11002/tcp, 45500/tcp,45501/tcp
nebula-docker-compose_storaged0_1   ./bin/nebula-storaged --fl ...   Up (health: starting)   0.0.0.0:32901->12000/tcp,0.0.0.0:32900->12002/tcp, 44500/tcp,44501/tcp
nebula-docker-compose_storaged1_1   ./bin/nebula-storaged --fl ...   Up (health: starting)   0.0.0.0:32903->12000/tcp,0.0.0.0:32902->12002/tcp, 44500/tcp,44501/tcp
nebula-docker-compose_storaged2_1   ./bin/nebula-storaged --fl ...   Up (health: starting)   0.0.0.0:32905->12000/tcp,0.0.0.0:32904->12002/tcp, 44500/tcp,44501/tcp
```

Next, we demonstrate scenario by scenario, from the process namespaces to the network namespaces. First of all, we need to have a handy image for debugging. There is no need to build one by ourselves since this is only for demonstration. Let's find a well-packed image from Docker Hub. If later we find this image not good enough, we can maintain a nebula-debug image and install all the debugging tools we want then. Here, we use a community solution from [nicolaka/netshoot](https://hub.docker.com/r/nicolaka/netshoot). Let's pull the image to the local host.

```shell
$ docker pull nicolaka/netshoot
$ docker images
REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
vesoft/nebula-graphd     nightly             c67fe54665b7        36hours ago        282MB
vesoft/nebula-storaged   nightly            5c77dbcdc507        36hours ago        288MB
vesoft/nebula-console    nightly             f3256c99eda1        36hours ago        249MB
vesoft/nebula-metad      nightly             5a78d3e3008f        36hours ago        288MB
nicolaka/netshoot        latest              6d7e8891c980        2months ago        352MB
Let's see what is going to happen if we run the image.
$ docker run --rm -ti nicolaka/netshoot bash
bash-5.0# ps
PID   USER    TIME  COMMAND
    1 root      0:00 bash
    8 root      0:00 ps
bash-5.0#
```

As shown above, this container does not have any NebulaGraph process. Let's add a few parameters to it and see what happens.

```shell
$ docker run --rm -ti --pid container:nebula-docker-compose_metad0_1 --cap-add sys_admin nicolaka/netshoot bash
bash-5.0# ps
PID   USER    TIME  COMMAND
    1 root      0:03 ./bin/nebula-metad --flagfile=./etc/nebula-metad.conf --daemonize=false --meta_server_addrs=172.28.1.1:45500,172.28.1.2:45500,172.28.1.3:45500--local_ip=172.28.1.1 --ws_ip=172.28.1.1 --port=45500 --data_path=/data/meta--log_dir=/logs--v=15 --minloglevel=0
  452 root      0:00 bash
  459 root      0:00 ps
bash-5.0# ls -al /proc/1/net/
total 0
dr-xr-xr-x    6 root     root             0 Sep18 07:17 .
dr-xr-xr-x    9 root     root             0 Sep18 06:55 ..
-r--r--r--    1 root     root             0 Sep18 07:18 anycast6
-r--r--r--    1 root     root             0 Sep18 07:18 arp
dr-xr-xr-x    2 root     root             0 Sep18 07:18 bonding
-r--r--r--    1 root     root             0 Sep18 07:18 dev
...
-r--r--r--    1 root     root             0 Sep18 07:18 sockstat
-r--r--r--    1 root     root             0 Sep18 07:18 sockstat6
-r--r--r--    1 root     root             0 Sep18 07:18 softnet_stat
dr-xr-xr-x    2 root     root             0 Sep18 07:18 stat
-r--r--r--    1 root     root             0 Sep18 07:18 tcp
-r--r--r--    1 root     root             0 Sep18 07:18 tcp6
-r--r--r--    1 root     root             0 Sep18 07:18 udp
-r--r--r--    1 root     root             0 Sep18 07:18 udp6
-r--r--r--    1 root     root             0 Sep18 07:18 udplite
-r--r--r--    1 root     root             0 Sep18 07:18 udplite6
-r--r--r--    1 root     root             0 Sep18 07:18 unix
-r--r--r--    1 root     root             0 Sep18 07:18 xfrm_stat
```

This time it's a bit different. We can see the metad0 process, which has the PID 1. Now we can easily do our work since we can see the process, for example, attaching it in the GDB. I don't have an image with the NebulaGraph binary file handy, so I'll leave the exploration to you.

We see that the PID namespace is shared already by setting `--pid container:<container_name|id>`. Next, considering we might need to capture a package sometimes, let's check if we can also see the network status. Run the following command.

```shell
bash-5.0# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      PID/Program name
```

There's nothing. It's a little bit different from what we expected, because we already have the metad0 process, and there should not be nothing here, not even one connection. To see the network namespaces within the original container, we need to set a few more options and run the following command to restart the debug container.

```shell
$ docker run --rm -ti --pid container:nebula-docker-compose_metad0_1 --network container:nebula-docker-compose_metad0_1 --cap-add sys_admin nicolaka/netshoot bash
bash-5.0# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 172.28.1.1:11000       0.0.0.0:*               LISTEN      -
tcp        0      0 172.28.1.1:11002       0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:45500           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:45501           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.11:33249       0.0.0.0:*               LISTEN      -
udp        0      0 127.0.0.11:51929       0.0.0.0:*                           -
```

This time the output is not the same with the preceding one. With the `--network container:nebula-docker-compose_metad0_1` option, we can check the connections in the metad0 container, capture packages, and debug.

# Summary

The key to being able to debug a container environment without installing extra tools in it is to run another container and have it share the PID/network namespaces with the original container. Some people in the community have even developed tools based on this method and made them easier to use. For more information, see [Docker-debug](https://github.com/zeromake/docker-debug).

# You might also like

1. [Compiling Trouble Shooting: Segmentation Fault and GCC Illegal Instruction](https://nebula-graph.io/posts/segmentation-fault-gcc-illegal-instruction-trouble-shooting/)
1. [Dev Log | How to Release jar Package to the Maven Central Repository](https://nebula-graph.io/posts/maven/)
1. [How to Reduce Docker Image Size](https://nebula-graph.io/posts/how-to-reduce-docker-image-size/)


This post shares with you a way to debug the Nubula Graph processes within the Docker container in the Sidecar mode just like the processes are running locally.

Debug the NebulaGraph Processes Inside the Docker Container


![Analyzing Relationships in Game of Thrones With NetworkX, Gephi, and NebulaGraph](https://user-images.githubusercontent.com/57335825/90602643-f2114a80-e1ae-11ea-9fea-0516b8bf71e1.png)

In the last post[1], we showed the character relationship for the _Game of Thrones_ by using NetworkX and Gephi. In this post, we will show you how to access data in NebulaGraph by using NetworkX.

# NetworkX

NetworkX[2] is a modeling tool for the graph theory and complex networks written by Python. With its rich, easy-to-use built-in graphs and analysis algorithms, it's easy to perform complex network analysis and simulation modeling.

In NetworkX, a graph (network) is a collection of nodes together with a collection of edges. Attributes are often associated with nodes and/or edges and are optional. Nodes represent data. Edges are uniquely determined by two nodes, representing the relationship between the two nodes. Nodes and edges can also have many attributes so that more information can be stored.

The following four basic graph types are provided in NetworkX:

- Graph: undirected graph.
- DiGraph: directed graph.
- MultiGraph: A flexible graph class that allows multiple undirected edges between pairs of nodes.
- MultiDiGraph: A directed version of a MultiGraph.

Create an undirected graph in NetworkX:

```python
import networkx as nx
G = nx.Graph()
```

Add nodes:

```python
G.add_node(1)
G.add_nodes_from([2,3,4])
G.add_node(2,name='Tom',age=23)
```

Add edges:

```python
G.add_edge(2,3)
G.add_edges_from([(1,2),(1,3)])
g.add_edge(1, 2, start_year=1996, end_year=2019)
```

In the last post (part one of this series), we have displayed the community detection algorithm Girvan-Newman provided by NetworkX.

## Graph Database NebulaGraph

NetworkX usually uses local files as the data source, which is totally okay for static network researches. But when the graph network changes a lot, for example, some central nodes are deleted or important network topology changes are introduced, it is a little troublesome to generate, load, and analyze the new static files. At this time, we'd better persist the entire change process in a database, and load subgraphs or whole graphs directly from the database for real-time analysis. This post uses NebulaGraph [3] as the graph database for storing the graph data.

NebulaGraph provides two methods to get graph structures:

1. Query to return a subgraph.
1. Scan the underlying storage to get the whole graph.

The first method is suitable for obtaining certain nodes and edges that meet the requirements through fine filtering and pruning in a large-scale graph network. Whereas the second method is more suitable for the whole graph analysis. It is usually used in the early stage of the project when conducting heuristic exploration. When you further understand the graph, you can use the first method for fine pruning analysis.

By reading the code of the python client for NebulaGraph, we know that the `nebula-python/nebula/ngStorage/StorageClient.py` and the `nebula-python/nebula/ngMeta/MetaClient.py` file are the APIs that interact with the underlying storage. These files contain a series of interfaces to scan nodes, scan edges, read columns, and so on. The following two interfaces can be used to scan all nodes and edges:

```python
def scanVertex(self, space, returnCols, allCols, limit, startTime, endTime)
def scanEdge(self, space, returnCols, allCols, limit, startTime, endTime)
```

1. Initialize a client and a ScanEdgeProcessor. The ScanEdgeProcessor decodes the edge data read from NebulaGraph:

```python
metaClient = MetaClient([('192.168.8.16', 45500)])
metaClient.connect()
storageClient = StorageClient(metaClient)
scanEdgeProcessor = ScanEdgeProcessor(metaClient)
```

2. Initialize the parameters for the scanEdge interface:

```python
spaceName = 'nba' # The name of the graph space to be read.
returnCols = {} # The edges (or nodes) and their property columns to be returned.
returnCols['serve'] = ['start_year', 'end_year']
returnCols['follow'] = ['degree']
allCols = False # Return all the property columns or not. When set to False, only the specified property columns in the returnCols are returned. When set to True, all property columns are returned.
limit = 100 # Maximum data entry number returned.
startTime = 0
endTime = sys.maxsize
```

3. Call the scanPartEdge interface to return the iterator for the canEdgeResponse object:

```python
scanEdgeResponseIterator = storageClient.scanEdge(spaceName, returnCols, allCols, limit, startTime, endTime)
```


4. Keep reading the data in the ScanEdgeResponse object which the iterator points to till all the data are read:

```python
while scanEdgeResponseIterator.hasNext():
    scanEdgeResponse = scanEdgeResponseIterator.next()
    if scanEdgeResponse is None:
        print("Error occurs while scaning edge")
        break
    processEdge(space, scanEdgeResponse)
```

In the preceding snippet, the `process_edge` is a custom function for processing the scanned edge data. This function first uses the `scan_edge_processor` to decode the data in the `scan_edge_response`. The decoded data can either be directly printed out or processed for other purposes such as importing these data into the calculation framework NetWorkX.

1. Process data. Here we import all the scanned data into Graph G in the NetworkX:

```python
def processEdge(space, scanEdgeResponse):
    result = scanEdgeProcessor.process(space, scanEdgeResponse)
    # Get the corresponding rows by edge_name
    for edgeName, edgeRows in result.rows.items():
        for row in edgeRows:
            srcId = row.defaultProperties[0].getValue()
            dstId = row.defaultProperties[2].getValue()
            print('%d -> %d' % (srcId, dstId))
            props = {}
            for prop in row.properties:
                propName = prop.getName()
                propValue = prop.getValue()
                props[propName] = propValue
            G.add_edges_from([(srcId, dstId, props)]) # Add edges to Graph G in the NetworkX
```

Nodes scanning is similar to the preceding process.

In addition, for some of the distributed graph calculation frameworks [4], NebulaGraph provides batch reading and storing based on the partitions. We will show you in the later post.

## Applying Graph Analysis in NetworkX

When importing all the nodes and edges into the NetworkX according to the preceding process, we can do some basic graph analysis and calculations:

1. Draw a plot:

```python
nx.draw(G, with_labels=True, font_weight='bold')
import matplotlib.pyplot as plt
plt.show()
plt.savefig('./test.png')
```

The plot you just drew looks like following:

![Graph of the Game of Thrones Plot](https://user-images.githubusercontent.com/57335825/92692639-4cf31900-f2f9-11ea-8c17-5a4cb9f51744.png)

2. Print out all the nodes and edges in the plot:

```python
print('nodes: ', list(G.nodes))
print('edges: ', list(G.edges))
```

Results printed:

```python
nodes:  [109, 119, 129, 139, 149, 209, 219, 229, 108, 118, 128, 138, 148, 208, 218, 228, 107, 117, 127, 137, 147, 207, 217, 227, 106, 116, 126, 136, 146, 206, 216, 226, 101, 111, 121, 131, 141, 201, 211, 221, 100, 110, 120, 130, 140, 150, 200, 210, 220, 102, 112, 122, 132, 142, 202, 212, 222, 103, 113, 123, 133, 143, 203, 213, 223, 104, 114, 124, 134, 144, 204, 214, 224, 105, 115, 125, 135, 145, 205, 215, 225]
edges:  [(109, 100), (109, 125), (109, 204), (109, 219), (109, 222), (119, 200), (119, 205), (119, 113), (129, 116), (129, 121), (129, 128), (129, 216), (129, 221), (129, 229), (129, 137), (139, 138), (139, 212), (139, 218), (149, 130), (149, 219), (209, 123), (219, 130), (219, 112), (219, 104), (229, 147), (229, 116), (229, 141), (229, 144), (108, 100), (108, 101), (108, 204), (108, 206), (108, 214), (108, 215), (108, 222), (118, 120), (118, 131), (118, 205), (118, 113), (128, 116), (128, 121), (128, 201), (128, 202), (128, 205), (128, 223), (138, 115), (138, 204), (138, 210), (138, 212), (138, 221), (138, 225), (148, 127), (148, 136), (148, 137), (148, 214), (148, 223), (148, 227), (148, 213), (208, 127), (208, 103), (208, 104), (208, 124), (218, 127), (218, 110), (218, 103), (218, 104), (218, 114), (218, 105), (228, 146), (228, 145), (107, 100), (107, 204), (107, 217), (107, 224), (117, 200), (117, 136), (117, 142), (127, 114), (127, 212), (127, 213), (127, 214), (127, 222), (127, 226), (127, 227), (137, 136), (137, 213), (137, 150), (147, 136), (147, 214), (147, 223), (207, 121), (207, 140), (207, 122), (207, 134), (217, 126), (217, 141), (217, 124), (217, 144), (106, 204), (106, 212), (106, 113), (116, 141), (116, 126), (116, 210), (116, 216), (116, 121), (116, 113), (116, 105), (126, 216), (136, 210), (136, 213), (136, 214), (146, 202), (146, 210), (146, 215), (146, 222), (146, 226), (206, 123), (216, 144), (216, 105), (226, 140), (226, 112), (226, 114), (226, 144), (101, 100), (101, 102), (101, 125), (101, 204), (101, 215), (101, 113), (101, 104), (111, 200), (111, 204), (111, 215), (111, 220), (121, 202), (121, 215), (121, 113), (121, 134), (131, 205), (131, 220), (141, 124), (141, 205), (141, 225), (201, 145), (211, 124), (221, 104), (221, 124), (100, 125), (100, 204), (100, 102), (100, 113), (100, 104), (100, 144), (100, 105), (110, 204), (110, 220), (120, 150), (120, 202), (120, 205), (120, 113), (140, 114), (140, 214), (140, 224), (150, 143), (150, 213), (200, 142), (200, 104), (200, 145), (210, 124), (210, 144), (210, 115), (210, 145), (102, 203), (102, 204), (102, 103), (102, 135), (112, 204), (122, 213), (122, 223), (132, 225), (202, 133), (202, 114), (212, 103), (222, 104), (103, 204), (103, 114), (113, 104), (113, 105), (113, 125), (113, 204), (133, 114), (133, 144), (143, 213), (143, 223), (203, 135), (213, 124), (213, 145), (104, 105), (104, 204), (104, 215), (114, 115), (114, 204), (134, 224), (144, 145), (144, 214), (204, 105), (204, 125)]
```

3. One of the common application is to calculate the shortest path between two nodes:

```python
p1 = nx.shortest_path(G, source=114, target=211)
print('The shortest path between Node 114 and Node 211: ', p1)
```

Results printed:

```shell
The shortest path between Node 114 and Node 211:  [114, 127, 208, 124, 211]
```

4. Calculate the pagerank value for each node in the graph to see their influences:

```python
print(nx.pagerank(G))
```

Results printed:

```shell
{109: 0.011507076520104863, 119: 0.007835838669313514, 129: 0.015304593799331218, 139: 0.007772926737873626, 149: 0.0073896601012629825, 209: 0.0065558926178649985, 219: 0.014100908598251508, 229: 0.011454115940170253, 108: 0.01645334474680034, 118: 0.01010598371500564, 128: 0.01594717876199238, 138: 0.01671097227127263, 148: 0.015898676579503977, 208: 0.009437234075904938, 218: 0.0153795416919104, 228: 0.005900393773635255, 107: 0.009745182763645681, 117: 0.008716335675518244, 127: 0.021565565312365507, 137: 0.011642680498867146, 147: 0.009721031073465738, 207: 0.01040504770909835, 217: 0.012054472529765329, 227: 0.005615576255373405, 106: 0.007371191843767635, 116: 0.020955704443679106, 126: 0.007589432032220849, 136: 0.015987209357117116, 146: 0.013922108926721374, 206: 0.008554794629575304, 216: 0.011219193251536395, 226: 0.013613173390725904, 101: 0.016680863106330837, 111: 0.010121524312495604, 121: 0.017545503989576015, 131: 0.008531567756846938, 141: 0.014598319866130227, 201: 0.0058643663430632525, 211: 0.003936285336338021, 221: 0.009587911774927793, 100: 0.02243017302167168, 110: 0.007928429795381916, 120: 0.011875669801396205, 130: 0.0073896601012629825, 140: 0.01205992633948699, 150: 0.010045605782606326, 200: 0.015289870550944322, 210: 0.017716629501785937, 220: 0.008666577509181518, 102: 0.014865431161046641, 112: 0.007931095811770324, 122: 0.008087439927630492, 132: 0.004659566123187912, 142: 0.006487446038191551, 202: 0.013579313206377282, 212: 0.01190888044566142, 222: 0.011376739416933006, 103: 0.013438110749144392, 113: 0.02458154500563397, 123: 0.01104978432213578, 133: 0.00743370900670294, 143: 0.008011123394996112, 203: 0.006883198710237787, 213: 0.020392557117890422, 223: 0.012345866520333572, 104: 0.024902235588979776, 114: 0.019369722463816744, 124: 0.017165705442951484, 134: 0.008284361176173354, 144: 0.019363506469972095, 204: 0.03507634139024834, 214: 0.015500649025348538, 224: 0.008320315540621754, 105: 0.01439975542831122, 115: 0.007592722237637133, 125: 0.010808523955754608, 135: 0.006883198710237788, 145: 0.014654713389044883, 205: 0.014660118545887803, 215: 0.01337467974572934, 225: 0.009909720748343093}
```

In addition, similar to the first post in the series, you can access to Gephi [5] to make the visualization look better.

Refer [6] for the code in this post.

## References

[1] [Analyzing Relationships in Game of Thrones With NetworkX, Gelphi, and NebulaGraph (Part One)](https://nebula-graph.com.cn/posts/game-of-thrones-relationship-networkx-gephi-nebula-graph/)

[2] [NetworkX](https://networkx.github.io/)

[3] [NebulaGraph GitHub Repository](https://github.com/vesoft-inc/nebula)

[4] [GraphX](https://spark.apache.org/graphx/)

[5] [Gephi](https://gephi.org/)

[6] [Code PR](https://github.com/vesoft-inc/nebula-python/pull/31)

This article describes in detail how to access NebulaGraph, the graph database used in this example, from NetworkX for data analysis.

Analyzing Relationships in Game of Thrones With NetworkX, Gephi, and NebulaGraph (Part Two)


![image.png](https://www-cdn.nebula-graph.com.cn/nebula-blog/spark-writer.png)

# Starting From Hadoop

With the rise of big data in recent years, distributed computing engines are emerging one after another. Apache Hadoop is a collection of open-source software utilities that is widely applied on many large websites. The core design of Hadoop comes from the Google MapReduce paper. It is inspired by the map and reduce functions commonly used in functional programming. Map is a higher-order function that applies a given function to each element of a functor, e.g. a list, returning a list of results in the same order. It is often called apply-to-all when considered in functional form. A reduce method performs a summary operation. The MapReduce algorithm is used to classify and process data.

# A Little About Apache Spark

Apache Spark is a lightning-fast parallel general-purpose cluster-computing framework. Originally developed at the AMPLab of UC Berkeley in 2009, the Spark codebase was donated to the Apache Software Foundation in 2010. Inspired by Hadoop, Spark inherits the advantages of distributed parallel computing and provides a rich set of operators.

Spark provides a comprehensive and unified framework for different data source managing, batch processing, and stream processing. It supports in-memory computing and outperforms Hadoop. It also supports programming in Java, Scala, and Python. Spark operates the distributed dataset by operating the local collections and provides interactive queries. In addition to the classic MapReduce, Spark also supports Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for incremental computation and stream processing.

![The Spark Stack](https://user-images.githubusercontent.com/57335825/92091361-81a92280-ed85-11ea-8c13-89a72c2f236a.png)

Resilient Distributed Datasets (RDD) is a fundamental data structure of Spark. It is an immutable distributed collection of objects. RDD is fault-tolerant and location-scheduling. Operating an RDD is like operating a local dataset. You need not worry about the scheduling and fault tolerance. When executing multiple queries, RDD allows users to explicitly cache the dataset in memory. Thus you can reuse the cached dataset in the subsequent queries. A Directed Acyclic Graph (DAG) is formed by a series of RDD conversions. The DAGs can be classified into different stages according to the different RDD dependencies.

Like an RDD, a DataFrame is an immutable distributed collection of data. Unlike an RDD, data is organized into named columns, like a table in a relational database. Designed to make large data sets processing even easier, DataFrame allows developers to impose a structure onto a distributed collection of data, allowing higher-level abstraction.

Dataset is a strongly-typed API that converts the functional or related operations in parallel. Conceptually, you can consider the DataFrame as an alias for a collection of generic objects Dataset[Row]. A Row is a generic untyped JVM object. Dataset is a collection of strongly-typed JVM objects and can be optimized with these types.

DataFrame and dataset trigger calculations only when performing operations. Essentially, a data set represents a logical plan that describes the calculations needed to generate the data. When performing operations, Spark's query optimizer optimizes the logical plan and generates an efficient parallel and distributed physical plan.

![History of Spark APIs](https://user-images.githubusercontent.com/57335825/92091467-a7362c00-ed85-11ea-99ce-501e150616b5.png)

# Spark Writer - Import Data from Spark to NebulaGraph

Spark Writer is a Spark-based distributed data importer for [NebulaGraph](https://github.com/vesoft-inc/nebula). It converts data from multiple data sources into vertices and edges for graphs and batch imports data into the graph database. Currently supported data sources are Hive and Hadoop Distributed File System (HDFS).

Spark Writer supports importing multiple tags and edges in parallel, and configuring different data sources for different tags and edges.

Spark Writer generates insert statements from the data through the configuration file and then sends it to the query engine and executes the insert operation. Insert operations are executed asynchronously in Spark Writer. The success and failure stats comes from the accumulator for Spark.

# How to Get Spark Writer

## From the Source Code

```bash
git clone https://github.com/vesoft-inc/nebula.git
cd nebula/src/tools/spark-sstfile-generator
mvn compile package
```

# Example Data for Spark Writer

## Format for the Tag Data File

A tag data file consists of multiple rows. Each line in the file represents a vertex and its properties. In general, the first column is the vertex ID. This ID column is specified in the mapping file. Other columns are the properties of the vertex. Consider the following example in JSON format.

```bash
{"id":100,"name":"Tim Duncan","age":42}
{"id":101,"name":"Tony Parker","age":36}
{"id":102,"name":"LaMarcus Aldridge","age":33}
```

## Format for the Edge Type Data File

An edge type data file consists of multiple rows. Each line in the file represents an edge and its properties. In general, the first column is the ID of the source vertex, the second column is the ID of the dest vertex. These ID columns are specified in the mapping file. Other columns are the properties of the edge. Consider the following example in JSON format.

```bash
{"source":100,"target":101,"likeness":95}
{"source":101,"target":100,"likeness":95}
{"source":101,"target":102,"likeness":90}
{"source":100,"target":101,"likeness":95,"ranking":2}
{"source":101,"target":100,"likeness":95,"ranking":1}
{"source":101,"target":102,"likeness":90,"ranking":3}
```

## Format for the Configuration File

Spark Writer adopts the Human-Optimized Config Object Notation (HOCON) format for its configuration files. HOCON is an easy-to-use and object-oriented format for configuration files. It consists of the Spark field, the Nebula field, the tag mapping field and the edge mapping field.

The Spark related parameters are configured in the Spark field. The username and password information for NebulaGraph is configured in the `nebula` field. Basic data source information for each tag/edge is described in the tag/edge mapping field. The tag/edge mapping field corresponds to multiple tag/edge inputting sources. Different tag/edge can come from different data sources.

The `nebula` configuration field mainly describes the query service IP, username, password and graph space for the nebula services.

```bash
nebula: {  
  # Query engine IP list.  
  addresses: ["127.0.0.1:3699"]  
  
  # Username and password to connect to NebulaGraph service.
  
  user: user  
  pswd: password  
  
  # Graph space name for Nebula Graph.
  space: test  
  
  # The thrift connection timeout and retry times. The default values are 3000 and 3 respectively.
  connection {  
    timeout: 3000  
    retry: 3  
  }  
  
  # The nGQL execution retry times. The default value is 3.
  execution {  
    retry: 3  
  }  
}
```

## The Configuration Field for NebulaGraph

Tag configuration fields are used to describe the imported tag information. Each element in the array represents a piece of tag information. There are two methods to import tags: file-based import and Hive-based import.

- You need to specify the file format for the file-based import.
- You need to specify the query language for the Hive-based import.

```bash
# Processing tags  
tags: [  
  
  # Loading data from the HDFS files. The data type is Parquet. The tag name is ${TAG_NAME}.
  # The field_0, field_1 in the HDFS Parquet file are written to the ${TAG_NAME}.
  # The vertex column is ${KEY_FIELD}.
  {  
    name: ${TAG_NAME}  
    type: parquet  
    path: ${HDFS_PATH}  
    fields: {  
      field_0: nebula_field_0,  
      field_1: nebula_field_1  
    }  
    vertex: ${KEY_FIELD}  
    batch : 16  
  }  
  
  # Similar to the preceding section.
  # Loaded from Hive. The execution command $ {EXEC} is the dataset.
  {  
    name: ${TAG_NAME}  
    type: hive  
    exec: ${EXEC}  
    fields: {  
      hive_field_0: nebula_field_0,  
      hive_field_1: nebula_field_1  
    }  
    vertex: ${KEY_FIELD}
  }  
]
```

Descriptions:

- The name field represents the tag name.
- The mapping relationship between the HDFS/Hive and Nebula are configured in the fields.
- The batch parameter stores the batch data records. You can configure it based on your needs.

Edge type configuration fields are used to describe the imported edge type information. Each element in the array represents a piece of edge type information. There are two methods to import edges: file-based import and Hive-based import.

- You need to specify file format for file-based import.
- You need to specify the query language for the Hive-based import.

```bash
# Processing tags  
edges: [  
  # Loading data from the HDFS files. The data type is JSON.
  # The edge type name is ${EDGE_NAME}.
  # The field_0, field_1 in the HDFS JSON file are written to the ${EDGE_NAME}.

  # The source vertex is source_field, the target vertex is target_field, the edge rank is the ranking_field.
  {  
    name: ${EDGE_NAME}  
    type: json  
    path: ${HDFS_PATH}  
    fields: {  
      field_0: nebula_field_0,  
      field_1: nebula_field_1  
    }  
    source:  source_field  
    target:  target_field  
    ranking: ranking_field  
  }  
  
  # Loaded from Hive. The execution command $ {EXEC} is the dataset.
  # The edge rank is optional.
  {  
    name: ${EDGE_NAME}  
    type: hive  
    exec: ${EXEC}  
    fields: {  
      hive_field_0: nebula_field_0,  
      hive_field_1: nebula_field_1  
    }  
    source:  source_id_field  
    target:  target_id_field  
  }  
]
```

Descriptions:

- The name field represents the edge type name.
- The mapping relationship between the HDFS/Hive and Nebula are configured in the fields.
- The source field is the source vertex of the edge.
- The target field is the target vertex of the edge.
- The ranking field is the rank of the edge.
- The batch parameter stores the batch data records. You can configure it based on your needs.

## Data Importing Command

```bash
bin/spark-submit \  
 --class com.vesoft.nebula.tools.generator.v2.SparkClientGenerator \  
 --master ${MASTER-URL} \  
 ${SPARK_WRITER_JAR_PACKAGE} -c conf/test.conf -h -d
```

Descriptions:

- `-c` specifies the path to the configuration files.
- `-h` specifies whether Hive is supported.
- `-d` checks whether the configuration files are correct. It doesn't process the data.

# You might also like:

1. [Automating Your Project Processes with Github Actions](https://nebula-graph.io/posts/github-action-automating-project-process/)
1. [Practice Jepsen Test Framework in NebulaGraph](https://nebula-graph.io/posts/practice-jepsen-test-framework-in-nebula-graph/)
1. [Integrating Codecov Test Coverage With NebulaGraph](https://nebula-graph.io/posts/integrate-codecov-test-coverage-with-nebula-graph/)

> Hi, I’m Darion, a software engineer in NebulaGraph. Currently I am involved in storage engine development. I hope this article is helpful and let me know if you have any thoughts and suggestions. Thanks!

Spark Writer is a Spark-based distributed data importer for NebulaGraph. It converts data from multiple data sources into vertices and edges for graphs and batch imports data into the graph database.

NebulaGraph Data Importer: Spark Writer


![Analyzing Relationships in Game of Thrones With NetworkX, Gephi, and NebulaGraph](https://user-images.githubusercontent.com/57335825/90602643-f2114a80-e1ae-11ea-9fea-0516b8bf71e1.png)

The hit series _Game of Thrones_ by HBO is popular all over the world. Besides the unexpected plot twists and turns, the series is also known for its complex and highly intertwined character relationships. In this post, we will access the open source graph database [NebulaGraph](https://github.com/vesoft-inc/nebula) with [NetworkX](https://networkx.github.io/) and visualize the complex character connections in _Game of Thrones_ with [Gephi](https://gephi.org/).

# Introduction to the Dataset

The dataset we used in this article is: _A Song of Ice and Fire_ Volume One to Volume Five[1].

- Character set (vertices set): Each character in the book is stored as a vertex, and the vertex has only one property, i.e. name.
- Relation set (edges set): If two characters connect directly or indirectly in the book, there is an edge between them. The edge has only one property, i.e. weight. The weight represents the intimacy level of the relationship.

The preceding vertices set and edges set constitute a graph, which is stored in the graph database NebulaGraph[2].

## Community Detection: The Girvan-Newman Algorithm

We used the built-in community detection algorithm Girvan-Newman provided by NetworkX[3] to divide communities for our graph network.

Below are some explanations for the algorithm:

> In the network graph, the closely connected part can be regarded as a community. Connections among vertices are relatively close within each community, while the connections between the two communities are loose. Community detection is the process of finding the communities contained in a given network graph.
> [Girvan-Newman](https://en.wikipedia.org/wiki/Girvan%E2%80%93Newman_algorithm) is a community detection algorithm based on the betweenness. Its basic idea is to progressively remove edges from the original network according to the edge betweenness until the entire network is broken down into communities. By removing these edges, the groups are separated from one another and so the underlying community structure of the network is revealed. Therefore, the Girvan-Newman algorithm is actually a splitting method.
> The algorithm's steps for community detection are summarized below:
> （1）The betweenness of all existing edges in the network is calculated first.
> （2）The edge(s) with the highest betweenness are removed.
> （3）Steps 2 and 3 are repeated until no edges remain.

With this explanation, let's see how to use the algorithm.

1. Detect communities with the Girvan-Newman algorithm. The NetworkX sample code is as follows:

```python
comp = networkx.algorithms.community.girvan_newman(G)
k = 7
limited = itertools.takewhile(lambda c: len(c) <= k, comp)
communities = list(limited)[-1]
```

2. Add a community property to each vertex in the graph. The property value is the community number where the vertex is located.

```python
community_dict = {}
community_num = 0
for community in communities:
    for character in community:
        community_dict[character] = community_num
        community_num += 1
        nx.set_node_attributes(G, community_dict, 'community')
```

## Vertex Style: The Betweenness Centrality Algorithm

Next we will adjust the size for the vertex and the size for the character name marked on the vertex. We will use NetworkX's Betweenness Centrality algorithm to achieve our goals.

> The importance of each vertex in the graph can be measured by the centrality of it. Different centrality definitions are adopted in different networks to describe the importance of the vertices in the network. Betweenness Centrality judges the importance of a vertex based on how many shortest paths pass through it.

1. Calculate the value of the betweenness centrality for each vertex.

```python
betweenness_dict = nx.betweenness_centrality(G) # Run betweenness centrality
```

2. Add a new betweenness property for each vertex in the graph.

```python
nx.set_node_attributes(G, betweenness_dict, 'betweenness')
```

## The Edge Size

The size of an edge is determined by the weight of the edge.

Through the preceding process, now our vertices have three properties: name, community, and betweenness. Edges only have one property: weight.

The code is as follows:

```python
import matplotlib.pyplot as plt
color = 0
color_map = ['red', 'blue', 'yellow', 'purple', 'black', 'green', 'pink']
for community in communities:
    nx.draw(G, pos = nx.spring_layout(G, iterations=200), nodelist = community, node_size = 100, node_color = color_map[color])
    color += 1
plt.savefig('./game.png')
```

emmm..., not quite good looking.

![Gephi Visualization](https://user-images.githubusercontent.com/57335825/90603223-d0fd2980-e1af-11ea-8b71-f80049b8e1e3.png)

Although NetworkX itself has many visualization functions, Gephi[4] looks better in interaction and visualization.

## Gephi - The Graph Visualization Tool 

Now let's export the preceding NetworkX data as a `game.gephi` file and import it into Gephi.

```python
nx.write_gexf(G, 'game.gexf')
```

![Access Gephi](https://user-images.githubusercontent.com/57335825/90603357-0ace3000-e1b0-11ea-8842-19756a8fcff3.png)

## Graph Display in Gephi

Open the `game.gephi` file you just exported into Gephi, and then modify the parameters in Gephi to get a prettier visualized picture:

1. Set the Layout to Force Atlas, modify the Repulsion strength to 500.0, and click the `Adjust by sizes` option to avoid vertices overlap as much as possible.

Force Atlas is a force-guided layout. The force-guided layout method can produce a fairly beautiful network layout and fully demonstrate the overall structure of the network and its automorphic characteristics. The force-guided layout imitates the gravitational and repulsive forces in physics, and automatically lays out until the forces are balanced.

![Set the Layout](https://user-images.githubusercontent.com/57335825/90603579-5a146080-e1b0-11ea-93bf-ad65e81cde46.png)

2. Color the divided communities.

Select Appearance, Nodes, Color, Partition, and community. The community here is the community number property we just added for each vertex.

![Color the divided communities](https://user-images.githubusercontent.com/57335825/90603677-7b754c80-e1b0-11ea-8ad6-8028e10d6c8e.png)

3. Set the size for vertices and the character name properties for the vertices.

Select Appearance, Nodes, Size, Ranking, and betweenness. The betweenness here is the betweenness property we just added for each vertex.

![Set the size for vertices and the character name properties for the vertices](https://user-images.githubusercontent.com/57335825/90603727-9051e000-e1b0-11ea-924d-a0ba8ac799fe.png)

4. The size of the edge is determined by the weight property of the edge.

Select Appearance, Edges, Size, Ranking, and Weight.

![The size of the edge is determined by the weight property of the edge](https://user-images.githubusercontent.com/57335825/90603783-a8296400-e1b0-11ea-952a-e687f37b2b62.png)

5. Export the visualized picture.

![Export the visualized picture](https://user-images.githubusercontent.com/57335825/90603823-bbd4ca80-e1b0-11ea-8c3e-b1c225844f01.png)

Now you've got a relationship graph for characters in _Game of Thrones_. Each vertex represents a character.

# What's Next

This article mainly introduces how to visualize your data with NetworkX and Gephi. Our next article will introduce how to access the data in the graph database NebulaGraph through NetworkX.

The code for this article can be accessed at [5] below.

This article is inspired by [6] below.

# References

[1] [_Game of Thrones_ dataset](https://www.kaggle.com/mmmarchetti/game-of-thrones-dataset)

[2] [NebulaGraph GitHub Repository](https://github.com/vesoft-inc/nebula)

[3] [NetworkX](https://networkx.github.io/)

[4] [Gephi](https://gephi.org/)

[5] [Source Code Repository](https://github.com/jievince/nx2gephi)

[6] [https://www.lyonwj.com/2016/06/26/graph-of-thrones-neo4j-social-network-analysis/](https://www.lyonwj.com/2016/06/26/graph-of-thrones-neo4j-social-network-analysis/)

This article show you how to analyze the complex and intertwined character relationships in Game of Thrones with NetworkX, Gephi, and NebulaGraph.

Analyzing Relationships in Game of Thrones With NetworkX, Gephi, and NebulaGraph (Part One)


![D3.js Examples for Advanced Uses - Custom Visualization](https://user-images.githubusercontent.com/57335825/90104877-da8f1900-dcf9-11ea-9e73-b96e3be620c6.png)


In the previous article [D3-Force Directed Graph Layout Optimization in NebulaGraph Studio](https://nebula-graph.io/posts/d3-force-layout-optimization/), we have discussed the advantages that D3.js has over other open source visualization libraries in custom graph  and the flexible operations on the document object model (DOM) with D3 js. Given the customizability of the D3.js, is it possible to achieve whatever I want by using it? In this article, I will show you how to take full advantage of the flexibility of D3.js to realize on-demand functions which are not supported by D3.js itself.

# Building the D3-Force Directed Graph

Here I won't elaborate on the principle of the particle physical movement module of the d3-force. You can refer to our previous [post](https://nebula-graph.io/posts/d3-force-layout-optimization/) if you are interested in this topic. Instead, I will focus on the practice of visualization in this article.

Now let me show you how I developed some new functions with the help of D3.js to better analyze the graph databases. Firstly, let's build a simple relationship network with the [d3-force](https://d3js.org.cn/document/d3-force/#installing) directed graph.

```Shell
this.force = d3
        .forceSimulation()
        // Allocate coordinates for the vertices
        .nodes(data.vertexes)
        // Link
        .force('link', linkForce)
        // For setting the center of gravity of the system
        .force('center', d3.forceCenter(width / 2, height / 2))
        // The gravity
        .force('charge', d3.forceManyBody().strength(-20))
        // The collision force, for preventing the vertices from overlapping
        .force('collide',d3.forceCollide().radius(60).iterations(2));

```

We can get the following vertices and relationships graph with the preceding code.

![Build a sample relationship network](https://user-images.githubusercontent.com/57335825/90103470-6b182a00-dcf7-11ea-9d52-aca261294ba3.png)

The preceding figure is a screenshot of the exploration tab of the graph visualization tool, NebulaGraph Studio. There you can select a certain vertex as the starting point of exploration by finding other vertices that are associated with it. For example, in this figure, vertex 100 and vertex 200 are connected with a single directed follow relation.

The problem is, if I have a super vertex which have thousands of edges, or if I want to display multi-hop query results, then the visual graph needs to display vertices and edges in huge density. It is also difficult to locate a specific vertex. Chances are users want to analyze only part of the data in the graph instead of the whole graph. Deleting the selected data will be a great solution to this scenario. You just delete the unwanted data and keep what you want.

# Deleting the Selected

Before introducing how this function is implemented, let me begin with the native APIs provided by D3.js. Yes, I mean the [enter()](https://github.com/d3/d3-selection/blob/v1.4.1/README.md#selection_enter) mentioned in the previous [post](https://nebula-graph.io/posts/d3-force-layout-optimization/) and [exit()](https://github.com/d3/d3-selection/blob/v1.4.1/README.md#selection_exit) I didn't cover last time. Here are some descriptions from the documentation:

> _When binding data, it's likely that the array has more (or less) elements than the DOM elements. Fortunately D3 can help in adding and removing DOM elements using the `.enter` and `.exit`_.
> _If the array is longer than the selection there’s a shortfall of DOM elements and we need to add elements with `enter`_.
> _If the array is shorter than the selection there’s a surplus of DOM elements and we need to remove elements with `exit`_.
> _There are three cases in data binding_:
> • _A shortfall of DOM elements_
> • _A surplus of DOM elements_
> • _The array is equal with the DOM elements_

According to the documentation, it seems simple to implement the "deleting the selected" function. I was so optimistic that I thought simply operating on the data level was enough. Thus I deleted some vertices directly from the vertices data, then removed the extra element with the `d3.select(this.nodeRef).exit().remove()` API. Now let's check the result of this operation:

![Deleted the selected nodes with d3.js](https://user-images.githubusercontent.com/57335825/90103845-0c06e500-dcf8-11ea-8d38-bff01e285be7.png)

The targeted vertices are deleted as expected. However, other vertices are messed up because the colors and properties of the vertices are inconsistent with the current `DOM` vertices. Why? I checked the documentation more carefully and was inspired by an idea. Why not print the exit().remove() vertices out to see which vertices are removed?

Sure enough my guess has been confirmed. It's the length change of the listening element that triggers the `enter()` and `exit()` function. That is to say, if two elements are taken out, the `exit()` will be triggered. But it won't process the data you want to delete. Instead, it processes the last two vertices of the current data. In another word, `enter()` and `exit()` are triggered by the data length. However, take `exit()` as example, when D3.js detects any data length changes, say N, it will cut all the elements between the last N and the last element. On the contrary, `enter()` will add N pieces of data after the last element in the array.

Therefore, although the vertices are deleted from the previously returned data (apparently they are not the last element in the current array), the `d3.select(this.nodeRef).exit()` will locate the last element in the existing graph. This is totally a mess. So how to deal with this issue?

The D3.js recommends that you solve this problem with the officially provided [merge function](https://www.d3indepth.com/enterexit/). But in our case, since we know the IDs of the vertices to be deleted, we operated directly on the DOM.

Here's my simple yet effective solution. Since the `exit()` API can't meet our requirements, I will process the vertices to be deleted separately. First I need to locate the DOM where the deleting operation is actually performed. To achieve this, I need to bind an ID to each vertex when rendering. Then I traverse. Find the corresponding DOM of the deleted vertices based on their ID. Following is my code:

```shell
componentDidUpdate(prevProps) {
    const { nodes } = this.props;
    if (nodes.length < prevProps.nodes.length) {
      const removeNodes = _.differenceBy(
        prevProps.nodes,
        nodes,
        (v: any) => v.name,
      );
      removeNodes.forEach(removeNode => {
        d3.select('#name_' + removeNode.name).remove();
      });
    } else {
      this.labelRender(this.props.nodes);
    }
  }
```

In fact, I  need to not only locate the actual DOM, but also delete all the associated edges and displaying properties of the deleted vertices. This is because an edge without a start or destination vertex is meaningless. The processing method of edges and properties are similar to deleting vertices. 

# The Zoom Button

In addition to deleting the selected, zoom is another common function in data visualization. The [d3.zoom()](https://github.com/d3/d3-zoom/blob/v1.8.3/README.md#zoom) API in D3.js is designed to solve such problems. And it performs stably because it has been  battle tested by many users. Then why do I bother reinventing the wheel?

In fact, zoom is purely an interactive function. Of course I can use the scroll wheel to zoom. But users who know little about NebulaGraph Studio can hardly find this hidden function, and the zoom proportion is hard to control with a scroll wheel. For example, the scroll wheel is useless if you want to zoom to a certain ratio, say 30% or 50%. In addition, during the process of implementing zoom by using the scroll wheel, I found that scrolling will affect the position offset of vertices and edges. What is the reason for this?

By checking the `d3.zoom()` code, I found that D3.js first gets the zoom value of the d3.event and then modify the transform value for the entire canvas. But such processing method won't change the x and y coordinates of the vertices and edges. Thus when implementing zoom in with d3.zoom(), the view will shift to the upper left (because compared with the x and y coordinates of the edge elements in the view, the canvas becomes smaller). And when implementing zoom out with `d3.zoom()`, the view will shift to the bottom right.

When zooming the canvas, we will process the vertex and edge location relative to the canvas size according to the zoom value.

We provide the users with a slider to control the zoom ratio of the canvas. Here we use the [Slider](https://ant.design/components/slider/) designed by antd. It can control the zoom ratio based on its slide values. Following is the code:

```Shell
<svg
  width={width}
  viewBox={`0 0 ${width * (1 + scale)}  ${height * (1 + scale)}`}
  height={height}
  >
 {/*****/}
</svg>
```

The scale parameter in the preceding code is generated based on the zoom value for the slider. We need to record the value to zoom in the canvas (svg element) so that a zoom out effect is achieved.

In addition, we also need the scale value when processing the above-mentioned vertex and edge offset. Therefore, we need to set an de-offset value for the vertex and edge. Simply put, if the canvas is enlarged by scale times, the coordinates of the vertices and edges must offset by scale times so that the relative locations of the vertices and edges are not changed in the canvas. Following is the core code of the offset:

```Shell
const { width, height } = this.props;
    const scale = (100 - zoomSize) / 100;
    const offsetX = width * (scale / 2);
    const offsetY = height * (scale / 2);
    // Set the offset for the partent DOM <g/> of the vertices and edges to be operated on 
    d3.select(this.circleRef).attr(
      'transform',
      `translate(${offsetX} ${offsetY})`,
    );
```

# Conclusion

In this article, I shared two practical and frequently used functions in data visualization: deleting the selected data and custom zooming (including optimizing the graph offset). There are still many issues to consider in visualizing a relational graph. D3.js is so highly customizable that we can implement whatever functions we want. Imagination is the only limit.

We will keep on optimizing and sharing our using experience on D3.js in the future. Stay tuned!

> _Hi, I'm Nico, the front-end engineer at NebulaGraph. I'm interested in data visualization and would like to share my experiences in this regard. Hope my post is of help to you. Please let me know if you have any ideas about this. Thanks!_

# You might also like

- [Automating Your Project Processes with Github Actions](https://nebula-graph.io/posts/github-action-automating-project-process/)
- [How to Reduce Docker Image Size](https://nebula-graph.io/posts/how-to-reduce-docker-image-size/)


This article will show you how to take full advantage of the flexiblility of D3.js to realize on-demand functions which are not supported by D3.js itself.

D3.js Examples for Advanced Uses - Custom Visualization


![Why Using Hugo as Static Site Generator -Hands-on Experience at NebulaGraph](https://user-images.githubusercontent.com/57335825/88053343-7e791080-cb10-11ea-95fa-8d8c465d6ca4.png)

Site generators are no longer a novel topic. Various programming languages have released their own popular website building frameworks. For example, Python released [Pelican](https://github.com/getpelican/pelican) , JavaScript released [Hexo](https://github.com/hexojs/hexo) and PHP released [WordPress](https://github.com/WordPress/WordPress), which has the largest market share. I have tried all these tools when deploying my personal blog. But when building the official [NebulaGraph Site](https://nebula-graph.io/), after evaluation, I chose [Hugo](https://gohugo.io/), which is written in Golang.

To clarify, there is not much difference in the frameworks written in various languages. It is personal preferences and aesthetics that matter, which is not the topic covered in this post.

# Website Requirements at NebulaGraph

## Blog Content Management

The NebulaGraph team frequently shares [technical articles](https://nebula-graph.io/posts/) on the company blog. Hugo has a flexible and powerful content management system. You can add various types of content, such as blogs, releases, and technical documents as required. Details will be introduced later.

## Product Proliferation

Commonly seen product proliferation pages include a home page, a newsroom page, and a company page, etc. This requires that the Content Management System supports various types of pages. These pages may share the same components such as the navi bar and footer. Details will be covered later as well.

## SEO

Hugo itself is a web framework similar to the service-end template, which supports server-end rendering natively.

## Multi-Language Support

NebulaGraph serves users worldwide. To ensure quality user experience, the website should support multiple languages. Hugo has us covered. As long as you have the corresponding corpus configuration, you can use the language on your site and manage it easily.

## Easy-to-Manage for Non-Technical Staff

Based on Hugo's powerful template system, the content managers can use it smoothly after the technicians have made custom development on the template.

## Hugo Feature Highlights

### Flexible and Powerful Content Management System

```markdown
...
├── config // The corpus your template needs.
|   ├── default
|   |   ├── config.toml
|   |   └── config.cn.toml
|   |   └── config.en.toml
|   |   └── footer.cn.toml
|   |   └── footer.en.toml
|   |   └── ...
├── content // Maintained by the content team.
|   ├── en
|   |   ├── posts // The built-in content type: post.
|   |   |   ├── post-01.md
|   |   |   ├── post-02.md
|   |   └── release // The customized content type: press releases.
|   |   |   ├── release-01.md
|   |   |   └── release-02.md
|   ├── cn
|   |   ├── posts
|   |   |   ├── post-01.md
|   |   |   ├── post-02.md
|   |   └── release
|   |   |   ├── release-01.md
|   |   |   └── release-02.md
...
├── themes // Theme of the site.
|   ├── nebula-theme // Theme name.
|   |   ├── layout // The template.
|   |   |   ├── _default // The default template.
|   |   |   |   ├── baseof.html // The definition of the rendered seed page.
|   |   |   |   ├── list.html  // The default post content type adopted by the blog home page.
|   |   |   |   ├── single.html // The default post content type adopted by the blog post pages.
|   |   |   ├── partials // Reused template fragments.
|   |   |   |   ├── head.html
|   |   |   |   ├── footer.html
|   |   |   |   ├── menus.html
|   |   |   |   ├── ...
|   |   |   ├── index.html // The template adopted by the homepage（'/').
|   |   |   ├── section
|   |   |   |   ├── release.html // The customized press release template adopted by the specific press release pages.
|   |   |   |   ├── news.html // The customized news coverage template adopted by the newsroom page.
...
```

The above is the project structure built with the powerful Hugo template. Personally I think it demonstrates the requirements of different sites intuitively. Refer to Hugo's official [Templates Overview](https://gohugo.io/templates/) for more information.

### Rich Built-in Tools

In addition to the powerful content management system, Hugo provides a lot of useful built-in templates and functions to improve the building efficiency. For example:<

- [Pagination](https://gohugo.io/templates/pagination/) for resources pages.

The Pagination template helps you paginate your resources pages such as blogs, press releases, and news coverage.

- [RSS](https://gohugo.io/templates/rss/) template.

RSS is indispensable for informational sites. Hugo has shipped with the default RSS template and all you need to do is adding a line of code to settle with the RSS. You can customize your own RSS as well.

- Various [Functions](https://gohugo.io/functions/) to process strings and contents.

Hugo supports replacing common strings in applications. Hugo also supports pipe to pass contents through layers, just like the commands input logic in Linux.

- Easy-to-use CLI tool

The built-in http server is convenient for local development. At the same time, it can package the entire site into a static resource, which facilitates the deployment and decreases the maintenance costs. You can initialize and start your project with one click. This makes Hugo an out-of-the-box tool and easy to get started.

- Powerful content management tools
   - [TableOfContents](https://gohugo.io/content-management/toc/#readout) - Quickly extract a ToC for blog posts
   The TableOfContents function generates a Table of Contents based on your current blog, saving the anchor generating time.
   - [Summary](https://gohugo.io/content-management/summaries/) - Generate a preview for your content.
   - [Image Processing](https://gohugo.io/content-management/image-processing/) - Obtain the image resources for your content.
   - [URL Management](https://gohugo.io/content-management/urls/) - Customize the URL for your content.

The above are some of the powerful Hugo tools that we have adopted in our website practice. Hugo provides far more abundant tools than these. Refer to the [Documentation](https://gohugo.io/about/) for more.

### Excellent Community Support

- There are a large number of templates for you to choose from.

Hugo has become the most popular site framework written in Golang. As more and more people are using Hugo to build static sites, Hugo encourages the community to open source their own themes and make sure these themes can be easily found by others on the Hugo official website. Non-technicians can also find topics that meet their needs without writing a single line of HTML code, and generate their own sites.

- Various channels to get help
   - As a project with 45k+ stars, Hugo has a large user base. You can find a solution to your problem by Googling most of the time.
   - The maintainers of the Hugo [forum](https://discourse.gohugo.io/) are active and your problem can be solved in a rather timely manner as long as it is expressed properly.
   - Hugo's [Documentation](https://gohugo.io/documentation/) is well-organized and easy to use.


# Hugo Best Practices from Hands-on Experiences

In addition to the above Hugo highlights, I would like to share some best practices out of my hands-on experiences using Hugo as the site generator.

## Use The Existing Themes

The quickest way to use Hugo is to find a theme that best suits your requirements in the theme market as the basis. Sometimes you don't even need to customize the themes. Even if customization is required, the learning curve of Hugo's project structure and running mechanism is relatively low because Hugo has defined the conventions quite clearly.

## Understand the Relationship Between Project Structure and Content Beforehand

With the earlier introduced content management system and Hugo's [Documentation](https://gohugo.io/documentation/) you can learn the mapping relationship between contents and layouts. Once obtaining an idea of the mapping relationship, you can develop in the way Hugo expects you to and find it easier to understand the doc.

## Customize Your Site

In addition to Hugo's own framework, specifications, and tools, you can introduce your customized code for every template as long as you are familiar with HTML/CSS/Javascript and Hugo's rules such as combination of various templates. In our practice, we have integrated with third-party plugins (such as data statistics, Discourse and ShareThis) and introduced some customized popup windows with JavaScript. So as long as you are familiar with the regular development of web pages, in addition to Hugo's basic functions, you can do anything to your site within your ability.

## Build a Static Site

Hugo will package your content to a static resource package so that you can deploy your site anywhere. For example, you can use the free GitHub [Pages](https://pages.github.com/) or the OSS, saving you from server and database maintenance. Deploying static resources is quite simple. Take Hugo as an example. Its routing relates with the file directories. Our site is available in both Chinese and English. And the two versions are developed in the same project and share the same template. When deployed, the two versions are packaged into different resource packages and are sent to different web servers to optimize user experience.

# Conclusion

The above are my experiences on using the Hugo framework to build [NebulaGraph Website](https://nebula-graph.io/). I hope that this article provides some ideas and references to those who need to quickly build sites. I have customized our site based on the existing themes. If you are curious about it, please go to the [GitHub Repository](https://github.com/vesoft-inc/nebula-website) for the details. Welcome to visit our graph database [NebulaGraph](https://github.com/vesoft-inc/nebula), thank you.

> _Hi, I’m Jerry, the front-end engineer at NebulaGraph. I would like to share my experiences in the front-end development. Hope my post is of help to you. Please let me know if you have any ideas about this. Thanks!_

# You might also like

1. [Automating Your Project Processes with Github Actions](https://nebula-graph.io/posts/github-action-automating-project-process/)
1. [How to Reduce Docker Image Size](https://nebula-graph.io/posts/how-to-reduce-docker-image-size/)

This article explains the reason why NebulaGraph chooses Hugo as the static site generator.

Why Using Hugo as Static Site Generator - Hands-on Experience at NebulaGraph

NebulaGraph has adopted GitHub Actions to automate testing and releasing workflows like Nightly building and branch release. This post shared some best practices with you in this regard.

Automate Workflows with Github Action in NebulaGraph


![D3-Force Directed Graph Layout Optimization in NebulaGraph Studio](https://user-images.githubusercontent.com/57335825/81265062-02c10900-9075-11ea-8049-bf3849df5791.png)

## What is D3.js

[D3.js](https://d3js.org/) is an open-source JavaScript library for producing dynamic, interactive data visualizations in web browsers using SVG, HTML and CSS.

In addition to D3, there are  other popular and powerful libraries such as [ECharts](https://echarts.apache.org/examples/zh/index.html) and [Chart.js](http://chartjs.cn/). However, they are **highly encapsulated**,leaving too little room for customization.

In the contrary, D3 is highly customizable thanks to its support for **event handling for SVG elements**. It can bind arbitrary data to the document object model (DOM) or directly operate [W3C DOM API](https://www.w3.org/DOM/DOMTR) on the DOM.

For those who want full control of their graph layout, D3 is definitely a go-to choice.

## D3-Force Directed Graph

For social networks analysis, D3-force directed graph is the best choice. It is a module realized in D3 by simulating the [velocity Verlet](https://en.wikipedia.org/wiki/Verlet_integration) numerical integrator for physical forces on particles. Each node in D3-force can be regarded as a discharge particle, and there is repulsion (Coulomb repulsion) among the particles. At the same time, these particles are connected by the edges among them, thus generating traction.

Because of the repulsion and traction, particles in D3-force continuously shift from the initial state of random disorder to a balanced and ordered layout. The [velocity Verlet](https://en.wikipedia.org/wiki/Verlet_integration) numerical integrator  controls the order of particles and  edges. The graph generated with d3-force contains only nodes and edges and there is only a small collection of [graph samples](https://observablehq.com/collection/@d3/d3-force) for reference because most graphs are customized.

The following is a popular network diagram I found that is realized with D3-force. Apparently it is too simple for some use cases. So if yours is more complex than that, then you have to create your own graph.

![force-directed graph example](https://user-images.githubusercontent.com/57335825/81265260-5895b100-9075-11ea-8dec-a4aeadbe33ab.png)

## Building D3-Force Directed Graph

In NebulaGraph Studio, we use the D3-force directed graph to analyze data relationships because the nodes and edges show the data connections intuitively and it allows graph exploration via [graph query language](https://nebula-graph.io/posts/graph-query-language-comparison-cypher-gremlin-ngql/). Moreover, the graph data can be updated by synchronizing the operations on the DOM to the database, which deserves another article to cover.

Let's build a simple D3-force directed graph to illustrate how D3.js displays data connections and share some layout optimization ideas based on this example.

```shell
this.force = d3
        .forceSimulation()
        // Assign coordinates to nodes
        .nodes(data.vertexes)
        // Connect edges
        .force('link', linkForce)
        // The instance center
        .force('center', d3.forceCenter(width / 2, height / 2))
        // Gravitation
        .force('charge', d3.forceManyBody().strength(-20))
        // Collide force to prevent nodes overlap
        .force('collide',d3.forceCollide().radius(60).iterations(2));
```

The code above generates a graph as below:

![force-directed graph in NebulaGraph Studio](https://user-images.githubusercontent.com/57335825/81265424-a8747800-9075-11ea-88fe-3f7a21de77d0.png)

### Graph Exploration Layout Optimization

The graph above displays only one-hop relationships from the starting nodes. How about two- or three-hop relationships? The answer is the D3.js [enter()](https://www.d3indepth.com/enterexit/#enter) API.

The D3.js [enter()](https://www.d3indepth.com/enterexit/#enter) API processes the new nodes independently. When new nodes are queried and pushed into the nodes array, the API renders them according to the coordinates assigned by the D3-force instance, without changing the information (x, y coordinates included) of the existing nodes.

From the perspective of API, this is definitely understandable. But the newly added nodes can not be handled just with a simple push to the  existing D3-force instance because the [d3.forceSimulation()](https://github.com/d3/d3-force/blob/master/src/simulation.js) module assigns location coordinates randomly.

The coordinates assigned by d3.forceSimulation().node() are random, so are the locations of the explored nodes. Together with the collied and link parameter, nodes associated with the new ones are close to each other under the influence of tractive force.  Also, there are collisions among other nodes in the process of approaching. When there are nodes on the force-directed graph, these newly added nodes will make the entire graph collide under the effect of the collision and traction until each node finds its own place. That means the movement only stops when the collision and traction both meet the requirement. Does it look like the Big Bang?

![force-directed graph movement](https://user-images.githubusercontent.com/57335825/81270806-d9f14180-907d-11ea-8aa1-cda0a3f8b1ef.gif)

There are two problems in the above process:

1. Adding a new node would result in the entire graph's constant moving
1. It takes a relatively long time to be stable

However, this is how the the enter() API is designed.

One solution is to process  new nodes and the existing ones separately, which means each node rendering requires traverse to determine whether it's new or existing. Not a performant solution, especially when the quantity is large.

Another common solution  is to reduce the collision and increase traction of the D3-force instance. In this way, the nodes find balance state more quickly so that the entire graph is stable. This is a better solution but the disadvantage is that it makes the connection among nodes varies remarkably in length and the  graph size is huge. So it's not an ideal solution for cases with huge data.

We've come up with a new solution.

The idea is to ensure that the new node is around the source node. Instead of D3.forceSimulation().node() allocation, set the coordinates of the new node the same as that of the source node. The node collision of the D3-force instance ensures that the appearance of new nodes will not be overwritten, and will eventually appear around the source node.

Although such solution still affects the nodes within a small range, it will not greatly affect the trend of the entire graph. The key codes are as follows:

```shell
# Set the new nodes coordinates as the source node center or the entire graph center
addVertexes.map(d => {
  d.x = _.meanBy(selectVertexes, 'x') || svg.style('width') / 2;
  d.y = _.meanBy(selectVertexes, 'y') || svg.style('height') / 2;
});
```

If there's no source node, then the new node appears at the center of the graph. This will pose less effect on the existing nodes thus is worth considering.

### Layout Optimization of Multiple Edges Display Between Two Nodes

When there are multiple edges between two nodes, the following problems occur:

1. The default straight line will overwrite each other. Therefore,  curve is a better choice under such circumstances.
1. How to define the curvature of the curve to ensure there are no overwritten?
1. When you have multiple curves, how to make sure the average semi-circular arc is not on a certain semi-circular?

Below is how we solve the above problems:

1. First count the edges between any two nodes and group them to a map. The key in the map is based on the name of the two nodes involved.
1. Then divide the edges in the same map into two groups by their direction. There are plenty of ways to determine the edge direction and here we adopt the method to compare the ASCII of source.name and target.name of a node. A linknum is set for each edge with a positive direction. Similarly, a -linknum is set for each edge with a negative direction. The linknum value we set is to determine the curvature and bending direction of the arc. 

Refer to the following code for better understanding:

```javascript
  const linkGroup = {};
  // Set the edges between two nodes as the same key based on their name property. 
  // Then add the key to the linkGroup, making all edges a group
  edges.forEach((link: any) => {
    const key =
      link.source.name < link.target.name
        ? link.source.name + ':' + link.target.name
        : link.target.name + ':' + link.source.name;
    if (!linkGroup.hasOwnProperty(key)) {
      linkGroup[key] = [];
    }
    linkGroup[key].push(link);
  });
  // Traverse each group to call setLinkNumbers to allocate linknum
  edges.forEach((link: any) => {
    const key = setLinkName(link);
    link.size = linkGroup[key].length;
    const group = linkGroup[key];
    if (group[group.length - 1] === link) {
      setLinkNumbers(group);
    }
  });
```

```javascript
// Divide the edges into linkA and linkB based on their directions.
// Then allocate two kinds of linknum to control the upper and lower elliptical arc.

export function setLinkNumbers(group) {
  const len = group.length;
  const linksA: any = [];
  const linksB: any = [];
  for (let i = 0; i < len; i++) {
    const link = group[i];
    if (link.source.name < link.target.name) {
      linksA.push(link);
    } else {
      linksB.push(link);
    }
  }
  let startLinkANumber = 1;
  linksA.forEach(linkA=> {
    linkA.linknum = startLinkANumber++;
  }
  let startLinkBNumber = -1;
  linksB.forEach(linkB=> {
    linkB.linknum = startLinkBNumber--;
  }
}
```

After allocating linknum to each edge, the sign of linknum is judged in the tick event function that monitors the edges. We only need to judge and set [the curvature and direction of the path](https://www.w3schools.com/graphics/svg_path.asp).

Here's how it looks:

![Optimize multiple edges between two nodes](https://user-images.githubusercontent.com/57335825/81265818-48320600-9076-11ea-9cad-18f91433beb9.png)

## Conclusion

This is how we optimized the D3.js force-directed graph. There are many issues to concern and optimize if you're building a complex relationships.

In this article we shared the two most common scenarios, i.e. presenting new nodes and multiple edges between two nodes. We will share more experiences in the future. Stay tuned!

Try NebulaGraph with NebulaGraph Studio to experience the D3.js visualization. Leave us a comment below or head over to our [forum](https://discuss.nebula-graph.io/) if there are any questions.

> _Hi, I'm Nico, the front-end engineer at NebulaGraph. I'm interested in data visualization and would like to share my experiences in this regard. Hope my post is of help to you. Please let me know if you have any ideas about this. Thanks!_

## You might also like

- [Automating Your Project Processes with Github Actions](https://nebula-graph.io/posts/github-action-automating-project-process/)
- [How to Reduce Docker Image Size](https://nebula-graph.io/posts/how-to-reduce-docker-image-size/)


NebulaGraph Studio has adopted D3-force directed graph to visualize data connections. This post has shared two ideas to optimize the force-directed graph layout.

D3-Force Directed Graph Layout Optimization in NebulaGraph Studio


![Integrating Codecov Test Coverage With NebulaGraph](https://user-images.githubusercontent.com/57335825/79863885-17fa2e80-840b-11ea-86a7-648626742972.png)

A solid testing strategy is a key point to the successful adoption of agile development. Test coverage is a metric used to measure how much of the source code of a program is executed by running a set of tests. It helps developers to identify the code in their application that was not tested.

Ideally, tests against software should define all behaviors of the software. However, this is rarely realized. That is how test coverage comes into play.

Test coverage is a visual measurement used to express which lines of code were executed by a test suite. This helps the software developers too locate where they should write new test cases to cover their source code completely.

In this post, we will introduce to you how NebulaGraph, a distributed graph database, performs test coverage.

# Collect Information for Test Coverage

NebulaGraph is mainly developed in C++. And it supports most Linux environments and gcc/clang compilers. Thus through the support provided by these tool chains, we can easily perform coverage analysis in NebulaGraph.

Both gcc and clang support gcov-style test coverage, which is also very simple to use. The main steps are as follows:

1. Add compilation option `--coverage -O0 -g`
1. Add link option `--coverage`
1. Run the test
1. Integrate the test coverage report with lcov. For example `lcov --capture --directory . --output-file coverage.info`
1. Filter out coverage data for  files that we aren't interested in. For example `lcov --remove coverage.info '*/opt/vesoft/*' -o clean.info`


By now the test coverage information has been successfully collected.

Next, you can generate a report through a tool such as genhtml and view the test coverage through your browser. See an example below:

![test coverage report sample](https://user-images.githubusercontent.com/57335825/79864195-a078cf00-840b-11ea-8f2a-9901d120451b.png)

But generating reports manually is very tedious. It is not automatic, and you have to perform such operations manually every time in the continuous development process. So the common practice is to set up code coverage for tests in CI and integrate with the third-party platforms (such as codecov, coveralls).

By doing so, developers do not have to care about the collection and report of test coverage, they only need to check the test coverage on the third-party platforms after committing their code.

What's more, some third-party tools support integrating the test report with pull requests, which is very convenient.

# Integrate with CI Github Action

There are many mainstream CI platforms, such as Travis, azure-pipelines and GitHub Action. NebulaGraph selects GitHub Action. In our previous post [_Automating Your Project Processes with Github Actions_](https://nebula-graph.io/posts/github-action-automating-project-process/), we have introduced GitHub Action in detail, you may take a look at it to gain some basic understanding about Action.

Compared with other CI platforms, GitHub Action integrates better with GitHub and supports many operating systems and CPUs.

Furthermore, Action is powerful, simple and easy to use. The following example config a test coverage job in the workflow file:

```yaml
- name: CMake with Coverage
  if: matrix.compiler == 'gcc-9.2' && matrix.os == 'centos7'
  run: |
    cmake -DENABLE_COVERAGE=ON -B build/
```

We can see that here we manage the coverage-related compilation options introduced above through a cmake option, which easily enables and disables the collection of coverage information.

For example, developers usually turn off coverage during development process, compilation, and testing to avoid the extra cost of compiling and running tests.

```yaml
- name: Testing Coverage Report
  working-directory: build
  if: success() && matrix.compiler == 'gcc-9.2' && matrix.os == 'centos7'
  run: |
    set -e
    /usr/local/bin/lcov --version
    /usr/local/bin/lcov --capture --gcov-tool $GCOV --directory . --output-file coverage.info
    /usr/local/bin/lcov --remove coverage.info '*/opt/vesoft/*' -o clean.info
    bash <(curl -s https://codecov.io/bash) -Z -f clean.info
```

The piece of code here collects, merges and uploads the test reports to the third-party platform. This has been described in detail in the previous section. The operation of CI is shown in the following figure:

![CI operation](https://user-images.githubusercontent.com/57335825/79864311-d9b13f00-840b-11ea-88ad-30b5f1fca4a9.png)

# Test Coverage Platform Codecov

NebulaGraph selected Codecov, a test result analysis tool, as its test coverage platform. For GitHub Action, it mainly executes the above  script in CI and uploads the produced data to Codecov.

First sign up/in on [Codecov](https://docs.codecov.io/), click 'Add new repository', choose the repository we want to apply the coverage for, and get an accessible token.

Use this token in your API requests, the coverage files can be uploaded to Codecov Details on API see the [Upload Doc](https://docs.codecov.io/reference#upload).

In addition to uploading reports, there are also APIs that list the pull requests or commits. You can customize your automated tools based on them.

Then you can view the test coverage information. For example, you can see NebulaGraph test coverage [here](https://codecov.io/gh/vesoft-inc/nebula).

Moreover, you can check the test coverage in different directories through the pie chart:

![test coverage result chart](https://user-images.githubusercontent.com/57335825/79864433-0c5b3780-840c-11ea-91e1-22c544983c68.png)

You can also open a specific file to see which lines are covered or not:

![test coverage results by code lines](https://user-images.githubusercontent.com/57335825/79864551-39a7e580-840c-11ea-9c9d-5fb3c37a2a50.png)

However, we use a CLI tool provided by Codecov instead of the APIs.

For example we use `bash <(curl -s https://codecov.io/bash) -Z -t <token>- f clean.info` to upload reports. The token here is the authentication token provided by Codecov. Set it as an environment variable `CODECOV_TOKEN` instead of inputting the plaintext token here.

Through the above operations, you can not only view the test coverage of your project on Codecov, but also see the coverage change if this pr is merged, which is convenient for gradually increasing test coverage.

Finally, you can also add the Codecov coverage badge in your README file just like [NebulaGraph](https://github.com/vesoft-inc/nebula).

![add test coverage result to your repo](https://user-images.githubusercontent.com/57335825/79864632-50e6d300-840c-11ea-8a79-c7ad53bdd03f.png)

# You might also like

1. [NebulaGraph Architecture — A Bird’s View](https://nebula-graph.io/posts/nebula-graph-architecture-overview/)
1. [Automating Your Project Processes with Github Actions](https://nebula-graph.io/posts/github-action-automating-project-process/)
1. [Compiling Trouble Shooting: Segmentation Fault and GCC Illegal Instruction](https://nebula-graph.io/posts/segmentation-fault-gcc-illegal-instruction-trouble-shooting/)

> _Hi, I'm Shylock. a software engineer at NebulaGraph. Hope my post is of help to you. Please let me know if you have any ideas about this. Thanks!_

In this post, we will introduce to you how NebulaGraph, a distributed graph database, performs test coverage with Codecov.

Integrating Codecov Test Coverage With NebulaGraph

![How NebulaGraph Uses Jepsen to Detect Data Consistency Issues in Raft Implementation](https://user-images.githubusercontent.com/57335825/79313524-0c9e9300-7f33-11ea-85cd-1fbc5407614c.png)

Data consistency is a global issue for all distributed systems. NebulaGraph is no exception as a distributed graph database.

Thanks to the separation between query and storage layers, NebulaGraph only exposes simple kv interfaces in the storage layer. Using RocksDB as a backend kv library, NebulaGraph [ensures strong data consistency](https://nebula-graph.io/posts/nebula-graph-storage-engine-overview/) among multiple replicas via the Raft protocol.

Although raft is meant to be more understandable than Paxos, the practice for it in a distributed system is tricky.

Another challenging problem is how to test a Raft-based distributed system. Currently, NebulaGraph verifies data consistency with Jepsen. In our previous post [Practice Jepsen Test Framework in NebulaGraph](https://nebula-graph.io/posts/practice-jepsen-test-framework-in-nebula-graph/), we have introduced in detail how Jepsen test framework works, you may take a look at it to gain some basic understanding about Jepsen.

In this post, we will explain how to verify data consistency of the distributed NebulaGraph kv stores.

# Strong Data Consistency Explained

Let's start with the definition of strong data consistency.  Quote the book [_Designing Data-Intensive Applications_](https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable-ebook/dp/B06XPJML5D/ref=sr_1_1?dchild=1&keywords=Designing+Data-Intensive+Applications&qid=1586310740&sr=8-1):

> _In a linearizable system, as soon as one client successfully completes a write, all clients reading from the database must be able to see the value just written._

The basic idea of strong data consistency or linearizability is to make a distributed system appear as if there were only one copy of the data, and all operations on it are atomic. Any read request from any client gets the latest data written to the system.

# Linearizability Verification with Jepsen

Take Jepsen timeline test as an example. In this test, we use the **single-register** model. Clients can only perform read or write operations on the register. All operations are atomic and there is no intermediate state.

Suppose there are four clients sending requests to the system at the same time. Each bar in the figure indicates a request made by a client, where the top of a bar is the time when the request was sent, and the bottom of a bar is when the response was received by the client.

![jepsen test](https://user-images.githubusercontent.com/57335825/79217801-043a4f80-7e82-11ea-85fc-a69c0518790d.png)

From the client's point of view, any request will be processed at any time between the time when it was sent and the time when the response was received. Seen from the figure above, the three operations, write 1 by client 1 (i.e. 1: write1), write 4 by client 3 (i.e. 3: write4), and read 1 by client 4 (i.e. 4: read1), overlapped in processing time. However, we know the real request processing sequence by the system via the responses received by different clients.

Here's how:

The initial value is empty yet client 4 gets 1 for its read request, which indicates that the read operation by client 4 must have begun after the write 1 operation by client 1 and that write 4 by client 3 must have happened before write 1, or the older value 4 would be returned for the read request by client 4. Thus the sequential order of the three operations is write 4 -> write 1 -> read 1. 

Although seen from the above figure, request read 1 is sent first, it's the last one processed. Later operations do not overlap in time, that is to say, they are sent sequentially. Read request by client 2 returns the last write value 4. 

The entire process does not violate the rules of strong data consistency, so the verification is passed.

If the read request by client 3 returns the older value 4, then the system is not strongly consistent. Based on the previous analysis, the last successful write value is 1. If client 3 gets the stale value 4, then linearizability of the system is broken.

Actually, Jepsen verifies strong data consistency of distributed systems with similar algorithms.

# Data Consistency Issues Found with Jepsen Test

The following gives an example of how a request is handled in a Raft group (three replicas) in NebulaGraph for better understanding of data consistency.

Read request processing is relatively simple as only the elected leader server can interact with the client. The leader, after checking it's still the leader, fetches the corresponding result from the state machine and sends back to the client.

Write request processing is a bit more complex. The steps are shown in the Raft Group Figure below:

1. The leader (the green circle) receives requests sent from the client and writes them into its WAL (write ahead log).
1. The leader sends the corresponding log entry in the WAL to its followers and enters the waiting phase.
1. The follower receives the log entry and writes it into its own WAL,  then returns success.
1. When at least one follower returns success, the leader updates the state machine and sends responses to the client.

![Raft groups](https://user-images.githubusercontent.com/57335825/79217666-cccba300-7e81-11ea-81ac-b55cd84cbfcc.png)

Now we will show you some NebulaGraph Raft implementing issues found in Jepsen test with examples below.

## Data consistency example 1: Stale value returned for read requests 

Shown in the above figure, A, B, C form a three-replica raft group. The circles are the state machines (we assume them as a single-register for simplicity) and the box stores the corresponding log entry.


- At the initial state, all the state machines in the three replicas are 1, the leader is A and the term is 1.

- After request write 2 is sent by the client, the leader processes it based on the previously described process, informs the client of a successful write, and  is then killed.

- Then C is elected as leader of term 2. However, since C has yet applied the log entry of write 2 to the state machine (the value is still 1 at the time being), 1 will be returned if C receives requests from the client. This absolutely violates the definition of strong data consistency because 2 has been successfully written while a stale value is returned.

Such an issue occurs after C is elected as leader of term 2. It is necessary to send heartbeat to ensure the log entry of the previous term is accepted by the quorum for the cluster. No read is allowed before the heartbeat success. Otherwise stale data is likely to be returned. Please refer to Figure 8 and Session 5.4.2 in this[Raft paper](https://raft.github.io/raft.pdf) if you want more details.

## Data consistency example 2: Leader ensures itself as a leader

Another issue we have found via Jepsen test is: how does a leader ensure itself still the leader? Such problems occur upon network partition. When the leader can't connect to the quorum due to network failure, if read is still allowed, a stale value is likely to be returned.

To avoid such cases, we have introduced the leader lease concept.

When a node is elected as leader, it needs to send heartbeats to other nodes periodically. If the heartbeat is received by most of the nodes successfully, a lease for some time is obtained and there will not be any new leaders during the term. Thus the data recency is guaranteed on the node and read requests are processed normally during this time.

![image](https://user-images.githubusercontent.com/57335825/79217878-259b3b80-7e82-11ea-9e51-dbe929e11753.png)

Being different from TiKV's process, NebulaGraph doesn't consider _heartbeat interval * some coefficient_ as the lease time because of the drift of the clock skew of different nodes.

Rather, we store 1) T1 = heartbeat intervals and 2) T2 = the time cost by the last successful heartbeat or   appendLog, and    get the lease duration by _(T1 - T2)_.

When network is partitioned, the old leader still processes read requests (write requests are failed due to the isolation) during the lease time in spite of isolation. When the lease is expired, both read and write requests would fail. A leader election takes place when a follower node times out while waiting for a heartbeat from the leader node.

# Conclusion

You need long-term pressure tests and failure simulations to discover problems in a distributed system. Jespen is a great helper for its ability to verify distributed systems under various failures.

In the future, we will introduce other chaos engineering tools to verify data consistency in NebulaGraph. This way we can continuously improve performance while ensuring high data reliability.

> Hi, I’m critical27, a software engineer in NebulaGraph. Currently I am involved in the storage engine development. I hope this article is helpful and let me know if you have any thoughts and suggestions. Thanks!_

# You might also like

- [Practice Jepsen Test Framework in NebulaGraph](https://nebula-graph.io/posts/practice-jepsen-test-framework-in-nebula-graph/)
- [An Introduction to NebulaGraph's Storage Engine](https://nebula-graph.io/posts/nebula-graph-storage-engine-overview/)

In this post, we will explain how to verify data consistency of the distributed NebulaGraph kv stores with the Jepsen test framework.