# Financial Fraud Detection: One of the Best Practices of Knowledge Graph

Hello, I'm Qingxin Kong from Bangsun Technology. I'm glad that I have this opportunity to share the topic **Financial Fraud Detection: Best Practices of Knowledge Graph **with NebulaGraph community members.

### Background: Problems with traditional fraud detection solutions

Financial frauds remain a hot topic in recent years. There's a new trend emerging that fraudsters are organized and ganged up on. It is difficult to verify whether the customer's information provided is fake or not. Such inconsistent information makes it difficult to identify the real frauds and leads to a lot of problems. The bank may control the passing rate by raising the requirements to handle loan applications due to the uncertainty of the customer's information, which is effective but brings losses.

The 2022 government work report said that financial inclusion will be further expanded and the customer base will include rural customers, which means there will be "white accounts" having no credit reports. Traditional fraudulent behaviors include fake credit reports, commission agencies, and collusion. 

Some agencies focus on a few banks for credit card applications or loans. Those agencies know the bottom line and red line of the banks. As long as the bottom line is not touched, they can successfully apply for loans. Those agencies may even know how to make the loan amount higher.

Traditional fraud detection solutions rely heavily on expert rules, that is, risk control experts set a batch of rules through experience and then adjust the trigger conditions to find a business balance between the pass rate and rejection rate. It costs little to forge things like ID numbers and bank card flows. There are no good means of control for mass and high-volume frauds.

### Empowering graphs: Paths to intelligent fraud detection solutions
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/financial-fraud-detection-one-of-the-best-practices-of-knowledge-graph/figure1.png)

Let's take a look at the development path of intelligent fraud detection solutions. 

The first step is to build expert rules, but it is difficult to cover all risks through expert experience. The second step is to build a machine learning model to prevent and control the overall risk and then use expert rules to help the model. Expert rules and machine learning models well complement each other. However, based on the current experience, we may face lots of problems when building the machine learning model. The first reason is insufficient data and the second is the limited growth of the sample data. Because there are probably be 6~7 ways to apply for loans or credit cards, manual inspection is required in the end, so the data growth is very low. There will be a case below to explain the problem.

Therefore, we need to use relational graphs to crack the problems of information inconsistency. Both group and agency frauds have a batch of application behaviors. We can use the visualized graphs to recognize fraud samples. In this way, we may capture dozens to hundreds of fraudulent applications. It is also a very effective way for sample replenishment. Graphs can also be used for consistency checks. The multiple sources of information prove the information inconsistency. Graphs are displayed from the dimension of relations which helps us to do secondary verification. For example, multiple applications sharing the same information like the same phone number and IP address are considered group fraud.

To detect these group frauds, we use graphs that help us find rules of relations. These rules are complementary to the original expert rules. On the other hand, we can extract graph information to optimize machine learning patterns which increases the pattern's ability to detect group frauds and label more suspicious objects. Therefore, the expert rules, machine learning models, and graphs construct an effective ecological closed loop and complement each other to detect frauds. 

In terms of the specific applications of relational graphs, I've listed the following applications:

**_Application Graph_**: An example is using graphs to detect fraudulent applicants for loans and credit cards. One characteristic of a group is that information is shared by the group members. From the visual queries of graph data, a lot of correlations will be found. The more common information includes identity card, cell phone number, device fingerprint, e-mail address, etc. We use this information to explore relationships among these data, and then visually analyze the specific fraudulent techniques.**_‍_**

**_Transaction Graph_**: For multi-level transfer relationships, we mainly focus on the final flow of funds and who are the beneficiaries. The recent gambling fraud issue is the main focus of concern for financial supervisors, which is also one of the main directions we focus on. We have many customer cases. Through the list of suspects provided by the supervision of our internal rule models, suspicious transactions can be detected. In this process, we use historical flow data to explore more layers of data starting with the collected accounts. We may also use the graph exploration feature or a graph platform to analyze existing data for incremental prevention and control. 

**_Enterprise Graph/Internal Control Graph_**: Nowadays, there are interconnections between two or more enterprise businesses. The impact of external risks on the enterprise is increasing. Using graphs can outline the whole picture of risks for an enterprise. Combined with the introduction of external risks, the impact of external risks on the enterprise can be predicted as early as possible. Internal control graphs are more often used for detecting ethical or operational risks, such as misuses or failure to do certain operations in accordance with rules and regulations. Moral risks occur more often in unusual financial transactions between employees and enterprises, such as illegal fundraising or misappropriation of funds. The prevention core is to use graphs to check the actual control accounts of employees. Because bank employees are well acquainted with the bank's business. Few employees take their own accounts for some crimes. The identification of these control accounts can be explored by the means of graphs. For example, the accounts of relatives. We have a case of a joint-stock bank. In this case, the target was located in the account of the employee's girlfriend, because from visual graphs we found that many funds actually flowed to the girlfriend's account. It is very difficult to detect a case like this by traditional means.

**_Money Laundering Graph_**: The money laundering problem has existed for a long time. There are many derivative money laundering ways in different periods. Popular ways include the underground money bank and the "order sneaking" platform. Criminals hide their money on the platform where order-sneaking customers' daily consumption transactions are normal, however, there is actually a fixed pattern on visual graphs to detect cases like this. So how creating business relations with graphs and making graph rules to perform real-time monitoring and detections have become an important task for Bangsun, taken as a graph technology platform, to empower business developers. We will talk about it later. 

**_Vehicle Insurance/Ops Graph_**: For the vehicle insurance/ops field, we combine the initialization network with the design of vertices and edges, then query for a subgraph where the data is relatively appropriate and related, and then combine the experience of business experts to perform some exploratory and analytical work.

### **Use case 1: Flow of funds**
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/financial-fraud-detection-one-of-the-best-practices-of-knowledge-graph/figure2.png)

Here, some specific application scenarios are also briefly listed. The first one is the flow of funds after the loan is approved.

We can focus on business and consumer loans through graphs, and track whether the approved loans flow into the housing market or the stock market. In this process, we will use the penetration ability of graphs to calculate the amount and proportion of transferred funds. Thereafter, we can easily find risks and violations and provide a basis for investigation. The regulators currently follow the principle of "three approaches and one guideline" for internet credits, so we establish real-time risk control through graphs and focus on the fund flows.

Next, we can show the specific methods. For example, loans flow directly into the blacklisted accounts or their own accounts are transferred to an account not under their name, but the account may be a controlled account. The account is used for investment and financial management. These can be analyzed with graphs and the specific pattern of the funds because the fund flows will form a very obvious distribution or pattern on visual graphs.

This pattern is difficult to portray with the previous expert rules, because the expert rules can only find a one-dimensional relationship, up to two-dimensional relationships, while the graphs are more adept at multi-dimensional relationship detecting. In massive transaction structures, we can also find some patterned structures. For example, the above figure shows 4-5 vertices, which involve some centralized transfer in, decentralized transfer out, and chain transaction structures like tree branches and ant nests, which are all abnormal funding patterns. Through visual graphs and combination with business knowledge, in many cases, we can see that it is problematic at a glance. As for where the specific problem is, we can check it through some functions provided by the graph, such as the k-degree query or the discovery algorithm of important nodes.

### **Use case 2: Insurance frauds** 
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/financial-fraud-detection-one-of-the-best-practices-of-knowledge-graph/figure3.png)

The second use case is frauds in insurance, such as vehicle insurance or health insurance. The primary vehicle insurance issues are collusive claim frauds. By monitoring and associating shared identity information, it is easier to find group members and excludes irrelevant data. As for health insurance, we generally focus more on the improper relationship between doctors and patients or providers. Once there is a large number of discrete values on a drug or disease, it represents fraud. This is the common way to detect individual fraud in the past. Now the group fraud problem is severe. Group fraudsters have a shift in thinking that dozens of accounts buy the same drug through individual payment accounts. There are also aggregation accounts and many other commonalities. Combined with graph analysis and algorithms, those group frauds can be detected.

### **Summary: Advantages of graph fraud detection**
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/financial-fraud-detection-one-of-the-best-practices-of-knowledge-graph/figure4.png)

The advantages of graph fraud detection mainly include the following four aspects.

Association analysis: This means that people take the initiative to explore and discover suspicious features, and then use graphs to make some visual associations. Business developers take the initiative to use graph tools for visual exploration and then combine the features of different business areas, such as money laundering, credit card applications, and loan applications to find some suspicious points. The suspicious points are certainly different in different areas, but the business developers who often use graph tools have a certain sensitivity to such suspicious points.

Graph rules: This aspect is more for incremental monitoring. Unlike the previous expert rules, graph rules use the means of graph exploration associated with knowledge comparison to draw some conclusions. We conclude graph rules from the dimension of relationships, and then the system uses the rules to detect risks.

Pattern analysis: We recently called the same model, because many fraud patterns have a solidified pattern, such as cash-out from credit cards. After analyzing these solidified patterns, we convert the pattern in a graph query language into a graph library for a traversal query. Compared to the graph rules introduced above, it is a method for detecting existing data.

Community analysis: As mentioned above, insufficient data is lacking for modeling graph data, the community analysis is actually a very good feature to discover fraudulent groups by algorithms and combination with data labels.

The above four graph fraud detection advantages can help us detect frauds in real-time, on time, and afterward. They are both the main graph functions for anti-frauds and are considered our strengths. These advantages are currently applied by about dozens of our customers.

This is all we have for today's article! Please look forward to the next topic Introduction of Bangsun Knowledge Graph Platform and Practical Cases of State-owned Banks.

‍

A knowledge Graph specialist from Bungsun shares his best Practices of Knowledge graph in the Financial Fraud detection field.

Financial Fraud Detection: One of the Best Practices of Knowledge Graph

Real-time recommendation is a very popular topic in the retail industry. The goal of real-time recommendation is to provide personalized recommendations for users immediately after they search for products on e-commerce websites. It is a challenging task since the user has not yet decided which product they want to buy, so it requires a deep understanding of their preferences and behaviors.

In order to achieve this goal, real-time recommender systems need to be able to process big data at near real-time speed. Real-time systems are often used as part of e-commerce websites or mobile apps, where they can recommend products based on what the user is currently doing. For example, if an e-commerce site is selling shoes, it might want to show the user different styles of shoes when they are looking at the "men's sneakers" page.

Graph technology is a good choice for real-time recommendation. It has the ability to predict user behavior and make recommendations based on it. Graph databases like NebulaGraph provide a flexible data model that allows you to represent any kind of relationship between entities. This includes not only the typical "product" and "user" relationships, but also any other relationships that are important to your application. For example, you can use a graph database to represent complex groupings of users (such as "friends") or categories of products (such as "books"). As long as there's an edge between two entities, you can use graph technology.

The ability to represent any kind of relationship also makes it possible for graph databases to be more expressive than relational databases. For example, if you have a large number of users who share similar interests but belong to different groups, then in relational databases this might mean having a separate table for each group and then joining them together into one big table at query time. In contrast, in a graph database, it would be possible for each user to have multiple edges representing their interests in different groups.

## **What is a graph database?**

A graph is known as a diagram that illustrates a relationship between two things. Thus, a graph database can be assumed to be a database to understand relationships between data.

Graph databases are unlike traditional Relational Database Management Systems (RDBMSs) most people are familiar with. RDBMSs house data in a table of columns and rows with little or no relation to each other.

Graph databases are fundamentally designed with a focus on the relationship between data sets. So, they require more intense and specialized processing capabilities. As a result, graph databases like NebulaGraph use sophisticated design and architecture.
 ![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/use-cases-of-graph-databases-in-real-time-recommendation/figure1.png)


Graph databases are designed to uncover important relationships between many big data sets. In other words, it can connect the dots between multiple datasets that might otherwise just be sitting useless in silos. And graph databases are already widely used for such purposes.

## **What is a real-time recommender system?**

recommender systems are used to predict customer preferences and provide them with products or services that they might like. It helps companies reduce costs and increase revenues by providing personalized recommendations for their customers.

Recommender systems are used in a variety of industries including retail, media and entertainment, travel, and the public sector.

For example, Netflix uses an algorithm to recommend movies based on what you’ve watched previously. Amazon uses recommendations to help shoppers find new products they might be interested in purchasing. And Facebook uses its own algorithm to show you relevant ads based on your interests and likes.

## **Why should you use NebulaGraph for real-time recommendation?**

NebulaGraph is a highly performant linearly scalable graph database available for use via a shared-nothing distributed model. In several real production environments, data mining and technical performance has been proven to beat the performance of competing graph databases by multiple times over.

The goal behind NebulaGraph is to unleash the power of exponentially growing connected data.

NebulaGraph securely processes data sets of at least twice the size and twice as fast as any competing graph databases

*   It is the only database that can store and process billions of data points with trillions of relational connections
*   NebulaGraph is designed for scalability and recovery without disruption, ensuring the best business continuity available
   ![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/use-cases-of-graph-databases-in-real-time-recommendation/figure2.png)

## **How graph technology is used in recommendation**

Recommender systems aim to provide users with personalized recommendations based on their tastes and preferences. The characteristics of a user can be represented by their preferences for movies and books, for example, or their shopping habits on an e-commerce website. There are two main types of recommender systems: 1) collaborative filtering methods and 2) model-based methods.

A collaborative filtering method, such as KNN, can predict the movie rating without knowing the attributes of the movies and users. To address this challenge, the graph factorization approach combines the model-based method with the collaborative filtering method to improve prediction accuracy when the rating record is sparse.

Graph factorization methods have been widely used in many online recommender systems. Graph factorization is a graph-based model that can be used to represent user preferences as well as the relationship between users, items, and attributes. The goal of graph factorization is to extract the latent features from user ratings and recommendations so that these features can be used to predict users' preferences in an unsupervised manner.

Graph factorization is done by breaking down the original dataset into smaller datasets or clusters. This process can be done using graph databases because they are designed to support highly connected data structures and relationships between data points.

## **How to build a recommender system using NebulaGraph?**

If you want to build your own recommender system, there are many different approaches you can take. One of the most popular is using a graph database like NebulaGraph. This section will walk through how to build a recommender system using a graph database.

**Define the data model**

In this step, we define the data model for our recommender system. The first thing we need to define is what type of data can be recommended. For example, we want to recommend movies so we will use movies as our entity. The second thing is what kind of information do we need about a movie? For example, if you are building a recommender system for books, maybe you need to know what genre it belongs to or what language it was written in. But in our case, all we need is the title, the year of release, the genre, and the country. So our movie entity is defined like this:

‍
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/use-cases-of-graph-databases-in-real-time-recommendation/figure3.png)

‍

**Define relationships**

The next thing we need to do is define what type of relationship exists between movies and users (or people). There are two types of relationships that exist between movies and users: user likes movie and user has watched movie (or user has not watched movie). In this example, we will simplify it to just user likes movie.

Now imagine two users James and Kelsey, who are both fans of Sci-Fi Movie A, which has a genre of Science fiction. James also likes Sci-Fi Movie B, but Kelsey hasn’t watched it, so we don’t know whether Kelsey likes it or not.
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/use-cases-of-graph-databases-in-real-time-recommendation/figure4.png)

But using this graph modeling of data, we may easily find out that Kelsey may like Sci-Fi Movie B. The recommender system would recommend Sci-Fi Movie B to Kelsey because James — who likes the same things as Kelsey — likes Sci-Fi Movie B.
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/use-cases-of-graph-databases-in-real-time-recommendation/figure5.png)

**Making recommendations**

There are a few graph algorithms that you can use to make recommendations within a graph database.

**The PageRank algorithm**: The [PageRank algorithm](http://pi.math.cornell.edu/~mec/Winter2009/RalucaRemus/Lecture3/lecture3.html) is used to rank web pages in search results. The purpose of this algorithm is to determine which web pages should be displayed first when someone searches Google or any other major search engine.

The basic idea behind PageRank is that if you link to another page, then that page is more important than your own page. So if you are linking to a page about shopping for shoes, and someone else links to your shoe shopping site, then that person thinks your site is more important than it really is.

If everyone is linking to the same sites, then the people who link most get more weight.&nbsp;

Continuing with the movie analogy, while watching movies one can either carry on watching movies of a similar genre (and hence follow their most expected journey), or skip to a random movie in a totally different genre. It turns out that this is exactly how Google ranks websites by popularity using the _PageRank_**_ _**algorithm.

The popularity of a website is measured by the number of links it points to (and is referred from). In our movie use case, the popularity is built as the number hashes a given movie shares with all its neighbors.

**Collaborative filtering:** Collaborative Filtering (CF) is a method for recommender systems based on information regarding users, items, and their connections. Recommendations are made by looking at the neighbors of the user at hand and their interests. Since they are similar, the assumption is made that they share the same interests.

Read more: [Build a Recommendation Engine With Collaborative Filtering](https://realpython.com/build-recommendation-engine-collaborative-filtering/)

## Conclusion

In conclusion, it's worthwhile to use graph databases to build real-time recommender systems.&nbsp; Since graph databases are well designed for representing the relationship among users and products in the recommender system, they can be used to build real-time recommendation systems. Graph databases' support of powerful data analysis makes it useful in constructing recommender systems that need to take into account users and products with different preferences.

Why graph database is a good choice for recommendation engines? How to implement a recommender system using graph technology? How graph algorithms like PageRank and Collaborative filtering are used in real-time recommendation?

Use cases of graph databases in real-time recommendation

I would like to share my opinionated reference data infrastructure stack with some of those best open-source projects with modern ETL, Dashboard, Metadata Governance, and Data Lineage Management.

Introducing a metadata management system that uses NebulaGraph as graph database

In this article, I will try to walk you through four different ways that you can make NebulaGraph and Apache Spark work together: Spark Connector, Nebula Exchange, Nebula Algorithm, and PySpark.

4 different ways to work with NebulaGraph in Apache Spark

Wordle is going viral these days. So is Handle, the Chinese Wordle. This post explains how to crack Chinese Wordle using knowledge graph.

How I cracked Chinese Wordle using knowledge graph


![NebulaGraph 2.0 Performance Testing](https://user-images.githubusercontent.com/57335825/122524996-5a3cf100-d04b-11eb-9587-beed62f97e65.png)

This article is shared by Fanfan from the NebulaGraph community. It is about his practice of the performance testing on NebulaGraph 2.0 and optimizing the data import performance of Nebula Importer. In this article, "I" refers to the author.

## 0. Background

I did some research on NebulaGraph and did tests to evaluate its performance. During the process, I got a lot of help from the NebulaGraph team. I would like to give thanks to them.

In this article, I will introduce my test process, hoping that it could help you get inspired in the performance testing on NebulaGraph. If you have other ideas, please share them with me. 

## Deploy NebulaGraph Cluster

In the test, four machines were used to deploy a NebulaGraph cluster: Machine 1, Machine 2, Machine 3, and Machine 4. 

Here is the hardware configuration of these machines:

- 1：meta，storage
- 2：storage
- 3：storage
- 4：graphd

NebulaGraph was installed with the RPM package. For more information, see [NebulaGraph Database Manual](https://docs.nebula-graph.io/2.0.1/4.deployment-and-installation/2.compile-and-install-nebula-graph/2.install-nebula-graph-by-rpm-or-deb/). Other tools were also needed: [nebula-importer-2.0](https://github.com/vesoft-inc/nebula-importer) and [nebula-bench-2.0](https://github.com/vesoft-inc/nebula-bench). Their source code was downloaded and compiled on every machine.

## Import Data

The data structure for the testing is composed of 7 tags and 15 edge types. The dataset is simple and not large in size, about 34 million records in total. It needed to be preprocessed to become vertex or edge tables.

A graph space was created, of which `vid` was set to 100, `replica_factor` was set to 3, and `partition_num` was set to 100.

### Optimize Data Import Performance of Nebula Importer

At the beginning of the test, I used [Nebula Importer](https://github.com/vesoft-inc/nebula-importer) to import data to NebulaGraph without any parameter configuration modifications. The speed was only about 30 thousand records per second. It was unacceptable.

To optimize the import performance, I read the documentation of Nebula Importer and found that only three parameters needed to be adjusted for optimization: `concurrency`, `channelBufferSize`, and `batchSize`.

I tried modifying the configuration of these parameters, but it did not bring a significant performance improvement. According to the feedback from the NebulaGraph team, I made the following changes to the YAML file:

```
concurrency：96 # the number of CPU cores
channelBufferSize：20000
batchsize：2500

```

After the changes, the import speed is increased to 70 or 80 thousand records per second. Great. I tried larger values, but the nebula-graphd server crashed. Therefore, these parameters should be set to a large value, but not too large.

The next check was on the disk and the network. Originally, mechanical drives and 1000 Mb/s network were used in my environment. I changed them to SSD and 10000 Mb/s network and the import speed was doubled to 170 thousand records per second. Therefore, hardware matters.

My last check was on the data. When I created the graph space, `vid` was set to a large value. How about a smaller value? The change failed because some vertices do need VIDs of the specified `vid` length. And then I turned my eyes to `partition_num`. According to NebulaGraph Database Manual, it must be set to a value 20 times of the disk capacity, so I set it to 15. The change worked. The import speed was increased to 250 thousand records per second. So far, I was satisfied with the import performance. It may be improved further with more modifications, but I think now it is acceptable.

### Summary

* `concurrency` must be set to the number of CPU cores. `channelBufferSize` and `batchSize` should be set to values that must be large enough to improve the performance but not exceed the load of the cluster.
* SSD and 10000 Mb/s network are necessary
* `partition_num` should be set to an appropriate value
* The VID length, the number of properties, and the number of nebula-graphd servers may affect the performance. This is only my guess because I have not done a test on them

## Pressure Testing

I picked a business indicator to do the pressure testing. 

Here is the query statement for the business indicator：

```
match (v:email)-[:emailid]->(mid:id)<-[:phoneid]-(phone:phone)-[:phoneid]->(ids:id) where id(v)=="replace" with v, count(distinct phone) as pnum,count(distinct mid) as midnum,count(distinct ids) as idsnum , sum(ids.isblack) as black  where pnum > 2 and midnum>5 and midnum < 100 and idsnum > 5 and idsnum < 300 and black > 0 return v.value1, true as result
```

This statement aimed to do 3-hop expansion and the conditional judgement. It queried two to four million concentrated data records. 

Nebula-bench needed to be modified as follows: Open the `go_step.jmx` configuration file of JMeter, set `ThreadGroup.num_threads` to the number of CPU cores, and then modify other parameters such as the loop associated parameters. The nGQL statement must be modified according to the actual situation. The variables in the nGQL statement must be replaced with `replace`.

Because the tested data was relatively concentrated, the speed is about 700 records per second. If the data on all the nodes are tested, the speed can be increased to higher than 6000 records per second. The concurrency performance looks good, and so does the query latency, the maximum of which is 300 ms. 

In my environment, only one node was used, so I wanted to add one more nebula-graphd server to do the check of the concurrency performance. After I started a new nebula-graphd server, it turned out that no improvement occurred.

At that point, NebulaGraph 2.0.1 was released, so I deployed it on the machines and imported data. On the new cluster, three nebula-graphd servers were deployed, the performance was tripled, the speed for concentrated data was higher than 2100 records per second, and the speed for data on all the machines was about 20 thousand records per second. It was weird. I guess it was caused because no `BALANCE` or `COMPACT` command was run after new nebula-graphd server was added. I should give it a try next time.

Additionally, I used Linux commands, but not some specific monitoring tools, to view the database metrics, so I cannot obtain the accurate metrics.

### Summary

*  Before the test, make sure that the `COMPACT` command has been run to ensure the load balance of the cluster.
*  Appropriately adjust the configuration of nebula-storaged by increasing the available threads and the cache memory capacity.
*  The concurrency performance is affected by both data size and data distribution.

## Modify Configuration

In my test, the default configurations of nebula-metad and nebula-graphd were used. Only the configuration of nebula-graphd was modified as follows:

```
rocksdb_block_cache=102400  # The NebulaGraph team recommend that 1/3 memory capacity should be used. In my test, 100 GB was used.
num_io_threads=48 # The number of available threads. It was set to a half of the number of CPU cores.
min_vertices_per_bucket=100 # The minimum number of vertices in a bucket.
vertex_cache_bucket_exp=8 # The total number of buckets was set to the eighth power of 2.
wal_buffer_size=16777216  # 16 M
write_buffer_size:268435456   # 256 M
```

I referred to some posts on the forum and read the source code of NebulaGraph before making the preceding changes to these parameters. These configurations were obtained from my attempts. They are not necessarily accurate. Besides, I did not modify other parameters.

The NebulaGraph team do not recommend users modify the configuration of other parameters. If you are interested in them, read the source code.

## Conclusion

In summary, my test is not professional. I did it to do a test for specific business scenarios and NebulaGraph turned out very well.

I haven't fully understood how to adjust the parameter configuration. I will keep study in the future. If you have more ideas for performance tuning, please share them with me.

Would like to know more about NebulaGraph? Join the [Slack](https://join.slack.com/t/nebulagraph/shared_invite/zt-7ybejuqa-NCZBroh~PCh66d9kOQj45g) channel!


This article is about the practice of the performance testing on NebulaGraph 2.0 and optimizing the data import performance of Nebula Importer.

Community Contribution | NebulaGraph 2.0 Performance Testing

This post introduces how Kuaishou has built their threat intelligence platform based on NebulaGraph database. 

Handling Tens of Billions of Threat Intelligence Data with Graph Database at Kuaishou

Zhou Peng, the senior big data developer at 360 Digitech, has shared their pain points with JanusGraph and why NebulaGraph is a better graph database solution for them.

Graph Database Evaluation ｜ Why 360 DigiTech Migrates from JanusGraph to NebulaGraph


![Import data from Neo4j to NebulaGraph](https://user-images.githubusercontent.com/57335825/105473413-10d4b980-5cd8-11eb-9689-ed380387f1cb.png)

The requirement for real-time data updating and querying increases continuously as the data size is surging. Under such case, Neo4j is sure to fall behind. Since the single-host Neo4j Community version doesn't support scale up, the linear scalability and read/write splitting is not possible. Besides, there are data size limitation for the community version, too. To make things worse, there is the leader node writing performance bottleneck for the causal cluster provided by the Neo4j Enterprise version.

Compared with Neo4j, NebulaGraph features in single leader host writing and read/write scalability owing to its shared-nothing distributed architecture. In fact, NebulaGraph is doing a perfect job with super large data contains hundreds of billions vertices and edges.

In this article, I will show you how to import data from Neo4j to NebulaGraph with the official ETL tool, [Nebula Exchange](https://github.com/vesoft-inc/nebula-java/tree/v1.0/tools). I also cover the import related problems and solutions in this article.

# Deployment Environment

Hardware configuration:
- CPU name：Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
- CPU Cores：40
- Memory Size：376 GB
- Disk：HDD
- System：CentOS Linux release 7.4.1708 (Core)

Software information:
- Neo4j: Version 3.4, five nodes causal cluster
- NebulaGraph：
  - Version: [NebulaGraph v1.1.0](https://github.com/vesoft-inc/nebula/releases/tag/v1.1.0), compiled by the source code
  - Deployment: three nodes cluster deployed on one host
- Exchange：[nebula-java v1.1.0](https://github.com/vesoft-inc/nebula-java/releases/tag/1.1.0)
- Data warehouse:
   - hadoop-2.7.4
   - spark-2.3.1
 
NOTE: Ports allocation when deploying multiple nodes on one host: each Storage Service adds 1 to the customized port number and calls it internally.

# Full and Incremental Import

## Full Import

Before importing data, create schema in NebulaGraph. The schema is based on the original Neo4j data. Note that you can create schema based on your actual business need. So it's likely some properties are `NULL`. Confirm all the properties with your co-workers in case you leave any properties behind. The schema in NebulaGraph is similar to that in MySQL, you can create and alter properties for the schema. All the tag and edge meta data are the same.

1. Create tag and edge in NebulaGraph

```
# Create a graph space with 10 partitions and 3 replicas.
CREATE SPACE test(partition_num=10,replica_factor=3);
# Use the graph space
USE test;
# Create tag tagA
CREATE TAG tagA(vid string, field-a0 string, field-a1 bool, field-a2 double);
# Create tag tagB
CREATE TAG tagB(vid string, field-b0 string, field-b1 bool, field-b2 double);
# Create edge type edgeAB
CREATE EDGE edgeAB(vid string, field-e0 string, field-e1 bool, field-e2 double);
```

2. Modify the configuration file for the Exchange

* At this time, Exchange doesn't support connecting to Neo4j with `bolt+routing`. If you're using causal cluster, choose a node to read data by using the `bolt` method to release the cluster load.
* Our vid in Neo4j is stored as the string type. However, NebulaGraph v1.x doesn't support string vid, so we use uuid() to gender vids.
* partition: the partition number Exchange pulls data from Neo4j.
*  batch: the batch size when importing data into NebulaGraph

```
{
  # Spark relation config
  spark: {
    app: {
      name: Spark Writer
    }

    driver: {
      cores: 1
      maxResultSize: 1G
    }

    cores {
      max: 16
    }
  }

  # NebulaGraph relation config
  nebula: {
    address:{
      graph:["xxx.xxx.xxx.xx:3699"]
      meta:["xxx.xxx.xxx.xx:45500"]
    }
    user: user
    pswd: password
    space: test

    connection {
      timeout: 3000
      retry: 3
    }

    execution {
      retry: 3
    }

    error: {
      max: 32
      output: /tmp/errors
    }

    rate: {
      limit: 1024
      timeout: 1000
    }
  }
  
  # Processing tags
  tags: [
    # Loading tag from neo4j
    {
      name: tagA
      type: {
        source: neo4j
        sink: client
      }
      server: "bolt://xxx.xxx.xxx.xxx:7687"
      user: neo4j
      password: neo4j
      exec: "match (n:tagA) where id(n) < 300000000 return n.vid as vid, n.field-a0 as field-a0, n.field-a1 as field-a1, n.field-a2 as field-a2 order by id(n)"
      fields: [vid, field-a0, field-a1, field-a2]
      nebula.fields: [vid, field-a0, field-a1, field-a2]
      vertex: {
        field: vid
        policy: "uuid"
      }
      partition: 10
      batch: 1000
      check_point_path: /tmp/test
    }
    # Loading tag from neo4j
    {
      name: tagB
      type: {
        source: neo4j
        sink: client
      }
      server: "bolt://xxx.xxx.xxx.xxx:7687"
      user: neo4j
      password: neo4j
      exec: "match (n:tagB) where id(n) < 300000000 return n.vid as vid, n.field-b0 as field-b0, n.field-b1 as field-b1, n.field-b2 as field-b2 order by id(n)"
      fields: [vid, field-b0, field-b1, field-b2]
      nebula.fields: [vid, field-b0, field-b1, field-b2]
      vertex: {
        field: vid
        policy: "uuid"
      }
      partition: 10
      batch: 1000
      check_point_path: /tmp/test
    }
  ]

  # Processing edges
  edges: [
   # Loading edges from neo4j
    {
      name: edgeAB
      type: {
        source: neo4j
        sink: client
      }
      server: "bolt://xxx.xxx.xxx.xxx:7687"
      user: neo4j
      password: neo4j
      exec: "match (a:tagA)-[r:edgeAB]->(b:tagB) where id(r) < 300000000 return n.vid as vid, n.field-e0 as field-e0, n.field-e1 as field-e1, n.field-e2 as field-e2 order by id(r)"
      fields: [vid, field-e0, field-e1, field-e2]
      nebula.fields: [vid, field-e0, field-e1, field-e2]
      source: {
        field: a.vid
        policy: "uuid"
      }
      target: {
        field: b.vid
        policy: "uuid"
      }
      partition: 10
      batch: 1000
      check_point_path: /tmp/test
    }
  ]
}
```

3. Run the import command

```
nohup spark-submit --class com.vesoft.nebula.tools.importer.Exchange --master "local" exchange-1.1.0.jar -c test.conf > test.log &
```

4. Confirm the imported data

```
./bin/db_dump --space=test --db_path=./data/storage/nebula/ --meta_server=127.0.0.1:45500 -limit 0 --mode=stat --tags=tagA,tagB --edges=edgeAB
```
**NOTE**: NebulaGraph 1.x only supports count data with db_dump. In NebulaGraph 2.0, you can use the nGQL statement to count data.

## Incremental Import

Incremental import is done through splitting the self-incremental `id()` for the Neo4j data. When importing the exec item, you need to add `id()` range limitation when running Neo4j Cypher statements. But before this step, stop the data deletion operation. This is because when doing incremental import, if you delete data, Neo4j uses the `id()` repeatedly, which will cause data loss. **If you support writing data from both Neo4j and NebulaGraph, no such problem is to be worried**.

```
exec: "match (n:user) where id(n) >= 300000000 and id(n) < 400000000 return xxx order by id(n)"
```

## Import Related Issues

Issue one: Exchange does not support the escape of special characters such as line break with enter. For example, the string data contains enter, so the insertion operation is failed.

![Neo4j 导入 NebulaGraph 的实践总结](https://www-cdn.nebula-graph.com.cn/nebula-blog/neo4j-nebula-graph-import-best-practice-01.png)

PR：[https://github.com/vesoft-inc/nebula-java/pull/203](https://github.com/vesoft-inc/nebula-java/pull/203) is merged to the exchange v1.0 branch.

Issue two: Exchange doesn't supports importing `NULL` data type. As mentioned previously, you can add properties to some tags or edge types. Other properties that are ignored is `NULL` at this time. So Exchange throws an error when importing.

Modify the `com.vesoft.nebula.tools.importer.processor.Processor#extraValue` configuration, add converted value for the `NULL` type.

```
case NullType => {
  fieldTypeMap(field) match {
    case StringType => ""
    case IntegerType => 0
    case LongType => 0L
    case DoubleType => 0.0
    case BooleanType => false
  }
}
```

# Optimize Import Efficiency

1. Improve importing efficiency by appropriately increasing the partition number and the batch value.
2. If your vid is the string type, use the hash() function. In NebulaGraph 2.0, string type vid is supported.
3. The official suggests modify the spark-submit commit command to `yarn-cluster`. If you don't use yarn, configure to `spark://ip:port`. We use the `spark-submit --master "local[16]"` method to increase spark concurrency. The import rate is raised by 4+ times. In our test (three nodes on one host), the max IO reaches 200-300 MB/s. But there are hadoop cache issues when setting `--master "local[16]"`. You can fix this by add the HDFS configuration `fs.hdfs.impl.disable.cache=true`.

# Conclusion

Although I have encountered some problems when importing, the community has provided fast helps. Many thanks to the community and NebulaGraph staffs. Hope the openCypher compatible NebulaGraph 2.0 come soon.

# References

1. https://nebula-graph.com.cn/posts/how-to-import-data-from-neo4j-to-nebula-graph/
2. https://github.com/vesoft-inc/nebula-java/tree/v1.0
3. https://docs.nebula-graph.com.cn/manual-CN/2.query-language/2.functions-and-operators/uuid/
4. http://arganzheng.life/hadoop-filesystem-closed-exception.html

This article shows you how to import data from Neo4j to NebulaGraph with the official ETL tool, Nebula Exchange. It also covers relevant problems and solutions during the data import process.

Import data from Neo4j to NebulaGraph via Nebula Exchange: Best Practices


![Exploring the S&P 100 Index Stocks Using Graph Machine Learning](https://user-images.githubusercontent.com/57335825/98957939-9a853100-24b6-11eb-9c54-56975578e2b8.png)

In a previous post of this series [1], we introduced how to use the Python-based graph analysis library NetworkX [2] and [NebulaGraph](https://github.com/vesoft-inc/nebula) [3] to analyze the character relationships in "Game of Thrones".

In this post, we are going to explore how to use the Java-based graph analysis library JGraphT [4] and the diagramming library mxGraph [5] to visualize the changes of correlation between the S&P 100 Index  stocks [6] over time.

![Exploring the S&P 100 Index Stocks Using Graph Machine Learning](https://user-images.githubusercontent.com/57335825/98920270-c38cce00-2484-11eb-8b1f-b9f3ea62d79b.png)

## Dataset Processing

The main analysis method used in this post refers to [7] and [8]. There are two datasets: the vertex set and the edge set.

### Stock Data (Vertex Set)

Select the following S&P 100 Index stocks, and model each stock as a vertex. The properties of each vertex are the stock code and the industry of the listed company.

Table 1: The vertex set sample

| Vertex ID | Code | Industry | Company Name |
| --- | --- | --- | --- |
| 1 | AAPL | Consumer Electronics | Apple Inc. (AAPL) |
| 2 | ABBV | Drug Manufacturers—General | AbbVie Inc. (ABBV) |
| 3 | ABT | Medical Devices | Abbott Laboratories (ABT) |
| 4 | ACN | Information Technology Services | Accenture plc (ACN) |
| 5 | AIG | Insurance—Diversified | American International Group, Inc. (AIG) |
| 6 | ALL | Insurance—Property & Casualty | The Allstate Corporation (ALL) |
| 7 | AMGN | Drug Manufacturers—General | Amgen Inc. (AMGN) |
| 8 | AMT | REIT—Specialty | American Tower Corporation (REIT) (AMT) |
| 9 | AMZN | Internet Retail | Amazon.com, Inc. (AMZN) |

### Stock Relationship (Edge/Relationship Set)

Each edge has only one property, i.e. the weight. The weight of an edge indicates the daily return rate similarity of the listed companies represented by the vertices (stocks) on both ends of the edge. The similarity algorithm refers to [7] and [8]: Analyze the time-series correlation of the individual stocks' daily return rate $ P_{ij} $ over a period of time (from 2014-01-01 to 2020-01-01), and define the distance between the individual stocks (i.e. the edge weight) as follows.

$ l_{ij} = \sqrt{2（1-P_{ij}）}  $

Through such processing, we get the range of the distance as [0,2]. That means the more distant the two stocks, the lower the correlation between their return rates.

Table 2: The edge set sample

| Source vertex ID | Target vertex ID | Edge weight |
| --- | --- | --- |
| 1 | 2 | 1.4206879982683 |
| 1 | 4 | 1.37090336100054 |
| 1 | 6 | 1.39684994985333 |

Such a vertex set and an edge set form a graph network, which can be stored in the NebulaGraph database.

## JGraphT

JGraphT is an open-source Java class library that provides not only a variety of efficient and common graph data structures but also many useful algorithms for solving the most common graph problems.
The features of JGraphT are as follows:

- Supports directed edges, undirected edges, weighted edges, non-weighted edges, etc.
- Supports simple graphs, multigraphs, and pseudographs.
- Provides dedicated iterators (DFS, BFS, etc.) for graph traversals.
- Provides a large number of commonly used graph algorithms, such as path lookup, isomorphism detection, coloring, common ancestors, wandering, connectivity, matching, cycle detection, partitioning, cutting, flow, and centrality.
- Provides easy import from/export to GraphViz. The exported GraphViz file can be applied by the visualization tool Gephi for analysis and demonstration.
- Supports convenient graph network generating while used with tools such as JGraphX，mxGraph，and Guava Graphs Generators.

Next, let's try it out.

1. Create a directed graph in JGraphT.

```java
import org.jgrapht.*;
import org.jgrapht.graph.*;
import org.jgrapht.nio.*;
import org.jgrapht.nio.dot.*;
import org.jgrapht.traverse.*;

import java.io.*;
import java.net.*;
import java.util.*;

Graph<URI, DefaultEdge> g = new DefaultDirectedGraph<>(DefaultEdge.class);

URI google = new URI("http://www.google.com");
URI wikipedia = new URI("http://www.wikipedia.org");
URI jgrapht = new URI("http://www.jgrapht.org");

```

2. Add vertices.

```java
g.addVertex(google);
g.addVertex(wikipedia);
g.addVertex(jgrapht);
```

3. Add edges.

```java
// Add edges to create a linked structure

g.addEdge(jgrapht, wikipedia);
g.addEdge(google, jgrapht);
g.addEdge(google, wikipedia);
g.addEdge(wikipedia, google);
```

## The NebulaGraph database

JGraphT usually uses local files as data sources. That's fine when you're doing static network research, but if the graph network is constantly changing, like a graph network of the stock data, which changes every day, it's a little bit of a hassle to generate a new static file, then load it and then analyze it every time. Ideally, the entire change process can be written to a database persistently, and subgraphs or the complete graph can be loaded directly from the database in real-time for analysis. In this post, we use NebulaGraph as the graph database for storing the graph data.

NebulaGraph's Java client nebula-java [11] provides two ways of accessing NebulaGraph. One is interacting with the query engine layer [13] through the graph query language nGQL [12], which is usually suitable for accessing subgraphs and supports complex semantics; The other is to directly interact with the underlying storage layer (the storaged process) [14] through the APIs, which is used to obtain the complete set of vertices and edges. In addition to accessing NebulaGraph itself, nebula-java provides examples of interacting with Neo4j [15], JanusGraph [16], Spark [17], and others.

In this post, we use the APIs to access the storage layer (the storaged process) directly to get all the vertices and edges. The following two interfaces can be used to read all the vertex and edge data.

```java
// space is the name of the graph space to be scanned, and returnCols is the vertex/edge and its properties to be read.
// Format of returnCols parameter: {tag1Name: prop1, prop2, tag2Name: prop3, prop4, prop5}
Iterator<ScanVertexResponse> scanVertex(
            Stringspace, Map<String, List<String>>returnCols);
Iterator<ScanEdgeResponse> scanEdge(
            Stringspace, Map<String, List<String>>returnCols);
```

1. Initialize a client and a ScanVertexProcessor. ScanVertexProcessor is used to decode the read vertex data:

```java
MetaClientImpl metaClientImpl = new MetaClientImpl(metaHost, metaPort);
metaClientImpl.connect();
StorageClient storageClient = new StorageClientImpl(metaClientImpl);
Processor processor = new ScanVertexProcessor(metaClientImpl);
```

2. Call the scanVertex interface, which returns an iterator for the scanVertexResponse object:

```java
Iterator<ScanVertexResponse> iterator =
                storageClient.scanVertex(spaceName, returnCols);
```

3. Keep reading the data in the scanVertexResponse object that the iterator points to until all of the data is read. The read vertex data is saved and later added to the graph structure of JGraphT.

```java
while (iterator.hasNext()) {
  ScanVertexResponse response = iterator.next();
  if (response == null) {
    log.error("Error occurs while scan vertex");
    break;
  }
  
  Result result =  processor.process(spaceName, response);
  results.addAll(result.getRows(TAGNAME));
}
```

Reading edge data is similar to the above process.

## Analyze the Graph in JGraphT

1. Create an undirected and weighted graph in JGraphT :

```java
Graph<String, MyEdge> graph = GraphTypeBuilder
                .undirected()
    .weighted(true)
    .allowingMultipleEdges(true)
    .allowingSelfLoops(false)
    .vertexSupplier(SupplierUtil.createStringSupplier())
    .edgeSupplier(SupplierUtil.createSupplier(MyEdge.class))
    .buildGraph();
```

2. Add the vertex and edge data read in the last step from the NebulaGraph space to the graph:

```java
for (VertexDomain vertex : vertexDomainList){
    graph.addVertex(vertex.getVid().toString());
    stockIdToName.put(vertex.getVid().toString(), vertex);
}

for (EdgeDomain edgeDomain : edgeDomainList){
    graph.addEdge(edgeDomain.getSrcid().toString(), edgeDomain.getDstid().toString());
    MyEdge newEdge = graph.getEdge(edgeDomain.getSrcid().toString(), edgeDomain.getDstid().toString());
    graph.setEdgeWeight(newEdge, edgeDomain.getWeight());
}
```

3. Like the analysis algorithms mentioned in [7] and [8], use Prim's minimum-spanning tree algorithm for the preceding graph and call the encapsulated drawGraph interface to draw the graph.

> Prim's algorithm is an algorithm in graph theory that searches for a minimum spanning tree in a weighted connected graph. In other words, the tree formed by the edge subset searched by this algorithm not only includes all vertices in the connected graph but also has the minimum sum of weights of all edges.

```java
SpanningTreeAlgorithm.SpanningTree pMST = new PrimMinimumSpanningTree(graph).getSpanningTree();

Legend.drawGraph(pMST.getEdges(), filename, stockIdToName);
```

4. The drawGraph method encapsulates parameter settings such as the layout of the drawing. This method renders stocks in the same sector in the same color, grouping close stocks together.

```java
public class Legend {
  
...
  
  public static void drawGraph(Set<MyEdge> edges, String filename, Map<String, VertexDomain> idVertexMap) throws IOException {
     // Creates graph with model
     mxGraph graph = new mxGraph();
     Object parent = graph.getDefaultParent();

     // set style
     graph.getModel().beginUpdate();
     mxStylesheet myStylesheet =  graph.getStylesheet();
     graph.setStylesheet(setMsStylesheet(myStylesheet));

     Map<String, Object> idMap = new HashMap<>();
     Map<String, String> industryColor = new HashMap<>();

     int colorIndex = 0;

     for (MyEdge edge : edges) {
       Object src, dst;
       if (!idMap.containsKey(edge.getSrc())) {
         VertexDomain srcNode = idVertexMap.get(edge.getSrc());
         String nodeColor;
         if (industryColor.containsKey(srcNode.getIndustry())){
           nodeColor = industryColor.get(srcNode.getIndustry());
         }else {
           nodeColor = COLOR_LIST[colorIndex++];
           industryColor.put(srcNode.getIndustry(), nodeColor);
         }
         src = graph.insertVertex(parent, null, srcNode.getName(), 0, 0, 105, 50, "fillColor=" + nodeColor);
         idMap.put(edge.getSrc(), src);
       } else {
         src = idMap.get(edge.getSrc());
       }

       if (!idMap.containsKey(edge.getDst())) {
         VertexDomain dstNode = idVertexMap.get(edge.getDst());

         String nodeColor;
         if (industryColor.containsKey(dstNode.getIndustry())){
           nodeColor = industryColor.get(dstNode.getIndustry());
         }else {
           nodeColor = COLOR_LIST[colorIndex++];
           industryColor.put(dstNode.getIndustry(), nodeColor);
         }

         dst = graph.insertVertex(parent, null, dstNode.getName(), 0, 0, 105, 50, "fillColor=" + nodeColor);
         idMap.put(edge.getDst(), dst);
       } else {
         dst = idMap.get(edge.getDst());
       }
       graph.insertEdge(parent, null, "", src, dst);
     }


     log.info("vertice " + idMap.size());
     log.info("colorsize " + industryColor.size());

     mxFastOrganicLayout layout = new mxFastOrganicLayout(graph);
     layout.setMaxIterations(2000);
     //layout.setMinDistanceLimit(10D);
     layout.execute(parent);

     graph.getModel().endUpdate();

     // Creates an image than can be saved using ImageIO
     BufferedImage image = createBufferedImage(graph, null, 1, Color.WHITE,
                                               true, null);

     // For the sake of this example we display the image in a window
     // Save as JPEG
     File file = new File(filename);
     ImageIO.write(image, "JPEG", file);

   }
  
  ...

}
```

5. Visualize the data.

The color of each vertex in Figure 1 represents its industry. We can see that the stocks with high business similarity have been clustered together, but some stocks with no obvious correlation have also been clustered together, the reason for which needs to be studied separately.

![Clustering based on the stock data from 2014-01-01 to 2020-01-01](https://user-images.githubusercontent.com/57335825/98921353-0d29e880-2486-11eb-82af-def12cef71a6.png)

Figure 1: Clustering based on the stock data from 2014-01-01 to 2020-01-01

6. Some other dynamic exploration based on different time windows.

The preceding conclusion is mainly based on the stock aggregation from 2014-01-01 to 2020-01-01. We also made other attempts: Use a sliding window of 2 years and the same analysis method to observe if the clustered groups would change over time.

![Clustering based on the stock data from 2014-01-01 to 2016-01-01](https://user-images.githubusercontent.com/57335825/98921452-2cc11100-2486-11eb-9dbe-ba34cfe0bd09.png)
 
Figure 2: Clustering based on the stock data from 2014-01-01 to 2016-01-01

![Clustering based on the stock data from 2015-01-01 to 2017-01-01](https://user-images.githubusercontent.com/57335825/98921505-3ea2b400-2486-11eb-978d-c5cf2091c900.png)

Figure 3: Clustering based on the stock data from 2015-01-01 to 2017-01-01

![Clustering based on the stock data from 2016-01-01 to 2018-01-01](https://user-images.githubusercontent.com/57335825/98921597-59752880-2486-11eb-908c-cd038eb974a7.png)

Figure 4: Clustering based on the stock data from 2016-01-01 to 2018-01-01

![Clustering based on the stock data from 2017-01-01 to 2019-01-01](https://user-images.githubusercontent.com/57335825/98921676-6f82e900-2486-11eb-8de5-ea984c98b37e.png)

Figure 5: Clustering based on the stock data from 2017-01-01 to 2019-01-01

![Clustering based on the stock data from 2018-01-01 to 2020-01-01](https://user-images.githubusercontent.com/57335825/98921795-8c1f2100-2486-11eb-8a1e-d9e28ff2201c.png)

Figure 6: Clustering based on the stock data from 2018-01-01 to 2020-01-01

According to a rough analysis, with the change of time window, the clustering of stocks within most industries maintains relatively good, which means that with the change of time, all kinds of stocks within the same industry maintain relatively high correlation.

## Disclaimer

This post should not be taken as investment advice.
Due to the situation of trade suspension, circuit breakers, trading limits, transfers, mergers and acquisitions, changes of the main business, etc., the data processing in this post may be incorrect. We have not checked all the data piece by piece.

Limited by time, this post only selects the data of 100 stock samples in the past six years, and only adopts the method of minimum expansion tree to do the clustering and classification. In the future, maybe we can use larger data sets (such as U.S. stocks, derivatives, and digital currencies) to try more methods in machine learning.

For the code used in this post, please see [18].

## References

[1] Analyzing Relationships in Game of Thrones With NetworkX, Gephi, and NebulaGraph (Part One) [https://nebula-graph.io/posts/game-of-thrones-relationship-networkx-gephi-nebula-graph/](https://nebula-graph.io/posts/game-of-thrones-relationship-networkx-gephi-nebula-graph/)

[2] Analyzing Relationships in Game of Thrones With NetworkX, Gephi, and NebulaGraph (Part Two) [https://nebula-graph.io/posts/game-of-thrones-relationship-networkx-gephi-nebula-graph-part-two/](https://nebula-graph.io/posts/game-of-thrones-relationship-networkx-gephi-nebula-graph-part-two/)

[3] NetworkX: a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. [https://networkx.github.io/](https://networkx.github.io/)

[4] NebulaGraph: A powerfully distributed, scalable, lightning-fast graph database written in C++. [https://nebula-graph.io/](https://nebula-graph.io/)

[5] JGraphT: a Java library of graph theory data structures and algorithms. [https://jgrapht.org/](https://jgrapht.org/)

[6] mxGraph: JavaScript diagramming library that enables interactive graph and charting applications. [https://jgraph.github.io/mxgraph/](https://jgraph.github.io/mxgraph/)

[7] Bonanno, Giovanni & Lillo, Fabrizio & Mantegna, Rosario. (2000). High-frequency Cross-correlation in a Set of Stocks. arXiv.org, Quantitative Finance Papers. 1. 10.1080/713665554.

[8] Mantegna, R.N. Hierarchical structure in financial markets. Eur. Phys. J. B 11, 193–197 (1999).

[9] [https://graphviz.org/](https://graphviz.org/)

[10] [https://gephi.org/](https://gephi.org/)

[11] [https://github.com/vesoft-inc/nebula-java](https://github.com/vesoft-inc/nebula-java)

[12] NebulaGraph Query Language (nGQL). [https://docs.nebula-graph.io/manual-EN/1.overview/1.concepts/2.nGQL-overview/](https://docs.nebula-graph.io/manual-EN/1.overview/1.concepts/2.nGQL-overview/)

[13] NebulaGraph Query Engine. [https://github.com/vesoft-inc/nebula-graph](https://github.com/vesoft-inc/nebula-graph)

[14] Nebula-storage: A distributed consistent graph storage. [https://github.com/vesoft-inc/nebula-storage](https://github.com/vesoft-inc/nebula-storage)

[15] Neo4j. [www.neo4j.com](http://www.neo4j.com)

[16] JanusGraph. [janusgraph.org](http://janusgraph.org)

[17] Apache Spark. [spark.apache.org](http://spark.apache.org).

[18] [https://github.com/Judy1992/nebula_scan](https://github.com/Judy1992/nebula_scan)

In this post, we are going to explore how to use the Java-based graph analysis library JGraphT and the diagramming library mxGraph to visualize the changes of correlation between the S&P 100 Index  stocks over time.

Exploring the S&P 100 Index Stocks Using Graph Machine Learning

This article shares the benchmarking tests conducted by the NLP team at Meituan. The test included data import and throughput/latency performance.

Benchmarking the Mainstream Open Source Distributed Graph Databases at Meituan: NebulaGraph vs Dgraph vs JanusGraph


> This article is written by Henson Wu from SeeYII, a financial service technology company
> It is originally published on the NebulaGraph forum:[https://discuss.nebula-graph.io/t/fast-deploy-nebula-graph-cluster-with-docker-swarm/693](https://discuss.nebula-graph.io/t/fast-deploy-nebula-graph-cluster-with-docker-swarm/693)

![Fast Deploy NebulaGraph Cluster with Docker Swarm](https://user-images.githubusercontent.com/57335825/97404495-71a84d80-18b3-11eb-8ac7-eaa79b55d844.png)

This post describes in detail how to deploy a NebulaGraph cluster with Docker Swarm. 

## Deploy a NebulaGraph Cluster

### 2.1 Environment Preparation

Prepare hosts as below.

| ip | Memory (GB) | CPU (# of Cores) |
| --- | --- | --- |
| 192.168.1.166 | 16 | 4 |
| 192.168.1.167 | 16 | 4 |
| 192.168.1.168 | 16 | 4 |

Please make sure that Docker has been installed on all the machines.

### 2.2 Initialize the Swarm Cluster

Execute the commands below on the host 192.168.1.166:

```
$ docker swarm init --advertise-addr 192.168.1.166
Swarm initialized: current node (dxn1zf6l61qsb1josjja83ngz) is now a manager.
To add a worker to this swarm, run the following command:
 docker swarm join \
 --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
 192.168.1.166:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
```

### 2.3 Add a Worker Node

Add a Swarm worker node per the notification message of the init commands. Execute the following commands on 192.168.1.167 and 192.168.1.168 respectively.

```
docker swarm join \
 --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
 192.168.1.166:2377
```

### 2.4 Authenticate the Cluster

```
docker node ls
 
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
h0az2wzqetpwhl9ybu76yxaen *   KF2-DATA-166        Ready               Active              Reachable           18.06.1-ce
q6jripaolxsl7xqv3cmv5pxji     KF2-DATA-167        Ready               Active              Leader              18.06.1-ce
h1iql1uvm7123h3gon9so69dy     KF2-DATA-168        Ready               Active                                  18.06.1-ce
```

### 2.5 Configure Docker Stack

```
vi docker-stack.yml
```

Configure Docker Stack as below:

```
version: '3.6'
services:
  metad0:
    image: vesoft/nebula-metad:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.166
      - --ws_ip=192.168.1.166
      - --port=45500
      - --data_path=/data/meta
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-166
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.166:11000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 11000
        published: 11000
        protocol: tcp
        mode: host
      - target: 11002
        published: 11002
        protocol: tcp
        mode: host
      - target: 45500
        published: 45500
        protocol: tcp
        mode: host
    volumes:
      - data-metad0:/data/meta
      - logs-metad0:/logs
    networks:
      - nebula-net

  metad1:
    image: vesoft/nebula-metad:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.167
      - --ws_ip=192.168.1.167
      - --port=45500
      - --data_path=/data/meta
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-167
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.167:11000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 11000
        published: 11000
        protocol: tcp
        mode: host
      - target: 11002
        published: 11002
        protocol: tcp
        mode: host
      - target: 45500
        published: 45500
        protocol: tcp
        mode: host
    volumes:
      - data-metad1:/data/meta
      - logs-metad1:/logs
    networks:
      - nebula-net

  metad2:
    image: vesoft/nebula-metad:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.168
      - --ws_ip=192.168.1.168
      - --port=45500
      - --data_path=/data/meta
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-168
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.168:11000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 11000
        published: 11000
        protocol: tcp
        mode: host
      - target: 11002
        published: 11002
        protocol: tcp
        mode: host
      - target: 45500
        published: 45500
        protocol: tcp
        mode: host
    volumes:
      - data-metad2:/data/meta
      - logs-metad2:/logs
    networks:
      - nebula-net

  storaged0:
    image: vesoft/nebula-storaged:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.166
      - --ws_ip=192.168.1.166
      - --port=44500
      - --data_path=/data/storage
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-166
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.166:12000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 12000
        published: 12000
        protocol: tcp
        mode: host
      - target: 12002
        published: 12002
        protocol: tcp
        mode: host
    volumes:
      - data-storaged0:/data/storage
      - logs-storaged0:/logs
    networks:
      - nebula-net
  storaged1:
    image: vesoft/nebula-storaged:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.167
      - --ws_ip=192.168.1.167
      - --port=44500
      - --data_path=/data/storage
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-167
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.167:12000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 12000
        published: 12000
        protocol: tcp
        mode: host
      - target: 12002
        published: 12004
        protocol: tcp
        mode: host
    volumes:
      - data-storaged1:/data/storage
      - logs-storaged1:/logs
    networks:
      - nebula-net

  storaged2:
    image: vesoft/nebula-storaged:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.168
      - --ws_ip=192.168.1.168
      - --port=44500
      - --data_path=/data/storage
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-168
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.168:12000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 12000
        published: 12000
        protocol: tcp
        mode: host
      - target: 12002
        published: 12006
        protocol: tcp
        mode: host
    volumes:
      - data-storaged2:/data/storage
      - logs-storaged2:/logs
    networks:
      - nebula-net
  graphd1:
    image: vesoft/nebula-graphd:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --port=3699
      - --ws_ip=192.168.1.166
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-166
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.166:13000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 3699
        published: 3699
        protocol: tcp
        mode: host
      - target: 13000
        published: 13000
        protocol: tcp
#        mode: host
      - target: 13002
        published: 13002
        protocol: tcp
        mode: host
    volumes:
      - logs-graphd:/logs
    networks:
      - nebula-net

  graphd2:
    image: vesoft/nebula-graphd:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --port=3699
      - --ws_ip=192.168.1.167
      - --log_dir=/logs
      - --v=2
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-167
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.167:13001/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 3699
        published: 3640
        protocol: tcp
        mode: host
      - target: 13000
        published: 13001
        protocol: tcp
        mode: host
      - target: 13002
        published: 13003
        protocol: tcp
#        mode: host
    volumes:
      - logs-graphd2:/logs
    networks:
      - nebula-net
  graphd3:
    image: vesoft/nebula-graphd:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --port=3699
      - --ws_ip=192.168.1.168
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-168
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.168:13002/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 3699
        published: 3641
        protocol: tcp
        mode: host
      - target: 13000
        published: 13002
        protocol: tcp
#        mode: host
      - target: 13002
        published: 13004
        protocol: tcp
        mode: host
    volumes:
      - logs-graphd3:/logs
    networks:
      - nebula-net
networks:
  nebula-net:
    external: true
    attachable: true
    name: host
volumes:
  data-metad0:
  logs-metad0:
  data-metad1:
  logs-metad1:
  data-metad2:
  logs-metad2:
  data-storaged0:
  logs-storaged0:
  data-storaged1:
  logs-storaged1:
  data-storaged2:
  logs-storaged2:
  logs-graphd:
  logs-graphd2:
  logs-graphd3:
```

Edit the nebula.env file by adding the items below:

```
TZ=UTC
USER=root
```

### 2.6 Start the NebulaGraph Cluster

```
docker stack deploy nebula -c docker-stack.yml
```

## Cluster Configuration for Load Balancing and High Availability

The NebulaGraph clients (1.X) don't provide load balancing capability currently. They randomly select any graphd to connect the database. Therefore, you need to configure load balancing and high availability on your own if you want to use NebulaGraph in production.

![Cluster Configuration for Load Balancing and High Availability](https://user-images.githubusercontent.com/57335825/97405081-5b4ec180-18b4-11eb-8df9-ac12f3cd8be9.png)

Seen from the figure above, the entire deployment can be divided into three layers, i.e. data layer, load balance layer, and high availability layer. 

The load balance layer is responsible for distributing the requests from the client to the data layer for load balancing purpose.

The HA layer is realized by HAProxy and it ensures that the load balancing service works properly so that the entire cluster works properly.

### 3.1 Load Balancing Configuration

HAProxy uses Docker compose for configuration. Edit the three files below respectively:

1. Dockerfile

```
FROM haproxy:1.7
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
EXPOSE 3640
```

2. docker-compose.yml

```
version: "3.2"
services:
  haproxy:
    container_name: haproxy
    build: .
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
    ports:
      - 3640:3640
    restart: always
    networks:
      - app_net
networks:
  app_net:
    external: true
```

3. haproxy.cfg

```javascript
global
    daemon
    maxconn 30000
    log 127.0.0.1 local0 info
    log 127.0.0.1 local1 warning

defaults
    log-format %hr\ %ST\ %B\ %Ts
    log  global
    mode http
    option http-keep-alive
    timeout connect 5000ms
    timeout client 10000ms
    timeout server 50000ms
    timeout http-request 20000ms

# Customize your own frontends && backends && listen conf
# CUSTOM

listen graphd-cluster
    bind *:3640
    mode tcp
    maxconn 300
    balance roundrobin
    server server1 192.168.1.166:3699 maxconn 300 check
    server server2 192.168.1.167:3699 maxconn 300 check
    server server3 192.168.1.168:3699 maxconn 300 check

listen stats
    bind *:1080
    stats refresh 30s
    stats uri /stats
```

### 3.2 Start HAProxy

```
docker-compose up -d
```

### 3.3 HA Configuration

Follow the configuration steps on 192.168.1.166, 192.168.1.167, and 192.168.1.168.

1. Install Keepalived

```
apt-get update && apt-get upgrade && apt-get install keepalived -y
```

2. Modify Keepalived's configuration file, i.e. /etc/keepalived/keepalived.conf. Note that the priority item should be set to different values for the three hosts so that there is a priority among them.**

**Note: Please be noted that vip (virtual IP) is required to configure Keepalive. In the configurations below, 192.168.1.99 is the virtual IP.**

a. Configuration on 192.168.1.166

```javascript
global_defs {
    router_id lb01 #An identifier；
}
vrrp_script chk_haproxy {
    script "killall -0 haproxy"    interval 2
}
vrrp_instance VI_1 {
    state MASTER
    interface ens160
    virtual_router_id 52
    priority 999
    # Set the interval (in seconds) for sync check between MASTER and BACKUP load balancers.
    advert_int 1
    # Configure the authentication type and password
    authentication {
    # Configure the authentication type, mainly PASS and AH
        auth_type PASS
    # Set the authentication password.The MASTER and BACKUP load balancers must use the same password within the same vrrp_instance to communicate properly.
        auth_pass amber1
    }
    virtual_ipaddress {
        # The virtual IP is 192.168.1.99/24; Bounded interface is ens160; Alias for both MASTER and BACKUP load balancers is ens160:1
        192.168.1.99/24 dev ens160 label ens160:1
    }
    track_script {
        chk_haproxy
    }
}
```

b. Configuration on 192.168.1.167

```javascript
global_defs {
    router_id lb01 #An identifier；
}
vrrp_script chk_haproxy {
    script "killall -0 haproxy"    interval 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    virtual_router_id 52
    priority 888
    # Set the interval (in seconds) for sync check between MASTER and BACKUP load balancers.
    advert_int 1
    # Configure the authentication type and password
    authentication {
    # Configure the authentication type, mainly PASS and AH
        auth_type PASS
    # Set the authentication password.The MASTER and BACKUP load balancers must use the same password within the same vrrp_instance to communicate properly.
        auth_pass amber1
    }
    virtual_ipaddress {
        # The virtual IP is 192.168.1.99/24; Bounded interface is ens160; Alias for both MASTER and BACKUP load balancers is ens160:1
        192.168.1.99/24 dev ens160 label ens160:1
    }
    track_script {
        chk_haproxy
    }
}
```

c. Configuration on 192.168.1.168

```javascript
global_defs {
    router_id lb01 #An identifier；
}
vrrp_script chk_haproxy {
    script "killall -0 haproxy"    interval 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    virtual_router_id 52
    priority 777
    # Set the interval (in seconds) for sync check between MASTER and BACKUP load balancers.
    advert_int 1
   # Configure the authentication type and password
    authentication {
    # Configure the authentication type, mainly PASS and AH
        auth_type PASS
    # Set the authentication password.The MASTER and BACKUP load balancers must use the same password within the same vrrp_instance to communicate properly.
        auth_pass amber1
    }
    virtual_ipaddress {
        # The virtual IP is 192.168.1.99/24; Bounded interface is ens160; Alias for both MASTER and BACKUP load balancers is ens160:1
        192.168.1.99/24 dev ens160 label ens160:1
    }
    track_script {
        chk_haproxy
    }
}
```

Relevant commands used in Keepalived:

```javascript
# Start Keepalived
systemctl start keepalived
# Auto start Keepalived
systemctl enable keeplived
# Re-start Keepalived
systemctl restart keepalived
```

## Deploy the Cluster Offline

How to deploy the NebulaGraph cluster with Docker Swarm offline? The answer is simple. Changing the Docker image to private image repo will do.

If you have any questions regarding deploying NebulaGraph with Docker Swarm, please leave your comments in this thread: [https://discuss.nebula-graph.io/t/fast-deploy-nebula-graph-cluster-with-docker-swarm/693](https://discuss.nebula-graph.io/t/fast-deploy-nebula-graph-cluster-with-docker-swarm/693)

## You might also like:

1. [NebulaGraph Architecture — A Bird’s Eye View](https://nebula-graph.io/posts/nebula-graph-architecture-overview/)
1. [An Introduction to NebulaGraph's Storage Engine](https://nebula-graph.io/posts/nebula-graph-storage-engine-overview/)
1. [An Introduction to NebulaGraph’s Query Engine](https://nebula-graph.io/posts/nebula-graph-query-engine-overview/)

This article introduces how to deploy a NebulaGraph cluster with Docker Swarm. It also talks about how to keep high availability and load balancing of the cluster.

Tag