The 5th KIR: S2WLAB_”S2_EYEZ”_Progress Report(2)

Klaytn Improvement Reserve: Progress Report

Last updated: 2021. 02. 27.

Summary

S2WLAB proposed a S2_EYEZ for Klaytn, which is a high-performance on-chain analysis platform to improve the data access quality to Klaytn Mainnet. We designed and implemented such a platform with three key challenges: 1) locality of reference in terms of ‘transactions’, 2) performance degradation from traditional ACID properties, and 3) burst read/write operations while traversing the entire blockchain. In order to handle these problems, we proposed a specialized on-chain analysis platform for Klaytn, called S2_EYEZ for Klaytn. We redesign a data management architecture from a flat database model to a graph based model, which allows users to access adjacency addresses or transactions in block traverses. S2_EYEZ for Klaytn allows loose ACID properties in graph operations that reduces unnecessary DB operations to manage the blockchain data. Also, we optimize memory lock mechanisms to reduce burst read/write operations.

In this progress report, we present a prototype implementation of S2_EYEZ for Klaytn and open public APIs to accelerate a rapid development of Klaytn on-chain analysis applications. It offers an abstracted development environment from a complex blockchain data architecture, which reduces the programming complexity and effort required to implement and design potential applications. S2_EYEZ for Klaytn exports tagging APIs and a graph query API. Tagging APIs help to manage preliminary knowledge about Klaytn addresses, used in filtering out meaningful results in graph queries. A query API provides a graph traverse across the entire Klayn on-chain with various conditions to filter out addresses and transactions. We provide such APIs with isolated environments to each user, which allows users to configure their own preliminary knowledge into our platform. To do this, we issue an access token to the users who want to use S2_EYEZ APIs.

Here, the below link provides a form to request a new access token.

Link: S2_EYEZ for Klaytn API Request Form

Project Milestones and Schedule

As a second key deliverable, this progress report provides S2_EYEZ APIs to Klaytn forum users. We implement a prototype of S2_EYEZ for Klaytn, deploy a high-performance graph backend in bare metal servers and open publicly accessible APIs deployed in Amazon EC2 with API gateway services. Please check the below table that describes the details of the timeline and key milestones.

Expected project completion time: 6 months

Key Deliverables

This section describes how to use S2_EYEZ APIs. The APIs require an access token that distinguishes environments to each user and has a daily quota for limiting DoS-like requests. The S2_EYEZ initializes daily quota (10,000 per day) at midnight (KST). Users can query over the HTTP (rpc) interface accessed to S2_EYEZ internal APIs. After reviewing the requests, we will send an email with an access API KEY for authentication, a token to access S2_EYEZ APIs, and the endpoint URL.

By default, all APIs require an access token and required parameters in each API. The common structure of API requests is described below.

# request
curl --request POST \
--url [ENDPOINT]/[FUNCTION] \
--header 'Content-Type: application/json' \
--header 'x-api-key: [ACCESS_API_KEY]' \
--data '{
"token" : [ASSIGNED_TOKEN]",
# other parameters
}'

The response includes two or three (if a data is included) as described below. Code and message fields represent a summarized status of each request. A data field includes query results such as a list of tag and graph search results.

# response
{
"code": 1000,
"data": {},
"message": "success"
}

A response message contains a predefined code and message, which indicates the status of queries. The following codes are provided.

1. Tagging APIs: knowledge curation interfaces

The tag namespace provides register/get/remove tags to Klaytn addresses. Each tag information is associated with an assigned token and not shared with other users.

1.1. Register

Register tag information to Klaytn address. A query API leverages the tag information in query operations.

  1. Request(POST)
  • Request path: [ENDPOINT]/tag/register

  • Request body schema: application/json

  • Body data: arguments for a tag and an associated address

  • Example

#request
curl --request POST
–url [ENDPOINT]/tag/register
–header ‘Content-Type: application/json’
–header ‘x-api-key: [ACCESS_API_KEY]’
–data ‘{
“token” : [ASSIGNED_TOKEN]",
“address” : “0x074fb65663b54770009b1acf2295529d3e010a81”,
“tag” : “general_user”
}’

  1. Response
  • code: integer, status code(1000 for success)
  • message: string, status message
  • Example

#response
{
“code”: 1000,
“message”: “success”
}

1.2. Get

Get a list of tag information defined by a user who has own access token.

  1. Request(POST)
  • Request path: [ENDPOINT]/tag/get

  • Request body schema: application/json

  • Body data: an argument of a request page

  • Example

#request
curl --request POST \
--url [ENDPOINT]/tag/get \
--header 'Content-Type: application/json' \
--header 'x-api-key: [ACCESS_API_KEY]' \
--data '{
"token" : [ASSIGNED_TOKEN]",
"page" : 1
}'
  1. Response
#response
{
"code": 1000,
"data": {
"data": [
{
"address": "0x074fb65663b54770009b1acf2295529d3e010a81",
"has_node": true,
"identifier": "24f9d2ccc67abbc7a7b5dd8a51ab576cbb678363",
"tag": "general_user"
}
],
"query_info": {
"total_document_num": 1,
"total_page_num": 1
}
},
"message": "success"
}

1.3. Remove

Remove tag information. Deleted tags are not used in graph queries anymore.

  1. Request(POST)
  • Request path: [ENDPOINT]/tag/remove

  • Request body schema: application/json

  • Body data: arguments for a tag and an associated address

  • Example

#request
curl --request POST \
--url [ENDPOINT]/tag/remove \
--header 'Content-Type: application/json' \
--header 'x-api-key: [ACCESS_API_KEY]' \
--data '{
"token" : [ASSIGNED_TOKEN]",
"identifier" : "7f508a9a5e0fb3ad5ba7916e035503a1c0f50344"
}'
  1. Response
  • code: integer, status code(1000 for success)

  • message: string, status message

  • Example

#response
{
"code": 1000,
"message": "success"
}

2. Graph search API: Graph-based on-chain analysis query interface

The graph traversal starts with a single origin address. Queries are controlled by the set of parameters of API and those parameters operate as logical conjunction. As an exception, multiple tags are passed with a single query, the system takes the operation of logical disjunction for tags. In some cases, the system retrieves the majority of an entire graph to deal with only a single query. To prevent the system resources occupied by just a few queries, we put the limitations of visible graph nodes for each query. During the traversal of a graph, the query processing will be stopped immediately and returns a result graph when it meets the limitation of visit.

  1. Request(POST)
  • Request path: [ENDPOINT]/analysis/graph-search

  • Request body schema: application/json

  • Body data: arguments for a graph traversal

  • Example

#request
curl --request POST \
--url [ENDPOINT]/analysis/graph-search \
--header 'Content-Type: application/json' \
--header 'x-api-key: [ACCESS_API_KEY]' \
--data '{
"token" : [ASSIGNED_TOKEN]",
"address" : "0x867593ffb80e3776464909e303ff33bf8a92a826",
"timestamp_from": "1603780270",
"timestamp_to": "1603780330",
"amount_min": "10000000",
"amount_max": "100000000000000000",
"transfer_direction" : "transfer_deposit",
"enable_tag_filter" : 0,
"hop_count_max" :1,
"tags": [ {"tag": "s2wlab"}, {"tag": “test"} ]
}'
  1. Response
  • code: integer, status code(1000 for success)

  • message: string, status message

  • data::query_result: json object, result graph data

    • remaining_quota: integer, the number of available query execution you have

    • nodes: array of json object, nodes for a result graph

    • edges: array of json object, edges for a result graph

  • Example

#response
{
"code": 1000,
"data": {
"query_result": {
"edges": [
{
"amount": 10000000000000000,
"from": "0x6d770545c81df8a313f1ce77a6e67a90455b1c8c782062159119466d6b151059",
"to": "0x867593ffb80e3776464909e303ff33bf8a92a826",
"type": "address-tx"
},
{
"amount": 10000000000000000,
"from": "0xdd00a8bab08067dd87b79b27ba07a7c4f7ddaef5",
"to": "0x6d770545c81df8a313f1ce77a6e67a90455b1c8c782062159119466d6b151059",
"type": "address-tx"
}
],
"nodes": [
{
"graph_id": "0x867593ffb80e3776464909e303ff33bf8a92a826",
"type": "address"
},
{
"graph_id": "0xdd00a8bab08067dd87b79b27ba07a7c4f7ddaef5",
"type": "address"
},
{
"graph_id": "0x6d770545c81df8a313f1ce77a6e67a90455b1c8c782062159119466d6b151059",
"timestamp": 1603780271,
"type": "tx"
}
]
},
"remaining_quota": 9992
},
"message": "success"
}

Challenges

In this section, we discuss two main challenges in our platform: 1) performance degradation caused by burst concurrent requests and 2) retrieving internal transactions not explicitly exposed to the blockchain.

Handling burst concurrent requests. Concurrency control is a well-known problem in the field of multithreading and our system is not an exception. Each thread of our system shares the graph data such as nodes and edges. To implement the mutual exception of a graph traversal, every single thread is forced to request to lock the shared resources before access. And then, after processing the locked resources, the thread should explicitly release the resources it holds. While the mutual execution is on its way, the other tasks for such shared resources are blocked until the resources are unlocked. The execution method for the blocked queries is first come, first served. Thus, there are many requests to access the same resources simultaneously, the system requires much more time costs to deal with each query than the idle status of it.

Retrieving internal transaction issues. S2_EYEZ for Klaytn parses value transfer transactions from the Klaytn Mainnet. Each transaction has a value field that represents the amount of transferred Klay between addresses. The blockchain parser in S2_EYEZ can directly parse such transactions, which are invoked by external accounts directly. Smart contracts can transfer Klay to another address, referred to as internal transactions. In contrast to transactions issued by external accounts, these internal transactions have 0 value and transferred amounts are not explicitly exposed to the Blockchain. In order to get internal transactions for tracing Klay transfers from smart contracts, the parser is required to use VM tracing for call traces (https://docs.klaytn.com/bapp/json-rpc/api-references/debug/tracing). According to our experiments, extracting call traces is too slow to parse all internal transactions from the Klaytn Mainnet. This makes it difficult to import internal transactions from the entire Blockchain in a graph initialized phase. If we retrieve such transactions with reasonable time, it is possible to import Klay transfers from smart contracts.

Budget

Task Weekly wage No. engineers Weeks Total
Design $5,000 per engineer 1 5 $25,000
Development $5,000 per engineer 2 5 $50,000
Testing & QA $5,000 per engineer 1 3 $15,000
SUM $90,000

The table describes the product development and maintenance fee. Engineers are assigned based on a schedule for each task. 1) For ‘Design’, one engineer is assigned for 8 weeks, and 3 weeks have progressed. ‘Design’ task has completed. For ‘Development’ and ‘Testing & QA’ tasks, we keep tracking the progress to meet the original plan. 2) For ‘Development’, two engineers are assigned for 15 weeks and 5 weeks have progressed.3) ‘Test & QA’ task has been started along with ‘Development’ task. This task will be conducted until the ‘Development’ task and overall project completes.

Out of the total $220,000 product development & maintenance fee, $155,000 have been spent so far. Product development & maintenance costs will be spent as scheduled.

We deployed the S2_EYEZ API service on the bare-metal servers, hosting a high-performance core engine, and Amazon EC2 and API gateway services, for a frontmost API interface to users. The budget for the Amazon services is expected to be about 6K USD, which will be placed in the next budget.

1 Like

@Prop_S2WLAB_SH The review of this report will be ended next week. :slight_smile:

1 Like

@Prop_S2WLAB_SH I can’t access the form to request a new access token. Could you change the access for public?
Link: https://cutt.ly/mlfHLb0

Could you visit the page again? We fixed the access permission.

Sorry for the inconvenience.

Thanks.
the link is accessible.

1 Like

Hello.

Have you ever used fastCallTracer for tracing internal transactions?
It is implemented by native golang. So it is faster than callTracer.
https://docs.klaytn.com/bapp/json-rpc/api-references/debug/tracing#tracing-options

Not yet. I will do an experiment for using ‘fastCallTracer’ and post the results.
Thanks!

1 Like