Technical solutions against transaction surges

As we shared in our Bot Transaction Analysis Report, the Klaytn mainnet has been experiencing sudden and sustained surges in transactions recently. This has caused great inconvenience to Klaytn users by congesting the network, misleading them to believe that the network was down or not operating correctly, when in fact the network was producing blocks as normal but the elevated transaction load was slowing down processing times . In this article, we would like to share the progress on our technological efforts to overcome this problem.

The Spam Throttler implementation

The Spam Throttler was created as a response to bot transactions around the end of 2021. It was found that these bot transactions were generated by a few entities, who almost took over the entire network for a few hours. While they used various sender addresses, the number of recipient addresses were limited. Over 99% of the transactions were intentionally being manipulated to be reverted. The Spam Throttler limits the speed at which the recurrently reverted transactions are propagated, thereby giving more opportunities for normal transactions to be included in a block.

Instead of “blacklisting”, whereby transactions from accounts and addresses are precluded from using the network with this centralized way of processing, we opted for a method by which a PN can analyze the newest blocks via an algorithm. We wanted the network resources to be decentralized and distributed evenly to a large number of users, instead of being concentrated on a few creating another layer of centralization. The criteria for spam transactions have also been defined to only cover substantial amounts of reverted transactions.

Such a solution that involves limiting system performance bears the risk of being exploited by malicious actors. To prevent such an ill-intentioned user from using the Spam Throttler for crippling the network, we also have put in place the following measures which will curb any unintended side effects of the Spam Throttler.

1. The Spam Throttler only limits propagation speed during periods of transaction overload on the Klaytn network
2. A certain level of TPS is guaranteed for Throttle Lists (Current: 100 TPS)
3. Non-faulty services can be whitelisted in case of emergency

The Spam Throttler was applied on the mainnet in early January 2022, which immediately led to a dramatic drop in reverted transactions. It also succeeded in regulating excessively occurring reverted transactions, decongesting the network by reducing these spam reversion transactions. However, while this feature was successful in handling the reverted transaction spam, it ultimately was not sufficient alone to completely eliminate bot-generated spam transactions. Since the introduction of the Spam Throttler feature, the bots have found a way to bypass reverted transactions by adjusting their logic to return gracefully instead of reverting outright. In sum, the Spam Throttler did reduce the number of reverted transactions, but did not alleviate the total number of bursty transactions.

Problems caused by bot transactions and technical solutions

Besides the Spam Throttler, the Klaytn Team is also testing out various technological solutions to mitigate the problem of transaction overload. To this end, we categorized the users’ inconvenience into three groups to implement the technological solutions suitable for each.

1. Error or transactions deployment failures when using Endpoint Nodes (ENs)
   This problem occurs when transactions suddenly increase to more than what an EN can accommodate. ENs are the primary transaction submission and gossip handling nodes within the Klaytn network. If you’ve ever seen the error message “Txpool is full” on Kaikas, a congested transaction pool on the EN was the cause. In order to mitigate this problem, the Klaytn Team is working on a solution to restrict the number of transactions received externally when each node stores transactions. At the beginning of January in 2022, many other experimental features have been applied to some ENs including the Kaikas EN. The number of users experiencing “Txpool is full” on Kaikas EN have significantly decreased. Through various experiments, this feature will evolve so that an appropriate number of external transactions can be manually set per EN by the operator.
2. Network monopoly of certain users/services
   This problem is what the Spam Throttler is intended to help mitigate, and is also the most technologically complex one to solve. For the long-term, we are working on ways to increase network throughput in terms of TPS and scaling solutions. We are also finding ways to advocate and enforce a fair utilization of network resources. Other solutions currently under consideration include first-come-first-served basis propagation and block inclusion, and fair management of multiple users’ transactions by nodes. Adjustments on the policy level such as transaction fee adjustments are also being taken into account.
3. Inability to prioritize transactions during network overload
   Klaytn’s policy of maintaining a fixed gas price, and in turn the deterministic gas fees, offers many advantages to developers. However, a critical downside is that the transactions cannot be prioritized during network congestion. The Klaytn Team is aware of this problem, and is contemplating an adjustment to the current price model. Possible options include a more flexible fee model, differentiated fee model, or fee burning model. Since this is a change that can greatly impact the ecosystem, we will make sure to base any decisions substantiated with sufficient preparations, research, real world data and communication.

The Klaytn Team also intends to collaborate with our ecosystem in solving these problems. We are aware that these issues have several add-on effects such as transaction freezes on exchanges, and bot transactions during NFT/DeFi project launches, which could cause Klaytn users great inconvenience. Since these are issues that cannot be considered only in the context of the Klaytn protocol layer alone, but affect the ecosystem as a whole, we will work jointly with the support and help from our ecosystem and community members. Klaytn ecosystem development is truly open-source and as such we welcome any contributions to the Klaytn ecosystem and communities via GitHub, Discord, or our developer forum.