Based on redis distributed cache implementation

Based on redis distributed cache implementation

First: What is Redis?

Redis is a high-performance storage system based on memory, durable log type, Key-Value database, and provides APIs in multiple languages.

Second: the background

Data structure (Data Structure) demand is increasing, but not in memcache, which affects development efficiency

Performance requirements need to be resolved as the number of read operations rises. The processes experienced are as follows:

Database read and write separation (M/S)-database uses multiple slaves-increase Cache (memcache)-go to Redis

Solve the problem of writing:

Horizontal split, split the table, put some users in this table, and some users in another table;

Reliability requirements

The avalanche problem of Cache makes people tangled

Cache faces the challenge of fast recovery

Development cost requirements

The cost of maintaining consistency between Cache and DB is getting higher and higher (first clean up the DB, then clean up the cache, no, it's too slow!)

Development needs to keep up with the influx of product demand

The most expensive hardware cost is the database-level machine, which is basically several times more expensive than the front-end machine, mainly IO-intensive, which consumes hardware;

Complex maintenance

Consistency maintenance costs are getting higher and higher;

BerkeleyDB uses B-trees and will always write new ones, and there will be no internal file reorganization; this will cause the files to become larger and larger; when they are large, they need to be archived, and the archiving operation should be done regularly;

In this way, a certain down time is required;

Based on the above considerations, Redis was chosen

Third: Application of Redis in Sina Weibo

Introduction to Redis

Support 5 data structures

Support strings, hashes, lists, sets, sorted sets

String is a good storage method for counting storage. sets are great for building index libraries;

KV storage vs KV cache

98% of Sina Weibo currently used are persistent applications, 2% are caches, and 600+ servers are used

The persistent application and non-persistent methods in Redis will not be very different:

The non-persistent is 80,000 to 90,000 tps, then the persistence is about 70 to 80,000 tps;

When using persistence, you need to consider the ratio of persistence and write performance, that is, consider the ratio of the memory size used by redis to the hard disk write rate;

Active community

Redis currently has more than 30,000 lines of code, the code is simplified, there are many ingenious implementations, the author has a technical clean

The Redis community is highly active, which is an important indicator to measure the quality of open source software. Generally, there is no commercial technical service support in the initial stage of open source software. If there is no active community to support, there is nowhere to ask for help once a problem occurs;

Redis basic principles

Redis persistence (aof) append online file:

Write log(aof) and merge it with memory to a certain extent. Append and then append, write to disk sequentially, which has very little impact on performance

Single instance single process

Redis uses a single process, so when configuring, an instance will only use one CPU;

During configuration, if you need to maximize the CPU usage, you can configure the number of Redis instances to correspond to the number of CPUs, and the number of Redis instances to correspond to the number of ports (8-core CPU, 8 instances, 8 ports) to improve concurrency:

In the single machine test, a single data is 200 bytes, and the test result is 80,000 to 90,000 tps;


Process: Data is written to master-master and stored in slave's rdb-slave loads RDB into memory.

Save point: When the network is interrupted, after connecting, continue to transmit.

The first synchronization under Master-slave is full transmission, followed by incremental synchronization;

Data consistency

The possibility of inconsistency between multiple nodes after long-term operation;

Develop two tool programs:

1. For data with a large amount of data, full inspection will be performed periodically;

2. Check whether the incremental data is consistent in real time;

The inconsistency caused by the failure of the master library to synchronize the slave library in time is called a delay problem;

For scenarios where the consistency requirements are not so strict, we only need to ensure the final consistency;

For the delay problem, it is necessary to analyze the characteristics of the business scenario and increase the strategy from the application level to solve this problem;


1. Newly registered users must first query the main library;

2. After the registration is successful, you need to wait for 3s to jump, and the background is doing data synchronization at this time.

Fourth: Architecture design of distributed cache

1. Architecture design

Since redis is a single point, it needs to be used in the project and must be distributed by itself. The basic architecture diagram is as follows:

Click to view original size image

2. Distributed implementation

Through the key to do consistent hashing, the distribution of the key corresponding to the redis node is realized.

Implementation of consistent hashing:

l Hash value calculation: By supporting MD5 and MurmurHash two calculation methods, the default is MurmurHash, which is an efficient hash calculation.

l Consistency realization: through java TreeMap to simulate the ring structure to achieve uniform distribution

3.Client's choice

The main modification of jedis is the modification of the partitioning module to support partitioning based on BufferKey. According to different redis node information, different ShardInfo can be initialized. At the same time, the underlying implementation of JedisPool is also modified to connect to the pool pool. Support the construction method based on key and value, create different jedis connection clients based on different ShardInfos, achieve the effect of partition, and call on the application layer

4. Description of the module

l Dirty data processing module, which handles cache operations that fail to execute.

l Shield monitoring module, for abnormal monitoring of jedis operation, when a node is abnormal, it can control the removal of redis node and other operations.

The entire distributed module uses hornetq to remove abnormal redis nodes. For the increase of new nodes, the increase can also be achieved through the reload method. (This module can also be easily implemented for new nodes)

The realization of the above distributed architecture meets the needs of the project. In addition, some redis nodes can be set separately for some more important uses of cached data, and specific priorities can be set. In addition, for the design of the cache interface, the basic interface and some special logical interfaces can also be implemented according to requirements. For cas-related operations, and some things operations can be achieved through its watch mechanism.