Experience summary report on emergency response guide

The ultimate effect of all standardized emergency guidelines is never to allow executors to replicate the process 1:1, but to find a dynamic balance between "the plan is underpinned without stepping on the red line, on-site contingency is not stuck to the provisions, and the review and iteration is not repeated." People who memorize the guidelines will most likely not be able to put out real fires, and those who completely skip the guidelines will easily turn small fires into accidents.

When I was on duty in the e-commerce operation and maintenance team for the 618 promotion last year, I encountered the dilemma of "getting stuck on the process". At two o'clock in the middle of the night the day before, 30% of requests on the core payment link suddenly failed. According to the emergency guidelines at the time, the first step was to cut off all non-core traffic and report it to the group's emergency center within 5 minutes. The expert team could not proceed until a formal solution was issued. But at that time, there were only 40 minutes left before the official launch of the big sale, and it would take at least 15 minutes to go through the group's submission and approval process. The pre-sale channel might be blocked by the time the expert group came up with a plan. At that time, Lao Zhou in the group directly made the decision. He first cut off 30% of the non-core traffic in the live broadcast area and the order recommendation area, leaving only the two core links of pre-sale and payment, and simultaneously sent reports to the group during the operation. The link was restored in the last 28 minutes, and the payment success rate for the day's big sale was 0.2 percentage points higher than last year.

When reviewing the matter afterwards, colleagues from the compliance department did have different opinions. As representatives of the "process-oriented group", their logic is completely tenable: if the core route of pre-sale was accidentally touched when cutting traffic, and the user was unable to pay the deposit, who would bear the loss? The reason why the standardized guidelines put the reporting process before the operation is essentially to allocate risks to the entire emergency system and avoid irreparable losses caused by a single person's decision-making mistakes.

This kind of controversy actually exists in all emergency fields. I had a meal with a friend from the fire protection system before, and he said that they often encountered this dilemma when dispatching police: the guidelines clearly require that before entering the scene of a high-rise fire, gas leaks must be checked and the structural stability of the building must be confirmed. But once when they arrived at the scene, they heard a child crying in the house, and the glass was burned. The team did not hold anyone accountable afterwards, because the highest priority in all emergency guides is "life first". This principle is written on the first page of the guide, but many people forget it when they memorize the process.

To be honest, I have been a "dogmatist" who clings to guides before. When I first joined the company, I encountered a server alarm. I saw that the disk was almost full, but I still followed the process and asked the leader to sign for approval before applying for capacity expansion. It was delayed for 20 minutes, resulting in the loss of all the uploaded materials of a group of users, and finally half a month's performance was deducted. It was then that I realized that the guide is to give you a bottom line, not a frame - just like the question bank for a driver's license test. You can't go straight to the highway and drive 120 yards after memorizing all the traffic regulations. But if you dare to hit the road without even memorizing the traffic regulations, something will happen sooner or later.

Later, when our team revised the internal emergency guide, we deliberately added a page of "priority determination table" at the front: user life safety > core data security > business continuity > process compliance. As long as all operations meet higher priority requirements, they can be "processed first and then supplemented." Last year, our emergency response time was shortened by an average of 42%, and there were no secondary accidents caused by random operations.

Of course, this does not mean that the process can be skipped casually. Last month, a young man who just joined the company encountered a cache service alarm and restarted without taking a snapshot according to the instructions. He lost half of nearly 3 hours of user order data. In the end, the whole team stayed up all night to restore from the backup. This loss was painful enough.

I recently sorted out last year’s emergency incident ledger and found that less than 30% of the incidents could really follow the guidelines 100%, and the remaining 70% required people on site to make adjustments based on the actual situation. To put it bluntly, the guide is dead and people are alive. You have to engrav the guide into your mind before you dare to jump out of the guide to solve the problem at the critical moment. Don’t treat the guide as a bible that cannot be changed, and don’t treat the guide as useless waste paper. This is the most practical experience I have gained from reading more than ten guides and handling dozens of emergency incidents.