Problem thinking and analysis process

I have encountered such problems in the process of applying for a job ： When the system fails , You check from top to bottom , Or from the bottom up

An interesting troubleshooting process

today , My colleague asked me to deal with a strange problem . He was in rsa When the private key configuration is normal , Can log in to most servers , Only one server cannot log in .

First , Think in terms of inertia , adopt ssh -vvv <hostanem> Direct view debug Information , Because most of the previous colleagues were unable to log in because they did not configure the private key or permission .

But it was later found that the problem was not in the private key , Then I log in to the server , Check the public key contents , Compare the public key of the server that can succeed , The content is the same .

Without a clue , I try to delete the public key , Resynchronize . An interesting problem was found in the process of synchronizing the public key , The public key cannot be written synchronously ~/.ssh/authorized_key, Of documents ownner It was modified .

Tread , Quickly correct the user permissions , Change a few random words from colleagues , Then try logging in again , Discovery still fails .

Now , Falling back into a clueless link , I will try to delete the entire user again , Try to re create the user . Now , The problem was finally exposed ：

sudo userdel -r tom
userdel: tom mail spool (/var/mail/tom) not found
userdel: /home/tom not owned by tom, not removing

user （ Is assumed to be tom） Of home The entire folder ownner Have been tampered with , This is what leads to ssh The reason why public-private key verification cannot be performed .

Subsequently amended ownner jurisdiction , Reconfirm with colleagues , This problem is caused by an operation .

Extended thinking

The whole problem investigation is complicated , Fortunately, it doesn't take up too much of my time , But this reminds me of my previous job search :

“ When the system fails , You check from top to bottom , Or from the bottom up ”

That's what I said ：

” From top to bottom , There are also bottom-up , Look at the alarm position , The alarm is at the front end , Check from top to bottom ; The alarm is in the database , Check from bottom to top “

I didn't have any ideas at that time , A random answer .

---

Now think about it. , With a new understanding .

When problems arise , You can go through the following steps From top to bottom Check and solve ：

1. Find the error point
2. You can use inertial thinking , common sense , Try a quick fix
3. If the problem is not solved , Try to play back the entire procedure , And add appropriate logs from the beginning to the end , analysis
4. Try from 0 To 1 Rebuild the files that each key point in the step depends on 、 Components etc.
5. Repetition goes deeper , Split multiple small keys with one key , Continue analysis

I haven't thought of “ Bottom up ” Analysis of the scene , Just to give an example , There may be a problem with the current end service , The back-end database also reports an error , At this time, the analysis may start from the database side , It is also from the bottom up .（ But the actual feeling is still used “ From top to bottom ” analysis , Because the affected point is the front-end service , The reason found for this is the database ）