Troubleshooting Mindset Series #009 Don't Restart First: Capture the Evidence Before You Fix the Problem
  

rizzuan Lv2Posted 2026-Aug-11 17:55

One troubleshooting habit that has saved me many times is simple:
Don't restart first. Investigate first.
When a server, VM, network device, or HCI node appears to have a problem, the easiest reaction is often to restart it.
Sometimes the restart fixes the symptom.
But there is one major problem.
You may also destroy the evidence needed to understand what actually happened.
Why Restarting Too Early Can Be Dangerous
Imagine a VM suddenly becomes unreachable.
The first reaction might be:
"Let's restart the server."
After the restart:
✅ Server is reachable again.
Problem solved?
Not necessarily.
We still don't know what caused the problem.
Was it:

  • Network connectivity?

  • DNS?

  • High CPU?

  • Memory pressure?

  • Disk latency?

  • Application failure?

  • Service stopped?

  • Firewall session issue?

  • OS-level problem?

The restart may have removed the symptom without identifying the root cause.
My Approach: Evidence First
Before restarting, I try to capture as much information as possible while the problem is still happening.
For example:
1. Connectivity
Check:

  • Ping

  • Packet loss

  • Latency

  • Gateway reachability

  • DNS resolution

2. Server / VM
Check:

  • CPU usage

  • Memory usage

  • Disk usage

  • Disk latency

  • Running services

  • System event logs

3. Network
Check:

  • Interface status

  • Interface errors

  • Switch port status

  • VLAN

  • MAC address

  • Routing

4. HCI
Check:

  • Node status

  • VM status

  • Cluster alerts

  • Resource utilization

  • Storage health

  • Recent events

5. Firewall
Check:

  • Traffic logs

  • Denied sessions

  • Interface events

  • Routing changes

  • HA events

Only after collecting the relevant evidence should we decide whether a restart is necessary.
A Real-World Example
In my recent HCI troubleshooting case, the platform reported:
"Node Disconnected from Gateway"
It would have been easy to restart the affected node or start changing network configuration.
Instead, I followed the evidence:
HCI alert
→ Check node health
→ Check VM status
→ Identify gateway
→ Check FortiGate
→ Check physical connectivity
→ Inspect network cable
→ Reseat cable
→ Monitor
The cable was found to be slightly loose.
After reseating it, the environment was monitored for more than 48 hours and the same alert did not reappear.
No HCI restart was required.
The Lesson
A restart is an action.
It is not a diagnosis.
If the system is unstable, restarting may sometimes be necessary to restore service. But whenever possible, capture the evidence before performing the restart.
Otherwise, we may end up with:
Problem disappeared
but not:
Root cause identified
My Troubleshooting Rule
Observe → Collect Evidence → Isolate → Fix → Validate
Not:
Problem → Restart → Hope
That small change in mindset can make troubleshooting much more useful, especially in production environments.
Final Takeaway
Don't just fix the symptom. Understand why the symptom happened.
The best troubleshooting result is not simply getting the system back online.
It is being able to explain:
What happened → Why it happened → What was changed → How we know it is fixed.
How do you normally handle a production server or VM that suddenly becomes unreachable?
Do you investigate first, or restart first?

Like this topic? Like it or reward the author.

Creating a topic earns you 5 coins. A featured or excellent topic earns you more coins. What is Coin?

Enter your mobile phone number and company name for better service. Go

ND Lv3Posted 2026-Aug-12 13:52
  
If it's a single dev VM, you can afford to investigate for an hour. If it's a production node serving customer traffic and there's no redundancy, the calculus shifts — SLA pressure sometimes forces the restart before the diagnosis is complete, and that's a legitimate tradeoff, not a failure of discipline.

In HA/clustered environments (like your HCI case), you often get investigation "for free" — failover keeps service up while the affected node stays in the exact broken state for as long as you need to examine it. That's without a doubt the biggest structural argument for designing redundancy: it turns "restart vs investigate" from a forced tradeoff into a non-issue.
Humayun Ahmed Lv4Posted 2026-Aug-11 18:49
  
Thanks to share!