How to Debug a Production Issue at a Customer Site in Under an Hour

The Situation Nobody Prepares You For

You are on-site at an enterprise customer. Their CTO is in the room. The integration your company shipped three weeks ago is throwing errors in production. Their team is losing revenue every minute it is down. They are looking at you.

This is the defining moment of a Forward Deployed Engineer's career. Not the demo. Not the implementation. This — the live production fire with a customer audience, unfamiliar infrastructure, and a clock running.

The engineers who handle this well are not necessarily the most technically brilliant people in the room. They are the ones with a repeatable process. They do not panic-grep through logs hoping to get lucky. They work a structured diagnostic sequence that finds the root cause fast, communicates clearly under pressure, and resolves or escalates with confidence.

This post is that process — the exact debugging sequence experienced FDEs use to go from "production is down" to "root cause identified and fixed" in under an hour.

🎯 Quick Answer (30-Second Read)

  • First move: Establish what changed in the last 24 hours — deployments, config changes, infrastructure events
  • Second move: Identify the blast radius — is this one customer, one region, one feature, or everything?
  • Third move: Read the logs from the error backward, not forward
  • Fourth move: Reproduce the error in a controlled way before touching anything in production
  • Key rule: Never make a change in production you cannot immediately revert
  • Communication rule: Update the customer every 15 minutes even if you have nothing new — silence is worse than bad news

The Mindset Before You Touch Anything

The biggest mistake FDEs make in a production fire is moving too fast. You open a terminal, start running commands, changing configs, restarting services — and 20 minutes later you have made the problem worse and lost the original error state.

Before you touch a single thing, spend three minutes establishing ground truth:

What is broken? Get one specific, reproducible description. "The integration is down" is not a problem statement. "POST requests to /api/sync are returning 502 with no response body since 09:14 this morning" is a problem statement you can debug.

What is not broken? Understanding the blast radius tells you where to look. If auth is working but sync is not, the problem is in the sync layer. If everything is down, the problem is in shared infrastructure.

What changed? Production systems do not spontaneously break. Something changed — a deployment, a config update, a certificate expiry, a third-party API change, a traffic spike. Find the change and you have found 80% of production issues.

The Debugging Sequence

Minute 0–5: Establish the Timeline

Your first question to the customer's team: "What changed in the last 24 hours?"

You are looking for:

  • New deployments (your product or theirs)
  • Configuration changes
  • Infrastructure events (scaling events, restarts, migrations)
  • Third-party dependency updates
  • Unusual traffic patterns

Pull the deployment log if you have access. Check your own product's release history for anything that went out in the last 48 hours. Check their cloud provider's health dashboard — AWS, GCP, and Azure all have public status pages that show regional incidents.

Most production issues are caused by a recent change. Finding the change is faster than finding the bug.

Minute 5–15: Read the Logs Correctly

Logs are read backward from the first error, not forward from the beginning. The error that caused the cascade is the first one chronologically — everything after it is downstream noise.

# Find the first occurrence of the error
grep -n "ERROR\|FATAL\|Exception" application.log | head -20

# Get context around the first error
grep -n "ERROR" application.log | head -1
# Note the line number, then:
sed -n '95,115p' application.log  # 20 lines around the first error

What you are looking for in the logs:

  • Timestamp of first error — does it correlate with a deployment or config change?
  • Error message — exact text, not a summary
  • Stack trace — the bottom of the stack is where the error originated, not the top
  • Request ID or correlation ID — follow one failed request through the entire system

If the customer's application has structured logging (JSON logs), use jq to filter:

cat application.log | jq 'select(.level == "error")' | head -20
cat application.log | jq 'select(.requestId == "abc-123")'

Minute 15–25: Check the Five Most Common Production Causes

While you are reading logs, run these checks in parallel if you have a second terminal or a second person helping:

1. Is the service actually running?

systemctl status your-service
# or
docker ps | grep your-service
# or
kubectl get pods -n your-namespace

2. Can the service reach its dependencies?

# Database connectivity
psql -h db-host -U user -c "SELECT 1;"

# External API reachability
curl -I https://api.dependency.com/health

# Redis/cache connectivity
redis-cli -h cache-host ping

3. Are environment variables and secrets correct?

# Check what the running process actually sees
cat /proc/$(pgrep your-service)/environ | tr '\0' '\n' | grep -i "api_key\|database_url\|secret"

4. Is the disk or memory full?

df -h          # disk usage
free -h        # memory usage
du -sh /var/log/*  # log directory size

5. Are there SSL/TLS certificate issues?

echo | openssl s_client -connect api.dependency.com:443 2>/dev/null | openssl x509 -noout -dates

Certificate expiry is responsible for more production outages than most engineers expect. It expires at a specific time, causes everything downstream to fail, and leaves an error message that is not always obvious about the cause.

Minute 25–40: Reproduce Before You Fix

Once you have a hypothesis about the root cause, reproduce the error in a controlled way before changing anything in production.

If the issue is an API call failing:

# Reproduce the exact failing request
curl -X POST https://customer-api.com/api/sync \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"test": true}' \
  -v 2>&1 | tee /tmp/debug-output.txt

The -v flag shows you the full request and response including headers — often the actual error is in a header the application is not logging.

If you can reproduce the error on demand, you can verify the fix on demand. Do not fix something you cannot reproduce — you will not know if the fix worked.

Minute 40–55: Fix With a Revert Plan

Every change you make in production needs a revert path defined before you make it.

If you are changing a config:

# Back up the current config before touching it
cp /etc/app/config.yaml /etc/app/config.yaml.backup-$(date +%Y%m%d-%H%M%S)
# Make your change
# Test immediately
# If it fails: cp /etc/app/config.yaml.backup-* /etc/app/config.yaml

If you are restarting a service:

# Know how to start it again before you stop it
systemctl restart your-service
# Watch the logs immediately
journalctl -u your-service -f

If you are rolling back a deployment — know the previous version tag before you start.

Minute 55–60: Verify and Document

After the fix, verify with the same reproduction steps you used to confirm the bug:

# Run the same curl command that was failing
# Check the logs for the same error — it should be gone
# Ask the customer to perform the action that was failing

Do not declare victory until the customer confirms from their side that the issue is resolved. Your curl test passing is not the same as their application working.

Communication During the Incident

This is as important as the technical work. A customer who is informed every 15 minutes feels significantly less pain than a customer who hears nothing for 45 minutes and then gets told it is fixed.

The 15-minute update format:

  • What you have found so far
  • What you are currently investigating
  • What the next step is
  • Estimated time to resolution if you have one

Even if you have nothing: "Still investigating — I've ruled out the database and the API gateway, currently looking at the authentication service. Will update in 15 minutes."

Never say "I don't know what's wrong" without a follow-up statement about what you are doing to find out.

The Right Approach vs The Wrong Approach

The right approach is working the sequence before touching anything. Establish ground truth, read logs correctly, reproduce the error, then fix with a revert plan. Document every command you run and every change you make — not for your own reference but because the customer's team will want a post-incident report and you will not remember the exact sequence under pressure.

The wrong approach is pattern-matching to a previous incident and jumping straight to the fix. "This looks like the certificate issue we had at the last customer" leads to changing the certificate when the real issue is a misconfigured environment variable. Fix the problem in front of you, not the problem you remember.

The other wrong approach is going silent while you debug. Customers in a production incident are not bothered by lack of progress — they are bothered by lack of communication. The silence feels like you are lost. The updates feel like you are in control.

My Take

The reason most engineers struggle with production debugging at a customer site is not technical ability — it is that the environment removes every safety net they rely on. No local reproduction, no staging environment, no familiar codebase, and an audience watching every command they type. The best FDEs I have seen work incidents treat the unknown environment as data rather than as a handicap — every unfamiliar piece of infrastructure they document as they go, every assumption they state out loud so the customer's team can correct them. The best outcome of a well-handled production incident is counterintuitive: the customer trusts you more after the incident than before it, because they watched you work calmly and systematically under pressure. The worst outcome is not failing to fix the bug — it is losing the customer's confidence by appearing reactive and disorganised even if you eventually find the issue. Right now, AI-assisted debugging is changing the FDE toolkit significantly — running a stack trace through Claude or feeding logs to an LLM as a first-pass triage step is cutting diagnostic time by 30–40% in practice. Where this is heading: FDEs who build personal runbooks of customer environment patterns and feed them to AI assistants will resolve incidents faster than those who rely on memory alone — the tooling advantage will compound the same way it does in software engineering.

The FDE Production Debugging Checklist

FIRST 5 MINUTES
☐ Get specific problem statement — not "it's broken", get exact error + timestamp
☐ Identify blast radius — what works, what doesn't
☐ Ask: what changed in the last 24 hours?
☐ Check cloud provider status page
MINUTES 5–15
☐ Find first error in logs (not latest — first)
☐ Get exact error message and stack trace
☐ Note timestamp of first error
☐ Correlate with any recent change
MINUTES 15–25
☐ Is the service running?
☐ Can the service reach its dependencies?
☐ Are environment variables correct?
☐ Is disk or memory full?
☐ Are SSL certificates valid and not expired?
MINUTES 25–40
☐ Form a hypothesis
☐ Reproduce the error on demand
☐ Confirm reproduction before fixing
MINUTES 40–55
☐ Define revert path before making any change
☐ Back up configs before modifying
☐ Make one change at a time
☐ Test after each change
MINUTES 55–60
☐ Verify fix with reproduction steps
☐ Get customer confirmation
☐ Note all commands run and changes made
THROUGHOUT
☐ Update customer every 15 minutes
☐ State what you have ruled out
☐ State what you are currently investigating
☐ Never go silent for more than 15 minutes

Frequently Asked Questions

What do you do if you cannot reproduce the error?

Intermittent errors that cannot be reproduced are almost always race conditions, resource exhaustion under specific load, or timing-dependent failures. Add more logging around the suspected area, set up an alert for the next occurrence, and monitor rather than blindly changing things. Making production changes to fix an error you cannot reproduce is how you create new problems.

How do you handle a situation where you do not have access to the customer's logs?

Ask immediately — this is a blocker and the customer needs to understand that. While waiting for access, work from the outside in: check what you can see (HTTP response codes, API gateway logs if you have them, your own product's outbound request logs). Document exactly what access you need and why so the customer can grant it fast.

What if the issue is caused by the customer's own infrastructure, not your product?

State it clearly but without blame: "Based on what I'm seeing in the logs, the issue appears to be in your authentication service rather than in our integration. Here's the specific evidence." Then offer to help diagnose even though it is outside your scope — the goodwill is worth more than the time.

How do you know when to escalate rather than keep debugging?

Escalate when you have been investigating for 30 minutes without a clear hypothesis, when the issue requires access or permissions you do not have, or when the fix requires a code change that cannot be deployed on-site. Escalating with a clear summary of what you have found is not failure — it is correct incident management.

How do you write a post-incident report after the fact?

Cover five things: timeline of the incident, root cause, how it was detected, how it was resolved, and what changes prevent recurrence. Keep it factual and blameless. The customer wants to understand what happened and feel confident it will not happen again — they do not want a confession or an excuse.

Conclusion

Debugging a production issue at a customer site in under an hour is not about being the fastest debugger in the room. It is about having a sequence that eliminates the noise quickly, communicates clearly under pressure, and fixes problems without creating new ones.

Work the sequence. Establish ground truth before touching anything. Read logs from the first error backward. Reproduce before you fix. Define the revert path before you make a change. Update the customer every 15 minutes regardless of progress.

The FDEs who become trusted by enterprise customers are not the ones who never have incidents. They are the ones who handle incidents in a way that makes the customer feel like they are in good hands.

Related reads: What Is a Forward Deployed Engineer · How AI Coding Agents Write and Debug Code Autonomously · CI/CD Explained: What It Is and Why Dev Teams Need It