← All Build Notes
Operations & Infrastructure Cross-Project Experience

Operating Web Software in Production: Practical Architecture & Reliability Lessons

Field notes on systemd service isolation, Gunicorn, PM2, Nginx reverse proxies, database persistence, and retaining human accountability in AI-assisted development.

1. The Reality Gap: Localhost vs. 24/7 Production

Writing software that runs correctly on a local development laptop is only the first step. The real test of an engineering stack begins once code is exposed to real users, varied network conditions, memory limits, and server operations.

Across projects like VScanner (live financial scanning), Jemo (restaurant operations), Netakit (graphic generation), and Chanakya (market risk terminal with over 3 million historical records loaded), maintaining reliable uptime requires treating deployment and operational maintenance as primary considerations.

2. Process Supervision and Reverse Proxy Architecture

A production web service must handle unexpected process terminations or worker interruptions gracefully. Practical architectural patterns for structuring these layers include:

Nginx as the Reverse Proxy Boundary: Placing Nginx in front of application runtimes is an effective boundary pattern:

  • Handling SSL/TLS termination and certificate management.
  • Applying standard security headers such as X-Frame-Options: SAMEORIGIN, X-Content-Type-Options: nosniff, and strict referrer policies.
  • Buffering client requests to prevent slow client connections from tying up application workers.
  • Serving static assets directly, reducing workload on application processes.

Process Supervision with systemd and PM2:

  • For Python WSGI applications (such as Gunicorn) and background task runners (such as Redis Queue workers), encapsulating processes as isolated systemd unit services allows automatic restart upon unexpected termination and centralizes logging via journalctl.
  • For Node-based services, process managers like PM2 offer convenient process monitoring, log management, and lifecycle controls.

3. Database Persistence and Operational Hygiene

Code can be redeployed quickly from version control, but handling state and persistent data requires careful discipline:

  • Connection Management: Web frameworks can quickly consume database connection limits if new connections are spawned unchecked. Tuning connection pools to match worker concurrency prevents database connection exhaustion under traffic bursts.
  • Index Verification: An unindexed query that runs instantly on a small test table can lock resources when operating against large datasets (such as Chanakya’s 3 million historical records). Reviewing execution plans and verifying composite indexes for common query filters avoids production query degradation.
  • Backup Routines: Implementing regular database dumps combined with offsite backups provides a reliable safety net. Periodic validation ensures that backup archives can be restored successfully if disaster recovery is needed.

4. Human Ownership in an AI-Native Engineering Workflow

Modern development tools like Claude and Codex allow rapid implementation and refactoring. However, AI assistance does not replace engineering accountability.

In my workflow, the division of responsibility remains clear:

  • AI Role: Accelerating syntax boilerplate, exploring implementation options, drafting tests, and diagnosing stack traces.
  • Human Developer Role: Defining requirements, choosing system architecture, verifying business logic, validating security and edge cases, and taking personal responsibility for deployed software.

When an operational issue occurs, an engineer who understands the system architecture and host environment can isolate and resolve the issue methodically.

Key Operational Takeaway

Reliable software operations rely on practical fundamentals: isolated process supervision, hardened reverse proxies, verified database indexing, consistent backup routines, and clear human accountability over system behavior.

Need help deploying, stabilizing, or operating your software?

I take applications from initial requirement through to live production deployment and ongoing operations. Let’s talk about your system requirements.