Building LNMP v1.5: Adaptive Baselines, Differential RCA, and Topology Engine

In the first part of this series, we built the core engine of the Lightweight Network Monitoring Platform (LNMP) using Python and TimescaleDB, focusing on dual-state classification and honest SLA calculations. While that foundation solved uptime tracking, deploying it against routed infrastructure revealed four major diagnostic limitations: the dynamic reconvergence race, intermediate transit blind spots, cascading alert storms, and static latency thresholds. LNMP Version 1.5 addresses these operational challenges by adding Adaptive Statistical Baselines, pure-Python FHRP inspection, Differential Root Cause Analysis (RCA), and an in-memory DAG topology engine.

---

Four Operational Challenges Left by v1.0

During testing, Version 1.0 left four distinct operational blind spots:

1. The Diagnostic Reconvergence Race: Dynamic routing protocols like OSPF or BGP reconverge within seconds. By the time an engineer receives an alert and runs traceroute, the failure path has already rerouted, destroying the diagnostic evidence.
2. The Intermediate Transit Blind Spot: In v1.0, every monitored IP was treated uniformly. The engine could not distinguish between a Layer 2 switch port failure on the local subnet and a multi-hop transit failure across a WAN carrier.
3. Cascading Alert Storms: When an upstream core router failed, all downstream endpoints failed their ICMP checks simultaneously, triggering dozens of separate alerts without root-cause awareness.
4. The Flaw of Static Latency Alerting: Hardcoded thresholds (such as $> 50\text{ ms}$) caused alert fatigue during peak business hours while missing severe anomalies during quiet off-hours when baseline latency was much lower.

LNMP v1.5 resolves these issues through the following architectural upgrades:

Architectural Area	LNMP v1.0 (Baseline)	LNMP v1.5 (Upgraded Architecture)
Performance Baselines	Static percentage thresholds on successful pings.	Adaptive Statistical Baselines: TimescaleDB continuous aggregates + Z-Score ($Z > 3.0$) across 168 weekly hourly bins.
Incident Diagnostics	No diagnostic tracing; required manual terminal troubleshooting.	Concurrent Asynchronous Background Tracing: Edge-triggered on first failed sub-cycle packet with asyncio.Semaphore(5) throttling.
Path & Failure Analysis	Blackbox availability (UP vs DOWN).	Differential Root Cause Analysis (RCA): Side-by-side comparison of Live Failure Trace vs Last-Known-Online Baseline Snapshot.
Network Boundary Awareness	None (Treated all target IPs uniformly).	4-Tier Boundary Classification: Pure-Python /proc/net/route and /proc/net/arp inspection + FHRP Virtual MAC regex matching (HSRP/VRRP/GLBP).
Topology & Dependency Mapping	Flat list of independent endpoints.	In-Memory Directed Acyclic Graph (DAG): Single-vertex Trie/Tree deduplication, orphan pruning, and INFERRED_DOWN alert suppression.
Graph API Latency	Slow SQL joins on read requests.	$O(1)$ In-Memory Cache: Pre-serialized JSON served in $< 1\text{ ms}$ with 0 database queries.
Security & Vulnerability Patches	Basic token generation; unpatched dependencies.	Dependency CVE patches (form-data CRLF, Vite NTFS traversal, Axios prototype pollution), CSV formula injection sanitization, and active token revocation.
Identity & Access Governance	Single user role; basic password checks.	Dual-Tier RBAC (ADMIN vs OPERATOR), forced first-login password resets, self-lockout prevention, and minimum password complexity rules.
Database & ORM Reliability	Standard SQLAlchemy parameter passing.	Explicit PostgreSQL UUID casting (CAST(:id AS uuid)) across all models to eliminate lookup and type-coercion errors.
Frontend & UI Presentation	Basic Vue dashboard with canvas jitter.	Vue 3 + PrimeVue, Canvas Physics Stabilization (freezing layout after 200 iterations), Interactive Inspector Drawer, and Theme CSS Variables.
Operations & Deployment	Manual install script; manual service restarts.	Automated Zero-Downtime Upgrade Utility (upgrade.sh) with pre-upgrade PostgreSQL backups, migration runners, and dependency checks (libcap2-bin).

---

Adaptive Statistical Baselines: The 168-Bin In-Memory Cache

Why Static Thresholds Fail

Take, for instance, an engineer monitoring an enterprise branch connection:

• Tuesday at 2:00 PM: With high user concurrency, an average Round Trip Time (RTT) of $42\text{ ms}$ is normal.
• Sunday at 3:00 AM: With the facility idle, normal RTT drops to $8\text{ ms}$. If latency jumps to $40\text{ ms}$ on Sunday morning, an interface has taken a sub-optimal path or a transit circuit is congested.

A static threshold configured to alert at $> 50\text{ ms}$ misses the off-peak routing anomaly completely. If you tighten the threshold to $30\text{ ms}$, you generate false alarms every weekday afternoon.

Materializing Hourly Baselines with TimescaleDB

I initially tested computing moving averages directly in Python memory, but calculating rolling statistics across dozens of endpoints during fast polling loops introduced event loop jitter. Moving aggregation into the database layer proved far cleaner.

Using TimescaleDB Continuous Aggregates, LNMP materializes hourly baseline statistics across a weekly cycle, following the PostgreSQL Materialized View model:

CREATE MATERIALIZED VIEW node_historical_baselines
WITH (timescaledb.continuous) AS
SELECT
    endpoint_id,
    EXTRACT(DOW FROM start_time)::integer AS day_of_week,
    EXTRACT(HOUR FROM start_time)::integer AS hour_of_day,
    AVG(avg_rtt_ms) AS historical_mean,
    STDDEV(avg_rtt_ms) AS historical_stddev,
    COUNT(*) AS sample_count
FROM endpoint_events
WHERE operational_state = 'UP' AND avg_rtt_ms IS NOT NULL
GROUP BY endpoint_id, day_of_week, hour_of_day;

The Compact 1D Array Cache ($O(1)$)

To prevent database queries from stalling the 6-second sub-cycle loop, the daemon loads these baselines into a compact 1D array in memory. A full week contains exactly 168 discrete hours ($7\text{ days} \times 24\text{ hours}$):

$$\text{Index} = (\text{DayOfWeek} \times 24) + \text{HourOfDay} \quad (0 \le \text{Index} < 168)$$

from typing import Optional

DEFAULT_STDDEV_MS = 15.0
DEFAULT_MEAN_MS = 50.0

def calculate_z_score(latency_ms: float, mean_ms: float, stddev_ms: float) -> float:
    """Calculate the number of standard deviations from the historical mean."""
    safe_stddev = stddev_ms if (stddev_ms is not None and stddev_ms > 0) else DEFAULT_STDDEV_MS
    return (latency_ms - mean_ms) / safe_stddev

def is_latency_degraded(
    latency_ms: Optional[float], 
    mean_ms: float, 
    stddev_ms: float, 
    k: float = 3.0
) -> bool:
    """Evaluate whether the current latency represents a 3-sigma anomaly."""
    if latency_ms is None:
        return False
    safe_stddev = stddev_ms if (stddev_ms is not None and stddev_ms > 0) else DEFAULT_STDDEV_MS
    return latency_ms > (mean_ms + (k * safe_stddev))

• Direct Index Lookup: Resolves (day_of_week * 24) + hour_of_day in $O(1)$ time with zero database queries.
• Z-Score Anomaly Evaluation: Evaluates $Z = \frac{x - \mu}{\sigma}$. Based on standard NIST Statistical Guidelines, a $Z > 3.0$ threshold flags measurements sitting outside $99.7\%$ of expected historical values for that exact hour.
• 7-Day Onboarding Grace Period: For endpoints onboarded less than 7 days ago, the cache falls back to baseline constants ($\mu_0 = 50.0\text{ ms}, \sigma_0 = 15.0\text{ ms}$) until sufficient operational data accumulates.

Symbol	Definition	Role in Telemetry Evaluation	Example Value
$x$	Live Measurement	Round Trip Time (RTT) in milliseconds recorded during the current poll cycle.	$42.0\text{ ms}$
$\mu$	Historical Mean	Mean latency for this node during this specific hour of the week across past cycles.	$8.2\text{ ms}$ / $42.0\text{ ms}$
$\sigma$	Standard Deviation	Expected variation of latency around the mean for this specific hour.	$1.4\text{ ms}$
$k$	Sigma Multiplier	Anomaly sensitivity factor ($k = 3.0$ establishes a 99.7% confidence boundary).	$3.0$
$Z$	Z-Score	Standardized deviation of the current measurement: $Z = \frac{x - \mu}{\sigma}$.	$> 3.0 \implies \text{Anomaly}$
$\tau$	Dynamic Threshold	Anomaly cutoff ceiling: $\tau = \mu + (k \cdot \sigma)$. An alert triggers when $x > \tau$.	$\mu + (3.0 \cdot \sigma)$

Example 1: Evaluation During the 7-Day Warmup Period

Before 168 hours of operational data are collected, the engine evaluates probes against fallback defaults ($\mu_0 = 50.0\text{ ms}$, $\sigma_0 = 15.0\text{ ms}$, $k = 3.0$):

$$\tau_{\text{warmup}} = \mu_0 + (k \cdot \sigma_0) = 50.0 + (3.0 \times 15.0) = 95.0\text{ ms}$$

• Normal Reading ($x = 65.0\text{ ms}$): $$Z = \frac{65.0 - 50.0}{15.0} = +1.0 \implies 65.0\text{ ms} \le 95.0\text{ ms} \quad (\textbf{NORMAL})$$
• Congestion Spike ($x = 110.0\text{ ms}$): $$Z = \frac{110.0 - 50.0}{15.0} = +4.0 \implies 110.0\text{ ms} > 95.0\text{ ms} \quad (\textbf{DEGRADED})$$

Example 2: Evaluation with Hourly Adaptive Telemetry

Once 7 days of historical baselines are materialized in the 1D array:

• Off-Peak Weekend Probe (Sunday 03:00, Index 3):

  • Baseline: $\mu = 8.2\text{ ms}$, $\sigma = 1.4\text{ ms}$
  • Cutoff: $\tau = 8.2 + (3.0 \times 1.4) = 12.4\text{ ms}$
  • Measured Latency ($x = 22.0\text{ ms}$): $$Z = \frac{22.0 - 8.2}{1.4} = +9.86$$
  • A static $50.0\text{ ms}$ threshold misses this off-peak routing detour ($22.0\text{ ms} < 50.0\text{ ms}$). LNMP detects that $22.0\text{ ms} > 12.4\text{ ms}$ ($Z = 9.86 > 3.0$) and immediately flags the anomaly.

• Peak Business Hours Probe (Tuesday 14:00, Index 62):

  • Baseline: $\mu = 42.0\text{ ms}$, $\sigma = 4.5\text{ ms}$
  • Cutoff: $\tau = 42.0 + (3.0 \times 4.5) = 55.5\text{ ms}$
  • Measured Latency ($x = 48.0\text{ ms}$): $$Z = \frac{48.0 - 42.0}{4.5} = +1.33$$
  • Because $48.0\text{ ms} \le 55.5\text{ ms}$ ($Z = 1.33 \le 3.0$), the system accepts the measurement as expected peak concurrency and avoids false alerts.

---

4-Tier Boundary Classification and Pure-Python FHRP Detection

Unprivileged Linux Kernel Inspection

Rather than requiring SNMP community strings or root privileges, LNMP inspects the Linux kernel state directly via the /proc filesystem:

• Default Gateway Extraction: Reads /proc/net/route to parse the host server’s active default gateway and interface.
• ARP Resolution: Reads /proc/net/arp to obtain live IP-to-MAC mappings for all Layer 2 adjacent hardware.

def get_default_gateway() -> Optional[str]:
    """Parse /proc/net/route to extract the default gateway IP in pure Python."""
    try:
        with open("/proc/net/route", "r") as f:
            for line in f.readlines()[1:]:
                fields = line.strip().split()
                if fields[1] == "00000000": # Destination 0.0.0.0 (Default Route)
                    gateway_hex = fields[2]
                    # Convert little-endian hex to dotted-decimal IPv4
                    gateway_ip = ".".join(
                        str(int(gateway_hex[i:i+2], 16)) for i in (6, 4, 2, 0)
                    )
                    return gateway_ip
    except Exception:
        return None
    return None

Decoding Virtual Router MAC Addresses

In redundant enterprise designs, default gateways run Cisco HSRP (RFC 2281), VRRP (RFC 5798), or Cisco GLBP. For a practical look at how these redundancy protocols interact with Layer 2 switching, see our earlier lab on HSRP and STP high-availability design.

To recognize virtual gateways and avoid false alerts when a failover shifts a virtual IP to a standby chassis, LNMP matches MAC addresses against compiled regular expression patterns:

import re

FHRP_PATTERNS = [
    ("HSRP_V1", re.compile(r"^00:00:0c:07:ac:([0-9a-f]{2})$", re.IGNORECASE)),
    ("HSRP_V2", re.compile(r"^00:00:0c:9f:f([0-9a-f]):([0-9a-f]{2})$", re.IGNORECASE)),
    ("VRRP_IPV4", re.compile(r"^00:00:5e:00:01:([0-9a-f]{2})$", re.IGNORECASE)),
    ("VRRP_IPV6", re.compile(r"^00:00:5e:00:02:([0-9a-f]{2})$", re.IGNORECASE)),
    ("GLBP", re.compile(r"^00:07:b4:([0-9a-f]{2}):([0-9a-f]{2}):([0-9a-f]{2})$", re.IGNORECASE)),
]

def classify_mac_address(mac_address: str) -> Optional[str]:
    """Identify if a MAC address belongs to an active FHRP virtual gateway."""
    clean_mac = mac_address.strip().lower()
    for protocol, pattern in FHRP_PATTERNS:
        if pattern.match(clean_mac):
            return protocol
    return None

The 4-Tier Classification Model

1. Tier 1 (L2_LOCAL_HOST): Target IP exists in the local ARP table or shares the local broadcast subnet.
2. Tier 2 (L2_L3_GATEWAY_FHRP): Target IP is an active virtual gateway running HSRP, VRRP, or GLBP. LNMP tracks Gateway MAC Drift to detect hardware failovers without raising false outage alerts.
3. Tier 3 (L2_L3_GATEWAY_DEFAULT): The host server’s active default gateway.
4. Tier 4 (L3_ROUTED_TRANSIT): Multi-hop destinations traversing routed campus backbones, remote sites, or WAN transit carriers.

---

Differential Root Cause Analysis: Catching the Failure Path

Baseline vs Failure Trace Differential Logic

When an outage occurs, a standalone traceroute full of timeout asterisks (* * *) only confirms that packets failed; it does not indicate where they were supposed to go.

LNMP v1.5 automates path differential analysis by comparing live failure traces against healthy baseline snapshots:

1. The Golden Baseline Snapshot: When an endpoint is healthy (UP), LNMP captures an asynchronous trace and stores the known-good path in PostgreSQL (refreshed automatically during scheduled midnight discovery passes).
2. The Edge-Triggered Failure Trace: The moment an endpoint fails its first sub-cycle packet, the diagnostic worker triggers an immediate live failure trace before routing tables reconverge.
3. Differential Alignment: The RCA engine performs a sequential array comparison:
   • Layer 2 Isolation: If is_l2_segment == True or hop count is 1, the failure is immediately tagged as a local host interface, cable, or switch access port issue.
   • Layer 3 Divergence Point: For multi-hop paths, the engine finds the divergence hop $K$:

$$\text{Hop } K \implies \mathbf{FAILURE\_POINT}, \quad \text{Hop } K-1 \implies \mathbf{LAST\_KNOWN\_GOOD}$$

Automatic Recovery Handling

When an endpoint recovers from DOWN back to UP, handle_endpoint_recovery() marks the active RCA incident as resolved (is_resolved = True), logs the total incident duration, and triggers a fresh baseline discovery pass to record any permanent post-convergence route modifications.

---

In-Memory DAG Topology and Alert Storm Suppression

Trie Graph Merging in RAM

Individual traceroutes produce linear hop lists. If you monitor 50 endpoints behind a single distribution router, naive visualization produces 50 duplicate representations of that router.

The TopologyGraphManager merges linear traces into an in-memory Directed Acyclic Graph (DAG):

• Single-Vertex Deduplication: When an intermediate hop IP matches an existing monitored endpoint, the edge connects directly to that node’s vertex.
• Anonymous Node Sanitization: Consecutive timeout hops (* * *) are collapsed into a single deterministic anonymous vertex (anon_prevHop_to_epId), keeping the topology clean.
• Ghost Pruning: When an endpoint is deleted or its baseline updates, orphaned transit vertices and stale edges are dynamically pruned from RAM.

The INFERRED_DOWN Suppression Algorithm

Picture a case where an intermediate distribution router $R_T$ loses power: all downstream switches connected behind it will simultaneously fail their ICMP checks.

LNMP resolves this at the graph level:

1. When outages occur, the RCA engine queries the in-memory DAG to identify shared upstream transit parents.
2. If $100\%$ of the monitored child nodes behind transit parent $R_T$ are unreachable, the engine marks $R_T$ as the root failure and tags the children as INFERRED_DOWN.
3. Outage notifications for the child nodes are suppressed. The operations team receives a single actionable notification identifying $R_T$ as the root cause, rather than dozens of cascading alerts.

$O(1)$ Zero-Query API Delivery

To keep the web interface responsive, the complete DAG topology is pre-serialized in RAM. When a user opens the Topology view, GET /api/v1/topology serves the full JSON payload directly from memory in less than $1\text{ ms}$, executing zero SQL queries against the database.

---

Concurrency Protection and Ephemeral Storage

Triggering automated diagnostic traces at scale introduces resource contention risks. If 50 endpoints drop offline simultaneously during a fiber cut, launching 50 parallel traceroute sub-processes would quickly exhaust host CPU cycles, file descriptors, and raw ICMP sockets.

LNMP implements three protective guardrails:

import asyncio

# Limit concurrent diagnostic traces to 5 sub-processes
traceroute_semaphore = asyncio.Semaphore(5)

async def execute_throttled_trace(target_ip: str) -> dict:
    async with traceroute_semaphore:
        proc = await asyncio.create_subprocess_exec(
            "traceroute", "-n", "-q", "1", "-w", "1", target_ip,
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE
        )
        stdout, _ = await proc.communicate()
        return parse_traceroute_output(stdout.decode())

1. asyncio.Semaphore(5) Throttling: Caps concurrent diagnostic sub-processes to 5, queuing additional trace requests until running sub-processes complete.
2. Non-Blocking Background Discovery: When an operator adds a new endpoint, the HTTP API returns an immediate 201 Created in under $50\text{ ms}$. Initial route discovery runs asynchronously in a background worker queue with $500\text{ ms}$ inter-probe spacing.
3. 14-Day Ephemeral Retention: Diagnostic traces are stored in a dedicated PostgreSQL table (endpoint_diagnostic_traces) as raw JSONB. A background task purges records older than 14 days, preventing diagnostic data from consuming disk space on the primary TimescaleDB partition tables.

---

Codebase Hardening and Core Bug Fixes

Transitioning from v1.0 to v1.5 involved addressing several practical database and framework edge cases:

Bug Issue	Technical Resolution
PostgreSQL UUID Type Coercion Error	Added explicit CAST(:id AS uuid) across all raw SQL queries in auth, endpoints, reports, and topology services.
FastAPI Parameter Collision	Fixed naming collision where Pydantic LoginRequest model shadowed Starlette’s Request object in the auth router.
Cyclic Parent Relationship Traversal	Added cycle detection with a visited set in manual_parent_id assignment to prevent infinite loops ($A \rightarrow B \rightarrow C \rightarrow A$).
Date Range Parsing & Timezone Drift	Standardized parse_datetime_param() with host local timezone and enforced a strict 730-day max query window.
Traceroute Output Format Parsing	Fixed regex parsers for both traceroute and tracepath outputs; added trailing timeout stripping and terminal hop validation.
Self-Reset Administrative Lockout	Added backend validation preventing admins from inadvertently disabling or self-resetting their own active credentials.

Explicit PostgreSQL UUID Casting

In raw SQL queries executed via SQLAlchemy, passing Python UUID objects without explicit casting caused PostgreSQL to throw operator does not exist: uuid = character varying errors during table joins.

In v1.5, all raw SQL queries enforce explicit database-level type coercion:

SELECT id, name, target_ip, operational_state 
FROM endpoints 
WHERE id = CAST(:endpoint_id AS uuid) AND is_deleted = false;

Cyclic Dependency Protection

When operators manually assign parent dependencies (manual_parent_id), accidental configuration loops can occur (such as Router A $\rightarrow$ Router B $\rightarrow$ Router C $\rightarrow$ Router A). Without validation, recursive graph traversal causes stack overflow errors.

v1.5 adds cycle validation using depth-first search with a visited set before any parent assignment is committed:

def validate_no_parent_cycle(endpoint_id: str, proposed_parent_id: str, db_session) -> bool:
    """Ensure assigning proposed_parent_id does not introduce a cyclic loop."""
    visited = {endpoint_id}
    curr = proposed_parent_id
    
    while curr is not None:
        if curr in visited:
            raise ValueError(f"Cyclic dependency detected: Node {curr} loops back to {endpoint_id}")
        visited.add(curr)
        curr = get_parent_id_from_db(curr, db_session)
    return True

---

Security Vulnerability Remediation and Access Governance

Version 1.5 also includes a comprehensive security audit of both frontend dependencies and backend authentication logic:

Vulnerability / Threat	Security Fix Implemented
Form-Data CRLF Injection (CWE-93)	Upgraded form-data to 4.0.6 (GHSA-hmw2-7cc7-3qxx).
Vite Path Traversal Vulnerability	Upgraded vite to 6.4.3 (GHSA-7w7f-f2mp-qvrm).
Axios Prototype Pollution	Upgraded axios to 1.18.1 (GHSA-gcfj-64vw-6mp9).
CSV Formula Injection Attacks	Telemetry streaming export automatically sanitizes leading control characters (=, +, -, @, \t, \r) with single quotes.
Zombie Session JWT Exploitation	Enforced live database user status validation on every token request; disabled users receive immediate 401 invalidation.
Initial Credential Governance	Enforced mandatory password changes on first login (must_change_password) and 8+ character length validation.

Neutralizing CSV Formula Injection

LNMP allows operators to stream historical telemetry data as downloadable CSV files for executive reports. As outlined in the OWASP CSV Injection guide, if an endpoint is named with spreadsheet formula triggers (=, +, -, @), opening the exported file in Excel could execute arbitrary commands on the user’s workstation.

In v1.5, the streaming CSV export automatically sanitizes formula characters:

def sanitize_csv_value(val: str) -> str:
    """Prefix dangerous formula initiation characters with a single apostrophe."""
    if not isinstance(val, str):
        return val
    stripped = val.strip()
    if stripped and stripped[0] in ("=", "+", "-", "@", "\t", "\r"):
        return f"'{val}"
    return val

Active JWT Validation and Two-Tier RBAC

Stateless JWT tokens present a well-known operational risk: if an administrator disables an operator account, existing unexpired tokens continue to work until their expiration time.

In v1.5:

• Every authenticated API request inspects the database to confirm is_active == True. If a user account is deactivated, all existing tokens receive an immediate 401 Unauthorized.
• We established strict Two-Tier RBAC: ADMIN accounts manage users, baselines, and configuration, while OPERATOR accounts have read-only access with manual diagnostic triggers.
• Newly provisioned accounts require an immediate password update on first login (must_change_password = True).

---

Frontend Physics Stabilization and UI Engineering

Taming Canvas Physics Jitter

LNMP uses vis-network to render the interactive Directed Acyclic Graph topology.

In early UI builds, force-directed graph physics caused an issue: every time a 60-second polling cycle completed and node colors refreshed, the physics engine recalculated spring repulsion. The entire topology map would shake and rearrange itself across the screen while an engineer was trying to inspect a node.

To fix this, we stabilized the layout on initial mount:

// Stabilize network coordinates and disable physics permanently
network.on("stabilizationIterationsDone", () => {
  network.setOptions({ physics: { enabled: false } });
  isGraphStabilized.value = true;
});

• The layout engine computes physical node positions for exactly 200 iterations during initial mount.
• Once stabilizationIterationsDone fires, physics calculations are disabled permanently.
• Node coordinates lock firmly in place. Subsequent ICMP status updates and state color shifts update in real time without causing the canvas to move.

The Node Inspector Drawer and Semantic Theming

• Node Inspector Drawer: Clicking any node in the topology opens a slide-out drawer presenting hardware classifications, live latency metrics, and the side-by-side Differential RCA comparison table.
• Semantic CSS Variables: Replaced hardcoded hex values with CSS custom properties (--bg-surface, --text-primary, --border-color, --status-up, --status-down). A MutationObserver on document.documentElement dynamically synchronises the canvas background and label fonts when toggling between dark and light themes.

---

Zero-Downtime Upgrade Pipeline

Upgrading production monitoring infrastructure should not require manual database surgery or service downtime. LNMP v1.5 includes an automated upgrade script (deploy/upgrade.sh) designed for enterprise Linux servers:

How the Upgrade Process Works

1. Pre-Upgrade Backup: Dumps the active PostgreSQL database into a timestamped archive (/var/backups/netmon/backup_pre_v1.5_TIMESTAMP.sql).
2. Capability Verification: Verifies the presence of traceroute and libcap2-bin. It grants Linux network capabilities (setcap cap_net_raw+ep) to the virtualenv Python binary, allowing low-level ICMP and diagnostic execution without running backend daemons as root.
3. Database Migrations: Runs Alembic migrations (alembic upgrade head) to construct the new diagnostic tables, continuous aggregates, and indexes.
4. Asset Rebuild and Service Restart: Compiles the optimized Vue 3 production bundle and restarts the systemd services (netmon-backend, netmon-poller, nginx).

---

Conclusion & Ongoing Edge Cases

Building LNMP v1.5 demonstrates how time-series aggregation, unprivileged kernel inspection, and graph deduplication can turn a basic ping poller into a root-cause-aware diagnostic platform. There are still edge cases we are working through: asymmetric routing paths can occasionally make return traffic appear divergent when forward forwarding is normal, intermediate carrier ICMP rate-limiting requires careful heuristics to avoid false timeout hops, and expanding unprivileged route discovery across Linux VRF namespaces remains an ongoing effort.

The complete open-source codebase for LNMP v1.5, including the polling daemon, FastAPI backend, Alembic migrations, and Vue 3 frontend, is available on GitHub:

• Repository: https://github.com/dc8official/lnmp.git