Tech Blog
Engineering insights, debugging stories, and product updates from the OrangeLabs team.
Bug Check 0x7E While Unloading a Driver — A Minifilter Calling Forbidden Completion Processing in FltMgr's DRAINING State
Why the Space Key Inserts No Space While Typing Korean over Remote Screen Control — The Unicode Character Channel and IME Composition
Report Only Recurrences After Recovery — Compressing Repeated TCP and DNS Failure Detection with a SUID Flag Table
An Analyst That Investigates PC Response Delays, and File Distribution (Monthly Review 2026-11-01)
Seven-Stage Streaming for Analyst Calls in LiveAgent, a New Server-Only Node Dashboard, Resumable Installer Uploads, and Consolidating N+1 APIs on the Manager Node Screen — Weekly Review (2026-10-26)
An Agent Update That Never Finished After 15 Retries on a Slow Link — Implementing Resumable Downloads with HTTP Range
Symptom List Queries Taking Seconds — Moving the Pipeline onto MongoDB's SBE Engine by Relocating $addFields
Diagnosing Active Directory Logon Delays and Failures — Narrowing the Cause by Port from the PC Side
A Node Disconnecting Every Minute Once MQTT Messages Exceeded the Broker Limit — Filtering Ahead of Time with CONNACK Maximum Packet Size
Launching the Agent's Active Analyst, Starting the Team-Lead Restructure, and Folding In 22 Missing Features from the Manager Network-Zone Redesign — Weekly Review (2026-10-19)
81% of 215 Blue-Screen Dumps Flagged as Bad Memory — It Was a False Positive from Running Checksum Verification Without Symbols
Finding the COM Caller's PID Without Symbols — A ServerCall vtable Table by combase Version
Blank User and Department Names in the Manager Alert Window on Repeat Symptoms — How a Lightweight detect2 Message Overwrote the Whole Grid Row
Process Handles Leaking in Step With Short-Lived Processes — A Reap Path That Permanently Exempted Anything Missing a Stop
Lazy Rule Loading in the Agent, Kernel Driver Bugcheck Fixes, and a 59% Smaller Remote-Control Binary on mbedTLS — Weekly Review (2026-10-12)
A Kernel Driver Bugchecking on Process Creation Under Memory Pressure — 0x1E From Reading a NULL Buffer Unchecked
A Minifilter Spiking CPU in File-Heavy Programs — Pinning PreCreate Filename Lookup at 28% With WPR and xperf
Cutting the Remote-Control Binary by 59% — Swapping OpenSSL for mbedTLS in the DTLS-SRTP Backend
A Minifilter Loading Quietly Even When Kernel Callback Registration Failed — The Per-Type Limit of 64 and a Monitoring-Free Load
A 0x4A Bugcheck From a Minifilter Returning While Still Holding a FAST_MUTEX — Lock Leak on the __leave Exception Path
Stabilizing orange.sys Kernel Driver Bugchecks and Handle Leaks, Tracing Kernel Crash Paths in the Dump Analyzer — Weekly Review (2026-10-05)
One Process-Kill Event Stopping the Agent's Entire ETW Intake — Per-Build Field Differences and a TdhGetProperty Exception
Picking the Minidump Even When a Kernel Dump Exists — Five Confirmed Defects in BSOD Dump Selection
A Detection Says the TCP Connection Took Over Three Seconds — Can You Trust the Number? Kernel Timestamps Instead of Processing Time, and SYN Retransmission
Precision for Multi-Target Commands, Root-Cause Analysis of Anomalous Nodes, and Command Retention Management — Monthly Review (2026-10-01)
WebRTC Both-Relay Remote Control Completed, Spread Prediction as a Service, Switch-Port Observation, Command Precision — Quarterly Review (2026 Q3)
Identifying Kernel Dump Crash Causes Without Symbols, and Browser Internet Connectivity Diagnosis — Weekly Review (2026-09-28)
The Agent Going Fully Offline When the Windows Terminal Service Hangs — sc start 1053 and an Unbounded ncalrpc Wait
A Lost Auth Response Splits a VM Node Into a New Document — Restoring the ticketb Comparison and Blocking Stale Ticket Writebacks
A Graceful Disconnect Discards the MQTT WILL and the Node Stays Stuck Online — paho Reason Codes and Dropping Out Mid-Reauthentication
Load Index Recast as Time a Person Waited, and Duplicate Nodes from Cloned Images — Weekly Review (2026-09-21)
Two PCs from the Same Disk Image Sharing One Node Document — Making Matching and Ticket Rotation Atomic in MongoDB
Installer Downloads Failing on the Stock Edge in Windows 10 1507 — Rolling async, fetch and CSS Variables Back to ES5
No Drive Selection Dialog on PCs with Hard-Recovery Software — Relaxing the NSIS Check to a Vendor Folder
Recalculating the Process Load Index and a Retention Policy for Repeating Commands — Weekly Review (2026-09-14)
90 Days for Late Arrivals, 5 Minutes for the Future: Why We Made Agent Clock Skew Asymmetric
How a MongoDB Regex Search Tied Up a Worker for 28 Seconds and Dropped Results Too
Fixing a MongoDB Query Where One Document Blocked the Whole List and Items Shifted Between Pages
Monthly Development Review — August 2026 (Aug 1–31)
Group Commands Reaching the Wrong PCs, and Swapping the Local LLM Engine — Weekly Review, 2026 Week 37 (Aug 31–Sep 7)
Endpoint Collection Must Continue Even When the Server Is Down — Removing a 13-Second Synchronous Server Registration Wait at Agent Startup
Why Resident-Process Telemetry Goes Missing on the Server — A Full Analysis of Five Combined Causes, Four in the Agent and One in the Server
Boot Without a Network and coturn Never Comes Back — systemd Retry Limits and a Docker Bridge Trap
A 1.4-Second WAL Checkpoint Landing at the Busiest Moment of Boot — Skipping the First Tick of a Periodic Loop
Zero Rows Deleted, 6.4 Seconds Spent — A SQLite <> Full Scan and a Cleanup Batch Repeating on Every Restart
Indexed and Still 1.2 Seconds Every Start — Catching a Dead Backfill Query with a Partial Index
Weekly Development Review — 2026 Week 36 (Aug 24–31)
The Rule Engine Said 900 Was Greater Than 1000 — Numeric Rules Compared as Strings
Putting a TTL on an Ever-Growing Time Series — Splitting First-Seen and Identity into a Dimension Table
Normal Traffic Erasing Detections, Normal RSTs Flooding Them — Two Fixes for ETW TCP Noise
In-House WebRTC Remote Control, the Server Data Pipeline, RBAC Across 68 Widgets, and Promoting Symptoms to Issues — Quarterly Review (2026 Q3)
Weekly Development Review — 2026 W34 (Aug 17–23)
Deleted Nodes Coming Back on Screen — Cascade in Aggregate Collections and the Rows That Resurrect
When Dump Analysis Blames ntdll.dll — Walking the Call Stack from the Exception Context
Misreading DNS ETW Event IDs — A 6.6× Detection Surge and a Failure Path That Was Dead
Windows Firewall Blocked It but Nothing Was Detected — The Domain Field Trap in ETW TCP Events
Finding the Real Culprit Driver Behind a 0x9F Bugcheck — Identifying the Miniport by Walking the Device Stack
The Checkbox Only Hid a Tooltip — "Disable Auto-Logout on Mobile" Never Actually Stopped the Logout
Manager Sessions Piling Up Without Expiry — Fixing It With an Idle Floor and an Absolute Timeout
Antivirus Flagged Our Watchdog as a Dropper — Removing PowerShell EncodedCommand in Favor of Native Execution
Late Telemetry Vanishing From Hourly Aggregates — A Four-Step Re-Aggregation Ladder Matched to the Arrival Curve
Remote Control Going Blurry With CPU at 100% — Stuck in Software Encoding After Missing an MF_E_TRANSFORM_STREAM_CHANGE Renegotiation
Sorting 305 Field Blue Screens Into Seven Categories — BSOD Categorization and the Measured Distribution
In-House Remote Screen Control Completed, coturn Standardized With Layered Connection Paths, Signal Command and Node Asset Bundle API, MQTT Topic Isolation — Weekly Review (Aug 3–10, 2026)
Turning Lightweight Commands Into Infrastructure — A Standard Path With No Result Aggregation, and the Principle That Lightweight ≠ Careless
Mass Agent Loss From a Lost Write During Update Installation — Root Cause and Hardened Self-Recovery
Honoring the Trust Result in Code-Signature Checks — Trust the Signer Name Alone and Tampered Signatures Walk Through
Catching Unbounded SQLite WAL Growth — A Disabled Checkpoint and Size Monitoring Under zipvfs Constraints
Debugging a WebRTC Black Screen Behind Double NAT — Forcing a Relay Switch on Media, Not on a False P2P Success
Building Our Own WebRTC Remote Screen Control — DXGI Capture, MF H.264, libdatachannel, the Secure Desktop and a TCP Relay
Wiring Up AI Remote Diagnosis and In-House Remote Screen Control, Load Testing With Virtual Agents, Three-Tier MongoDB Instrumentation, Resolving the mosquitto FD-Exhaustion Crash Loop — Weekly Review (Jul 27 – Aug 3, 2026)
Server-Autonomous AI Remote Diagnosis, the Network Zone Location Engine, and a Root-Cause Fix for the orange.sys IRQL Bugcheck — Monthly Review (July 2026)
Why PowerShell Process Collection Intermittently Half-Failed — A Race After the Query and a Null in Split-Path
Minimizing Remote-Control CPU Load on Virtual GPU Nodes — Integer BGRA→NV12 Conversion Plus Dirty-Rect Partial Updates
Auto-Recovery Never Ran Once — The Recovery Installer Was Killing Its Own Parent
A mosquitto 2.1 Watchdog False-Positive Crash Loop — When FD Exhaustion at 1024 Silently Drops sd_notify
Duplicate PC Names and Delivery Optimization in a Master-Image Cloned Environment
Why a React Widget Panel Flickers on First Load — ResizeObserver Runs After Paint
Why Load-Metric OHLC Candles Don't Connect — Fixing Bucket Rollover to Carry the Previous Close
Why an Unattended Update Script Hangs on DB Migration — Tracing a Swallowed stdin Prompt
Builds Hanging on GitHub API Instability — Hardening the Python Bundle Step in a Batch Script
A Kernel BSOD Exposed by Driver Verifier — Why You Must Not Call ZwClose While Holding a Lock (0xC4 IrqlZwPassive)
Why a Workgroup PC Reports an Empty Domain Name — Discarding DsRole's DomainNameFlat for a Service-Dependent API
Why IPs and Versions Sort Backwards in AG Grid — A Codebase-Wide Fix for Missing Comparators
Why a VM Node Forks From Itself on Every Restart — Suppressing Ticket Rotation on the Shutdown POST
Same SUID for lsass, svchost, and Defender — An Exhaustive Analysis of 32-bit Hash Collisions
Tracking a Kernel Driver Handle Leak — Why __finally Doesn't Run in a Process Exit Callback
A Long Command Line Swallows the Whole Process Event — Capping Kernel Messages at 4KB to Stop 12KB Drops
Improving CPU Measurement Accuracy for Short-Lived Processes — Kernel/Agent Clock Sync and the PID→PUID Switch
A Process Identifier That Beats PID Reuse — Moving the Kernel Driver's PUID to the Windows StartKey
Adding Disk and Network I/O to System Performance Metrics — From Summed Process Approximations to Direct OS APIs
Measuring a Handle Leak in the Field — 16.71 Million Handles in Fingerprint Security Software (FMService64) and Detecting Memory-Heavy Processes
Version-Locking System Packages on Air-Gapped Ubuntu Servers + Offline Security Upgrades (.deb Closure Bundle + lock.json)
Local Icons Break After Switching to an https WebView2 Virtual Host — Fixed With a Second Virtual Host Mapping
The cmd echo Trap That Killed Builds With 401 After a Docker Desktop Update — A Space Before the --password-stdin Pipe
Post-Mortem of a Windows Kernel Freeze — Forcing a Memory Dump With an NMI Keyboard Trigger
Detecting Failed Deletion of a Running EXE (STATUS_CANNOT_DELETE) With a Minifilter — Identifying Deleter, Runner, and Launcher
Per-Chrome-Major PC Impact Analysis — Measuring Impact Across 150 (PC × Major) Pairs From 45 PCs and 9 Majors
A Self-Wait Regression Stalling Exactly 3 Seconds Per Process at Agent Startup — Analyzing an FltSendMessage Block
Hourly sprocess Aggregation With a Shared Rotate Entry Point, Blocking a 31k-SUID Surge With MQTT Subscriber-Driven Publish, and Establishing the Manager Web Design System — Weekly Review (2026-06-01 – 2026-06-08)
Detecting Zombie Processes in the Windows Kernel — Judging EPROCESS Survival and PID Reuse Safety
Impact Analysis of a New Software Rollout — Measuring How 980 AhnLab Processes Affected Resources and Symptoms Across 77 Company PCs
Four-Repo Implementation of the Process-Impact Widget, Migration to MongoDB Transaction Infrastructure, and a 40-Million-Fold File Event Throttle — Monthly Review (2026-05)
Working Around the Broker's OpenSSL 3.0.x Residual Path, Cleaning Up the Process-Impact Widget Across Four Repos, and a 40-Million-Fold File Event Throttle — Weekly Review (2026-05-25 – 2026-06-01)
Migrating MongoDB to a Single-Node Replica Set — Enabling Transactions With Zero Downtime
A Year of Problems Solved — BSOD Analyzer, DB Migration Framework, Manager RBAC, Kernel Driver Deadlock, and Manager Diagnostics — Annual Review (2025-05-26 – 2026-05-26)
Infinite 1006/1011 Reconnect Loops on FastAPI WebSocket Auth Failure — The Close-Before-Accept Trap and Sending 1008 Properly
DB Migration Framework, Five SNMP Monitoring Domains, the Manager Status Panel, and a Kernel Driver Deadlock — Quarterly Review (2026 Q2)
The wasClean Trap — Overnight WebSocket Loss and Per-Close-Code Reconnect and Logout Policy
Unrelated Nodes Drop Off Mosquitto — An OpenSSL 3.0.x Record Layer Bug (0A000123) and the Abrupt-Close Workaround
ResizeObserver Loop Limit Exceeded — Breaking the Cycle With setState Guards in a Multi-Widget Dashboard
Why EPROCESS SignatureLevel Is Zero — The Limits of MS Signature Detection for Medium-IL Processes, and an Authenticode Workaround
The Real Causes of Two Laptop Blue Screens — Intel Ethernet D3 Transition (0x9F) and WSL2 Memory (0x7E) in WinDbg
Why Mosquitto 2.0 Says 'Unable to Write pid File' — Privilege Drop vs. systemd Directory Ownership
Existing Applications Break After an Azure AD Migration — FILE_NOT_FOUND / ACCESS_DENIED From Account-Bound Files and Registry Keys
Which Process Killed Mine? — Tracing Forced Termination With ETW Kernel-Audit-API
A Field Guide to Diagnosing PC Slowdowns — Telling a Culprit Process From a Culprit Driver
Hidden Causes of Bluetooth Disconnection: A Guide to Tracing Battery Power Policies
Getting a Windows Kernel Driver MS-Signed (Attestation) — From Build to Partner Center Submission
A Git Flow + Jira Branch Strategy — Automatically Tracking Commits, Branches, and PRs by Issue Key
Tracing the Root Cause of High CPU Usage by WMIPrvSE.exe
Client PC Performance Analysis Report 2
Client PC Performance Degradation Case Analysis Report
Laptop CPU Overload and Wi-Fi Drops on Meeting-Room HDMI — Optimus (iGPU/dGPU) Switching and Battery Power Limits
A 43% Drop in Endpoint Load From Upgrading Antivirus (Symantec Endpoint Protection) — A Measured Comparison Across 49 Machines
The Hidden Cause of dxgmms2.sys CPU Usage — A Full Analysis of a Remote-Control Virtual Display (Mirage Driver)
How Do You Detect System (PID 4) CPU Usage? — Identifying Kernel Thread and Driver Load
Mirage Driver Found on 28 of 44 Machines — A Fleet Survey of How Remote-Control Leftovers Create dxgmms2.sys Load
A Comprehensive IT Incident Analysis Across 45 Corporate PCs — The Real Causes of CPU Usage, Forced Termination, and Blue Screens (May 2025)
A Focused Analysis of Antivirus (SEP) CPU Overuse — Recurring on 17 of 41 Machines (41.5%), With Module Distribution and Recommendations
An 'All-in-One Driver' Installer Brought In by USB and Flagged as Malicious — A Full Forensic Timeline
OrangeLabs | Endpoint Performance & Failure Management
Orange The Client — Endpoint Performance & Failure Management
Pricing
Orange The Client User Manual
Product Videos
Release Notes
Downloads
Tech Blog
About Us
Careers
Partners
Support
Contact Us
Privacy Policy
Terms of Service