easily when User Interface (UI) elements shift or screen resolutions change. Furthermore, standard RPA cannot reason; it merely repeats. This paper proposes a solution: MSA (Mining Software Agents). We present a hybrid architecture that places "Guardrails" around Generative AI. By coupling an LLM with a formal planner and a vision-based execution engine, we create an agent that understands natural language but executes with mathematical determinism. This approach allows for Non-Intrusive Interoperability, enabling engineers to command complex software routines through a centralized dashboard, effectively democratizing automation for non-programmers. 2. SYSTEM ARCHITECTURE The MSA framework is built upon a decoupled microservices topology that separates reasoning (The Brain) from execution (The Body). This ensures scalability and security, as the reasoning engine can be deployed in the cloud or on-premise servers, while the execution agent remains local to the engineer's workstation. 2.1 The Cognitive Orchestrator (The Brain) Deployed as a Dockerized service, the Orchestrator acts as the central nervous system. It is built using FastAPI for high-performance asynchronous communication and PostgreSQL for persistent storage of the "Knowledge Base." • Knowledge Base: Unlike linear scripts, the system stores atomic knowledge. Using SQLAlchemy ORM models, we define Facts (states of the software, e.g., "Menu_Open") and Actions (tasks defined by Preconditions and Effects). • Symbolic Planner: The core intelligence is a STRIPS (Stanford Research Institute Problem Solver) algorithm implemented in Python. This planner receives a "Start State" and a "Goal State" and mathematically derives the optimal sequence of actions to bridge the gap, ensuring no logical violations occur. 2.2 The Haptic Agent (The Body) The Local Agent is a lightweight background service running on the Windows workstation. It exposes a local REST API that accepts atomic commands (e.g., click_on_image, type_text, wait). • Visual Perception and Coordinate Extraction: To overcome the brittleness of coordinate-based RPA, the agent employs a dynamic Computer Vision pipeline. When the Orchestrator commands an action, the Local Agent executes the following sequence: 1. Screen Capture: MSS captures the current state of the workstation monitor in real-time, converting it to a grayscale matrix to reduce computational load. 2. Template Matching: The agent uses OpenCV's cv2.matchTemplate() function utilizing the Normalized Cross-Correlation (cv2.TM_CCOEFF_NORMED) method. It slides the Visual Anchor (the reference image from the Knowledge Base) across the screen matrix to find the highest correlation. 3. Threshold Verification: If the maximum correlation value exceeds the predefined confidence threshold (typically above 0.85), the system extracts the bounding box coordinates (x, y, width, height). 4. Centroid Calculation: The agent calculates the centroid of the bounding box (x + width/2, y + height/2) to ensure the haptic injection (mouse click) occurs exactly in the center of the UI element, regardless of where the window is positioned on the screen.
RkJQdWJsaXNoZXIy MTM0Mzk2