Toward Co-Working with Autonomous Agents: Rethinking Operations of Automation Systems

The growing complexity of commissioning industrial automation systems has increased the demand for simpler and more intuitive operation, often requiring domain-specific expertise. This paper explores task-oriented natural-language interaction with automation systems, a paradigm that is rapidly gaining traction particularly due to the advent of large language models (LLMs). Leveraging a multi-agent system (MAS), the authors introduce a two-phase approach for controlling an industrial gantry for pick-and-place operations. In phase one, the MAS generates an execution plan based on the user input; in phase two, the plan is executed deterministically on real-world hardware. Our preliminary results demonstrate the advantages of a chat-based human-in-the-loop (HiL) workflow for refining and validating execution plans, and they highlight the promise of natural-language interfaces for operating pick-and-place systems (Demo: Video 1, Video 2).