Salman Munaf, who serves as a Site Reliability Engineer at TikTok, pointed out that AI Agents have undergone a transformation, evolving from basic large language model (LLM) invocations into complex distributed systems. Consequently, a distributed system mindset is indispensable during their construction. His key perspectives are as follows:

• AI Agents, functioning as coordinators of external systems, have reshaped system boundaries and risk landscapes.
• The context (state) that influences actions is susceptible to expiration and conflicts. Therefore, memory should be regarded as cache that is prone to failure and must be accompanied by source information.
• A model that is initially harmless can turn perilous if it gains the ability to execute unsafe operations.
• Instead of relying solely on logs, it is crucial to comprehensively record the decision-making rationale and actions of AI Agents.
• As probabilistic coordinators, the steps of AI Agents, such as planning, actions, and observations, all traverse system boundaries. Each step necessitates persistence and well-defined transaction boundaries.
• In the event of tool invocation failures, mechanisms like request IDs, idempotency keys, and status queries should be implemented. Retry storms can be prevented by capping the number of retry attempts and setting exponential backoff.
• Moreover, it is essential to establish permissions, budgets, and rate limits for AI Agents. Enhanced observability should also be put in place to ensure that their behavior can be constrained, monitored, and recovered from errors, with a clear tolerance level for system errors they might commit.