The rise of large language models (LLMs) has accelerated the need to design meaningful human-agent interactions (HAI), ensuring agency, trust, and accountability are preserved while leveraging the capabilities of these systems. This reflection explores the temporal and social nature of agency, the paradox of automation and control, the emergence of trust through patterns of use, and the role of metaphors in shaping user expectations, drawing on contemporary HAI literature and design-oriented perspectives.
An agent is typically characterized by its capacity to understand, interpret, and perform tasks autonomously. However, agency is not merely functional; it is a social phenomenon with temporal qualities. Emirbayer and Mische (1998) argue that agents draw on past patterns to inform possible futures while evaluating decisions in the present. This view emphasizes that LLMs operate within temporally situated contexts where past data and learned patterns shape future reasoning, underscoring the importance of aligning system behavior with human values and expectations during design.
Tsiakas and Murray-Rust (2024) identify the opacity of LLMs as a central challenge in designing HAI systems. The 'black box' metaphor aptly illustrates the complexity of these models, making their internal reasoning difficult to interpret. To mitigate this, metaphors become crucial in conceptualizing system behavior and fostering user understanding, particularly when designing for intermittent, continuous, or proactive interactions.
Automation introduces a paradox: the more automation is enabled, the more human resources are required to monitor these systems (Shneiderman, 2020). This 'automation and control paradox' implies that fully automated systems are not inherently preferable but should instead be balanced according to context and task criticality. Tasks requiring rapid, life-critical decisions may benefit from high automation, whereas aesthetic or judgment-heavy interactions may be better left to human control. Thus, thoughtful integration, placing systems in the 'Goldilocks zone' of 'thoughtful automation,' is key for effective HAI.
Maintaining a human-in-the-loop (HITL) is critical for accountability and trust, especially in high-stakes domains like healthcare. Instead of viewing LLMs as replacements, Handa et al. (2025) advocate for an augmentation model where LLMs complement rather than substitute human work, preserving human agency and mastery. Augmentation further supports task iteration and validation, positioning the human as the primary decision-maker while the AI acts as a collaborator, offering suggestions and identifying gaps. This approach shifts the narrative from full automation toward hybrid systems that blend human judgment with computational efficiency, essential in contexts like augmented journaling in healthcare.
Trust in digital systems develops through repeated patterns of use, aligning with the perspective that meaning arises from embodied engagement with systems (Dourish, 2001). Norman (2013) similarly highlights the formation of mental models through repeated interaction, which help users anticipate system behavior. Trust in AI emerges from understanding system workings over time, while reliability, honesty, and cooperation further support trustworthiness (Shneiderman, 2020). As AI systems gain more agency, building trust becomes essential to avoid eroding user confidence and to support effective workflows (Krakowski, 2025).
Metaphors are powerful tools for shaping user expectations about AI capabilities. Four metaphors—assistant, intern, co-pilot, and agent—help conceptualize different levels of autonomy and required supervision. Each role implies a distinct mode of interaction: assistants perform simple tasks requiring explicit commands, interns handle routine work under supervision, co-pilots collaborate actively, and agents operate with high autonomy. This dynamic framing supports a research-through-design approach where sketching, prototyping, and iterative refinement guide the development of hybrid HAI systems.
Designing effective human-agent interactions requires acknowledging the temporal and social nature of agency, balancing automation with human control, fostering trust through repeated engagement, and employing metaphors to align user expectations. Rather than viewing LLMs as replacements, framing them as collaborative partners within workflows supports user mastery while leveraging the strengths of computational systems. By keeping humans in the loop and prioritizing augmentation over substitution, designers can create HAI systems that enhance trust, accountability, and meaningful user engagement across diverse contexts.