As Infrastructure engineering deploys AI at scale, organizations can take a better approach with AI and LLMS. This list here discusses those actionable items that have gained widespread nod in the community.

1. Observability becomes more critical and empowering as LLMs grow and mature. The types of teams working in the Generative AI have become more diverse. While data scientists are savvy about continuous monitoring, many others are not. If they perform prompt engineering and retrieval augmented generation (RAG) they will need to know what’s going on behind the scenes. They learn quickly with their prompt engineering, then might do more experimentation with language models, then proceed with RAG, before tuning its models to a specific domain knowledge and grounding outputs with searches. Observability helps with every stage of the life cycle.

2. Simpler use cases are called for before sophisticated ones with multi-agents. This aligns with an organization’s human resources and skills experiences. And since the landscape changes quickly, developing “evergreen” skills in evaluation and implementation remain essential. Leveraging context windows for RAG is an example. As context windows get larger, will RAG remain relevant? Organizations could do well to have not only a goal but a path to the AI maturity for their production deployments.

3. Gaining customer confidence in the AI models output is crucial. Customers neither trust the AI model to learn enough to make a decision based on its output nor the data on which the AI models train and test. Fetching some telemetry, accessing certain logs, and presenting them along with the AI output garners some trust and confidence. Techniques like factual grounding help here.

4. The most powerful use cases for LLMs are often the simplest as in the case of summarizing driving productivity gains. Summarization is popular because it is an easy way to build trust in LLMs with its straightforward evaluation. They are also powerful because they add instant value in many use cases. Multimodality is also important to consider here. Building a multimodal model from the ground up helps summaries to be drawn from text, image, and a variety of sources.

5. Tooling must keep up with the AI maturity roadmap. As production-oriented systems are developed, controls and human evaluation are necessary at each stage of the game. Organizations know that usage and maturity must evolve without disruption.

6. Balancing compute time, compute cost, and model quality requires cutting edge observability. Organizations must drive quality and performance of their observability capabilities because there are a number of integrations that need to work collaboratively and effectively. Even networking performance monitoring can become imperative.

7. Prioritizing the metrics that matter is more important than ever. With changes happening quickly, metrics serve as guardrails to ensure that new approaches do not chew up time and effort for the promise of productivity and introduce new risks. Establishing clear service level objectives works in this case.