Google has announced the general availability of AlloyDB AI functions, featuring a proxy model architecture that allows database queries to run at local speeds without relying on external large language model calls.
The new system, which is now in general availability, trains a lightweight local model based on outputs from a larger LLM. This proxy model then processes queries directly within the database, eliminating the need for external API calls and reducing latency.
According to Google, the approach achieves significant performance improvements through a technique called smart batching, which the company says delivers a 2,400x throughput improvement over conventional methods.
During the preview phase, the proxy model was reported to reach speeds of 100,000 rows per second. However, Google has clarified that these benchmark numbers apply only to the ai.if function in internal testing environments and may not reflect real world performance in all scenarios.
How the Proxy Model Works
The proxy model architecture addresses a common bottleneck in AI powered database applications: the cost and delay of sending queries to external LLMs. By training a smaller, specialized model directly on the outputs of a larger LLM, the database can perform inference locally.
This method allows developers to run complex natural language queries or AI driven analytics without leaving the database environment. The local model is designed to approximate the behavior of the larger LLM while operating at database level speeds.
Google stated that the general availability release includes support for the AI functions across AlloyDB instances, enabling users to integrate AI capabilities directly into their database workflows.
Performance and Limitations
The 2,400x throughput improvement cited by Google is based on smart batching, which groups multiple queries together for more efficient processing. The company emphasized that these numbers come from controlled internal tests.
While the preview achieved 100,000 rows per second, Google noted that actual performance will vary depending on workload, data volume, and specific use cases. Users are advised to conduct their own benchmarks in production environments.
The proxy model approach is particularly relevant for applications requiring real time AI responses, such as fraud detection, personalized recommendations, and automated customer support within databases.
Implications for Developers
For developers working with AlloyDB, the new AI functions reduce dependency on external services, potentially lowering costs and simplifying architecture. By keeping inference inside the database, organizations can also improve data security and compliance by avoiding data transfer to third party LLM providers.
Google has not disclosed the specific architectures used for the proxy models or the training methodology beyond stating that they are trained from LLM outputs. The company recommends that users test the feature against their specific workloads to evaluate performance.
The announcement was made by Steef-Jan Wiggers, who reported on the release and its benchmark results.
Looking ahead, Google is expected to continue developing AlloyDB’s AI capabilities, with potential expansions to support additional LLM integrations and more sophisticated proxy models. No specific timeline for future updates has been provided.
