Multi-Modal Performance Prediction for Next-Generation AI Model Serving Platforms
Keywords:
multi-modal prediction, AI model serving, inference platforms, performance modeling, observability, sustainability, fairness, governance, edge computing, systems architectureAbstract
Next-generation artificial intelligence model serving platforms must manage heterogeneous accelerators, dynamic request mixtures, increasingly large foundation models, and stringent service-level expectations across cloud, edge, and hybrid infrastructures. Performance prediction in such environments can no longer rely on a single latency or throughput metric. It must become multi-modal, combining telemetry from hardware counters, request traces, model metadata, scheduler state, network conditions, energy usage, and governance signals. This paper examines multi-modal performance prediction as a socio-technical capability for AI serving platforms. It argues that prediction quality depends less on isolated algorithms than on architectural observability, data contracts, deployment feedback loops, and policy-aware governance. The discussion reviews foundational constraints in warehouse-scale and inference serving systems, then analyzes how architectural layering, feature modalities, modeling choices, fairness, robustness, and sustainability interact. It considers trade-offs among accuracy, interpretability, responsiveness, and operational cost. The paper also explores cross-domain comparisons with stream processing, edge computing, and federated learning, where similar tensions appear between centralized prediction and local autonomy. A central claim is that multi-modal prediction should be treated as a platform service rather than a model artifact, with explicit interfaces, versioning, auditability, and fallback behavior. Such treatment supports capacity planning, autoscaling, carbon-aware scheduling, incident response, and equitable service delivery. The conclusion outlines research directions for open benchmarks, uncertainty-aware prediction, and governance frameworks that align technical performance with broader institutional accountability.
References
1. Barroso, L. A., Clidaras, J., & Hölzle, U. (2013). The datacenter as a computer: An introduction to the design of warehouse-scale machines (2nd ed.). Morgan & Claypool.
2. Dean, J., & Barroso, L. A. (2013). The tail at scale. Communications of the ACM, 56(2), 74-80. https://doi.org/10.1145/2408776.2408794
3. Romero, F., Li, Q., Govindaraju, V., & Guo, Y. (2021). INFaaS: Automated model-less inference serving. In 2021 USENIX Annual Technical Conference (pp. 397-411). USENIX Association.
4. Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., & Stoica, I. (2017). Clipper: A low-latency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (pp. 613-627). USENIX Association.
5. Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., & Mace, J. (2020). Serving DNNs like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (pp. 443-462). USENIX Association.
6. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (pp. 611-626). ACM.
7. Zhang, J., & Gao, H. (2026, March). LLM inference delay prediction based on machine learning algorithms. In 2026 International Conference on Generative Artificial Intelligence and Information Security (GAIIS) (pp. 37-41). IEEE.
8. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2023). Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 606-624.
9. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M. (2021). Efficient large-scale language model training on GPU clusters using Megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1-15). ACM.
10. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L. M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021). Carbon emissions and large neural network research. arXiv preprint arXiv:2104.10350.
11. Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 3645-3650). ACL.
12. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J. F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems 28 (pp. 2503-2511). Curran Associates.
13. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 220-229). ACM.
14. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86-92.
15. Crankshaw, D., Sela, O., Mo, X., Zumar, C., Stoica, I., Gonzalez, J. E., & Tumanov, A. (2020). InferLine: Latency-aware provisioning and scheduling for machine learning inference pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing (pp. 1-14). ACM.
16. Kalavri, V., Liagouris, J., Hoffmann, M., Dimitrova, D., Forshaw, M., & Roscoe, T. (2018). Three steps is all you need: Fast, accurate, automatic scaling decisions for distributed streaming dataflows. In 13th USENIX Symposium on Operating Systems Design and Implementation (pp. 783-798). USENIX Association.
17. Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems 35 (pp. 30318-30332). Curran Associates.
18. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., ... & Kaplan, J. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073.
19. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610-623). ACM.
20. Reddi, V. J., Cheng, C., Kanter, D., Mattson, P., Schmuelling, G., Wu, C. J., Anderson, B., Gu, W., Bhandare, A., Sokolov, A., & Tang, H. (2020). MLPerf inference benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (pp. 446-459). IEEE.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 International Journal of Artificial Intelligence Engineering and Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
