9253-GRP Efficient Memory Management for LLM Inference on Constrained Memory Devices

Presenter Information

Location

https://campus.kennesaw.edu/colleges-departments/ccse/events/computing-showcase/fa26-cday-program.php

Document Type

Event

Start Date

20-11-2026 2:00 PM

Description

Large Language Model inference on memory-constrained devices suffers from delay due to thrashing. Since models with billions of parameters often cross the memory limit, it becomes imperative to keep the weights on disk and fetch them on demand. As a result, the inference procedure faces delay due to the constant swapping in and out of model parameters between disk and memory. In this paper, we establish this problem by comparing the existing methods in the literature. We also propose a solution to this problem, which is an efficient eviction and prefetching scheme.

This document is currently not available here.

Share

COinS
 
Nov 20th, 2:00 PM

9253-GRP Efficient Memory Management for LLM Inference on Constrained Memory Devices

https://campus.kennesaw.edu/colleges-departments/ccse/events/computing-showcase/fa26-cday-program.php

Large Language Model inference on memory-constrained devices suffers from delay due to thrashing. Since models with billions of parameters often cross the memory limit, it becomes imperative to keep the weights on disk and fetch them on demand. As a result, the inference procedure faces delay due to the constant swapping in and out of model parameters between disk and memory. In this paper, we establish this problem by comparing the existing methods in the literature. We also propose a solution to this problem, which is an efficient eviction and prefetching scheme.