Photo credit: venturebeat.com
As organizations increasingly integrate large language models (LLMs) into their operations, enhancing the factual accuracy of these models while minimizing instances of hallucination has become a significant challenge. Recent research from Meta AI introduces a concept known as “scalable memory layers,” which may address this critical issue.
Scalable memory layers incorporate additional parameters into LLMs, thereby boosting their ability to learn without the need for more computational resources. This innovative architecture is particularly beneficial for scenarios where there is available memory for factual information, while still maintaining the inference speed typical of more agile models.
Understanding Dense and Memory Layers
Conventional language models rely on “dense layers” to encapsulate large volumes of information within their parameters. In such dense layers, every parameter is utilized fully and is frequently activated during the inference process. While these dense layers can model complex functions, scaling them necessitates increased computational and energy costs.
Conversely, simpler architectures utilizing memory layers are more effective for encoding straightforward factual information. These memory layers operate with basic sparse activations and employ key-value lookup systems for knowledge retrieval. Although taking up more memory overall, sparse layers activate a smaller fraction of parameters at any one time, offering greater computational efficiency.
While memory layers have been around for some time, their application in modern deep learning is limited and often not optimized for existing hardware accelerators.
The latest advanced LLMs typically utilize various forms of a “mixture of experts” (MoE) framework, which possesses some parallels to memory layers. MoE models comprise numerous smaller expert components, each tailored to specific tasks. During inference, a routing mechanism identifies which expert will be activated, based on the given input. Recently, Google DeepMind’s PEER architecture has expanded this model to include millions of experts, offering finer control over which parameters are activated during the inference stage.
Enhancing Memory Layers
Memory layers are characterized by low computational demands but substantial memory requirements, which can pose challenges within current hardware and software environments. The researchers at Meta proposed several key adjustments to overcome these hurdles, enabling large-scale implementation.
Memory layers can simultaneously store knowledge across multiple GPUs without affecting model performance (source: arXiv)
Initially, the researchers adapted memory layers for parallel processing, allowing them to span multiple GPUs and handle vast quantities of key-value pairs without altering the remainder of the model. They also created a specialized CUDA kernel to manage high-bandwidth memory operations and established a parameter-sharing framework that uses a single set of memory parameters across various memory layers within a model, facilitating shared lookups.
These enhancements allow for the effective inclusion of memory layers in LLMs without compromising performance.
“By leveraging sparse activations, memory layers effectively complement dense networks, enhancing the capacity for knowledge acquisition while maintaining low computational overhead,” the researchers noted. “They can be scaled efficiently, providing a promising avenue for balancing memory usage with computational load.”
To evaluate the efficacy of memory layers, the researchers modified Llama models by substituting one or more dense layers with shared memory layers. They compared these enhanced models against both standard dense LLMs and MoE and PEER architectures across various tasks, including factual question answering, scientific knowledge, common-sense reasoning, and coding.
A 1.3B memory model (solid line) trained on 1 trillion tokens nears the performance of a 7B model (dashed line) in factual question-answering, as it increases memory parameters (source: arXiv)
Results indicated that memory-augmented models significantly outperformed standard dense baselines and competed successfully with models requiring two to four times the computational resources. In tasks demanding factual recall, a 1.3 billion-parameter memory model approached the abilities of the Llama-2-7B, which had been trained with double the tokens and tenfold the compute resources.
Additionally, the researchers observed that the advantages of memory models were consistent across various model sizes, having successively scaled their evaluations from 134 million to 8 billion parameters.
“In light of these findings, we strongly recommend the integration of memory layers into the architectures of next-generation AI,” the researchers concluded, while emphasizing that further improvements are still possible. “We particularly look forward to new learning methodologies that could enhance the functionality of these layers, enabling reduced forgetting, diminished hallucination, and sustained learning.”
Source
venturebeat.com