Brains over Brawn: Small AI Labs in the Age of Datacenter-Scale Compute

Abstract

The prevailing trend towards large models that demand extensive computational resources threatens to marginalize smaller research labs, constraining innovation and diversity in the field. This position paper advocates for a strategic pivot of small institutions to research directions that are computationally economical, specifically through a modular approach inspired by neurobiological mechanisms. We argue for a balanced approach that draws inspiration from the brain's energy-efficient processing and specialized structures, yet is liberated from the evolutionary constraints of biological growth. By focusing on modular architectures that mimic the brain's specialization and adaptability, we can strive to keep energy consumption within reasonable bounds. Recent research into forward-only training algorithms has opened up concrete avenues to include such modules into existing networks. This approach not only aligns with the imperative to make AI research more sustainable and inclusive but also leverages the brain's proven strategies for efficient computation. We posit that there exists a middle ground between the brain and datacenter-scale models that eschews the need for excessive computational power, fostering an environment where innovation is driven by ingenuity rather than computational capacity. 2 OUTLINE OF OBJECTIVES The human brain's ability to perform complex tasks with remarkable efficiency is a compelling in vivo demonstration that intelligent computation is possible within a severely energy-constrained regime. We argue that large datacenter-scale models solve a different version of the cognition problem in which neu-a https://orcid.org/0000-0003-1122-6366 b https://orcid.org/0009-0000-7071-8534 c https://orcid.org/0000-0001-8755-0002 d https://orcid.org/0000-0002-7047-5867 ral networks are designed with the assumption that energy and data are plentiful and compute resources abundant. The brain, on the other hand, operates under severe energetic and evolutionary constraints. In effect, the brain and datacenter-scale models represent two extremes on a spectrum representing different implementations for intelligent systems. This implies there could be in-between solutions that perform intelligent computation with a reasonable energy consumption while also avoiding the evolutionary burdens of biological systems. It follows then that AI development should in principle be possible through research conducted even in small labs. The outline of the paper will be as follows: • Addressing the computational arms race: the paper will begin by examining the trend towards larger models requiring extensive computational resources. It will discuss the impact of this computational arms race on smaller research groups, the challenges they face and the potential stifling of diverse, innovative contributions to the field. • Defining a reasonable power budget: we will describe what constitutes a reasonable power budget for AI research in smaller labs. • Brain-inspired modular approaches: this section focuses on investigating brain-inspired, modular approaches as a solution to computational limitations. The paper will argue for a Brains over Brawn (BoB) philosopy, drawing on analogies between the human brain's efficient architecture and how similar principles can be applied to the design of AI systems to enhance their efficiency. • Interesting research avenues: the paper will offer practitioners in small labs some interesting research directions, informed by the latest neuro-science research. It will discuss how structures such as the thalamus, the hippocampus, and the cerebellum could inspire the development of similar modules for artificial neural networks (ANNs), while at the same time striving for as much generality as possible.