Universal Approximation Theorem;Neural Network;Activation Function

This paper presents a structured overview and novel insights into the universal approximation property of feedforward neural networks.We categorize existing results based on the characteristics of activation functions — ranging from strictly monotonic to weakly monotonicand continuous almost everywhere — and examine their implications under architectural constraints such as bounded depth andwidth. Building on classical results by Cybenko [ 1], Hornik [ 2], and Maiorov [ 3 ], we introduce new activation functions that enableeven simpler neural network architectures to retain universal approximation capabilities. Notably, we demonstrate that single-layernetworks with only two neurons and fixed weights can approximate any continuous univariate function, and that two-layer networkscan extend this capability to multivariate functions. These findings refine the known lower bounds of neural network complexityand offer constructive approaches that preserve strict monotonicity, improving upon prior work that relied on relaxed monotonicityconditions. Our results contribute to the theoretical foundation of neural networks and open pathways for designing minimal yetexpressive architectures.

This paper presents a structured overview and novel insights into the universal approximation property of feedforward neural networks.We categorize existing results based on the characteristics of activation functions — ranging from strictly monotonic to weakly monotonicand continuous almost everywhere — and examine their implications under architectural constraints such as bounded depth andwidth. Building on classical results by Cybenko [ 1], Hornik [ 2], and Maiorov [ 3 ], we introduce new activation functions that enableeven simpler neural network architectures to retain universal approximation capabilities. Notably, we demonstrate that single-layernetworks with only two neurons and fixed weights can approximate any continuous univariate function, and that two-layer networkscan extend this capability to multivariate functions. These findings refine the known lower bounds of neural network complexityand offer constructive approaches that preserve strict monotonicity, improving upon prior work that relied on relaxed monotonicityconditions. Our results contribute to the theoretical foundation of neural networks and open pathways for designing minimal yetexpressive architectures.