Abstract
Real-world datasets often comprise a large number of features or variables, encompassing billions of samples. Manipulating such high-dimensional data is computationally expensive. Although some advanced strategies leverage high-performance and parallel computing capabilities, such approaches could entail complex and costly resources for users. In response, dimensionality reduction techniques have been developed that transform highdimensional data into a low-dimensional space while preserving properties critical to the original data. Dimensionality Reduction has found widespread use across various domains, including pattern recognition, clustering, classification, and accelerating numerical simulations of complex physical phenomena. Regardless of the strategy adopted, available techniques heavily rely on computationally intensive matrix factorizations, such as Singular Value Decomposition. These techniques are thus quickly becoming intractable as the world adapts to a new norm where massive datasets have become everyday commodities. In the first part of this research, we introduce a novel dimensionality reduction strategy, HYBRID: HYper-reduced Basis Reduction via Interactive Decomposition, which offers remarkable efficiency and includes an error indicator certifying the accuracy to which the properties of the original data are preserved. HYBRID draws upon reduced basis decomposition and recent advancements in reduced basis methods, which have garnered attention as efficient dimensionality reduction tools for solving parametrized partial differential equations. Specifically, we propose speeding up the construction of the reduced basis by adopting the concept of “reduced residual,” enabling efficient error measurement on a subset whose dimensionality is proportional to the intrinsic dimension of the given dataset. We present numerical examples to demonstrate the performance of the proposed HYBRID technique and its competitive edge over existing dimensionality reduction techniques. In the second part of this research, we focus on electricity demand forecasting. In a highly competitive electricity market, securing the largest share through high-quality services demands meticulous planning, particularly in accurately forecasting future loads to prevent shortages. While oversupplying can satisfy demand, the costs of energy storage technology are prohibitively high. Therefore, effective and purposeful planning is necessary for companies to ensure precise load forecasting, crucial in maintaining the reliability and resilience of electricity services. Here, we apply HYBRID to a large-scale utility dataset encompassing 3.7 million customers of Commonwealth Edison (ComEd) in the state of Illinois to accurately forecast demand.