"Manually designing features for a complex task requires a great deal of human time and effort; it can take decades for an entire community of researchers." - Deep Learning by Ian Goodfellow, Yoshua Bengio, & Aaron Courville
Hong Kong SAR, 04-08-2025
So much has happened in the past few months, and my maturity in topics related to ECE and computer science, I feel, has also grown tremendously. Due to the diversity of topics that I have come to appreciate in the previous months, perhaps I will separate them into sections with brief recalls and descriptions. I hope that I can publish a technical blog post sometime next month, and that I can try to maintain some level of consistency, although such appears to be surprisingly difficult.
I would also like to warn the reader of such a low quality post. It is merely meant to provide a basic update.
Operating Systems Theory: The Potential for A.I.
Previously, our main appraisal of the utility of operating systems theory came in the notion that appropriate introduction of layers of abstraction achieves reductions in working memory loads for end-users, yet, there are other things that have become increasingly interesting as of late. In fact, I can state here that, recently, I find myself looking for A.I. applications, or interpreting STEM phenomena from the perspective of A.I. more and more so. Once, I would interpret virtually all STEM phenomena through a mathematical lens, relatively easy to do due to the generality of mathematics, but I find myself gradually changing and growing in breadth.
Consider that, in operating systems theory, there are many resource management algorithms including those related to page tables, and over time, one will begin to notice more and more algorithms that are "speculative", for instance, as in speculative algorithms in page table data swaps, or, in the context of resource allocation as a means of minimizing chances of resource lockup. Then, if these methods were to become slightly more sophisticated and subtle, it becomes the case that statistical methods are invoked, and, that "past history" is increasingly appraised, allowing one to call on the methods of certain stochastic processes. In fact, the notion of "locality of reference", exploited in many standard OS algorithms, exploits standard statistical phenomena.
And so, it becomes so clear that the right machine learning methods can be applied, especially those that utilize stochastic methods in their models, those that learn from past data, and possibly even data as personalized. Indeed, it should be completely unsurprising that, if some time in the future, many conventional operating systems may become in part managed by sophisticated A.I. modules, those that learn the habits and tendencies of their users.
Computer Architecture: Superscalar CPUs and the Potential for A.I.
Superscalar CPUs are, in a certain sense, CPUs that are able to perform more instructions per second than their clock rates would seem to allow, seemingly impossible. However, in continuing an intuitive point of "speculative" procedures in the previous part, speculative execution of instructions or parts of instructions should not come as a surprise. Yet again, there is such an obvious point here in that, as some of these speculative algorithms become more sophisticated and are to leverage statistical phenomena, the door to the proliferation of machine learning is completely opened. Like previously, then, it should not be entirely too surprising if it is the case that, some time in the future, many conventional CPUs have in them A.I. modules that assist in speculative execution, those that may even learn the habits and tendencies of their users.
Signals and Systems: Features Engineering in Machine Learning
An important point in machine learning comes in features engineering, essentially, on the introduction of quantitative rigor by the apt mathematization of phenomena in general. As an example, if there was an interest in cluster analysis in the clustering of, say, PDFs, then PDFs may be characterized via a "word of bag" approach, and so associated to each PDF is a mathematical object, a vector, that then allows quantitative rigor and corresponding quantitative analysis. And so, a problem presents itself, given the complexities of this world, there are many ways to design features, to engineer features as a means of quantitative rigorization, and many options appear arbitrary, ad-hoc, lacking in general principles. If so, could there actually exist general approaches, paradigms, that aid in features engineering in machine learning, at least for certain classes of machine learning algorithms?
I recall marvelling at the reality that the relatively sophisticated topics of mathematical analysis of the 2nd half of the 20th century had trickled down to such an incredible myriad of STEM subfields that in engineering, all of the major methods of the theory of generalized functions, functional analysis, and what not, were essentially formalized in a subfield that is commonly referred to as "signals and systems". Such a subfield was so sufficiently general that to pursue signals and systems as a prerequisite was even popularized in contexts of computational neurobiology. But then, if we were, say, interested in features engineering in computational neurobiology, what would be standard approaches and paradigms that would be pursued? As is clear, signals and systems provide a general paradigm, and such an interpretation is not ad-hoc.
Data Structures, Object-Oriented Programming, and Computational Complexity
From mathematics, there is the notion of "Kolmogorov complexity". Although certain irrational number may never be described fully and explicitly with finite information as, for instance, irrational numbers possess decimal representations that are non-periodic and non-terminating, the entire set of natural numbers may be described with a simple axiomatic system, along with a simple scheme that allows the iterated recovery of the entire set of natural numbers in principle. In other words, in a certain sense, the infinite set of natural numbers may be described to be "simpler" than certain irrational numbers.
In high-level programming, the notion of Kolmogorov complexity is repeatedly met, for instance, one can contrast iterators versus generators, where iterators may utilize further space complexity relative to generators, and where generators will set out to describe entire classes of objects, implicitly, without sacrificing space complexity. In fact, by leveraging appropriate data structures along with operations on them, both time complexity and space complexity can be dramatically reduced in a diversity of contexts, including by exploiting generators over iterators.
Mathematics and A.I.
With my increasing familiarity with certain topics of machine learning in recent months, I found myself continually thinking back to the time spent with mathematics, and relatively rigorous mathematics compared to standard results in engineering as encapsulated by such subfields as signals and systems. Increasingly, it appears that developments in artificial intelligence come with applications of mathematical methods, that, fantastically, artificial intelligence today demonstrates to be that particular subfield that is so applied mathematical in a fundamental research sense. Whether one is interested in applications of the methods of differential geometry on the appraisal of geodesics, or in the application of certain methods of first-order logic so as to provide logical reasoning supplementation, it appears increasingly so that the potential is vast.
Indeed, there are a few well-known mathematical subfields that has proliferated in A.I. and machine learning in particular, being, multivariable calculus, differential geometry, matrix theory, and stochastic analysis. Many methods as in those related to the theory of transforms, spectral theory, harmonic analysis, and so on, also appear increasingly common, although as typically realized under the guise of signals and systems. In fact, recently, one had discovered the supposed popularity of the Hilbert transform in communications, a transform that I had first encountered in Elias Stein's exposition on Singular Integrals and Differentiability Properties of Functions - in other words, it should be unsurprising that the Hilbert transform is utilized in features engineering in communications engineering.
Perhaps more so than mathematical physics, A.I. is the ultimate applied mathematical STEM subfield.
No comments:
Post a Comment