As a programmer and epistemology enthusiast, I\u2019ve been studying some statistical modeling techniques lately! It\u2019s been boodles of fun, and might even prove useful in a future dayjob if I decide to pivot my career away from the backend web development roles I\u2019ve taken in the past.<\/p>\n
More specifically, I\u2019ve mostly been focused on multi-layer parameterized graphical function approximators, which map inputs to outputs via a sequence of affine transformations composed with nonlinear \u201cactivation\u201d functions.<\/p>\n
(Some authors call these \u201cdeep neural networks\u201d<\/a> for some reason, but I like my name better<\/a>.)<\/p>\n It\u2019s a curve-fitting technique: by setting the multiplicative factors and additive terms appropriately, multi-layer parameterized graphical function approximators can approximate any function<\/a>. For a popular choice of \u201cactivation\u201d rule which takes the maximum of the input and zero<\/a>, the curve is specifically a piecewise-linear function. We iteratively improve the approximation f(x, \u03b8) by adjusting the parameters \u03b8 in the direction of the derivative of some error metric on the current approximation\u2019s fit to some example input\u2013output pairs (x, y), which some authors call \u201cgradient descent\u201d<\/a> for some reason. (The mean squared error<\/a> (f(x, \u03b8) \u2212 y)\u00b2 is a popular choice for the error metric, as is the negative log likelihood \u2212log P(y | f(x, \u03b8)). Some authors call these \u201closs functions\u201d for some reason.)<\/p>\n Basically, the big empirical surprise of the previous decade<\/a> is that given a lot of desired input\u2013output pairs (x, y) and the proper engineering know-how, you can use large amounts of computing power to find parameters \u03b8 to fit a function approximator that \u201cgeneralizes\u201d well\u2014meaning that if you compute y\u0302 = f(x, \u03b8) for some x that wasn\u2019t in any of your original example input\u2013output pairs (which some authors call \u201ctraining\u201d data for some reason), it turns out that y\u0302 is usually pretty similar to the y you would have used in an example (x, y) pair.<\/p>\n It wasn\u2019t obvious beforehand that this would work! You\u2019d expect that if your function approximator has more parameters than you have example input\u2013output pairs, it would overfit<\/a>, implementing a complicated function that reproduced the example input\u2013output pairs but outputted crazy nonsense for other choices of x\u2014the more expressive function approximator proving useless for the lack of evidence to pin down the correct approximation<\/a>.<\/p>\n And that is what we see for function approximators with only slightly more parameters than example input\u2013output pairs, but for sufficiently large<\/em> function approximators, the trend reverses<\/a> and \u201cgeneralization\u201d improves\u2014the more expressive function approximator proving useful after all, as it admits algorithmically simpler functions<\/a> that fit the example pairs.<\/p>\n The other week I was talking about this to an acquaintance who seemed puzzled by my explanation. \u201cWhat are the preconditions for this intuition about neural networks as function approximators?\u201d they asked. (I paraphrase only slightly.) \u201cI would assume this is true under specific conditions,\u201d they continued, \u201cbut I don\u2019t think we should expect such niceness to hold under capability increases. Why should we expect this to carry forward?\u201d<\/p>\n I don\u2019t know where this person was getting their information, but this made zero sense to me. I mean, okay, when you increase the number of parameters<\/a> in your function approximator, it gets better at representing more complicated functions, which I guess you could describe as \u201ccapability increases\u201d?<\/p>\n But multi-layer parameterized graphical function approximators created by iteratively using the derivative of some error metric to improve the quality of the approximation are still, actually, function approximators. Piecewise-linear functions are still piecewise-linear functions even when there are a lot of pieces. What did you<\/em> think it was doing?<\/p>\n To be clear, you can do a lot with function approximation!<\/p>\n For example, if you assemble a collection of desired input\u2013output pairs (x, y) where the x is an array of pixels depicting a handwritten digit<\/a> and y is a character representing which digit, then you can fit a \u201cconvolutional\u201d multi-layer parameterized graphical function approximator to approximate the function from pixel-arrays to digits\u2014effectively allowing computers to read handwriting.<\/p>\n Such techniques have proven useful in all sorts of domains where a task can be conceptualized as a function from one data distribution to another: image synthesis, voice recognition, recommender systems\u2014you name it. Famously, by approximating the next-token function in tokenized internet text, large language models can answer questions, write code, and perform other natural-language understanding tasks.<\/p>\n I could see how someone reading about computer systems performing cognitive tasks previously thought to require intelligence might be alarmed\u2014and become further alarmed when reading that these systems are \u201ctrained\u201d rather than coded in the manner of traditional computer programs. The summary evokes imagery of training a wild animal that might turn on us the moment it can seize power and reward itself rather than being dependent on its masters.<\/p>\n But \u201ctraining\u201d is just a suggestive name<\/a>. It\u2019s true that we don\u2019t have a mechanistic understanding of how function approximators perform tasks, in contrast to traditional computer programs whose source code was written by a human. It\u2019s plausible that this opacity represents grave risks, if we create powerful systems that we don\u2019t know how to debug.<\/p>\n But whatever the real risks are, any hope of mitigating them is going to depend on acquiring the most accurate possible understanding of the problem. If the problem is itself largely one of our own lack of understanding, it helps to be specific<\/em> about exactly which parts we do and don\u2019t understand, rather than surrendering the entire field to a blurry aura of mystery and despair.<\/p>\n One of the exciting things about multi-layer parameterized graphical function approximators is that they can be combined with other methods for the automation of cognitive tasks (which is usually called \u201ccomputing\u201d, but some authors say \u201cartificial intelligence\u201d for some reason).<\/p>\n In the spirit of being specific about exactly which parts we do and don\u2019t understand, I want to talk about Mnih et al.<\/em> 2013\u2019s work on getting computers to play classic Atari games<\/a> (like Pong<\/em><\/a>, Breakout<\/em><\/a>, or Space Invaders<\/em><\/a>). This work is notable as one of the first high-profile examples of using multi-layer parameterized graphical function approximators in conjunction with success-antecedent computation boosting (which some authors call \u201creinforcement learning\u201d<\/a> for some reason).<\/p>\n If you only read the news\u2014if you\u2019re not in tune with there being things to read besides<\/em> news\u2014I could see this result being quite alarming. Digital brains learning to play video games at superhuman levels from the raw pixels<\/em>, rather than because a programmer sat down to write an automation policy for that particular game? Are we not already in the shadow of the coming race<\/a>?<\/p>\n But people who read textbooks and not just news, being no less impressed by the result, are often inclined to take a subtler lesson from any particular headline-grabbing advance.<\/p>\n Mnih et al.<\/em>\u2019s Atari result built off the technique of Q-learning<\/a> introduced two decades prior. Given a discrete-time present-state-based outcome-valued stochastic control problem (which some authors call a \u201cMarkov decision process\u201d<\/a> for some reason), Q-learning concerns itself with defining a function Q(s, a) that describes the value of taking action a while in state s, for some discrete sets of states and actions. For example, to describe the problem faced by an policy for a grid-based video game, the states might be the squares of the grid, and the available actions might be moving left, right, up, or down. The Q-value for being on a particular square and taking the move-right action might be the expected change in the game\u2019s score from doing that (including a scaled-down expectation of score changes from future actions after that).<\/p>\n Upon finding itself in a particular state s, a Q-learning policy<\/a> will usually perform the action with the highest Q(s, a), \u201cexploiting\u201d<\/a> its current beliefs about the environment, but with some probability<\/a> it will \u201cexplore\u201d by taking a random action. The predicted outcomes of its decisions are compared to the actual outcomes to update the function Q(s, a), which can simply be represented as a table with as many rows as there are possible states and as many columns as there are possible actions. We have theorems to the effect that as the policy thoroughly explores the environment, it will eventually converge on the correct Q(s, a).<\/p>\n But Q-learning as originally conceived doesn\u2019t work for the Atari games studied by Mnih et al.<\/em>, because it assumes a discrete set of possible states that could be represented with the rows in a table. This is intractable for problems where the state of the environment varies continuously. If a \u201cstate\u201d in Pong<\/em> is a 6-tuple of floating-point numbers representing the player\u2019s paddle position, the opponent\u2019s paddle position, and the x- and y-coordinates of the ball\u2019s position and velocity, then there\u2019s no way for the traditional Q-learning algorithm to base its behavior on its past experiences without having already seen that exact conjunction of paddle positions, ball position, and ball velocity, which almost never happens. So Mnih et al.<\/em>\u2019s great innovation was\u2014<\/p>\n (Wait for it \u2026)<\/p>\n \u2014to replace the table representing Q(s, a) with a multi-layer parameterized graphical function approximator! By approximating the mapping from state\u2013action pairs to discounted-sums-of-\u201crewards\u201d, the \u201cneural network\u201d allows the policy to \u201cgeneralize\u201d from its experience, taking similar actions in relevantly similar states, without having visited those exact states before. There are a few other minor technical details<\/a> needed to make it work well, but that\u2019s the big idea.<\/p>\n And understanding the big idea probably changes your perspective on the headline-grabbing advance. (It certainly did for me.) \u201cDeep learning is like evolving brains; it solves problems and we don\u2019t know how<\/a>\u201d is an importantly different story from \u201cWe swapped out a table for a multi-layer parameterized graphical function approximator in this specific success-antecedent computation boosting algorithm, and now it can handle continuous state spaces.\u201d<\/p>\n When I solicited reading recommendations from people who ought to know about risks of harm from statistical modeling techniques, I was directed to a list of reputedly fatal-to-humanity problems, or \u201clethalities\u201d<\/a>.<\/p>\n Unfortunately, I don\u2019t think I\u2019m qualified to evaluate the list as a whole; I would seem to lack some necessary context. (The author keeps using the term \u201cAGI\u201d without defining it, and adjusted gross income<\/a> doesn\u2019t make sense in context.)<\/p>\n What I can say is that when the list discusses the kinds of statistical modeling techniques I\u2019ve been studying lately, it starts to talk funny<\/em>. I don\u2019t think someone who\u2019s been reading the same textbooks as I have (like Prince 2023<\/a> or Bishop and Bishop 2024<\/a>) would write like this:<\/p>\n Even if you train really hard on an exact loss function, that doesn\u2019t thereby create an explicit internal representation of the loss function inside an AI that then continues to pursue that exact loss function in distribution-shifted environments. Humans don\u2019t explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn\u2019t produce inner optimization in that direction.<\/strong> [\u2026] This is sufficient on its own [\u2026] to trash entire categories of naive alignment proposals which assume that if you optimize a bunch on a loss function calculated using some simple concept, you get perfect inner alignment on that concept.<\/p><\/blockquote>\n To be clear, I agree that if you fit a function approximator by iteratively adjusting its parameters in the direction of the derivative of some loss function on example input\u2013output pairs, that doesn\u2019t create an explicit internal representation of the loss function inside the function approximator.<\/p>\n It\u2019s just\u2014why would you want that? And really, what would that even mean? If I use the mean squared error loss function to approximate a set of data points in the plane with a line (which some authors call a \u201clinear regression model\u201d for some reason), obviously the line itself does not somehow contain a representation of general squared-error-minimization. The line is just a line. The loss function defines how my choice of line responds to the data I\u2019m trying to approximate with the line. (The mean squared error has some elegant mathematical properties<\/a>, but is more sensitive to outliers than the mean absolute error<\/a>.)<\/p>\n It\u2019s the same thing for piecewise-linear functions defined by multi-layer parameterized graphical function approximators: the model is the dataset<\/a>. It\u2019s just not meaningful to talk about what a loss function implies, independently of the training data. (Mean squared error of what?<\/em> Negative log likelihood of what?<\/em> Finish the sentence!)<\/p>\n This confusion about loss functions seems to be linked to a particular theory of how statistical modeling techniques might be dangerous, in which \u201couter\u201d training results in the emergence of an \u201cinner\u201d intelligent agent. If you expect that, and you expect intelligent agents to have a \u201cutility function\u201d<\/a>, you might be inclined to think of \u201cgradient descent\u201d \u201ctraining\u201d as trying to transfer an outer \u201closs function\u201d into an inner \u201cutility function\u201d, and perhaps to think that the attempted transfer primarily doesn\u2019t work because \u201cgradient descent\u201d is an insufficiently powerful optimization method.<\/p>\n I guess the emergence of inner agents might be possible<\/a>? I can\u2019t rule it out<\/em>. (\u201cFunctions\u201d are very general, so I can\u2019t claim that a function approximator could never implement an agent.) Maybe it would happen at some scale?<\/p>\n But taking the technology in front of us at face value, that\u2019s not my default guess at how the machine intelligence transition would go down. If I had to guess, I\u2019d imagine someone deliberately building an agent using function approximators as a critical component, rather than your function approximator secretly having an agent inside of it.<\/p>\n That\u2019s a different threat model! If you\u2019re trying to build a good agent, or trying to prohibit people from building bad agents using coordinated violence (which some authors call \u201cregulation\u201d for some reason), it matters what your threat model is!<\/p>\n (Statistical modeling engineer Jack Gallagher has described his experience of this debate as \u201clike trying to discuss crash test methodology with people who insist that the wheels must be made of little cars, because how else would they move forward like a car does?\u201d)<\/p>\n I don\u2019t know how to build a general agent, but contemporary computing research offers clues as to how function approximators can be composed with other components to build systems that perform cognitive tasks.<\/p>\n Consider AlphaGo<\/a> and its successor AlphaZero<\/a>. In AlphaGo, one function approximator is used to approximate a function from board states to move probabilities. Another is used to approximate the function from board states to game outcomes, where the outcome is +1 when one player has certainly won, \u22121 when the other player has certainly won, and a proportionately intermediate value indicating who has the advantage when the outcome is still uncertain. The system plays both sides of a game, using the board-state-to-move-probability function and board-state-to-game-outcome function as heuristics to guide a search algorithm which some authors call \u201cMonte Carlo tree search\u201d<\/a>. The board-state-to-move-probability function approximation is improved by adjusting its parameters in the direction of the derivative of its cross-entropy<\/a> with the move distribution found by the search algorithm. The board-state-to-game-outcome function approximation is improved by adjusting its parameters in the direction of the derivative of its squared difference with the self-play game\u2019s ultimate outcome.<\/p>\n This kind of design is not trivially safe. A similarly superhuman system that operated in the real world (instead of the restricted world of board games) that iteratively improved an action-to-money-in-this-bank-account function seems like it would have undesirable consequences, because if the search discovered that theft or fraud increased the amount of money in the bank account, then the action-to-money function approximator would generalizably steer the system into doing more theft and fraud.<\/p>\n Statistical modeling engineers have a saying: if you\u2019re surprised by what your nerual net is doing, you haven\u2019t looked at your training data closely enough. The problem in this hypothetical scenario is not that multi-layer parameterized graphical function approximators are inherently unpredictable, or must necessarily contain a power-seeking consequentialist agent in order to do any useful cognitive work. The problem is that you\u2019re approximating the wrong function and get what you measure<\/a>. The failure would still occur if the function approximator \u201cgeneralizes\u201d from its \u201ctraining\u201d data the way you\u2019d expect. (If you can recognize<\/em> fraud and theft, it\u2019s easy enough to just not use that data as examples to approximate, but by hypothesis, this system is only looking at the account balance.) This doesn\u2019t itself rule out more careful designs that use function approximators to approximate known-trustworthy processes<\/a> and don\u2019t search harder than their representation of value can support<\/a>.<\/p>\n This may be cold comfort to people who anticipate a competitive future in which cognitive automation designs that more carefully respect human values will foreseeably fail to keep up with the frontier of more powerful systems that do search harder<\/a>. It may not matter to the long-run future of the universe<\/a> that you can build helpful and harmless language agents today, if your civilization gets eaten by more powerful and unfriendlier cognitive automation designs some number of years down the line. As a humble programmer and epistemology enthusiast, I have no assurances to offer, no principle or theory to guarantee everything will turn out all right in the end. Just a conviction that, whatever challenges confront us in the future, we\u2019ll be a better position to face them by understanding the problem in as much detail as possible.<\/p>\n Bishop, Christopher M., and Andrew M. Bishop. 2024. Deep Learning: Foundations and Concepts<\/em>. Cambridge, UK: Cambridge University Press. https:\/\/www.bishopbook.com\/<\/em><\/p>\n Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. \u201cPlaying Atari with Deep Reinforcement Learning.\u201d https:\/\/arxiv.org\/abs\/1312.5602<\/em><\/p>\n Prince, Simon J.D. 2023. Understanding Deep Learning<\/em>. Cambridge, MA: MIT Press. http:\/\/udlbook.com<\/em><\/p>\n Sutton, Richard S. and Andrew G. Barto. 2024. Reinforcement Learning<\/em>. 2nd ed. Cambridge, MA: MIT Press.<\/p>\n","protected":false},"excerpt":{"rendered":" A Surprising Development in the Study of Multi-layer Parameterized Graphical Function Approximators As a programmer and epistemology enthusiast, I\u2019ve been studying some statistical modeling techniques lately! It\u2019s been boodles of fun, and might even prove useful in a future dayjob … Continue reading Multi-layer Parameterized Graphical Function Approximators Have Many Exciting Applications<\/h2>\n
An Example of Applying Multi-layer Parameterized Graphical Function Approximators in Success-Antecedent Computation Boosting<\/h2>\n
Risks From Learned Approximation<\/h2>\n
\nBibliography<\/h3>\n