Jason McInerney’s Lablog

Jump to:
Temporal Difference Models of Learning | Emergence and Information | Dynamics and Synchronization of Subthreshold Oscillators | Neurospy | Interfaces Graduate Program | Sleep and Simulation | Multi-Scale Modeling | The Virtual Instrument | Teaching

Temporal Difference Models in Vision Reinforcement Learning top
with Leanne Chukoskie and Mike Arnold

Intrinsic to learning is the need for a certain timespan to learn within. Most models of reinforcement learning incorporate a single timespan or time step during which states are evaluated which lead to actions, which in turn may or may not lead to a reward. That decision process relies on certain learning rules contained in the model which apply weights to the links between states and actions based on the presentation of a reward following the action. However, a reward might only occur after several different or repeated actions over many of the time steps in the model. If the model does not account for that, these actions which add up to a reward will be mis-weighted as not leading to a reward. The basic temporal difference model takes this into account and offers a method to predict a reward based on a state and to measure the error in that prediction. Using this model, future rewards are thus predicted, with more distant rewards being discounted by a certain factor, and so the inclusion of varying or multiple time steps is taken care of. However, reward availability or probability can change, and enhancements to the model have allowed accounting for that. The real interest in this model is that it seems dopamine neurons provide prediction error-based feedback in the same or similar manner, including the ability to use context and conditional probability in reward prediction. Dopamine neurons project to neurons controlling saccadic eye movements: Dopamine neurons in the substantia nigra pars reticulata (SNr) in the basal ganglia inhibit the superior colliculus (SC), which the cortex excites (from various regions) . Inhibition of the SNr by the caudate nucleus (CD) in the basal ganglia causes disinhibition of the SC and allows the SC neurons to act on the excitation from the cortex, initiating saccades. In a sort of loop, the cortex also excites the CD. There’s more to it, but this simplified system shows that the TD model can be applied to reinforcement learning in saccade movement. The current model simulates an experiment that Leanne in Tom Albright’s lab is doing, in conjunction with Howard Poizner, with various subjects. The experiment involves a visual search task on a screen for a hidden target, with a reward once the target is found. More info in the modeling category.

Emergence and Information top

I have been having thoughts lately on the not-so-new idea of emergence and how it can be utilized to store and convey information. Most notably, it seems rather obvious that “weak” emergence (where the observed pattern of the group can be broken down to a set of rules for the individual) can cause repeatable patterns that a cell connected to 10,000 others would be suited to read (I’m thinking of variations on Misha Rabinovich’s work with winnerless competition networks governing behavior). This weak form has been shown in simulations to explain flock movement in birds, schools of fish moving, and of course is responsible for the image you see right now (check out StarLogo if you wanna play with such simulations). More challenging is “strong” emergence, where the individual rules do not necessarily cause the emergent pattern, and this is the more common use of the term. This describes things like how we piece together mannerisms, speech sounds, and word choice to decipher our friends’ moods, or how a detective can develop a reliable event sequence from a crime scene. In fact, many learning behaviors emerge from interactions of our brains and the external world and this emergence can be modeled computationally (recently by Jochan Triesch and colleagues on gaze following in infants). Emergence can also be seen more as a way of considering or describing the relationship between these things, or between ideas and needs, and concepts that emerge from them. The needs emerging from a society and those of the individual can cause us to redefine what privacy means in debating issues of federal law, privacy being the emergent concept. We do not have a framework to understand how to look for and explain these patterns that emerge or if they are even definable (the idea of emergence is pervasive and abstract). Once I find a suitable system, I hope to address that in the near future with a multi-scale mathematical framework to analyze large simulations and their dynamics, hopefully with VI, using the phenomenon of emergence as a guiding inspiration.

Dynamics and Synchronization of Subthreshold Oscillators top
with Mike Arnold and Gert Cauwenberghs (as part of his BGGN260 course)

Subthreshold oscillations are an interesting and common occurrence in certain neurons. While their effects are numerous and implicated in many diseases, their method remains somewhat elusive. Simulation tools such as NEURON help analyze these systems by allowing us to take experimentally determined parameters during such oscillations and create models which we can easily manipulate. I have taken current experimental data for the neurons of the inferior olive and their channels and built models in MATLAB and NEURON to investigate what the oscillations and their synchronization can convey, as in spike timing dependent plasticity (STDP). I am hoping that an emergent pattern in the dynamics may be found which can relay information that we are not yet seeing when we look at the individual neurons or networks. The dynamics of the olivary-cerebellar (OC) system in particular may be providing a mechanism for some forms of learning (STDP in particular). Mike has created a model of the system capable of learning, but lacking in certain dynamics. We’re hoping these two models will combine to produce an interesting and more complete picture of the OC system.

Neurospy top
with Dejan Vučinić

Another software project, and Java as well, Neurospy enables computer imaging and recording of very fast neuronal activity. It has been implemented on a inexpensive, bench-top multiphoton imaging system built by Dejan, which is capable of fast imaging in three dimensions. The software is available from Sourceforge under an open source license. The scope setup is the real jewel, though, and I hope to be involved in building the next one. Check out the braintool wiki for more info on this (and much more) and the neurospy page for a bit more on the software.

Interdisciplinary Research and Multi-Scale Biology top
with the UCSD Interfaces Graduate Program

I am an Affiliate, Ambassador, webmaster, and symposium co-chair for the newly-formed Interfaces Training Grant HHMI program which encourages students across many disciplines to share and learn together for the betterment of human health and understanding. See the UCSD Interfaces website I built for more info. This year’s first annual symposium was great success and the program is moving along well. Ultimately it will provide an official multi-scale specialization to the participants in their individual disciplines. I am trying to get an online journal for the entire HHMI Interfaces program started, as well.

Sleep as a Body and Life Simulation Environment top

It has been proposed and fairly well received that sleep in some part serves to rehash the day’s events, in effect sifting through experiences, clearing out unneeded responses and cementing others. An obvious step from there is to consider sleep as a simulation environment, both physiologically and in dreams for mental aspects like learning, mood adjustment, anxiety, fear, and other things. A veritable Second Life (Inner Life?) where wild things can happen, and the mind can run parameter sweeps testing the body, its reactions, and neural responses. Within that simulation, powerful prediction based on lifelong experience can result in highly efficient and meaningful exercises to better learn tasks and prepare for hideous beasts jumping out from cracked doors. All fun aside, this perspective can allow us to apply many of the principles we have learned from computer simulations to investigate dreaming, sleep, and correlations with wakeful behavior. It also allows for several specific predictions, including some on lucid dreaming. More thoughts on this are here in my lablog. Watch the sleep & dreaming category for updates.

A Framework for Multi-Scale Modeling top
with Mike Arnold, Don Spencer, Tom Bartol, Ping Wang, and Elaine Zhang

A typical problem in setting up simulations, especially in such a multidisciplinary lab that spans the world from atoms through cells and systems to social behaviors, is accounting for varying time, size, and distance scales. Several studies have worked at trying to come up with a suitable framework for describing how these scales can be brought together in meaningful ways. I revisited the dynamical grammar proposed by Eric Mjolsness and Guy Yosiphon (read Eric’s overview on arXiv) and I have been working on a way to use that framework to create a specific working example. Essentially the investigator defines various methods for solving certain nodes of the model to be studied, as described by the dynamical grammar. VI could then be used to organize it and intelligently choose the solvers and other aspects based on the availability and the model in the grammar. This can also help abstract the specific model so people modeling other systems, like climate change, ocean currents, stellar atmospheres, and molecular dynamics, can all discuss approaches and share techniques meaningfully and using the same terminology. It should also help in instructing programs like MCell and NEURON on what to do. This type of framework will be needed to study complex dynamics and patterns in emergence, and also serves my non-intellectual goal of encouraging cross-disciplinary collaboration and sharing of general ideas regardless of specialization.

The Virtual Instrument Project top
with Mike Arnold, Tom Bartol, and Justin Kinney

Started many years ago, spawned from use of the Grid at SDSC, the Virtual Instrument (VI) aims to provide an intelligent software package to enable automated control over large, complex, and interacting simulations. Currently a work in progress, VI utilizes several Java packages, JPOX, MySQL, and a custom package to facilitate the organization of tasks involved in managing and running large simulations. Dependencies, input/output files, sequential process starting and stopping, real-time status updating, XML- and web-based input and monitoring, and much more are built into the API. These features enable the running of complicated jobs with simple inputs, allowing parameter sweeps, step-by-step logging, job creation and process execution, and interaction with other programs and daemons across multiple systems and physically separated working storage space. We’ve finished the code and I’m currently working on updating the package to the latest version of Java and setting up a demo based on simulations being performed in the lab by Dan Keller. This will be a powerful tool for what would otherwise be taxing and time-consuming simulation work, and will help us all concentrate on the research and results instead of tracking and maintaining input/output and simulation settings. More info in the Virtual Instrument (VI) category.

Teaching top
with UCSD

I was fortunate enough to be asked to TA the Interfaces graduate course “Numerical Analysis for Multi-Scale Biology”. I learned much more than I taught and I hope to be involved next time. Of course, I did the course website, too.

I was also a TA for the undergraduate UCSD course, BIPN148 “The Cellular Basis of Learning and Memory”. That was great!

I also have many ideas for courses. Recently I had a series of discussions based on misunderstood information reported in the media, which spawned two ideas:

• A seminar on popular science taking its material from the media and reporting the current state of that particular research.
  • This could be geared toward non-science majors, but taught by graduate students in the relevant fields.
  • It would offer non-science students the much-needed benefits of being exposed to real scientific data and processes, and in turn offer the science student lecturers a chance to present to non-scientific audiences.
• A course on how to recognize, correct, and prevent misunderstanding by the media and general public on topics in your own field.
  • See my thoughts page on this for my ideas.

I apologize for not providing all links and references. They usually appear soon after initial posting once I find the best ones to link to.