# Blog

On the other hand, the naive strategy has an expected loss of a much larger $1.6, with a standard deviation of 1¢ on that estimate. Using ttest2, we can assert that the MDP strategy is better with near perfect confidence. While the difference between the MDP performance and that of the cheat sheet is very marginal, the likeliest for why is performs worse is that I made a small error in painstakingly filling out the almost two million state transitions probabilities in $P$. However, what is clearly evident is that the exercise does generate a near optimal strategy for playing Blackjack, one that performs much better than a naive decision policy. ## Conclusion The purpose of this project was to get familiar with using Markov Decision Processes to provide optimal strategies in discrete, finite state stochastic environments. Blackjack seemed to be a perfect candidate to try the algorithm on. To formulate the game in a manner acceptable to an MDP solver, I first specified the probabilities for transitioning from each gamestate to every other under all actions using either explicit analysis or Monte Carlo simulation. I then specified the rewards for transitioning from game states to terminal states under the different actions. With these matrices specified, I used the MDP solver to provide the optimal strategy. I compared this strategy to a widely available “Blackjack Cheat Sheet”, which claims to be the optimal blackjack strategy. Repeatedly simulating the strategies in the game of Blackjack, I found that both the MDP’s strategy and that of the cheat sheet lose only about 5¢ per$10 bet on average, while a more naive strategy loses \$1.6 on the same bet. According to wizardofodds.com, the “house advantage” for a game of Blackjack with the assumed rules is around 0.5%. The house advantage indicates the expected value of the dealer assuming the player is playing optimally, and thus provides evidence that the strategy given by the MDP is indeed close to optimal.

I did this project with the help of Dilip Ravindran, a close friend and graduate student at Columbia Economics.

## Spatial Heterodyne Infrared Raman Spectrometry

As part of the Harvey Mudd College Clinic Program, I worked on a project for the Kinohi Institute. Over the course of a year, we designed and built the first spatial heterodyne Raman spectrometer capable of operating at near-infrared excitation light. The team also tested and validated the performance of this device.

Raman spectroscopy is a common technique used to determine the material composition of a substance. Currently, the most common Raman spectrometers are slit spectrometers (which have low light throughput), and Fourier Transform spectrometers (which require sensitive moving parts). Spatial Heterodyne Spectrometry (SHS) has no moving parts and high light throughput, which means it is robust and non-destructive to the sample. The Spatial Heterodyne Infrared Spectroscopy (SHIRS) technique developed in this project can be employed to detect cancer, observe protein folding in real time, and could even be used for the non-destructive detection of extraterrestrial life.

In this post, I will describe how the device excites Raman signals, isolates the signal from noise, and identifies the Raman signal through spectroscopy. I will then discuss the results of the effort.

Exciting Raman Signals (A)

Raman signals are a form of light that reflects of molecules with a change in energy that acts as a signature for the molecule. Hence, if we shine light of a single, pure wavelength onto any molecule and identify the spectrum of the resulting Raman signals, we can correctly identify the molecule. However, this isn’t quite so easy. For one, Raman shifts are a very low probability event, which results in their signals being extremely weak. In addition, the unchanged reflection of the input laser (Rayleigh scattering) and fluorescent reflections are sources of noise that flood out the Raman signals. The method for dealing with this noise is discussed in the next section.

Isolating Raman Signals (B)

Aside from the advantages of causing little damage to organic samples, using a near infrared excitation laser has the additional advantage of causing much less fluorescence than the more conventionally used visible or UV excitation lasers in Raman or SHS systems. Additionally, since Raman shifts are known to be longer wavelength than the excitation laser, a Long-Pass Filter (LP on Figure 1) and Long-Pass Dichroic Mirror (D) are used to ensure that all light of lower than the excitation wavelength are heavily attenuated. A Notch Filter (N) is used to heavily attenuate the Rayleigh scattering from the excitation laser. The remaining light is mostly composed of just the Raman signal at a sufficiently high Signal-to-Noise ratio. The next section will discuss how a Spatial Heterodyne Spectrometer recovers the spectra of this light.

Identifying Raman Signals (C)

With reference to Figure 1, the SHS works by first using a 50:50 Beam Splitter (BS) to direct the light towards two Diffraction Gratings (DG). The light then reflects of these gratings at a wavelength-dependent angle. Since the angle of reflection depends on wavelength, there is a wavelength-dependent separation between the the two reflected wavefronts that recombine on the Camera’s Sensor (C). From the theory of interference, we know that the frequency of oscillations in an interference pattern depends on the angle separating the wavefronts. Thus, the wavelength of light is encoded in the frequency of oscillations produced on the camera’s sensor. Performing a Fourier Transform on the resulting image recovers the spectrum of the input light.

The figure above shows examples of fringe patterns created by single-frequency sources (top), and the spectra that are generated from the fringe patterns (bottom). The generated spectra are compared to spectra gathered from an off-the-shelf spectrometer. We can see that for two test cases, the device functions as an accurate and precise spectrometer compared to an off-the-shelf alternative.

Results

The complete SHIRS system was validated as a standalone spectrometer on various test sources such as Hydrogen, Neon, and Sodium Vapor Lamps. The spectrometer demonstrated a bandwidth of 131 nm and a resolution of 0.1-0.6 nm, which satisfy the constraints for the desired applications of the Kinohi Institute.

The figure above shows the spectrum of the sodium vapor lamp captured by the team’s SHS and the ScienceSurplus slit-based spectrometer. Notice that only the team’s SHS is able to resolve sodium’s characteristic double peak.

The complete system was validated by testing elemental Sulfur, which is known to have a strong Raman signal and is a typical first-test case.

This figure shows the Raman spectrum gathered using the team’s prototype from a sulfur sample. The peak seen matches the peak expected from published Raman spectra to within 1 cm-1. This spectrum was reproducible across multiple trials.

I completed this project on a team with Alex Alves, Achintaya Bansal, Erica Martelly, and Apoorva Sharma. The team was advised by Professors Philip Cha and Gregory Lyzenga of Harvey Mudd College. If you would like to apply this technology to any of your work please contact me or the liaison of this project from the Kinohi Institute, Dr. Michael Storrie-Lombardi (mike@kinohi.org).

## Magnetic Levitation with an FPGA

In this project, a small disc magnet is levitated by a solenoid that is PID controlled by an FPGA at 156 kHz sample rate. A slower open loop is driven by a Raspberry Pi to stably move the magnet between set points. The Pi communicates the set point, steady state voltage, and gains, thus slowly making big transitions by commanding smaller jumps with accurate linearization. A video of the system in action can be found here.

Figure 2 shows a block diagram of the system components. As shown in Figure 1, the physical system is composed of a solenoid above a disc magnet. A PID controller is implemented on an FPGA to control the position of the disc magnet. The 10-bit position of the magnet is measured by the Hall-effect sensor, which is located about an inch below the solenoid so it avoids saturation from the solenoid’s magnetic field but still detects the disc-magnet’s field. The voltage from the sensor is passed through an ADC and then through a pre-calibrated lookup-table to estimate the position of the magnet. The position is then compared to the desired setpoint and used to compute a voltage across the solenoid. The voltage is computed by a PID controller that implements discrete-time derivative filtering for improved performance. This voltage is then passed through a PWM to generate a binary control signal. This control signal leaves the FPGA and is applied to a transistor which controls the current flow to the solenoid.

The linearized gains and feed-forward voltages most applicable to a given setpoint were found in calibration and stored on a Raspberry Pi. The Raspberry Pi takes a desired setpoint and passes the associated gains and feed-forward voltage to the FPGA’s PID module over SPI. In order to create the up-and-down motion seen in the video, the Pi takes small, discrete steps to march the magnet to the endpoints, holding each command for a time long enough for the magnet to reach the desired intermediate location and dissipate its transient response.

The project was done as a joint final project for the Harvey Mudd College courses Microprocessor Design (E155) and Systems Simulation (E205). The project was done in collaboration with Vaibhav Viswanathan.