Markov Decision Processes
Yahtzee Die Roll
UNO Card Sorter
Naive Wiggle Stereoscopy Gifs
AI-Battleship
HEXBUG Tarantula
In this MDP I try to model a more simple and traditional path finding problem with a slight twist on movement. I define a 5 by 5 grid world with cells that have positive and negative reward for landing on them. An agent is placed at the bottom of the board and has to make its way up toward the reward square. Its movement is limited to one unit to the left, right, and up. If the agent tries to move off the board, it is bounced back to its current position. The alteration I made to the movement is the slight possibility of overshooting. With probability p, the agent will move 1 space but with probability 1-p, it will overshoot and travel 2 spaces in the chosen direction (unless it would be placed outside the board). The board is shown below.
I used value iteration, policy iteration, and Q-learning and compared their results. In a Markov decision process, Q-learning finds an optimal policy that maximizes the expected value of the total reward over the traversal of the grid.
This Markov Decision Process is inspired by the game Yahtzee in which a player has 3 tries to roll dice in order to score points depending on what the final outcome is. In my version, the player first rolls two dice, one blue and one red. The player then has two more opportunities to either reroll the blue die, the red die, both die, or neither. After the 3 tries have been used, the player is awarded points depending on the values on the dice. If the dice match, the player is awarded 10 points, otherwise, the player is awarded points equal to the value on the blue die minus the value on the red die. Generate a MDP example based on a modified yahtzee inspired die game. This function is used to generate a transition probability (A × S × S) array P and a reward (S × A) matrix R that model the following problem. The game is managed by two dice numbered {1-6} with each turn consisting of four actions: - Re-roll blue die {action 0} - Re-roll red die {action 1} - Re-roll both {action 2} - Re-roll None {action 3}
The original scope of the project was to recreate the results in the paper “Kinetic Depth Images: Flexible Generation of Depth Perception”. In it, the researchers expand on the concept of wiggle stereoscopy by creating smooth transitions between the two stereoscopic images rather than jumping between images. They do this by producing a depth map of the scene from the two stereoscopic images and transforming the images to render intermediate frames between the locations of the two images. In addition, to this, they try to maximize the perceived 3D effect of the gif by calculating an energy map for the image using depth, inverted saliency, and centrality. Using this energy map, a pivot axis is established which enhances the perceived 3D effect. I intended to recreate these elements to produce my own smooth 3D wiggle gifs.
It became clear almost immediately that the scope of the project would have to be reduced as soon as I set out to code and implement the different elements of the KDI paper. The main issue was that there was very little guidance or explanation behind the calculations for saliency and the depth mesh. The paper also briefly named the method for producing intermediate frames without providing resources or explaining the method. Overall it was infeasible to implement all the elements that went into the paper under those circumstances.
Rather than attempting to implement all the aspects of the paper, I decided to approach the project holistically. I needed to produce a gif that would smoothly transition between the stereoscopic images. To do this, I needed to produce the intermediate frames. I focused my attention only on implementing the concepts and methods in the paper that would directly aid in producing these frames. Thus, I limited the scope of the project to generating depth maps and constructing the transport maps that generated those frames.
Rules:
-
Each player arranges ships according to fleet
-
Take turns firing a shot
-
Mark Hits and Misses
-
Call out when a ship has been sunk
-
Sink all to win
Ships:
-
Carrier - 5
-
Battleship - 4
-
Cruiser - 3
-
Submarine - 3
-
Destroyer -2
-
10 Rows x 10 Columns
This Project is a sample app that simulates an UNO card game. Current functionality shows the deck you have and allows you to add card, filter them, and sort them. Functionality is to be expanded into a fully functioning card game.
This project showcases only the parts where commands were being sent to the HEXBUG Tarantula to move it forward, backwards and to turn right or left.
Sumobot Robot Controller
This project was made with the purpose of creating a robot that will compete in a tournament to push the enemy robot out of the arena. The robot uses a QTI sensor to detect whether it is in or out of bounds of the arena and a sonar to detect the respective opponent's location.
The main strategy of this robot is to use the 3 sonar's attached to it to help find the opposing robot. The idea behind the 3 sonars is to take advantage of its wider field of vision. This will help it quickly find the opponent and start lunging towards it, before the opponent finds it. It has a defensive strategy that moves away from the bound if it detects it and an offensive strategy that will push the opponent until it is out of bounds.
Strategy
2018
2018
2018
2017
2017
2017
2017
-
After successfully linking to the Infrared Remote Control, the HEXBUG Tarantula itself was pulled apart and resoldered to an ESP8266 WiFI Board. The IR Receiver was removed and the board was mounted to it. The goal was to use the WiFi board and communicate with the Tarantula using only python scripts. To do this I programmed using a library called PySerial to communicate through the Arduino's serial port.
Using Arduino UNO to Connect to HEXBUG Controller
-
The arduino folder contains code used to program an Arduino board soldered to the HEXBUG Tarantula's IR Controller. By wiring to the metal contact plates in the controller, the Arduino board can directly push commands to the Tarantula.
Using the ESP8266 WiFi Board to Connect to HEXBUG
Using Deep Learning to Correct Radar Beam Blockage
2020-2021
Radar beam blockage due to terrain or other obstructions is a major source of degradation in weather radar measurements across several areas of the country, particularly mountainous regions in the Western US. This degradation affects several widely used radar products, including Vertically Integrated Liquid (VIL), Composite Reflectively, Echo Tops, Quantitative Precipitation Estimates (QPE) and others. Beam blockage introduces biases into these products and can limit their applicability in many safety critical applications, such as Air Traffic Control and Quantitative Precipitation Forecasts.
In this work, we apply deep learning to reduce the effects of radar beam blockage. The deep learning Convolutional Neural Network (CNN) architecture used is based on an established architecture called the Resnet model. The model trained in this work combines multiple sources of data to compute a correction to VIL mosaics affected by beam blockage. These input data include degraded VIL, along with visible and infrared satellite imagery, and lightning flash detection. This presentation will explain this methodology and provide examples of the new beam blockage algorithm’s performance in removing degradation, as well as the improvements this Resnet-inspired architecture was able to make against the older architecture.
Work & Course Projects
Below are a few projects I can share showing some of the work I have implemented throughout my undergraduate and graduate coursework as well as at my current workplace.