Bounties for Autonomous Traffic and Smart Grid

So, David reminded me of the problem of Coopetition and how that would be an issue with using bounties with the cars.  Also, I love this problem (I’ve looked at it off and on since undergrad). Coopetition is basically the problem is how can competing teams learn to cooperate.  So, here are my thoughts on how I think that a solution may emerge due to how bounties work and who the bondsmen are.

So, on to bounties and the idea of conflicting goals. This is why people shouldn’t be driving :). People don’t compromise because we have no way to really communicate our goals with other drivers other than through “signaling” by essentially going slower or faster and annoying some people. It would be interesting to show in a simulator what the throughput through intersections, time to destination, fuel economy, etc. is when the agents can’t communicate, are greedy and have different objectives. I think that bad things happen. However, it would be good to get a base line of when things start going bad for the scenario.

The main problem I think is that bounties are more locally greedy and that the “world” might want to put a bounty out that helps to cause cooperation among conflicting goals. These could adapt as the “world” observes the agents. The world could be an intersection or a road or something. And the road puts a bounty out for cars to get into some lane or turning left etc.

This could be an entirely different way to work the problem. The cars place their constraints like fuel efficient or fast and their destination and then the world directs them by alerting them to bounties that they may be interested in doing in order to satisfy their constraints in a more cooperative manner. Then this becomes a multiagent planning problem. So, one of the roads could put a bounty out for x number of cars with these properties and then some intersections might want to put out bounties to those cars to change lanes and turn onto that road (that have the properties that the road desired). So, then the car would have to learn whether to do one of the bounties or not and continue on its original route.

I think that this sort of framework would be applicable to smart grid charging and discharging of batteries and consumer devices (like electric cars, air conditions, etc) in order for the grid to be optimal. The cars are the electrons 🙂 wires are the roads, batteries are intersections and houses/appliances are destinations.  Or some variation….

So, essentially I am separating those that are doing the competition (the humans/cars) and those that are cooperating (the road system).  By doing so, the components of the road system are able to use the competitiveness of the agents to manipulate the system.

 

Another idea is that in large networks we may need a broker of bounties.  This would incorporate the work I did with Nil.  The packages in this case would be Bounties….

Games

Would be interesting if console games included smartphone apps that had more game content to augment the game on the console.  Like mini-games to get more gold/money/experience/characters/weapons etc to use in the console game.  Also, by playing the real game different smartphone minigames could be played.

To make money some of the games could offer in app purchases.

Watch Dogs did this but all you did was play against others that were playing it on the pc.

Accident Notification System

So, a friend of mine was just in a really bad car collision (not with another car thankfully).  He was taken to a hospital and was unconscious and had fractured his skull.  It took about 6 hours until his family was notified!  It was on the website of our local newspaper like within the hour of the crash.  I googled it and it seems like others have experienced the same thing where the emergency contacts aren’t called for hours after the collision.

So, my idea is that we could connect a raspberry pi to a bluetooth OBD-II adapter.  Then when the OBD signals that the airbag is deployed the raspberry pi would notify a smartphone to call 911 and any emergency contacts.  However, I did a little googling and found that getting whether the airbag has been deployed would be different for each manufacturer and is proprietary so there is no easy way to get it!  I guess that is why OnStar has got basically a monopoly on providing services like this.

So, I was also thinking that airbags would probably not be the only system affected when in a crash that would cause the airbags to deploy.  Would be interesting to collect data from the OBD to find out what other things are indicators…

So, this idea seems worthy of a kickstarter for Procyon :).  Also, I’ve seen most of the better kickstarters have cool youtube videos.  The guy who was in the accident has a degree in film and has made commercials so maybe we could get him involved.  Would be fun to get into autonomous vehicles eventually.  This might be a good start.

However, back in 2010 someone tried doing this sort of thing only based on the smart phone’s ability to measure Gs by the accelerometer.  Pretty cool.  However, it seems like they had troubles with false positives.  http://mobilware.org/2010/presentations/Car%20Accidents.pdf  But, since then, they haven’t done anything to continue the work.  They did have a neat idea where they would also broadcast location of wreck to a map service so that people could route around it.

EDIT: Well I found someone that offers this as a service http://my-911.com/.  They make you purchase the dongle and then they offer their software as a service that you subscribe to.  They don’t seem to have a big market.  I think that if we could sell it to taxi/limo companies.  Then passengers could provide emergency contact info.  And if the user has the app they would be able to just get into any car and it would be able to interface with the bluetooth device automatically.

EDIT 2: Well I no longer think we should even work on this.  It has been done and done well by https://www.automatic.com/.  It is just about how I would have done it too.  They have a built in accelerometer in the dongle that attaches to the OBD port, so they don’t have to figure out how to know if the air bag has been deployed.  Also, they have really good support so that it isn’t just your phone automatically calling they have a nice protocol set up.  I emailed them and told them about Procyon and an idea I had while I was looking at their site that I think we could provide and a broken link I found on their site.  Cool if they respond.

Decision Theory Paradox

http://en.wikipedia.org/wiki/Ellsberg_paradox is a decision theory paradox.  One solution to the paradox is to use this Choquet integral.  I don’t have time to read about it right now, but it seems like something that would be interesting to observe in multi-agent learning algorithms.  What decision do they learn to take, what would make them learn the Choquet integral method?  Is it stable, meaning do they always chose a particular action in line with a particular method or not?

A paper on the subject: http://ces.univ-paris1.fr/membre/cohen/pdf/gradef.pdf

Seems like it is useful in multi-objective combinatorial optimization: http://eric.univ-lyon2.fr/~arolland/publis/LustRollandMIWAI2013.pdf

Maybe I was asking the wrong questions as no one is looking at it from a MAL perspective.

Automatic feature and target selection

So, I think that I will need to augment how we make features, behaviors and targets in order to get the automatic feature selection (and now automatic target selection as well.  I forgot about how features can take targets!).

We need to annotate them.  Otherwise, I think that we won’t be able to get a good clustering.  Also, annotation will help to pick the features and the corresponding targets.  I think that maybe that induction from the targets for the behaviors to the features might be something to think about.  The annotations are essentially providing the semantics that we as humans take for granted.  Otherwise the clustering would probably not work.

I also think that targets will need to be chosen based on the overall semantics of the learned behavior.  So, like the behavior for determining whether to play on offense or defense was based on a feature distanceTo(me, centerField) I would have to say that this behavior is semantically a location based behavior.  This would then limit the set of behaviors and targets the feature and target selector would have to choose from.  I think that this is enough.  In the future this could be even more generalized.

If this method doesn’t work to pick the targets for the features, then the targets could be chosen in post by the trainer and then the decision tree could be built based off of that.

Automatic feature selection for Horde

I think that fractal clustering could be used to cluster data streams of electricity data.

What about distributed clustering algorithms?  So, we have massive data streams from different parts of the grid coming in to their respective hubs.  How do you make sense of the whole?  I think fractal clustering would work here.  I believe that the data would be self-similar enough to warrant such an algorithm.

Using the Fractal Dimension to Cluster Datasets 

Fast Feature Selection using Fractal Dimension (another interesting paper).  This one made think that maybe it would be possible for Horde to automatically select the correct features to be using to make its decisions on.  This would be extremely useful, however highly unlikely that it would work with the few examples it is currently classifying on.  I think that in order for automatic feature selection it would need a live stream of the features and “spike” the features on transitions between behaviors.  I would then be able to continue doing horde as normal but would base my decision tree off of the new feature set.

Would this also be useful in creating the hierarchies?  I think so.  I will have to think a bit more about this aspect.

Trying out draftin

So this is a note I made at draftin.com, I don’t know if I will actually use this or will end up being a waste of time…

Well this is sort of nice, but I would have to come here to another website and I already get this from my wordpress blog?

Well I don’t get a couple things

  1. privacy
  2. backups
  3. exports easily
  4. pictures
  5. comments/sharing

Well I could just write in google docs, but I don’t get the privacy.

Well this is totally awesome now!!! I can publish to my wordpress blog!! And to twitter, and a bunch of other places.

So, I am able to insert an image!!!  The main problem is that my wordpress and draftin won’t be in sinc when I deleted the image here now and add this text.

So, I think that it would be nice maybe once it gets a bit more mature.

Note to self

Ohh how I wish I had read through my posts before going to Robocup.  I had the chance to talk with Peter Stone!!!  I could have discussed some of my ideas with him and maybe coauthored a paper with him on this.  Oh well maybe I can do it at GMU.

Well, I just remembered I wrote a couple months ago about this topic I have been talking about.  https://drewsblog.herokuapp.com/?p=1471 (Dynamic lane reversal and Braess’ Paradox).  The paper referenced there even has a nice name for what I have been talking about.

WOW…

So, at least I finally remembered.  I also took a more multiagent perspective to the problem.  I am now intrigued to find out if my method would scale compared to their ILP approach.

I also found the first authors notes!  http://www.cs.utexas.edu/~mhauskn/projects/aim/notebook.html.  They are very detailed and even mention the Braess’ Paradox!  He apparently encountered it.  It seems like the project just stopped.  Many interesting questions are still open regarding the problem.  It is certainly not solved.  Also, in there experiments they never had a road fully become one-directional there was always at least one lane that went in the other direction.  The other thing was that lane was always on the side that it “should” be on.  That lane never could change sides or be in the middle or something really strange.  Which would be possible in my system.

It seems like bounties might be able to work in this environment.  I have to think about it a bit.  Peter has been doing more recent work on the subject and using Auctions to allow drivers to bid on things.  He has also created larger scale simulations with a nice simulator for Austin Texas.  http://www.cs.utexas.edu/~mhauskn/publications.html

Cool RL reading group at UT https://www.cs.utexas.edu/~eladlieb/RLRG.html