I built my very first colony. However I am unhappy with it's location. Before I continue in a place I dont want to be, is there a way to delete the colony and start fresh on the map?

Yes there is, if you click your colony and click on the profile button, the last option should read "delete colony" there you will get the details of it.

IF all you want to do is move your colony to new location, and not restarting you can build an outpost where you want to be and relocate the colony. This cost oil based on the distance, and a some energy. And your units will NOT move with your base. That way you don't have to start over for just moving

Another way is to build a outpost where you would want to be and then if it doesnt get attacked then go to it click on it and click on relocate my colony here but it will not take your units with you and you need alot of oil but it depends on the distance :] hope that helps