Probably the biggest factor holding most new riders from the race-track is fear. Whether it be a fear of injury, smashing up your pride and joy, or simply a fear of the unknown, there seems to be a pathological connection in the uninitiated between trackdays and catastrophe. A few may even be scared of damaging their image; no-one wants to start off in the slow group in front of their friends, especially with all the macho bullshit that surrounds sport motorcycles.
To be fair, none of these are totally irrational fears. People do hurt themselves on trackdays, they do write off their motorcycles, and I can confirm that sitting in the pit-lane before your first ever session can almost induce vomiting, such are the nerves.
However, you'd have a job to convince anyone that riding on race-track is more dangerous than riding on the road these days. For a start, a track is a lot more predictable. You know where it goes, there's nothing coming the other way, fewer tractors, city buses and dangerous taxis, and if someone does happen to lay down a sneaky oil slick there's a nice man ready to wave a pretty flag to warn you. If a trackday still sounds too scary, then a track-based school is an even saver and more controlled way to get in on the action, with the benefit of loads more ways to learn. Most race-tracks have some sort of school or training available, and sometimes you don't even have to use your own motorcycle.
A major obstacle to your enjoyment that you will need to overcome is what to do with your concentration. Most motorcycle racing schools teach a lot on this, but the basics are about concentrating on the right things. Don't get distracted by riders around or behind you and what they're doing.
On the road you have a lot more things to spread your concentration over, so you will have to learn how to focus it more at the track. There are still dozens of things to think about, but at the most basic level it's chosen line, chosen speed, and direct hazards. Take care of these and you'll do fine.