One of the great pleasures of caving is finding unknown or less explored caves, and striking out on your own.
When getting started however, bring water and multiple light sources... and most of all, prepare to get dirty!
Hard Hat with Light
You will need a hardhat with a light that fastens around the hat. You can find a hardhat at most hardware stores for about $6. The light is more expensive. If you don't want to buy one of these lamps, and you're not caving that often, try taping a small flashlight to the top of the hat with duct tape (although this usually comes off after the 2nd or 3rd trip out).
You could also buy a caving hat for about $40 which is brimless and has holes that enable you to thread the light's headband through.
Many cavers prefer a brimless hat because the brim sometimes gets in the way when you're crawling through a tunnel on your stomach and you try to look up but you only have a few inches above your head. To save money, consider cutting the brim off of a regular construction hat using a hacksaw.
It's recommended that you always bring at least 3 sources of light with you: a head lantern, some kind of flashlight, and matches.
Any boots which provide strong support and can get 'trashed' will work. Caving boots can become permanently caked in mud, so any inexpensive pair of boots that have a tread on them will do.
Unless you're planning to walk through a cave, knee and elbow pads are a must (or at least the knee pads anyway). You can cave without them, but once you've worn them you won't go back to not using them. You can usually get them at any sporting goods store. Personally, I prefer the wrestling pads over the rollerblading ones because they provide a lot more knee and elbow coverage.
Bring water as you would for hiking. The problem is carrying it.
Take an old pair of jeans and and a shirt. These are the jeans you wear to clean the garage or wash the car--the ones with the ripped pockets and paint splatters all over them. In smaller caves, you might scrape your legs on rocks, so long pants are usually best.
GLASSES--Some people prefer to wear glasses over contact lenses because of the dust. Others complain that glasses fog too much... and still others don't wear glasses at all since one's visual range in a cave (depending on the flashlight) may not go beyond 6 to 18 feet (2 to 6 meters) anyway.
BANDANAS--I've found bandanas very helpful for keeping cave crickets out of my hair. In many of the central Texas caves in the summer, the ceilings seem to be filled with cave crickets. Tying a bandana around your head before putting your hat seems to keep them from getting caught in one's hair.