Well, for one it would really depend a lot on your TV. HDMI will produce a better picture compared to the component 360 cables you are using, but whether or not you will notice the difference is debatable. HDMI is a digital cable, and transmits the digital signal put out by the 360. The cables you are using are analog, so the signal has to be downgraded from digital to analog before it is displayed on your TV, hence the reason people say HDMI gives a better picture.
The other question would be can your TV output in 1080P? If it can, very few TVs can accept a 1080P signal over component cables, so if your TV can output in 1080P, which is said to be the best resolution, you're probably not going to be able to view it in 1080P without an HDMI cable. Plus, HDMI is nice because it is one cable using one connection, compared to the huge cable using five connections.
Well, if I haven't bored you to death and you're still reading, lol, the quick answer to your question would be yes, there is a difference. That being siad, you shouldn't go out to your local Best Buy and purchase an HDMI cable, though. They charge an outrageous price, as high as $100. Browsing online, HDMI cables can be had for around ten bucks after shipping. eBay may even be cheaper, and if you have a Big Lots near you I have seen HDMI cables there for twelve bucks in my area. Don't get suckered in by stores trying to upsell you: A 12 dollar HDMI cable works just as well as a 100 dollar cable. Hope I helped...