When you're stacking luggage on your bike. What should you worry about? First you need to make sure the luggage is suitable for your motorcycle. That it is securely fastened and not overloaded and the motorcycle's set up is adjusted accordingly. Most modern motorcycles manuals have a section about attaching luggage, so make sure you read it.
The three obvious areas of soft luggage are tank bags, tailpacks and panniers. Tank bags and tailpacks are located where the weight displacement can be easily accommodated by the motorcycle. They should not unsettle your machine as long as they're not excessively loaded. Soft panniers must be fitted properly, read the instructions, and all the fastenings must be secure. Once again, these must not be excessively loaded.
If panniers are overloaded or if they are located too far back handling will be seriously affected. My guess would be that the panniers will give you the most problems. I regularly use all three luggage types myself and if fitted correctly will not upset the handling of your machine. I do, however, throw a cargo net over the tail bag and panniers... Just in case.
You cannot be over cautious in this area, as sadly I'm aware of fatalities which have occurred through luggage slipping off the bike. Finally, you need to consider your tire pressures and suspension settings, which may have to be adjusted to compensate for the extra weight you're carrying.