No reason for the gallbladder to cause pain if it is just sitting there doing nothing. Blocked ducts and gallbladders squeezing on stones cause the classic intense pain. Your discomfort at the moment will be caused by difficulty digesting certain foods due to insufficient bile flow, or from an excess bile flow causing irritation and possibly reflux.
Probably a bit late for you to try fixing your gallbladder now that it is completely broken. Although you could try adapting your lifestyle and diet to live with occasional discomfort, you should be aware that you most likely have a dead organ inside you and that can lead to serious complications. Can you spell g-a-n-g-r-e-n-e?
If you really don't want the surgery, perhaps you could discuss with your doctor the risks of keeping your gallbladder and whether you want to just wait for possible emergency surgery if it bursts. Nobody can force you to have it removed immediately.