That will depend on how much muscle mass you have, if you have a lean muscular body as it is, no, you won't lose much. If you have a body builder's body with biceps the size of people's heads, etc, and decide to cut back on lifting significantly and switch over to cardio, yes, you will lose quite a bit of it.
Remember that your body is a highly efficient machine, it adjusts to what is required based on the immediate situation. Muscles use up a large amount of energy even at rest. If you cut back significantly on exercises that require larger muscle fibers for an extended amount of time, in essence, the body will "realize" that you no longer have the need for such mass and cut back on energy loss.