risk of cancer recurrence with pregnancy ??

I am 35 years old and it has been 4 years since my diagnosis of stage three colon cancer. It has been three years since finishing my chemo and radiation and I have regular menstral cycles. My husband and I would love to have more children. When I was diagnosed, my doctors casually told me how fortunate I was to have already had two children (two beautiful girls now 6 & 8 years old). I do feel blessed and often feel guilty for wanting another child and tempting fate. They also mentioned that there may be some information about increased risk of cancer recurrence by getting pregnant. I think it has to do with the female body going into rapid growth mode during pregnancy which is not good for possible dormant cancer cells. I have not been able to find any information on this topic. Can anyone help me?

Thanks so much!

Katie