gf wants x to be part of kids life not his kid

my gf i been with now since feb 07 2010 has a little girl whom i grown to love dear and i want to be part of there lifes fully i am 23 and my gf is 31 but my gf was married and just divorced in march separated since jan and he dont seem like he wants to be part of the little girls life dont really but forth effort and it drives my gf nuts little girl is 8 and my gf's x been around for 7 yrs of her life she calls him daddy but tells me that he a mean daddy and i like a nice daddy but misses him but she says she dono why well things were going good for a while he kinda stayed away and we grew but my gf got wind that her x thought she didnt want him part of little girls life so she had to goto his house and tell him whats for and now he trying to.. i dont really want him to be....he has tryed to get my gf his x to sleep with him and we still togather i dont really want her around him or near him but she just dont wanna talk bout it and gets mad when i bring him up at all now am i wrong for wanteding him out or should i just let it go?