He Breaks His Silence in Out in the Open

My heart and soul was completely shattered. He wants to take my baby girl to live with him for the next school year. Of course I said, "no." But...

He's basically telling me I am punishing him for him not being there for the first 3 years of her life. He doesn't understand.... My daughter is my life. She is the reason I wake up in the morning and continue to breathe through out the day. She's my everything. If she goes there for next school season I will NEVER see her. I work every weekend. I seriously would never see her, or talk to her, or read to her, or etc... It would KILL me. My heart is broken now. My mind is crumbling. Why does he want to take my baby girl away from me? He says he wants to be part of her life. I get that. He takes her every weekend. (except the ones where he is gone) He even has taken her for a week at a time. I try to work with him any time he wants her. But this? This is too much. She has been with me for 5 years. How can I give away the only child I have? He has another child. He doesn't want him to live with him. Just my daughter. It's not fair. IT'S NOT FAIR! I loved him. I gave him the option to love me and be with me and her as a family. He turned that option down. Now he wants the little girl I raised by myself for the first 3 years and leave me with NOTHING.
I am beginning to hate him.
What in the world would I do with out my little girl around? What reason would my life have?
I hate him for making me go through this. HATE him.
Am I over reacting? Am I being selfish? Who's right? Who's wrong?
THIS is why people get married BEFORE children. We sure did screw this up.