Combat skills and SP bug

Ok, what gives? I wanted to be a mage, right? So i reallotted my stats (decreasing strength for intelligence and dexterity) and combat skills (sword and gun for magic) and got it the way i wanted. My SP went up when i first re allotted my stats. It was fine until i changed my combat skills. I decreased my gun and sword skills to 1 and made my magic 112 and I LOST 500 or so SP. Something's not right. Help? D: