It’s still unclear how exactly players will unlock spells in Hogwarts Legacy. However, we’d imagine they’re both tied to story progression and available through optional side quests. Many of the core spells needed for battle will likely be introduced while taking Defence Against the Dark Arts.
The official Hogwarts Legacy website states that players can “complete challenges throughout the world to earn experience and level up their abilities.” It also says you can mix and match spells to create your own unique play style. Beyond spells, you’ll also be able to unlock magical gear, talents, and other skills to help you face all the challenges Hogwarts Legacy throws your way.