Category and tags are frequently used synonymously. Though often times a category (in terms of hierarchy) is the top-level definition and a tag may be a more specific classification beneath that.
For example, a blog about apple pies may have a category of baking and a tag of pies.
Categories help you define your structure (even after you've been writing for awhile). Once you notice you're writing about something a lot, you can create a link somewhere on your site that allows for people to find more information about your topic. Let's say you're writing a recipe. You can tag your ingredients (apples, pie crust) as well as the course (dessert). Then, you can organize your site to list categories so your apple pie recipe comes up whether the person clicks on the ingredient or the course.