Seasonal allergies but only inside the house?

Every summer since we moved here (12 years ago) I've had terrible seasonal allergies. They say it is from the pollen, however I can ride my bike outside all day and be fine, but almost the second I walk into our house it hits me. But if I am allergic to something in the house, then why does it only affect me in the summer? We have a dog and we live next to a farm, so the house is dust year round. I am an hour south of Portland, Or.