Yes, you can downgrade your Agency Plan at any time. The changes to how your plan works and the number of available domains will take effect at the end of your billing period. When you downgrade to a more limited plan, the only change is that you will have fewer domains available to use in the following year. All domains from your current Agency Plan will remain available.
For example, if you have an Agency Pro plan with 48 websites/domains per year and downgrade to the Lite plan with 24 websites/domains per year, you will still keep your 48 domains from the current plan. In the following year, your new Lite plan will give you 24 new domains to use.