Do I have to apply to the CAA to get an A2 CofC

No, your RAE (UAVHUB) will send you your certificate on behalf of the CAA

When you have completed your A2 Certificate of Competency Examination, UAVHUB will issue you with your A2 CofC via email. You will receive a digital PDF and this will be valid for 5 years.

After 5 years (or whatever is shown as the expiry date on your certificate), you will have to renew your A2 CofC.