Losing Pea Gravel from underneath small footer for greenhouse

We poured a small footer (9" wide and 4" deep) for a greenhouse to tie the bottom plates of the walls to. After removing the forms and in the process of all the pounding from the framing the pea gravel is falling out from under the concrete. Is this going to be a problem in the long run? If its not obvious, this is my first time doing something like this.