To grout flagstone patio

About 7 years ago I laid a flagstone patio in my backyard. I compacted the ground, laid a layer of about 3 inches of sand, then laid my stone. Between the stone I spread some red crushed gravel that was supposed to hold the stones in place. For the first year or so it looked great. But the soil is clay and any sand or gravel spread out on it disappears into the soil after about 18 months. For the first two years I spread more gravel each spring as the ground between the stones seemed to shrink away - the stones seem to set about a half inch above the ground. For 3 years we tried planting moss between the stones but that never took hold. I am wondering if grouting the areas between the stones would not hold them better in place and give a more level and even appearance. Any disadvanted to grouting at this late date? Any alternatives? Thank you.