Frost inside closet

The closet wall was adjacent the exterior wall in a 1' by 1' area where I had a frost build up. Obviously when the weather changed it turned the drywall in that area to mold. Insulation had some discolouration but not much. I ripped out drywall. Couldn't really notice a draught but once I taped the vapour barrier back up I noticed it being sucked in and pushed out with any sorta windy condition outside. The house was built in 73. I have no clue what to do or how to find this issue to correct. Looking for help!!