fixing this by yourself is very hard. it is nearly impossible to see behind you when you are doing movements. i have tried mirrors and videocameras on myself and they are ok but still hard.  and it gets harder still if you have your t-spine curved in the wrong direction when your torso is upright. 

generally the scapula should move along the scapular “plane” which is not a flat plane but the plane created by the curve of your ribcage. it is possible to feel that and some of the exercises in that youtube channel can help with that. but you may need to find a professional to help you with this. i took a Dynamic Neuromuscular Stabilization class which dealt with this – you may be able to find someone near you here: http://www.rehabps.com. 

as for which muscle is dysfunctional – i would encourage you not to think of which muscle is dysfunctional because the focus is how to encourage the proper motor control of all the muscles of the scapula to fire properly and in the right sequence.  if you put the scapula in the right place and have someone teach how it should move, then everything will work correctly. trying to strengthen this or that muscle is not a good way to approach this problem.