Have you started addressing your ankle?
There are some Pro videos that address different aspects of ankle mobility
Free Your Heel, Free Your Mind
Ankle Positional Fault Fix: Jill Miller Style
Squat Quick Test: Is it your Ankles or Hips?
Case Study: Tight Ankles = Bad Squatting

Adductors
Episode 208: Wide Stance Prep or Fixing Sumo+Decompression
Episode 14: Celebrity Goat Death Match: Chris Spealer

It should not be painful. There is a difference between pain and a stretch or mobilization.