It sounds like you are not creating torque.
Push Up Butt Acuity Test (from BASL p 56)
Get into the push up start position by making your hands as parallel as possible positioning your hands underneath your shoulders and pinning your feet together.
Next, go through the bracing sequence: Squeeze your butt, align your rib cage over your pelvis and get your belly tight.
Key is to make sure your wrist, elbow, shoulders are aligned and feet are together.
If your feet are apart it is difficult to engage your glutes to stabilize the trunk.
If your glutes are disengaged you can’t organize your trunk which could be the cause of the issue you are seeing at your shoulder.
When your butt is offline you’re susceptible to spinal faults.
A broken midline places additional stress on your shoulders.

You need to create torque through your hands(hands straight as possible elbow pits forward) to stabilize your shoulders and thoracic spine. If your hands are in a position where you can’t create torque its hard to stabilize and generate force.