How long is a piece of string …..?
A little movement at the wheel (top) end of the column is likely to have minimal effect at the bottom but a lot of movement might put undue strain on the rack.
If all the loose rubber bits have now fallen out I would guess it is unlikely to get worse – but sod’s law will always prevail so assume it will.
The answer has to be – it depends how bad it already is and how far you are going. Only you can tell.
B………………..
Bernie