Yes, US units wouldn't matter in sentence where actual magnitude is not important for meaning of the sentence. But if the audience is supposed to imaginary feel the mass of weight to be lifted then it's a different case.
Most non-US people are familiar with US miles. So when told to walk 5 miles, one have an idea in the head how far it is: 5 km and around half more, so it's under 2 hours of swift walk.
Some other US units are obvious by it's name. So when non-US person is told that someone is 6 feet tall, he looks at his own foot and imagine 6 of these. Medieval units sound weird but you have an idea how much it is.
Then there are units most non-US never heard of, like volumetric ounce, mass ounce and some other I've never heard of. So the only thing that goes through head is WTF.
Honestly, when I read the sentence "You begin with 50lb and do a set of ten." my head images lifting 50kg (hard-core start) even through I know that 1lb is just 0.45kg