I was thinking in terms of using the guns as fully fledged field or anti tank guns. A 4.1 inch gun would have made a handy field gun or self propelled gun, given that the gun and cartridge were known quantities.
Part of the reason might be that 4" guns were too much gun/carriage for too little shell weight. The closest approximation to 4" naval gun performance was probably the German 10 cm schwere Kanone 18, which was not liked and often relegated to coast defense duties. A carriage, SP or towed, that could take 4" gun was seen as better employed with a 5.5" gun or 155mm howitzer.
For anti-tank use, the heavies that would require a 100mm gun only showed up in numbers in 44-45 and by then the US/UK preferred improved ammunition for existing calibers (90mm HVAP and 17lbr APDS) to larger guns firing full caliber AP rounds, since they had those guns developed as anti-tank guns and for use in AFV, the US moving away from towed ATG before they could get a towed 90mm in service. When both of those countries looked at heavier ATG and AFV guns, they went to anti-aircraft guns (32lbr/105mm/120mm) as the basis instead of naval weapons.
Most countries seemed to decide that you had to get up in the 4.7" - 5" range to have a heavy enough shell, even for long range counter-battery fire. That's where you see the Soviet 122mm guns and the late war German fascination with 128mm as not just an AT gun, but also a field piece. The US and UK worked with slightly smaller, 4.5" guns, but they didn't seem that popular, had low muzzle velocities compared to most naval guns, and were dumped as soon as the war ended in favor of the 5.5"/155mm howitzers that could be employed using the same carriages and tractors.
The Soviet 100mm gun is an exception, but by the time it came into play, 1944, it was a dual purpose field/anti-tank gun and the Soviets wanted a heavier anti-tank punch than even the contemporary 85mm field gun could provide. The Soviet faith in ATG persisted with 100mm guns, while the US/UK never got any conventional ATG fielded in numbers larger than 76mm. Even given all that, for field gun purposes the Soviets moved up to 130mm in the decade after WWII.