With guns against armored ships, there are basically two approaches:
- Penetrate the armor and explode a charge, even a relatively small one, deep inside the ship.
- Attack the "outside" of the ship causing enormous damage to the unarmored (or just lightly armored) parts and, if you're lucky, starting fires. This is analogous to trying to get a mission kill on a tank with artillery not by penetrating the armor, but by destroying vision blocks, fire control equipment, running gear, etc., all of which require expensive and time consuming repairs.
The first option is the one that sometimes produces catastrophic damage, e.g., HMS Hood, and seems to work best (as per one of T. A. Gardner's posts and I think every navy's practice). HESH may penetrate an outer layer of armor (either literally or via spalling), but the combination of layered armor and critical spaces (e.g., ammunition and power plant) being deep in the ship seem to make HESH unlikely to produce the effects we're looking for in Option 1.
But option 1 is not always available. There may be situations where the combination of the enemy's armor and the range and the power of your own guns takes it off the table. Then the question becomes what's the optimal ammunition choice for Option 2: HESH, HE, or something else?
HE damages not just by blast but also by fragmentation, so it's very much aligned with Option 2 and it has a bonus of being much easier to spot fall of shot at long range than it is with an AP round. Again T. A. Gardner has already posted a good example of HE's work.
HESH might have the advantage of being more destructive to the outer armor layer than HE while still doing a great deal of blast damage. This is traded off for less fragmentation. It's hard to guess where that tradeoff winds up, but I suspect that, given the blast hardening anything outside the armor envelope of a battleship has to have, just to survive the muzzle blast of a main gun salvo, the added penetration of fragments might be important.
I'd note that neither HE nor HESH is designed to start fires, which are particularly dangerous to ships, so I'd wonder if some sort of very large high explosive - incendiary round might have been do-able, but that would probably be an over-specialized round, if it were possible at all.