There are other letter-like symbols that could be used, if one is willing to poach characters from other languages. The basic ASCII alphabet doesn't have any of those, but extended ASCII and Unicode support them. This isn't as helpful as one might suppose, however; the advantage to ASCII is its universality, and furthermore, the primary advantage of using letters if that monsters are more easily recognizable by using the first letter of their name to represent the foe. Most players speak English, and there just aren't many monsters iconic enough for inclusion that use diacritical marks on their first letters.

Yet, despite all these options, it's still not enough for some games. Nethack, in particular, has a scheme whereby monster letters (upper- and lower-case considered separately) indicate a general monster type, and color depicts species. A comment in the source code notes the pattern behind these: monsters with an elemental affinity are to use an appropriate color (red for fire, white for ice), and leader or royal monsters are purple. Yet under this system there are some monsters that appear identical on an ASCII display. The most troublesome result of this happens near the end of the game, when the player reaches the elemental plane of Earth. Upon entry, the first two monsters the player meets are always another reincarnation of the Wizard of Yendor and an elflord, both represented by a purple "@". The elflord is a middle-level foe, not dangerous to a player who has literally been to hell and back, but the Wizard could be quite a pest.

Second problem: Finding monsters for the less-common letters

Fitting all the monsters into 26 letters isn't exactly easy, but worse is finding monsters for the less-common letters of the alphabet.

ADOM is maybe the game least burdened by these considerations. While it does try to conform the monsters into families represented by particular letters, there are many letters that don't obviously match family names. Grues are 'x'es, while humans are generally '@' barbarians are 'K's, bugbears are 'g', bears are 'N', golems are 'Y', sea monsters seem to be 'A', and so on.

I've been thinking a bit about this problem myself, and started compiling a list of monsters that begin with different letters. A few letters are easy. It turns out there are an abundance of G monsters: Gnome, Griffin, Gargoyle, Gremlin, Gorgon, Golem. V, a fairly uncommon letter, has more monsters than one might expect: Vampire, Vrock and Vortex.

A few letters are troublesome, in particular: J, K, Q, X, Y and Z. It is entirely because of the letter Y that the Yeti appears in so many roguelike games. The difficulty of finding good monsters for K, Q and Y, ultimately, was responsible for the beginnings of a roguelike tradition: the inclusion of weird monsters in order to fill out the whole alphabet.

To elaborate: the first versions of Rogue borrowed much of the opposition from Dungeons & Dragons. Later revisions switched out some of those monsters for a more idiosyncratic set. The rumor is that this was done in order to foil Rog-O-Matic, the early Rogue-playing borg, but to me it seems at least possible that it was to distance the game from D&D around the time the game was being sold commercially by Epyx.

When the monsters were changed, they had to find new monsters for some of the hard-to-fill letters. This was the point that Lewis Carroll's Jabberwock became a Rogue monster, as did the Quagga, an extinct relative of the zebra, the Kestral, a type of falcon one might not expect to find underground, and my personal favorite, the Xeroc, replacing mimics. (Think about that name for a moment, in relation to copying things.)

Artist's rendering of a Yeek

Not even Angband is immune to this. One of that game's trademarks is a race of very-low-level humanoids called Yeeks, which actually originated in Moria. Yeeks are something of an unofficial mascot for the games, a race of monsters that are comically weak. According to the deleted Wikipedia page on them, they're called Yeeks because that's the sound they make when stepped on. Yeeks have a kind of popularity in Angband culture; the basic game includes the King and Prince of the Yeeks (Boldor and Orfax) as unique monsters, and variants add the Yeekish Queen and President.

Some recent variants even make Yeeks a playable race. Being so weak, they gain experience levels rapidly, but unfortunately they must live with a -5 to Luck. Why would that be? Because, if you were born a Yeek, it's not exactly like the laws of chance were on your side.

Others:

Nethack: commercial-at: Human or Elf, space: Ghost, colon: Lizard, semi-colon: Sea Monster, apostrophe: Golem, Amphersand: Demon

Angband: period: Lurker, comma: Plant Monster, dollar-sign: Creeping Coins, various symbols: Mimic

Sources:
The Rogue Vede-Mecum:
http://www.monmouth.com/~colonel/rvm.html
Wikihack:
http://nethack.wikia.com/
Angband monster spoilers:
http://www.thangorodrim.net/spoilers/monsters0.html

The ampersand demons above come from the Geoduck Tileset, a clever Nethack modification that makes monster graphics tiles into appropriately-customized ASCII versions.
Image of a quagga (R.I.P.) from the Wikipedia entry on that animal.
Image of Fobby (shamelessly pawned off as a Yeek) is from Earthbound, copyright by Nintendo. Used here because Earthbound is awesome. Pic stolen from Fobbies Are Borange.