Well, after working at this almost non-stop today, I think I've managed to figure out how the dms.dat file is structured. I figured I'd post my conclusions so that if anyone else wants to try adding custom dynamic music to their levels then at least they'll have this as a base.
For starters, as far as I can tell, the 'etr' tracks do nothing. I came to this conclusion after listening to the 'etr' files in assets0.pk3 and noting that I never seemed to hear them in-game. I always assumed that the 'etr' tracks were transition music from action to explore, and the 'atr' files were transition music from explore to action. After much experimenting, however, it seems to me that on encountering an enemy the music switches straight from explore to action, and then when all enemies in the surrounding area are dead, the 'atr' tracks play, thus the 'atr' tracks must be transition music from action to explore.
With regards to the 'marker' and 'time' figures, this is what I've been able to determine. The explore entry must have a number of markers equal to the number of 'atr' tracks. The explore markers are positions that the explore music will start playing at after an 'atr' track, depending on which 'atr' track has been allocated to that marker (done with the command 'nextmark'). For example, if you have two 'atr' tracks, with the first allocated to marker0 and the second allocated to marker1, then if the first 'atr' track plays and then finishes, the explore music will resume at the position specified by its marker0 figure. Likewise, if the second 'atr' track plays and finishes, then the explore music will resume at the position specified by its marker1 figure. As for the action markers, it seems that these are positions in the action music that will be randomly selected and switched to directly from the explore music once an enemy is encountered. (I may, however, be wrong about the random selection.)
The 'time' figures (for the 'atr' tracks) are positions in the action track that indicate when the 'atr' track can play, once no enemies are nearby and the action is over. What this means is that once the action is over, an 'atr' will play as soon as the next time index is reached. The 'atr' that will play will depend on which time index is reached as soon as the action is over and to which 'atr' that time index is allocated. Ideally, the entire action track should be covered in this way by the 'atr' tracks so that you don't have the action track playing on while no action is actually taking place.
That's pretty much what I've found. I am in no way saying this is all completely true and correct; all I'm saying is that these are the conclusions I've come to based on my experimentation. Hopefully, anyone who uses my findings will have an easier time of implementing custom dynamic music than I have!