About a half mile into your hike, you'll pass a turn off for the Legion of Honor. You can take the trail up the hill to the right if your destination is the museum -- or if you want to parallel the Coastal Trail on the El Camino del Mar Trail above.
For this tour, continue on the Coastal Trail, past the Mile Rock Beach marker as well. If you have time for the Mile Rock detour, it's worth a trip to the ocean below, and to see the Mile Rock labyrinth. The hike down involves many stairs -- more than 100 -- so keep that in mind as you ascend back to the Coastal Trail.