Japan, who had also shocked Germany in their opener in Qatar, finished top of Group E to reach the last 16, a stage where they have faltered three times before.
Germany that won in 2014 have crashed out at the group stages in the two world cups since.
Spain edged Germany on goal difference to take the second spot.
Russia 2018 was the first time that Germany had failed to pass the group stage since 1938 - which was a different format.
Mesut Ozil has kept silent about Germany on the day of their fight for survival in the world cup.
'I expect from Costa Rica a defensive performance as they did it against Japan. We have to find solutions against that,' said the German coach.
His recent Bundesliga scoring form has now spilled over to the World Cup and 'The Gap', as he is know due to a space between his front teeth, is being hailed as Germany's improbable hero.
Spain top the standings with four points and now need only a draw in their final Group E game against Japan to progress.
Germany need a win to keep their World Cup hopes alive.
Asked whether the four-time world champions were still the mighty tournament team of the past as they face the prospect of a second early World Cup exit, he said the game on Sunday would determine their reputation.