The three points lifted the Burgundy side to 17th placed from the bottom of the log.
Guided by Congo coach Jean-Guy Wallemme, Auxerre struck first on the half hour after French star Roy Contout headed home a cross by Burkina Faso winger Alain Traore.
Auxerre then hung on to their lead until the 86th minute when the burly Ndinga waltzed past the Dijon defence and slotted home through the legs of the goalkeeper to seal the win.
Wallemme and Ndinga are expected to lead Congo's campaign for a place at the 2013 Africa Cup of Nations and the 2014 World Cup.
The Red Devils travel to Burkina Faso before hosting Niger in two World Cup qualifiers in the first two weekends of June. they then travel to Kampala to try and protect a 3:1 lead over Uganda in their Afcon qualifier.
Young Nigeria midfielder Rabiu Ibrahim was denied a goal on his debut for Celtic when his effort was helped over the line by team-mate Gary Hooper who was offside.
Rabiu joined the fray in the 76th minute and impressed for the Scottish champions who beat St. Johnstone 1:0 in the Scottish Premier League game.
The U-17 World Cup winner arrived in Celtic in January after he was released by Dutch side PSV Eindhoven.
Other Africans in the game were Celtic duo Kenya midfielder Victor Wanyama and Sierra Leone striker Mohamed Bangura.
Bangura was playing his first game since December last year when he was injured. The striker tested the goalkeeper before hitting woodwork.
- All Thursday's African goalscorers overseas