Toyota added that it would temporarily suspend the service, with Toyota's president saying the event shows that autonomous vehicles are not yet realistic for normal roads.
The Japanese athlete, Aramitsu Kitazono, will be unable to compete in his 81kg category this weekend after being left with cuts and bruises following the impact with the "e-Palette" vehicle.
The move is also embarrassing for Toyota which has been using its sponsorship of Tokyo 2020, Toyota as a showcase for autonomous vehicles via a shuttle service, which has been running around the clock in the athletes' village.
On Thursday, however, one of the buses pulled away from a T-junction and drove through a pedestrian crossing while Kitazono, a visually impaired athlete, was walking across.
Tokyo police said that vehicle operators had told them they "were aware that a person was there but thought [the person] would [realize that a bus was coming] and stop crossing the [street]", according to the Asahi Shimbun newspaper.
Ironically the bus was under manual control at the time of the accident, adding that the vehicle "was barely moving, but it still managed to collide with a visually-impaired athlete at the Paralympic Games, raising potential concerns about the limitations of autonomous driving technology".