Cornell researchers have developed a robotic feeding system that makes use of laptop imaginative and prescient, machine studying and multimodal sensing to soundly feed individuals with extreme mobility limitations, together with these with spinal twine accidents, cerebral palsy and a number of sclerosis.
“Feeding people with extreme mobility limitations with a robotic is troublesome, as many can not lean ahead and require meals to be positioned straight inside their mouths,” mentioned Tapomayukh “Tapo” Bhattacharjee, assistant professor of laptop science within the Cornell Ann S. Bowers School of Computing and Data Science and senior developer behind the system. “The problem intensifies when feeding people with extra complicated medical circumstances.”
A paper on the system, “Really feel the Chunk: Robotic-Assisted Inside-Mouth Chunk Switch utilizing Strong Mouth Notion and Bodily Interplay-Conscious Management,” was introduced on the Human Robotic Interplay convention, held March 11-14, in Boulder, Colorado. It obtained a Greatest Paper Honorable Point out recognition, whereas a demo of the analysis staff’s broader robotic feeding system obtained a Greatest Demo Award.
A pacesetter in assistive robotics, Bhattacharjee and his EmPRISE Lab have spent years educating machines the complicated course of by which we people feed ourselves. It is a difficult problem to show a machine — every little thing from figuring out meals objects on a plate, selecting them up after which transferring it contained in the mouth of a care recipient.
“This final 5 centimeters, from the utensil to contained in the mouth, is extraordinarily difficult,” Bhattacharjee mentioned.
Some care recipients could have very restricted mouth openings, measuring lower than 2 centimeters, whereas others expertise involuntary muscle spasms that may happen unexpectedly, even when the utensil is inside their mouth, Bhattacharjee mentioned. Additional, some can solely chew meals at particular areas inside their mouth, which they point out by pushing the utensil utilizing their tongue, he mentioned.
“Present know-how solely appears at an individual’s face as soon as and assumes they are going to stay nonetheless, which is usually not the case and may be very limiting for care recipients,” mentioned Rajat Kumar Jenamani, the paper’s lead writer and a doctoral pupil within the area of laptop science.
To handle these challenges, researchers developed and outfitted their robotic with two important options: real-time mouth monitoring that adjusts to customers’ actions, and a dynamic response mechanism that allows the robotic to detect the character of bodily interactions as they happen, and react appropriately. This allows the system to tell apart between sudden spasms, intentional bites and consumer makes an attempt to control the utensil inside their mouth, researchers mentioned.
The robotic system efficiently fed 13 people with numerous medical circumstances in a consumer examine spanning three areas: the EmPRISE Lab on the Cornell Ithaca campus, a medical heart in New York Metropolis, and a care recipient’s dwelling in Connecticut. Customers of the robotic discovered it to be protected and comfy, researchers mentioned.
“This is likely one of the most intensive real-world evaluations of any autonomous robot-assisted feeding system with end-users,” Bhattacharjee mentioned.
The staff’s robotic is a multi-jointed arm that holds a custom-built utensil on the finish that may sense the forces being utilized on it. The mouth monitoring methodology — educated on hundreds of pictures that includes numerous individuals’ head poses and facial expressions — combines knowledge from two cameras positioned above and under the utensil. This enables for exact detection of the mouth and overcomes any visible obstructions brought on by the utensil itself, researchers mentioned. This bodily interaction-aware response mechanism makes use of each visible and power sensing to understand how customers are interacting with the robotic, Jenamani mentioned.
“We’re empowering people to manage a 20-pound robotic with simply their tongue,” he mentioned.
He cited the consumer research as probably the most gratifying side of the mission, noting the numerous emotional impression of the robotic on the care recipients and their caregivers. Throughout one session, the mother and father of a daughter with schizencephaly quadriplegia, a uncommon start defect, witnessed her efficiently feed herself utilizing the system.
“It was a second of actual emotion; her father raised his cap in celebration, and her mom was nearly in tears,” Jenamani mentioned.
Whereas additional work is required to discover the system’s long-term usability, its promising outcomes spotlight the potential to enhance care recipients’ stage of independence and high quality of life, researchers mentioned.
“It is superb,” Bhattacharjee mentioned, “and really, very fulfilling.”
Paper co-authors are: Daniel Stabile, M.S. ’23; Ziang Liu, a doctoral pupil within the area of laptop science; Abrar Anwar of the College of South California, and Katherine Dimitropoulou of Columbia College.
This analysis was funded primarily by the Nationwide Science Basis.