In this episode of ImpacTech, Precision in Motion, we shine a spotlight on the interdisciplinary team behind the Hand Assessment Tool (HAT). This innovative, camera-based system automates hand range of motion assessments in under 90 seconds. Developed by students from UC Berkeley, this low-cost, user-friendly tool earned runner-up honors in the RESNA Student Design Challenge and the IMPACT Award for its potential to transform hand rehabilitation.
Host(s): Dr. Mary Goldberg, Co-Director of the IMPACT Center at the University of Pittsburgh
Guest(s): Student Design Team - HAT, University of Washington: Yuka Fan, Emily Boeschoten, Adria Gonzalez, Sasha Portnova
IMPACT Center | Website, Facebook, LinkedIn, Twitter
HAT Team Information | Video, Website, Publication
Discussion Topics
SPEAKERS
Dr. Mary Goldberg, Yuka Fan, Emily Boeschoten, Adria Gonzalez, Sasha Portnova
Mary Goldberg 0:01
The IMPACT Center at the University of Pittsburgh, supported by the National Institute of Disability Independent Living and Rehabilitation Research, proudly presents ImpacTech. Welcome to season four of the ImpacTech podcast. On today's episode, Precision in Motion, the student innovators behind the hand assessment tool. We spotlight a team of bright minds tackling a real world clinical challenge with creativity and compassion. These engineers and health science students from the University of California, Berkeley, earned honorable mention in the RESNA student design challenge and were selected as runners up for the Impact Award for their development of hat, a low cost camera based tool that automates hand range of motion tracking in under 90 seconds, with the potential to improve rehabilitation outcomes, streamline clinical workflows and make hand therapy more accessible for both patients and providers, that exemplifies the kind of innovation that makes assistive technology more inclusive and impactful, recorded remotely from my soundproof bedroom closet in Pittsburgh. PA, this is your host, Dr, Mary Goldberg, and welcome to our 30th episode of The emphatic podcast series. All right, so please tell us a little bit about your team in particular. Do you have different backgrounds, and how did you all come together to work together as a team?
Sasha Portnova 1:42
I can probably start with this question. Our team is extremely diverse. My name is Sasha, and it kind of started when Andrea P, T, o, t, p, G, student in rehab medicine and I was a post doc at the University of Washington. At the time, we were working on different projects related to quantifying hand function in individuals with upper body disabilities, and we have been working with wearable technology to capture all of the variability in hand, all The various degrees of freedom in that single hand. And we came to discussing how one way to measure hand function, or improvements of hand function in the clinical realm, has been through manual gonneometry. Which are those like plastic goniometers, angle measuring tools, and you have to, like, apply them to every single joint on the hand, which there are quite a lot of them so and how it's just time consuming. And being the engineer I am, like, why not try to apply the engineering tools to try to address that problem? And then we got more expertise on our team. We had Yuga fan join us. She's now has a master in human center design and engineering, and she's a UX design professional on our team to make sure that the product we're creating meets the usability standards and is all pretty and very user friendly. And we also have Emily Bon who has both experience of the clinician she's currently in OT school getting her ot education, and she also is a user who uses hand therapy, and kind of knows firsthand how cumbersome those Hand assessments can be. So we have pretty well rounded, very diverse team.
Mary Goldberg 3:44
Wonderful. Thank you so much for the introduction, and it sounded like Sasha, based on what you shared, that Andrea's experience firsthand, clinically. Yeah, pun intended, I guess clinically is really what motivated the design challenge of the hand function assessment. Audrey, I'm wondering if there are a couple cases that stand out to you in particular that motivated you to take on this particular challenge.
Adria Gonzalez 4:11
Yes, I think that going back a little bit, I remember Sasha, I think that when we came out with this, we were doing a study with World technology on hand, and I had invited one of the hand therapies, where, at that time I was doing like a brilliant job at a hand clinic. And I had invited them to participate, actually into the study and to get their feedback. And after we finished the study, we the three of us sat down, and I don't remember if Emily was there too, because Emily, at that point was also like helping us collect the data, and we were having a conversation, and I think that it came out as a way of sometimes at the hand clinic, we see people very far especially here, like in the warming region, right, like that. It's like Central Washington and Idaho and Alaska even like very. Remote places. And just to give a little bit more context, the clinic where I used to work, it's a traumatic hand injury. So it's people that has had like surgery for usually, some sort of like accident on their hand. And they people from all these different states would come here to harbor V, which is like a level one child center get their surgery, which is usually, like, very intense, and then they get sent out to like, different places, right? So we sometimes they have to, depending on how it works, they have to travel here, like to Washington State, or like to Seattle to, like, get their therapy, and that's difficult for them. So I think that the conversation started in a way that, how can we have more tools to, for example, have an assessment on these people while they're at home, right? And then we started discussing the webcams and then the leap, and then, like other tools that were one option, for example, we have these people that are far away, and can, can we measure the range of motion that's so important in that specific setup without having them to travel, like, all the way over here, right? So maybe we can deliver our therapy, but we can get, like, still reliable numbers, if we inst them to do that. And I think that that was one of the reasons, and then a sexual saying too. I think that there is a sort that there's a shortage of pain therapies here in the area, in Seattle, and I don't know nationally, but I know that here in Seattle area, at least at that point, there was a shortage of hand therapies and taking measurements, like all the time, that's like, time consuming. And then you need to, like, introduce this data into, like, the patient chart. So the idea of having something that can kind of like aid that participant, or I keep saying participant, and I will be using participant and patient interchangeably. I'm just doing research, so everybody's participants, but we're talking about patients now. So I think that having some sort of a tool that that patient can arrive and just can get that first thing that we do watch therapies, right? We just like, measure your own range of motion by themselves. That could be like time shortening and then being able to extract that data and put it like, directly into their chart, and then also as a way for I think visually, it's really good to have a way to communicate that information to our patients as well. Thinking right now, I wear my smart watch, right? And we can see, we can track progress with it, right? We can see graphs. So think that this tool brings, brings that to and then we were also thinking on reliability, usually in my experience and and I'm not a hand therapist. I've had like experience in hand therapy, but I'm not a certified therapist, so what I've experienced is, like, in the clinic where I was there were like, three regular hand therapist OTS working there I will go per diem, like I was trained by them. So the way that we would obtain our range of motion was, like, very similar, I would say, like, within five degrees of difference, right? So we have reliability within the team, but then we also have times when we will see a patient, and they will be follow up, like in another city or in another town, and then they may come back to see like the surgeon again, right? And maybe see us one more time. So how do we find a way that we can now, like a consistent way of like collecting data in between these different healthcare professionals that may not work at the same place. Does that make sense?
Mary Goldberg 8:23
Nice. It absolutely does. Yeah, when we think about the value proposition of it, it certainly seems very efficient for those clients who get to see you in person, but then also for some remote assessment as well. So seems like there's really great potential there. Would one of you be able to walk us through how the hand assessment tool, or you all use the acronym of hat works?
Yuka Fan 8:49
I can talk about that. We start with the patient input. This can be done by the patient or the clinician, and after the initial logging, you come to this setup page where you can choose to do left hand or right hand or both hand and pre or post therapy, because you want to able to see the difference, and also which device you're using so leap or camera, webcam based camera. So after this set up page, which is the second screen, the patient goes to the task selection. And on this page, we have six tasks in total that measures up to 19 joints. And after selecting task, the assessment begins. And assessment, it starts with visual instruction, with text instruction to show how to orient your hand. So that's the first step of the instruction, and the second step is how to position your hand. So we have these two separated because we want to make sure they're having the correct orientation first and then the correct pose. And. And we always ask to ask the participant to hold their post for two seconds, so that's when the recording happens and tracks the data real time. And after all this assessments, patient comes to the data visualization page where they can click around which finger which joint and see their progress in a visualized line graph
Sasha Portnova 10:22
I can probably add to that one step back is so hat. It's an interface we develop for computers. The information that it takes can be in two different ways, like the way it tracks the hand movement itself. One is through what's called like a leap a leap motion tracker. It's a very commonly used tool, mostly for, like, game development, interaction and VR and AR that people have used. It's also, like, webcam based, but it has infrared cameras. So the idea would be, is that in a clinic, this would be placed so they're, like, about $200 per piece. It will be placed right next to the computer or iPad, whatever it is the person would record using this or the second way is we developed another option for clinics that potentially don't have extra money to spend on these devices, or don't have access to them in one way, shape or form, is to do it via just webcams, because right now, the motion tracking through webcams have really gone waste in the past probably five years. And so the second option would be to use a webcam with which the patient would be able to record these hand movements that they're directed to do, and each movement is associated with a specific joint range of motion. And then they get to the visualization page, and then there's reports that can be exported.
Mary Goldberg 11:57
Thanks for the description. So when Sasha was talking, she was holding up the camera. Is it right that it looks to be about the length of a finger, ish in length, and certainly very lightweight, is a cell phone camera able to be used as well?
Sasha Portnova 12:13
Well, we don't have the code to rate to export it from cell phone cameras. It could potentially, yeah, because the model that we use for tracking hands can be used with any webcam. Technically, yeah, it's just the implementing this capability into the interface itself.
Mary Goldberg 12:35
Got it. Okay. Yeah, I know many therapeutic devices that use sensors or wearables instead of the camera based system. Could you describe a little bit about why you decided to go the route of the camera?
Sasha Portnova 12:48
Yeah, I think one of the ways is that you minimize the number of things that are being put on the patient, which I think one of the priorities that we aimed for in terms of the development of this interface was to reduce the amount of time clinician spends performing a certain task that is very important to the hand therapy, because understanding whether the person has improved or their results have deteriorated is very important. It's important to kind of incentivize insurance companies to continue paying for therapy if you can show progress. But the problem is that right now, traditional assessments can take about 1015, minutes, with the standard treatment being about 4045, minutes. And so that's a very big chunk of time that we want to minimize. And so when it comes to other wearable technologies, like, for example, there are wearable gloves out there that Adria and I have used in our study. They just take a long time to be fitted onto the individuals to Don and then doff it, and then ensure that, because no true glove will be one size fits all. So we have to go around ways to really kind of have this one size truly fit all, and that requires a lot of fitting. So this is just a matter of kind of reducing time and simplicity for a given clinic, too.
Mary Goldberg 14:21
I know that you did some early reliability testing with participants, and I'm wondering what your key takeaways were from that early validation exercise.
Sasha Portnova 14:30
It's been variable. I think there's, there's quite a lot of improvements that can be done, which is what I'm working on right now. Computer Vision, while it has improved significantly over the past five years, as I mentioned, it still has quite a lot of challenges when it comes to such intricate systems like human hands. Human Hands are very complex, and I think that is why all of us in this team really enjoy working with hands. I think that's the reason. Usually, when it comes to like the rehab world, people kind of split in the middle. They usually either work on lower extremities or they work on upper extremities. And a lot of reason for people not working on upper extremities has to do with the fact that they're complex. But the opposite we like that. It's complexity. And so because of that complexity, there's quite a lot of improvement that can be done in terms of motion capture. Was video cameras, because, like, depending on different angles, your one finger can be or two fingers can be occluding other fingers, and then if the camera doesn't see it. It's hard for it to predict. And then the other part, as Emily is showing it's really most of the models have been trained on very average people with very average hands, and so most of the models just kind of try to fit that to a hand that they see, which they can struggle with. When it comes to two hands that are either having a lot of spasticity or a lot of contractions or, you know, if we're talking about post surgery, they're just not the way that a model has been trained to look at them, so the results were variable. The one thing that we really hit the nail on, I think the measures are consistent, so it's this thing that Adria mentioned in the beginning, is this idea that if you were to go to one clinician, or, let's say you just had a surgery, and then your surgeon is kind of testing your range of motion in their clinic with their tools. And then you get referral to go to a hand clinic, and then the hand therapist there is doing it their own way, which is technically the same things. It's usually using the manual goniometer, but the variability in that is very high. But with these tools, what we found like with the leap and the web count the as long as you measure it like is the hands replaced consistently, then the measures are very, very consistent, very agreeable with each other.
Mary Goldberg 17:18
Thank you. Although I know you said that some of the testing has been managed in kind of a general way or or with average hand models, so to speak. I know that you've been very purposeful in designing this system with accessibility and inclusivity in mind, and you certainly mentioned that that Emily herself is going is an OT, or pursuing OT and will be ultimately, one of these clinicians who I believe also experiences some low finger dexterity. So Emily, I'm I'm interested in your perspective on this and how your insights have helped shape the design.
Emily Boeschoten 17:56
Yeah, so it's interesting because the context, I've been a hand therapy clinic patient since 2018 so I have a I've gone to like, three different hand therapists in different state, and so I've seen all the different interpretations of goniometry and all the different approaches to hand therapy assessment. So it's been good in that regard, because I know, like, how long things are supposed to take, and then I also know how data has been presented to patients in the past. So I very much appreciate the hat experience from a patient perspective, but the biggest thing for me, Well, I'll start with the patient perspective in terms of accessibility. So I've been around the disability community since I was 15, which has made it so I've luckily learned about a lot of different disabilities and how to make things accessible, and just done stuff that from that regard. But Sasha has as well. So between all of us, we made sure to have both visual audio and different descriptors for all the instructions to make it so no matter what the patient or the clinician is dealing with in terms of disability, they'll be able to interact with the interface, and then also making it so that the visuals are there's good contrast, making sure that people can navigate the website and making sure that it's also efficient from all perspectives. So that's also part of why we didn't want to do wearables, because the cumbersomeness of putting it on and taking it off, especially for clinicians like myself, from the clinician perspective, there is a big push to get more people with disabilities in occupational therapy. However, there is a big barrier when it comes to technical standards, and so we learned a lot about accommodations assistive technology, and that's in regard there is good tools in regards to documentation, like voice to text, like a lot of the. Clinician side software now has good accessibility features. However, if you need to be able to measure someone's hand, it's going to take me patients entire appointment to measure all 19 angles in each hand. And no one wants to do that. I don't want to do that, and patient doesn't want to sit through that. And also, the employer doesn't want to pay for that. And so there's a big barrier where there's not a lot of assistive technology in terms of the technical standards that come with occupational therapy. And that's the biggest barrier for employment with occupational therapists with people with disabilities. And so there's this big underlying current in the OT world that doesn't really get talked about, and people don't really want to say it, but there's a big understanding that, from the OT perspective, our patients can do anything except our job, and so that is my hope, one step towards addressing this issue. Because I don't know about you, but I personally have gotten a lot more value from just talking with my fellow disability members than I have from a lot of the occupational therapists I've gone to and physical therapist over the years. And so I think it'd be great to have more representation in the OT community. And I really have enjoyed working with this team, because you can I even met at like, a summer camp for teens with disabilities, and so that's why I was like, Oh my gosh, you think you have to come join us, because you've actually, like, been around the community where you have some understanding of what different disabilities look like, and can come across
Mary Goldberg 21:35
That's so cool. And I love the diversity of your team and the different ways that you've been able to weigh into the design and CO create this product. Adria, please go ahead.
Adria Gonzalez 21:49
Yes, I was thinking, well, while Sasha was discussing on my perspective as a clinician or as a researcher, when we use wearable technology, I've been using wearable technology for a couple of years now, with Sasha and with Emily tours. And I think that when you ask the question about, like, what do you use cameras and way to use, like a visual system instead of like something on, I think that there are a few things I wanted to comment on, and Sasha touched on on one of them, which is, like time like that, that time consuming, of like, putting something on someone. It's very like diverting to like, the first time like this is it's not feasible, right? Like, to get, like, very accurate data. Also, I want to put in perspective, like, when we go and this tights up into like that design of like technology sometimes based on typically moving hands, or typically dimension hands. Many times, people that come to the clinic or people that come to research, they have pain, they have wounds, they may not be able to move their fingers as much as as they can. That's why we're measuring their fingers. So I think that this is a really good solution to break all these barriers. I also see like another, especially like after covid, for example, at the clinic, right? Like hygiene wise, right? Like having, like, something that somebody doesn't need to even touch, right? So we minimize the the risk of, like contagious, especially on people that have had surgeries or have like wounds or pain. So I think that these are all like advantages that I think that are important from like, a feasibility perspective on this tool. And then I also wanted to say like that I agree with everything that Emily is saying too on the fact that making like these tools like accessible for people, so people with disabilities can join us too. I think that is important.
Mary Goldberg 23:38
Thank you for joining us for this episode of ImpacTech, we've loved chatting with the hat team, and our next episode will explore where the hat team goes next, including their future work, as well as the impact of the resident design competition and their advice for innovators like them to help get technologies to market. If you like impactic, please review us on Apple podcasts or wherever you listen to podcasts, thank you again for tuning in and continue to make an impact in whatever you do.
Mary Goldberg 24:22
A quick note from our sponsors, IMPACT initiatives are being developed under a grant from the National Institute on Disability Independent Living and Rehabilitation Research. NIDILRR is a center within the Administration for Community Living Department of Health and Human Services. IMPACT initiatives do not necessarily represent the policy of NIDILRR, ACL, or HHS, and you should not assume endorsement by the federal government, and the same goes for the University of Pittsburgh. We would like to thank our ImpacTech guests and our production team led by Dr. Michelle Zorrilla at the University of Pittsburgh, Department of Rehabilitation, Science and Technology.