A evaluation of various approaches to vision-based rep counting
On this article, I attempt to clarify my exploration of various vision-based repetition counting methods and talk about their execs and cons. Particularly, I spotlight 5 main methods through which laptop imaginative and prescient has been employed for rep counting.
Wearable sensors have been fairly widespread for reps and set counting. Owing to the truth that these sensors are costly and, generally, are solely restricted to monitoring a selected physique half, these days, quite a lot of focus has been on utilizing vision-based approaches for rep counting.
From numerous functions in exercise monitoring, sports activities, and gaming to serving to achieve perception into the variety of occasions a organic occasion (heartbeat, pulse rely, and so forth.) happens, Rep counting is an issue actively being solved in each academia and trade.
Key phrases: Rep Counting, Pc Imaginative and prescient, Pose Estimation.
RepNet: Class Agnostic rep counting within the WildRule-based train rep counting utilizing Pose EstimationExercise rep counting utilizing concepts from Sign ProcessingGymCamRep counting utilizing a DL-based Optical Stream Method
A lot of the methods we talk about within the weblog received’t be generic however moderately unique to a particular downside (for instance, exercises). Additionally, for a deeper understanding of the approach, please consult with the references offered.
paper: Counting Out Time: Class Agnostic Video Repetition Counting within the Wild [1]
One of the distinguished works round Rep Counting has been the RepNet, an end-to-end deep studying mannequin that may precisely predict counts on a broad vary of repetitive actions.
The RepNet mannequin takes in a video stream as enter and predicts two outputs:
Per-frame interval size: For every body that is part of repetitive motion, we wish to know the interval size (in time models) of that motion.
Per-frame periodicity: a rating indicating whether or not the present body is part of repetition or not.
A number of the key highlights of the RepNet mannequin embrace: A Temporal Self-similarity Matrix (TSM):
TSM is the spotlight of this rep counting approach. It’s the info bottleneck of the RepNet structure. This matrix helps relate the frames to one another by computing a pairwise similarity perform between two embeddings.One may also infer (utilizing heuristics) the variety of repetitions from these TSMs, which makes predictions from the RepNet mannequin interpretable.Various real-world repetition movies guarantee these TSMs are fairly various, and therefore RepNet has a pool of functions apart from simply rep counting.
One of the spectacular issues about this rep counting methodology is that it’s class agnostic (generic) and helpful to a variety of repetitive motions. RepNet mannequin is a classical utility of widespread Transformers in Pc Imaginative and prescient.
Nevertheless, the mannequin is constrained within the sense that the variety of frames within the enter video needs to be restricted. This may be attributed to the truth that the scale of the TSMs is the same as the variety of enter frames.
The mannequin is sort of heavy and sophisticated; therefore deploying this on a cell app or any manufacturing atmosphere could be fairly difficult and might need latency points.
weblog: Successful Interactivity Utilizing Pc Imaginative and prescient [2]
That is the commonest concept utilized in trade. Various well being and health startups have been engaged on constructing correct, light-weight, state-of-the-art pose estimation fashions which can be utilized to precisely rely the reps throughout train and supply posture correction suggestions, and so forth.
Main Steps concerned:
Given a particular train, you first give you definitions (guidelines) for states in that train. There could be a number of states in an train. A squat train, for instance, could be damaged into two states, say a decrease state and an higher state. Through the course of motion, the particular person doing train will shift from one state to the opposite. These state guidelines could be regarded as representing activation areas throughout motion.E.g., for a squat, these guidelines could be (th refers to threshold values):down: (left_knee_hip_dist_y < th1 and right_knee_hip_dist_y < th2)up: (left_knee_hip_dist_y > th3 and right_knee_hip_dist_y > th4)Throughout inference, we begin by computing the metrics (angles, distances normalized) utilizing pose-keypoints from the mannequin in real-time and verify whether or not a selected rule will get activated or not, and carry out rep counting utilizing the flag.
One of many main upsides of the method is that rep counting is quick and correct, and latency could be very low. Nevertheless, some main downsides embrace the next:
It’s not a generic rep counting.The pose estimation mannequin is extremely delicate to background noise and therefore rep counting as properly.Scalability Points: writing guidelines manually is a time-intensive course of. We additionally want to check the foundations with totally different variations in angle, orientation, posture, and so forth. Think about writing guidelines for 100s of workout routines within the corpus.
weblog: Constructing an Train Rep Counter Utilizing Concepts from Sign Processing [3]
Purpose: Use Sign Processing concepts like zero-crossing and peak detection to make an train rep counter.
This method is similar to rule-based rep counting aside from the trouble of manually writing the foundations for various states throughout the rep. This method semi-automates the state calculation method by inferring a reference line (which could be regarded as a state boundary) for a particular motion/train utilizing a coach’s video after which utilizing the reference line for counting reps of any video of that train.
Right here, we contemplate train as a set of waves of metrics of keypoints. These Metrics embrace angles and distances between a mixture of various physique keypoints, and the keypoints are computed utilizing a pose estimation mannequin (Tensorflow’s Movenet pose estimation mannequin).
Main Steps concerned:
We first compute metrics (distances and angles) between a mixture of keypoints utilizing a coach reference video (as enter). These metrics characterize a sign temporally.We filter out all of the stationary indicators and create a mixed sign of the non-stationary ones. Then we compute the reference line utilizing the imply of the summed-up sign.Throughout inference, we begin by once more computing the metrics on the take a look at person enter video and compute an general mixed sign in real-time.We create a fixed-size transferring window and verify for the intersection of the general sign (from 3) with the reference line (from 2). This intersection offers a sign that the rep is full.
This method is quick, straightforward to implement, and pretty correct. Nevertheless, some main downsides embrace the next:
Rep counting is unique and non-generic.Extremely delicate to background noise.Scaling points: One must calculate the zero-crossing line utilizing a reference video for any exercise (additionally making certain the video doesn’t have any noise).
paper: Detecting, recognizing, and monitoring simultaneous workout routines in unconstrained scenes [4]
GymCam is a vision-based system used for automated train rep counting and monitoring. It’s based mostly on the belief that any repetitive movement contained in the fitness center is a few form of train. Once more, right here the enter to the system is a video stream from the digicam, and the output is a number of exercise-related metrics, together with rep rely.
Abstract of the Steps Concerned
Detect all potential movement trajectories in a video utilizing a dense optical stream algorithm. A movement trajectory could be a results of non-exercise actions, too, for instance, warm-up, customers’ gait, roaming right here and there, and so forth.Detect all train movement trajectories in a scene. How do they accomplish that? Firstly, they carry out a characteristic extraction step that includes extracting handcrafted options from a 5-sec window of any trajectory. They use an MLP-based binary classifier mannequin, which takes within the enter characteristic and outputs a likelihood of whether or not that enter trajectory (characteristic) is an exercise-related exercise or not.Clustering train movement trajectories in area and time. After clustering, a median movement trajectory is generated by combining all trajectories belonging to a given cluster. Be aware right here that the variety of clusters is pre-defined. These common trajectories are then used for train rep counting and monitoring.Rep Counting and Train Recognition: Common trajectories are then transformed into characteristic vectors, that are then fed to an MLP Regressor and an MLP Classifier mannequin to deduce rep counts and train labels, respectively.
A number of the noteworthy options of this technique are: It’s an end-to-end system that performs rep counting in a real-world setting. Optical Stream identifies all actions, and therefore it could be adequate to trace the train and carry out rep counting even when the person is barely seen.
Points with this technique:
A number of customers overlap in a video whereas doing the train. And therefore it turns into very troublesome to determine the precise boundaries of those customers and infer the rep counts.Noise Delicate: noisy human habits similar to warming up, relaxation, person’s gait, and so forth., may exhibit periodicity and therefore, can have an undesired contribution to the rep rely.Rep counting is just not generic: the system is proscribed to only the train rep counting.
weblog: Exercise Motion Counting App utilizing Deep Studying and Optical Stream Algorithm [5]
One other fascinating concept using imaginative and prescient to resolve rep counting is the Optical stream method.
Main Steps Concerned
Discover color-coded representations of video frames in a repetitive exercise utilizing a dense optical stream algorithm. Right here, the catch mainly lies in the concept totally different states of a repetitive motion can have totally different coloration codings.
For particulars in regards to the optical stream algorithm, please consult with the opencv doc right here (together with the implementation).
2. Dataset Creation: Subsequent step is to generate a dataset of color-coded photographs and movies and label them with totally different states of the motion (say up or down).
3. Mannequin Coaching: Subsequent step includes coaching a vanilla CNN mannequin to carry out a multiclass classification of the frames. At take a look at time, color-coded frames from optical stream are then fed to the mannequin, which predicts one of many motion states and in addition captures the category label. That is mainly a color-matching downside however by way of a mannequin, because the mannequin is extra sturdy.
The method is correct and simply deployable in manufacturing. Nevertheless, the cons simply outweigh the professionals of the method:
Rep counting is unique and class-dependent.Scaling points: one must annotate the dataset and prepare a mannequin every time a brand new train will get added to the corpus.Orientation Delicate: Identical actions in several orientations can have totally different coloration encodings leading to a incorrect mannequin prediction. This is likely one of the main limitations of the method.Noise Delicate: Any slight noise within the background would change these coloration encodings and therefore the mannequin’s prediction.
