Skip to content

improving writeups #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mohitdamke wants to merge 1 commit into
base: main
Choose a base branch
from
+26 −21
Open

improving writeups #74

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 26 additions & 21 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,26 @@
[![Website][website]][website-url]
[![Discord Shield][discord]][discord-url]

| :exclamation: KalidoKit is now officially deprecated and solutions will be directly integrated into [MediaPipe](https://mediapipe-studio.webapps.google.com/demo/face_landmarker) :exclamation:|
| :exclamation: KalidoKit is now officially deprecated, and its functionalities will be directly integrated into other solutions [MediaPipe](https://mediapipe-studio.webapps.google.com/demo/face_landmarker) :exclamation:|
|--------------------------------------|

## Face, Pose, and Hand Tracking Calculator

Kalidokit is a blendshape and kinematics solver for Mediapipe/Tensorflow.js face, eyes, pose, and hand tracking models, compatible with [Facemesh](https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection), [Blazepose](https://github.com/tensorflow/tfjs-models/tree/master/pose-detection), [Handpose](https://google.github.io/mediapipe/solutions/hands.html), and [Holistic](https://google.github.io/mediapipe/solutions/holistic.html). It takes predicted 3D landmarks and calculates simple euler rotations and blendshape face values.
- Kalidokit is a blendshape and kinematics solver for Mediapipe/Tensorflow.js face, eyes, pose, and hand tracking models, compatible with [Facemesh](https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection), [Blazepose](https://github.com/tensorflow/tfjs-models/tree/master/pose-detection), [Handpose](https://google.github.io/mediapipe/solutions/hands.html), and [Holistic](https://google.github.io/mediapipe/solutions/holistic.html). It takes predicted 3D landmarks and calculates simple euler rotations and blendshape face values.

As the core to Vtuber web apps, [Kalidoface](https://kalidoface.com) and [Kalidoface 3D](https://3d.kalidoface.com), KalidoKit is designed specifically for rigging 3D VRM models and Live2D avatars!

NOTE: This is a JS library meant for developers using Mediapipe pretrained models and not a complete app by itself. Please visit [Kalidoface](https://kalidoface.com) for the full vtuber experience 😉!
- As the core to Vtuber web apps, [Kalidoface](https://kalidoface.com) and [Kalidoface 3D](https://3d.kalidoface.com), KalidoKit is designed specifically for rigging 3D VRM models and Live2D avatars!

<a href="https://glitch.com/edit/#!/kalidokit"><img src="https://github.com/yeemachine/kalidokit/blob/main/docs/kalidokit_glitch.gif?raw=true" alt="Kalidokit Template" width="48%"/></a>
<a href="https://glitch.com/edit/#!/kalidokit-live2d"><img src="https://github.com/yeemachine/kalidokit/blob/main/docs/kalidokit-live2d_glitch.gif?raw=true" alt="Kalidokit Template" width="48%"/></a>


NOTE: This is a JS library meant for developers using Mediapipe pretrained models and not a complete app by itself. Please visit [Kalidoface](https://kalidoface.com) for the full vtuber experience 😉!

[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/B0B75DIY1)

## Install

#### Via NPM
### Via NPM

```
npm install kalidokit
Expand All @@ -39,15 +40,15 @@ import * as Kalidokit from "kalidokit";
import { Face, Pose, Hand } from "kalidokit";
```

#### Via CDN
### Via CDN

```js
<script src="https://cdn.jsdelivr.net/npm/kalidokit@1.1/dist/kalidokit.umd.js"></script>
```

## Methods

Kalidokit is composed of 3 classes for Face, Pose, and Hand calculations. They accept landmark outputs from models like Facemesh, Blazepose, Handpose, and Holistic.
- KalidoKit comprises three classes for Face, Pose, and Hand calculations, which can process landmark outputs from models like Facemesh, Blazepose, Handpose, and Holistic.

```js
// Accepts an array(468 or 478 with iris tracking) of vectors
Expand Down Expand Up @@ -76,7 +77,7 @@ Pose.solve(poseWorld3DArray, poseLandmarkArray);
Hand.solve(handLandmarkArray, "Right");
```

Additional Utils
- Additional Utils

```js
// Stabilizes left/right blink delays + wink by providing blenshapes and head rotation
Expand All @@ -94,23 +95,23 @@ Kalidokit.Vector();

## VRM and Live2D Sample Code

Quick-start your Vtuber app with these simple remixable examples.
- Quick-start your Vtuber app with these simple remixable examples.

### VRM Glitch Template

Face, full-body, and hand tracking in under 350 lines of javascript. This demo uses Mediapipe Holistic for body tracking, Three.js + Three-VRM for rendering models, and KalidoKit for the kinematic calculations. This [demo](https://glitch.com/edit/#!/kalidokit) uses a minimal amount of easing to smooth animations, but feel free to make it your own! VRM model from [Vroid Hub](https://hub.vroid.com/en/characters/6617701103638326208/models/8892157442595230149).
- Face, full-body, and hand tracking in under 350 lines of javascript. This demo uses Mediapipe Holistic for body tracking, Three.js + Three-VRM for rendering models, and KalidoKit for the kinematic calculations. This [demo](https://glitch.com/edit/#!/kalidokit) uses a minimal amount of easing to smooth animations, but feel free to make it your own! VRM model from [Vroid Hub](https://hub.vroid.com/en/characters/6617701103638326208/models/8892157442595230149).

<a href="https://glitch.com/edit/#!/kalidokit"><img alt="Remix on Glitch" src="https://cdn.gomix.com/f3620a78-0ad3-4f81-a271-c8a4faa20f86%2Fremix-button.svg"></a>

### Live2D Glitch Template

This demo uses Mediapipe FaceMesh for face tracking, Pixi.js + pixi-live2d-display for rendering models, and KalidoKit for the kinematic calculations. This [demo](https://glitch.com/edit/#!/kalidokit-live2d) uses an official sample Live2D model, [Hiyori](https://www.live2d.com/en/download/sample-data/).
- This demo uses Mediapipe FaceMesh for face tracking, Pixi.js + pixi-live2d-display for rendering models, and KalidoKit for the kinematic calculations. This [demo](https://glitch.com/edit/#!/kalidokit-live2d) uses an official sample Live2D model, [Hiyori](https://www.live2d.com/en/download/sample-data/).

<a href="https://glitch.com/edit/#!/kalidokit-live2d"><img alt="Remix on Glitch" src="https://cdn.gomix.com/f3620a78-0ad3-4f81-a271-c8a4faa20f86%2Fremix-button.svg"></a>

### Local VRM and Live2D Examples

Interested in contributing? If you clone this project, the `docs` folder includes both VRM and Live2D KalidoKit templates. Run the `test` script to build the library and start up the dev server. The VRM demo will be hosted on `localhost:3000` and Live2D on `localhost:3000/live2d/`.
- Excited about contributing? By cloning this project, you can access the `docs` folder with VRM and Live2D KalidoKit templates. To get started, run the `test` script to build the library and launch the development server. The VRM demo will be available at `localhost:3000`, and the Live2D demo at `localhost:3000/live2d/`. Happy coding!

```
npm install
Expand All @@ -121,7 +122,7 @@ npm run test

<img src="https://github.com/yeemachine/kalidokit/blob/main/docs/kalidokit-pipeline.png?raw=true" alt="Kalidokit Template" width="100%"/>

The implementation may vary depending on what pose and face detection model you choose to use, but the principle is still the same. This example uses Mediapipe Holistic which concisely combines them together.
- The way you implement it might differ based on your chosen pose and face detection model, but the underlying principle remains the same. This example utilizes Mediapipe Holistic, which efficiently combines both functionalities.

```js
import * as Kalidokit from 'kalidokit'
Expand Down Expand Up @@ -162,7 +163,7 @@ camera.start();

## Slight differences with Mediapipe and Tensorflow.js

Due to slight differences in the results from Mediapipe and Tensorflow.js, it is recommended to specify which runtime version you are using as well as the video input/image size as a reference.
- Due to slight differences in the results from Mediapipe and Tensorflow.js, it is recommended to specify which runtime version you are using as well as the video input/image size as a reference.

```js
Kalidokit.Pose.solve(poselm3D,poselm,{
Expand All @@ -186,7 +187,7 @@ Kalidokit.Face.solve(facelm,{

## Outputs

Below are the expected results from KalidoKit solvers.
- Below are the expected results from KalidoKit solvers.

```js
// Kalidokit.Face.solve()
Expand Down Expand Up @@ -259,16 +260,20 @@ Below are the expected results from KalidoKit solvers.
RightLittleDistal: {x: 0, y: 0, z: -0.1}
}
```
[![Kalidoface virtual webcam](https://raw.githubusercontent.com/yeemachine/kalidoface-live2d-models/main/promo/TW-Promo-short.gif)](https://kalidoface.com) [![Kalidoface Pose Demo](https://cdn.glitch.me/29e07830-2317-4b15-a044-135e73c7f840%2Fkalidoface-pose-dance.gif?v=1633453098775)](https://3d.kalidoface.com)

## Community Showcase + Contributing

If you'd like to share a creative use of KalidoKit, please reach out or send a pull request! Feel free to also use our Twitter hashtag, [#kalidokit](https://twitter.com/search?q=%23kalidokit).
## Update
- We're improving the library and welcome contributions! Plans include better facial blendshapes, full leg tracking, and improved efficiency.

Kalidoface/KalidoKit also now has a [Discord server][discord-url]! Come and join us!
## Community Showcase
- We warmly welcome you to showcase your innovative creations using KalidoKit on [Twitter](https://twitter.com/search?q=%23kalidokit)) using the designated hashtag [#kalidokit](https://twitter.com/search?q=%23kalidokit).

The current library is a work in progress and contributions to improve it are very welcome! Some plans include: better facial blendshapes, full leg tracking, and improved efficiency.
## Contributing
- If you'd like to share a creative use of KalidoKit, don't hesitate to reach out or send a pull request. Your contributions and imaginative implementations of KalidoKit are highly valued, and we can't wait to see what you come up with

- We are excited to announce that Kalidoface/KalidoKit now has its very own Discord server! Join us in the community and be part of the conversation by clicking on the [Discord server][discord-url]

[![Kalidoface virtual webcam](https://raw.githubusercontent.com/yeemachine/kalidoface-live2d-models/main/promo/TW-Promo-short.gif)](https://kalidoface.com) [![Kalidoface Pose Demo](https://cdn.glitch.me/29e07830-2317-4b15-a044-135e73c7f840%2Fkalidoface-pose-dance.gif?v=1633453098775)](https://3d.kalidoface.com)

[npm]: https://img.shields.io/npm/v/kalidokit
[npm-url]: https://www.npmjs.com/package/kalidokit
Expand Down