Making Face Filters - PCD 2020

# Making Face Filters (inside a browser!)
### Shailendra Paliwal ([@ziscore](http://shailendra.me) 👨🏼‍💻, 🏃🏼‍♂️, 🚵🏼‍♂️)

---

## `shailendra.me/face-filters-demos`

![](assets/qr_link.png)

---

# Workshop Agenda

By the end of this workshop, you will know how to track faces and draw objects on them. That's a face filter, right?
But, before we arrive at the end, we need to know a few things...

1. What happens under the hood with face filters?

---

# Workshop Agenda

By the end of this workshop, you will know how to track faces and draw objects on them. That's a face filter, right?
But, before we arrive at the end, we need to know a few things...

1. What happens under the hood with face filters?
1. Face Detection vs Face Tracking vs Face Recognition
1. Using an External Library

---

# Workshop Agenda

By the end of this workshop, you will know how to track faces and draw objects on them. That's a face filter, right?
But, before we arrive at the end, we need to know a few things...

1. What happens under the hood with face filters?
1. Face Detection vs Face Tracking vs Face Recognition
1. Using an External Library
1. [Hands On] Tracking a Face
1. How do these libraries work?
1. [Hands On] Recreating some classic Face Filters

---

# Face Detection

# Face Tracking

# Face Recognition

---

# Using the webcam

```js
function setup() {
  // add a webcam capture
  capture = createCapture(VIDEO);
  
  // show this capture
  capture.show();
  
  // create a canvas for this capture
  createCanvas(400, 400);
}
```

---

# Is this enough though?

Wouldn't it be cool if we could add trails like a Vector to this as well?

---

# Is this enough though?

Wouldn't it be cool if we could add trails like a Vector to this as well?

```js
function setup() {
  createCanvas(400, 400);
  capture = createCapture(VIDEO);
  capture.hide();
}

function draw() {
  // display at half opacity
  tint(255, 20);
  image(capture, 0, 0, width, width * capture.height / capture.width);
}
```

---

# Using External Libraries

Programmers create black boxes that other folks can use without going into the details of how does the black box works.

![](assets/giants.jpg)

---

# Tracking a face

## Enters *CLMTracker*

> clmtrackr is a javascript library for fitting facial models to faces in images and video, and can be used for getting precise positions of facial features in an image, or precisely tracking faces in video.

Seems good enough?

---

# Tracking a face

```js
function setup() {
  // setup camera capture
  var videoInput = createCapture(VIDEO);
  videoInput.size(size, size);
  videoInput.position(0, 0);
  
  // setup canvas
  var cnv = createCanvas(size, size);
  cnv.position(0, 0);

// setup tracker
  ctracker = new clm.tracker();
  ctracker.init(pModel);
  ctracker.start(videoInput.elt);

// define colors
  noStroke();
  fill('red')
}
```

---

# Tracking a face

```js
function draw() {
  // clear the canvas
  clear();

// get array of face marker positions [x, y] format
  var positions = ctracker.getCurrentPosition();
  
  for (var i=0; i < positions.length; i++) {
    // set the color of the ellipse based on position on screen
    circle(positions[i][0], positions[i][1], 5);
  }
}
```

---

# How does it work?

```js
  // get array of face marker positions [x, y] format
  var positions = ctracker.getCurrentPosition();
```

![](assets/clmtrack.png)

---

# Recreating Classic Face Filters

### Ambika's Section 144
#### Yesterday's showcase

[link to tweet](https://twitter.com/ziscore/status/1224543093958963200)

How do we do it?

---

# Recreating Classic Face Filters

### Ambika's Section 144

Add a global variable
```js
var points_of_interest = [62];
```

Change what we do with array `positions`
```js
  for (var i=0; i < positions.length; i++) {
    // check if are we interest in this point
    if(points_of_interest.includes(i)) {
      // write a large emoji at this location
      textSize(250);
      textAlign(CENTER);
      text('🤩', positions[i][0], positions[i][1]);
    }
  }
```

---

# Recreating Classic Face Filters

### Dog Filter?????

![](assets/dog-filter.jpg)

---

# Dog Filter?????
### How do we make it?

Elements of the filter,
1. Puts dog ears on top of your face
2. Puts a dog nose
3. If you open your mouth, there's also a dog tongue!

---

# Dog Filter?????
### How do we make it?

```
> Implementation of this filter has been left as an exercise for the reader
```

---

# Dog Filter?????
## Resources

Adding an image,

```js
let dog_nose;
function preload() {
  dog_nose = loadImage('path/to/an/image/from/the/internet.jpg');
}
function setup() {
  // image(image_name, position_x, position_y)
  image(dog_nose, 0, 0);
}
```

---

# Spacer

[link](https://editor.p5js.org/zscore/present/Dru3FOH3)

---

# Eye Trails

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">My goal is to make ig regret giving me access to filters <a href="https://t.co/VROJwTSp7J">pic.twitter.com/VROJwTSp7J</a></p>— zach lieberman (@zachlieberman) <a href="https://twitter.com/zachlieberman/status/1129416333425336320?ref_src=twsrc%5Etfw">May 17, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

How do we do this?

---

# Eye Trails

How do we crop the canvas?

```js
var croppedCanvas = get(x, y, width, height);
```

---

# Eye Trails

Something like this?

```js
var left_eye = [27], right_eye = [32];
var lastFrameLeft, lastFrameRight;
```

/// 
    if(right_eye.includes(i)) {
        lastFrameRight = get(positions[i][0] - 10, positions[i][1] - 10, 20, 20);
        image(lastFrameRight, positions[i][0], positions[i][1]);
    }
  }
```

---

# Eye Trails

More trails!

```js
  for (var i=0; i < positions.length; i++) {
    // check if are we interest in this point
    if(left_eye.includes(i)) {
        lastFrameLeft = get(positions[i][0] - 10, positions[i][1] - 10, 20, 20);
        image(lastFrameLeft, positions[i][0], positions[i][1]);
        image(lastFrameLeft, positions[i][0] + 5, positions[i][1] + 5);
        image(lastFrameLeft, positions[i][0] + 10, positions[i][1] + 10);
        image(lastFrameLeft, positions[i][0] + 15, positions[i][1] + 15);
    }

/// 
    if(right_eye.includes(i)) {
        lastFrameRight = get(positions[i][0] - 10, positions[i][1] - 10, 20, 20);
        image(lastFrameRight, positions[i][0], positions[i][1]);
        image(lastFrameRight, positions[i][0] + 5, positions[i][1] + 5);
        image(lastFrameRight, positions[i][0] + 10, positions[i][1] + 10);
        image(lastFrameRight, positions[i][0] + 15, positions[i][1] + 15);
    }
  }
```

---

classes: middle

# Thanks!

---

# Resources

1. CLMTracker - [link to repo](https://github.com/auduno/clmtrackr)
2. Computer Vision Examples by Kyle McDonald- [link](https://kylemcdonald.github.io/cv-examples/)