medusa.containers.fourD

Module with core 4D functionality of the medusa package, most importantly the Data4D class, which stores reconstructed data from videos and other (meta)data needed to further process, analyze, and visualize it.

The data can be saved to disk as a HDF5 file (using h5py) with the save method and loaded from disk using the load classmethod.

Module Contents

class medusa.containers.fourD.Data4D(v, mat, tris=None, img_idx=None, face_idx=None, video_metadata=None, cam_mat=None, space='world', device=DEVICE)

Data class which stores reconstruction data and provides methods to preprocess/manipulate them.

Parameters:

• v (np.ndarray, torch.tensor) – Numpy array or torch tensor of shape T (time points) x nV (no. vertices) x 3 (x/y/z)

• tris (ndarray, torch.tensor) – Integer numpy array or torch tensor of shape n_t (no. of triangles) x 3 (vertices per triangle)

• mat (ndarray) – Numpy array of shape T (time points) x 4 x 4 (affine matrix) representing the ‘world’ (or ‘model’) matrix for each time point

• face_idx (ndarray) – Integer numpy array with indices that map vertices to distinct faces

• cam_mat (ndarray) – Numpy array of shape 4x4 (affine matrix) representing the camera matrix

• space (str) – The space the vertices are currently in; can be either ‘local’ or ‘world’

save(path, compression_level=9)

Saves (meta)data to disk as an HDF5 file.

Parameters:

• path (str) – Path to save the data to

• compression_level (int) – Level of compression (higher = more compression, but slower; max = 9)

Examples

Save data to disk:

>>> import os
>>> from medusa.data import get_example_data4d
>>> data = get_example_data4d(load=True, model="mediapipe")
>>> data.save('./my_data.h5')
>>> os.remove('./my_data.h5')  # clean up

apply_vertex_mask(name)

Applies a mask to the vertices (and triangles).

Parameters:

name (str) – Name of masks (one of ‘face’, ‘lips’, ‘neck’, ‘nose’, ‘boundary’, ‘forehead’, ‘scalp’)

static from_video(path, **kwargs)

Utility method to directly initialize a Data4D object by calling the videorecon function.

Parameters:

• path (str, pathlib.Path) – Path to video that will be reconstructed

• **kwargs – Keyword arguments passed to videorecon

Returns:

data – A Data4D object

Return type:

Data4D

classmethod load(path, device=None)

Loads an HDF5 file from disk, parses its contents, and creates the initialization parameters necessary to initialize a *Data object.

Parameters:

path (str, pathlib.Path) – A path towards an HDF5 file data reconstructed by Medusa

Return type:

An initialized Data4D object

to_local()

Converts the data to local space.

to_world()

Converts the data to world space.

project_to_68_landmarks()

Projects to 68 landmark set.

Return type:

v_proj

get_face(index, pad_missing=True)

Get the data from a particular face in the reconstruction.

Parameters:

index (int) – Integer index corresponding to the face

decompose_mats(to_df=True)

Decomponses a time series (of length T) 4x4 affine matrices to a numpy array (or pandas DataFrame) with a time series of T x 12 affine parameters (translation XYZ, rotation XYZ, scale XYZ, shear XYZ).

Parameters:

to_df (bool) – Whether to return the parameters as a pandas DataFrame or not (in which case it’s returned as a numpy array)

Returns:

params – Either a DataFrame or numpy array, depending on the to_df parameter

Return type:

pd.DataFrame, np.ndarray

Examples

Convert the sequences of affine matrices to a 2D numpy array:

>>> from medusa.data import get_example_data4d
>>> data = get_example_data4d(load=True, model="mediapipe")
>>> params = data.decompose_mats(to_df=False)
>>> params.shape
(232, 12)

compose_mats(params)

Converts a sequence of global (affine) motion parameters into a sequence of 4x4 affine matrices and updates the .mat attribute. Essentially does the opposite of the decompose_mats method.

Parameters:

params (np.ndarray) – A 2D numpy array of shape T (time points) x 12

Examples

Convert the sequences of affine matrices to a 2D numpy array and uses the compose_mats function to reverse it.

>>> from medusa.data import get_example_data4d
>>> data = get_example_data4d(load=True, model="mediapipe")
>>> orig_mats = data.mat.copy()
>>> params = data.decompose_mats(to_df=False)
>>> data.compose_mats(params)
>>> np.testing.assert_array_almost_equal(orig_mats, data.mat)  # passes!

filter_faces(present_threshold=0.1)

Filters the reconstructed faces by the proportion of frames they are present in.

Parameters:

present_threshold (float) – Lower bound on proportion present