YANAI Lab

Research

At Yanai Lab, we study deep learning methods for understanding, generating, and editing images and videos. Our research goes beyond recognizing objects: we seek to understand human actions and relationships between objects, connect visual content with language, and create or modify images and videos based on user instructions.

Our recent work focuses on multimodal large language models and generative models, including diffusion models. We explore how recognition and generation can support each other, with applications in food image analysis, text and font generation, and 3D reconstruction. Our research spans fundamental methods and practical systems.

Research Keywords

Image and Video Understanding / Multimodal Learning / Vision and Language / Generative AI / Image and Video Generation and Editing / Food Image Analysis / Text and Font Generation / 3D Reconstruction / Efficient Deep Learning

Main Research Topics

1. Multimodal Image and Video Understanding

We combine visual and language information to develop a deeper understanding of images and videos. Our research includes human–object interaction detection—identifying who is doing what with which object—and temporal action detection, which locates when actions occur in a video. We also investigate methods that use language instructions and reasoning to identify and segment relevant regions in images.

2. Image and Video Generation and Editing

We study image and video generation and editing using diffusion models and other generative approaches. Topics include editing images while preserving a subject’s identity, controlling the motion of objects in generated videos, and colorizing long videos based on text instructions. We also explore how representations learned by generative models can support recognition tasks such as image segmentation.

3. Food Image Analysis and Cross-Modal Food Computing

We develop methods for analyzing food photographs, including food recognition, segmentation, portion estimation, and calorie estimation. Our work combines reasoning with multimodal large language models and geometric information, such as food volume, 3D shape, and depth measurements from smartphone LiDAR sensors. We also study cross-modal retrieval between food images and recipes, recipe-conditioned food image generation, and calorie-aware food image editing.

4. Text and Font Generation and Style Transfer

We investigate methods for generating characters from a small number of examples while capturing the characteristics of a target font or writing style. Our work includes Japanese kuzushiji and Bengali font generation, style transfer for text in natural images, and font stylization guided by language or sound. We use diffusion models and differentiable rendering to model character shapes and create diverse visual styles.

5. 3D Reconstruction and Geometric Understanding

We study how to recover 3D shape from a single image. Our research includes reconstructing the geometry and real-world scale of food, as well as generating point clouds from individual images. By recovering depth and shape from 2D observations, we aim to support visual understanding and applications such as food portion estimation.

6. Efficient Deep Learning and Practical Applications

We investigate methods that make deep learning models more efficient to train and run. Topics include network pruning and parameter-efficient adaptation of pretrained models. We also develop practical applications, including smartphone-based food analysis and interactive systems for text generation and image editing, to make our research accessible through usable tools.