AI Photo Editing with Inpainting

Overview

This notebook demonstrates how to perform AI photo editing using inpainting with the Stable Diffusion model. The notebook is divided into three main parts:

1. undefinedSegmentation Mask Creation with SAM (Segment Anything Model)undefined
2. undefinedInpainting Setup and Executionundefined
3. undefinedInteractive Web Appundefined

Segmentation Mask Creation with SAM

1. undefinedLoading the Pre-trained SAM Model:undefined

   • The notebook starts by loading a pre-trained SAM model from Facebook/Meta. The model is moved to the GPU and configured with torch_dtype=torch.float16.

2. undefinedGenerating the Mask:undefined

   • A function uses SAM to produce a segmentation mask. The function takes an image and points indicating the subject in the image to generate the mask.

3. undefinedTesting the Mask Generation:undefined

   • The mask generation is tested on a sample image of a car. The image is resized to 512x512 pixels, points are defined to indicate the car, and the mask is generated and visualized.

Inpainting Setup and Execution

1. undefinedLoading the Inpainting Pipeline:undefined

   • The inpainting pipeline is set up using the diffusers/stable-diffusion-xl-1.0-inpainting-0.1 pre-trained model and AutoPipelineForInpainting.

2. undefinedInpainting Function:undefined

   • A function performs inpainting. The function takes the raw image, the mask generated by SAM, a text prompt for the infill, optional negative prompt, optional seed for repeatability, and the Classifier-Free Guidance Scale (CFGS).

3. undefinedTesting the Inpainting:undefined

   • The inpainting function is tested on the mask obtained earlier with SAM to produce the final edited image.

Interactive Web App

1. undefinedCreating the Interactive App:undefined

   • An interactive app allows users to upload an image, run SAM, and generate a new background through a text prompt.

2. undefinedRunning the App:undefined

   • Instructions are provided to run the app and access the public URL for using the app.

Example Results

• undefinedGradio Page Screenshot:undefined

  

• undefinedSource Image:undefined

  

• undefinedPrompt:undefined

  a cat in the mountains
  

• undefinedSAM Segmented Mask:undefined

  

• undefinedOutput Image:undefined

  

Note: The subject can be changed depending on the use case.

Environment Setup

1. undefinedCreating a Conda Environment:undefined
   • Create a Conda environment with Python 3.10 using the following commands:

     conda create --name inpaint python=3.10
     conda activate inpaint
     pip install -r requirements.txt
     

Summary

For detailed code and step-by-step execution, refer to the cells in the notebook. Each section is well-commented to provide additional context.