Wired
This article has a total of 1800 words and the estimated reading time is 10 minutes.
Copy the URL below to Safari to download the model:
https://www.icloud.com.cn/iclouddrive/009f24Yad4Ijw9JJ69pmecn4Q#realisticVision_V51_8bit
1. Write in front
With mobile phone manufacturers such as Meizu and Oppo announcing “All in AI”, the concept of “AI mobile phones” has become popular. In the author’s opinion, “AI mobile phones” are nothing new: Take the iPhone as an example. Its “AIization” at the hardware level can be traced back to the A11 chip equipped with the “Neural Engine”, which provides the basis for running AI algorithms locally.
In the past, AI functions, such as text prediction and photo cutout, were in a tepid state, but the emergence of large models and AIGC has completely ignited the public’s attention to AI.
Embarrassingly, the weak performance of mobile phones is not suitable for large-scale model horoscopes, but AI services such as chatGPT and Wen Xinyiyan can be called free and quickly on any Internet-connected device. This makes “AI mobile phone” a false proposition. What everyone actually uses is “AI browser”.
However, there is still a gap in the “AI-generated images” segment. Adobe’s FireFly and Meitu Xiuxiu’s AI photo retouching are both paid services, and functions such as removing debris from photos and changing photo proportions are exactly what mobile phone users use every day. As a result, mobile phone manufacturers have successfully taken a share of the AI wave by relying on the “free AI photo editing” card.
The image retouching function based on AIGC can be divided into 5 major directions:
– Elimination: Generate parts obscured by debris
– Expand image: Generate the part outside the border of the photo
– Change picture: superimpose new content on the photo
– Super score: complete the details of the photo
– Creation: directly generate a photo according to user needs
Currently, Xiaomi, Oppo, and Vivo have all launched the functions of eliminating and expanding images, and what may surprise you is that the iPhone supports all five functions, even locally. The above sample images were all generated with one click on the iPhone 15 Pro Max in my hand.
Considering the length, this article only introduces the operation method of “AI elimination” on iPhone. I will make all the AI photo editing functions into a series for interested friends to read on demand.
2. Introduction to Apple CoreML and Draw Things software
In a nutshell, the iPhone’s image generation function is driven by Stable Diffusion (SD). This large model has nothing to do with Apple. It is the result of the joint efforts of the University of Munich, Runway Company, and many “alchemists.”
Apple’s contribution is that by optimizing the native AI framework CoreML, almost all SD models (including the latest Stable Cascade) can run fully, quickly and energy-savingly on A14 and later Apple chips.
Make a simple comparison: (SD1.5 baseline model, 20 steps, 512 x 512 size)
– Using Termux to run the model on a mobile phone, it takes more than ten minutes to generate a picture;
– Qualcomm claims that 8Gen2 can generate a picture in just 15 seconds under the Qualcomm AI framework, but there is no actual product yet;
– In the actual test, the A14 took 15 seconds to generate images, while the chip at the same time was the 865;
– Today, the A17 Pro takes only 7 seconds to generate a picture, which is fully satisfactory for daily use.
Note that the actual SD model we used is far more complex than the baseline model, so doubling the speed from A14 to A17 Pro is critical. But old iPhone users don’t need to worry. We only need to lower the image quality, and the time consumption will not be much different from that of the A17 Pro.
Compared with cloud-based retouching models, local, open-source SD has many benefits:
– No content restrictions, no “cannot modify” error;
– No usage restrictions, old models can also be installed, and the functions are not emasculated;
– Highly customizable, including comics, oil paintings and other styles.
Today, there are many Wenshengtu Apps that call CoreML and SD on the App Store. If you want to choose the best one, I strongly recommend the Draw Things client.
I don’t want to go into details about the advantages of exquisite UI and completely free. The great thing about Draw Things is that its “infinite canvas” integrates the 5 major AI photo editing functions mentioned above. Users only need to import photos once. Complete all operations.
Friends who are new to SD may feel overwhelmed by the dazzling array of parameters and buttons in Draw Things. Don’t worry, the working principle of SD is actually very simple:
1. First, you need to download an SD model (Checkpoint) and import it into Draw Things. This tutorial focuses on the post-retouching of mobile phone photography, so we recommend Realistic Vision, a model that focuses on realistic style. The download link is at the beginning of this article;
2. With models, Draw Things has the ability to “generate drawings”. But in order to “retouch”, we also need to use the control plug-in (ControlNet). These plug-ins can be selected and downloaded directly from the menu in the App. The plug-in used in this article is Inpainting (SD v1.x, ControlNet 1.1);
3. Everything is ready, all you need is a spell. For “elimination”, we don’t need prompt words, and the model can “guess” the content covered by debris, which is quite convenient. There are not many parameters we need to adjust:
– Basics – Image Size (Size) = 512 x 512
– Basics – Step = 8
– Basics – Text Guide (CFG Scale) = 7
– Advanced – Sampler = DPM++ 2M Krass
– Keep the rest of the parameters as default.
3. Complete AI elimination in 3 steps
Draw Things has two ways to import photos. At the bottom of the App, click the “Camera” icon and you can select photos in the album. I personally prefer dragging photos from the album to Draw Things, which saves the process of flipping through the album.
Step 1: Zoom
Use two fingers to pinch to move the “clutter” that needs to be removed to the range of the “canvas”. Please note that do not let the canvas contain only clutter. You must ensure that there is a complete “scene” around the clutter so that the AI can reason about the content behind the clutter;
Step 2: Frame selection
Click the “eraser” icon at the bottom of the App to enter “free drawing” mode. When erasing debris, don’t forget to include the shadows, reflections, etc. of the debris;
Step 3: Generate
Click the “Generate” button and wait about 20 seconds for the AI elimination to be completed. It’s that simple. Click the “Download” icon in the upper right corner of the App and you can save the repaired photos to the album.
At this point, if you want to continue editing the next picture, just go back to the photo album. If you’re not satisfied with the image generated by the AI, you can have it generate it again until you’re satisfied. Click the “clock” icon at the bottom of the App to enter the history, select the photo that has been framed before generation, and you can generate it again.
Next, I will briefly compare the effects of local photo editing on iPhone and cloud photo editing on Android phones. The materials are derived from digital reviews on Station B and Weibo. Please note that AI retouched images are produced one at a time, and the retouched quality in the example does not represent the highest quality:
4. Advanced skills
AI is fallible. When we can’t get the image we want, it’s often because the parameter input is incorrect, rather than because the model has insufficient capabilities. For example, for the car pictured below, multiple brands of cloud-based photo editing failed to eliminate the car. Instead, they replaced it with another one:
At this time, open source SD has an advantage. You just need to fine-tune the parameters in Draw Things and let the AI correct its mistakes. For example, the above-mentioned “failed to eliminate, but added new” problem also occurs from time to time in SD:
– As shown in the picture, I want to eliminate the dolls on the left side of the sofa;
– But I generate multiple times and can only get a new doll;
– At this point, if you zoom out a little, the AI will know that you don’t want new dolls, but want to eliminate them.
But what should we do if the clutter in the photo occupies a large area and we have no room to shrink the photo? I have summarized the following coping methods:
– Divide the debris into multiple parts and generate them multiple times to eliminate;
– Adjust the aspect ratio of the canvas to allow more of the original photos to fit into the frame;
– Use prompt words lightly to tell the AI what content you want to generate.
5. Summary
How mobile phone manufacturers connect with AIGC is a difficult question to answer. Frankly speaking, AI development and mobile phone development are not on the same track at all. “All in AI” for mobile phone manufacturers is equivalent to business transformation. If you want to focus on both “mobile phone hardware and AI software”, the end result will probably be that your R&D capabilities cannot keep up, and your products will never be on par with industry leaders.
In the situation where OpenAI is unparalleled, let alone mobile phone manufacturers, even established AI powerhouses like Google dare not pat their chests and say: “The AIGC I provide is better.” Therefore, it is a more realistic development route to target OpenAI’s closed-source and conservative disadvantages and work on the “customization” of large models.
In the subdivision of “image generation”, Stable Diffusion 3, which is still in the development stage, can compete with OpenAI’s DALL-E3. However, DALL-E3 sometimes has “prompt word violation” and sometimes “system is busy”. In actual experience, SD is more in line with user needs. And this is also an opportunity for mobile phone manufacturers.
Whether it is connecting to open source large models or developing a “customized” large model solution, as long as users can get an experience from the “mobile phone model” that is more free than the cloud and less troublesome than the computer, then “mobile phone + AIGC” There is a place to stand.
