Grzegorz Baran here, this time I am going to present you surface scanning with the drone. To rise the bar with my photogrammetry experience I decided to involve the drone for my surface scanning workflow I was wandering is it good enough to fully replace the DSLR camera with the tripod in long term or is it rather complementary piece of equipment. When I did this scan I had no idea what results I am going to get and I also took into account that it might be a total disaster and waste of money. I have heard already that the drone isnt able to deliver image quality which is enough for detailed surface capture. So this video presents the first scan I made using a drone trying to answer all those questions I had before The plan I had was to use the drone for surface capture and turn captured data using photogrammetry reconstruction into fully functional PBR material. and this is where the story starts In details, I am going to present the capture part where I used the Mavic 2 Pro drone to capture a cliff wall Next I am going to show preproduction part, where I tweaked captured RAW images and flattened the color removing ambient occlusion shadows during this stage. Next I used the Metashape for photogrammetry reconstruction to get a highpoly model of captured cliff wall Next I built and uv-mapped the lowpoly plane for baking in ZBrush the PBR texture baking part was done I Substance Designer baker Next I used the Artificial Intelligence in Artomatix to remove seams, fix tiling and tweak overal material logic as well as generate missing PBR maps. And finally I applied the final result to the mesh I usually use in Marmoset Toolbag 3 to preview the final PBR material. Hope you will enjoy the video and let’s start it It took me over a year to make a final decision and spend money to buy a drone. Thanks to one of my friends I had an opportunity to test images taken with Mavic Air before. Unfortunately the camera wasn’t good enough to deliver images in expected quality. After a few more months of research I decided finally to get Mavic 2 Pro. but before that, since the drone is a freaking expensive piece of equipment I decided to get some flying practice and learn how to fly the drone first For this purpose I bought a used Tello drone on ebay. The drone had 2 batteries. Each lasts for just maximum 12 minutes of flying and since it wasn’t enough I decided to get a third one. This drone also had a controller with steering similar to the bigger drones. Even if Tello Drone is just a toy, it is still pretty advanced one, and with the full .. even if a bit simplified.. drone functionality. It is very fast and has about 100m range and go up 30 meters. I learnt about battery life, wind, birds and trees and the worst drone enemy in my opinion so far.. tree branches I crashed it a few times experimenting but since this is a toy.. it handled those crashes quite well. During this time I lost a few propellers A few times I thought the drone is totally dead and my training is over.. but after recalibration it was still fully alive. Finally after a few weeks without any crash I decided that I am ready to get a real, a bigger drone. I decided to get Mavic 2 Pro.
Since it has a 20Mpx camera and 1 inch sensor it gave me the hope that this drone actually can do the job. Unfortunately the basic package contains just the drone, one battery, controller and a few cables. Battery lasts for maximum of 30 minutes of fly, what in practice probably means about 25. So to make the purchase complete I also decided to get a ‘Fly more Kit’. With this one I got two more drone batteries which means more flying time during the same trip. With this kit I also got a bag which secures the drone during the transport. The drone itself is a bit fragile and I can see that its first crash will be probably the last one so I decided also to get a refresh insurance which replaces the drone if I destroy it. Since the drones camera is mounted very low to the ground, I also decided to buy a landing pad and energy bank to be able recharge batteries, mobile and a controller when I am outside Initially I bought a 75 cm wide landing pad. Unfortunately during this scan I realised that it is way too small for the drone I have. Even if feels like an airport for the Tello, the coverage is not enough for the Mavic 2 Pro and doesn’t allow to any landing mistakes. During the landing, the drone pushed by wind landed on the edge of the landing pad and the propellers blade hit the stake and stopped on it
Hopefully blade was ok as had quite low speed at the last phase of landing but it was really close to have it damaged. It pushed me to order a bigger one this time, a 110 cm wide. So far it does the job way better and gives enough space for landing even during the strong wind. Unfortunately even when folded, cant fit into any of my backpacks and has to be attached outside which makes traveling a bit tougher. I can imagine it is going to be even less convenient while carrying both, the light reflector and the landing pad together. I decided to buy a quite expensive energy bank from China since it was recommended on drone forums. The brand name is Yx. and it took almost a month to have it delivered to the UK. The energy bank itself is quite heavy but has capacity of 25.000 mAh To compare the Mavic 2 Pro battery has a capacity of 3850 mAh. the energy bank I’ve got in theory should fully recharge the Mavic 2 battery, or controller or the mobile 6 times before it is dead. In theory of course. Unfortunately after I unpacked the package I have found that I made a mistake as I didn’t read the description very well and cables delivered don’t match with the Mavic2 batteries as are compatible with the Mavic Air only. I have contacted the seller and asked for help with cables but so far it looks like I have a fancy energy bank to recharge just the controler Which is still fine but definitely nothing as planned. At least not as long I won’t find a proper cables. The first week after I bought a drone it was raining in the UK. So it gave me an opportunity to spent a few days going through manuals and youtube videos before I even took off. when the rain was over I took a few really short flights and decided to pack the entire equipment and go to the place I call a photogrammetry heaven to get the final answer: Is it possible to scan surfaces to create high quality materials using the drone I bought I call this place as a photogrammetry heaven since no matter what weather or lighting conditions are, there is always a nice surface to scan somewhere. The first thing I noticed after I watched all those drone videos and never cared about before were birds. I saw them many times before and I even quite like them but not this time. These birds are really big and sometimes very aggressive. The can dive very fast when above the drone and often appears from nowhere. I have noticed when they see the drone flying they start shouting like crazy and after a while there is 15 of them circling around. I think I can understand them but its something new for me as I never thought about birds, planning my photogrammetry trips before. The air space belongs to them for sure and I am a stranger violating it. But whatever reason they have to be hostile, I can imagine them taking my drone out during a flight and I need to make sure it is not gonna happen. I have found a nice cliff wall in a next spot .. but after since this time I took much more attention to birds I noticed pigeons having nests in a holes of the cliff wall. I have read that birds might be even more aggressive while are defending their nests which makes sense. So I decided that it would be safer to pick a different location for a first scan The place where I can purely focus on scanning .. not birds. Since I was in a ‘photogrammetry heaven’ getting to a new, better spot wasnt hard at all To the other side of the walking path I found a nice rocky wall to capture. The one I always struggled to scan before due to limited access and quite high elevation. I picked a really nice looking and quite flat part of the wall hidden in a shadow and set up the landing pad next to it. The wall was high enough and the sun was angled the way I was sure that the drone’s camera wont face the sun during capture. This time I used 75cm wide landing pad. To make sure the wind wont affect its position I used a stake. The one mentioned before. and the one which helped me to find out that the landing pad is too small in long term and it would be better to get a bigger one. After I secured the landing pad I unpacked and prepared the drone for a flight. I removed gimbal cover, plugged my mobile into the controller and finally turned everything on. I checked the app if everything was ok, and I had enough of satellite coverage to have GPS mode active. Finally the drone was ready for scanning. This part took me a while but I believe I just need a bit more practice. I took off to take a test approach to the wall, tried a tripod mode, checked sensors in action and how stable the drone is next to the wall. I took a few test images and came back to land. And it was the moment when I almost broke the propeller, you can hear the sound how the propellers hits the stake and stops on it I decided that the other one to the other side is enough and I removed this one After this small accident I turned engines on again, took off the drone and made a small circle around to make sure that everything works fine. By the way, those angled stripes you can see on the drone screen are there to inform about overexposured areas and can be turned off in a drone setting Since the drone behaved ok I decided to continue with the capture Scanning with a drone is quite easy.
This one has sensors in each direction what helps in navigation as they tell me exactly how away I am to the wall Sensors would stop the drong from hitting a wall in direct forward flight and wont let me to fly into a wall If I push the stick forward but while the drone is pushed by a wind in hover mode they wont stop it Since the wind there was quite strong and definitely was pushing the drone towards the wall I had to keep the distance under control
and every time the drone was too close to the wall I pulled it back a bit and continued capture. As I was scanning the surface without any direct access I wasn’t able to add any scale reference elements or markers Those would help me to get sense of scale during reconstruction but also help to navigate during the capture Even without them I was trying to keep my standard capture pattern by simply moving in lines. and since the capture part took me 8 minutes lets speed this video up a bit due to lack of my experience in drone scanning and without any clear navigation the last part of the capture was a bit chaotic but even thou I managed to get 92 very high quality images and all were in quite hight quality for photogrammetry based surface reconstruction Since the drone also detects me as an obstacle I had to step back over 2 meters during the landing. As you can see this landing also wasn’t the best. The drone has landed on the edge of the landing pad and if there would be a stake or high grass, the propelers blades would hit them for sure again. I also realized that I didn’t take any calibration images yet so took off again to get them It wasnt an issue at all since after this scan I was planning to scan the other part of the wall to the left. So I took off again and this time I captured the pages with color clips visible for future calibration in photo editing software and I took a shot of the grey card from X-Rite Checker for optional white balance setting A DJI Mavic 2 Pro camera is definitely a really good camera. It has one inch sensor with the 20 Mpx matrix to store the images In result can capture images in 5472 per 3648 pixels resolution It has quite fast electronic shutter. It stores images in JPEG but also in DNG which is a RAW file format According to DJI support it has 30bits of color depth for still images. But it important to know that this is not a DSLR camera. A DSLR camera wins with the image quality when compared in every aspect, but only when the image was taken from the same spot. And this is the field where the drone wins as it gives a total freedom for camera positioning. With the drone I can shot from angles which were impossible when I was equipped with a tripod or a monopod It also gives access to spots I could only dream about before. There is no chance I could take as consistent shots from such close distance and such consistent, perpendicular to the surface angle without using a drone or other really sophisticated equipment. Unfortunately the Mavic 2 Pro camera has terrible perspective distortion. Here is an example how big the distortion is before and after the fix. So definitelly this is something that can be fixed by Photoediting software easily but that kind of fixes always cost the quality as are based on stretching and interpolation. As long I know that, I can take closer and better angled shots to compensate that quality loss. Anyway, since I have to fix that perspective distortion for photogrammetry reconstruction, the image postproduction for images captured with this drone is a must. Hopefully this is something what is autofixed by Photolab2 automatically since Mavic 2 Pro camera is fully supported by this app. Just to explain why do I work on RAW files 8bits – has only 256 unique colors per each RGB channel It means 256 shades of RED color, 256 shades of GREEN and 256 shades of BLUE and this is fine since this is the amount of information every photogrammetry sofware I know use for 3D reconstruction. So even if we use more denser RGB data stored in more sophisticated format, for reconstruction it is going to be baked into 8 bits anyway. So what is the purpose of shooting in format which stores more color information to 8 bits? Lets get a bit of theory about differences when we shot RAW and when we capture JPG. Whatever file format we use it can be represented by a histogram LCD monitors we usually use are a information bottleneck for any color information. Any color or brightness judgment based on what we see on any screen is always more or less inaccurate and depends on many factors. Screens we use are very limited and even those really the best show us maximum depth of color information just around 8 bits. Most of the LCD screens is not able to present any difference in bottom spectrum, that one around blacks. You can check it easily by trying to notice the difference between RGB 0,0,0 or 12,12,12 For many screens the first visible color change is noticeable above RGB 16,16,16. And this is where HISTOGRAM comes in handy. It is a chart which is pure, mathematical visualisation of pixel distribution across the full available tonal range. The bottom, horizontal line represents the tonal range from pure black to pure white. The vertical line represents the amount of pixels for certain value. Left side represents dark values, usually shadows. The middle section represents the midtones Highlights or speculars are represented to the right side. Everything outside of this graph should be considered as information lost forever. We lose it usually by shooting overexposured or underexposured images. The accuraccy we get between the 0 to 1 depends on color depth measured in bits. If the color depth is stored in 8 bits, it means we store just 256 tonal values for each RGB color between 0 and 1 it means 256 color shades to represent luminance level for RED, 256 for GREEN and 256 for BLUE color If we multiply those values in result we get over 16 milion colors. So to sumarise Low Dynamic Range file stores 8 bits of RGB color information which gives us 16 milion colors To compare, 12 bits on the other hand, stores 4096 colors per each RGB channel It gives over 68 bilion colors. The DSLR camera I use stores images in 14bits depth RAW files. 14 bits contains 16,385 colors per each RGB channel which gives over 4 trillion colors. It is also worth to mention that some RAW files stores additional dynamic range, sometimes even up to 2 additional EV. this information might help us as a backup data in case of slight over or underpexposure. This is why RAW is called a losless format as we have really a lot of data to play with and to pick from.
While doing any color modifications on LDR 8bit is not recommended as this is color data stripped to minimum already. So the plan I had was to tweak, modify and re-shift color distribution while still working on high density data. Since I have thousands times more information to what I actually need, any changes I do don’t affect any data quality et the end. So I reset the white balance, tweak colors, reduce shadows and highlights. I bring missing information back if needed. And finally I shift the color distribution towards the proper middle value for albedo if needed. It is important to know that this value is not in the middle of the graph at 0.5 but slightly above 0.7 The reason for that is that while setting the Albedo we work in gamma space, not linear space. So even if mathematicaly 128 is in a half way from 0 to 255, this is the correct value just for linear space. Linear space makes sense when we work on maps like height map, ambient occlusion or normal map. Since the albedo is not pure mathematical value it depends on Gamma Space.
So it is important to know that in gamma space in RGB counted from 0 to 255, RGB 187 is considered as neutral color, not 128. Since as I mentioned at the beginning, any pohotogrammetry reconstruction software use just 8 bits of color data for reconstruction, after the tweaking is over I bake the results into 8 bits and save it as noncompressed JPG files. From this moment, any further color related tweaks made on 8bits data are going to cost us the quality. Hope that makes sense. So far I am not really sure what is the Color Depth for the Mavic 2 Pro. I estimate it might be something around 12 to 14bits due to the file size each RAW file has. But whatever those parameters are, I can tell the camera delivers quite solid data to work with so far. Let’s go back to image postprocessing stage. Due to perspective distortions all images have to be processed by Photoediting Software. PhotoLab2 fixes it by default when the RAW file is opened. In next steps I select all images I want to use for reconstruction and tweak them the way I can make them even more useful for photogrammetry and material creation. I set the White Balance by selecting neutral grey clip from the Color Checker I shift the colors the way described before, remove shadows playing with blacks and shadow options, I am also calming down highlights using Highlight slider I remove Vignetting and chromatic abberation level the tonal range distribution for all images with Smart Lighting and I implement noise reduction correction the PhotoLab After all those changes are set I bake them into 8 bit non-compressed JPG files for photogrammetry reconstruction. The baking part takes some time since some of those settings need some time to be calculated. When done, I can close the Photolab and proceed with Photogrammetry Reconstruction software.