Processing large data set keeps failing

hedaja

Hi everyone

I am trying to process a fairly large data set. I took a drone survey of riparian zones of two streams using a DJI Mavric 3M.
The result is about 13.300 jpg images with about 128 GB (about 4.400 images for the smaller stream).

I fortunately have access to a pretty good computer:
Win 11
Threadripper Pro 3975WX 64 cores
NVIDIA GeForce RTX 3090
128 GB RAM
Data Harddrive is unfortunately a HDD

I first processed each flight separately and that worked pretty well. Unfortunately I get quite a bit of distortion at the boundaries between two flights.
I therefore had hoped to process the whole data set at once.
I used fast-ortho and split:2000 as settings.
Unfortunately the process failed. I also tried to at least get each stream on its own but it also fails.
I checked the logs and it always seem to break after it creating the cutline and trying to compute the mask raster.
See attached log file.

Is the rather complex geometry of the long winding streams causing this issue (working on getting a plot of all camera positions right now...will add when ready)? Is there anything I can do/try?

Thanks in advance

console_output_one_stream.txt — view on Discord

12 Replies

pierotofy

Yeah corridors are memory heavy; try lowering split further?

hedaja OP

Thanks I will give it a try
From the log it seems like the gdal cache is set to a maximum of 60GB
Is there a way to adjust that from within WebODM? I got the version without manually installing docker

hedaja OP

I tried two more times. First reducing split to 1500 and then 1000 images. Still fails with the same error message

hedaja OP

any other ideas?

Aeret

Hi, it looks like a nice set. Are the images RGB or MS? Are the images taken on the same date / height ?

What is your required resolution? Fast orthophoto is a nice setting to get a quick and fast overview, but imho it's not the best setting to do calculations / analysis on the data.

For 13000 images you need a big(ger) computer I guess https://docs.webodm.org/installation/#hardware-requirements especially if you want to process the images in defaulf or high res.

hedaja OP

hi @Aeret

hedaja OP

Sorry for poor editing. discord on a mobile browser is a piece of crap. images are jpg RGB. They were taken across multiple days. Most of them at the same height but the first flights was slightly higher (I tried a set without it as well though). I need full resolution since I need to identify individual flowers . since its mostly a flat area I had hoped to get away with fast-ortho. I still wonder if the issue is related to the limited GDAL cache but I don't know how I could adjust that. Unfortunately it also doesn't seem to save the intermediate steps so I could just try to run that gdal command separately

Aeret Best answer

Thanks for the additional information. Fast ortho doesn't give you the required resolution. I run WebODM on a VPS. For (really) large mappings I usually add a (temporary) NodeODX environment, which processes the images, using cloud recources, such as Google Cloud / Compute Engine. Based on the amount of images I set up a VM with the required resources.

I guess on of the bigger mappings I did were 8539 images from a DJI M3E with a required GSD of 2.0

Maybe you can try a subset via Lightning? https://webodm.net/pricing
Or if you can post a (small)data set, no problem to process these for you. Can you tell us a bit about the size / bounding box of your area? What are the dimensions of the mappings (square mile / km?)

💛 1
hedaja OP

Thank you for the offer.
I have been able to process subsets (each flight by itself with up to around 800 images) without many issues. But with the amount of distortion in the overlapping areas (edge of the flight path), they are not well suited for trying to merge them later on.
GSD should be around 2.5 cm according to DJI flight planner

I guess I misunderstood the fast-ortho option. I thought all it did was skip dense point cloud reconstruction and DEM . Which would as far as I understand result in somehwhat less precision when de-distorting the images. I just saw, that selecting the Fast Orthphoto preset defaults the GSD to 5cm. Do I understand it correctly if I set it to 1 but my recorded resolution was 2.5, the resolution will be 2.5 cm/px at the end?

Thanks for your help

Aeret

Yes, it will give you the flown GSD of 2.5. I don't have any experience with the following plugin, but maybe you can try. First you process each subset and then merge them with : https://github.com/uav4geo/QRasterMerge

hedaja OP

With split set to set to 1000 images it finally worked. But it also took 35 hours.
I also had to realize that the whole orthophoto gets too big for further processing. There are also areas that didn't get reconstructed well (densly forested without enough matches)
I am now testing different ideas for the individual flights.

Thank you guys for your help

Need more help?

This discussion happened in the WebODM community Discord. Join and ask your own question.

Join View Original Thread

This page mirrors a public discussion from the WebODM Discord. How to request a removal.