EPT builder seems really slow - can Potree converter be used instead?

cwood256

When processing large datasets with 200M+ point clouds, the EPT build stage can take hours or days.

The ODX option to build EPT is hidden in the WebODM UI, apparently because EPT output is used by the embedded Potree 3D point cloud viewer.

However, if EPT is just used for Potree, why not use PotreeConverter Version 2 instead to convert the LAZ file? It's orders of magnitude faster. Then, pc-ept could be enabled only when users really need it, saving huge amounts of time otherwise.

5 Replies

pierotofy

It's really strange that it's taking that long. 200M points aren't that big and it should be faster. Are you running an antivirus software or some other software that might be scanning/slowing down entwine in the background?

cwood256 OP

No anti-virus. I ran several tests to better see what the bottleneck is. For these tests I used a 540M point cloud that takes a little over 7 hours to convert from LAZ in WebODM.

I ran entwine build stand-alone on the georeferenced LAZ file, using both WebODM's fork of entwine (and its environment), as well as official distribution (via conda). The box used for testing has 24 cores (up to 4 GHz), 384 GB RAM, disks are all SSD, and runs Windows. The test SSD was 2 TB, with nothing else on it.

Here's what I found:

  1. Entwine build puts only a tiny load on the system - CPU load is minimal and RAM usage is small
  2. Entwine build gets progressively slower in terms of Mpoints processed per hour
  3. Varying the arg --threads from 10 to 48 made no real difference
  4. Varying the args --sleepCount and --cacheSize by factors of 2 from their default sizes made no real difference
  5. Using subsets (--subset) significantly improves performance: 4 reduced runtime by about a factor of 2.5, 16 reduced runtime by about a factor of 3.0, and 64 reduced runtime by about a factor of 2.7
  6. Running PotreeConverter 2 on the same LAZ file was by far the fastest: it completed in less than a minute
pierotofy

Interesting; unfortunately Potree converter cannot be used as-is, especially because v2 lacks compression, which is generally not a good tradeoff for speed (imo)

pierotofy

I would be interested in optimizing Entwine at some point though

cwood256 OP

For comparison, here are file sizes for my test case:

  1. original LAZ file: 2.9 GB
  2. EPT (same for any merged subset): 3.1 GB
  3. PotreeConverter Ver. 2: 17.5 GB
  4. 7z archive of PotreeConverter Ver. 2: 4.4 GB

The increase in size for the 7z-archived PotreeConverter V2 output is significantly outweighed by the 7-hour savings in processing time, at least for my use cases.
It would be nice to have some well-though out options for these tradeoffs available in the UI, although it's not too hard to hack the code for any particular use case.

Need more help?

This discussion happened in the WebODM community Discord. Join and ask your own question.

Join View Original Thread

This page mirrors a public discussion from the WebODM Discord. How to request a removal.