Cars BMW Z4 3.0i E85 with photogrammetry (WIP)

Discussion in 'Assetto Corsa Modding Discussions' started by Mr Whippy, Jun 11, 2016.

  1. Mr Whippy

    Mr Whippy

    After struggling with all the hidden curves and shapes on this car, and with blueprints that didn't really stack up with reality in many places, I decided I wanted to just spline cage model the car and use photomatched cameras and photo backgrounds in my 3D application to achieve that.
    But after struggling with all the hidden curves on this car that wouldn't even show up in spline cages, I decided to just have a bash with full on photogrammetry capture!

    I've never used this approach before, so getting really sharp results was key to getting a really great and accurate model, so I spent many hours shooting and testing.

    I think I did about 30 different tests with things like dusting the car, various lighting techniques, wrapping in paper, adding feature points (tape marks all over the bonnet), and about 5 different full car sets before refining the process to the point I was happy modelling off the data I got.

    Also it seems a few people are using photogrammetry for cars a bit more, so I thought I'd share the techniques I'd found worked really nicely for capturing my Z4!

    So I'll just start to babble and hopefully it's all making sense to those familiar with the technique. For everyone else then hopefully the pictures will be of interest :)

    Firstly, the camera I use is a Ricoh GR with 16mpx. I also have a circ polariser stuck on there (literally with black-tack :) ). It's on a very hefty tripod so I can shoot away for long exposures without issue, or even adjust things on camera between multiple shots (will explain later) and not worry too much about movement.

    Another little detail worth mentioning for those who saw it stuck on there, is my little radius gauge. Very handy for measuring radius of panel edges and bevels etc. No need to guess when you can measure, since even these photo scans or laser scans can't capture bevels all that well.

    Here is an image of the car so far which shows mostly test panels and parts to check the method works nicely.

    So on to how I captured the imagery.

    Those familiar with Agisoft Photoscan will glean a lot from the screenshots and what I've done and the results I've got.

    So for this part of the car I took 128 shots. I just used a generic vertical block on the polariser.

    Each shot is taken in RAW format. They're then batch converted through Photoshop RAW, levels adjusted and so on, then exported to TIF format for Agisoft Photoscan. TIF preserves all the details and provides excellent base imagery to reconstruct from.

    I think they're all shot at F8 so as deep a DOF as possible without suffering from blurring due to photon issues on the sensor (you get more DOF at higher f-stop settings, but also more noise and blur!)

    I'm using a feature on the camera which captures as wide a dynamic range as possible by using a higher ISO (320 vs 100).

    I'm using manual focus and keeping the camera a very specific distance from the car, so as much of the surface in shot between images is in focus.

    If you use auto-focus even on this type of camera (fixed aperture), the frame of the shots and thus the calibration of the camera in the Photoscan software will change.
    Since Photoscan assumes all shots you provide in a chunk are based on identical camera properties, this can cause inaccuracies. So if you change focus, all shots using a different focus need to be in a different chunk and aligned later.

    And the key to my results being so good is good old British weather! Rain and overcast skies!

    As you can see the rain drops act as a nice static (be careful here as they can evaporate in low humidity or move if it's windy or rainy, so work as fast as is possible) reference all over the cars surface. However probably due to some optical properties they don't really appear to show up as large humps all over the scan unless you only capture the features from steep angles.
    So if you shoot generally 'flat' through the drops then they reconstruct flat.

    To get good on-car beading you need to wax the car.

    Also small marks, tar spots, debris etc act as features to track if you have a good enough camera up close.

    Here is a good example of an image.

    So sharp details all over the surfaces facing us, good contrast of the features to track.

    Here is a less good example of an image.

    This is bad because the steep elements of the car bodywork are now reflecting the gravel floor, and that results in the missing data you can see in the whole car shot above.
    These reflections basically break the photogrammetry process, and reconstruct as if the panel goes under the car parallel with the gravel surface.
    The water drops present don't really stand out against this gravel reflection backdrop.

    So to fix this problem I had to re-shoot the side skirts another time. This was a good test to see how nicely I could reintegrate new chunks of car scan with my original one.

    So I took a line of shots along the skirt another day. No water drops this time. But I used a polariser set very carefully to remove as many reflections as possible, and didn't go beyond about 45 deg angle with the surface.
    Beyond 45 deg diffuse reflections are still significant, but with the polariser under 45deg you seem to be able to cut most of them out.

    An interesting thing I thought would work and was a primary reason I bought the polariser, but didn't use in this project, is taking multiple shots from each cam position.
    Rotate the polariser for each primary surface chunk that is reflecting the environment.
    Paint and mask each chunk in/out in Photoshop to get a really low environment reflection image to feed into Photoscan.
    The reason I didn't do this became clear on a basic test shoot. The Ricoh GR lens body moves relative to the sensor. Mounting the polariser on it meant it moved between shots! Arghh!

    I did try build a small mount to hold the polariser apart from the lens body but it wasn't working so well.
    So if you want to try that technique, think about a rigid mount to the tripod mount for the polariser, not one where you have to touch anywhere near the lens!

    You can see here that the reflections are well controlled here with just a basic single polarising angle but it's also a flat ish area too.
    Also some dirty marks on the skirt act as nice tracking details. Also at this close range, tar spots and other feint features are trackable which all helps with a good reconstruction.
    Last edited: Jun 11, 2016
    • Like Like x 7
  2. Mr Whippy

    Mr Whippy

    So once that was reconstucted I cut out the skirt.

    With this chunk of point cloud exported I now used Cloud Compare and the 'align point clouds' feature, to align this chunk with the original point cloud chunk. This worked well enough to get the point clouds very closely aligned.

    You'll also note the roof is missing in the original scan, and this too was captured separately.

    However since the roof is like a different part of the car, I built it separately after alignment. For this part of the build I generated a high density mesh from the pointcloud and used the Wrapit plugin from Matt Clark for 3DS Max.
    This is a great plugin for modelling over high density meshes and suited the task of this more organic and 'rough' form of a convertible roof. Also given the fairly high frequency detail of a convertible roof (seams and feint details), having a high density mesh to bake to a normal map is useful.

    To use the pointclouds to build from, firstly I draw 2d splines around panels. So for example the side skirt.

    I generate a nice topology and quad flow for the panel in question, helped with smoothing and flow tools like WrapIt once again.
    The edges are snapped to my spline outline (not shown here). The edges are then inset by around 0.5cm and removed. This serves two purposes.

    Firstly when you conform the mesh to the point cloud, the edges are obviously not entirely clear to the conformer as they fold away along the direction of the conform, so conforming to the flat surface away from the edge gets much better results.
    But also the edges need an edge treatment bevel too. So again using the WrapIt plugin, I can select the edges after conforming, and extend them by 0.5cm, and then extrude them again by 0.5cm to give the panel 'depth'

    As you can see this side skirt looks great. Infact this is one area where blueprints were useless as this part is almost totally hidden except for the 'side' view.
    Also in every other game I've seen this panel is clearly wrong in one way or another. It's really hard to get correct unless you have reference material like a laser or photoscan!

    The conform process is also not always perfect for whatever reason so can need some small fine adjustments. However the conform process, which multi-samples the point cloud to find the correct 'depth' for a vertex to be placed, usually gives very very nice results.

    Here are a few flow-check shots of conformed panels. As you can see they're pretty much perfect. At most I might push or pull one or two verts on a whole panel by 1 or 2 mm at the most to perfect the flow.

    Finally wheels and tyres.

    How I captured these is pretty self-explanatory in the screenshot. I've built off these using a combination of WrapIt with a high density mesh, and the pointcloud. You can't really conform to these high density details and shapes so just getting in there and modelling with reference meshes and point clouds is the only way to go.

    And to see how detailed I could capture a tyre, here is a tyre capture.

    As you can see it took 73 million points to get lots of nice details, but the mesh is at a level (after testing) to get some really nice normal map data for the tyre sidewall. Obviously it'll be hardly visible in the finished car but it's nice to have a true 3D source for the normal map and diffuse map to match.
    Unwrapped properly it's pretty much round. The only deviation is a feint one at the top of the tyre, and a matching but more dramatic one at the bottom of the tyre (contact with floor). I'll re-capture this with the wheel off the car and laying on it's side facing upwards then I can get it perfect.

    Here is a large image showing the cross-section of the car through the bonnet area just near the windscreen spraying nozzles!


    You can see the spraying nozzles clearly (though not perfectly sharp), and also the dips at each side of the bonnet where the wing panels are very marginally higher.
    So it's not pin sharp but the surface forms are perfectly clear to see... and even the overall shape and size of panel creases are visible enough to get more accurate than a blueprint build.

    Also here is a video just showing the point cloud cross-section being scanned through which gives an idea of how fine the surface itself is.

    With the radius gauge at the ready panel edge bevels can be recreated accurately too (though a ~25% oversize seems best for game engine models using triangle based geometry approximations)

    This data quality is sufficient for making any 3D model of the Z4 to a level of accuracy perfectly good for games/renderings in my view.

    In testing it became clear that to get a solid surface without wobbles and lumps you needed lots of detail, and you won't get that from far away due to the types of features you'd get on a car paintwork being generally small (water drops, tar spots, dust spots etc).
    Being close with good tracking points just works.

    I actually think with the right lighting (very even overcast) and very high quality camera, you could reconstruct based purely on the topology of the base coat (under the lacquer)

    Even on my Z4's silver paintwork in some tests (closer still), the metallic flake detail under the lacquer, between nearby shots, was consistent to be tracked.

    Ultimately, as long as features can be followed across 3 or 4 images, and the camera positions as a whole can be reconstructed in the overall scene, things should work ok.

    Really this whole technique of photogrammetry is arbitrary. No one says you have to do X and Y to get Z.

    I thought about shooting in IR for example, since a cars IR appearance will be significantly different to it's visible light appearance.

    Or even projecting patterns on to the car using a projector in my garage. Using a controlled environment you could remove reflections (except parts of the car body reflecting itself I guess) too, and possibly get a really great result.

    So many techniques and ideas to try. Just use your imagination :D

    If anyone has any questions please feel free to ask away and I'll try add more detail around specific points.
    • Like Like x 6
    • Love Love x 3
  3. Repa24


    Welcome to the forum! Interesting technique, excited for more!
  4. Mr Whippy

    Mr Whippy

    I have to do the interior too so I'll get that posted soon.

    Since this project has been running for about 7 years, heavily as a test bed for ideas, techniques and so on for Free Racer, there is oodles of work already created towards sounds, physics etc.

    I just need to finish up the model to a high standard, which I can now do with such excellent reference!

  5. Reference92


    Hi. Great thread, just read through and its really useful having so much detail about each stage that you went through. Good luck for your projects. 92
    • Like Like x 1
  6. Adi Sahar

    Adi Sahar

    MrWhippy + BMW = WIN. I can't forget that legendary M3. :)
  7. Mr Whippy

    Mr Whippy

    13 years ago! Yikes.

    That car got me an interview to make 3D cars for a living at the time.

    Last edited: Jun 11, 2016
    • Like Like x 2
    • Haha Haha x 1
  8. Adi Sahar

    Adi Sahar

    Yes, we all grew up since then.. or maybe not? :roflmao:
    • Haha Haha x 1
  9. Tyber325


    Man, I wish this mod was still being worked on :/ There isn't a single BMW Z4 E85 mod for assetto corsa except one but its the GT3 version.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.