How can I extract part id or design id from a photo?

Question

Is there any software/site that tells me which LEGO parts exist in a photo? I would love to get a csv of parts from a photo.

Answer 1

As far as I know, no such project or software or site exists. The closest one I could find is (was?) a thread in the Eurobircks forum, where exactly this topic came up. The members there tossed about various ideas, mentioning Akiyuki's brick sorter device (with the factors that made its job easier than the one you propose: uniformly lit background with known color, precision scale, only one piece at a time, a few hundred possible parts), some proof-of-concept code was written to test the capabilities of the OpenCV library, but then the interest faded and the topic was abandoned.

There are several problems in this proposition that are highly difficult (if not straight impossible) to overcome:

1. Lighting conditions

The optical recognition process depends heavily on sensing the difference in the brightness and color of adjacent points. If the lighting conditions are allowed to vary (that is, not requiring the use of a light box) then the differences between each color and the background can't be set in the program. One could still overcome this limitation if the image contains a piece of a known color, but then the calibration must be done for each image.

2. Overlap

If pieces are allowed to overlap then they can be easily mistaken for similar pieces, or if sufficient parts are overlapping then clear identification can become impossible. If you can only see a part of a Technic axle, you can't tell how much of it is under the 6x6 plate, so the lenght of the axle can't be determined. The same analogy works for many kinds of pieces (bricks, tiles, plates).

3. One point of view

If this system can work using only a single image (taken with an ordinary camera) then some pieces can become too similar to tell apart. For example, from straight up, a 2x2 brick and a 2x2 plate look exactly the same, just like a 2x2 brick and a 3x2 brick if both are standing on their end. Shadow analysis could help with this matter, but only if there are clear shadows cast from a known point in space related to the camera and the surface on which the pieces are laid out. So we have essentially arrived at the "very precisely controlled and known lighting and photographing setup", called light box, but also added a ton of processing logic and advanced mathematics. Not to mention the problem of dealing with shadows cast on other pieces, which can render shadow analysis moot or even obscure some pieces so they won't be recognized correctly or in extreme scenarios maybe not even seen.

4. Differences only visible from one side

This might not concern you, but in some cases pieces only differ from each other in a very small detail that can only be seen from a certain point of view. For example, if a minifig head was laying on its side, it is optically impossible to tell if it has a hole on the top or not.