S4D: Providing proper glasses (sensor control) would be better than solely depending on over-training the brain (neural network)!
2023.08 - current
[Key Concept]
Current AI improvement methods focus solely on the 'brain' components.
[ImageNet-ES]
In contrast to conventional robustness benchmarks that rely on digital perturbations, we directly capture 202k images by using a real camera in a controllable testbed. The dataset presents a wide range of covariate shifts caused by variations in light and camera sensor factors. Download ImageNet-ES here
[ ImageNet-ES ] A new distribution shift dataset, comprising variations in environmental and camera sensor factors by directly capturing 202k images with a real camera in a controllable testbed.
[ES-Studio]
To compensate the missing perturbations in current robustness benchmarks, we construct a new testbed, ES-Studio (Environment and camera Sensor perturbation Studio). It can control physical light and camera sensor parameters during data collection.
Computer vision applications predict on digital images acquired by a camera from physical scenes through light. However, conventional robustness benchmarks rely on perturbations in digitized images, diverging from distribution shifts occurring in the image acquisition process. To bridge this gap, we introduce a new distribution shift dataset, ImageNet-ES, comprising variations in environmental and camera sensor factors by directly capturing 202k images with a real camera in a controllable testbed. With the new dataset, we evaluate out-of-distribution (OOD) detection and model robustness. We find that existing OOD detection methods do not cope with the covariate shifts in ImageNet-ES, implying that the definition and detection of OOD should be revisited to embrace real-world distribution shifts. We also observe that the model becomes more robust in both ImageNet-C and -ES by learning environment and sensor variations in addition to existing digital augmentations. Lastly, our results suggest that effective shift mitigation via camera sensor control can significantly improve performance without increasing model size. With these findings, our benchmark may aid future research on robustness, OOD, and camera sensor control for computer vision. Our code and dataset are available at this https://github.com/Edw2n/ImageNet-ES
@inproceedings{imagenet-es,author={Baek, Eunsu and Park, Keondo and Kim, Jiyoon and Kim, Hyung-Sin},title={Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains},booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},year={2024},month=jun,tags={s4d}}
