Publications

2024

1. Contrastive Pretraining for Visual Concept Explanations of Socioeconomic Outcomes

   Ivica Obadic , Alex Levering , Lars Pennig , and 3 more authors

   In Proceedings of the 2024 IEEE/CVF International Conference on Computer Vision , Jun 2024

   Abs

   Predicting socioeconomic indicators from satellite imagery with deep learning has become an increasingly popular research direction. Post-hoc concept-based explanations can be an important step towards broader adoption of these models in policy-making as they enable the interpretation of socioeconomic outcomes based on visual concepts that are intuitive to humans. In this paper, we study the interplay between representation learning using an additional task-specific contrastive loss and post-hoc concept explainability for socioeconomic studies. Our results on two different geographical locations and tasks indicate that the task-specific pretraining imposes a continuous ordering of the latent space embeddings according to the socioeconomic outcomes. This improves the model’s interpretability as it enables the latent space of the model to associate concepts encoding typical urban and natural area patterns with continuous intervals of socioeconomic outcomes. Further, we illustrate how analyzing the model’s conceptual sensitivity for the intervals of socioeconomic outcomes can shed light on new insights for urban studies.

2. Landscape quality assessments using deep learning

   Alex Levering

   Jan 2024

   Abs

   Landscapes are an ever-present aspect of life, and whether for daily living or for relaxation, the qualities of landscapes influence the individuals that use them. They are important contributors to the health and well-being of humans and provide a number of important ecosystem services. Therefore, there is a need to map the qualities of our landscapes. Landscape qualities can be studied visually through photographs, with the help of AI models. Research performed in this thesis focuses on making assessments more broadly accessible, and easier to use. It introduces interpretability methods which make models easier to understand, tests methods for modelling using only a handful of examples, and studied methods which make use of multiple information sources to provide new insights and new ways of mapping and exploring landscape qualities.

2023

1. Time Series Analysis of Urban Liveability

   Alex Levering , Diego Marcos , and Devis Tuia

   In 2023 Joint Urban Remote Sensing Event (JURSE) , May 2023

   ISSN: 2642-9535

   Abs

   In this paper we explore deep learning models to monitor longitudinal liveability changes in Dutch cities at the neighbourhood level. Our liveability reference data is defined by a country-wise yearly survey based on a set of indicators combined into a liveability score, the Leefbaarometer. We pair this reference data with yearly-available high-resolution aerial images, which creates yearly timesteps at which liveability can be monitored. We deploy a convolutional neural network trained on an aerial image from 2016 and the Leefbaarometer score to predict liveability at new timesteps 2012 and 2020. The results in a city used for training (Amsterdam) and one never seen during training (Eindhoven) show some trends which are difficult to interpret, especially in light of the differences in image acquisitions at the different time steps. This demonstrates the complexity of liveability monitoring across time periods and the necessity for more sophisticated methods compensating for changes unrelated to liveability dynamics.

2. Predicting the liveability of Dutch cities with aerial images and semantic intermediate concepts

   Alex Levering , Diego Marcos , Jasper Vliet , and 1 more author

   Remote Sensing of Environment, Mar 2023

   Abs

   In order to provide urban residents with suitable living conditions, it is essential to keep track of the liveability of neighbourhoods. This is traditionally done through surveys and by predictive modelling. However, surveying on a large scale is expensive and hard to repeat. Recent research has shown that deep learning models trained on remote sensing images may be used to predict liveability. In this paper we study how well a model can predict liveability from aerial images by first predicting a set of intermediate domain scores. Our results suggest that our semantic bottleneck model performs equally well to a model that is trained only to predict liveability. Secondly, our model extrapolates well to unseen regions (R2 between 0.45 and 0.75, Kendall’s τ between 0.39 and 0.57), even to regions with an urban developmental context that is different from areas seen during training. Our results also suggest that domains which are directly visible within the aerial image patches (physical environment, buildings) are easier to generalize than domains which can only be predicted through proxies (population, safety, amenities). We also test our model’s perception of different neighbourhood typologies, from which we conclude that our model is able to predict the liveability of neighbourhood typologies though with a varying accuracy. Overall, our results suggest that remote sensing can be used to extrapolate liveability surveys and their related domains to new and unseen regions within the same cultural and policy context.

2022

1. Geoinformation Harvesting From Social Media Data: A community remote sensing approach

   Xiao Xiang Zhu , Yuanyuan Wang , Mrinalini Kochupillai , and 9 more authors

   IEEE Geoscience and Remote Sensing Magazine, Dec 2022

   Conference Name: IEEE Geoscience and Remote Sensing Magazine

   Abs

   As unconventional sources of geoinformation, massive imagery and text messages from open platforms and social media form a temporally quasi-seamless, spatially multiperspective stream, but with unknown and diverse quality. Due to its complementarity to remote sensing (RS) data, geoinformation from these sources offers promising perspectives, but harvesting is not trivial due to its data characteristics. In this article, we address key aspects in the field, including data availability, analysis-ready data preparation and data management, geoinformation extraction from social media text messages and images, and the fusion of social media and RS data. We then showcase some exemplary geographic applications. In addition, we present the first extensive discussion of ethical considerations of social media data in the context of geoinformation harvesting and geographic applications. With this effort, we wish to stimulate curiosity and lay the groundwork for researchers who intend to explore social media data for geoapplications. We encourage the community to join forces by sharing their code and data.

2021

1. Cross-Modal Learning of Housing Quality in Amsterdam

   Alex Levering , Diego Marcos , Ilan Havinga , and 1 more author

   In Proceedings of the 4th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery, GeoAI 2021 , Nov 2021
2. Detecting Unsigned Physical Road Incidents From Driver-View Images

   Alex Levering , Martin Tomko , Devis Tuia , and 1 more author

   IEEE Transactions on Intelligent Vehicles, Mar 2021

   Conference Name: IEEE Transactions on Intelligent Vehicles

   Abs

   Safety on roads is of uttermost importance, especially in the context of autonomous vehicles. A critical need is to detect and communicate disruptive incidents early and effectively. In this paper we propose a system based on an off-the-shelf deep neural network architecture that is able to detect and recognize types of unsigned (non-placarded, such as traffic signs), physical (visible in images) road incidents. We develop a taxonomy for unsigned physical incidents to provide a means of organizing and grouping related incidents. After selecting eight target types of incidents, we collect a dataset of twelve thousand images gathered from publicly-available web sources. We subsequently fine-tune a convolutional neural network to recognize the eight types of road incidents. The proposed model is able to recognize incidents with a high level of accuracy (higher than 90%). We further show that while our system generalizes well across spatial context by training a classifier on geostratified data in the United Kingdom (with an accuracy of over 90%), the translation to visually less similar environments requires spatially distributed data collection.

3. On the relation between landscape beauty and land cover: A case study in the U.K. at Sentinel-2 resolution with interpretable AI

   Alex Levering , Diego Marcos , and Devis Tuia

   ISPRS Journal of Photogrammetry and Remote Sensing, Jul 2021

   Abs

   The environment where we live and recreate can have a significant effect on our well-being. More beautiful landscapes have considerable benefits to both health and quality of life. When we chose where to live or our next holiday destination, we do so according to some perception of the environment around us. In a way, we value nature and assign an ecosystem service to it. Landscape aesthetics, or scenicness, is one such service, which we consider in this paper as a collective perceived quality. We present a deep learning model called ScenicNet for the large-scale inventorisation of landscape scenicness from satellite imagery. We model scenicness with an interpretable deep learning model and learn a landscape beauty estimator based on crowdsourced scores derived from more than two hundred thousand landscape images in the United Kingdom. Our ScenicNet model learns the relationship between land cover types and scenicness by using land cover prediction as an interpretable intermediate task to scenicness regression. It predicts landscape scenicness and land cover from the Corine Land Cover product concurrently, without compromising the accuracy of either task. In addition, our proposed model is interpretable in the sense that it learns to express preferences for certain types of land covers in a manner that is easily understandable by an end-user. Our semantic bottleneck also allows us to further our understanding of crowd preferences for landscape aesthetics.

4. Liveability from Above: Understanding Quality of Life with Overhead Imagery and Deep Neural Networks

   Alex Levering , Diego Marcos , and Devis Tuia

   In Proceedings of the 2020 International Geoscience and Remote Sensing Symposium , Jul 2021

2020

1. Interpretable Scenicness from Sentinel-2 Imagery

   Alex Levering , Diego Marcos , Sylvain Lobry , and 1 more author

   In Proceedings of the 2020 International Geoscience and Remote Sensing Symposium , Jul 2020

   Abs

   Landscape aesthetics, or scenicness, has been identified as an important ecosystem service that contribute to human health and well-being. Currently there are no methods to inventorize landscape scenicness on a large scale. In this paper we study how to upscale local assessments of scenicness provided by human observers, and we do so by using satellite images. Moreover, we develop an explicitly interpretable CNN model that allows assessing the connections between landscape scenicness and the presence of specific landcover types. To generate the landscape scenicness ground truth, we use the ScenicOrNot crowdsourcing database, which provides geo-referenced, human-based scenicness estimates for ground based photos in Great Britain. Our results show that it is feasible to predict landscape scenicness based on satellite imagery. The interpretable model performs comparably to an unconstrained model, suggesting that it is possible to learn a semantic bottleneck that represents well the present landcover classes and still contains enough information to accurately predict the location’s scenicness.

2018

1. Regional coastal erosion assessment based on global open access data: a case study for Colombia

   Joost Stronkhorst , Alex Levering , Gerrit Hendriksen , and 2 more authors

   Journal of Coastal Conservation, Aug 2018

   Abs

   Many coastlines throughout the world are retreating, as a result of erosion and sea level rise. The damage incurred to property, infrastructure, coastal flood defence, and the loss of ecosystem services and agricultural land have substantial economic repercussions. For many coastal regions located in developing countries, the assessment of the spatial extent of coastal erosion is very time-consuming and is often hampered by lack of data. To investigate the suitability of global open access data for coastal erosion assessments at regional scale six biogeophysical variables (geological layout, waves, sediment balance, tides, storms, and vegetation) were integrated using the Coastal Hazard Wheel approach (CHW). Original datasets with global coverage were retrieved from the internet and from various research institutes. The data were processed and assigned to the CHW classes, so that the CHW method could be applied to assess coastal erosion hazard levels. The data can be viewed in the Coastal Hazard Wheel App (www.coastalhazardwheel.org) that also allows the coastal erosion hazard levels to be determined for each point at coastlines around the world. The application of the CHW with global open access data was tested for the Caribbean and Pacific coasts of Colombia and revealed a high to very high erosion hazard along 47% of the Caribbean coast and along 23% of the Pacific coast. The application provides additional information on capital stock near the coast, as a tentative indication of assets at risk. This approach provides a straightforward and uniform erosion hazard identification method that can be used for spatial planning on coastal developments at a regional scale.