Earth Observation: Recent Episodes
mapscaping.com
Computer vision is everywhere! But teaching an algorithm to identify objects requires a lot of data and this is definitely the case when we think about GeoAI
But it is not enough to have a lot of data we also need data that is labeled
If we are looking for cars in images we need a lot of images of cars and we need to know which pixels are the car!
Of course, I am oversimplifying but I hope you get the idea,
Now imagine that you can automatically generate a large labeled data set of realistic images of cars based on the specifications of a specific sensor.
These data sets are often referred to as synthetic data or fake data and to help us understand more about this I have invited Chris Andrews from Rendered AI on the podcast.
Here are a few previous episodes you might find interesting
Computer Vision And GeoAI
https://mapscaping.com/podcast/computer-vision-and-geoai/
In this episode, the discussion is aimed at an increased understanding of the differences between computer vision and the AI that is used in the Earth Observation world.
Labels Matter
https://mapscaping.com/podcast/labels-matter/
What it takes to create labeled training data manually. If you are new to the idea of labeled data sets this is a good place to start.
Fake Satellite Imagery
https://mapscaping.com/podcast/fake-satellite-imagery/
This is a good episode if you want to know more about Generative AI and Generative Adversarial Networks.
Also, check out this website https://thisxdoesnotexist.com/ to get an idea of where and how these Generative Adversarial Networks can be used. Look for a website called This City Does Not Exist http://thiscitydoesnotexist.com/
On a silently similar note try uploading an image to https://bard.google.com/ … it's pretty interesting!
How do we get data from a satellite down to Earth? How do we task a satellite?
Today the answer is likely to be via radios and a system of downlink sites or ground stations. As the satellites pass overhead or within “line of sight” data can be sent via radio from the satellite to the receiver on the ground.
If you don’t want to wait until the satellite can see the ground station, you can send your data to a geostationary satellite that can always see a ground station and let it send the data back to Earth.
Radios are tried and tested, they have been used for this purpose since the inception of satellite communication and radio waves can pass through Earth's atmosphere without significant loss!
But … the frequency spectrum for radio waves is strictly regulated, which can limit available channels for communication, and the bandwidth of radio frequencies is limited, which can reduce the volume of data transmission.
What about lasers?
You can send more data faster with a laser, you don’t need to worry about interfering with someone else part of the radio spectrum, and ground stations can be much smaller even human-portable!
But … lasers struggle with clouds and the technology is still relatively new
So what is the best way to communicate with satellites? Radio or Laser? The answer is … it depends ;)
Jordan Wachs, Director of Business Development for SpaceRake.net does a great job adding context to this discussion but perhaps the bigger question here is what will we do when satellites become internet devices, part of the Internet of Things?
What if they were always on always connected in the same way your phone is always on, always connected? What will this enable?
This episode was sponsored by Sponsored by Sinergise, as part of Copernicus Data Space Ecosystem knowledge sharing
People who liked this episode also liked …
How to keep your satellite pointing at earth
https://mapscaping.com/podcast/how-to-keep-your-satellite-pointing-at-earth/
Hyperspectral v’s Multispectral
https://mapscaping.com/podcast/hyperspectral-vs-multispectral/
Sentinel Hub
https://mapscaping.com/podcast/sentinel-hub/
Swing by our website sometime https://mapscaping.com/
Computer vision is a field of artificial intelligence (AI) that enables computers and systems to derive meaningful information from digital images.
You might think that this is exactly what we are doing in earth observation but there are a few important differences between computer vision and what some people refer to as GeoAI.
This week Jordi inglada is going to help you understand what those differences are and why it's not always possible to use Computer vision techniques in the field of Remote Sensing.
Listen out for these key points during the conversation!
- Why plausible or realistic data is not always a substitute for actual measurements, except when it is ;)
- In computer vision we can learn from the data, in earth observation we know the physics
- To do interesting work in data science you need to - Computer science, applied math, and domain expertise. You don’t need to be an expert in all three but you need to be interested in all three
- Vectors in the machine learning world don’t necessarily have anything to do with points lines and polygons ;)
Sponsored by Sinergise, as part of Copernicus Data Space Ecosystem knowledge sharing. dataspace.copernicus.eu/ http://dataspace.copernicus.eu/
Related Podcast Episodes
Super Resolution
https://mapscaping.com/podcast/super-resolution-smarter-upsampling/
Fake Satellite Imagery
https://mapscaping.com/podcast/fake-satellite-imagery/
Sentinal Hub
https://mapscaping.com/podcast/sentinel-hub/
Google Earth Engine
https://mapscaping.com/podcast/introducing-google-earth-engine/
Microsofts Planetary Computer
https://mapscaping.com/podcast/the-planetary-computer/
BTW MapScaping has started a Job Board!
it's in the early stages but it's live
Jobs - Mapscaping.com
When comparing multispectral and hyperspectral data it is not simply a case of “more data more better”!
With hyperspectral you have “The curse of Dimensionality” but you also get more flexibility to pick exactly what bands you want to use!
With multispectral you have less noise but you also have less data!
This episode is designed to be a beginner's guide to the differences between hyperspectral and multispectral satellite data. Sponsored by Sinergise, as part of Copernicus Data Space Ecosystem knowledge sharing. dataspace.copernicus.eu/ http://dataspace.copernicus.eu/ You can reach out to Gordon Logie here: https://sparkgeo.com/blog/team/gordon/
Here are some courses that focused on hyperspectral and offer further training
https://eo-college.org/courses/beyond-the-visible/ https://eo-college.org/courses/beyond-the-visible-imaging-spectroscopy-for-agricultural-applications/ https://www.enmap.org/events_education/hyperedu/
Geospatial Technology Is No Longer A Luxury – It Is A Necessity!
In this episode, the discussion is about the role of geospatial at the UN World Food Programme (WFP). The combination of remote sensing, the use of drones, and GIS offer the best solutions that the WFP turns to for evidence-based information about a disaster, which helps response teams in their operations. Geospatial helps to answer the questions such as: which areas face a risk of food insecurity, who are the affected population, and how WFP and its partners can quickly reach them.
Want to stay ahead of the geospatial curve? listen to our podcast! About The Guest Rohini Swaminathan is the head of the geospatial Support Unit at the UN World Food Programme. She holds a bachelor's in Geoinformatics and a master's in Geomatics with a focus in Remote Sensing. Her work experience spans a lot of different applications ranging from environmental science and biological applications to disaster management. Currently, her focus is more on sudden onset disasters that happen at a large scale such as floods, earthquakes, and tropical storms, among others.
The focus is not only to respond after the disasters happen but also to prepare adequately for the disasters before they actually happen.
Disaster Preparedness – Advanced Disaster Analysis and Mapping (ADAM) Today, WFP focuses more on disaster preparedness as much as post-disaster emergency response. With the ADAM system that issues early warning alerts for any country that may be hit by a certain disaster, WFP receives beforehand information on the extent to which a certain disaster would be hazardous.
ADAM provides a holistic picture of what the potential impact of a disaster looks like, how many people would be exposed, and where they would potentially be displaced to. With this information, the welfare program, other UN agencies, and NGOs can plan on how to provide the much-needed response as quickly as possible.
Where Does WFP Get Mapping Data? To be able to serve the areas where the welfare program operates, WFP needs up-to-date mapping data for those areas. Although the entire world is not fully mapped, there are many mapping programs that are making efforts to map the world. Even so, the data is not all gathered and coordinated in one single database that can easily be accessed. The WFP does not usually collect data itself but relies on partners (different mapping programs) to provide the data. WFP then gathers the data, cleans it, and stores it in a way that is quickly accessed during response operations.
The UN WFP Hunger Map The WFP hunger map shows what the food insecurity of a population looks like at any given day. A dedicated unit for vulnerability analysis and mapping conducts surveys within a population to determine its exposure to food shortage. Gathering information through multiple different ways including mobile phones and online data collection tools, internally developed machine learning.
Geospatial Technology Is Not A Luxury – It Is A Necessity! Evidence-based information is required to be able to efficiently reach people in need. This makes geospatial a critical part of disaster preparedness and response efforts. With this awareness, geospatial technology is no longer thought of as a luxury thing that could possibly add value. Today, as soon as a disaster such as a flood occurs, the first immediate request is to get geospatial-based evidence information that could be used. Geospatial technology is a necessity that needs more integration and institutionalization.
Acknowledging Limitations in Geospatial Technology There is only so much that geospatial technology can do. The technology has limitations. But oftentimes, people tend to overestimate the capabilities of this technology. Rohini sums this up in the following words:
“Sometimes, it is possible that people's expectation on what the technology could do is more than what it can actually do.”
A good example is drones – whose potential is often overestimated. In a massive disaster, a drone would be limited in how much area it can cover and how fast the area covered can be processed. Turning to satellites, the imagery coverage might not always capture the whole extent of what a disaster looks like. There is still a need to research how various satellite data that overlap by time and space can be combined in a way that gives a holistic representation of how things are.
Optical imagery is often misunderstood as well. Right after a flood, there would be a lot of cloud cover which will make it impossible to use optical imagery for data collection. In such a scenario, radar data may offer an alternative, but it also comes with its own challenges. Trying to translate these technical difficulties is not something most people would be able to grasp. Not so many people are aware of the difference between optical and radar imagery.
Overlooking these limitations is what often leads to other people overestimating what can be achieved with existing geospatial technology. Geospatial professionals should be as realistic as possible in the way they translate the highly technical products and the limitations they have into something useful, usable and understandable by people with no technical background at all.
Skills You Need For a Successful Geospatial Career in the Humanitarian Space When seeking to work in the humanitarian space one of the things to do is to demonstrate your passion for that kind of work. There are different ways to do this, either through working in NGOs, volunteering, or getting hands-on and doing some work on your own i.e. taking some datasets and trying to understand how it can be useful in a certain context.
Humanitarian agencies such as the UN WFP have a goal of making a product that could be as wholesome as possible in its usefulness at the last mile response operations. One of the main skills to have is an improved understanding of the application of the technology. In addition to an understanding of the technology side of GIS and the data, an understanding of the context of how the technology can be used is also critical for a successful career in the humanitarian space.
Personally, I don't feel like aerial imagery gets the attention it deserves! So I invited Michael Bewley - Senior Director of AI Systems at Nearmap back on the podcast to help bring us up to speed on the state of the art of capturing, processing, and building a business around aerial imagery.
If you don’t care about aerial imagery, think of this as a story about turning unstructured data into structured data into insights and building a business around that.
You can connect with Micheal on Twitter and LinkedIn
Listen out for the following highlights
- It's not a camera it's an imaging system!
- Detecting change is not hard, detecting meaningful change is hard
- Are human abilities still a good benchmark for AI systems?
- How to determine if an AI system is a prototype or production ready
Previous Interview with Michael Bewley
Stratospheric Balloons As Remote Sensing Platforms
NICFI Satellite Data Program – Finding Values That MatterTropical forests play a critical role in reducing the amount of carbon in the Earth’s atmosphere. Despite this, we have not been able to sufficiently measure and understand one of the biggest threats to our climate – deforestation. Featured on this show is Tara O'Shea, the Senior Director of Forest and Land Use at Planet. Tara shares how Planet and other partners have come together under NICFI’s (Norway's International Climate and Forest Initiative) satellite data project to make high resolution satellite data available for tropical forest monitoring systems and researchers across the world.
Deforestation CrisisSince our earliest years, we have been taught that trees are important, and why. Trees act as carbon sinks, absorbing carbon dioxide from the air and giving back oxygen, in addition to securing soil, and acting as flooding mitigators, amongst countless other benefits. Yet for many people, the value of forests remains solely the economic incentive of converting forest resources into products. As a result, the rate of deforestation happening globally is unsustainable. We are quickly turning one of the largest terrestrial carbon sinks into one of the largest sources of climate changing greenhouse gas emissions.
Why Do We Need High Resolution Imagery?A higher resolution image can reveal what would not be visible in lower resolution imagery. Lying at the root of managing the climate and sustainability crises, is the inability to measure the things that matter. Capturing information and the scale of change at the pace at which it is happening on the ground is critical for better management of forest areas. The forestry sector has struggled to manage some aspects that matter because they could not be measured using low resolution imagery. As the saying goes, “you can’t fix what you can’t see.” Finding a way to value and allocate capital on the services offered by forests will help to reduce the value placed on economic activities that destroy forests.
What Is NICFI?Administered by the Norwegian Ministry of Climate and Environment, NICFI (Norway's International Climate and Forest Initiative) is an initiative to save the world’s tropical forests and improve the livelihoods of its inhabitants. Since September 2020, the NICFI satellite data program has helped to make high resolution satellite imagery available to tropical forest monitoring systems around the world. The data is distributed under a license by the Norwegian government, giving free access to NGOs, governments, and any other users that are doing some work around deforestation in the tropics. The program’s lifespan is 3 years, and is set to run through September 2023.
NICFI Satellite DataPlanet is one of the key partners that captures high resolution satellite imagery that is then made available through the NICFI satellite data program. Every day, Planet runs satellite missions to image the full Earth using hundreds of small satellites. The imagery is multispectral and has a spatial resolution of less than five meters. Four bands are captured in the imagery; the Visual (Red, Blue, Green) bands and the Near Infrared (NIR), which is critical in vegetation monitoring. From the daily imagery captured over an entire month, the best images are selected and stitched together into a single analysis-ready data layer. Choosing from the daily scans to provide a monthly dataset makes it possible to pull cloud free pixels, even in very cloudy places, which is significant given that the tropics are generally a very cloudy region.
Data Access LevelsIn order to reach a wide variety of users in the tropical forest monitoring community, the data is provided at different access levels, each offering a different product. The levels are designed to serve different sectors of users with varying degrees of technical capacity and need. The data access levels are categorized as follows:
Level 0:Available at this level is a visual mosaic product which has been optimized for display and visual interpretation. It is a three-band (RGB) optical product, particularly useful for journalists and locals in indigenous communities.
Level 1:Level 1 data is an analysis ready mosaic product with all four bands (Visible RGB and NIR). Optimized for analysis, the product is fully downloadable and can be accessed through Planet APIs in whichever operational environment a user is working in. Level 1 users also have access to a data archive that dates back to December 2015.
Level 2Users at this level have access to the scenes data that went into the mosaic products of the lower levels. Additionally, they also have access to archived data that dates back to 2002. Level 2 membership is only for a selected number of users named by the Norwegian ministry.
Who Is Allowed to Access NICFI Data?The goal of the NICFI satellite data project is to provide a common, improved high resolution dataset that can inform stakeholders’ work flows instead of everyone making decisions using different information. As such, anyone can access the data under a non-commercial license. The data usage must align with NICFI purposes to reduce and reverse tropical forest loss, combat climate change, conserve biodiversity, and facilitate sustainable development.
Where Can Users Access NICFI Data?Efforts have been made to “meet the users where they are” by making data available in existing operational and decision making environments that are already being used. Adding to Planet’s data platform, the data can also be accessed through a number of its partner platforms. For instance, the level zero visual data can be freely accessed through Sentinel Hub, Global Forest Watch, and SEPAL, among others. The analysis ready data is available to users in Google Earth Engine, and those working in ArcGIS, QGIS, or other GIS environments can use Planet’s API and existing integrations to pull the data into their working environment.
How Is NICFI Data Currently Being Used?Several governments across the tropics are using the NICFI data to honor their Paris Agreement on Climate Change commitments. Additionally, the data is also useful in domestic policy, planning, and enforcement.
In the private sector, several entities are using the data to gain insights into parts of their supply chain that have risks of deforestation. On the other side of the coin, local and indigenous communities use the data to monitor their territories. Media groups are finding the visual data helpful in reporting as well.
Bringing Stakeholders TogetherThere are greater benefits in getting different people from different disciplines to use the data in their workflows. Integrating different datasets can draw more insights into land use change. We will not only be able to understand what the forests are being turned into, but also to identify the supply chain that produces that change.
Is Culture Holding Us Back In The Fight Against Deforestation?The role of forests in the ecosystem is a concept that indigenous communities and cultures seem to have understood and put into practice for thousands of years. However, modern cultures struggle with this aspect. A lot of value is placed on the economic incentives from deforestation more than the important role played by forests in preserving the health of our planet. We need a change of culture to one that understands the services of forests and places more value on them.
Artificial Intelligence In Simpler TermsThe guest on this episode is Daniel Whitenack. He is a data scientist at SIL International, a teacher of AI, and a co-host of the Practical AI podcast. With his background in computational physics and his day to day role as a data scientist, Daniel has developed expert-level modelling and math skills. Currently, he is working on AI technology that benefits local language communities (i.e. speech recognition for local languages, machine translation, and different natural language processing techniques). On today’s show, he helps to demystify questions around Artificial Intelligence (AI), machine learning, and deep learning.
What Is AI?Put simply, Artificial Intelligence can be thought of as a function of a software. In programming, a function is the logic or algorithm that performs a certain transformation on an input, and gives a certain output. Usually, a function is created by a developer who curates the logic associated with that function. The developer specifies exactly what happens to the input, how it is transformed, and how the output is provided. In the case of AI, instead of a human specifying all the logic of a function, they create a function whose internal parameters are not set. These parameters are left to be set by the computer itself later on. This is the main distinction of AI functions from other conventional functions in software engineering, the computer is trained to select its own parameters to achieve a goal.
What Is Algorithm Training?When an AI model is created, its functions are un-parameterized. This means that although the structure for handling the desired data transformation is defined, the parameters that should be used are not specified. The computer has to learn and figure out the right parameters that will transform the input into the desired output so that it can fill them out itself for future runs.
In algorithm training, the computer learns to set the optimal parameters for an AI function through a trial and error cycle. The training data provided to the AI algorithms includes samples of the expected input to the model, as well as samples of the outcome that the model is expected to produce. Using the sample data, the computer sets the parameters for the algorithm. Tests are conducted to see which parameters produce the best results in comparison to the sample output, and the model’s settings are refined through an iterative process until this is achieved.
What Are Transferable Models?In the world of AI models, transferability means that an existing model that solves a certain problem, and can be used to solve another different, but very similar problem. Tweaking the parameters a little bit can make the model useful to another use case. For instance, by just tweaking the parameters of a model trained to recognize dogs in images, it could be used to recognize cats in images. Transferable models are more computationally favorable since training does not start from scratch, but instead makes use of the knowledge and efforts already developed in the parameter and training sample set.
What Are Model Architectures?An architecture of models is a configuration of a neural network that is composed of different layers bolted together. Each layer in the neural network is better than its counterparts in processing particular types of data. Model architectures are useful in applications like Natural Language Processing (NLP) which often uses recurrent layers to process the sequences of text. In NLP, text is treated as a sequence of words and characters. The neural networks process the order in the sequence and the relationship between parts of the sequence in order to pull meaning from the individual words to create a sentence, and impose context.
When Should AI Be Used?There are typically two instances that would tell whether AI is beneficial for a certain application. The first is if the data transformation that needs to happen is so complicated that a human cannot seem to be able to do it reliably every time. An example of this is detecting mental health issues from a person’s voice. While it is nearly impossible for a human to do this due to the very complicated data transformation that needs to happen, an AI model would do it very efficiently.
The second is the scale of the task. When the scale of the task is too large and repetitive, a human may not be the most efficient option for doing it. For instance, it would be tedious for a human to have to classify two million aerial images, breaking them down into different land use types. At such large scales, it would be best to automate with a model.
What is the Difference Between Machine Learning and Deep Learning?The main distinction between machine learning and deep learning is the scope at which the models are operating. Machine learning models operate within a smaller scope, and do not usually utilize neural network architectures (e.g. decision trees, random forests, naive Bayes, among many more). On the other hand, deep learning occurs at a larger scope, and requires less human intervention than machine learning.
Deep learning usually involves neural network architectures that can take in millions or even billions of parameters. Significantly more data is required to properly fit the parameters of functions in deep learning than what would be required in machine learning models. Similarly, more computing power is needed in deep learning than in machine learning.
Making the Choice Between Machine Learning and Deep LearningThere are some situations where deep learning, though applicable, may not be the best option for your use case. One is when we require clear interpretability of the decisions we make. Certain industries, i.e. in finance or health care, there could be a high burden or government regulation that requires you to be able to explain and audit the decisions that you are taking. This means clear documentation on every step taken in a process. In cases such as this, a simpler model may be more appropriate since it is possible to actually tell how a decision was made, you aren’t plugging everything into a black box.
The other element is the cost involved. In certain cases, a lot of specialized, expensive hardware may be required in a deep learning AI project which you may not already have access to. If a simpler machine learning model can be trained on a laptop to solve the same problem, then there shouldn’t be a need to spend thousands of dollars on specialized AI hardware clusters for deep learning.
Do You Need Specialized Hardware to Use AI?The phases of working with AI models can be broadly put into two categories – training and inferencing. The training side is often the more time and resource intensive side that would require specialized hardware, especially if it is deep learning. A lot of computations are required to set the parameters of a model and extra Graphical Processing Units (GPUs) may be necessary depending on the scale of the data.
Inferencing is when the model is making predictions after it has already been trained. Specialized hardware is not usually required, save only for certain use cases. In real time processing and data feeds, where instant results are required, specialized inference hardware might be required. Otherwise, many AI models can run on CPUs since inferencing is not as computationally intensive as training.
Demystifying AIFor quite some time, the AI industry has faced two extremes: a heightened hype of unwarranted expectations, and on the other end, a total shutdown on the applicability of AI in certain cases. This is well captured in the words of Daniel Whitenack;
“If you are on the side of things where you are thinking AI is the solution to every problem, then you are overestimating its utility and where it is at the moment. On the other hand, if you are running a business of any size, operating at least some type of technology, and you think AI cannot solve any of your problems, then you might be mistaken as well.”
Applications of thermal imaging from space include monitoring wildfires, urban heat islands, economic activity, and the built environment.
But it's not easy ;)
Connect with Robin Cole at
https://robmarkcole.com/
Check out the Earth Observation Hub!
https://geoawesomeness.com/eo-hub/
More Geospatial Podcasts episodes
Recommended Podcast Earth Observation Podcasts
Fake Satellite Imagery
https://mapscaping.com/podcast/fake-satellite-imagery/
The LandSat Program
https://mapscaping.com/podcast/the-landsat-program/
How To Keep Your Satellite Pointing At Earth
https://mapscaping.com/podcast/the-landsat-program/
It turns out that monitoring atmospheric pollution from space is really hard! But if you can do it will help you understand air quality, solar energy, ozone Layer and UV radiation, emissions and surfaces fluxes, and climate forcing
Contact Mark on Twitter
https://twitter.com/m_parrington
Copernicus Atmospheric Monitoring Service
https://atmosphere.copernicus.eu/
https://twitter.com/CopernicusECMWF
Check out the Earth Observation Hub!
https://geoawesomeness.com/eo-hub/
More Geospatial Podcasts episodes
Recommended Podcast Earth Observation Podcasts
Fake Satellite Imagery
https://mapscaping.com/podcast/fake-satellite-imagery/
The LandSat Program
https://mapscaping.com/podcast/the-landsat-program/
How To Keep Your Satellite Pointing At Earth
https://mapscaping.com/podcast/the-landsat-program/
SAR Technology: Looking Beneath the Earth from Space
The guest on today’s show is Lauren Guy, the CTO and founder of Asterra. He has a background in Geophysics and discovered the technology on which Asterra is built while during his masters. Lauren was involved in a project that used radar sensors orbiting Mars to search for water on the planet. The Synthetic Aperture Radar (SAR) signals used penetrate the ground to give a clearer view of what lies beneath and whether there is any indication of the presence of water.
How Bad Are Water Leakages?
Most of our drinking water is transported from far-off locations like reservoirs or even desalination plants near the oceans.
About 30-40% of the water that is moved around the world is lost through leaky pipes.
It is not only water that is lost in leakages – a huge amount of energy (used to pump the water) is lost as well. Considering the growing impacts of climate change, and looming future water scarcity issues, this is incredibly significant.
How Does Synthetic Aperture Radar (SAR ) Find Water Leakages on Earth?
Different materials have varying dielectric constants and therefore, electrical conductivity. These properties cause materials to reflect SAR signals differently. Treating drinking water gives it a distinct salinity level from other makeups of water.
Since salinity affects conductivity, it reflects a distinct SAR signal that differentiates drinking water from other kinds of water sources in the ground.
The assumption made is that there is no other source of drinking water in the ground – it has to come from pipes. This makes it possible to use SAR to create an underground map illustrating leakages.
Even with the capability to accurately isolate drinking water from other kinds of water, there are still possibilities for false positives as the same water is used for a lot of different uses like watering lawns, gardens, or filling swimming pools. The isolation has to go a notch higher to be able to distinguish the drinking water that is coming from pipes.
One way to do this is by calibrating the algorithm to only show moisture in the ground that has been accumulating for more than 48 hours. The thinking behind this is that most people do not water their lawn for more than 48 hours at once. This helps to avoid wasting time on false triggers.
How Far Can SAR Penetrate into the Ground?
SAR can only penetrate a few meters into the ground, depending on the soil type, and the top covering (i.e. asphalt, pavement, etc.). Generally, the depth of SAR penetration is about 2m in cities, 5m in more rural locations, and up to 10m in very sandy soils. The penetration depth of SAR is suitable for this application since water pipes are usually laid at a depth of 1m.
Georeferencing SAR Images
Georeferencing is a critical part of working with SAR images. The images need to be georeferenced in order to figure out where they are on the surface. Finding the exact location where there is a leak as shown in a SAR image is very important to avoid sending a crew to the wrong place.
Since SAR sensors are usually pointed at Earth at an oblique angle, georeferencing SAR images can be difficult. Georeferencing algorithms have to undergo a robust training phase in order to achieve the level of accuracy required. As we continue to see the popularity of SAR technology grow, we may see this get easier.
Tackling Signal Noise in Urban Environments
Telecommunications in urban environments creates a lot of noise for SAR sensors as they use the same frequencies. Reflective surfaces also create a lot of noise. This problem can be overcome by using different polarizations. A SAR signal can be sent in three ways: Vertical, horizontal, or as an alternating combination of the two in some cases.
When the signal is bounces off different materials or noises, the polarised signal goes from vertical to horizontal and vice versa. Detecting these polarization changes, and measuring their magnitude makes it possible to identify the source of noise that caused the change of polarizations and correct for it.
Building a Business in Around SAR Tech
The problems that Asterra faced while building a business around SAR technology are quite the same for many companies that are developing new solutions in tech. Despite putting in a lot of effort to convince clients that the technology actually works, it is even more difficult to convince a client to create a new budget to buy that solution.
For Asterra, these were passive utility companies that were not actively looking for leakages in their infrastructure, but primarily relied on citizens to report suspected leakages. One of the more common ways a customer might notice a leak is a sudden spike or even gradual increase in their water bill, without having changed their habits. As utility companies did not have an existing budget for a field team that looked for leakages, it was difficult for them to justify a new expense.
Why You Should “Speak the Same Language” As Your Clients
The best kind of clients are the ones that are already solving a problem. It is easier to convince a client to buy your solution if you can acknowledge their existing ideas and solutions, and then offer your ‘new’ solution as a complement to theirs. It will not sound realistic to them to ask them to throw everything out – what is often very expensive equipment – and replace it with your solution. A general good tip in convincing anyone of something, is to make them feel like they came up with it themselves.
For Asterra, it was easier to convince utilities that already had a field team that were actively looking for leakages in their infrastructure, and already had a budget for it. The companies could just reappropriate the budget towards buying the new solution, and even save costs as Asterra’s solution is often cheaper than the original way.
There were also advanced utilities that not only have a field team, but also very expensive IoT equipment for finding leakages. Due to the high costs involved, it is not economical to install the equipment throughout an entire city. For these cases, Asterra’s solution could be adopted as a complement to these devices in order to identify the most problematic areas in the city, and shift the more costly equipment to those locations where it is needed most.
Embracing Competition in Tech Businesses
For most businesses, the thought of competitors may be dreadful, but in tech businesses, competition might just be the force that drives the business forward. Being the only player in the game may be viewed by clients and investors to mean that the market is not viable. It leaves them with many questions of why there are no other solutions in that area already if there is so much money to be made. Competition fuels more discussion around a technology, which is an opportunity for the newest solutions to gain exposure.
Other Applications of SAR Technology
SAR technology is also useful in monitoring other important infrastructure like highways and railways for potential issues. Since SAR is looking underground, the issues it identifies may not be apparent yet from ground level. Accumulated water causes most of the issues in infrastructure. Pinpointing locations with very high soil moisture can help railway companies, or a country’s Department of Transportation to identify infrastructure that may fail soon. SAR’s ability to penetrate the ground at night and in any kind of weather can be harnessed in many other applications, such as mineral explorations, defence, and identifying contaminated soils. Who will be the ones to make it happen?
Recommended Podcast episodes
How to Keep Your Satellite Pointing at Earth
The guest on this show is Jack Reed, a PhD student at the MIT Media Lab. He started out in mechanical engineering and then later on moved into aerospace engineering. At the MIT Media Lab, he is part of an interdisciplinary research group alongside lab mates with backgrounds in data science, ethics, and art. Together they work on making space sustainable, and using space-based assets and imagery to help promote sustainability on Earth.
Why do We Need to Orient Satellites?
We tend to assume that since space is a vacuum, then there is nothing that would make the satellite drift from its original course. If this is the case, then there should really be no need to worry about orientation since the satellite should keep pointing to the same direction.
In reality, satellites are affected by forces like atmospheric drag, especially in low Earth orbits. Atmospheric drag slows down satellites and pulls them out of orbit.
Another reason to orient satellites is that as they revolve around the Earth, at 180 degrees they would be pointing directly away from the Earth. Unless they are rotated it will have to go another 180 degrees before they point directly at earth again.
In order to keep the satellite pointing at Earth at all times, they need to be rotated constantly, otherwise, they lose half of their utility.
How Satellites Orientate Themselves in Space?
The ability to control a satellite’s position in three dimensions, and where it is pointing is critical to increasing its lifespan. Controlling it in three-dimensional space helps to keep the satellite in the correct orbit, while controlling where it is pointing to ensure that the satellite is capturing the right data, and sends communications back in the right direction.
If it goes off its orbit, or points in the wrong direction and is unable to get back, its lifespan would be cut short, and a huge amount of resources would be wasted.
Propulsion is used to move a satellite through three dimensions. It enables satellites to get back to orbit if they get off. Propulsion can be achieved by having rockets or thrusters at different corners of a spacecraft for turning. Large spacecraft also use propulsion to control where they are pointing. Ideally, engineers want to achieve propulsion with the least possible amount of fuel, as the weight of the fuel at launch can make the endeavour far more expensive and complicated.
When talking about flight, attitude is the information about an object’s orientation and position on its axis in relation to the plane below it. These properties are pitch, roll, and yaw.For attitude control, most satellites use reaction wheels. Speeding up or slowing down a reaction wheel on a spacecraft, will cause the spacecraft to rotate in the opposite direction. By having at least three wheels, a satellite can precisely orient itself towards wherever it needs to be. You could hypothetically call this an attitude adjustment.
How Do Satellites Determine Their Attitude?
In order to keep pointing at Earth, satellites need to know how they are oriented relative to the Earth’s location at any particular moment. A spacecraft subsystem, the ADCS (Attitude Determination and Control System) serves the purpose of maintaining a satellite's optimal orientation. It consists of IMUs (Inertial Measurement Units) and a variety of other sensors working together to tell where a satellite is pointing in relation to the Earth’s location.
Some of these sensors include:
Horizon Trackers
Since the Earth is warmer than space, infrared cameras can see the cut off of the earth’s horizon in space. In this way, a satellite is able to figure out where the Earth is. The downside though is that due to the horizon’s huge size, it does not give a precise point as to where all satellites will be pointing, producing a more generalized target direction.
Sun Trackers
The sun can be tracked through sensors that look for the hottest thing in the sky. Unlike the Earth’s horizon, which may appear as a huge circle of horizon, the sun tends to be in a very particular spot in the sky that the satellite can point at. By tracking the sun, a satellite can determine how they are oriented in space. During the times in its orbit when the Earth is between the sun and the satellite, sun tracking cannot be used.
Star Trackers
Star tracking is the most accurate method that satellites use to determine their attitude and figure out the location of the Earth. In fact, even the early astronauts using sextants, an instrument used for measuring the positions of stars since the 18th century. Star trackers compare the position of very particular stars against a star catalogue in order to tell the direction in which a satellite is pointing.
Magnetometers
Satellites on Earth orbits can use the Earth’s magnetic field to orient themselves. A magnetometer which detects the Earth’s magnetic field can tell a satellite which direction is north, and with this, a satellite can determine its orientation.
Earth based Ground Stations
Broadcasted signals from ground stations on Earth can be used by satellites to figure out where the Earth is relative to their current position. A major limitation to this method is that the signals’ strength fades out the further the satellite travels, and many ground stations are needed to keep continuous coverage and not lose data.
Can GNSS Satellites Be Used to Orientate Low Orbit Satellites?
GNSS constellations live about 20,000 km above Earth, and hypothetically they can potentially help orient low orbit satellites, which are usually below 2000 Km. While it is possible, there are certain caveats that limit using GNSS satellites from orienting lower satellites. Firstly, GNSS satellites are broadcasting specifically to Earth, so it is possible to receive a signal from at least four of them at any point on Earth, and triangulate an accurate position. This is how we use GNSS for earth navigation systems, and GPS. As you move higher, the field of view gets narrower and satellites which are very high up may not receive signals from enough GNSS satellites to triangulate an accurate position. Additionally, GNSS predominantly gets a satellite’s position, but not its attitude or orientation.
Want to learn more about using GNSS on earth? Listen here to learn about Where Does the Blue Dot Come From?
Edge Computing
Tones of high-resolution imagery is being captured by earth observation satellites every day, and edge computing has an important role to play here. It is not necessarily feasible to transmit everything that a satellite captures, since some of the data may not be usable. For instance, running algorithms on captured images out in space will help to sort out the ones that have too much cloud in them, and only downlink the images that would be usable. Algorithms can also complete some light processing and correction of the images, so the data is already in a more usable state before being sent down to a ground station on earth.
Space and Geospatial
Compared to previous generations, we are certainly living in a golden age of both geospatial data, and the space industry. There are a lot of commercial players that are designing, building, and launching new Earth observation satellites; and many others who are figuring out new and innovative ways of processing that data and turning it into useful products for various applications. Earth observation satellites are surely becoming an important pillar for many geospatial applications.
Want to learn more about the growth of earth observation, remote sensing, and imagery? Listen to our podcast with Dr. Aliastair Graham
Recommended Podcast episodes
This is a story about bathymetric Lidar... and how geo-tagged sharks led to the discovery of a huge nature-based carbon sink in the Bahamas.
More on this case study here: https://r-evolution.com/r-initiatives/oceans
The Ocean of Things
https://www.darpa.mil/program/ocean-of-things
Recommended Podcast episodes
Mapping The Ocean Floor
Mapping Oceans With Sound And Mapping The Sound In The Oceans
PDAL -Point Data Abstraction Library
Whitebox Tools Is The Backend To Many Frontends
consider supporting this podcast on Patreon
https://www.patreon.com/MapScaping?
Or go to MapScaping.com to find out about sponsoring our website
reach out on Twitter https://twitter.com/MapScaping
or LinkedIn https://www.linkedin.com/in/danielodonohue/
Sentinel Hub – The Cloud API for Analysis Ready Satellite Data
Gregor Milcinski is the CEO and co-founder of Sentinel Hub and has worked in the geospatial field for about 20 years. He shares his knowledge of the Earth observation industry and gives us an in-depth explanation of what it is that the Sentinel Hub does.
Sentinel Hub is a cloud API for satellite imagery. It uses its APIs to enhance access to satellite data from missions such as Sentinel, Landsat, and other commercial satellite projects. Sentinel Hub users make use of its APIs to process the data they are interested in, and access it in a format that best provides the information they need.
This helps take off the operational load in terms of time and effort required to process the large, ever-changing volumes of satellite data. Sentinel Hub enables users to obtain Analysis Ready Data (ARD).
This means having data in a form that is ready for a particular workflow process and does not require additional cleaning. Sentinel Hub estimates that the capabilities they provide cover about 80% of what users may want to do as far as data processing is concerned.
Among others, these include Ortho-rectification, data transformations, rescaling, re-projection, as well as applying some machine learning models. Using a simple API request, users can get immediate access to full and global archives of the relevant satellite missions partnered in Sentinel Hub.
A Peek into How Sentinel Hub was Built The idea that eventually transformed into Sentinel Hub was born from operational difficulties in a previous project. This triggered the realization that the technology which was already available for working with satellite data was not overly suitable for processing the ever-expanding, continually updating, large volumes of data collected by satellites. A prototype was developed and subsequently iterated to evolve into the now Sentinel Hub.
This journey of growth is an inspiration to those who may want to build something, but are unsure on how they should start. Do not get stuck in the idea that you have to start big straightaway. Start where you are confident, and once you figure out what is working, keep building on it.
How Sentinel Hub Works Sentinel Hub works ‘on the fly’. It does not store pre-processed data that can be ordered off-the-shelf. Rather, the processing happens on the fly as requested by the user. This approach was chosen since it is quite impossible to accurately predict what a users’ needs will be in terms of geographical, temporal, and spectral aspects.
This is also one of the reasons why Sentinel Hub does not cache processed results (apart from privacy policies). Considering these aspects, there is a high risk of storing petabytes of data that may never be needed. Instead, Sentinel Hub worked to optimise their processing steps as much as possible for speed.
Currently, it only takes a couple of seconds for the most usual requests (like above) to process. This is typically fast enough that users can integrate these APIs directly in their applications in an interactive manner, without even having to store the data on their side.
Keeping operations to "on-the-fly" also allows Sentinel Hub to expose their new capabilities and improvements to their users by simply deploying a new version.
If data was pre-processed, there would be a fixed dataset which would need processing in order to introduce the new improvements. This could be expensive, time-consuming, and inconvenient.
How is Sentinel Hub Different? The description of what Sentinel Hub does may have you thinking that it is the same as Google Earth Engine, or Microsoft’s Planetary Computer. Ideally, it actually falls somewhere in between the two. See the sections below to see how they compare.
Sentinel Hub vs Google Earth Engine Google Earth Engine was designed as a search platform where people could do analyses, and then make use of the results. It was not primarily designed to power other applications.
Even though there are workarounds regarding this, Google Earth Engine was mainly designed for people to work with the results of its analyses. On the flip side, Sentinel Hub is designed to power other applications.
It offers a set of APIs which users can easily integrate into their procedures and workflows. It removes limitations on what users can do, and where they can do it. Building your workflow on top of Sentinel Hub, you can do just about everything, since it is possible to fine-tune what you'd like to get from Sentinel Hub APIs.
Sentinel Hub vs Microsoft’s Planetary Computer Microsoft’s Planetary Computer can be viewed as a platform that is mainly for making geospatial data available in a cloud-native way. It does not abstract the complexity of satellite data, which makes it difficult to perform tasks like stitching scenes. Microsoft’s Planetary Computer provides APIs to access the metadata of the satellite data for users’ workflows being run on virtual machines in Azure. This in itself is very limiting for users who may want to run it on a different cloud. On the other hand, Sentinel Hub allows users to be more flexible as they can use the program in their own environments.
Businesses that have their own infrastructures and proprietary data that they do not want to use outside their environments can simply use Sentinel Hub as a data stream in their procedures from wherever they are.
Who Are the Users of Sentinel Hub? The vast majority of Sentinel Hub users are application developers and data scientists. The application developers are mostly those working in the agricultural sector.
One reason for this is that Sentinel-2 provides a powerful collection of data with a resolution that is most useful for agriculture and vegetation monitoring. Typically, many of the Precision Agriculture applications are powered by Sentinel Hub.
The use of Sentinel Hub by data scientists leans towards agriculture primarily for monitoring activities and climate change analyses. Data scientists are the largest consumers of Sentinel Hub since they perform large scale operations using machine learning. Despite making up only 5% of the paid users of Sentinel Hub, data scientists consume about 80% of the volume of data.
Apart from the agricultural sector, Sentinel Hub also serves the energy, defence, environment, and mining sectors.
Partnering with Sentinel Hub Satellite companies can partner with Sentinel Hub through its “Bring Your Own Data” service. This service allows satellite companies to plug in their data to Sentinel Hub in any cloud native format supported by Sentinel Hub (i.e. GeoTIFF, XAR), and basically any kind of raster data. Sentinel Hub does not copy this data, but merely stores the metadata in their database for faster access, like a catalog.
Users can then make use of the same set of features offered by Sentinel Hub to distribute data efficiently. The satellite providers are responsible for charging users for using their data as Sentinel Hub does not support this functionality at the moment.
Sentinel Hub’s EO browser The EO browser is a Google Maps-type application that runs in a browser. It provides access to the satellite missions supported by Sentinel Hub, and is free for non-commercial use. In the EO browser, you can check for the most recent data in any part of the world and visualize it interactively (i.e. zooming in and out, viewing different band combinations, doing time lapses, switching between satellites), and much more. No account is needed to use these capabilities in the browser.
The EO browser showcases what Sentinel Hub’s APIs can do when integrated into workflows. It is a perfect display of how satellite data can be used. If you want to get familiar with or know more about satellite data, the EO browser is a useful resource for that.
An Overview of the Earth Observation (EO) Industry The usefulness of satellite imagery has grown over the years as experts explore more applications where the use of satellite data would improve efficiency, and insights. Coupled with developments in IT and data capture technologies, the Earth Observation industry has exposed new opportunities but also faces some setbacks.
Opportunities in the EO industry Spatial resolution in satellite imagery has improved in recent years and has unlocked new opportunities for using satellite data. Some satellite missions, such as Sentinel-2, have launched a revolution by providing very useful, good quality, free data set that can be used as the basis for a number of geospatial workflows.
Machine learning is also worth mentioning due to its contribution to the EO industry. It has revolutionized the interaction with the normally large volumes of satellite data, and processing it. There is still much to be done in terms of developing machine learning procedures for highly dynamic satellite data, and the wide variety of more niche uses that consumers crave.
Setbacks in the EO Industry The complexity in the nature of satellite data, coupled with the unique traits of geospatial applications introduces difficulties in using machine learning to fully realize opportunities that emerge in the EO industry. This uniqueness hinders the development of off-the-shelf machine learning algorithms. There are a very limited set of applications for which machine learning procedures can be offered as a service (i.e. ship or building detection). In most other cases, there would still be a need to tailor the procedure to a specific workflow or niche.
In an effort to contribute to the development of machine learning in geospatial applications, Sentinel Hub have open-sourced all their internally developed machine learning procedures, and hosted them on GitHub. They allow other people to refine these procedures and use them as a starting point for their own machine learning projects.
Making the decision on how to balance temporal and spatial resolutions during data capture is also a defining line on whether the missions will generate sufficient value.
For instance, applications in the construction industry may require high spatial resolution (i.e. 1 meter) as well as a high temporal resolution (i.e. 1 day) to monitor construction progress. For applications like vegetation monitoring where there will be no significant change within a day, using these exact resolutions will be overkill.
It is highly likely that the value generated may not be sufficient to compensate for the cost used in processing.
Another challenge is the fact that satellite data is only periodically produced. It is unlikely that a user can find what they want in their area of interest at the particular time that they are interested in. A satellite can only be tasked for the future, not the past.
Advancing the Earth Observation Industry Sharing knowledge has always been one of the pillars for new developments that can take any industry to the next level.
In the geospatial field, more and more companies are cultivating the culture of sharing knowledge. Sentinel Hub is already making strides in this by sharing a lot of what they do (i.e. openly sharing their machine learning procedures, which they could easily choose to make proprietary).
Sentinel Hub explained that they are doing this because they believe the Earth Observation field is in a phase where contributing to its growth is more important than growing one’s position within it.
The Future of the Earth Observation Industry Growth is expected in the Earth Observation industry for the foreseeable future. With the introduction of the monitoring approach in almost all elements of our lives, satellite data will be needed since it is a rich, objective, and useful primary source.
The industry will grow since this data will be needed in a form that makes it easier to extract information, and be processed it in a smart way. Sentinel Hub acknowledges that they are only doing a tiny part in the industry. It will need effort from the geospatial community to grow all the tiny parts and realize growth across the entire industry.
Recommended Podcast Episodes
Our guest today is Jakub Dziwisz, the CEO and founder of Orbify, a web-based SaaS solution for earth observation applications. Although he does not come from a GIS background, Jakub has always enjoyed problem-solving through technology. Originally, this consisted of being a software engineer focused on expanding people’s horizons and perspectives through the travel industry. Now, Orbify helps expand the geospatial capabilities of everyday people, solving not-so everyday problems.
Why Merge GIS and SaaS? First of all, what is Software as a Service (SaaS)? Well,
SaaS products are traditionally web-based software that greatly reduce the amount of time it takes for customers to get up and running with their own products by utilizing the SaaS product’s infrastructure.
Generally, these products follow a ‘pay only for what you use’ format, very similar to how some cell phone providers offer custom mobile data plans.
SaaS products can be great for GIS because they leverage the SaaS provider’s hardware and software resources, allowing customer applications to live and run in the cloud. This lowers the barrier to entry in terms of resources for creating an application. Most SaaS products allow users to select their application’s function or design from a series of preset templates, or DIY in some other modular way. This also lowers the barrier by reducing the amount of time and expertise necessary to get started on a project.
Another huge time saver with SaaS products is that, due to being web-based, there is no installation needed. Organization administrators can configure all their user settings on the backend, and easily regulate access to their application. This makes it a very popular platform with large companies who have a huge and constantly changing employment pool.
Although the pay-as-you-go method has been popularized in cloud storage, and to a certain degree in the ArcGIS platform for some analysis using credits, SaaS providers that are creating infrastructure for GIS workflows are few and far between.
The Need for SaaS in EOS: An Analogy In the 1920s, Henry Ford introduced the world to the raw efficiency of the assembly line. Before this, the process of building a car was slow, and expensive. By introducing the assembly line, cars could now be built in a streamlined fashion which was faster and easier, ultimately making vehicles accessible to a much wider market.
In the 1990s, it was customary to go to a travel agency if you needed to go on a trip. The travel agent would list for you the often limited available flights and fares, and you could take it or leave it. With the introduction of budget travel services like Expedia and Hotwire, consumers were provided with options to modify their plans to their needs. This resulted in a more competitive airline market, and democratized the space by putting the consumer in control of what they consumed.
Finally, in 2007, Steve Jobs unveiled the iPhone. People had been familiar with the telephone, even the cell phone, for decades. Similarly, they had been familiar with MP3 players for years. The idea, however, of being able to have both of these things in one changed how people judged the capabilities of technology.
For decades, GIS users and earth scientists have been working across dozens of applications, tools, script, and data sources.
By bringing SaaS to this niche, geospatial data workflows can become faster, easier, and more accessible. As it succeeds and more players enter the space, consumers will find they have more competitive options, and that their needs and dollars are in control of what happens next, and they shouldn’t be afraid to ask for what they want.
Creating a SaaS User Experience The best SaaS products are intuitive, and feel almost effortless to navigate. This is in stark contrast to the alternatives- environment-specific scripts, custom desktop installations, and coordinating and managing server resources. Planning and designing a great SaaS product is heavily dependent on knowing what your clients need, and what their customers want.
In order to build an app, there are four main elements, data, functionality, the output, and the packaging/user interface.
Normally, it takes quite a lot of time, code, and infrastructure to build an application like this. It often even requires a full team of developers. Using SaaS software, something like this can be configured by only one developer, and relatively quickly.
Let’s take a look at an example, using Orbify as our SaaS platform.
Orbify was developed with low tech skilled end users in mind, who need to solve a spatial question on a regular basis.
The Orbify platform is a bit complicated for someone with little technology experience to configure, so the most likely situation is that a developer has been contracted to build this application. Our end user in this case is an ecology graduate student who needs an application to monitor shipping vessel activity to better understand how this overlaps with known whale migration paths.
A user-friendly experience is key to the speed here. This starts with the data aspect. An interface like Orbify comes packaged with access to a catalog of existing cloud data that the developer can connect to and use in their application.
As the source code is completely customizable, they have the flexibility to connect to their own data as well.
For our scenario, the developer is able to connect to hosted AIS ship tracking data, and import their own data for their whale migration paths.
The functionality needed for our application is to visualize the ship and whale data in the context of time. This is easily accomplished in Orbify by selecting the ship tracking application template, and enabling the time slider. The output for this project is a map, which is accomplished through the template. Finally, the user interface will largely be directed by the template, but the developer can select from a number of existing no-code buttons and tools to include in the interface, and customize the look of the application as desired.
In the end, our graduate student has a simple, custom application that was created with minimal man-hours, and will only be billed for what they use.
This scenario can be extended to all kinds of potential customers for whom the traditional custom application would be too expensive, farmers, foresters, firefighters, etc.
By reducing the barriers to entry for using custom GIS applications, more people can be exposed to the wonders of geospatial, and we can collectively work towards solving problems that weigh on everyone from the individual, to the global organization.
Joining the podcast today is Eric Jensen, currently a data scientist with Climate Engine. After obtaining his BA in Geography, he completed a masters in Ecology adding context to his research with remote sensing through the University of Montana. Eric has gained a variety of experiences during his career, ranging from working with web tools, to conservation and research work, to a data analysis focus, and even to creating public-facing geospatial educational media. He is going to share his experience and expertise with approaching the geospatial job market, and finding what you’re looking for. Making a Job Transition Making a career move can be scary, whether it is within your current company, or trying somewhere, or something new. The first step in making a transition is deciding if now is the right time, and if not now, then what would the right time look like? What can you do to move up that timeline? Here are some things to consider.
Finding your dream job starts with identifying what your dream job is, and who is hiring for it.
A company’s focus, industry, and culture have a huge influence on what the day to day of that role may look like, and if it’s a good fit.
This includes considering who your coworkers and clients might be, and if you would be compatible. If your preference is for the research and innovation associated with academia, you might like a company that employs a lot of scientists. If you like ambitious colleagues with a range of experiences, a startup might be a good choice.
Identifying the specific job you want is going to rely on your experience, interest, and the perceived opportunities.
GIS is often described as a tool, and many people find greater success using GIS as a secondary skill to support whatever their expertise in their main industry is.
If you are not sure exactly what niche to climb into, start asking yourself some deep questions about where you could see yourself going, or better yet, have someone else ask you these questions and garner their feedback. Spend some time reflecting in order to identify exactly what your next move looks like.
An essential, yet intimidating element of changing jobs is discussing that move with your boss or supervisor. Some bosses aim for stability, and may not react as well to the idea of a change, but some supervisors might surprise you and end up being an excellent resource in your job search.
A boss that can act as a reference is immensely valuable, as they can provide your next boss with specific examples of your skills and what you bring to the table. If you have a boss that is supportive of your growth, start that conversation with them, it might be easier than you think. Tools for Success in Finding a Geospatial Job Once you have decided it is time to make a change, it’s time to get actionable. Entering an iterative process of research and self-reflection can help you to uncover some truths about where your strengths, weaknesses, and interests lie.
The most productive tools for learning more about career growth in GIS are informational interviews.
An informational interview is an informal conversation with someone you find to be a role model in your industry.
These low-stakes conversations allow you to source honest answers on questions you have about nearly anything. You may discuss questions surrounding salary, curating a social media presence, or get some insight on how they approach tough interviews.
Preparing for an informational interview first requires finding someone to interview. You might not have to look far, however, as your existing network might already hold some great options.
As a general rule, most people enjoy talking about themselves, so do not be afraid to reach out to request interviews.
In order to prepare questions, try checking out your interviewee’s LinkedIn, or cruise through their company’s website. Use the list of questions you build to help guide the conversation. As a natural part of the conversation, you may find a lot of these questions reflected back at you. It is okay to not have answers for all of them, especially if you are still relatively young in your career. Take this opportunity to notice gaps in your knowledge, both in industry topics, and in knowledge of your own professional priorities and interests.
After concluding an informational interview, reflect on what you have discussed.
Did you get what you came for? Distill the main takeaways from your conversation into a few sentences and write them down somewhere for later to jog your memory if needed.
Another great resource for building insight on how people in GIS approach their careers is by listening to podcasts like this one! The MapScaping Podcast hosts a curated archive of years of conversations with industry professionals, and we even allow you to filter for Career Focused episodes. GIS podcasts like MapScaping, Seeing From Above, Minds Behind Maps, and Geodorable can provide a lot of the same value as conducting your own informational interview, but without all the effort. How and Where to Look For a GIS Job After identifying your niche in GIS, start identifying companies that need it filled. You might start by looking within your existing network, or by following companies you have interest in on LinkedIn or Twitter.
Plan to take a look at GIS job boards which may allow you to get even more niche.
ClimateBase, for example, posts jobs specifically surrounding the climate industry. Similarly, if you are interested in a company, follow their public job board postings to be notified as soon as new positions become available.
Before applying for your potential new job, spend some time refreshing your portfolio, resume, and CV. HR softwares are notorious for filtering out applications based on the presence of keywords, so take the extra time to tailor your resume to the job description.
Another popular addition to a CV or resume is including a link to your portfolio website (use a PDF, never a Word doc!).
When a decision-maker pulls up your site, you gain immensely valuable ‘facetime’ with them, and a chance to impress with your well-curated portfolio.
Of course, in business, not everything is entirely based on merit. While it is certainly possible to get a job through the traditional application process, anytime you can expand on a personal connection you have with the hiring company it will help your chances of making it to that final interview.
An added benefit of job-seeking within your network is it gives you a chance to gain some background and insight on your potential future company.
You can ask questions about the culture and growth opportunities that might not feel as natural when you reach the formal interviews.
GIS is an ever-expanding industry, revealing potential new applications through an iterative process of innovation. By investing some time into reflecting on how you fit into the larger schema of the field, you can be more confident that you are making the best choice for your future as you pick your next move.
Our guest today is Ron Hagensieker Ph.D., the founder, and CEO of OSIR.IO, an artificial intelligence and earth observation company. Ron has had an interest in remote sensing in the beginning, starting with his BA in geography, all the way to his PhD in remote sensing. All along the way, he has found ways to incorporate machine learning. One project, in particular, was thiscitydoesnotexist.com, a site that uses AI to randomly generate a false Landsat-style image of, you guessed it, a city that does not actually exist. How is Fake Aerial Imagery Created? In order to create fake imagery, first, you must obtain a great deal of real imagery. These images will be used to train your neural network to recognize the unique landscape characteristics that make a city a city. The machine learning models learn to identify and recreate the patterns we associate with small-scale views of cities (i.e Landsat or Sentinel imagery) mimicking the transitions from city center, to suburbs, to farmlands. They can even be trained to create fake DEMs.
ThisCityDoesNotExist.com’s algorithm actually involves creating two competing machine learning algorithms, the generator, and the discriminator.
The generator algorithm generates images, attempting to fool the discriminator model which dubs each image as a ‘real’ or ‘fake’ city.
Based on the responses of the discriminator model, the generator will adapt its next image, and try again. In programming, this technique is called a Generative adversarial network (GAN). GANs are especially powerful algorithms, and can even be pit against other algorithms to expose their weaknesses.
The positive feedback loop the generation and discrimination process creates can hypothetically go on forever, but will ultimately be limited by computer resources, or be influenced by the finite number of images originally used to train the model.
These algorithms use unsupervised classification, which is when the computer groups like pixels into a prescribed number of classes.
Unsupervised classification distinctly does not require drawing bounding boxes around features, and therefore may be faster to prepare than supervised methods.
It is possible to add some influence to the process with supervised classification. This involves inputting a dataset of city center points, telling the model “here, these are cities”, potentially improving results. Why Create Fake Imagery? In a data rich world with so much access to all varieties of raster data, why would we want to add fake imagery into the mix? Well, the truth is we aren’t quite sure yet. When plenty of real data exists, there is little desire to inject energy into simulating data. While there are no especially compelling uses for large amounts of fake imagery, doctored imagery is a different story.
There may be military applications for faking parts of images, but not necessarily whole landscapes. This could be identifying and disguising sensitive military operations in satellite imagery, or maybe even simulating the result of damage to landscape elements.
Randomly generated AI landscapes don’t have much use on their own, but once you add more channels to the data (Elevation models, population density, etc), and start adding constraints, things can get really interesting.
The trick here is the channels must be coregistered to be meaningful. Coregistering means that the images have all been lined up with each other spatially.
By combining information from many channels, you can create a living modeling environment. It gets even more exciting if you can pull off adding time. This allows predictive simulation of urban planning scenarios, deforestation patterns, and even of events resulting from climate change. By simulating different population counts and densities, we can begin to identify common patterns in how people grow into their cities.
Fun Fact: If you drop oat flakes into slime mold at the positions of train stations in Tokyo, the resulting fungal network grows to look just like the actual Tokyo Rail System.
The reason landscape modeling does not qualify as a compelling application of fake imagery is that it is incredibly costly in terms of computing resources. Without adequate financial backing, it does not make sense to create a program like this. For now, coders are just having fun without funding. What Are the Risks of Fake Satellite Imagery? The most obvious risk associated with creation of imagery from machine learning models is the risk of people using this technology maliciously. Deep faking images may allow you to represent extreme scenarios on a landscape. One popular one is predicting the same landscape in all four seasons, but you could also simulate flooding, or wildfires.
Luckily there is not much genuine risk of misinformation (amongst free media at least) about events as these things are so easily verifiable.
If someone actually takes the time to simulate media coverage of a catastrophic event somewhere using deep-faked images, fact-checkers can verify from half a dozen public imagery sources if the event actually occurred (or even simply just call someone there).
So if you come across a fake image in the wild, how would you be able to tell it was fake? Well, one characteristic is that fake imagery often looks downscaled. Depending on the base logic in the algorithm used, sometimes you can pick out a fake image by the lack of truly straight road segments. This also applies to the lack of natural angles on building corners, or the presence of especially blurry building footprints.
Characteristics like haze or odd colors in areas are not reliable tells of fake imagery as these elements are normal in most real imagery as well due to atmospheric effects.
Overall, fake imagery has the potential to be very powerful, but in the current technical landscape, there is not much need for it. Due to the high skill, time, hardware, and financial investment necessary, it is unlikely we will see too much development growth in this area. The one thing we have learned from artificial intelligence, however, is to expect surprises.
Our guest on the show today is Chris Holmes, the Vice President of Product and Strategy at Planet. Chris entered the geospatial arena almost 20 years ago as an early contributor to the GeoServer project. In the beginning, he was writing code, but eventually discovered his talents were better spent helping to build a community and awareness around GeoServer and the greater Open Geospatial Consortium (OGC). Chasing innovation, he now works to promote wider spread adoption of cloud native geospatial solutions such as the SpatialTemporal Asset Catalog (STAC), and use of cloud optimized geotiffs (COGs) from his position at Planet.
The Basics of Cloud Native Geospatial As one of the newest technologies on the GIS scene, cloud native geospatial can seem a bit intimidating. Realistically, it is a lot of familiar industry staples repackaged to take advantage of huge advancements in computing technology.
At its core, cloud native geospatial (CNG) is your classic geospatial infrastructure, without all those pesky computational power and storage limitations.
By leveraging the power of AWS, BigQuery, Snowflake, Google Earth Engine, or ArcGIS Server, users can access and analyze global-scale datasets without needing to purchase and maintain the physical servers traditionally used in on-premise setups.
This shift reduces the barrier to entry for scientists, and even casual users, allowing more spatial questions to be asked and answered.
CNG systems are flexible and scalable to most needs. One can choose to host some, or all of their data in the cloud, then access it for local analysis, complete that analysis in the cloud, or take a hybrid approach.
GIS work has traditionally followed a desktop-centered workflow. Using cloud native geospatial, it does not matter if you access and analyze your cloud data through a browser, or desktop application, although each platform will come with its own natural limitations. GIS Data Formats and the Cloud Cloud technology alone has existed for a while, but took a bit of a learning curve to adopt into GIS due to the specific requirements and preferences of cloud infrastructure.
This means that some people took this rare opportunity to essentially start from scratch, and build data formats that are optimized specifically for cloud systems, but often still maintain the flexibility to be backwards compatible to desktop and enterprise workflows.
It is not possible to talk about CNG without talking about Cloud Optimized GeoTIFFs (COGs). COGs are the backbone of cloud native geospatial, and are essentially responsible for starting the GIS industry’s race to the cloud. The beauty of COGs is that they can be used as a regular GeoTIFF in a desktop setting, or leveraged in the cloud to unlock fantastic real-time data streaming and analysis.
The key element of COGs is their compatibility with range requests. The efficiency of range requests is what enables streaming in a lot of our favorite applications, like Spotify, and even Netflix or YouTube. A range request is when a client reaches out to a server for information in its HTTP header to first know if the server supports range requests.
If range requests are supported, this means that the client can query what is essentially a table of contents for the data to retrieve only the data that the client is interested in, then stream it back for use.
If you are familiar with Python, you can think of this almost like slicing a list [x:y]. By pulling only the data relevant to the query, performance is greatly increased as fewer packets can be transferred back to the client.
At this time, streaming optimized geospatial formats are mostly limited to raster and point cloud data. There are nevertheless hopes that we will see options for optimized vector formats in the not too distant future. Open vs Closed Geospatial Data Standards In the past, GIS was often more or less siloed within an organization, making it reasonable to work in closed or proprietary formats. Today, in an increasingly interconnected world, sharing data and preparing it with interoperability in mind is essential.
Open data standards have been embraced by many as they can be used as-is, or maybe modified to work with existing infrastructure to create a better fit for an organization’s needs.
The Open Geospatial Consortium in particular has been instrumental in expanding the prevalence of open formats by publishing and documenting open data standards.
Having this groundwork in place gives developers a good spot to start from when creating a custom implementation, and leads to greater potential for innovation.
Although open data standards and formats have come to dominate the industry, closed data standards still have a presence. The best example is Google Earth Engine. Within its system, Google Earth Engine ultimately handles data in a closed proprietary format.
They know, however, that consumers today are generally unwilling to accept the risks of holding all of their data in a closed format.
After all, if the provider went out of business, then the consumer would lose usability of their data. Google remedies this by allowing other data formats to essentially port into their own, allowing clients the flexibility and security they would get if they utilized open standard formats. What's Next for Cloud Native Geospatial? CNG has already brought a huge paradigm shift to the industry, removing traditional access, storage, and processing barriers for those who enter the game. As more and more datasets are uploaded into the cloud, it begs the question of what the next great leap forward will be.
One potential development we may hope to see in the future is data that is optimized for retrieval by search engines. At this point, the data exists, but it needs to be described in a way that allows the search engines to find it, and match it to user needs. This means fully populated metadata, and plain text descriptions that allow data to be matched to a query.
More accessible and queryable geospatial datasets could play a huge role in bringing geospatial to the masses, rather than following the current theme of only being used by those involved in the GIS industry. The SpatioTemporal Asset Catalog is a great example of what this may look like.
Another glimpse into the future of CNG is that eventually, pretty much all the data we could want will be available in the cloud, in open data standards. This can allow the focus to shift from data aggregation, to creative data analysis and applications. As young people come into the industry, they will not be clouded with the ideas of what cannot be done, but will rather see the wealth of options and resources available, and take it and run with it, hopefully leading to the next big thing.
Daniel:
Hi Michael. Welcome to the podcast. Thank you so much for taking the time to join me today. I really appreciate it. So in just a second, we're going to talk about open-source geospatial journalism, and we'll dive into exactly what that means in just a second for the listeners. Before we get there, can you just take the time to introduce yourself, and perhaps explain how you got involved in geospatial and journalism?
Michael Cruickshank:
I'm Mike Cruickshank, a journalist, originally from Australia, and I actually got into doing geospatial through journalism, which I suppose is a strange entry point to geospatial. This happened because I got involved in this kind of journalism called open-source intelligence. This uses a lot of geospatial data for cross-referencing things by using techniques such as geolocation.
This first brought me in contact with satellite imagery through Google Earth and that sort of thing. I just became more and more interested in what this data is, how can I use it to do more than just have something to look at? What stories are there that kind of lie beneath the pixels, so to speak, of the data and how can I use it in more interesting ways? From there I taught myself how to program, and work more deeply with this data.
I've learned how to use all these different geospatial tools and programs. I'm also doing academic research now, I'm doing a master's thesis where I'm using this kind of data to investigate links between climate change and conflict. I’m also working professionally at a company in Berlin called LiveEO, where we use geospatial data to provide information for industries on things like environmental issues or forestry, as well as monitoring industrial assets. It's quite a broad range of things that I'm involved in these days.
The Link Between Journalism and GISDaniel:
It sounds like you're moving away from journalism and into geospatial. What kind of skills can you already see that are going to be really helpful in that move, that you'll take with you from journalism and over to the geospatial side?
Michael:
I wouldn't so much say I'm moving away from journalism as I am trying to tell a different kind of story to a different kind of person. In the past, my thinking was all about how do I inform people about what is going on? I came to the conclusion that, while I still want to inform as many people as much as I can about what is going on, I also want to be able to reach more of the kind of people who are actually making the decisions in the world, and provide them with information on how things will go into the future.
I'm particularly interested in looking at the effects of climate change and its security implications. I want to be able to provide insights on this prediction, analysis, etc., using geospatial, as well as some of my background in journalism on thinking about the real effects of these things. I want to present this kind of information in a way that's engaging to people so they actually take it seriously, and just reach in a more targeted way the kind of people who can actually make a change in this world.]
Daniel:
You had this great sentence there, you said something like, "I want to tell different stories to a different kind of people." What was it about traditional journalism that you felt like wasn't working, or could be done better?
Michael:
I felt that in many ways, people were flooded by the same story. I was working in conflict journalism so perhaps you could go to a country, you can go to the front line of their wars, you can tell a story of, oh, it's very awful. It definitely is awful what is going on in these places, but the unfortunate matter of fact, is that so many people are telling these kind of stories, people just tend to switch off.
You need to find a different way of reaching them with a different kind of story, one that engages them in a different kind of way, and perhaps tells a bigger picture or a different picture.
Daniel:
I think you're absolutely right with that. I think that's an absolutely brilliant insight. I guess that the danger is when we think about engagement, we sort of fast forward this trend towards the short, incredibly clickable link bait. I really do feel like we are a world trending shallow, as opposed to deep. We see this in all the social media trends, generally. It's quick, short snippets. Do you feel like that's the way to go? If we think about engagement, the numbers speak for themselves. That seems to be what people want, what people engage with. When I think about journalism, I kind of expect something deeper.
Michael:
You're well within your rights to. I certainly think that going after the kind of engagement that you are talking about is a bad thing for publications and for journalists to do. I suppose when I talk about engagement, I don't mean someone just reading something, clicking it, sharing it, liking it, reacting to it. What I mean is someone who's really taking stock of what is written, what is seen, etc., and thinking about it and incorporating that knowledge into their life and their decision-making.
Daniel:
So I have to ask here, we are overwhelmed with media, we are flooded with information and data. How do you think your media, your style, this idea that you are going after here is going to cut through the noise?
Michael:
I'm not going to put myself up on some pedestal and say I could solve this problem on my own. What I think is it forms part of a body of information which is more verifiable, and forms more of a ground truth, or a reality of what has happened or what is going to happen, rather than just 1,000 different conflicting reports from different directions within the political sphere or different interested groups, etc. I think that part of the way that we combat this flood of information is simply providing people with better information, rather than just increasing the amount of it or telling them to consume less. Instead, we should suggest and provide the option for them to just consume better information.
The Tools of the TradeDaniel:
Yeah. I totally agree. I feel like that volume game that you're talking about inevitably leads to a lesser quality of information, or lesser quality of work because it's a volume game, right? It's about getting the amount out there as much as possible.
I know you've come a long way in your journey with geospatial, I know that you program in a couple of different languages now. Could you explain to me what it was like right at the start? Were you using tools like Google Earth just to look at the images and see what was on them and try and geolocate stuff there, or were you using something completely different than that?
Michael:
No, I was using Google Earth or Google Maps, those sorts of platforms, and just purely analyzing things on a visual level. There was no sort of looking at the data behind it, it was just all about identifying things that appeared in pictures, using video locations. I was using platforms like Google Earth, Wikimapia, and a few others to get different kinds of images or different dates of images. Beyond that, I hadn't gone particularly deep into it.
Daniel:
Okay, so that's where you came from. You started off using these very simple visualization tools like Google Maps, I believe you said. Where are you now? What tools are you using and where do you get your data from when you think about doing open-source geospatial journalistic work?
Michael:
These days I'm using a large number of open data sources, but I think the primary and most useful one is Google Earth Engine, which of course is very different from Google Earth itself. Google Earth Engine, as some of the listeners might be aware, is a multi-petabyte catalog of huge amounts of open-source satellite imagery from sources like the European Copernicus programme, Sentinel satellites, NASA's Landsat program, and many others. The beauty of Earth Engine is that it allows people to basically program with the data. You don't have to download these very large raster images and all that to your computer, you can work with it all remotely. The processing is done off your computer. So this is quite powerful because it means that you don't have to mess around with huge amounts of files and huge amounts of processing locally.
Daniel:
So maybe I made a mistake. I've been saying open-source geospatial here, Google Earth Engine is not open-source, even though some of the data might be. Are there any other tools you've been using, or have I made a mistake on my end?
Michael:
By open-source, in this case, I was referring to the data itself. Earth Engine is certainly not open-source, and you can only use it for non-commercial applications, as far as I'm aware at least. In terms of fully open-source programs, I use QGIS as well, so when I'm doing more visual-related output, more than processing, more than programming, more than analytical stuff, when I'm doing visual, I will use QGIS. For the more programming side of things, I'll usually use Google Earth Engine.
Daniel:
So let's stay with Google Earth Engine for a second here. What are you doing? I know you can make these amazing time lapses in Google Earth Engine. What kind of analysis are you doing in there?
Michael:
The kind of analysis that I'm doing, really depends on the use case. Basically, you can do any kind of transformation of the data. If you imagine these satellite images are effectively raster arrays, or just data points, you can do any kind of mathematical operations on the data. You can compare bands or images over time. You can create custom visualizations, and programs that just access this data and then work with it in other ways. You can create front-end and backend applications of this data. Recently, for instance, I wrote a small web program that enabled users to work with Sentinel-1 imagery of some of these Russian military bases where they were building up troops before they invaded Ukraine.
This was taking data from Earth Engine, specifically from the Sentinel-1 satellites, and then creating time lapses out and creating different kinds of visualizations out of this that were presented in such a way that a user could do this without any background knowledge of how to actually interpret the data. Often, especially with synthetic aperture radar (SAR), it's very difficult to get down to what you're actually looking at unless you are experienced with working with that kind of data.
Using Geospatial Intelligence Techniques for JournalismDaniel:
When we say Sentinel-1, are we talking about the SAR band?
Michael:
So Sentinel-1 is actually two satellites. One of them is currently non-operational. I'm not sure whether it's dead or just not working currently. So it's two satellites and they're both synthetic aperture radar, SAR, satellites.
Daniel:
What were you doing with this data? What kind of analysis could you do on SAR data that would give you information about what's happening with the Russian military in this case?
Michael:
The resolution of Sentinel-1 is quite low, it's approximately 10 meters per pixel. Obviously, you can't see something like a tank with this; a tank is smaller than a 10 meter square. If you know the areas where the tanks are being stored, at these bases, you have the polygons of the base. When more vehicles move into the base, the mean reflectance in the synthetic aperture radar increases. You can plot the mean of a time to get an idea of whether the number of vehicles in the base is increasing or decreasing. Moreover, if you have high resolution optical imagery, you can get a baseline of what a certain level of mean reflectance represents in terms of actual real numbers of vehicles.
Daniel:
How do you ground truth, something like that? Or can you use any other data sources to get an idea of perhaps a more precise number of how many tanks, in this case, that there are in an area? Can you incorporate other data sources into this kind of analysis?
Michael:
Absolutely. In my case, I have been using very high-resolution optical data as a ground truth that was taken on the same day. Obviously, this is not always possible because of clouds, especially in the more recent months in winter in Europe, it's very cloudy. You have to kind of wait until you catch a break. This usually can be done with a very high-resolution optical image.
Daniel:
How do you document this, and what are the results of this kind of journalism? Has it been cited anywhere? Are people using it as evidence anywhere? I guess what I'm looking for is- are people referring back to this, almost like a peer review?
Michael:
Looking at these Russian military bases, this was getting a lot of play in the media. I wasn't the only one doing it, so obviously I'm not going to take all of the credit for this. There were a large community of people within the open-source intelligence community who were sharing this kind of analysis. Many of the initial stories about this buildup were coming from within this community, and then were taken up later by the more mainstream media who were then tasking even higher resolution satellite imagery for their stories.
This really did get picked up in the lead up to the invasion. It helped in many ways to lend credence to what countries like the United States were saying when they were saying, "There's definitely going to be an invasion. This is how it's going to happen. They have all these troops here." The advantage of having people doing this kind of open-source intelligence journalism, using things like geospatial data is that we can then have a second data point and say, "Well, they're saying this, but now we can check if there really is data to back this up." In this case, there certainly was, and we've seen what's happened since then.
Using Social Media as a Data SourceDaniel:
Twitter is my social media drug of choice. At the moment, my account is flooded with images and short video clips of tanks and military operations in the Ukraine. At the same time, I'm constantly wondering, is this real, this thing that I'm looking at? Does this make sense? Is it misinformation? Is it disinformation? Can I believe this? Maybe before we dive into this as a topic, perhaps you could explain the difference between misinformation and disinformation.
Michael:
I'll start with misinformation. Misinformation is when someone shares a piece of information that is untrue, however, they don't necessarily know that that information is untrue. More often than not, they will probably believe that it is true. Disinformation is the case when someone is maliciously and with intent sharing something that they know to be untrue. The differentiation lies in whether the person sharing knows whether what they're sharing is true or not.
Daniel:
So this has been a huge problem, obviously, not just in the current situation we are in now with Russia and Ukraine, but also in other political theaters around the world. Is there any opportunity here to sort of prove or disprove some of this kind of information that we are seeing in social media feeds, for example, by doing the kind of journalistic work that you are doing? Or can we use this data together with what you are doing?
Michael:
Absolutely. Using open-source intelligence is one of the best ways that we can actually prove or disprove whether something is indeed misinformation. The primary tool is geolocation, where you can compare elements within, say, a video or an image to satellite imagery to work out whether it really is where it is said to have happened. This often makes it very easy to filter out if something is misinformation or disinformation. You know something is misrepresented, because you can immediately see that, okay, this is indeed where it's said to have happened, which then lends credence to the idea that that is indeed true.
You can take geolocation further and move on to chronolocation, where you start using things like elements of the shadows within the videos and backing this up with geospatial data on the exact position to calculate the exact time that a video or an image was taken. This further leads you down the path of whether this is true, or something which is misrepresented.
Daniel:
I can definitely see the power of geolocation here, but how do you do that? Are you looking for a place name in the video? Are you looking at some sort of metadata in the video, perhaps the geolocation tag, that kind of thing? Or are you doing something completely different to locate that image or that video?
Michael:
If there is a name, place name, shop name, or a business name within the video, this obviously makes it very easy. More often than not, that's not present, or the video has such low quality that it's difficult to actually read what's written. The more common way this is done is by cross-referencing visual elements within the video to visual elements within satellite imagery. This would be things like the architecture of the buildings, the specific layout of streets, of plants, and that kind of thing. (Much like Geoguessr). You then think about what that same scene would look like from above, and then cross-reference that scene with satellite imagery of the location where the video or image is said to have taken place. You then see if you can work out exactly where this happened. More often than not, you can work out not just where, but what angle the camera was pointing in, what time of the day, what time of the year, etc..
Daniel:
This reminds me of trying to orientate myself to a map. Looking at a map, looking for features that I can identify on a map, and then looking up in the real world and saying, okay, where are those features relative to me? Can I use them to orientate myself if I'm out hiking, for example? So this makes sense to me, but we live in a time of deep fakes. You will have seen these videos on the internet before. Is it possible to fake a location, when we think about the kind of information we can glean from social media?
Michael:
It is possible to fake a location, or at least fake a plausible looking location. Geolocation is possible 99% of the time in any video. If you can't find where somewhere is, if you can't get a fix on it, you have a pretty strong indication that this may indeed be a fake location. I haven't specifically heard any reports of videos that have been shared from things like conflict zones where the background has been completely faked using these kind of deep fake techniques, but it's certainly possible.
There's a website called This City Does Not Exist that sort of generates these deep fake landscapes, and there's no reason someone couldn't do something like this. It would take a lot of effort, and looking at some of the misinformation or disinformation that's being spread around, most recently in the conflict between Russia and Ukraine, most of it is pretty low effort. This sort of stuff takes a lot more effort and I have yet to see it being used, especially at scale.
Doing the Analytics and ResearchDaniel:
I'm imagining that the power of this kind of journalism is when you can start joining the dots. You see something happening over here, and you can connect it to an event somewhere else, or you can see that things are moving. How do you keep track of these events? How do you know for example that, oh, okay, I'm looking at tanks here now. I've used my SAR data. I can see that the values are changing over time, so there's change over time. I've confirmed this one location using social media, and georeference some of the videos or images that I've seen there.This is sort of giving me a picture in my mind and understanding of what's happening. Tomorrow, the tanks are gone, things have moved. How do you link that with other events, or how do you find where they've gone?
Michael:
I approach it kind of like a scientific, or a proof statement almost. You're trying to prove that something happened or didn't happen, or trying to work out what the limits of what you can know are. You try to create a chain of evidence going from the open data that you have, and then see how those things link together, how you bring this all together, and how each of these things mutually reinforce, or weaken the argument of the other. In terms of your question you asked, how would you know where the tanks have gone to? Well, maybe you've seen the mean SAR values at this base decrease, then maybe a day or two later, you start seeing videos pop up on TikTok of large columns of military vehicles moving around another town, somewhere else in the same region. So you think, okay, well maybe the vehicles are moving towards that area, but you don't know where they're going.
In my case, recently, I created a map of an entire region, Oblast in Russia, just doing basically change detection in SAR and seeing which areas specifically had had large increases within the last week or two. Obviously you get a lot of false positives; you get things like snow melting, lakes melting, or different landscape changes. Some of these places, however, you might see a very large area of change; a polygon that looks perhaps somewhat artificial. Then you can sort of cross-reference this with other data and think, okay, well maybe there is something there. Then you can download very high-resolution satellite imagery, optical imagery of this area to try to cross reference and determine, is this really something which is interesting or suspicious?
Daniel:
I've got to tell you that this idea of using geospatial in this way for journalistic research is kind of new to me. It makes perfect sense though. It seems like a great tool for this kind of application. Is this different from what you were doing when you were working as a journalist? Is this completely different from the kind of work that journalists would generally be doing?
Michael:
I think in some ways, it is very different from traditional journalism. Traditional journalism is all about building human sources, and your strength of your argument, or of the story that you're writing. It is almost based on your level of access that you have to reputable or trusted or well-positioned human sources. That is the skill; it's a networking skill in many ways, it's being in the right place at the right time, etc..This kind of journalism is about pulling together a whole lot of things that everyone knows, or everyone could know, and then seeing how those things all work together to prove something and drawing connections that other people aren't.
This is unlike traditional journalism, where it's all based on trusting that this is a reputable publication that wouldn't lie to you. You've got to trust that this anonymous source is really a real person, etc.. The opposite is true with open-source intelligence. Everything is effectively a proof statement. You don't need to trust the journalist. You can read through the way they've brought together their argument and their evidence and see if you agree with it. There's nothing that's hidden. Everything is in the open.
Daniel:
For me, this begs the question, why aren't big publishing, media, journalistic companies doing this? Or are they doing this? Do they have their open-source and intelligence center within their magazine or their media company?
Michael:
Over the last few years, there have been more and more mainstream media outlets that have been starting to use these techniques. It is growing in popularity, but it's also a relatively new thing. This really only started probably in 2012, 2013. That was when the ball started rolling. It has just grown more in importance over time, and in popularity and sort of the credibility that it has.
Taking the Story to the PeopleDaniel:
I'm pleased you brought up credibility. What do people say when they push back on this, when they don't believe it's a great idea? What kinds of arguments do they come with?
Michael:
It depends. A lot of the time they simply just don't really understand what you are doing. They'll just say, "Well, this isn't forensic enough, this isn't scientific. You're letting your own biases get in the way. Your analysis is wrong." They don't really understand how these things prove each other and so they just say, "Well, you're just hand waving things around and saying that this equals this," and they don't really understand why this equals this, so they dismiss it. To be fair, most of the criticism I've come across is more or less just bad faith criticism in the sense that these are people who have motivated reasoning and start with a position of "You're definitely wrong, because you're saying something I don't like." I find that there's little good faith criticism, and it's mostly just bad faith, politically-motivated criticism.
Daniel:
You brought up a great point there, that people didn't understand the analysis you did; perhaps they didn't understand the techniques, they didn't understand the data that was being used. You're telling people, "Yeah, I looked down from space and watched the change over time." I mean, it sounds pretty fantastic. How do you explain that to people?
Michael:
This is sometimes the harder part, because you have to explain a bunch of things that aren't always intuitive. I suppose for people who work in areas like geospatial, thinking in a spatial manner is very natural to us. For other people, sometimes it's not. Imagining what a scene might look like from above, imagining how things could look over time, etc., or how things can be abstracted in a 2D or 3D format, can be difficult for some people. You need to make this easy, and so having really good visualizations is incredibly important for this. I've seen some groups that make really nice videos that are sort of linking things together with nice effects to sort of show how things transition over time. It's all in the kind of visual communication you use.
Daniel:
Oftentimes, I think that the people that work in geospatial, we get stuck in only doing analysis for big companies or municipalities, organizing and maintaining the data. When I meet people like you that are doing work like this, I think, wow, here is another opportunity, here is something that people should know about, because it makes so much sense. We can use these tools that we know about, that we're comfortable with, and we can use them to tell a story. For me, the skill sounds like investigative journalism that tells a story, and backing it up with evidence. Could you imagine a time where new journalists need to learn these skills just in the same way they need to learn interviewing skills or other media skills, making that data analysis a part of their education?
Michael:
Absolutely. It will be critical going into the future. It's the only way that journalism can push back against the kind of flood of misinformation and disinformation that we're currently facing. This is a perfect way to combat it. Rather than just retreating back to ideas of, well, I'm a reputable outlet, or, you have to trust me at my word, instead, you're saying, no, don't trust me. You're right to not trust me, but here, I can prove it. I think this will hold much more weight than simply asking people to take you at your word. If more and more stories are presented in this way, then it won't be so much that people trust the media more, but the correct information will get out more. In the end, that’s what's important.
Daniel:
If you do a great job of documenting this kind of work, do you think this will help people think critically about how people draw conclusions when they see other types of journalistic work?
Michael:
Absolutely. I think this idea of thinking very forensically, in terms of how you prove something, and what proof really means, and where the edges of what can be proven and can't be proven lie really helps you narrow down where the gaps in our knowledge are, and where the area for debate lies. That way we don't spend so much time talking about things that are kind of extraneous, or are red herrings within the conversation.
Daniel:
Michael, I think we could probably round things off here. I'm curious if you have any recommendations or references. If I want to learn more about this open-source intelligence, if I want to be a part of this, is there a community I can join? Is there a newsletter I can follow? Is there somewhere I can go?
Michael:
In terms of communities, there's a great Discord community called Project Owl, which I think currently has about 25,000 users. It's growing very rapidly with hundreds of people working on all sorts of different projects all around the world, all brought together by this kind of usage of collecting information from open-sources, especially about conflict zones, but also about many other different areas- bringing this together and synthesizing this information and seeing what can we prove? What can't we prove? etc..
In terms of other sources online, I think the best group in the world who's doing this is Bellingcat. They're a UK-based publication started by Eliot Higgins, and they've done some really great investigative work. I believe they've also won a Pulitzer Prize, if my memory serves me correctly. They were looking at MH-17, some of the Novichok poisoning attacks in the UK, they've done all sorts of interesting looks at different conflict zones around the world; in Syria, in Yemen. I've also written a few articles for them myself, looking specifically at Yemen and Ukraine. There are lots of great people associated with this and they're doing really good work all the time.
Daniel:
So I really hope you enjoyed that episode with Michael Cruickshank. I found a few other articles that I want to link to and share with you. And one of them is called The Growing Problem with Deep Fake Geography: How AI Falsifies Satellite Pictures, or Satellite Images. There's another one here along the same lines, When is Satellite Imagery Fake? And the third one, Why Newsroom People Need Expertise in Remote Sensing. I'm also going to include a link to a newsletter on Substack that I found that I think you might find interesting. It's called actualcontrol.substack.com. If I read the headline of this blog newsletter, it says A Blog About Satellite Imagery, Social Media, and Other open-source Information from All Corners of the Internet. There will also be a link to the publishing house that Michael mentioned, called Bellingcat.
Before I let you go, I just want to highlight one of Michael's insights, and that was at some stage during the start of the conversation he said something like, "I guess I wanted to tell a different story to a different kind of person." He used this idea of telling better stories. I think this is really important: telling better stories. We talk about data stories and we talk about customer journey stories, but we never talk about telling better stories. Michael wanted to tell better stories. He wanted to do that because he discovered that the old stories weren't working anymore, they were being drowned out. They were too similar. Everyone was telling the same story so it wasn't sinking in anymore. It was blending in, it was becoming part of the background noise. That sounds really simple, right? Tell a better story. What does a better story look like? I'm not completely convinced you need to know what a better story looks like, or what the right story looks like. I'm more convinced that you need to try a new story. If you try enough new stories, you'll find the right story.
Okay. That's it for me. That's it for another episode of the Mapscaping podcast. I'll be back again next week with a new story. I hope that you'll join me then.
Daniel:
Hi, Jeff. Welcome to the podcast. So you are the director of user experience at a company called Element 84, and you've got this amazingly rich background in design, and now you're working in geospatial. For the sake of context, could you just take a couple of minutes to introduce yourself to the audience and perhaps explain to us how you got involved with design, your background in it, and how that led you to work with a geospatial company?
Jeff Siarto:
Absolutely. I'm Jeff Siarto. I'm the director of user experience (UX) at Element 84. E84 is a small software engineering company that specializes in geospatial and science systems. I got here through a sort of roundabout way. My educational background is not in geospatial, or science. I studied general design and user experience in college. After that, I did freelance web design for a while. During that time, I wrote a couple of books for O'Reilly and their Head First series. In that time, I met Dan and Tracey Pilone who run Element 84. They were co-authors in that same series so I was able to get to know them early on in my career.
Before I came to E84, I started a small social media analytics company, which I ran for about five or six years. We did early social media analytics for consumer product brands. I worked with companies like Wrigley, and New Balance, and Radio Flyer, and we essentially monitored their social media traffic, and then helped their marketing teams kind of figure out messaging and how to engage on early social media channels. That was sort of my initial background. I worked on the software there, and I worked on the design of those projects. After I was done with that, Element 84 reached out to me and asked if I'd be interested in working on this NASA project. I said, "I don't actually have anything going on right now. That sounds really interesting." That's sort of how I got here. I came onto that project knowing very little about remote sensing and earth observation, but bringing my design background to bear on their problems. They had never actually worked with any designers before, so I was kind of new to them as well. I sort of just learned as I went through some of their early projects.
Design and Earth ObservationD:
So we're going to talk a lot more about design and earth observation in just a second here. I'm curious, when you think back to when you started at Element 84, you said that they'd never worked with a designer before. What was it like for you coming in? Were they doing the right things? I guess a lot of the semantics around earth observation was new to you. When you looked at the products that they were making, the things that they were doing, was it all brand new, or could you see a lot of the design elements that you were used to working with, or was it something totally different?
J:
They were getting there. I was actually the first designer at Element 84, and then I was also the first designer to come onto this NASA program, so they had some foundational pieces there. They had great backend systems. They were working in the right direction, and I was able to come in and sort of help them improve the interface, or get their interfaces to the point where they were in the same league as some of the backend systems that they had. They were kind of getting to that point where they had solved a lot of the early problems, but now more of those user experience problems were starting to surface. That has ended up being a trend throughout my career here is once those early problems get solved, those user experience problems sort of creep up and become the major players.
D:
Now that you've been working in the industry for a while, when you think about earth observation and design, do you think a lot of the companies that are doing similar work to what you are doing are facing the same problems? I mean, does it surprise you in any way that we weren’t thinking about design earlier?
J:
I do think that more companies are starting to look at that. As we start to solve those kind of middleware early problems, the piping, and the user interface problems start to creep up. I don't necessarily think that it's surprising. If you step back and look at the timeline of the early internet and where all of that went, design became more important as the web matured. A lot of people sort of moved from that graphic design discipline onto the web. They brought those skill sets to bear in an area that was highly technical before it was a focused design medium.
I think we're seeing that in geospatial, particularly as the tooling gets more web-based and cloud-based, or you're interacting with these applications and APIs through the internet, you're going to encounter more user interfaces. I think the importance of those user interfaces is in making the job of whoever that end-user is easier.
Reducing the Time to ScienceD:
In one of your tweets, I saw this great catchphrase, and it was reducing the time to science. My first question is, how is design going to reduce the time to science? I think design could mean a lot of different things for a lot of different people, depending on what it is that we are working on. If we think about the geospatial stack, we could divide it up broadly into the backend and the frontend. I'd like to start with the backend. You've been talking about it as pipes and infrastructure up until this point. How do we bring design to bear on that? What bits can you come in and design to make it better?
J:
Time to science is super important. To define it a little bit, it is the amount of time that a particular scientist or researcher will spend getting to the answer of the question they initially posed. In the past, scientists have spent 50, 60% of their time doing data wrangling, or data processing- getting all of the bits together so that they can answer the question.
The first part of there in reducing time to science, is sort of smoothing out that backend process. Maybe that's going from having to deal with level zero data, or level one data, up to a more refined data product that is properly subsetted or properly re-projected, and taking care of that for the scientists. Now they don't have to do all of that specialty work on the data, and that gets them one step closer. If you sort of continue on with that, particularly as we move into a cloud data paradigm, maybe we go even further. Maybe we take it and we put these data products into a format that can be easily visualized, or easily manipulated in a way that does require programming. So we've sort of eliminated some of the lower level tasks from the workflow and we sort of shorten that time to science. As a result, now maybe they only have to spend 30% of their time getting the data in the right place.
If you go back even further, some 60% of data wrangling time was just literally waiting for data to download. I would hear stories when I first started where we would go out and talk to scientists, and we'd watch them work on their projects. We'd look at their workflow and see how they interact with the data. Oftentimes, they'd just have to set up a download, and just let it go, and come back a couple days later when all their files were downloaded. Some of that time to science was literally just kind of sitting around and waiting.
Even just improving how you interact with the data, and just generally improving the speed of the internet that people have access to is user experience as well. The speed of the internet access, the way in which they have to get the data onto their machine, and what formats that data is in. They all are related to the overall user experience.
Data Science Infrastructure as a CommodityD:
When I think about the backend, apart from the data manipulation to get it to that analysis ready-state, I think about infrastructure like blob storage and the cloud. I think about infrastructure like cloud optimized GeoTIFFs and other cloud optimized data formats so that we can stream data directly to a client without having to run a database somewhere in the background. I think about things like the stack interface. It seems to me that these pieces are sort of slowly coming together, and that at some stage, they're going to be a commodity. It's going to be “of course you're running with these types of standard components”. I'm wondering how you see that and think about that in terms of being a commodity. Do you think it is at the moment? Are we there yet? Of course you're using these things. What else would you be using?
J:
Yeah, I think we're moving in that direction. I would not say we're quite at a commodity yet, but I think we're getting there. I think you can tell that we're getting there in a couple of different ways. You're starting to see the major cloud players building these systems up. I'm speaking specifically of the AWS public data sets and the Microsoft Planetary Computer. You have these big cloud players that are kind of investing in these open geospatial systems. They are building this backend, this foundational layer of data products and APIs that then can be built on top of in a standardized way.
I think we are in the building stage for that layer. I would say I don't really know where we are in that building stage, maybe halfway through. I think in the decade-ish that I've been working in this, over the last three to four years, you've really seen an uptick in cloud data adoption, and seeing investment in those platforms that are going to maybe become a commodity in the short term. I don't think we're quite there yet, we're still in the building phase.
D:
It sounds like you are thinking in perhaps the same sort of lines as I am- that this will be a commodity at some stage. When I think about earth observation, and I think about software companies that are looking to build a business around it, I think it's going to be pretty hard for them to create the data themselves. There will be satellite platforms which are going to create the data. It's going to come down to earth. It's going to go through a commoditized system, and it's going to end up in the frontend. I think the place where a lot of value would be created, would be in the frontend. Do you think that I'm on the right track there?
J:
Yeah, absolutely. Part of the issue that we're facing right now is that Earth observation and geospatial data is fairly complex to use. We're taking a step in the right direction by moving that to the cloud and creating standardized systems on top of that, and now we've sort of eliminated that initial barrier. Once we have that, now the companies, and the startups, and the government agencies that want to start spending more time answering specific questions can do that. If we've got a commercial interest or issue that we want to answer a question about, now, you have the foundational pieces in line to be able to answer those questions, and focus on taking the data that is in the standardized format and manipulating it in a way to answer a specific question for a specific customer.
All you have to do is sort of focus on what that question is and how to build or manipulate the system to answer it. But you no longer have to worry about “Okay, where am I going to get this data? What data set do I need to use?”. You won't even really need to worry about what the sensor is or what the platform is. You'll have the normalized data ready to go. So I feel like that allows you to really focus on the customer problem.
I think a good example of this is a company like Twilio, which is sort of like a communication and SMS/voice API over a bunch of lower level technologies. If I want to then build a product that utilizes some sort of SMS messaging, or some sort of voice system, I don't have to then go interact with the SMS layer, or figure out how I'm going to do voice over the internet. I can just build on top of Twilio's system. They've sort of standardized out that lower level tech for me. I can focus on my AI chat bot, or my SMS messaging tool or something like that, without having to worry about that slightly lower level. It frees you to think about the higher level problem, as opposed to where you're going to acquire the data, and how you're going to get it into the right format.
The Future of Frontend Design in Geospatial Data ScienceD:
When I think about the frontend opportunity, we have these great platforms we can build on. I think about the brand-ability of a frontend. The way you can dress something up, you can change the look and feel of it. You can add an identity to it, and that can become a selling point in itself. It doesn't feel like those in the geospatial industry are that interested in changing the way we do frontends, and designs. It seems to me that we are pretty committed to using the tools we've always used. Firstly, I'd like to know what you think about that statement. And then secondly, I'd like to say do you see any opportunities to change things here? Any things that could be done differently?
J:
Yeah, I do agree with that. There's a pretty consistent set of tooling that's been used forever. I think there's a lot of standardized patterns that we've seen used quite frequently. I'm kind of talking about a map interface where you can drag around a map, you can create a bounding box, and you can sort of interact with a Google Maps style interface, which has become fairly ubiquitous. Ubiquitous both on the sort of geospatial earth observation side, but then also the consumer side as well. From a geospatial perspective, when you're looking at a map with a bounding box and a sidebar of layers, you're still thinking- “I need to pull data from around this area. Give me all the stuff inside of my bounding box.” I think there are parts of that that are going to carry over. I do think that we're going to need to move into a direction where we are building interfaces that are answering specific questions as opposed to just building interfaces that are allowing me to grab a bunch of stuff inside of a bounding box.
There's always going to be that. In geospatial and earth observation, you're always going to have to tell the system where on earth you want to get data from. I'm not necessarily convinced that it always has to be on a map. I'm also not convinced, after spending many, many years watching folks interact with map products, particularly in the earth observation and earth science world, that that's really the best thing. I think people still struggle with it. I think they're still a little inconsistent, and I think we could do better.
Yes, these tools are ubiquitous. But we still have not ironed out all the UX issues of the map-based interface, let alone moving into a new paradigm. I'm not convinced that it's the best interface once we get into that higher level “what question do you want answered?” phase.
D:
Every time I see a new data portal, it feels like it's a total one-size-fits-all without too much thought that's gone into it. It's a map. It's the panel on the left hand side of the screen. It's all the usual things. Part of me thinks seeing standard products, like standard design, things that I know and have used before makes it easy for me. I don't have to learn something new, so I get that side of it. It just feels like this one-size-fits-all product where no one has really come along and thought- Can we do it differently? Is there something else? We even see Google Maps doing the same thing. It feels like maybe it's not a technical problem. Maybe it's more of a cultural problem.
J:
Yeah, absolutely. I don't think it's a technical problem. There have been great strides on the technical side of internet map making. There are all sorts of great tools to make creating layers and getting the map in your browser easier. I certainly would rather be building map interfaces now than map interfaces a decade ago. We've come very, very far in the developer experience with how easy it is for a software engineer to put a map interface together, but I think that's where we've been iterating. We've been iterating on- How can we make this map interface a little bit easier to use? or a little bit easier to develop? or easier to serve information on top of? But I don't think we've ever really stopped and thought, is a map really the right interface for this particular application?
I think that's sort of where I see us going- Maybe a map's not the best interface for this particular piece of software. And if not a map, then what?
D:
I've often heard people get really excited about having an output they can put directly into a spreadsheet, because that's what they are used to working with. Something that’s going to go directly into a database, or talk nicely with other spreadsheets, and they didn't actually want a visual output at all.
J:
I think that's true. You're kind of always thinking look, I can give you this picture of the land area. We always want to put things into a visualization. Doing visualizations is an important part of this., but not all science users want their output in that format. Sometimes they want it output in code. Sometimes they want it output in a spreadsheet. The flexibility in that output is important. Not assuming that your user wants it in any particular format, but making things sort of extensible in a way that they can pull it into whatever format they're most comfortable with is the way to go. Whether that's a statistical programming language or, like you said, just a standard spreadsheet.
Going back to what we were talking about earlier, with having that middle piece built out and matured, that lets us think about those interface problems. We can spend more mental energy and man hours on those types of problems, because we're not having to sort through the lower level building blocks.
UI/UX Standardization in Geospatial D:
MapScaping used to be an eCommerce website. We ran on a platform called Shopify. Are you familiar with Shopify?
J:
Yes, absolutely.
D:
So Shopify showed up and made it incredibly easy for people like me to start an eCommerce website. There is an app environment, so you can just add these plugins, these apps to your Shopify store from a standardized box. You can quickly make a custom thing, and it's absolutely amazing. It's interesting with eCommerce. Their whole thing is to get someone into the shop and out of the shop as quickly as possible. If the checkout takes more than five seconds, it's too long. There's this real focus on “get the customer what they want as quickly as they can”. I don't feel like we've got there yet with geospatial. I'm wondering if there's any lessons that you can see in eCommerce, perhaps from Shopify that we could take over to the products that we build in the geospatial world?
J:
Shopify is a great example, because they've done a handful of things really well. One of the things that I love about Shopify is that they've sort of standardized the user experience patterns around the checkout experience. When you're checking out on a Shopify site, you know that it's a Shopify site. A lot of people probably don't know it's a Shopify site, but they know that they've seen this pattern before. They know they've seen this particular way the credit card has to be entered into the form. They expect when they start to type their address, that a dropdown is going to appear, and they're going to be able to select their address, and it's going to autofill. They don't have to do all that. That experience is seamless and it's sort of expected. The end user knows what they're going to get. I think one of the problems I see with geospatial interfaces now is that you're not always sure what you're going to get.
My one example of this is the way that an area of interest selector, or a bounding box selector on a map interface behaves. Sometimes, you have to click and drag it. Sometimes, you have to click and add points like a polygon. Once you have the area selected, the way in which you perform actions on that selection are also different. I think there are ways that we can standardize that. I think the group, or company, or people that figure out the best user experience will have standardized it in a way that makes it kind of crazy to not use that. That's one thing that Shopify's done really well is you really have to convince someone to not use the Shopify platform for their store. The people that are using Shopify, they're using it because they don't want to deal with the eCommerce aspect. They want to sell their T-shirts. They want to sell their eBooks. They want to sell their information products. They don't want to have to learn a new system to do that, or put anything out there that is going to make it difficult for their customers to buy their product.
I think with geospatial interfaces we need to move to that point. We need to make it so that there's an obvious choice for some subset of these user actions, but not everything. There are certainly people who choose to roll their own eCommerce site for various reasons, and that's always going to exist. I would love to see some of these patterns mature in geospatial user interfaces so that our science users and our municipal users who are trying to get questions answered for their county or state don't have to wonder what they're going to get when they try to pull data to answer a question. They know there's going to be some standard level of interaction there.
There are two sides here. It's great for the developers because they don't have to reinvent that wheel. They don't have to come up with a new way to do a bounding box, or a new way to do a map interface. On the customer side, they don't have to wonder, "Oh my gosh, this is a new geospatial system. What am I going to get here? What is this going to be like?". I think right now, there's still some of that.
A lot of the science users that I talk to, they're certainly apprehensive about new systems, because oftentimes it takes them a long time to become proficient in the one that they're already using. They do not like to have a new system designed by these new designers that just came in. I'm totally putting myself into those shoes. At one point, I was the new designer that came in and I was like, "You guys are doing all this wrong. You need to do this, and this, and this." That kind of freaks people out, especially people whose entire career is based around doing this type of science work. That specific data is very important to them, both to their career and whatever agency they're working for. There's a lot of uncomfortableness with that change. I think some of that is growing pains, and we've got to get through that.
D:
So we talked about Shopify, and one of the beautiful things about it is that establishment of a standard set of patterns. We know what a checkout looks like. We understand this. Three steps, there's autofill forms, there's this, there's that. Then lastly, we push our credit card numbers in. We get that. Do you think the lack of standard patterns in geospatial speaks to how niche it still is?
J:
There's not as many designers working on specific geospatial problems. I think that's certainly changing. I think a lot of the new interfaces that we see coming out from a lot of the new commercial companies that are appearing, they're taking design a lot more seriously. There just hasn't been a lot of people working on these problems.
There obviously have been great designers working on mapping problems for quite a while. Look at Google Maps, or Apple Maps, or Mapbox. I think on the earth science side specifically it gets pretty niche, especially because of the complex science that's going on there. I think the consumer side is a little more mature, but I'm not entirely convinced that the patterns developed on the consumer side necessarily fit the scientific model or even the municipal GIS worker model. Those are different people that need to navigate in a car or look at an aerial imagery of their house versus a user who's trying to work out land use patterns, or monitor sea surface temperature, or things like that. There are different requirements.
Breaking the Status Quo with New UI/UX Functionality in Geospatial D:
I think this gets back to the idea that one size doesn't fit everyone. We need specialized maps or specialized interfaces, depending on what problems we're trying to solve. I want to stay with the Shopify example just for a second here. In our store, we had this option to install a plugin. That plugin was going to provide a little mini recommendation engine- People that like this also like that. I think in the past, a big argument for having the map was, "We can see the data to decide if that's the data we want." It's an incredibly quick way of filtering a massive amount of data visually. I would love to see some sort of recommendation engines come into these portals to help us filter through it. I would like to see some voice interfaces built into these things. From a design perspective, can you see anything that's stopping us from integrating some of these ideas?
J:
I don't think there's anything specifically stopping us from integrating these ideas. I think we are moving towards that in some areas. We have this notion on the earth science side of “usage based discovery”. This is finding data sets based on what you're trying to do. What is the end goal of the data set, and categorizing things in that way.
I think as the metadata systems, that middle piece that we're in the building phase of now, I think as that gets improved we can expand the metadata available for these data products. Then those alternate discovery mechanisms become easier to build.
Part of the problem is that, in order to build those higher level discovery systems, you still have to have the information about the data. You still have to have the metadata about the products. As we're building this middle piece out, those metadata are becoming more refined, more detailed. Again, because we're not dealing with the lower level problems, people have more time to refine and incorporate more specific metadata into these programs. We can start to build smarter systems that sit on top of that data and allow us to have some sort of AI, or machine learning algorithm make recommendations for data that we may not have thought about. Maybe we have an interface that isn't a map, and you're just sort of asking the system a question based on a region of the globe. Then it can suggest other questions related to that. Or it can say, "Hey, these other researchers have posed a similar question. Here are their results." I think there's a ton of space for those types of interfaces once we have the sort of data infrastructure there, but we aren’t quite there yet.
D:
Is there anybody out there at the moment that's doing inspirational work in this space that you can point to and say, "Well, if you want some examples of great design, great interfaces, look over there"?
J:
Yeah. So a couple examples off the top of my head. I think the folks at UP42 are doing some really great design and user experience work, from a product perspective as well. They're sort of an example of the higher level system that's sort of interfacing with a bunch of these other data providers and then providing an interface or a system on top of that to streamline a workflow to get a faster answer to a question. I really like what they're doing.
I also love this little project called Placemark by Tom MacWright. He's doing some really interesting user interface work in a niche space. Unfolded.ai is another really great one. Their mapping and general design interfaces are really, really good. They're built up on top of Uber's H3 geospatial library, which is a hexagon-based geospatial library. This is being pulled out of the consumer, commercial side, and adopted now into more of the data analytics and geospatial side of the house.
Obviously Mapbox, I'm sure everybody knows about Mapbox. They're doing really, really wonderful stuff. They have fantastic designers. Their 3D rendering and map rendering is really amazing. They are importing consumer and commercial tech into the data analytics and geospatial side. Those are all great examples of fantastic design and user experience.
I'm excited because all of these new companies and projects that are cropping up look great. They're all seeing design as a first class citizen. I think we can just continue to build on that momentum and raise that bar a little bit. Then those expectations for where design and user experience need to be for these systems are just going to keep getting better. We're going to see the same trajectory that we've seen on the consumer and commercial internet side. Pick your industry. All of those systems have been improved with designers having a seat at the table.
Design and/or User Experience?D:
You mentioned design and user experience a few different times there. Do you see them as being two different things?
J:
Yeah. This is like an endless debate. I feel like I was just having a conversation on Twitter with somebody about the difference between design, user experience, and user interfaces. If you're a user experience designer, does that mean you also design user interfaces? And if you're only a user interface designer, are you also a user experience designer? I'm not sure if that's exactly the right argument.
I think I see design as the whole of everything. I consider myself a designer. There are a multitude of practices under ‘designer’. I think a designer could be a pure art illustrator. I would call architects designers. I think industrial designers sort of run the gamut.
User experience for me is the entire end-to-end workflow of engaging with a product. Oftentimes that is not necessarily the way the buttons look, or the way the map looks, or the way the interface looks. It's the process by which a user gets through the system. What are the steps that they have to take to get there? What is the language that you're using in the system? What words did you pick to describe different things? What icons did you choose to represent particular things within the interface? It even goes even further than that. What is it like to engage with your company? So if you're selling geospatial data and you have to send an email to a salesman and then jump through a bunch of hoops to get any information on a data product, well that's part of the user experience too. That's the early, early interactions.
Oftentimes at Element 84, when I'm looking at the user experience, I'm not only focused on the UX of the products that we're developing, or the tools that we're building for customers. I'm looking at the user experience of the company itself. What is it like to interact with Element 84 on social media? What is it like to interact with Element 84 when you're engaging as a new customer? Is the contracting goofy, or is it really hard to get a response from someone? Does the person you had to deal with from a project management standpoint, are they treating you well?
Specifically for geospatial and earth observation stuff, the user experience sort of starts at the engagement with the organization that is brokering the data. What's the UX of going to NASA for earth observation data? It starts pretty early on in the cycle, and our user interfaces are only one tiny part of that overall experience.
D:
So it sounds like what you're talking about is the promise. What is the promise here? When I interact with a company, when I interact with a piece of software, what's the promise that is made? And is the promise kept throughout the interaction?
J:
Part of that promise is that you're setting some level of expectation. You need to be careful. It's easy to talk about a particular feature, process, whatever, and set that high expectation. Then you've got to deliver on that throughout the entire user experience of that individual interacting with your system. That system could just be a little piece of software, or a website, or an entire company. I think keeping that promise is important. When you get into complex systems, particularly in earth science, it is very hard to keep that promise all the way through. There are so many places in that process where you can sort of lose your promise. It's really important to be vigilant about that. It’s the hardest part of the job. Making the interface look good and polished honestly is the easy part. The really hard problem that we have to tackle, is really how do we keep that promise from the initial interaction all the way through to I've got the answer to the question I had.
The Future of UX/UI Design in GISD:
I feel like we've covered a lot of this already during the conversation. When we look out into the future, if you had to sum up things into some bullet points, what do you think we can expect from user interfaces when we think about geospatial?
J:
First and foremost, there's going to be kind of a layering over geospatial. I think we're going to worry less and less about the sensor technology, the resolution, all the tech specs of the geospatial data itself. I think we're going to get to a point where we're not going to have to worry about the low level earth observation data. So there's going to be a pretty good obfuscation of that. I think you're going to get to a point where you're going to get transparent sub-setting and re-projection. It's just going to be like having to understand the intricacies of SMS messaging or any of the HTTP layers on the web.
I really think we're going to be done with downloading data. I think the future data sets are going to be enormous, like terabytes, petabytes. Huge, huge data sets. It's not going to make sense to pull that down over even your best gigabit ethernet pipe. Everything is going to be an API layer. Everything is likely going to be sitting with major cloud providers. You could have a whole other discussion on how you feel about centralization of the cloud providers. That's beyond the topic of this conversation, but I think that's where we're headed, at least from what I can see.
I also think particularly on the earth science side, you're going to get into these low code or no code science tools. There are a lot of really great projects going on right now. Pangeo is one of them where they're building a lot of standardized Python tool sets and libraries to sort of deal with earth observation and geospatial data at a code level. Now we don't have to write this low level code anymore. We can write this higher level elegant Python to do some of the data processing. As we move into the future, I think we're even going to get a step above that. We're going to get to the point where I could be a data scientist or an earth scientist and not have to deal with spinning up AWS instances, or making sure I have a particular Python library. Looking 10, 20 years into the future, I think the low code, no code science tools are going to be the big step there.
D:
Do you think the community is going to push for this? Do you think the pull is going to come from outside of the geospatial community? Do you think scientists and geospatial users are wanting this today, is this going to make their lives easier? Or do you think that it's going to be people from outside the community saying, "We would love to use that, but you're going to have to do these five things first"?
J:
I don't think we're quite there yet. I think the pull right now is “Give me better code tools. Give me better Python libraries. Make this whole spinning up cloud instances go away. Get me out of that.” I think we're moving in the right direction with things like Jupyter Notebooks and JupyterHub standardizing and making it much easier to do data processing at scale in the cloud. So I think we're getting there, and I think we are answering the call to improve developer tools, but we aren’t there yet.
I think that the push is going to come from the other end though. A good example of this, the future scientists, think people that are maybe in high school or middle school now that in 15 years, they're going to be taking over desks at NASA, and NOAA, and USGS. They're going to have grown up on low code and no code tools. They're going to have expectations that the science tools are as easy to use or of a similar capacity as what they've seen in other parts of their digital life. I think the push is going to come from that direction.
In order to get the low code or no code science tools, first you've got to have really great coding tools. The developer experience has to be really, really good, then the developers can stop worrying about the lower level code. Then they can start thinking about, "How can I write this piece of software where my scientist doesn't really have to do any coding at all, and they can just be an expert at land use, or they can be an expert oceanographer without also having to be a software engineer Python expert?"
You could think of other commercial interests full of people that don't have any desire to do code. They're going to be pushing for this too. You see this in commercial web products, like website builders, and even low code or no code tools for building mobile apps and all sorts of complex web interfaces. We're getting to the point now where the infrastructure for writing code on the web has gotten so good, that we are thinking one level higher now. We're able to think about how we can build tools that'll grant that same power to an end user that the developer has without them having to be a software developer.
Daniel:
I think this is a really, really great place to round off the conversation. I want to say thank you very much for walking us through this. I've never talked to a designer before on the podcast. I've really enjoyed the discussion. It's opened my eyes to a lot of things.
Before I let you go, if people want to reach out to you, if they want to ask more questions about any of the things we've talked about today, where can they go to do that?
Jeff Siarto:
The best place to get in touch with me is probably Twitter. They can get me @jsiarto. So first initial, last name on Twitter. Or you can also find me @Element84, which is our company's handle. Both of those places are great. DMs are open as they say.
You don't have to be a big organization to create a business or side hustle around your geospatial skills. If you are looking for examples and ideas ... this episode is for you.
If you enjoyed this episode check out "Geospatial Side hustles"
https://mapscaping.com/podcast/geospatial-side-hustles/
Please support this podcast on Patreon!
https://www.patreon.com/MapScaping?
Join the mailing list!
https://mapscaping.com/podcasts/
Daniel:
Hey, Chris, welcome to the podcast. Today we're going to be talking about mapping the acoustic marine environment and using acoustics to map the marine environment. I wonder if for the sake of context, if you could take the time to just introduce yourself to the audience, please?
Chris:
Thank you very much for having me. Happy to be here. My name's Chris Verlinden, I'm am oceanographer, and acoustician. I'm originally from Portland, Oregon in the United States. I spent about 14 years in the coast guard as an officer, primarily serving on icebreakers in the Arctic. I spent the last few years of my career on loan to the Navy doing ocean acoustics research, everything from finding submarines to whales. I got out of the military about four or five years ago and helped found a small company called Applied Ocean Sciences, where I currently serve as the CTO. We do research in ocean acoustics and marine environmental protection.
Differncen between a Soundscape and a Soundshed
D:
I noticed during the introduction there, you refer to it as ocean acoustics. A lot of the time I hear people calling acoustic environments, soundscapes. For me it would make much more sense to call them soundsheds- the same way we talk about a watershed or a viewshed. I imagine that this acoustic environment has geographic limits. Could you explain to me why we don't often refer to this as being a soundshed?
C:
I actually love that question because I'm secretly starting the movement to start the term soundshed. I’m referring to the study of ocean acoustics, which is really just the study of sound in seawater and how sound propagates, its impact on the environment, and how you can use sound to sense the environment. When I use the word soundscape, I think of it as a way to describe everything that you'd hear in the ocean.
It's sort of the cacophony of man-made sounds from ships in industry, oil exploration, the sound of biologics like marine mammals, fish, snapping shrimp, flocking fish, things like that. As well as natural sounds, everything from breaking ice, breaking waves, and wind and seismic activity like earthquakes and volcanoes.
I think of soundscape the same way I think of a landscape. In a landscape, if you're standing in an environment, is everything that you can see. Well, the soundscape is everything that you can hear.
I think of the term like viewshed or watershed. A watershed is an area where all the water in the area flows to a single point or, a viewshed is all the places that I can see from a point. I think an appropriate use of the term soundshed might be to refer to the sound footprint from a single source, or maybe the sound footprint of a single receiver. Practically, if I have a microphone or an underwater microphone called a hydrophone in this location, what are all of the things I can hear?
If I look at the footprint of everything I could hear, that would be the soundshed of that microphone.
Similarly, if there was a whale, let's say a blue whale calling at its usual noise level and frequency, there would be a certain area over which you could hear that whale. That would be the whale’s soundshed. The soundscape would be the full abundance of sound in the ocean. It's the cumulation of all of the various sounds that you're hearing.
A soundscape might not have a clear definition because, depending on the time or what is happening or what the dominant source of noise is, the variability of the soundscape will change. For example, if a hurricane is moving through the ocean or within a few 100 miles, all you're going to be able to hear is breaking waves and wind and the noise from the hurricane. Whereas similarly, if you're in an area near a shipping port, all you're going to be able to hear is these ships, but that could change.
Maybe the port closes at night, and over the course of the day that soundscape changes. So to me soundscape describes all of the sounds that you can hear in a given area and those geographic boundaries will change and shift over time.
Mapping using Active and Passive Acoustics
D:
When you were describing soundscapes, you were talking about all of the different noises you could hear. You also mentioned this idea of using sound to sense the environment. Oftentimes when we think about remote sensing from satellites we'll think about active and passive sensors. When we think about active sensors and sensing the marine environment, people may be familiar with using acoustic measurements to measure bathymetry. Could you give us a few examples of passive sources of acoustics that we can use to measure things, and what we can measure with them?
C:
This is a question I've devoted my life to, so I get really excited, I think it's a pretty interesting topic. First of all, just like with remote sensing from space where you have active sensors like synthetic-aperture radar (SAR) and passive sensors just collecting imagery, you have that same paradigm in the ocean.
Fundamentally, when you're studying the ocean, you have to use sound because sound is how we see underwater.
What I mean by that is all the electromagnetic radiation, all the visual band optics. The light just doesn't propagate well underwater, and neither do most frequencies of radio that we can use for communications. What that means is that if we want to image the bottom of the ocean, whereas on land, we could use radar or LiDAR- all the DARS. DARS don't work underwater. We pretty much have to use sound to image the environment. Similarly, marine mammals use sound to navigate, find mates, communicate.
If you want to communicate, if you want to see and sense your environment, if you want to navigate, sound really is your only option. Just like in space and air where you have passive and active sensors to do all of those things, similarly underwater, you have active and passive acoustics.
Active acoustics would be where you transmit a sound and use the way that that sound interacts with the environment to learn something about your environment. Just like you pointed out, the real quintessential example of that is bottom mapping sonars or multi-beam sonars, bottom-penetrating sonars. Where you put out a pulse of sound, you wait for the reflection off of the bottom of the ocean and sub bottom layers. You can also image what's underneath the sea floor. You use the travel time of that reflection back to your sensor to determine the distance to those things. If you're determining the distance to all things in all directions, you start to construct an image of your environment.
Active acoustics are a very effective way to measure certain things in the environment, but there are still downsides. One is that some of these active acoustic techniques can be damaging to the environment. It can be harmful to marine mammals and fish that you sound to navigate. In this new paradigm of swarms of autonomous vehicles exploring the ocean, every watt of power matters. If you have to put out active of pulses of sound, you're going to kill your battery a lot quicker.
Passive sensing techniques have the advantage of being far less energetically expensive, and the equipment is far less expensive too. The easiest example is passive sonar compared to active sonar. If I'm looking for a submarine with active sonar, I put out a ping. I wait for that ping to reflect off of objects such as submarines. If I want to define that submarine passively, I just have to listen very, very carefully for the sound that submarine makes and then use triangulation, with my array of hydrophones to determine the location of the source. Passive acoustic sensing isn't limited to just hearing whale song or finding submarines. You can do a lot of really cool things passively.
There was a really wonderful paper written in the nineties by a scientist at Scripps, Mike Buckingham. He was actually on my committee and he's a brilliant and wonderful guy. He wrote a paper called Acoustic Daylight. What he did was he drew a comparison to the way that we use our eyes to see and sense our environment in air to what we might be able to do with acoustics underwater.
Think about the way that we interact with and that we sense our environment around us above water. We use our eyes. Our eyes aren't active sensors, we don't have laser beams shooting out of our eyes like LiDAR measuring reflections off of everything. We just take the ambient electromagnetic radiation or light that's reflecting off of objects and scattering all around us. We receive that light on arrays (our eyeballs) and do some very clever signal processing in our brain. We are able to reconstruct an image of what the world around us looks like from nothing but back-scattered ambient light.
Michael Buckingham wrote this paper of how we might be able to use acoustics to do the same thing. He used noise from things like breaking waves on a pier to image objects underwater. It's a non-trivial problem, it’s very challenging to do. He opened the door for a lot of really interesting work in using passive acoustics to sense the environment.
We have used noise from ships to measure the temperature and salinity, and possibly even the pH of the ocean using the way that the temperature, salinity, and pH of the ocean affects how sound propagates through seawater. This can help with all kinds of things, from climate and weather forecasting to military operations, to understanding ecosystem dynamics.
Another really interesting example, Dr. Aaron Thode wrote a paper a while back where he used blue whale calls to measure seawater and bottom properties of the ocean.
Creating Underwater sensor networks
D:
You mentioned this idea of using an array of sensors to gather this information. You talked about it being a non-trivial problem. When you were talking about using the acoustics from a ship to measure seawater temperature, what kind of sensor array do we need to have in place in order for that to make sense?
C:
I'll let you know when I figure that out, it's a non-trivial problem and there are a lot of really bright people working on it. In fact, I have a current ongoing research program with the Office of Naval Research studying exactly that problem, which is what sensors and what combination, what configuration do we need in order to make use of those types of sources to measure the ocean?
What I want to do is I want to create algorithms that can take an arbitrary system of hydrophones or sensors anywhere in the ocean, and quantify how well we can use those sensors to measure the ocean.
You're not going to pursue a new remote sensing technique unless you really understand how well it's going to perform. Some of the mathematics that go into inverting ocean profiles and seawater properties using noise from ships are not non-trivial because you don't know exactly where that ship is. You need to localize the ship. You need to account for any type of timing or position offsets in both your sensors and the receiver, and you need to measure the properties of the ocean all in one big mathematical calculation.
There's a whole field of mathematics that really center on how you can cheat in that problem by using prior information. Things like AIS ship tracking data and the fact that your array is only so long, so your hydrophones can only vary in certain positions. I've really made a career out of finding clever ways to cheat in order to do more with less information on measuring the ocean with sound. I'm currently working on trying to find ways to determine the information content of sound on arrays of hydrophones so that we can use that to answer your exact question of what exactly we need to measure the environment. The answer is as many hydrophones as possible, as perfectly surveyed as possible, with as quality of synchronization as possible. How good is good enough and how many hydrophones do you really need? That is a really an open question and something that we're hoping to answer.
Getting Good GIS Data from Aquatic Sensors
D:
I heard you mention AIS data. AIS data is this broadcast that comes off ships and it gives you speed, direction and a little bit more information about the actual ship and its position. Can we use that and say, "Oh, we know that ships travel along these lanes here," and put an array of hydrophones along those shipping lanes and start doing something there? We know where the ship was, we know the size of the ship. We can make some assumptions about how much noise it was producing with an array of hydrophones that hopefully weren't moving. Is that a feasible way forward when we think about using these passive sources to map the marine environment?
C:
That's something that I've been wanting to do for ages. We've got colleagues at University of Hawaii who've been thinking about that same thing for a long time as well. Instrumenting underwater cables and things that are already on the bottom of the ocean in areas where you have access to sources of sound like ships in order to do exactly that.
The advantage of fixed sensors is you only need to survey their location once and they don't move. There's a lot of advantages to those types of fixed sensors. So yes, I think 100% the more hydrophones that are out there in the ocean the better. You shouldn't put them out randomly. There will be places where we have more data, like next to a shipping lane. Then you also have to think about places where we really need those measurements. There are a lot of places where we just kind of know what the ocean looks like, either because we have a lot of sensors already in place or because the variability of the ocean in those areas isn't so significant.
I've done a lot of acoustic experiments off the Pacific Northwest. And I could probably draw for you what the temperature and salinity profiles look like 500 miles off of The Olympic Peninsula in Washington state right now, and be within a degree or two. The variability is very predictable and very well understood there. So with a couple sea surface temperature measurements, a few floats, a good oceanographer can tell you really what going on with the dynamics in that area fairly accurately.
If you look at a place like the north Atlantic Ocean off of the UK, for example, that is a complicated ocean. If you take a measurement in one place and you take a measurement 10 kilometers away, I don't know what they're going to look like, but I guarantee they're going to look different. There are a lot of really complicated oceanographic processes, deep convection sites, where water sinks in from the ocean bottom, and deep water masses spread throughout the ocean at ease. Currents meander hundreds of kilometers on a time scale of a few days. Oceans like that are very important to get more measurements from.
The real holy grail of ocean measurements right now is the Arctic. The Arctic is for much of the year covered with ice, and that makes it very difficult to get research vessels and sensing platforms that we have access to under the ice. The Arctic is one of the most critically important places to measure because it's one of the places that's changing the fastest due to climate change. This means acoustic measurements for measuring the ocean properties under the Arctic are incredibly important and challenging, because there are not a lot of ships in the Arctic. If you need to know as much as you can about the sensors and the environment in order to use those sensors to make meaningful measurements of the environment, well, the Arctic where there are a lot of unknowns is a challenging place to work.
Your question was a really good one. "If we just put these fixed sensors where we know exactly where they are all over the ocean, can we just use all the sound sources that are already in the ocean to map and measure ocean properties?"
Yes, but there will be places where that's not practical. That's why I think we need to do a lot more work on using moving platforms and sensors that might not be perfect. There was an interesting study done by DARPA here in the United States recently called Ocean of Things, where they wanted to put tens of thousands of biodegradable floats out in the ocean and instrument them with sensors, including potentially hydrophones, where these things are just drifting around and moving all over the place. That creates challenges because you need to really get the most accurate position of those sensors, you need to get timing synchronized perfectly. If you can do that, you could potentially use all of those sensors like the cube sats, but in the ocean.
There is a lot of really exciting work going on in this area. There's a company I love to work with called Subsea 7 that's making this autonomous sailing vessel that's a fraction of the cost of anything else that's out there. You could put hundreds, if not thousands of them out all over the ocean and position them optimally in order to get the best sample measurements of the ocean everywhere. It is a silent sensor because it's a sailing vessel and its semi-submersible nature makes it a really quiet platform. You could also instrument platforms that are already out there, like whales.
I used to teach oceanography at the Coast Guard Academy and I'd always start off with the description of why it's so important to get these measurements. I always use El Niño as the example. El Niño is a phenomena where warm water from the Western Pacific slashes across the ocean into the Eastern Pacific. It creates a big pool of warm water, which causes air to rise, and precipitation which disrupts global weather patterns. It can cause floods, famine, disruption to crop cycles, and disruption in ecosystems all over the world.
It can put thousands of people out of work, it can cause starvation, billions of dollars in economic disruption, thousands of lives impacted, people dying from the impact of El Niño. It is dynamics that we understand, we get geophysical fluid dynamics, we get how water moves and yet we cannot predict El Niño. This is a solvable problem in our time. If we understand the ocean better, if we make better measurements, if we make advances to ocean modeling techniques and ocean sensing techniques, including the acoustics ones we're discussing today, it's achievable to do things like predict El Niño.
We can save lives. We can preserve the economy. These are solvable problems in our time if we can just make better measurements of the ocean.
Using Sea Creatures as Spatial Sensors
D:
You're so enthusiastic about the problems that you're trying to solve. You said a couple things that you need to go back and clarify for me. You talked about the ocean of things, and it sounded like you were talking about using marine animals to crowdsource this mapping work that needs to be done. Maybe putting sensors on whale. Can you tell me more about that please? Earlier in the conversation, you talked about using whale’s acoustics to passively sense the environment. Are we then talking about putting a sensor on it and using a whale like a cube sat?
C:
First of all, DARPA’s Ocean of Things- it was a program that they created in order to see if you could make tons of measurements of the ocean in order to resolve all the things that we just talked about. If you could instrument the ocean to that degree, you could do things like predict El Niño. You could track where whales are in order to more accurately predict and mitigate human impacts on ecosystems. Having more sensors in the ocean is critically important.
Sensors, electronics, data exfiltration, these things have all gotten cheaper and more available over the years. It used to be that if you wanted a temperature sensor in the ocean, you better have 70 grand. Now there's an Open CTD project. It is a conductivity, temperature, and depth sensor. You can make it for a couple hundred dollars worth of parts and a PVC pipe. There's access to technology that we never had before. There are hacker websites, there's Arduinos, Raspberry Pis, Odroids, Teensys. Similarly, there is the Iridium satellite constellation so you can get data off of any of these instruments.
There is Starlink of course, and Kepler Space, which is a really exciting company out of Canada that's making a really nice communications constellation. The ability to interact with low-powered, low-cost sensors all over the world has never existed before, and it does now. I think in oceanography, we're entering into a new paradigm. It's kind of a revolution right now.
The space industry I would say is still about 10 or 20 years ahead of us. In space, it used to be that there were are a few dozen satellites. They cost billions of dollars, they were the size of school buses, and they had really nice cameras or sensors, but you only had a few of them. Now we have tens of thousands of sensors in communications hubs. Each of those cameras might not be as good, but when you combine the information from tens of thousands of lower end, lower cost, lower size platforms, you get a lot more information. We just got that memo in oceanography, and we just realized that having more sensors is good and having more sensors needn't be expensive.
I think it's established that having lots of sensors, even if they're poor quality sensors that don’t report as often, or in the case of whales, have complete control over where they go, are still a really good thing. So now let's talk a little bit about what would happen if you instrumented a whale or an elephant seal with a sensor. I say elephant seals, because they're one of the deepest diving marine mammals. They have a higher blood oxygen content and a higher concentration of hemoglobin in their blood than any other animal I'm aware of. They can stay down forever.
Put a temperature sensor, a CTD, like we just talked about on elephant seal, you'll get really incredible profiles of the ocean. You can listen to what the animal is doing. It is really important to understand the way that animals interact with their environment, like how whales use acoustics. Just the biological information we could gain from that would be incredibly valuable. If we can localize those animals using the acoustics around us and other technologies, perhaps we can analyze across groups of animals just like an array of hydrophones.
I think there are two really critical things to understand here and that's that you can use ambient noise to measure the environment. You could use the noise of ships like a sonar to localize things underwater or to measure temperature, salinity, things like that. You can also use the differences in ambient noise, from things like ships and waves recorded on groups of hydrophones to measure how much time it took for the sound to travel between those two instruments. Temperature and salinity affect the speed of sound, and that can give you estimates of the temperature and salinity of the ocean. You can also use passive acoustics of sources like whales and ships to measure the ocean bottom’s location, shape, and material.
A good example of that is a paper by Martin Siderius at Portland State who created something called the passive fathometer. Essentially, he takes an array of hydrophones, no sources of sound anywhere, and he does something called beamforming, which is a signal processing technique that allows you to listen in just one direction. He beamforms up and he beamforms down. "All the energy coming from the sea floor is coming from the surface, just reflected off the sea floor, because all the things that make sound are at the surface." "The time difference between the up beam and the down beam has to be the travel time between the hydrophone array in the bottom and back up.” So just like an active sonar, he's able to use the ambient noise of breaking waves and things at the surface in order to image the bottom. It's not inconceivable to think you could do passive multi-beams, or passive sub-bottom profilers. Martin and I have actually written papers about measuring bottom loss, which can tell you what the bottom's made out of.
If you have passive hydrophones anywhere and everywhere you can get them, on floats, on animals, or fixed platforms all over the ocean, you can make very meaningful measurements of both the soundscape and the environment, including temperature, salinity, pH and bottom properties. This matters for ecosystem dynamics, economics, oil and gas exploration, national defense, etc. This stuff matters, and I do think having hydrophones everywhere, including potentially on biological cube sats would be really beneficial.
Comparing Earth Observation From Space To Remotely Sensing The Deep Ocean
D:
This is a real eyeopener for me, this idea of using biological cubesats as you call them. You were referring to the idea of using groups of these in a mesh network to image and sense our environment. You started making those comparisons to what's happening in the space industry at the moment. Why do you think we are pouring money into space and earth observation platforms as opposed to earth observation platforms in the water? Is it more difficult to put things in the water than in space?
C:
I apologize if I offend any of my space colleagues now, but yeah, I think that's exactly why. Don't get me wrong, the assets we're putting in space are very valuable. This cube sat revolution has changed the world in a positive way. That being said, those space guys just need to stop complaining. When they put something into space, they act like it's this engineering marvel. They've got one atmosphere of pressure to deal with between the inside of a platform and the outside of the platform. Try putting something on the bottom of the ocean, where you have hundreds of atmospheres of pressure. Everything needs to be perfect to millimeters, otherwise it'll literally explode or implode. We're putting our sensors in acidic baths of some of the most corrosive stuff known to being- salt water. Yes, space is sexy. Space is important. Space-based sensors and space-based communication assets can really improve our lives, but understanding the ocean is critically important as well. It is very challenging and it's very expensive.
You hear people say things like, "We know more about the surface of Mars than we do the floor of the ocean on our own planet." I once wanted to examine how true that statement was. I was hosting a panel for the American Geographical Society between Don Walsh, Bob Ballard, Sylvia Earle, and Kathryn Sullivan, four pillars in the ocean exploration community, and we had this exact discussion. I asked them, "To what degree is it true that we know more about the surface of Mars than we know about the sea floor on our own planet?" And it really depends on what you mean.
We have a bathymetry map of the whole world. There aren't places where we know nothing because of the space people. Satellite-derived gravity measurements, gravitational anomalies, the Shuttle Radar Topography Mission, because of things like that we've been able to map out with at least very poor resolution everywhere on the surface of the earth. If you want to talk about resolution that's good enough for things like seismic exploration, or understanding ecosystem parameters, or where mineral deposits are or what the bottom's made out of well enough to be able to model all the acoustics in order to do things like find submarines or whales better, then you're talking about the need for maps of the seafloor with resolution better than 10 meters, and we don't have that. Depending on who you ask, I think it's between 5 and 15% of the ocean is mapped well enough to do that, whereas the entire surface of Mars is.
By at least some very meaningful metrics, we do know more about the surface of Mars than we do about the seafloor on our own planet, and I think that's problematic. There's a lot of really exciting efforts going on and I think Seabed 2030 is one where there's a lot of academic institutions, government programs, research labs, nonprofits, and companies that have combined together to try to create a comprehensive seafloor map of the entire world by 2030. It also represents this level of international cooperation, which I think is rare and incredibly powerful. We should use that as an example of ways that we can share data more effectively on a lot of different levels, acoustic data, temperature, salinity measurements, things like that.
Kathryn Sullivan, the former director of NOAA, the first person to have walked in space and gone to the bottom of the Mariana's Trench, a low key hero of mine. She once said, "We need to API the ocean." What she meant is we need to work on data availability, data sharing, just like those folks who are working on the seafloor map for 2030, and we need to do that for all of our ocean measurements.
I think that people are seeing that sensors are becoming more affordable. The problems that we can solve by understanding the oceans better are coming into our daily lives more. Things like understanding if storms are going to get worse or better with climate change, things like knowing which fisheries are healthy, or knowing which coral reefs are going to die and which sites might be viable for coral reefs in the future. People are starting to really understand how these things map to their daily lives. As a result, we're going to see a real increase in how much people care about and think about ocean exploration and ocean sensing in the future.
Understanding Marine Ecosystems
D:
I just want to try and make one more point here about this before we move on and perhaps talk about noise pollution and what that means for the ocean. You talked about the bathymetry model that we have for the deep oceans, and you were saying it's not good enough to accurately model the acoustic environment. Early on in the conversation, we were talking about how acoustics is the way we see underwater. Is that actually a limiting factor when we think about the bathymetry model that we have today? Is that the limiting factor when we think about seeing underwater, about sensing our marine environment?
C:
Yeah it is, think it this way, all over the world people are talking about 5G communications now. 5G communications could be very wonderful, very high bandwidth comms everywhere, but it's a challenging media to work with because 5G doesn't travel through objects. Heaven forbid that there are leaves in the way, so having really incredibly accurate maps of the landscape is really important for understanding this 5G communication infrastructure.
There's been a really sharp increase in the commercial, as well as government need for very accurate maps of the surface of the planet. In order to understand what you can communicate with, what you can see with acoustics underwater, you need to be able to understand what the bathymetry looks like.
Similarly, radio propagation depends on atmospheric properties. For certain things like long-range radio communications and over-the-horizon radar, you need to understand those things intimately.
However, for knowing whether or not you're going to get Wi-Fi somewhere, you don't necessarily need to pull out your temperature sensor and measure the humidity, but you do need to do that underwater. If you want to know where you can hear a humpback whale, you put out a hydrophone say, near a shipping lane because you want to be able to warn mariners if there are a lot of whales in the area so they can slow down. Nobody wants to hit a whale. If you want to know where your hydrophone can hear whales, you need to know the bathymetry perfectly. As well as you possibly can, within a few meters. You need to know that the depth accuracy everywhere, you need to know what the bottom is made of. You need to know if it's sand, is it rock? Is there sand with a little bit of rock underneath it?
You need to know the temperature and salinity of the ocean everywhere, because temperature and salinity affect the speed of sound, and the speed of sound can cause it to refract differently and bend back up to the surface or possibly down into the bottom. Finally, you even need to know the pH because the attenuation of sound is dependent on the pH. If you really want to understand your soundshed, going back to what we talked about earlier today, you need to know as much as possible about the ocean.
There's hope that if you have hydrophones out in the ocean, you can learn more about the ocean, and that improves your models. In order to do ocean sensing, we need to know the ocean as well as we possibly can. For example, whales when they interact with their environment, they're doing it acoustically. They can't see very far underwater, so if a beluga whale wants to know where it's food is, it's finding its food with acoustics. It knows its food likes ocean environments of certain types, maybe they hang out on ocean fronts, boundaries between warm and cold water, or maybe the food likes to live above ocean bottoms made of certain properties. Whales know what the sea floor is made out of. They know where ocean fronts are. They know how to navigate. They know how to find mates. They know how to find prey. They know how to find dangers in their environment.
Whales know a lot about their environment from what they're learning acoustically. We've really only scratched the surface of what we can do with acoustics. Yeah, we can find a submarine, a giant hunk of steel underwater, but we don't know where every single ocean front is everywhere all the time. We don't know what the bottom is made out of everywhere. We spend a lot more money than beluga whales do. Beluga whales keep me up at night and they should keep you up at night too.
I helped with a study by some researchers at Woods Hole, including a gentleman named Aaron Mooney, just a really brilliant scientist who was looking at how well beluga whales could direction find. One of the things they've found is that they can determine the direction sound is coming from, something like an order of magnitude, 10 times more precisely than they should be able to given the size of their sensing organ. They can defy modern mathematics in ways that we don't understand. If we can understand that a little bit better, then we can figure out how to use ocean acoustics to sense our environment a little bit better.
We can understand the environment better for its own sake. Understanding ecosystem dynamics, marine environmental protection- If you know where coral reefs are dying, you know where to focus. If you know the viability of the ocean for certain species, you know where to concentrate your efforts. There is just an abundance of discoveries to be made that have world-changing applications. To me, that's why studying sound in the ocean is so exciting right now.
Managing Underwater Noise Pollution
D:
I'd like to move on now and talk about mapping and measuring the sound in the marine environment. When we think about sound in the marine environment, a lot of us will think about pollution, noise pollution. My question is, how do we define noise pollution? Is it simply all anthropogenic noise that is put into that environment? Is it a certain frequency over a certain time period? Could you give us a working definition of noise pollution when we think about the marine environment?
C:
If you're in the European Union, you define noise pollution as ambient noise created by humans that can be harmful to the environment defined as marine organisms or anything else. We don't define underwater sound as a pollutant in the United States as of yet. The Noise Control Act of 1972 in the United States really only defines noise pollution as it affects human beings. Noise pollution for us is defined as creating sounds that are harmful to either creating stress or physical damage to human beings. Underwater sound where there aren't a lot of human beings is not currently classified as a pollutant in the United States. And I think that's going to change very soon. And I think we've already seen the beginnings of it.
There's been a lot of development lately in offshore wind, which I think is a really wonderful here in the United States. It's existed for quite a while in much of Europe, but we're really starting to hit it hard in the United States, which I think is a really good thing. These offshore wind developers have come from Europe, where they've already had to go through the process of determining what the environmental impact of these platforms is going to be. They've already got the procedures in place. They already have done studies to see how the sound from these offshore wind platforms could impact the marine environment. When they're coming to the United States, the Bureau of Ocean Energy Management is requiring them to do that same work. Even though noise isn't technically a pollutant here in the United States, they're really treating it as one in environmental impact assessments.
If you want to put in a new pier and you're going to do pile driving, pile driving's loud. You have to be able to determine what your impact is going to be on the environment and take necessary mitigation actions. I think that's a really good thing, and it's been forward-leaning to require that of the wind companies that are doing the work, and who are already prepared to do that work.
There's a wonderful company called Jasco based out of Canada, they've got a U.S. office as well. We used to compete with them, but we got tired of losing, so now we work with them as often as possible.
They do just a great job at understanding the environmental impact of industrial activity on the habitat on marine mammals. So if we take a step back again, you said, "How do you define noise pollution?" There are legal definitions, and that's what we just talked about, but there's just working definitions as well. I think underwater noise pollution will be sound made by human beings that impacts the environment. That can be more nuanced and more complicated than it sounds on the surface.
Considering the original example I gave of sound in air, it's pretty easy to determine if that's having a negative impact on human beings. Human beings are difficult animals to work with. That's why I became a physicist, but it is possible to ask a person, "How are you feeling?". It's possible to ask them to come in for regular tests so you can evaluate their health. You can determine if regular loud noises are causing stress hormones like cortisol and aldosterone to increase in their systems. It's possible to stick a camera in their ear or whatever doctors do to determine what sort of impact you've had on their hearing.
Whales are a little more difficult to get them to come in for regular appointments where you can ask them how they're feeling. Moreover, the way they use acoustics is so much more complicated and more nuanced than the way human beings use acoustics. It's hard to know how the sound that we produce impacts them.
We use sound to hear and it's about it. We use our ears to communicate with each other. Maybe find things generally, but the whales use sound for everything. Think about this, if I put you in a room with nails on a chalkboard, just constantly going for a month, the sound of nails on a chalkboard, you would lose your mind. Your stress hormones would be elevated, you'd lose sleep, and there would be significant health impacts on you. There is no physical reason why that should be true. The elevated levels of that sound, the frequencies, there is nothing physically about that sound and how it interacts with your ears drums that should mean that that's really painful for you. The only way that I know that nails on a chalkboard are painful for some people is because we can talk to people. We can't talk to whales about that. We don't know if there's an equivalent of nails on a chalkboard for a whale, because the way that they perceive sound is so much more nuanced and so much more complex than human beings.
That being said, we can do our best. I've recently been very lucky to participate in a study in collaboration with the World Wildlife Fund, and NOAA, here in the United States and Penn, a working group on the Arctic Council to predict future noise pollution levels in the Arctic from changes in shipping traffic. We ran some really incredible acoustic models that we're very proud of. We modeled every ship everywhere in the world all the time. Every minute, every day, every month, every year in order to create statistics on what the sound has looked like over the last 10 years and what it might look like over the next 10 years. We showed our results to our partners at the World Wildlife Fund and they were like, "Cool, nerds. What does that mean?" They didn't say that, they're very nice. What we realized is we had all these statistics and decibels and charts, but it didn't mean anything because it didn't tell us how that sound might actually impact the environment.
We asked one of our lead marine biologists, Dr. Kerri Seger, if she could put together some sort of a map by species. What she did is she took all the information that we had on the spatial extents for all of these critical species, so we had maps of where they were. Then she took all the information we had about the audible range of these species as best we could tell, and integrated the sound energy on their audible bands in the area where that animal lives. We were able to start coming up with risk scores by species and region that told us how much sound energy there would be.
Of course, there are differences. A gunshot or an explosion is a really intense sound that lasts for a second, but it can irreparably damage your ears and your eardrums. Similarly, if I played really loud, really bad music at 130 decibels for a month straight and never gave you any reprieve, that would have different physiological impacts on you. You would be stressed, you wouldn't be able to communicate, you wouldn't be able to talk. If you're a whale, you wouldn't be able to navigate, find food, find mates, etc. How you quantify how bad sound is for an organism has to be classified in different senses. There's a whole set of guidelines created by NOAA here in the United States, and there are similar guidelines in Europe and all over the world that help us understand what the impact of sound is on marine mammals.
One of my favorite studies was by a group called JAMA in Northern Europe, where they wanted to understand the impact of shipping noise on marine mammals in the North Sea. One of the statistics they reported on was the percentage of the time that sound was above certain thresholds that would be either damaging or uncomfortable to marine mammals. That is really important from a stress perspective, how often can this whale find mates, etc. Maybe if a ferry crosses through that whale's habitat once every few hours, it's not a big deal because they can still find mates, food, and communicate in between trips. If it's all the time though, that really matters a lot. So, to your original question, how do we define sound pollution? Well, it's man-made sound that affects the environment. But how do we decide what the impact of that pollution is on the environment? I think there's a lot of research that still needs to be done.
Bubbles as Sound Barriers
D:
I used to work for a company that was building offshore wind packs, and they were experimenting with this idea of using a bubble net, blowing bubbles up from the seafloor to protect the area that they were working in from letting too much noise escape from the point source. When you think about the future, is there a world where we noise proof our shipping lanes, noise proof our other activities? Can we set up barriers in the ocean? Can we use bubble nets? Is there any way to protect the marine environment from the noise that we are creating?
C:
We just submitted a grant this week to see about the viability of doing bubble curtains around ships. It's a pretty far-out idea. I would say a lot of experts in the field have said it would be very challenging. First, why might it work? Well, if you're doing pile-driving and let's say, you've determined that it's going to be really loud. You've determined that there's a lot of marine mammals in the area. You've done your homework and your due diligence and you don't want to impact those marine mammals. One of your options is to essentially lay hosing down on the bottom of the ocean. It's more complicated than this, but basically, it's a tube with a bunch of holes poked in it that you pump air through. It emits these big strings and curtains of bubbles around the pile-driving thing, and that blocks the sound. You might think to yourself, "Well, that seems silly. Why do bubbles block sound?".
If you've ever been in a swimming pool, could you hear the people talking above the swimming pool? Probably not. Similarly, if you were in a swimming pool and made a bunch of noise, the people outside the pool couldn't really hear you. That's because the pressure difference, density difference, and speed of sound difference between water and air is so great that sound waves see that boundary between air and water as almost like a perfect reflection surface. Very little energy can travel from water to air and vice versa. If you start putting all these little bubbles in the ocean, you create a wall where the sound reflects off that wall and it can't propagate through it. So in theory, bubble curtains can be a really effective mitigation strategy.
There's a researcher at The National Oceanography Center (NOC) in Southampton named Tim Leighton. We call him the bubble king. He wrote an 800-page book called The Acoustic Bubble, about the acoustics of bubbles. They are indescribably important. Bubble acoustics can cause sound to be trapped on the surface. It affects underwater communications, it affects everything. Bubble acoustics is an entire discipline of study that is really important for reasons like this.
To your question, can we just soundproof entire shipping lanes? Most people would just flat out say, "No," because it's going to be too expensive, shipping lanes are massive. But maybe just in places where it really matters. Maybe just in places where you have a shipping lane going through a critical habitat. Look at the Santa Barbara channel in the United States. LA Long Beach is the highest volume port in the United States, and for a lot of vessels, the most efficient route is to go through the Santa Barbara channel. It is a habitat for a ton of highly endangered marine mammals and other species that are sensitive to sound. If you're in a ship that operates often in areas where species might be really sensitive to sound, could you inject a bubble curtain around the whole of your ship? It would be very challenging, there are a lot of significant engineering challenges to be able to do something like that.
Icebreakers actually inject bubbles along the whole of their ship to reduce the friction with the ice. So there are systems, albeit a lot less complex, that do things that are similar. I don't think it's impossible, and there might be other ways to reduce the sound from ships. There are really good engineers out there who specialize in making things quiet. You would never accept a car that was so loud that you could never hear anybody while you were inside of it. That's because really smart engineers tried very hard to make those things quieter. I think there's probably a lot of gains to be had just purely in engineering of ships.
I forget the name of the company, but some folks have designed these sort of... They're not springs, but basically shock absorbers that you mount to things like generators and engines. That decouples the vibrations from the machinery with the whole of your ship, which prevents some of the acoustic emissions. There's a lot more that can be done. The Navy's been trying to make submarines quieter and quieter for going on 80 years now. Taking some of that engineering and applying it to commercial shipping, it'd be expensive, but I think there's a lot more work to be done there. As more and more people start to think of sound as a pollutant, there are going to be requirements that come out that make us take some of those engineering steps.
D:
We have come a long way in this conversation and I'm really enjoying it. This has been absolutely great. You're working on a ton of different things, you're exposed to a ton of different ideas. What's the thing out there that's got you most excited for this year and maybe for the next five years?
C:
Oh, man, that's a tough one. You might be able to tell, I get very excited about my work. I love oceanography and I love ocean acoustics for all the reasons we talked about. There are so many unsolved problems that are really, really important.
D:
Let me just stop you there. You talk about unsolved problems that are really important, and it sounds like you're excited by this. You're not overwhelmed by the challenges, you're excited by these unsolved problems?
C:
Yeah, of course I'm excited. I don't know about you, but if I do something that I think is impactful, it makes me happy, it makes me think I made a difference. If I can make tiny little advances in our understanding of underwater sound and of the ocean, I think that there are huge benefits to society and to the world. There are so many problems that I could contribute in a small way to solving that can make the world a better place, and that can help us understand how to protect whales better, help national defense, that can help us address climate change. The amount of unsolved problems that are solvable in our time and the impact those solutions would have to humanity is what gets me up in the morning. So to your question, there are two projects that have me most excited right now, one of them, because it's just so darn cool, and the other one, because it's so darn hard.
The one that is cool is the project I told you about already with the World Wildlife Fund and the Artic Council and NOAA. First of all, our government program manager is wonderful. She is a solving hard problem, she's pushing us hard to really do the best job possible to understand how shipping noise in the Arctic is likely to affect Arctic ecosystems in the future. We have just gone all out. We've taken the most sophisticated acoustic propagation models we've ever made and analyzed and optimized them for all sorts of computing architectures, to be able to simulate every ship, everywhere, all the time.
We've brought in marine biologists, psychologists, physicists, and atmospheric scientists to work with the World Wildlife Fund. The work we're doing is good. We've gotten to work with world experts in everything from economics who will drive how shipping will change, to some of the best ecologists and marine biologists there are, and department of transportation people who just get shipping traffic. The work is good, and it will be delivered by the World Wildlife Fund and these Arctic Council people to decision makers who will get to look at these maps of, "Oh, there's a shipping lane here and there's a critical habitat for this whale there." Maybe if we just moved the shipping lane, or maybe if we made this regulation or that regulation, it would have this huge impact on the future of conservation in that really critical environment. I think the combination of how good the work is, the fun people we get to work with, the potential impact the study has, makes it all really exciting to me.
The second project that has me really excited is one we've alluded to, it's a basic research program funded by the Office of Naval Research here in the United States. It's about the information content of ocean noise. It was inspired by work done by my PhD advisor, Dr. Bill Cooperman at Scripps a few years ago where we're trying very hard to understand everything you can learn about the ocean, from sound that's already in the ocean. Be it from ships, be it from whales, regardless of the source. How can we better understand what we can learn about the ocean from underwater sound?
I got a chance to collaborate with brilliant scientists at Scripps, at Georgia Tech and Woods Hole and Portland State, University of Washington, and in my own company, at Applied Ocean Sciences and it's so darn hard, the math, the work you need to do to try to synchronize all your clocks and position all your ships and your sensors and parameterize the ocean in really clever ways to reduce that mathematical search space. It's so darn hard and therefore very rewarding when we make progress. So yeah, those are the two most exciting things I'm working on, the ocean conservation, noise pollution stuff in the Arctic, because of the impact and the information content of ocean noise stuff for the Navy, because it's so hard and so interesting, and I also do think critically important.
D:
Chris, I really want to thank you for your time. I have thoroughly enjoyed talking with you. You're brilliant, you're enthusiastic, and you're human. This has been a really inspirational conversation for me. So I really appreciate it. Let’s say somebody else is listening to this and they think, "Well, who is this guy? How do I get in touch with him? How can I reach out to him?" Where would they go to do that?
C:
If anybody after this still to hear more, so the Applied Ocean Sciences website has contact info. It's appliedoceansciences.com. My email is chris.verlinden@appliedoceansciences.com. I'm always happy to talk acoustics with anyone and everyone. Feel free to cold call me or email me, I love talking about this stuff.
Our guest today is Danny Arribas-Bel, PhD. He is the Senior Lecturer of Data Science at the University of Liverpool, and Deputy Programme Director for Urban Analytics at the Alan Turing Institute. Although his focus now is in urban planning and development, he began his academic pursuits with a PhD in economics. Building this background in the economic driving factors behind cities and communities has allowed Danny valuable insight and context into why we see certain patterns in these spaces. Through the Alan Turing Institute, Danny contributes to the Urban Grammar project, a spatial data science project which aims to classify the world’s cities based on the form and function of their unique organizations.
What is Urban Grammar?Looking across the world, and across time, we have seen many cities rise and fall. While none of these cities are close to identical, they share an intriguing number of similarities in how they are built, populated, and utilized by their residents. Those who study the urban sciences, from planning to architecture to economics, have great interest in being able to quantify, qualify, and compare these places across the dimensions of time and space. This is where the Alan Turing Institute’s Urban Grammar project comes in.
In the same way that we can break down the grammar and syntax of a language, we can begin to break down the characteristics of cities. Before this can happen, however, there needs to be a more or less consistent dataset to pull from for analysis. Considering the characteristics of cities and their residents are constantly changing, it is a huge challenge to find data with the temporal granularity necessary to be useful to track changes.
The best and most complete data for Urban Grammar’s purposes are sources like the Census, or Ordnance Survey data. Official sources like this are rich with attributes, and reliable and consistent in their collection methods. The issue, of course, is that Census data is only collected every 10 years. In order to fill in some of the temporal gaps, researchers utilize spatial data generated from satellite imagery. This may take the form of crowdsourced OpenStreetMap datasets, which may come with built in attributes, or datasets generated through earth observation and artificial intelligence workflows.
Artificial intelligence (AI) practices like deep learning and machine learning are invaluable to Urban Grammar’s mission to make data available to study spatial changes in our urban landscapes. GIS workflows using AI generally consist of collecting and labeling a large amount of imagery in order to create training data samples. These samples are then fed into an algorithm to train it to begin to generate future samples and data on its own when provided similar imagery.
Again, the biggest challenge here is the availability of satellite imagery at a temporal scale that is useful to track change, and of high enough quality to perform deep learning workflows. Generally, this can be combated by combining data from many sources, including government provided satellite imagery, Census and Ordnance Survey data, and crowdsourced geospatial data.
Data Classification and SignaturesOnce the necessary data has been aggregated, it is time to move into the equally large task of analysis. As mentioned, analysis at the scale necessary to understand entire cities requires deep learning and artificial intelligence workflows in order to match the pace at which the underlying data becomes available. These workflows require, and have the ultimate goal of classifying data.
Considering the diversity of study areas researchers deal with, how are these classes created?
Well, first and foremost, Urban Grammer approaches categorizations of cities through the framework of what they call signatures. Signatures are the resulting classes of analyzing the form and function of different spaces using concepts related to morphometrics. Form in this context is what does the space “look like”. This can be the building footprints, street network, or significant natural features revealed in imagery. Essentially, if you were learning a language and viewing flashcards with pictures of different types of buildings and features, form would be what the structure “is”.
Function is the other side of the coin here. This is what the structure “does”. For example, while the form of a house is a building, the function of the house is a living space. The form of a highway may be a street, but the function of that street is transportation. Oftentimes, form and function go hand in hand, one follows the other. Form and function can be distilled into signatures through a plug and play sort of process using AI algorithms, fine tuning it for optimal I/O.
Considering urban spaces are unique and different from each other, the inputs and outputs of these algorithms will be similarly unique, and likely will not result in uniform classifications across the world. This is where humans enter the system, providing checks and “ground truthing” on the resulting data by adding the context and knowledge of human and spatial sciences.
Applying the Sciences to Spatial DataUnderstanding the function and form of the world’s cities is not a new concept. This has been a goal, direct or indirect, of many disciplines including architecture, urban planning and design, civil engineering, political science, archeology, GIS, history, etc.
If you have ever visited ruins on vacation, you have likely found yourself theorizing the day to day uses of different spaces on site through your modern lens. This can be a fun conversation starter with your traveling companions, and you may find yourself getting a bit creative with assigning applications to the space only to be surprised by an informational board that provides some correcting historical context.
Every dataset acts as a snapshot in time. Maps themselves become outdated as soon as they are put into print. By collecting, categorizing, and documenting datasets over time, researchers can build more nuanced pictures of the past, present, and future. A large part of this process for modern scientists is writing and utilizing code. Where once upon a time physicists used mathematical equations and the written word to document processes, now, researchers find code and the algorithms themselves to function as reusable and portable documentation for their research.
The building blocks of cities are pretty similar all around. Streets are streets and buildings are buildings. Every society has the same basic needs, food, lodging, work, recreation, etc. For this reason, AI practices like transfer learning, which promotes interoperability of algorithms by reusing the most base levels of code, then customizing it based on needs, are growing in popularity amongst data scientists.
Of course, these algorithms need to be adapted based on location to accommodate for differences in source data, and the technical and cultural variations inherently seen across our world. Adding regional context may add some steps to the process, but ultimately it results in richer, more relevant data for the area. Data may be beautiful, but it hardly exists just for the sake of looking pretty. We need data to make decisions. More locally relevant data results in better policy, and resource management decisions by those in charge. It allows for more nuanced predictions for the future, and a more acutely critical lens of the past.
A note from the author: If you are interested in learning more about how form and function have changed over time at the scale of day-to-day life, Bill Bryson’s book “At Home: A Short History of Private Life” does an excellent job of deconstructing the changes in use of our built world.
